Pages

Extract field name and values from Java Objects

 Suppose you have to write some generic utility which should extract the field name and values from Java DTO. 


Problem: There is a tracing library where you need to add a few critical fields and their values which will make the debugging very easier in any environment. Say, Jaeger.

Jaeger is a library which helps trace various class level calls in a microservices environment.

How would you convert the fields into Key value structure:

public static Map<String, String> extractFields(Object obj) {
    Map<String, String> map = new HashMap<>();

    if(obj == null) return map;

    Class<?> cls = obj.getClass();
    for(Field field: cls.getDeclaredFields()) {
        String name = field.getName();
        if("String".equals(field.getType().getSimpleName())) {
            field.setAccessible(true);
            try{
                String val = (String) field.get(obj);
                if(val != null) {
                    map.put(name, val);
                 }
            } catch(IllegalAccessException e) {
                Logger.error("Error extracting {} field", field);
            }
        }
    }
    return map;
}

Binary Search and Gotchas

 Binary search is an important algorithm because it helps find certain element's position fairly quickly even when there are vary large number of elements.

Given a sorted integer array, and a target, find the index of target in the array. If its not present, return -1.

Say there might be duplicates in the array. Return the leftmost index of target in the array.

    int findLeft(int[] nums, int target) {
        if(nums.length==0) return -1;
        if(nums.length==1) return nums[0]==target ? 0 : -1;
        
        int left = 0, right = nums.length-1;
        while(left < right) {
            int mid = left + (right-left)/2;
            
            if(nums[mid] > target) {
                right = mid-1;
            } else if(nums[mid] < target){
                left = mid + 1;
            } else {
                right = mid;
            }
        }
        return nums[left] == target ? left : -1;
    }
    
  
To find the left most index, the above code will work fine. There are some number properties working in the background to make the above code successful.

The mid is calculated dividing the diff of right and left by 2. What if the difference is 1. In that case the mid is same as left. But in this case, right is assigned to mid, so the algorithm will terminate in the next step.

What if you have to find the right most index of the target element in the array.
    int findRight(int[] nums, int target) {
        if(nums.length==0) return -1;
        if(nums.length==1) return nums[0]==target ? 0 : -1;
        
        int left = 0, right = nums.length-1;
        while(left < right) {
            int mid = left + (right-left)/2;
            
            if(nums[mid] > target) {
                right = mid-1;
            } else if(nums[mid] < target){
                left = mid+1  ;
            } else {
                left = mid;
            }
        }
        return nums[right] == target ? right : -1;
    }
Do you think the above code will work? Why or Why not?
All the calculations are same as the previous code except, when the mid and target are equal, you are assigning mid to left. This is because there may be other duplicated on the right of mid which you intend to search. But if there's not, you still want the right to be able to come to left before terminating.

But what about the calculation of mid? say the diff between right and left is 1 at some step. Then the mid will be same as left. Then in action, you again assign left to mid. So there will be an infinite loop.

How to solve this issue?
How about adding 1 to the diff of right and left before dividing it by 2. Do you think that has any adverse effect in the result?
Why or Why not?
    int findRight(int[] nums, int target) {
        if(nums.length==0) return -1;
        if(nums.length==1) return nums[0]==target ? 0 : -1;
        
        int left = 0, right = nums.length-1;
        while(left < right) {
            int mid = left + (right-left+1)/2;
            
            if(nums[mid] > target) {
                right = mid-1;
            } else if(nums[mid] < target){
                left = mid+1  ;
            } else {
                left = mid;
            }
        }
        return nums[right] == target ? right : -1;
    }
For the intermediate mids, this algorithm will pick the right one among the two middles. But the overall result will be correct and this algorithm will terminate as well. Why?

Because when diff of right and left is 1, mid will be left + 1, which would be larger than target (if left is the rightmost target). In this case, right will decrease to left and the loop terminates in the next step.

Query DSL - start

 build.gradle

ext {
javaGeneratedSources = file("build/classes/generated")
}

idea {
module {
outputDir file('build/classes')
sourceDirs += javaGeneratedSources
generatedSourceDirs += javaGeneratedSources
}
}

compileJava {
doFirst {
javaGeneratedSources.mkdirs()
}
options.compilerArgs += [
'-parameters', '-s', javaGeneratedSources
]
}

// QueryDSL
implementation "com.querydsl:querydsl-jpa:4.2.2"
annotationProcessor("com.querydsl:querydsl-apt:4.2.2:jpa")
annotationProcessor("org.springframework.boot:spring-boot-starter-data-jpa")


Repository.java class
extend the Repository with QueryDSL repository interfaces
public interface SomeRepository extends
JpaRepository<SomeEntity, Long>, QuerydslPredicateExecutor<SomeEntity>, QuerydslBinderCustomizer<QSomeEntity>
create a method
@Override
default void customize(QuerydslBindings bindings, QSomeEntity root) {
bindings.bind(String.class).first((SingleValueBinding<StringPath, String>) StringExpression::containsIgnoreCase);
}

Neo4j Graph Database | Design tips

One thing I learned from working using Neo4j graph database is that we need to avoid replication as much as possible in graph database. Replication could lead to a log of inconsistency and could be the reason of a lot of problems.(or maintenance issue).

We were using graph database to represent the logical progression of states of let's say a smart phone. The phone is in certain state, which means, the phone has certain set of module hardwares, and those hardwares are running certain set of softwares to help us properly use the smartphone.

What I learned from deploying my Node application in AWS cloud

Well something needs to be done in the time of Corona (NCovid 19) virus. The schools are shutdown, and the work offices are shutting down as well. I has some free time and so I started getting restless as I always do.

Long story short, I has an idea for a simple application which I started building and then ended up creating the basic functionality over a 3 day weekend. The feature was pretty straight forward. It is a live story pointing application where a bunch of people would join and point stories based on their complexity. And I deployed it on an Aws application over my own sub-domain. Now other questions starts coming.

  1. How to make it more usable?
  2. How to add additional features?
  3. How to make is secure?
  4. What if the server restarts?
  5. How to make it scale if all the people in the whole world start using it?
  6. How much am I going to be billed?
  7. How to make it Https?

1. How to make it more usable?

Well there some work required to make an application work, but there's a lot and lot more work required to make an application more usable. I was able to develop a working application in a few hours(4-5 hrs). But to make it simple usable took the other two days (10-15 hrs). There's just a lot of scenarios that could occur in the real world. And I was able to think about a few of them. Rapid testing and fixing the bugs that could occur was the way to go. Then there was work to be done about the looks and feel of the application. I have never been a proper UI-designer so this is always a struggle for me. However, I was able to complete it using some of my previous designs. 

2. How to add additional features?

This was one of the important question in my mind. The first MVP product I have developed was using simple nodejs server and html and javascript. There's a lot of work that you need to do in the plain languages and if this project has to go anywhere, I need to use some kind of frameworks. There are frameworks for everything nowadays. Now I have to decide the backend framework, and then the css framework, and the frontend application framework etc. Then there is the effort required to port the application to the framework. Its a decent amount of work and there's a significant learning curve.
I ended up using express for the backend and bootstrap for css and React for the frontend .

3. How to make it secure?

Security was not a huge deal for this application. It does not have any sensitive information nor it deals with any kind of financial transaction. So it was not a big deal. But there is always a thinking about what if someone able to hack my system and application. What if someone is able to get into the ec2 instance, or shut down the application etc. Well, I did what I could to make it secure.

4. What if the server restarts?

Everybody should think about this point if they are deploying their application in the cloud. In the cloud, based on the resource contention, the server can restart any time. If that happens, you have to make sure, your application re-starts and is automatically available on the web. Previously I was using a linux tool called screen to run the application in a background shell. This would not have worked.
I found pm2 instead, which runs on background and can starts when the server restarts. I still need to figure out how to see the logs though.
# to keep the process running even after a server restart
npm i -g pm2
pm2 start --name "app-name" npm -- start
pm2 startup # to make sure pm2 starts when the server starts
pm2 save
pm2 ls
# to stop and delete a running instance
pm2 stop app-name
pm2 delete app-name

 5. How to make it scale if all the people in the world start using it?

This question runs on the back of my brain all the time. What would happen if a lot of people start using it. Right now I am running my node application on an ec2 micro instance with 1 GB or ram and 8 GB of memory. I choose Node because it has very less memory footprint and can serve huge number of requests rapidly. It's not good for long running tasks which this application is not. I do store some information in the server but clean it regularly and have gone about this strictly couple of iterations over my code. So I am pretty confident that my micro instance can handle significant amount of load. Also, I can start utilizing the 8 GB disk space that I have instead of relying on the memory which will make it slightly slow but I think overall it will help handle some more data. One thing I am not sure is how much of the parallel request (TCP connection) can the ec2 instance handle at a time. I believe that could be a hard limit and I have to go to other options of Auto scaling and so on. But let's see if I ever reach there.

6. How much am I going to be billed for this?

This is always one of a major concern. But I started feeling this way. If I am always worried about the bill, I will never do anything cool with my life. Beside, I cannot be billed like thousands of dollar for this application. I am still eligible for free tier and if the resources consumption starts increasing because of load, there will always come some way to pay for it. So I am calm about this. However I am ready to pay like a hundred dollars a month or so.

7. How to make it HTTPS?

Now a days, https is must for any web application. Even google will not rank you properly if you don't have https for your site. Because of this, I had to make my application use a certificate. There were other problems also while running the application on ec2. It was not allowing my application to use PORT 80 or 443 which are the default ports for http and https. So I had to use a Nginx server to route my http and https traffic to my local port on which the application was running on ec2.
# install nginx
sudo apt-get install nginx
# edit the config
sudo vi /etc/nginx/nginx.conf
# auto restart nginx on system restart
sudo systemctl enable nginx
-- to start nginx 
sudo systemctl start nginx.service
sudo systemctl stop nginx.service

## route traffic in port 80 to local 8080
sudo vim /etc/nginx/nginx.conf
server {
  listen 80;
  server_name myservername;
  if ($http_x_forwarded_proto = 'http'){
       return 301 https://$host$request_uri;
  }
  # you can add this in the sites-available/default file as well
  location / {
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  Host       $http_host;
    proxy_pass        http://127.0.0.1:3000;
  }
}
# restart again
sudo systemctl restart nginx.service

Now if the security group in your EC2 has just 80 port open, will still work.
Now researched and found that I could use aws certificate manager and configure certificate on Elastic load balancer and point the load balancer to this ec2 instance. Then I can point my domain to the elastic load balancer. Now my application is more safe, secure and load balanced as well.


Liquigraph as a data migration tool for Neo4j

I have been working a graph database recently called Neo4j. Its interesting. Since its a graph database, you have to think differently. And so, it might be intimidating for those who are used to the sql databases. Especially because they are familiar with how it works and all the tools required to work with it. I was particularly focused on Application database migration tools.

Directly to the point:
There are a couple of tools for automatic data migration for sql databases

  • Liquibase
  • Flyway
But since we are also using Neo4j (graph database), above migration tools won't work.
Well I found a tool called 
  • LIQUIGRAPH
Its basically similar to Liquibase. 

How it works?

Basically most of the getting started tutorial is here. 
  • Add the dependency in the spring project. 
  • Configure the datasource. (Tricky is multiple databases)
  • Add migration scripts in changelog.xml at your resources/db/liquigraph directory
Now the new changes logged in the changelog.xml will be applied to the graph data. 

Will it apply the change on every application start?

No. It creates some nodes in your graph database (_liquigraphchangelog, _liquigraphchangeset, _liquigraphquery). These are to track the previous migrations. Liquigraph knows which changes are already applied to the graph and which are the new changes, and applies only the new changes.

What are the dependencies?

Add the two dependencies

implementation ('org.liquigraph:liquigraph-spring-boot-starter:3.1.1')
implementation ('org.neo4j:neo4j-jdbc-bolt:3.5.1')

Add the properties in the properties file

liquigraph:
  user: neo4j
  password: pswd
  url: jdbc:neo4j:bolt://localhost:7687

Put the changes in the changelog.xml
<?xml version="1.0" encoding="UTF-8"?><changelog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"           xsi:noNamespaceSchemaLocation="http://www.liquigraph.org/schema/1.0/liquigraph.xsd">
    <changeset id="hello-world" author="you">
        <query>CREATE (n:Sentence {text:'Hello monde!'}) RETURN n</query>
    </changeset>
    <changeset id="hello-world-fixed" author="you">
        <query>MATCH (n:Sentence {text:'Hello monde!'}) SET n.text='Hello world!' RETURN n</query>
    </changeset>
    <changeset id="set-author-in-sentence" author="you">
        <query>match(n: Sentence) set n.author= 'rajan' return count(n)</query>
    </changeset>
</changelog>

Now if these changes are not already applied in the graph, They will be applied at the application restart.

Problems/Confusions 

  • I was confused about specifying the liquigraph properties explicitly. I thought since I have already configured the spring.data.neo4j properties, it would pick those properties to configure the Datasource. But it does not.
  • The tutorial tells you to put spring-data-jdbc to the dependencies. I was afraid that it would mess with spring-data-jpa that I had in my project. So the second way of creating a Datasource manually won't work either. There were miltiple databases in my project (MSSQL, Mongodb, Neo4j). So faced some issue as to which Datasource the Liquigraph was picking. I had to dig inside the library to find out that the liquigraph property could be the solution, and it worked.
  • Pick proper version of the libraries