Java 8 Features | stream processing and lambda expressions

There are lots of programming languages in the world today, and some of them are quiet popular. Each language has their own features, syntax and rules. There is one method of programming called functional programming, where you declare what do you need rather then how do you do it. Lots of languages has these features, for example python, bash, c#. In spite of being one of the most popular languages, java has been behind in putting stream processing and lambda expressions into itself for almost four years and finally it did it.

Below are some of the code examples of stream processing and lambda expressions combined that, if you were unfamiliar with it, you will be blown away.

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.function.BiFunction;
import java.util.stream.Collectors;

public class BiFunctionExample {

 public static void main(String[] args) {
  biFunctionStringConcat();
  biFunctionExample();
  byFunctionFilterExample();
 }
 public static void byFunctionFilterExample(){
  List<Double> lst = new ArrayList<Double>();
  lst.add(2d);
  lst.add(200d);
  lst.add(6d);
  lst.add(60d);
  lst.add(122d);
  lst.add(876d);
  lst.add(738d);
  lst.add(333d);
  lst.add(112d);
  lst.add(98d);
  lst.add(0d);
  lst.add(3d);
  lst.add(1d);
  lst.add(6666d);
  
  List<Double> resultList = lst.stream()
   .sorted()
   .filter( x -> x > 20d )
   .collect(Collectors.toList());
  
  Double sumResultList = resultList.stream().reduce(0d,  Double::sum);
  Double max = resultList.stream().max(Comparator.comparing(x -> x)).get();
  String plusItems = resultList.stream()
    .map(x -> x.toString())
    .reduce((x, y) -> x + " + " + y)
    .get();
  
  System.out.println(resultList);
  System.out.println(plusItems);
  System.out.println("Sum = " + sumResultList);
  System.out.println("max = " + max);
 }

 public static void biFunctionExample() {
  BiFunction<Double, Double, List<Double>> bi = (x,y) -> {
   List<Double> list = new ArrayList<Double>();
   list.add(Math.pow(x, y));
   list.add(x * y);
   return list;
  };
  
  System.out.println(bi.apply(2d, 3d));
 }
 
 public static void biFunctionStringConcat(){
  BiFunction<String, String,String> bi = (x, y) -> {      
        return x + y;
      };

      System.out.println(bi.apply("rajan", " upadhyay"));
 }
}

Enjoy!!

Java Serialize and DeSerialize | Json

I was trying to persist a complex object in java and re-use it later after re-running the program. Java supports traditional Serializable interface which is also cool. But it was rather looking for a better way to do it and I found this awesone API called json-io. You could also download the jar file from its maven repository, include it in the classpath and use it. I am writing a simple code to demonstrate it for my reference but you could always go to its site and learn more.

Well, I should mention this. I was trying to implement a desktop application which has some settings. The application should save its settings and perform according to them. The settings should persist, the next time the application is started. I could simply create a map, for the options, update the maps in run time and save them in a file on exit and reload from the same file.

import com.cedarsoftware.util.io.JsonReader;
import com.cedarsoftware.util.io.JsonWriter;
import java.io.IOException;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author Rajan Prasad Upadhyay
 */
public class Dog {
    Barks bark = new Barks();
    String name ;

    public Barks getBark() {
        return bark;
    }

    public void setBark(Barks bark) {
        this.bark = bark;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
    
    public static void main(String[] args) {
        
        String json = "";
        
        
        
        try {
            Dog dog = new Dog();
            dog.setName("Tommy");
            json = JsonWriter.objectToJson(dog);
            System.out.println(json);
        } catch (IOException ex) {
            Logger.getLogger(Dog.class.getName()).log(Level.SEVERE, null, ex);
        }
        
        try{
            String sjon = json;
            Dog dg = (Dog) JsonReader.jsonToJava(sjon);
            System.out.println(dg.getName() + " sounds like "+ dg.getBark().getSound());
            
        }catch (Exception ex) {
            Logger.getLogger(Dog.class.getName()).log(Level.SEVERE, null, ex);
        }
        
    }
}

class Barks{
    String sound = "woo woo";

    public String getSound() {
        return sound;
    }

    public void setSound(String sound) {
        this.sound = sound;
    }
    
    
}

How to read text content from a url | java

I was involved in this project where the task was to grab the contents from a url, and then to do other stuffs on that content. Other things will be discussed later for this post is dedicated to grabbing the text content of the URL. Initially, we implemented our own logic to do this task, but later I found this amazing API, of which I am going to give an introduction.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author Rajan Prasad Upadhyay
 */
public class URLReader {
    public static String getUrlContent(String url){
        StringBuilder builder = new StringBuilder();
        try {
            URL uri = new URL(url);
            BufferedReader in = new BufferedReader(new InputStreamReader(uri.openStream()));
            String line;
            while((line = in.readLine()) != null){
                builder.append(line);
                System.out.println(line);
            }
            in.close();
        } catch (MalformedURLException ex) {
            Logger.getLogger(URLReader.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException ex) {
            Logger.getLogger(URLReader.class.getName()).log(Level.SEVERE, null, ex);
        }
        return builder.toString();
    }
    
    public static String htmlToTextFilter1(String rawHtml){
        
        String text = rawHtml.replaceAll("\\<.*?>","");
        
        return text;
    }
    
    public static void main(String[] args) {
        String rawHtml = getUrlContent("http://rajanpupa.blogspot.com/");
        
        System.out.println(htmlToTextFilter1(rawHtml));
    }
}

There are better ways to grab texts from the web. Rather then implementing your own code/logic, you could simply use standard API's specifically designed for these kind of purpose. I used opensource api JSoup for the same task and its performance and accuracy is absolutely amazing.

Download the JSoup library from here, and import it in your project.
Here is a snapshot of the code.

     try {
            Document doc = Jsoup.connect("http://rajanpupa.blogspot.com").get();
            String title = doc.title();
            System.out.println(title);
            System.out.println(doc.text());
        } catch (IOException ex) {
            Logger.getLogger(JSourURLReader.class.getName()).log(Level.SEVERE, null, ex);
        }

Efficient way to build strings | Java

In java, the string is a immutable data type. Immutable is something that could not be modified once it is created. There are immutable data objects in other languages also, eg tuples in python.

Now lets get to the point.
I used to do the straight way to concat strings in java with the += operator. Even though I have to build a string using long algorithms. I even used the same way for my document parser in the previous, which I may soon re-implement. The point is that, every time I add another character to a string, the original string is destroyed, and an entirely string is created with the character added in the original string. Now the overhead of object creation may not be huge for a few string concatenation but say if our algorithm is such that we are performing 50 or 100 or thousands of strings additions or subtractions, then it has a huge impact. The solution is to use the StringBuilder object.

What this class does is, it provide append, delete kind of methods. It does not create new string objects on every append or delete and thus the object creation overhead is removed. It creates an string object only when called the toString() method.

Here is an example.

        StringBuilder sb = new StringBuilder();
        sb.append("rajana");
        sb.deleteCharAt(sb.length()-1);
        sb.append(" upadhyay");
        System.out.println(sb.toString());

Text Analysis | Document parser

Long back, I was involved in a project which deals with Text Analysis. It was a preety interesting project which I realized later, when the project was actually completed. Anyways, recently I was surfing odesk to see if there is something I could make a few bucks easily. I found an assignment project where the basic requirement was to be able to parse a text file, and generate a report of number of unique words, their counts etc. I did not get hired for the job, buy I decided to post my work here.

Backthen, the problem was actually to group news article. There are many sources of news in the internet. Our idea was to channel these pieces of articles into our application, run some analyzing and scoring algorithms, make a cluster of related news, and to present the related news in a more suggestive way according to the users interest.

package testapplications;

import java.io.IOException;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;
import java.util.StringTokenizer;
import java.util.Vector;

/**
 *
 * @author Rajan Prasad Upadhyay
 */
/*
 * This class holds one piece of text document. It splits the text into 
 * words along with their count, so that distinct words could be easily found 
 * out. used for indexing, and calculating similarity between documents in collection
 * class
 */
public class Document {

    /*
     * This class is made as a data-structure for holding the words along with their
     * counts in the text when the text is splitted into words in vector.
     */
    private class StringCounter {

        public String word = "";
        public int count = 1;

        public StringCounter(String str) {
            word = str;
        }

        public void increaseCount() {
            count++;
        }

    }

    String text;
    public Vector<String> words = new Vector();
    public Vector<StringCounter> count = new Vector();

    public void sentenceSplitter(String speech) {
        /*default found in java, not used here*/
        /*This will break the text into tokens, words*/
        //String speech = "My name is Rajan Prasad Upadhyay.I am twenty-two years old.";
        StringTokenizer st = new StringTokenizer(speech, ".");
        System.out.println(st.countTokens());
        //st.nextToken(".");
        while (st.hasMoreTokens()) {
            System.out.println(st.nextToken());
        }
    }

    public Document(List<String> lines) {
        text = "";
        for (String str : lines) {
            text += str;
        }
        parse();
        //System.out.println(text);
    }

    public Document(String text) {
        this.text = text;
        parse();
    }

    public Document() {
    }//never use like this
    /*internal operations methods not needed outside*/

    private boolean isValid(char c) {
        //
        if (c == '.' || c == '\'' || c == ',' || c == '?' || c == '!' || c == ':' || c == '\t' || c == '\n') {
            return false;
        } else {
            return true;
        }
    }

    private void parse() {
        /*split the text into words and store in vector*/
        String word = "";
        char c;
        int i = 0;
        int n = text.length();
        while (i < n) {
            c = text.charAt(i++);
            //System.out.print(c);
            if (c != ' ' && isValid(c)) {
                word += c;
                if (i == n) {
                    this.insertWord(word);
                    word = "";
                }
                continue;
            } else {
                if (word.length() > 0) {
                    //System.out.println(word);
                    this.insertWord(word);
                    word = "";
                } else {
                    //ie word.length=0
                    continue;
                }
            }//end else
        }//while

    }
    /*
     * These methods may be needed outside 
     */

    public void insertWord(String word) {
        /*
         * if word is not present in the vector then insert it into the vector
         * and allocate count 1 , otherwise increase the count associated with 
         * the word by 1.
         */
        word = word.trim().toLowerCase();
        //if you want case sensitive, comment the above line
        if (!words.contains(word)) {
            words.add(word);
            count.add(new StringCounter(word));
        } else {
            count.elementAt(words.indexOf(word)).increaseCount();
        }
    }

    public void printWeighedWords() {
        /*
         * This is just a debugging method
         */
        int max = getDistinctWordsCount();
        for (int i = 0; i < max; i++) {
            System.out.println(words.elementAt(i) + " :" + count.elementAt(i).count);
        }
    }

    public int getDistinctWordsCount() {
        /*
         * returns the total number of distinct words
         * ie. if the word "rain" is repeated many times
         * it returns as only one word
         */
        return words.size();
    }

    public int getTotalWordsCount() {
        /*
         * returns total number of words, counts
         * same word multiple times if it is repeated 
         * multiple times
         */
        int i = 0;
        for (int j = 0; j < count.size(); j++) {
            i += count.elementAt(j).count;
        }
        return i;
    }

    public int getCountOfTerm(String term) {
        /*
         * returns the count of term if it is present
         * otherwise returns zero
         */
        int ind = words.indexOf(term);
        if (ind > 0) {
            return count.elementAt(ind).count;
        } else {
            return 0;
        }
    }

    public String toString() {
        String doc = "[";
        int max = getDistinctWordsCount();
        for (int i = 0; i < max; i++) {
            doc += words.elementAt(i) + "(" + count.elementAt(i).count + ")";
            if (i < max - 1) {
                doc += ", ";
            }
        }
        doc += "]";
        return doc;

    }

    public static void main(String[] args) throws IOException {
       //Document doc = parseFile("assignment1.txt");

        Document doc = new Document("Hello World, My name is Rajan. and Hello world, "
                + "I enjoy doing this kind of works.");

        System.out.println("Total words count: " + doc.getTotalWordsCount());
        System.out.println("Distinct words count: " + doc.getDistinctWordsCount());
        System.out.println("Weighed words: \n");
        System.out.println(doc);

    }

    public static Document parseFile(String filepath) {
        // String filepath = "assignment1.txt";
        Charset ENCODING = StandardCharsets.UTF_8;

        Path path = Paths.get(filepath);

        //System.out.println(Files.readAllLines(path, ENCODING));
        Document doc = null;
        try {
            doc = new Document(Files.readAllLines(path, ENCODING));
            int count = doc.getDistinctWordsCount();
            System.out.println("Words count:  " + count);
            System.out.println("Words:  " + doc.words);
            doc.printWeighedWords();
        } catch (IOException e) {
            System.out.println("error");
            e.printStackTrace();
        }
        return doc;
    }
}

Sorting of objects based on their fields | Java

We experience the problem of sorting certain arrays or values quiet often in the world of programming. Well, When it is the inbuilt data types like int or char or even string, there are inbuilt methods that will solve the problem.

int[] c={1,5,3,7,3,9,100,50,40,30,20,10,11,9,12};
        
        //sorting of normal array
        Arrays.sort(c);
        System.out.print(Arrays.toString(c));

Now the problem is that, how do we solve a problem of sorting a list of custom objects like this one.

public class CustomSortingTest{
 
    int value=0;

    public CustomSortingTest() {
     }
    public CustomSortingTest(int a){
    value=a;
    }
}

The solution is pretty simple actually. I remember my effort some years back trying to create a separate utility method and passing the list of this object and writing whole logic there to sort the list and return another one.

Java when sorting a collection, looks for the method "compareTo(T..)" which is the function for an inbuilt interface which most of the inbuilt classes happens to be implementing. When we write a custom class, we normally do not implement this interface, and so while trying to sort the list, java never finds the interface method. Now you could simply implement this interface and override the method and write your custom logic of sorting there. Below is an example of this one.

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

/**
 * 
 * @author Rajan Prasad Upadhyay
 *
 */
public class CustomSortingTest implements Comparable<CustomSortingTest> {
 
 int value=0;

    public CustomSortingTest() {
    }
    public CustomSortingTest(int a){
        value=a;
    }
    public static void main(String[] args) {
        List <CustomSortingTest> a=new ArrayList();
        int[] b={1,5,3,7,3,9,100,50,40,30,20,10,11,9,12};
        int[] c={1,5,3,7,3,9,100,50,40,30,20,10,11,9,12};
        
        //sorting of normal array
        Arrays.sort(c);
        System.out.print(Arrays.toString(c));
        
        //sorting a list of objects based on their certain fields
        for(int i=0;i<b.length;i++){
            a.add(new CustomSortingTest(b[i]));
        }
        Collections.sort(a);
        for(int i=0;i<a.size();i++){
            System.out.print(" "+a.get(i).value);
        }
    }

    @Override
    public int compareTo(CustomSortingTest o) {
        if(value<o.value){
            return -1;
        }else if(value==o.value){
            return 0;
        }else{
            return 1;
        }
    }
}

The logic is preety simple. Just determine when to return -1 and when to return 1 or 0.

Best of luck

Selenium Web Driver Downloading video

G’Day people. Well as the heading of this article suggests, I am going to share with you some of my adventures I have had in the recent months. The adventure is in the virtual world of computers and internet rather than the real world.

The Real problem

Every story begins with a problem. The problem in my case is that Internet is very expensive. And so, as magic comes with a price, so does the internet in modern age. I have to download some video tutorial series from youtube and I prefer not to do that from my home. I know a place where I could but I could not spend too much time sitting there and going to each and every video url and then downloading the video. Now, the quest starts. How to download the videos without spending much time there? I have two choices.

· Get someone to download them for me.

· Discover of an automation, that could do that for me.

It’s not that I could not go with solution number 1, but what if I have to download another set of videos again after some time. Every time finding some to do the job for me was not an option I would prefer. I am a type of person who would rather do some learning, some discovery, some hits and trials and go to solution #2.

First steps

Now I happen to know of a tool, called a webDriver. A web driver is a tool which could communicate with a browser and tell it to perform what actions you desire through programming. I have done clicks, navigations', hovers, execution of javascript etc through it, and I truly believed that I could go one step further and do file downloads as well. I started digging.

Second thought

OK, that was the beginning of the story. The overall steps and happenings of the story are not always funny or adventurous as you might have been thinking. Keeping this in mind, lets directly go to the interesting parts of findings and problem solving and skip the boring part of blood sucking labors and hopeless efforts.

The technical problem

To find some videos in youtube is not really a big problem. It is very easy to write a selenium script that will open a browser, navigate to the youtube site and search for a term in the search box. Now when the search box is available. The real problem begins. Lets say we reached the video page where the video is being played.



The problem are as pointed below.

1. Youtube does not have a direct link to download the video.

Luckily there are plugins which you could just install to the Firefox and it will add a download link to your youtube video page. You just have to click it and then save it. Now another problem which I later found was that, Firefox had these things called the profiles. When you install Firefox for the first time(I am using windows7), a default profile is made. When you install extensions on Firefox from default profile, it is available only for the instance of Firefox opened from default profile. Now selenium opens Firefox with a new profile by default. So the add ons are not available. I had three options.

· Configure Firefox to open with the default profile.

· Create a different profile and configure Firefox to use this one each time.

· Pass Firefox with the add-on files each time to add to itself.

Solution number was not a good idea, mixing normal browsing and development profiles as same. So I went for solution #2.

Steps.

Ø Create a new profile called selenium.

o Hit ctrl + r on windows

o Type “Firefox –profilemanager”

o Create a new profile named selenium, directory to somewhere other than the default place like E:FirefoxSeleniumProfiles

o Now while creating instance of selenium, configure it to use this profile.

FirefoxProfile profile = new FirefoxProfile(new File("E:ProgrammingFirefox-selenium-profile"));

WebDriver driver = new FirefoxDriver(profile);

Ok, the Firefox started with this WebDriver will use that particular profile. You could install the youtube downloader add-ons on that profile by opening its instance and installing it there.

The problem to the point where there is no download button available on the video page of selenium is solved. Now when you manually click the download button and then the mp4 option, a window appears asking you about what to do with the file. Two options are whether to open it with some registered program, or to save it to some location. Now this window is part of native windows and selenium is unable to handle the window.

So now, the problem is “How to force Firefox to save the file in disk quietly instead of asking for option”. And this happen to be easy. Just while instantiating Firefox, add some more options for that profile.

FirefoxProfile profile = new FirefoxProfile(new File("E:ProgrammingFirefox-selenium-profile"));

profile.setPreference("browser.download.folderList",2);

profile.setPreference("browser.download.manager.showWhenStarting",false);

profile.setPreference("browser.download.dir","E:ProgrammingDownloads");

profile.setPreference("browser.helperApps.neverAsk.saveToDisk","text/csv,video/mp4");

WebDriver driver = new FirefoxDriver(profile);

 

These options override the default behavior of Firefox download-manager, specify a default download directory and also the mimetypes of files for which to quietly save the files.

The problem is almost solved. The only logic remaining to write is about which keywords to search, how to find the video urls, and how to iterate through those urls. These are yours choice. I have added a sample code to iterate to the urls of the searched videos.

import java.io.File;
import java.util.List;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.Firefox.FirefoxDriver;
import org.openqa.selenium.Firefox.FirefoxProfile;
public class WithExtension {
 static WebDriver getExtendedFirefox(){
 FirefoxProfile profile = new FirefoxProfile(new File("E:ProgrammingFirefox-selenium-profile"));  profile.setPreference("browser.download.folderList",2);  profile.setPreference("browser.download.manager.showWhenStarting",false);  profile.setPreference("browser.download.dir","E:ProgrammingDownloads");  profile.setPreference("browser.helperApps.neverAsk.saveToDisk","text/csv,video/mp4");
 WebDriver driver = new FirefoxDriver(profile);
 return driver;
}
public static void main(String[] args) {
 WebDriver driver = getExtendedFirefox();
 driver.get("http://www.youtube.com/watch?v=2V9h1aa-Cro");  driver.navigate().to("https://www.youtube.com");
 driver.findElement(By.id("masthead-search-term")).sendKeys("java tutorial advanced leveln");
 longWait(5000);
 WebElement videoSection = driver.findElement(By.id("gh-activityfeed"));
 List videos = videoSection.findElements(By.className("yt-lockup"));
 System.out.println("The number of videos is " + videos.size());
 for(WebElement vcontent : videos){
  WebElement vpanel = vcontent.findElement(By.className("yt-lockup-content"));
  WebElement vtitle = vpanel.findElement(By.className("yt-lockup-title"));
  System.out.println(vtitle.getText());
  System.out.println(vtitle.findElement(By.tagName("a")).getAttribute("href")+"n");
}
}
public static void longWait(long a){
 try { Thread.sleep(a); }
 catch (InterruptedException e) {}
}
}

Best Database connector code for java and oracle.

package database;


import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author Rajan Prasad Upadhyay
 */
public class OracleConnection {
    /* parameters */
    private String username=null;
    private String password=null;
    private String host=null;
    private String port=null;
    private String sid=null;
    
    private Connection con = null;
    
    
    
    /* Query execution */
    public void executeUpdate (String query) throws SQLException{
        if(con != null){
            PreparedStatement ps = con.prepareStatement(query);
            ps.executeUpdate();
        }
    }
    public ResultSet executeQuery (String query){
        ResultSet rs = null;
        if(con != null){
            try{
                PreparedStatement ps = con.prepareStatement(query);
                rs = ps.executeQuery();
            }catch(Exception e){
                
            }
        }
        return rs;
    }
    
    /* parameters part */
    public void setConnection (String conString){
        try {
            Class.forName("oracle.jdbc.driver.OracleDriver");
            con = DriverManager.getConnection(conString, username, password);
        } catch (Exception ex) {
            //return null;
            System.out.println("Connection could not be established");
            ex.printStackTrace();
        }
    }
    public Connection getConnection() {
        return con;
    }
    public void setDefaultParameters(){
        username = "USER1";
        password = "oracle";
        host    =   "localhost";
        port    =   "1521";
        sid     =   "XE";
    }
    public String getConnectionString(){
        return "jdbc:oracle:thin:@"+host+":"+port+":"+sid;
    }
 
 
 
 /* main method for unit testing */
    public static void main(String [] args) throws SQLException{
        OracleConnection con = new OracleConnection();
        //con.setDefaultParameters();
        con.setHost("localhost");
        con.setPort("1521");
        con.setSid("XE");
        con.setUsername("USER1");
        con.setPassword("oracle");
        
        //System.out.println(con.getConnectionString()); return ;
       
        con.setConnection(con.getConnectionString());
        
        ResultSet rs = con.executeQuery("select sysdate from dual");
        
        if(rs != null){
            while(rs.next()){
                System.out.println("The date is " + rs.getString(1));
            }
        }
    }

    /* Getters and Setters */
    public String getUsername() {
        return username;
    }
    public void setUsername(String username) {
        this.username = username;
    }
    public String getPassword() {
        return password;
    }
    public void setPassword(String password) {
        this.password = password;
    }
    public String getHost() {
        return host;
    }
    public void setHost(String host) {
        this.host = host;
    }
    public String getPort() {
        return port;
    }
    public void setPort(String port) {
        this.port = port;
    }
    public String getSid() {
        return sid;
    }
    public void setSid(String sid) {
        this.sid = sid;
    }
}

My Artificial Intelligence Project (The Routine Generator)

Click here for downloading the java project (The routine generator). The code is quiet simple enough that you will learn it easily. Best of luck

However due to the polularity of this page, i am going to descrive the concept of planning of the program in brief in a abstract way, just for helping the interested ones about general concepts of arranging the components.