Quick Groovy Scripting

Recently I have to port some data from mainframe database to SQL based db for testing purposes. I have started with some text report files generated from mainframe. I have fond of using unix awk, grep for these kind of data munging. Also used perl and ruby for some scripting activities in the past. But given that I had to do this on windows and also with fading knowledge of perl, thought of getting in donw with groovy. Since eclipse also supports groovy it became easy to start with.

I got something running which spits SQL statements (using println) for every line of the input. Sooner my eclipse console started eating the output because of the buffer size for console display I had in my settings! Though I had the huge monolithic script which works fine, I cannot able to get the output in single shot. I had to rerun them in parts to get the final collective output. This slowed me on tweaking the final script. Given we didn't have much re-factoring support in eclipse, I couldn't either easily extract them as functions as I could in Java. But I am able to use a more powerful tool i.e. define a closure immediately and redirect the inputs to the println statements to a File without much changes to the original script.

println "insert into table_name (col1, col2, col3) into values (${col1},'${col2}', ${col3})"

def file = new File( "C:\output.txt")

def println = { line ->  file.append(line)}

Just adding these two line saved me a lot of time, also now I can switch back to see the output in command line or to capture them in a file very easily.

Other things that helped me to get things done quickly are the ability to refer the variables inside the string directly as " '${col2}'". This is especially useful where I have to qualify the column of string data type with quotes, otherwise for which I have to use endless escaping and + con-catenations!

Also for the next script I did, I started writing in small classes than single file, so made things easier to change at last minute.  Another gotcha for beginner for the groovy script is the use of '=='. Remember in groovy use of '==' is actually converted to this.equals(that) before the execution. I ran into endless self-recursive calls as I used the == for reference comparison as we do in Java.

As I got the script completed there were lot of duplicate SQL statements in the output. As we get errors due to integrity constraints in database, I have to find some way to remove duplicate statements. In unix, I normally use `uniq` to get this done. Since I have to get that done quickly, i just looped thru the output file and added each line to the Set and dumped it back out to remove the duplicates.

Being used Perl, and Ruby in the past I know the libraries support in perl or ruby are far huge when compared to groovy. But the single fact that I have used to Java in past years and have to work with windows, Groovy was a life saver!

N.B. No data conversion is possible without effective use of Regular Expressions. I did used regular expressions to format the input files before running groovy scripts against them. I used Textpad to do  find/replace with regular expressions. The regular expression support in eclipse editor find/replace tool still needs improvements before could it could be really useful.

Testing Naturally, and Agile

Behaviour-Driven Development is what you are already doing, if you are doing Test-Driven Development properly...

Test-Driven Development is commonly used everywhere, however the term 'test' makes people to think it as something they do extra to their coding activity. The same makes it hard to convince managers about the value of the unit-testing. Also the term unit-testing means different to different people.

How BDD, helps in: In BDD you capture every test you write under a specification of the requirement. The user stories could be directly translated to test specification. The greatest value here is this grouping of unit-test cases to a behavior that matches to a requirement from user story. So its more traceable on test coverage, from the business perspective. This business perspective helps management team to understand the value of testing.

The BDD concept is quite spoken around for years, but people misunderstood it as something where they don't have to do testing. But it actually the same test-driven development. This makes again to think that, BDD has nothing new.

BDD could be considered as DSL for testing. It uses concepts such as Story, Behavior (terms that are more common in agile practice and Object Oriented Design*) to directly describe the test cases that we write. These test cases are described using common terms called as Specification.

Dan North first described this term and he had taken efforts to come up with specification framework in java, called as JBehave. If you check the API, you will know that its more are like JUnit, just with test method naming conventions. In JUnit, we have every test-method starting with 'test' keyword. Here it starts with 'should'. As I told you earlier, its not different in API much, but its concepts and the domain terms you use to describe the test-cases makes them better. Also here you should note that from JUnit4, ou dont need to have the methods named starting with test. Also later versions of JUnit also introduces higher level domain terms such as Theory, etc.,

rSpec is a ruby testing framework which uses behavioral driven development concepts.

Since lately there are many such common abstraction is coming up in development testing, and practices. And finally one of the abstraction is going to be commonly, used in organizing your test cases.

Ok coming back to the point, what makes me to write about this now. The ruby or any dynamic language I liked most is because of the human readable form they give to the code. But still Java gives the more robust JVM platform, that is tested and more compatible with any middleware which is required to support production level high-volume transactions. The dynamic languages are something that is most available for writing tools, scripts that helps to speed up development. I have used perl to write to monitoring scripts which automate as bot logging into production system, as a test user, and executes standard test cases. And it checks back the response to confirm the functionality of the code in production. We have used them even to load test applications in small-scale. With the rich library the perl/ruby has you can automate any task in simple steps. Using java for these is a over-kill.

Unit Testing can also be one such activity, which could be handled well by these languages provided a good flexibility between them. Said this naturally Groovy and JRuby are natural choice for testing our Java code.

And very recently, actually just weeks back there are some projects started on this perspective.

easyb, a project from Andrew Glover, would be a good choice given that its build on Groovy. Go check out yourself. I had written below some of my first hand experiences with easyb.

jtestr, released by Ola Bini, a Thoughtworks employee of UK. This framework is about coding in Java, and Testing in Ruby = > JtestR. I didn't got my hands dirty with this yet, will try this soon and write about it.

easyb: EasyB has very good documentation, so I am not gonna repeat here on using it. I just show you a sample test case Story.

package com.bdd.test;

given "new IM instance", {
im = new com.bdd.InstantMessenger()
}

when "somebody logs in", {
im.login()
}

then "status message should show Available", {
ensureThat(im.getStatusMessage().equals("Available"),eq(true))
}

Test Result:

.
Time: 0.568s

Total: 1. Success!

Instead of the simple output we can generate, striping the closure method definition and could generate something like:

Login Story: Success

given new IM instance

when somebody logs in

then status message should show Available

Since we use groovy closures to wrap a given definition, we can pass that around the Test Story; thereby reusing blocks of code.

As you could see, this way the test report becomes more natural, and next time when a test fails; even your business team can see what's failing without digging into the method to see the details. Off-course you can cheat, but we wont with all intentions of a good developer!

LINKS:
Google Tech Talk from rSpec developer:
http://video.google.com/videoplay?docid=8135690990081075324
*Object Design: Roles, Responsibilities and Collaborations:
http://www.amazon.com/Object-Design-Responsibilities-Collaborations-Addison-Wesley/dp/0201379430