But it works on my machine!!

I've heard this phrase numerous times while testing and communicating an issue/bug to a developer: "But it works on my machine!". For some of them, it's the first thing they'd say, sometimes even before I finish describing the exact sequence of events. And you'd think that I would have learned to handle this situation gracefully by now, but I still have to resist the urge to smack them and drag them to my desk or wherever the tests ran, and show them the error.

Well...I wouldn't be writing this just for that. I recently faced this issue myself. I've been working on creating an MQTT keyword library for Robot Framework. This library provides keywords to publish/subscribe to an MQTT broker. Source code is here: https://github.com/randomsync/robotframework-mqttlibrary

One of the keywords that is a part of this library is 'unsubscribe'. This lets a durable client (one which subscribed with clean session set to false) unsubscribe from a topic so that it doesn't receive any further messages published to the broker on that particular topic. If the client doesn't unsubscribe and disconnects, the subscription is still valid and the broker will deliver all messages received when the client next reconnects.

A test for this keyword is:
Step 1. Connect, Subscribe and Unsubscribe from a topic with a durable client (Client A)
Step 2. Publish messages to the topic with a different client (Client B)
Step 3. Connect as Client A, Subscribe and ensure that messages published by Client B are NOT received.

I wrote the test using Robot Framework and it worked on my mac. To run these tests, I'm using a local mosquitto broker and also a public broker provided by eclipse project at: http://iot.eclipse.org. While running the tests from my mac on both local broker and the eclipse broker, it verified that after unsubscribing and reconnecting, no messages were delivered. I pushed the change.

I also have the project set to build on travis-ci.org: https://travis-ci.org/randomsync/robotframework-mqttlibrary.  To my dismay, that test failed on travis-ci. WTF? "But it works on machine!!"

Typically, unless there's something obvious that you overlooked, the only way to tackle these kind of issues is process of elimination. We try to account for differences between local vs. remote server and determine if any one, or a combination of those differences might be the culprit. Of course, in these kind of scenarios, it helps if the local machines that you build on are as similar to the build/deploy servers as possible. (At Amazon, all engineers are given a RHEL VM instance to develop on, which is what is used for production deployments as well)

In my case, differences were:
Local environment: Mac, Python 2.7.6, pip 1.5.6,
Travis build instance: Ubuntu 12.04, Python 2.7.9, pip 6.0.7

Other dependencies were installed through pip and *should* be the same:
paho-mqtt: 1.1
robotframework: 2.8.7

Target server iot.eclipse.org is running mosquitto version 1.3.1 and locally, I have version 1.3.5 running.

So the first thing I could eliminate easily was the broker. I ran the tests from my machine using iot.eclipse.org as the target and they passed. Still, I went through the release notes for mosquitto server to see if there were any changes between 1.3.1 and 1.3.5 that might provide a clue.

Next thing I looked into was to somehow re-create locally the VM instance travis uses so I can better debug, because not having access to any logs or the machine where the tests fail is a major hinderance. I found some helpful articles [1] [2] [3]. There's also an option to upload the build artifacts to S3 as described here.

At that time, I didn't get a chance to try any of these. Ideally and as I mentioned before, you should have a build environment easily accessible that is as close to production as possible. So long term, it will help in debugging build issues to have a local instance similar to what travis-ci uses. In this case, I found that the tests failed when running on a local windows platform as well. So that made it easier to debug.

One of the things I had a hunch about right from the start was that I was not waiting long enough for 'unsubscribe' to complete. What if I send a 'disconnect' very quickly before the broker even finishes processing the 'unsubscribe' packet. I was able to confirm this by adding a 1 second sleep after unsubscribing on the windows machine. After adding that, the tests passed.

Obviously adding sleeps is not the correct fix. Paho client's documentation suggests to use one of the 'loop*' functions: http://eclipse.org/paho/clients/python/docs/#network-loop. These allow you to wait and confirm that the message was sent or received. I had overlooked these before but I went ahead and added those to the connect and subscribe functions (still need to do that for publish, disconnect) and was able to verify that the unsubscribe test worked without the sleep.

Conclusion:

1. Inconsistent test failures are the bane of test automation. They undermine the value provided by test automation. Follow these as best practices:
1. Design robust automated tests. DO NOT add an automated test if it's not 100% reliable. I would much rather have 1 reliable test than 10 unreliable tests.
2. Have a build environment available locally which is very similar (if not the same) as the one used by your CI hosts
2. But, just because a test is failing inconsistently doesn't always mean it's a test issue. It can be a bug in the code, as seen above. It definitely helps if the test automation engineers know how the application is implemented and can look at and understand the code. Sometimes, just looking at the code gives you ideas on what kind of edge conditions to test for. Sometimes, you just get lucky and find an issue which may have been overlooked.
3. I still don't know why the tests pass on a mac and fail on a windows/ubuntu (travis) machine consistently. Python version is different but I didn't get to evaluate that. Could there be a difference in how the network packets are sent/received in whatever libraries the 2 versions of Python are using? There's also a slight chance that there's a bug in some client/broker implementations if the tests fail inconsistently.
   Next steps:
• Setup a virtualenv on mac so I can use different versions of python 
• Setup a local image used by travis-ci

How I’m NOT going to delete my Windows profile folder again?

So I deleted the complete profile folder under C:\Documents and Settings\<my user id> on my work machine. Everything under that was gone, except for a few read-only folders that survived. My Documents, Application Data, Desktop, Local Settings, Favorites…gone. My Outlook archives, my OneNote notes and a few years worth of accumulated documents…all done for. Plus all my Java projects, since I configured git under that folder and all my git projects were there.
How did I do that? Well…I was creating a new Java project using Eclipse to demo the use of excel-testng. But since I didn’t want to use the default workspace location, I specified the location where I keep the git projects, which happens to be the profile folder. Usually when I use the default location, eclipse creates a sub-folder under the default location with the same name as the project as below:

But when I didn’t use the default location, it didn’t create sub-folder with the project name. It created the project right under that location!

And having not realized that and since I wanted to remove the project (can’t remember why), I proceeded to delete the project.

Now, to be fair to myself and going back to that moment just before I clicked OK, I have never ever before needed to not delete the project contents from disk. The whole idea of deleting it from eclipse workspace has always been to get rid of the project, completely. And that’s what I did. At that time, I found it a little surprising that it was taking a little longer that usual. But it was 5pm on Friday evening and things usually get a little strange around that time, so I was ready for anything. Just not ready for what happened next.
It probably took me less than a few seconds to realize what had happened since I had an explorer window open on that folder, and I saw blank white screen where I had my profile folder contents previously. And I’m not going to mention the words that came out of my mouth next, or the next few hours I spent trying to restore the contents unsuccessfully and recreate the git projects.
There’s a happy ending to this: I had got my laptop replaced just a week ago and had a backup of most of that folder. I’ll most probably never find out if there was anything I was NOT able to recover, and that’s a good thing!

excel-testng: driving TestNG tests through MS Excel

Over last few days, I have been working on a small project in my spare time. It's called excel-testng and it provides a way to drive TestNG tests through MS Excel. The code repository is here: http://code.google.com/p/excel-testng/ and the jars can be downloaded from here. I also put together a small automation project using Selenium WebDriver to demonstrate its use here: https://github.com/randomsync/excel-testng-demo.

Introduction

During functional testing projects, after we create tests for a certain application, we need to review them with the rest of the team (developers, analysts, project managers etc). We may use a test management tool to document the test cases and then export them into an easily distributable format like MS Excel or PDF. Sometimes, we may also create the tests in Excel directly specifying the test name, description, parameters and other data in the spreadsheets. And then finally, we may automate a part (or all) of the tests.
So after the tests are automated and when executing the tests, we need to specify which tests to run and the test data (parameters). In Quality Center, it involves creating a test set (kind of like a test suite) and then adding tests to it. If they are automated in QTP which integrates with QC, they can be executed by running the test set. But we have started using Selenium WebDriver for its cross browser capabilities and that means we need to specify the tests in a format that can drive the Selenium tests. We use TestNG as the framework for test execution, assertions and reporting.

Extending TestNG

TestNG is a great framework for test execution and its input can be in form of an XML file that specifies which tests to run, where to find the test classes/methods, test parameters and a whole lot of other features that gives you a fine grained control over each test execution. However for our UI functional tests, we wanted to be able to specify the tests and test executions in an easily distributable format (like MS Excel) as mentioned above. The good thing is that TestNG provides the capability to extend it so that its input can be created and executed programmatically. So this way, the test specification in Excel files can be parsed and driven through TestNG. And this is exactly what excel-testng does. It externalizes all Excel parsing and TestNG XmlSuite creation so that the focus can be in creating the test classes and methods. Once that is done and Excel specifications created, all that needs to be done is to provide a main method that does this:

   1: ExcelTestNGRunner runner = new ExcelTestNGRunner("input.xls"); // this can be a single file

2: // or a directory, in which case all spreadsheets in that directory are parsed

3: runner.run(); // run the tests

With ExcelTestNGRunner class, you just specify the input location of the Excel file(s) that constitute the test specifications and then call the run() method, which parses the excel file using default parser (an instance of ExcelSuiteParser) into XmlSuites, creates a TestNG object if not already created and then runs them. After that TestNG takes care of test execution and reporting.

Excel Test Specification

ExcelTestNGRunner parses each worksheet in the Excel file(s) into a separate suite using the included parser (ExcelSuiteParser). Each suite can have suite level parameters specified in the worksheet and the test specifications that specify which tests will be run, their name, description, parameters and the test classes. Here's what a test suite in Excel looks like (a demo spreadsheet can also be downloaded from here and can be used as a starting point):

Specifying TestNG tests in Excel File

The top few rows in the worksheet provide the suite information. ExcelSuiteParser looks for the string "Suite Name" and retrieves the name of the suite from next cell in the same row. Similarly, it looks for string "Suite Parameters" and retrieves suite parameters from the next cell. "Suite Configuration" is not currently used. You can customize the location of these values by providing your own map to the parser. See "Customizing Input" for more details.

To retrieve the tests that will be executed, it looks for the 1st row containing "Id" in the 1st column. This will be the header row, below which each row is a separate test. The header row and tests must have columns specifying Id, Test Name, Test Parameters and Test Configuration (which specifies the classes containing the Test methods). If "Id" is left blank, the test will not be added to the suite. Finally, it parses each row under the header row into a TestNG XmlTest and adds it to the suite. The test specifications are provided as:

• Test Name: generated by concatenating Id & Test Name
• Test Parameters: retrieved from "Test Parameters" column and need to be provided in valid properties (<key>=<value> etc.) format. You can also specify functions as parameter values and then add the logic to parse and evaluate the functions in your test classes (maybe a Base Test class)
• Test Classes: specified under "Test Configuration" column as classes property. Currently, you can only specify a single test class of which, all @Test annotated methods will be executed as a part of this test execution. I'm going to add the ability to select the test methods in later releases.

Customizing Input

If your test cases are specified in Excel but in different format, there are 2 levels of customizations you can do with ExcelTestNGRunner on how to parse the input spreadsheet(s):

1. Custom Parser Map (currently not implemented): You can use the in-built parser but specify your own parser map, which tells the parser where it can find the suite and test data
2. Custom Parser: You can create your own parser by implementing IExcelFileParser interface. You need to parse the spreadsheet file and return a list of TestNG XmlSuites.

ExcelTestNGRunner also provides helper methods to customize the TestNG object it uses to execute tests. For example, you can specify any custom listeners using addTestNGListener() method. If you need to have more control, you can create your own TestNG object and then pass it to ExcelTestNGRunner. Please see javadocs for more details.

Putting it together

You can see the project at https://github.com/randomsync/excel-testng-demo for a complete working demo of excel-testng to parse the input test specifications in Excel. It uses Selenium WebDriver to automate the testing of basic Google search functionality.

Update (3/12/2012): I'm now using google code to host this project because of the provision to host the downloadable jars, javadocs and wiki easily. I'll keep it synced with github but you can find the project documentation and downloads there.

Puzzler Solution: Prisoners and Hats and a Jungle

Can’t believe it’s been 2 months since I posted the puzzle. I’ve been involved in test automation using Selenium WebDriver over last couple of months and it has been a great experience designing and coding the test suite. The 1st step is complete - smoke tests for 2 of our applications have been automated and can be run on different browsers concurrently. Now we’re moving to the next steps (which are still TBD). I have lots of ideas in my head but I have to evaluate their feasibility and to break them into different phases so we can realize the added value as we continue to improve the automation suite.

Back to the puzzler solution. For reference, here is the puzzle statement and the code. My implementation of the guessStrategy method is below.

The strategy is that the 1st person to guess speaks the color of the odd hats ahead. So if the (number of black hats ahead) % 2 == 0, the guess is “Black” (lines 4-8). How does it help that person? It doesn’t, that prisoner is unlucky enough to be the 1st one to guess (end of the line) and has a 50% chance. But the rest of the prisoners can guess the color of their hat correctly now that they know the initial guess, all guesses since then and the color of hats ahead. So for example, if the 1st prisoner guesses black, there must be odd black hats ahead. Now if the 2nd prisoner sees odd black hats ahead, s/he must be wearing a white hat because the previous prisoner also saw odd black hats. And the logic goes on like that…if the nth prisoner sees odd number of black hats, and previously, odd number of people have guessed black, then s/he must be wearing a white hat (lines 13-23).

   1: public static char guessStrategy(String prevGuesses, String remArr) {

2: int numB = getNumOfChars(remArr, 'B');

3: int numPrevB;

4: if (prevGuesses.length() == 0) { // initial guess

5: if (numB % 2 == 0)

6: return 'W'; // if B is even, return W

7: else

8: return 'B'; // else return B

9: } else {

10: // strategy: if previously even number of people have said black,

11: // and I see even black hats, I have a white hat, else black

12: numPrevB = getNumOfChars(prevGuesses, 'B');

13: if (numPrevB % 2 == 0) {

14: if (numB % 2 == 0)

15: return 'W';

16: else

17: return 'B';

18: } else {

19: if (numB % 2 == 0)

20: return 'B';

21: else

22: return 'W';

23: }

24: }

25: }

26: 

27: /**

28: * @param arr

29: * @param ch

30: * @return number of times the specified char appears in the array

31: */

32: public static int getNumOfChars(String arr, char ch) {

33: int num = 0;

34: for (int i = 0; i < arr.length(); i++) {

35: if (arr.charAt(i) == ch)

36: num++;

37: }

38: return num;

39: }

Dennis Ritchie

http://www.nytimes.com/2011/10/14/technology/dennis-ritchie-programming-trailblazer-dies-at-70.html 

No doubt, comparisons will be made with Steve Jobs and how much media buzz was created.

iPhone has had a huge impact in my life, but not as much as C. C has defined my thinking. During college, most of the programming I did was in C and K&R was THE reference book I had with me most of the time (I still have it). The precise and succinct way the book is written helped me appreciate the beauty of coding. Every function was defined in clear and concise terms. That not only guided me in my programming adventures, but somehow it also helped me put a different perspective around life and accumulate a no-nonsense attitude of holding myself to much higher standards than anyone else.

From the article linked above:

Colleagues who worked with Mr. Ritchie were struck by his code — meticulous, clean and concise. His writing, according to Mr. Kernighan, was similar. “There was a remarkable precision to his writing,” Mr. Kernighan said, “no extra words, elegant and spare, much like his code.”

In essence, this guy created something wonderful and powerful in the world of computing. And I hope his legacy lives on.

-Gaurav Gupta

Puzzler: Prisoners and Hats and a Jungle, Oh My!

This puzzler was mentioned in Car Talk sometime ago and I really liked it. In brief, it goes like this:

A prison has 30 prisoners sentenced to be executed and the warden, who has the authority to pardon, decides to give them a chance to escape the punishment. He will stand all the prisoners in a straight line with each prisoner able to see the heads of all prisoners in front of him but not of those behind him. Next, he will put either a white or black hat on each prisoner’s head and ask them to guess the color of their hat one by one (starting with the 1st prisoner in the back of the line who can see all 29 other prisoners’ heads). If he guesses correctly, he’s pardoned. What is the strategy that the prisoners can use to maximize their chances of being pardoned?

The answer I came up with was grossly wrong so when these guys gave the answer (link to which I’m not posting here but can be found easily), I was intrigued. When listening to the answer, it sounded very simple but when I actually thought about it some more, I had to listen to it again to understand the strategy. For example, does the current prisoner need to know all the previous guesses or only the most previous guess?

I decided to write a simple program for this. The line of prisoners here is a StringBuffer of predefined size (in this case, 30) that is randomly filled with ‘B’ or ‘W’ chars. The objective is to write a method that will be called for each character in the StringBuffer with all the previous guesses (as a String) and remaining series (as a String). The method has to return the current character and should be implemented in such a way as to maximize the correct answers. Here’s the code:

   1: import java.util.Random;

2: public class Test {

3: static int SIZE = 30;

4:

5: public static void main(String[] args){

6: int correctGuesses = 0;

7: StringBuffer pRow = new StringBuffer(SIZE);

8: StringBuffer prevGuesses = new StringBuffer(SIZE-1);

9: char currGuess;

10:

11: Random rnd = new Random(System.currentTimeMillis());

12: //---print the series

13: System.out.print("Series:\t");

14: for (int i=0;i<SIZE;i++){

15: if (rnd.nextInt(2) == 0) //0=black, 1=white

16: pRow.append('B');

17: else pRow.append('W');

18: System.out.print(pRow.charAt(i) + " ");

19: }

20: //---strategy

21: System.out.print("\nStrat:\t");

22:

23: for (int i=0;i<SIZE;i++){

24: currGuess = guessStrategy(prevGuesses.toString(),pRow.substring(i+1));

25: System.out.print(currGuess + " ");

26: if(currGuess == pRow.charAt(i)) correctGuesses++;

27: prevGuesses.append(currGuess);

28: }

29: System.out.print("\nCorrect Guesses=" + correctGuesses);

30: }

31: 

32: public static char guessStrategy(String prevGuesses, String remArr){

33: //random

34: Random rnd = new Random(System.currentTimeMillis());

35: if (rnd.nextInt(2) == 0) //0=black, 1=white

36: return 'B';

37: else return 'W';

38: }

39: }

First, I generate the series and print it (lines 13-19). And then I call the guessStrategy method for each character in the series with all the previous guesses and the remaining series. But currently, there is no strategy and each time, it randomly returns ‘B’ or ‘W’. And obviously, result is that correct guesses average around 15. Your job, should you decide to accept this assignment is to provide a better implementation of the guessStrategy method which maximizes the chances of guessing correctly. If you’re stuck, you can look at the Car Talk website or search on internet to find the answer and then try to implement it.

I’ll post my implementation of the method in a few days. Hopefully it provides enough of a challenge to some people to work on this besides their otherwise busy life.

Friday Lunch: Content-Type, Content-Disposition HTTP Headers and MIME type handling in IE

We have a scripting tool that is used to create web-based scripts that simulate application scenarios. These scripts are run through a monitoring tool on a continuous basis to assess the health of these applications. The scripts & scripting tool are also used to validate the scenarios if the monitoring tool issues an alert. The scripting tool uses IE to run the HTTP request/response scenarios. When recording the script for a recent new application, it was noticed that after submitting the request which was an HTTP POST with XML data, the tool was bringing up a dialog box to save the response file, something you’d get when clicking on a HTML link in the browser to a file (maybe a word .doc etc) and if you don’t have an application to open that kind of file. We checked the response and it contained a standard XML that should usually be displayed within the browser (like if you click on Amazon’s AWS service). The fact that it was bringing up a download prompt and not displaying the XML in the browser was a problem because the non-technical team that uses it to validate the application will not be able to determine whether the script ran successfully or not.

The application is a servlet that takes the initial POST request and returns an XML response. After further investigation with the tool and looking at the response headers, it was found that the application was returning “application/x-www-form-urlencoded” in “Content-Type” header erroneously. I haven’t found something specific that would say that content type is not correct for HTTP response, but it is more common to use this content type when submitting form-submission data that is URL encoded (see here).

Investigating with WebScarab

My first conjecture was that the file download prompt was being displayed because of this Content-type header’s value. If we could intercept the response and replace its value with something like “text/xml”, the tool or browser will be able to display the content within the application instead of bringing up the file download prompt. And to validate that, I used WebScarab to intercept the request/responses. But for some reason (maybe because of invalid Content-Type header), it didn’t intercept the response. It would intercept the request and after accepting, it would just complete the request and we’ll end up seeing the file download prompt in the tool. Clearing the “Only MIME-Types matching:” didn’t help.

So the workaround was to write a BeanShell script within WebScarab to replace the “Content-Type” header with a proper value instead of manually intercepting the response. That turned out to be quite straightforward. All we needed to do was put:

response.setHeader("Content-Type","text/xml");

in WebScarab’s Bean Shell interface under the fetchResponse method. As is obvious, this replaces the value of Content-Type header in the response to “text/xml”.

Once we tried it out, it worked and the browser displayed the XML response without bringing up the file download prompt. This proved that the Content-Type header was the problem. But what exactly was the problem? What makes the browser display a prompt to the user to download a file instead of just displaying the content? Further searching revealed this article: http://www.jtricks.com/bits/content_disposition.html

So basically the Content-Disposition header along with Content-Type specifies the MIME type of the file returned and whether the browser should display it or prompt the user to download it. The RFC has all the details but the gist is that the Content-Disposition header can be set to “attachment” for the browser to display the file download prompt. But the response we were getting didn’t have a Content-Disposition header. And IE was still displaying the file download prompt. Further reading warranted.

How does IE handle MIME types in a response, anyways?

I kept on searching and reading on how and when a browser brings up the file dialog prompt. The Content-Type header still bothered me. Then I came across this MSDN article: MIME Type Detection in IE. It provides details on what steps IE takes to detect the content-type of the content before deciding what to do with the file:

The purpose of MIME type detection, or data sniffing, is to determine the MIME type (also known as content type or media type) of downloaded content using information from the following four sources:

• The server-supplied MIME type, if available
• An examination of the actual contents associated with a downloaded URL
• The file name associated with the downloaded content (assumed to be derived from the associated URL)
• Registry settings (file name extension/MIME type associations or registered applications) in effect during the download

If you look at the known MIME types for FindMimeFromData, “application/x-www-form-urlencoded” is not one of them. It is also not an ambiguous MIME type ("text/plain," "application/octet-stream," an empty string, or null) either. So with the MIME type as unknown,

If the "suggested" (server-provided) MIME type is unknown (not known and not ambiguous), FindMimeFromData immediately returns this MIME type as the final determination.

And what happens after that is in this MSDN article: Handling MIME Types in IE. It doesn’t mention exactly what happens if the MIME type is unknown, there’s no application associated with it in the registry and there’s no Content-Disposition header. But I’m guessing that the only thing for it left to do is to let the user decide by showing the file download prompt as was happening in our case. And we were able to prove that. I looked up the registry key for “text/xml” under HKEY_CLASSES_ROOT\MIME\Database\Content Type and added it to a new key “application/x-www-form-urlencoded”. The new registry key looked like this:

[HKEY_CLASSES_ROOT\MIME\Database\Content Type\application/x-www-form-urlencoded]
"CLSID"="{48123BC4-99D9-11D1-A6B3-00C04FD91555}"
"Extension"=".xml"
"Encoding"=hex:08,00,00,00

This tells IE that for responses received with Content-Type equal to “application/x-www-form-urlencoded”, use extension “.xml” and open it however XML files are opened (which is to display them inline). Once added, we went back to the tool and ran the script. And this time, it displayed the response xml without bringing up the download prompt.

So through this exercise, I learned about how IE displays different MIME types and about Content-Type and Content-Disposition headers.