But it works on my machine!!

I've heard this phrase numerous times while testing and communicating an issue/bug to a developer: "But it works on my machine!". For some of them, it's the first thing they'd say, sometimes even before I finish describing the exact sequence of events. And you'd think that I would have learned to handle this situation gracefully by now, but I still have to resist the urge to smack them and drag them to my desk or wherever the tests ran, and show them the error.

Well...I wouldn't be writing this just for that. I recently faced this issue myself. I've been working on creating an MQTT keyword library for Robot Framework. This library provides keywords to publish/subscribe to an MQTT broker. Source code is here: https://github.com/randomsync/robotframework-mqttlibrary

One of the keywords that is a part of this library is 'unsubscribe'. This lets a durable client (one which subscribed with clean session set to false) unsubscribe from a topic so that it doesn't receive any further messages published to the broker on that particular topic. If the client doesn't unsubscribe and disconnects, the subscription is still valid and the broker will deliver all messages received when the client next reconnects.

A test for this keyword is:
Step 1. Connect, Subscribe and Unsubscribe from a topic with a durable client (Client A)
Step 2. Publish messages to the topic with a different client (Client B)
Step 3. Connect as Client A, Subscribe and ensure that messages published by Client B are NOT received.

I wrote the test using Robot Framework and it worked on my mac. To run these tests, I'm using a local mosquitto broker and also a public broker provided by eclipse project at: http://iot.eclipse.org. While running the tests from my mac on both local broker and the eclipse broker, it verified that after unsubscribing and reconnecting, no messages were delivered. I pushed the change.

I also have the project set to build on travis-ci.org: https://travis-ci.org/randomsync/robotframework-mqttlibrary.  To my dismay, that test failed on travis-ci. WTF? "But it works on machine!!"

Typically, unless there's something obvious that you overlooked, the only way to tackle these kind of issues is process of elimination. We try to account for differences between local vs. remote server and determine if any one, or a combination of those differences might be the culprit. Of course, in these kind of scenarios, it helps if the local machines that you build on are as similar to the build/deploy servers as possible. (At Amazon, all engineers are given a RHEL VM instance to develop on, which is what is used for production deployments as well)

In my case, differences were:
Local environment: Mac, Python 2.7.6, pip 1.5.6,
Travis build instance: Ubuntu 12.04, Python 2.7.9, pip 6.0.7

Other dependencies were installed through pip and *should* be the same:
paho-mqtt: 1.1
robotframework: 2.8.7

Target server iot.eclipse.org is running mosquitto version 1.3.1 and locally, I have version 1.3.5 running.

So the first thing I could eliminate easily was the broker. I ran the tests from my machine using iot.eclipse.org as the target and they passed. Still, I went through the release notes for mosquitto server to see if there were any changes between 1.3.1 and 1.3.5 that might provide a clue.

Next thing I looked into was to somehow re-create locally the VM instance travis uses so I can better debug, because not having access to any logs or the machine where the tests fail is a major hinderance. I found some helpful articles [1] [2] [3]. There's also an option to upload the build artifacts to S3 as described here.

At that time, I didn't get a chance to try any of these. Ideally and as I mentioned before, you should have a build environment easily accessible that is as close to production as possible. So long term, it will help in debugging build issues to have a local instance similar to what travis-ci uses. In this case, I found that the tests failed when running on a local windows platform as well. So that made it easier to debug.

One of the things I had a hunch about right from the start was that I was not waiting long enough for 'unsubscribe' to complete. What if I send a 'disconnect' very quickly before the broker even finishes processing the 'unsubscribe' packet. I was able to confirm this by adding a 1 second sleep after unsubscribing on the windows machine. After adding that, the tests passed.

Obviously adding sleeps is not the correct fix. Paho client's documentation suggests to use one of the 'loop*' functions: http://eclipse.org/paho/clients/python/docs/#network-loop. These allow you to wait and confirm that the message was sent or received. I had overlooked these before but I went ahead and added those to the connect and subscribe functions (still need to do that for publish, disconnect) and was able to verify that the unsubscribe test worked without the sleep.

Conclusion:

1. Inconsistent test failures are the bane of test automation. They undermine the value provided by test automation. Follow these as best practices:
1. Design robust automated tests. DO NOT add an automated test if it's not 100% reliable. I would much rather have 1 reliable test than 10 unreliable tests.
2. Have a build environment available locally which is very similar (if not the same) as the one used by your CI hosts
2. But, just because a test is failing inconsistently doesn't always mean it's a test issue. It can be a bug in the code, as seen above. It definitely helps if the test automation engineers know how the application is implemented and can look at and understand the code. Sometimes, just looking at the code gives you ideas on what kind of edge conditions to test for. Sometimes, you just get lucky and find an issue which may have been overlooked.
3. I still don't know why the tests pass on a mac and fail on a windows/ubuntu (travis) machine consistently. Python version is different but I didn't get to evaluate that. Could there be a difference in how the network packets are sent/received in whatever libraries the 2 versions of Python are using? There's also a slight chance that there's a bug in some client/broker implementations if the tests fail inconsistently.
   Next steps:
• Setup a virtualenv on mac so I can use different versions of python 
• Setup a local image used by travis-ci

excel-testng: driving TestNG tests through MS Excel

Over last few days, I have been working on a small project in my spare time. It's called excel-testng and it provides a way to drive TestNG tests through MS Excel. The code repository is here: http://code.google.com/p/excel-testng/ and the jars can be downloaded from here. I also put together a small automation project using Selenium WebDriver to demonstrate its use here: https://github.com/randomsync/excel-testng-demo.

Introduction

During functional testing projects, after we create tests for a certain application, we need to review them with the rest of the team (developers, analysts, project managers etc). We may use a test management tool to document the test cases and then export them into an easily distributable format like MS Excel or PDF. Sometimes, we may also create the tests in Excel directly specifying the test name, description, parameters and other data in the spreadsheets. And then finally, we may automate a part (or all) of the tests.
So after the tests are automated and when executing the tests, we need to specify which tests to run and the test data (parameters). In Quality Center, it involves creating a test set (kind of like a test suite) and then adding tests to it. If they are automated in QTP which integrates with QC, they can be executed by running the test set. But we have started using Selenium WebDriver for its cross browser capabilities and that means we need to specify the tests in a format that can drive the Selenium tests. We use TestNG as the framework for test execution, assertions and reporting.

Extending TestNG

TestNG is a great framework for test execution and its input can be in form of an XML file that specifies which tests to run, where to find the test classes/methods, test parameters and a whole lot of other features that gives you a fine grained control over each test execution. However for our UI functional tests, we wanted to be able to specify the tests and test executions in an easily distributable format (like MS Excel) as mentioned above. The good thing is that TestNG provides the capability to extend it so that its input can be created and executed programmatically. So this way, the test specification in Excel files can be parsed and driven through TestNG. And this is exactly what excel-testng does. It externalizes all Excel parsing and TestNG XmlSuite creation so that the focus can be in creating the test classes and methods. Once that is done and Excel specifications created, all that needs to be done is to provide a main method that does this:

   1: ExcelTestNGRunner runner = new ExcelTestNGRunner("input.xls"); // this can be a single file

2: // or a directory, in which case all spreadsheets in that directory are parsed

3: runner.run(); // run the tests

With ExcelTestNGRunner class, you just specify the input location of the Excel file(s) that constitute the test specifications and then call the run() method, which parses the excel file using default parser (an instance of ExcelSuiteParser) into XmlSuites, creates a TestNG object if not already created and then runs them. After that TestNG takes care of test execution and reporting.

Excel Test Specification

ExcelTestNGRunner parses each worksheet in the Excel file(s) into a separate suite using the included parser (ExcelSuiteParser). Each suite can have suite level parameters specified in the worksheet and the test specifications that specify which tests will be run, their name, description, parameters and the test classes. Here's what a test suite in Excel looks like (a demo spreadsheet can also be downloaded from here and can be used as a starting point):

Specifying TestNG tests in Excel File

The top few rows in the worksheet provide the suite information. ExcelSuiteParser looks for the string "Suite Name" and retrieves the name of the suite from next cell in the same row. Similarly, it looks for string "Suite Parameters" and retrieves suite parameters from the next cell. "Suite Configuration" is not currently used. You can customize the location of these values by providing your own map to the parser. See "Customizing Input" for more details.

To retrieve the tests that will be executed, it looks for the 1st row containing "Id" in the 1st column. This will be the header row, below which each row is a separate test. The header row and tests must have columns specifying Id, Test Name, Test Parameters and Test Configuration (which specifies the classes containing the Test methods). If "Id" is left blank, the test will not be added to the suite. Finally, it parses each row under the header row into a TestNG XmlTest and adds it to the suite. The test specifications are provided as:

• Test Name: generated by concatenating Id & Test Name
• Test Parameters: retrieved from "Test Parameters" column and need to be provided in valid properties (<key>=<value> etc.) format. You can also specify functions as parameter values and then add the logic to parse and evaluate the functions in your test classes (maybe a Base Test class)
• Test Classes: specified under "Test Configuration" column as classes property. Currently, you can only specify a single test class of which, all @Test annotated methods will be executed as a part of this test execution. I'm going to add the ability to select the test methods in later releases.

Customizing Input

If your test cases are specified in Excel but in different format, there are 2 levels of customizations you can do with ExcelTestNGRunner on how to parse the input spreadsheet(s):

1. Custom Parser Map (currently not implemented): You can use the in-built parser but specify your own parser map, which tells the parser where it can find the suite and test data
2. Custom Parser: You can create your own parser by implementing IExcelFileParser interface. You need to parse the spreadsheet file and return a list of TestNG XmlSuites.

ExcelTestNGRunner also provides helper methods to customize the TestNG object it uses to execute tests. For example, you can specify any custom listeners using addTestNGListener() method. If you need to have more control, you can create your own TestNG object and then pass it to ExcelTestNGRunner. Please see javadocs for more details.

Putting it together

You can see the project at https://github.com/randomsync/excel-testng-demo for a complete working demo of excel-testng to parse the input test specifications in Excel. It uses Selenium WebDriver to automate the testing of basic Google search functionality.

Update (3/12/2012): I'm now using google code to host this project because of the provision to host the downloadable jars, javadocs and wiki easily. I'll keep it synced with github but you can find the project documentation and downloads there.

Friday Lunch: Content-Type, Content-Disposition HTTP Headers and MIME type handling in IE

We have a scripting tool that is used to create web-based scripts that simulate application scenarios. These scripts are run through a monitoring tool on a continuous basis to assess the health of these applications. The scripts & scripting tool are also used to validate the scenarios if the monitoring tool issues an alert. The scripting tool uses IE to run the HTTP request/response scenarios. When recording the script for a recent new application, it was noticed that after submitting the request which was an HTTP POST with XML data, the tool was bringing up a dialog box to save the response file, something you’d get when clicking on a HTML link in the browser to a file (maybe a word .doc etc) and if you don’t have an application to open that kind of file. We checked the response and it contained a standard XML that should usually be displayed within the browser (like if you click on Amazon’s AWS service). The fact that it was bringing up a download prompt and not displaying the XML in the browser was a problem because the non-technical team that uses it to validate the application will not be able to determine whether the script ran successfully or not.

The application is a servlet that takes the initial POST request and returns an XML response. After further investigation with the tool and looking at the response headers, it was found that the application was returning “application/x-www-form-urlencoded” in “Content-Type” header erroneously. I haven’t found something specific that would say that content type is not correct for HTTP response, but it is more common to use this content type when submitting form-submission data that is URL encoded (see here).

Investigating with WebScarab

My first conjecture was that the file download prompt was being displayed because of this Content-type header’s value. If we could intercept the response and replace its value with something like “text/xml”, the tool or browser will be able to display the content within the application instead of bringing up the file download prompt. And to validate that, I used WebScarab to intercept the request/responses. But for some reason (maybe because of invalid Content-Type header), it didn’t intercept the response. It would intercept the request and after accepting, it would just complete the request and we’ll end up seeing the file download prompt in the tool. Clearing the “Only MIME-Types matching:” didn’t help.

So the workaround was to write a BeanShell script within WebScarab to replace the “Content-Type” header with a proper value instead of manually intercepting the response. That turned out to be quite straightforward. All we needed to do was put:

response.setHeader("Content-Type","text/xml");

in WebScarab’s Bean Shell interface under the fetchResponse method. As is obvious, this replaces the value of Content-Type header in the response to “text/xml”.

Once we tried it out, it worked and the browser displayed the XML response without bringing up the file download prompt. This proved that the Content-Type header was the problem. But what exactly was the problem? What makes the browser display a prompt to the user to download a file instead of just displaying the content? Further searching revealed this article: http://www.jtricks.com/bits/content_disposition.html

So basically the Content-Disposition header along with Content-Type specifies the MIME type of the file returned and whether the browser should display it or prompt the user to download it. The RFC has all the details but the gist is that the Content-Disposition header can be set to “attachment” for the browser to display the file download prompt. But the response we were getting didn’t have a Content-Disposition header. And IE was still displaying the file download prompt. Further reading warranted.

How does IE handle MIME types in a response, anyways?

I kept on searching and reading on how and when a browser brings up the file dialog prompt. The Content-Type header still bothered me. Then I came across this MSDN article: MIME Type Detection in IE. It provides details on what steps IE takes to detect the content-type of the content before deciding what to do with the file:

The purpose of MIME type detection, or data sniffing, is to determine the MIME type (also known as content type or media type) of downloaded content using information from the following four sources:

• The server-supplied MIME type, if available
• An examination of the actual contents associated with a downloaded URL
• The file name associated with the downloaded content (assumed to be derived from the associated URL)
• Registry settings (file name extension/MIME type associations or registered applications) in effect during the download

If you look at the known MIME types for FindMimeFromData, “application/x-www-form-urlencoded” is not one of them. It is also not an ambiguous MIME type ("text/plain," "application/octet-stream," an empty string, or null) either. So with the MIME type as unknown,

If the "suggested" (server-provided) MIME type is unknown (not known and not ambiguous), FindMimeFromData immediately returns this MIME type as the final determination.

And what happens after that is in this MSDN article: Handling MIME Types in IE. It doesn’t mention exactly what happens if the MIME type is unknown, there’s no application associated with it in the registry and there’s no Content-Disposition header. But I’m guessing that the only thing for it left to do is to let the user decide by showing the file download prompt as was happening in our case. And we were able to prove that. I looked up the registry key for “text/xml” under HKEY_CLASSES_ROOT\MIME\Database\Content Type and added it to a new key “application/x-www-form-urlencoded”. The new registry key looked like this:

[HKEY_CLASSES_ROOT\MIME\Database\Content Type\application/x-www-form-urlencoded]
"CLSID"="{48123BC4-99D9-11D1-A6B3-00C04FD91555}"
"Extension"=".xml"
"Encoding"=hex:08,00,00,00

This tells IE that for responses received with Content-Type equal to “application/x-www-form-urlencoded”, use extension “.xml” and open it however XML files are opened (which is to display them inline). Once added, we went back to the tool and ran the script. And this time, it displayed the response xml without bringing up the download prompt.

So through this exercise, I learned about how IE displays different MIME types and about Content-Type and Content-Disposition headers.

QTP Automation: A Generic Function to perform mouse click

For all the automation testing I’ve been doing in recent months, I’ve built a set of function libraries in QTP that contain functions doing a lot of different things from clicking on a button, validating properties of different web objects to parsing data in excel files. One of these libraries contains all the WebBrowser related functions and one of those is BrowserButtonClick. It takes the title of the browser and name/index of the button as input and does the obvious – clicks on the button within the specified browser. It does all that through descriptive programming, not needing to have any objects in the GUI repository and gives me the advantage of reusing this library framework in different projects/applications. This has allowed me to automate functional tests of different applications very quickly.

But this post is not about that function or about the framework. I’ll write about that later - how it all fits together to provide an automation framework that can speed up functional automation. In this post, I want to share another QTP VBScript function that I wrote to perform a mouse click on any web object. This function forms the basis of descriptive programming that I’ve used in my libraries and provides greater flexibility instead of tying it down to a specific object type.

Like I described, the BrowserButtonClick takes the name of the button as parameter. This works fine as long as the button object has the name property defined (which most of them do) and I know it when creating the tests. For one of the application page however, there were 2 buttons with no name specified but with “html id” property. This led me to come up with another function that will take the property names (“html id” or “name” etc) and property values as input parameters, search for a button object with specified properties and perform the action (left button click) on that. But thinking about it a little more, since the class of the object (“micclass”) is also another property of the object, it doesn’t have to be restricted to a “WebButton”. So the end result was this generic function that takes the list of appropriately delimited properties and values, finds the object and performs a left-click on the object. The input parameter is in the format “propName1=propValue1;propName2=propValue2…”. For example, “micclass=WebButton;html id=Search”. This way, it can be used to perform a click on any object and similar functions can be used to perform any action on any GUI object.

Here’s the function.

Part 1: Function Definition

' Function BrowserObjectClick
' ------------------
' Perform a mouse click on an object within the specified Browser
' Parameter: browserTitle - Title of the browser
' Parameter: props - a list of name-value pairs of properties and their values, semi-colon delimted. ("propName1=propValue1;propName2=propValue2…”)
' Parameter: objIndex - the index (0-based) of the object that needs to be clicked (if multiple objects match)
'@Description Perform a mouse click on the object that matches specified properties within the specified Browser. Returns 0 is successful, -1 if no object is found or the object is disabled
'@Documentation Perform a mouse click on object that matches <props> properties within <browserTitle> browser
Public Function BrowserObjectClick(ByVal browserTitle, ByVal props, ByVal objIndex)

As I already explained above, the function performs a mouse click on any object that matches the specified properties within the specified browser. The browser title needs to be provided as input parameter and so do the properties, which are name-value pairs delimited by semi-colon. For example, to find a WebButton object with name “Search”, the input will be “micclass=WebButton;name=Search”. For a link with outertext property “Log Off, the input will be “micclass=Link;outertext=Log Off”. If multiple objects match the identification properties, objIndex (0-based) will specify which one of those objects is clicked.

Part 2: Getting the browser objects

    On Error Resume Next
    If (props <> "") Then
        indx = CInt(objIndex)
        If (Err <> 0 Or indx < 0)Then
            indx = 0
            Err.Clear
        End If

        'Get the browser objects
        Set desc = Description.Create()
        desc("micclass").Value = "Browser"
        Set bObjs = Desktop.ChildObjects(desc)
    
        For i = 0 To bObjs.Count -1
            If Instr(1, bObjs(i).GetROProperty("title"), browserTitle, 1) > 0 Then

First we make sure that the properties are not blank in the input parameters. If they are blank, there’s not much we can do so we just exit out of the function. Next thing to do is to make sure the objIndex is an integer >=0. In case it isn’t, we take the default value of 0, which means that the first object matching the properties will be clicked. Then we get all existing browser objects and iterate through them until we find the one with matching title. If none are found, we’ll exit out with appropriate return values. If multiple browsers are found, we’ll use the 1st one that matches the title. In the applications I’ve automated so far, I haven’t come across a case where multiple browsers match the title…but if needed (in case the application pops up multiple windows, for example), the function can be easily modified to handle that.

Part 3: Getting the Object and performing the Action

                desc("micclass").Value = "Page"
                Set pObjs = bObjs(i).ChildObjects(desc)
                'Make sure you got one
                If (pObjs.Count > 0) Then
                    'load the properties in an array
                    Dim propArray : propArray = Split(props,";",-1,vbTextCompare)
                    Dim prop
                    Dim propsDict    'As Scripting.Dictionary
                    Set propsDict = CreateObject("Scripting.Dictionary")
                    For Each prop in propArray
                        propsDict.Add Split(prop,"=",-1,vbTextCompare)(0), Split(prop,"=",-1,vbTextCompare)(1)
                    Next
                    
                    'Get the Object
                    Dim j, pKeys : pKeys = propsDict.Keys
                    For j = 0 to propsDict.Count-1
                        desc(pKeys(j)).Value = propsDict.Item(pKeys(j))
                    Next
                    Set propsDict = Nothing
                    Set objs = pObjs(0).ChildObjects(desc)

So once we find the browser with specified title, we get its “Page” child objects since all other objects are child objects of the page object. We split the properties that we got as input by the semi-colon “;” and load it in an array. At this point, the array is populated with the name value pairs of the properties delimted by an equals sign “=”. I’m not doing much error handling here because I trust the input provided will be appropriately delimited.

Next, we split each of the array values by an equals sign “=” and load it in a “Scripting.Dictionary” object as key-value pairs, where the key is the name of the property. We create a Description object, load all the properties in it and find all matching child objects.

Part 4: Performing action on the object

                    If (objs.Count > indx) Then
                        'Execute the event
                        If (objs(indx).GetROProperty("disabled")) Then
                            BrowserObjectClick = -1
                        Else
                            objs(indx).Click
                            BrowserObjectClick = 0
                        End If
                        'Free up the objects
                        Set desc = Nothing
                        Set bObjs = Nothing
                        Set pObjs = Nothing
                        Set objs = Nothing
                        Exit Function
                    End If

Now that we have a collection of all objects that match the properties, we’ll make sure that the index is not more than the number of objects returned, that it is not disabled and then perform the click action. If it is disabled, we’ll return a –1 so that caller can handle it appropriately. If not, we perform the mouse click by calling the click method of the object and return a 0. Finally, we do some cleanup and exit function.

PostScript

This function provides a generic method to implement a mouse-click on any object using descriptive programming and is particularly useful where the standard properties of an object (name) are not available. It can be further extended to implement any action, not just clicks and provide other functionality as well.

If you have any questions or suggestions to improve or simplify the function, let me know.

Real Life Testing Scenario – Bank ATMs

I think a lot of real life defects you come across would have to do with Bank ATMs. Maybe because of complexity of coding the human-machine interface logic, the plethora of usage scenarios and/or real-time nature of the transactions that it is really hard to make a defect free (or almost) ATM machine.

Yesterday I faced one such scenario when trying to deposit some cash in an ATM machine. This was one of those BoFA ATMs that don’t need an envelope to deposit. You just put the cash in a slot in the ATM and it does the rest. It scans and counts the bill and tells you the total amount deposited.

So as I put all the cash in the slot, it closed the slot door and scanned and counted all the bills. But after it was done, It gave me a message on the screen that some of the bills couldn’t be accepted and opened the slot door for me to take the bills out. There was only 1 bill there and all others were accepted. I took out the one bill it couldn’t accept and it closed the slot door and showed the amount that it had accepted correctly. Pretty neat…so far.

On the same screen, it asked whether I wanted to add more money to the deposit. I pressed “Cash” and it opened the slot door again. After pressing and straightening the bill it had rejected previously. I put that in the slot. It didn’t accept the bill this time either. It gave me the error message and opened the slot door. I took the bill out and it closed the door. But…the error screen didn’t go away. It continued beeping and complaining that it couldn’t accept the bill even after it had closed the slot door. There were no options available…cancel the transaction, return the card…nothing. Just the error message and no buttons to select. At this point, my card was inside the ATM and so was the cash and I had no proof of the deposit! I didn’t have the customer service number because it is printed on the back of the card.

Well, after fruitlessly pressing random buttons including the cancel button and including keeping the cancel button pressed for sometime, I gave up. I approached the lady at next ATM slowly lest she thought I’m going to mug her. But she probably saw the ordeal I was going through and gave me the customer service number from the back of her card. And after spending about 40 minutes on the phone during which I faced numerous call transfers and 1 disconnect and had to talk to the amazingly annoying automated voice response system, I got to talk to a real person who seemed to understand what I was going through and gave me a temporary credit pending research. The previous card was rejected and a new card will be sent.

So…here are the steps:
1. Deposit Card in the ATM slot and go through entering the PIN etc.
2. Select “Deposit”…and “Cash”
3. Once the cash slot door opens, enter several bills at least 1 of which will definitely be rejected.
4. After it complains about the bills, take the bill(s) out.
5. Once the slot door closes and it counts and shows the amount, press the option to add more money to the current deposit.
6. After the slot door opens, add the rejected bill(s)
7. After it shows the error screen again and opens the slot door, take out the rejected bill.

The expected result at this point is for the machine to detect that I have taken the rejected bill out, close the slot door and continue with the amount that was previously determined. It should show me the options again that were shown at Step 5 and let me choose whether I want to try adding more cash again or just go ahead with whatever amount that has been added already. No matter how many times I add previously rejected bills, it should either accept the bills or reject them. This seems to be a normal usage scenario to me, which should’ve been tested thoroughly. This kind of scenario is very likely to happen and it’s not something a user would have to try real hard to go through. I honestly doubt that this issue can be replicated again by following the exact same steps. There has to be some other variable which caused the machine to malfunction.

If I had to test the scenario, there are several things I would want to know:
Firstly, how does the bill scanning logic works. How is it decided whether a bill should be accepted or rejected. Once I know that, I can test with several combinations of different amounts, with different combinations of good and bad bills and different degree of bad quality bills. Maybe also with junk (coins? fake bills? other paper? leaves? lint?)instead of bills!
Secondly, how does it detect whether the bills have been added or removed from the slot to close the slot door. Based on this, I can test with different number of bills to validate if it detects the change.
There are a lot of other exception scenarios also that come to mind…for example, what if somebody props up the slot door open by jamming something through the door? What if somebody continues to add rejected bills 5 times? 10 times? 100 times? (of course, not all the scenarios are realistic so the likelihood of the scenario would have to be weighed against its criticality).

It seems that in this case, it did recognize that I had taken the rejected bills out because it closed the slot door. This is assuming that it closes the slot door only after it detects that the bills have been removed. If it closes the door after a fixed time (I hope not), then this statement is invalid. Anyways, assuming it detected the change, it seems to have failed to continue to the next step which is to carry on with the transaction and provide the options to the user to choose what to do next. This is exactly where I would look to identify the problem by running various scenarios or a combination of scenarios.