LoadRunner: Creating manual web_custom_request for HTTP file uploads

There was a recent forum post in Advanced LoadRunner yahoo group asking how LoadRunner captures the file upload requests and if there was a way to customize that request. At that time, I replied that it may be possible to generate a manual web_custom_request by using the underlying raw HTTP request. I hadn't tried it then but it sounded interesting enough to be marked as to-do. I recently got some time to try it and it was fun.

Problem:

As mentioned in the forum post, LoadRunner captures any form based file uploads as web_submit_data with the file name (and location) within the list of data within the request. An example request:

web_submit_data("FileUpload",
        "Action={URL}",
        "Method=POST",
        "EncType=multipart/form-data",
        "TargetFrame=",
        "RecContentType=text/html",
        "Mode=HTML",
        ITEMDATA,
        "Name=File", "Value=C:\\testdata\\readme1.txt", "File=yes", ENDITEM,
        LAST);

There are 2 obvious drawbacks to this:

1. For this kind of script, the files to be uploaded have to be transferred to each of the Load Generator machines being used in the scenario. Or they have to be transferred to a shared network location so that both the machine used for creating the script and the load generator machines can reference it.

2. As mentioned in the forum post, any parameterization in the file contents is not possible. If the scenario requires files to be uploaded iteratively with unique names, that many files have to be created manually.

Solution:

There are 2 solutions to this and one is more universally acceptable (by web servers) than the other. I'll describe both and let you decide which one you want to use. I also highly recommend reading RFC1867 which provides excellent background on how multipart file uploads are implemented in HTML form based submissions.

1. web_custom_request POST: To arrive at the solution, I did exactly what I usually do when I'm stuck with some HTML script or want to see the raw HTTP request being sent to any web server. I opened Webscarab to capture the HTTP request that was being sent for file uploads. (Webscarab related post here).

Here's a screenshot of the HTTP post request as captured by Webscarab:

  

And here's the relevant portions of the raw request:

Content-Type: multipart/form-data; boundary=---------------------------327572003712859
Content-length: 228

-----------------------------327572003712859
Content-Disposition: form-data; name="File"; filename="readme1.txt"
Content-Type: text/plain

testdata
readme
readme
123456789
-----------------------------327572003712859--

The Content-Type HTTP header is what specifies the submission to be a multipart data submission. The "boundary" is a string that doesn't occur in the file data and is used to mark the boundaries of the data being sent. Content-length, as the name implies is the length of the data that is being sent.

The body of the request starts with the boundary string and is followed by the 2 headers to specify the content-disposition which includes the name of the file to be uploaded. Content-type specifies the encoding of the data being submitted. Following that within the body is a blank line and the actual contents of the file that will be uploaded. After the contents is the boundary once again to signify end of the submission data.

So this kinda makes it easier to create a web_custom_request. The Content-Type HTTP header is specified by "EncType" attribute of the function. Content-length header can be ignored since web_custom_request itself generates it by calculating the length of the data at runtime. The rest goes in the "Body" attribute starting with the boundary. Content-Disposition specifies among other things the name of the file to be uploaded and this can be parameterized. Content-Type is the type of file being uploaded and can be text/plain, image/gif etc. And in the end is the boundary once again. The final request looks like this:

web_custom_request("{rndString}.txt",
        "URL={testURL}",
        "Method=POST",
        "EncType=multipart/form-data; boundary=---------------------------17773322701763",
        "TargetFrame=",
        "Resource=1",
        "RecContentType=application/octet-stream",
        "Body=-----------------------------17773322701763\r\n"
        "Content-Disposition: form-data; name=\"File\"; filename=\"{rndString}.txt\"\r\n"
        "Content-Type: text/plain\r\n"
        "\r\n"
        "testdata\r\n"
        "readme\r\n"
        "readme\r\n"
        "{rndString}\r\n"
        "-----------------------------17773322701763--\r\n",
        LAST);       

I have parameterized the file name to be a random string so I can run this script iteratively without having to worry about server rejecting duplicate files. I can also parameterize the contents of the file within the body to whatever I like so that each file is unique or based on some other logic.

2. web_custom_request PUT: The second option is more straightforward but less universally accepted. I've read that not all web servers implement this. It involves using the HTTP PUT to create a web_custom_request. For more on different HTTP methods, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html. It mentions that:

The fundamental difference between the POST and PUT requests is reflected in the different meaning of the Request-URI. The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended and the server MUST NOT attempt to apply the request to some other resource.

I've also read that PUT is more efficient so if your test site implements the uploads using POST, better do it that way instead. Nevertheless, here is the web_custom_request with PUT:

web_custom_request("{rndString}.txt",
        "URL={URL}/{rndString}.txt",
        "Method=PUT",
        "Resource=1",
        "RecContentType=application/octet-stream",
        "Body=testdata\r\n"
        "readme\r\n"
        "readme\r\n"
        "{rndString}\r\n",
        LAST);

The resulting function is straightforward, URL attribute is the URI identifying the file to be uploaded and Body is the exact contents of the file.

Of Code Reviewers and Food Critics

For last 2-3 months, I was involved in testing our in-house customizations of Sun's IdM product. My responsibilities included not only leading the QA effort for the customizations but also to support the development effort by reviewing and suggesting improvements to the code/design.

So engrossed in my newfound glory as a unit tester/code reviewer, I realized that it may be easy for me to look at the code and find out a problem with it. But if I had to write something like that myself, I'd end up spending much more of my mind and time than it usually would take an average programmer. That realization reminded me of something I had recently heard in a movie.

The movie was "Ratatouille" and the scene was when Anton Ego starts writing his critique after visiting Gustaeu's Restaurant and having a meal cooked by the new but unknown chef Remy:

In many ways, the work of a critic is easy. We risk very little yet enjoy a position over those who offer up their work and their selves to our judgment. We thrive on negative criticism, which is fun to write and to read. But the bitter truth we critics must face, is that in the grand scheme of things, the average piece of junk is more meaningful than our criticism designating it so.

The parallels between a food critic as described above and what I was doing seemed surrealistically close, so much that I had to finish this post (after realizing it had been lying in my drafts for a few weeks). In some ways, that was exactly what I was doing. I enjoyed a position over those who wrote the code because finding out a problem with a piece of code drew much more attention than the original act of writing it. I thrived on negative criticism because it would get immediate credit, instead of the work of a developer which was the reason my role existed in the first place.

There are some fundamental differences as well. One being that Anton Ego, as a food critic does not have to care about the success of the Restaurant he is reviewing. This affords him the luxury of being overtly aggressive and have a dismissive attitude. My role on the other hand (both as a tester and code reviewer), is as responsible for the success of the product as any other stakeholder. That is the reason I'm as much pained when I find an issue in some code that I previously have reviewed and may have overlooked as the developer with whom I have to revisit the information once again. But the basic responsibility of both is still the same: to challenge the creator of an artifact to produce a better product which aligns with or exceeds the expectations of the users/consumers.

----------------------------------------------------------------------------------

The project is nearing its completion now and in hindsight, this is what I have to say:

This was a dream project for me. The kind of project that comes like once in a few years and offers amazing challenges with tremendous learning opportunities. Right kind of people, right kind of responsibilities and right kind of control over what I wanted to do. As much as I enjoyed working every moment in this project, I have to realize that in my enthusiasm, I may have stepped over some boundaries. I may have offered some unwarranted advices that were probably irrelevant or I may have criticized something to hide my lack of competence. I will take full responsibility for all the misgivings I caused, in however form I may deserve. But I do hope it will be realized that most of whatever I did was aimed at the common cause of project's success, and the fact that I enjoyed every bit of it and tried to make it enjoyable for everyone else should make it all the more worthwhile.

Calculating Virtual User Ramp up and Pacing for Load Tests

I was working on performance testing for a project not long ago that had different VUser types each with different load requirements. This meant that in one load test scenario, there were several user groups having unique transactional load characteristics (like desired TPM), not based on the whole scenario but by individual groups. At that time I created a simple spreadsheet to calculate the ramp-ups, pacing and actual load for each of the group and totals given the expected load and the number of virtual users. I recently got some time to improve it a little and am sharing it now.

So here's that table with some sample values:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	100	1	30	2.00	2	200	30.00
2	34.2	14	1	34.2	1.75	2	28	30.00
3	24.6	14	1	24.6	2.44	2	28	30.00
4	19.8	10	1	19.8	3.03	3	30	20.00
5	1.5	1	1	1.5	40.00	40	40	1.50
6	9.3	5	1	9.3	6.45	6	30	10.00
7	6.9	3	1	6.9	8.70	9	27	6.67
8	34.2	14	1	34.2	1.75	2	28	30.00
9	24.6	14	1	24.6	2.44	2	28	30.00
Total	185.1	175		185.1				188.17

A cleaner Google Spreadsheet read-only version is available here;
and an excel spreadsheet for your own use can be downloaded from here

How to use this table:

Basically, this table represents one load test scenario. So each of the row entry is a separate VUser group that is included in this scenario. Under VUser Group Column, the name of user group can be entered. If less or more VUser groups are needed to include in test scenario, rows can be removed or added accordingly.

There are 3 values that need to be entered for each user group: the base load that is desired for that user group, the number of virtual users that are to be used, and a multiplier (which in most cases can be 1). These are explained below:

Base Load: This is the load (in transactions per minute) at which that particular user group will be run. This is a user provided value based on actual requirements of the load test. The Total in last row gives the total load (in TPM) for this whole scenario. The total may or may not mean much but it does provide the total TPM at a glance.

Number of Virtual Users: This is the number of virtual users that will be used to run that user group. This is also a user provided value. Determining this will be a little trickier and there will have to be some estimation involved if you have a small number of VU licenses available (as I do). You will need the estimated time taken by a complete iteration of the transaction represented by this VUser group. Once you have that, you'll have to use enough virtual users to keep VU Pacing of this group above that number. Actually, you should use more Virtual users than that because in a load test, transaction time will vary depending on how stressed the test servers are and may be higher than the times you see when running a single user.

Let me use an example:
For my user group 1, I need to generate a load of 30TPM and I know by running the script with a single user that it takes around 10 seconds to complete one iteration. I want to keep the pacing much above 10 seconds, say at least 60 seconds. Its not difficult to see that I'll need to use 30 users to achieve a pacing of 60 seconds. I can just put 30 in the Number of VU Column and find out what the ramp up and pacing values will be.

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	30	1	30	2.00	2	60	30.00

Also, if more virtual users are available, they can be used based on the requirements and/or a desire to increase the VU pacing even more. So if I want to use 100 users for this user group, the numbers will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	100	1	30	2.00	2	200	30.00

On the other hand, if I have only 15 users available for this group, the values will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	15	1	30	2.00	2	30	30.00

Now to achieve 30TPM with 15 users, I will have to use a pacing of 30 seconds which is much closer to 10 seconds. But in real life with limited number of virtual users available, I may have to go with these values and hope that the iterations don't take more than 30 seconds during the test.

There is another thing to keep in mind: With this table, the values will be accurate if the expected load for each user group is less than 60. If it is more, the expected ramp up will be less than a second and it will be rounded off to 0 or 1. If it is zero, you'll get a division by zero error in Actual Load Column and if it is 1, you'll get the maximum Actual Load of 60.

So the thing to do if you want to calculate the values for loads above 60TPM is to break the group into 2 different groups. For example, to generate the values for 90TPM load, you can break it up into 2 groups of 60TPM and 30TPM. That way, you can generate 90TPM accurately. Other thing that can be done is to ramp up more than 1 user per interval. So with 90TPM, the expected ramp up will be 0.67 per user which means 3 users in 2 second ramp up scheme. This calculation will have to be done manually as the spreadsheet doesn't take care of fractional ramp up values and rounds them to nearest integer value. (see more of this under Actual Ramp Up)

Multiplier (Load Factor): This column solely exists because I had to run different tests at different loads (for example, a stress test that was run at 5 times the load of general load test) and I wanted to quickly find out how many users will be needed at different load factors and what will be their ramp up and pacing values. For example, if I had a user group with 1.5TPM load and 1 user with a pacing of 40 seconds and wanted to find out how many users will be needed to have the same pacing but with 5 times the load, I can use a Load Factor of 5 to find out that I'll need 5 users to maintain a pacing of 40 seconds, as shown in table below:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	1.5	5	5	7.5	8.00	8	40	7.50

So based on these 3 values entered for each user group, the remaining values, namely Ramp up time, Pacing and Actual TPM achieved are calculated.

Expected Load (TPM): This is calculated by simply multiplying base load with the load factor.

Expected Load (TPM): This is the theoretical ramp up time that is required to achieve the expected load and is calculated by dividing 60 by expected load. For example, if expected load is 30TPM, the ramp of course will be 60/30 = 2 users per second.

Actual Ramp up: Since the theoretical ramp up calculated in previous column can be a fractional value (indicating millisecond ramp up times) and we can have actual ramp up times only at a 1-second granularity, the actual ramp up is calculated by rounding off the expected ramp up to nearest integer. For example, if the expected load is 18TPM, the expected ramp up will be 3.33 seconds. This has to be rounded off to 3 seconds. This has the obvious effect of making the Actual Load different than the desired load.

Another case to watch out for, as I mentioned earlier is when the expected load is above 60TPM. In this case, the expected ramp up time is always going to be less than 1 and the actual ramp up will get rounded up to 0 or 1. The maximum Actual Load will be 60 as I'm not accounting for ramp ups greater than 1 user per second.

However, for both the above cases, there are 2 ways you can achieve the actual load that is closer to the desired load:

1. Break the group into multiple groups: For example, a 90TPM group can be broken into 2 groups - one with desired load of 60TPM and other with 30TPM. This way, an actual total TPM of 90 can be achieved.

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	60	60	1	60	1.00	1	60	60.00
2	30	30	1	30	2.00	2	60	30.00

In case of 18TPM, it can be broken into 2 groups of 15 and 3TPM. The values will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	15	30	1	15	4.00	4	120	15.00
2	3	6	1	3	20.00	20	120	3.00

2. More users per second ramp-up: Another way is to have multiple users per ramp up interval. For example, 90TPM with a single user group can be achieved by having a ramp up of 3 users every 2 seconds. Similarly, 18TPM can be achieved by having a ramp up of 3 users every 10 seconds.

I usually prefer the 1st method and choose the start time so that the ramp ups are even between the multiple groups of same user type. However it does make it more tedious as I have more user groups to keep account of. The 2nd method has the drawback of introducing a relative burst of users every ramp up interval.

VU Pacing: This is the interval between iterations and is calculated by multiplying the ramp up value with the number of users. Again, you have to make sure that the pacing value is comfortably above the single user transaction time; otherwise if the transactions take longer than the pacing value and next iteration is started immediately after the previous iteration, it may effect the load pattern.

Also, I'm assuming that you are pretty much content with having a stable pacing. If you want to add certain randomness to the the pacing times, that can also be done using this value. See my previous post for more on this here.

Actual Load: Finally, the Actual load is calculated by dividing 60 by the actual ramp up value. This is the actual load that you can hope to generate during your load test for this user group. What is actually generated during the load test depends on a lot of other factors like how stressed your system is etc. This can be found out by analyzing the results of the load test.

So that's that. It turned out to be a longer post than I expected, but hopefully it helps. If you have any comments or think of any suggestions to improve it, let me know.

Rubik Timer

I'm currently working on writing a Java based Rubik Timer. A Rubik timer is a specialized timer that can be used to track the time it takes to solve a Rubik's Cube. There are some implementations available currently but the ones that I tried lacked some much needed features. And since the current projects I'm working on are not proving challenging enough, I needed something else to keep my life exciting.

Even though I've written some code already and am faced with some design decisions, it came to my mind that it'll be a good opportunity to put myself in different shoes - as a requirements/business analyst and document the requirements, as a programmer/designer and design/develop the solution, as a tester and test the implemented solution and finally as a user and use the application. Inextricably entangled with all these roles will be a project management role that I will have throughout the project. So I plan to follow and document the project as it goes through its own lifecycle, in an attempt to learn more about different phases that a project goes through. In the process, I also want to understand the roles of all these different actors and the challenges they face. I know it is easier said than done and at times, it may seem like an undiscerning adventure. But I want to at least give it a try and even though the project is smaller in terms of effort required as compared to real life ones, I believe that it'll prove to be an enjoyable and learning experience. And the fact that I'll be playing all those roles will make it as challenging as (or even more than) those I face in real-life.

In upcoming related posts, I plan to detail the progress as I find time to work on this project. As it seems, resource availability will be the toughest challenge this project faces since my current projects, even though minimally challenging can be substantially time consuming and resource (in this case, my mind!) draining. And every now and then, smaller challenges have a tendency to pop-up and provide exciting distractions.

Business Case for Testing Tools & Automation

This post had been lying in my drafts for a long time. For some reason, I decided not to publish it. But it's still relevant...except that I have submitted the business case since then and it's lying at somebody at Finance's desk or email Inbox for review/approvals.

----------------------------------------------------------------------------------

I'm currently working on a business case to invest in testing tools and functional test automation. The case will be sent to upper management and finance for approval and once they do, we're going to start working on building an automation framework. I foresee a lot of hurdles and am not keeping my hopes high based on the tightening of budget and how the business case is coming out to be.

One of the things that I have to add in the business case is financial analysis which includes an estimate on return of investment. And it has been a very "enlightening" process. Based on the current estimates on the capital and labor costs, it'll take us 2.9 years to get a return on investment. And of course, it's based on a lot of assumptions and I'm kind of skeptic when claiming that number for ROI.

You see, one of things that are never in short supply at my workplace is the list outstanding tasks. And with that comes extreme work pressure and very little time to invest in exploring new technologies and coming up with new ideas. I'm quite sure that designing the automation framework will be a time consuming and learning process and I will have to be devoted at least half-time (if not more) in this effort. And I'm quite sure that when we actually start working on that effort, the estimates that I put in for labor in the business case will start looking like best case scenario.

The brighter side is that I have now under my belt the experience of creating a business case. And once/if approved, we'll be undertaking the effort of functionally automating the test cases for our applications. We're also planning to expand our QA service by offering the testing tools and/or automation to other teams.

My colleague and I had been talking about the design of the framework since we first conceptualized the idea of automation. We had this whole idea in our minds on how to make it reusable, to maximize customizability with least rework. He has experience in building these kind of frameworks and I have had a brief encounter at my last job. I recently looked up online and found that it is now one of the established practices, one of the buzzwords surrounding test automation process - Keyword Driven Test Automation Frameworks. It must have been a long time since both of us were in automation business because what I read matched what we had been planning to do. Seems like a lot of people have written their experiences with building automation frameworks with this idea. Seems like we'll be able to use some of the knowledge to our benefit.

Coming up...

A few weeks ago, I spent some time going over JMeter and building some advanced web test plans. But then I got distracted and buried under some intense projects, including a 1-week training on a product that I didn't know much about and that I have to be an advanced user of within a few weeks. (BTW...if you want to know why attending a 1-week advanced training on a product without having a solid background is a bad idea, let me know).

Now I have some time again but it's right before my vacation and I just don't feel like getting immersed into it again. But when I come back, I plan on posting some of my notes on how to build an advanced web test plan in JMeter. I plan on using a more advanced scenario than the one with basic Get requests, probably something similar to my previous post about using an MD5 library to create a LoadRunner script. If I can't find something else, I'll use that very example. I also plan on exploring some webservices and pure XML over HTTP kind of scripts. Stay tuned...

Oh... and I should also mention my reasons for looking into JMeter. It has been increasingly frustrating to borrow time on LoadRunner Controller from an external team, partially because of having to justify to the project team the additional charge back incurred to the project. I can't believe how many emails I have to write to justify a few 100 dollars. But anyway, turning misery into a learning experience, I wanted to explore if JMeter provides enough capabilities and is robust enough to be used for production performance testing.

LoadRunner Scripting Challenge - LiveJournal

My last few posts have unabashedly been about LoadRunner even though I vowed to diversify my writings into more distinct areas. But this one again is about LoadRunner and is too interesting to pass...

Background:

I had an old journal in LiveJournal back from the times when 'blogging' wasn't a very familiar word or activity and most of the popular blogging sites like Blogger and WordPress (these are the only free ones that I've liked so far) didn't exist. I came across LiveJournal and was immediately hooked. I wrote prolifically at that time and I should also mention that my creative writing abilities then were much superior to my current showcase. But over time, I've outgrown LiveJournal because their free version doesn't provide the most needed features and the one that does is ad-supported. To get rid of the ads, you have to pay a nominal amount. I think that wasn't the only reason and I was a paid member for sometime supporting their efforts. But I thought I needed a more traditional 'blog' rather than a journal. I also haven't been writing much lately and have no communities that I would like to keep track of in LiveJournal.

So the result was some of my most cherished writings were stored in LiveJournal's history of posts and I wanted to import all of them to a WordPress blog. WordPress allows importing an XML file of LiveJournal formatted entries. The problem however is that LiveJournal allows exporting the posts only by month. So if I wanted to export my posts from back in 2002, I would have to export it month by month for over 6 years worth of posts. That was certainly not very exciting. But soon it became pretty exciting when I decided to use LoadRunner to export all my posts month by month in XML format and I learned something very interesting in the process - which is of course the whole idea of this post.

Challenge:

When I recorded the script by going to the export page and logging in, I noticed in the script that my password wasn't hard coded anywhere. I expected to find it with the request when I submitted the form containing my username and password. Instead, there were the "chal" field and the "response" field and the password field was blank.

web_submit_data("login.bml",
"Action=http://www.livejournal.com/login.bml?ret=1",
"Method=POST",
"RecContentType=text/html",
"Referer=http://www.livejournal.com/?returnto=/export.bml",
"Snapshot=t5.inf",
"Mode=HTML",
ITEMDATA,
"Name=returnto", "Value=/export.bml", ENDITEM,
"Name=mode", "Value=login", ENDITEM,
"Name=chal", "Value=c0:1193421600:2223:300:DsqGiIDCk0A4hs9sNsmH:c7f09b46d0ba2d82bb68945ea84532fc", ENDITEM,
"Name=response", "Value=b84434e3c2d193c4a1c345d7874a89a3", ENDITEM,
"Name=user", "Value=myusername", ENDITEM,
"Name=password", "Value=", ENDITEM,

Honestly, that was surprising to me. At that time, I couldn't come up with any explanation of how my user ID can be authenticated and successfully logged in without sending the password in form submission. I could smell the hint of another challenge. If you want to try it before reading on to get a better handle, try recording a script on LiveJournal (http://www.livejournal.com) by logging in and making it work.

Solution:

On thinking a little more and looking at the script again, I realized that they could be using JavaScript to generate the response which was some kind of hashed form of the password. On viewing the source of the initial page, I found a js file with this code:

var pass = pass_field.value;
var chal = chal_field.value;
var res = MD5(chal + MD5(pass));
resp_field.value = res;
pass_field.value = ""; // dont send clear-text password!

So on further investigation, I figured what was happening:

1. On initial navigation to any page with the login form, a challenge string is sent to the client. I guess it has to have some kind of expiration time because if you use it sometime later with the correct response, it'll return that the challenge has expired.

2. An event handler is registered with the submit button of the login form.

3. When the submit button is pressed, the event handler takes the field values including the hidden challenge string. It takes care of creating an MD5 digest of the challenge string concatenated with the MD5 digest of the password and clearing the actual password field value. This is the value that you see in the "response" form parameter in the code above. The values are then submitted and the server takes care of validating the response value.

This page explains this in high-level: http://blog.paranoidferret.com/index.php/2007/07/22/secure-authentication-without-ssl-using-javascript/

So to make the LR script work, I had to do 2 things:

1. Get the value of the challenge by correlating appropriately

2. Once I have the challenge string, create the 'response' value by getting the MD5 digest of the challenge string concatenated with digest of the password string.

The MD5 digest calculation is based on the MD5 Message Digest Algorithm by Ron Rivest of MIT. There was no way I was going to write an MD5 function myself. A quick search revealed some implementations of this algorithm. The one I used was from http://www.fourmilab.ch/md5/ because this was in C and saved me from including a lot of legalese. All I had to do was include the header and C file and use the MD5 functions. So this is the outline of final script. I have left out the details which should serve as an interesting exercise.

1. Firstly, get the MD5 digest of the password. This will later be used to concatenate to the challenge string.

MD5Init(&md5c);
MD5Update(&md5c,(unsigned char *)mesg,strlen(mesg));
MD5Final(signature,&md5c);

where mesg is the password string.

2. Correlate the script to save the challenge string at the appropriate place. This is the string that is sent by the server as a hidden form parameter of the login form.

3. Once you have the challenge string, concatenate the MD5 digest of the password (calculated in Step 1) to it. Remember that this digest has to be concatenated as a lower-case hexadecimal number. Get the MD5 digest of this concatenated string.

4. Pass this MD5 digest calculated in Step 3 as the value to 'response' parameter in the login step:

"Name=chal", "Value={chal}", ENDITEM,
"Name=response", "Value={response}", ENDITEM,

That should do it. Now I can parameterize the year and month values in export step and export all my journal entries by month in XML files. It was a simple task to concatenate all the files into 1 big XML file and importing it through WordPress blogging features. I'm a happy WordPress blog user now...we'll just have to see how long that lasts before I start looking for other options.

I guess I should also add that the exercise gave me a great opportunity to learn how web site authentication can be handled without using SSL and how I can script that in LoadRunner.

Note 1: Since creating this script, I explored LiveJournal's server protocol documentation and found that they have documented the authentication mechanism pretty well. Great work in encouraging other users to build custom clients.

Note 2: I felt I should mention that LoadRunner licensing agreement specifically prohibits using the software to run tests on public domain sites. It basically means that you cannot load up your LR Controller with the script you created above and run a load test with any number of virtual users.