Calculating Virtual User Ramp up and Pacing for Load Tests

I was working on performance testing for a project not long ago that had different VUser types each with different load requirements. This meant that in one load test scenario, there were several user groups having unique transactional load characteristics (like desired TPM), not based on the whole scenario but by individual groups. At that time I created a simple spreadsheet to calculate the ramp-ups, pacing and actual load for each of the group and totals given the expected load and the number of virtual users. I recently got some time to improve it a little and am sharing it now.

So here's that table with some sample values:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	100	1	30	2.00	2	200	30.00
2	34.2	14	1	34.2	1.75	2	28	30.00
3	24.6	14	1	24.6	2.44	2	28	30.00
4	19.8	10	1	19.8	3.03	3	30	20.00
5	1.5	1	1	1.5	40.00	40	40	1.50
6	9.3	5	1	9.3	6.45	6	30	10.00
7	6.9	3	1	6.9	8.70	9	27	6.67
8	34.2	14	1	34.2	1.75	2	28	30.00
9	24.6	14	1	24.6	2.44	2	28	30.00
Total	185.1	175		185.1				188.17

A cleaner Google Spreadsheet read-only version is available here;
and an excel spreadsheet for your own use can be downloaded from here

How to use this table:

Basically, this table represents one load test scenario. So each of the row entry is a separate VUser group that is included in this scenario. Under VUser Group Column, the name of user group can be entered. If less or more VUser groups are needed to include in test scenario, rows can be removed or added accordingly.

There are 3 values that need to be entered for each user group: the base load that is desired for that user group, the number of virtual users that are to be used, and a multiplier (which in most cases can be 1). These are explained below:

Base Load: This is the load (in transactions per minute) at which that particular user group will be run. This is a user provided value based on actual requirements of the load test. The Total in last row gives the total load (in TPM) for this whole scenario. The total may or may not mean much but it does provide the total TPM at a glance.

Number of Virtual Users: This is the number of virtual users that will be used to run that user group. This is also a user provided value. Determining this will be a little trickier and there will have to be some estimation involved if you have a small number of VU licenses available (as I do). You will need the estimated time taken by a complete iteration of the transaction represented by this VUser group. Once you have that, you'll have to use enough virtual users to keep VU Pacing of this group above that number. Actually, you should use more Virtual users than that because in a load test, transaction time will vary depending on how stressed the test servers are and may be higher than the times you see when running a single user.

Let me use an example:
For my user group 1, I need to generate a load of 30TPM and I know by running the script with a single user that it takes around 10 seconds to complete one iteration. I want to keep the pacing much above 10 seconds, say at least 60 seconds. Its not difficult to see that I'll need to use 30 users to achieve a pacing of 60 seconds. I can just put 30 in the Number of VU Column and find out what the ramp up and pacing values will be.

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	30	1	30	2.00	2	60	30.00

Also, if more virtual users are available, they can be used based on the requirements and/or a desire to increase the VU pacing even more. So if I want to use 100 users for this user group, the numbers will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	100	1	30	2.00	2	200	30.00

On the other hand, if I have only 15 users available for this group, the values will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	30	15	1	30	2.00	2	30	30.00

Now to achieve 30TPM with 15 users, I will have to use a pacing of 30 seconds which is much closer to 10 seconds. But in real life with limited number of virtual users available, I may have to go with these values and hope that the iterations don't take more than 30 seconds during the test.

There is another thing to keep in mind: With this table, the values will be accurate if the expected load for each user group is less than 60. If it is more, the expected ramp up will be less than a second and it will be rounded off to 0 or 1. If it is zero, you'll get a division by zero error in Actual Load Column and if it is 1, you'll get the maximum Actual Load of 60.

So the thing to do if you want to calculate the values for loads above 60TPM is to break the group into 2 different groups. For example, to generate the values for 90TPM load, you can break it up into 2 groups of 60TPM and 30TPM. That way, you can generate 90TPM accurately. Other thing that can be done is to ramp up more than 1 user per interval. So with 90TPM, the expected ramp up will be 0.67 per user which means 3 users in 2 second ramp up scheme. This calculation will have to be done manually as the spreadsheet doesn't take care of fractional ramp up values and rounds them to nearest integer value. (see more of this under Actual Ramp Up)

Multiplier (Load Factor): This column solely exists because I had to run different tests at different loads (for example, a stress test that was run at 5 times the load of general load test) and I wanted to quickly find out how many users will be needed at different load factors and what will be their ramp up and pacing values. For example, if I had a user group with 1.5TPM load and 1 user with a pacing of 40 seconds and wanted to find out how many users will be needed to have the same pacing but with 5 times the load, I can use a Load Factor of 5 to find out that I'll need 5 users to maintain a pacing of 40 seconds, as shown in table below:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	1.5	5	5	7.5	8.00	8	40	7.50

So based on these 3 values entered for each user group, the remaining values, namely Ramp up time, Pacing and Actual TPM achieved are calculated.

Expected Load (TPM): This is calculated by simply multiplying base load with the load factor.

Expected Load (TPM): This is the theoretical ramp up time that is required to achieve the expected load and is calculated by dividing 60 by expected load. For example, if expected load is 30TPM, the ramp of course will be 60/30 = 2 users per second.

Actual Ramp up: Since the theoretical ramp up calculated in previous column can be a fractional value (indicating millisecond ramp up times) and we can have actual ramp up times only at a 1-second granularity, the actual ramp up is calculated by rounding off the expected ramp up to nearest integer. For example, if the expected load is 18TPM, the expected ramp up will be 3.33 seconds. This has to be rounded off to 3 seconds. This has the obvious effect of making the Actual Load different than the desired load.

Another case to watch out for, as I mentioned earlier is when the expected load is above 60TPM. In this case, the expected ramp up time is always going to be less than 1 and the actual ramp up will get rounded up to 0 or 1. The maximum Actual Load will be 60 as I'm not accounting for ramp ups greater than 1 user per second.

However, for both the above cases, there are 2 ways you can achieve the actual load that is closer to the desired load:

1. Break the group into multiple groups: For example, a 90TPM group can be broken into 2 groups - one with desired load of 60TPM and other with 30TPM. This way, an actual total TPM of 90 can be achieved.

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	60	60	1	60	1.00	1	60	60.00
2	30	30	1	30	2.00	2	60	30.00

In case of 18TPM, it can be broken into 2 groups of 15 and 3TPM. The values will be:

VUser Group	Base Load (TPM)	Number of VU	Multiplier (Load Factor)	Expected Load (TPM)	Expected Ramp Up (seconds)	Actual Ramp Up (seconds)	VU Pacing (seconds)	Actual Load (TPM)
1	15	30	1	15	4.00	4	120	15.00
2	3	6	1	3	20.00	20	120	3.00

2. More users per second ramp-up: Another way is to have multiple users per ramp up interval. For example, 90TPM with a single user group can be achieved by having a ramp up of 3 users every 2 seconds. Similarly, 18TPM can be achieved by having a ramp up of 3 users every 10 seconds.

I usually prefer the 1st method and choose the start time so that the ramp ups are even between the multiple groups of same user type. However it does make it more tedious as I have more user groups to keep account of. The 2nd method has the drawback of introducing a relative burst of users every ramp up interval.

VU Pacing: This is the interval between iterations and is calculated by multiplying the ramp up value with the number of users. Again, you have to make sure that the pacing value is comfortably above the single user transaction time; otherwise if the transactions take longer than the pacing value and next iteration is started immediately after the previous iteration, it may effect the load pattern.

Also, I'm assuming that you are pretty much content with having a stable pacing. If you want to add certain randomness to the the pacing times, that can also be done using this value. See my previous post for more on this here.

Actual Load: Finally, the Actual load is calculated by dividing 60 by the actual ramp up value. This is the actual load that you can hope to generate during your load test for this user group. What is actually generated during the load test depends on a lot of other factors like how stressed your system is etc. This can be found out by analyzing the results of the load test.

So that's that. It turned out to be a longer post than I expected, but hopefully it helps. If you have any comments or think of any suggestions to improve it, let me know.

Coming up...

A few weeks ago, I spent some time going over JMeter and building some advanced web test plans. But then I got distracted and buried under some intense projects, including a 1-week training on a product that I didn't know much about and that I have to be an advanced user of within a few weeks. (BTW...if you want to know why attending a 1-week advanced training on a product without having a solid background is a bad idea, let me know).

Now I have some time again but it's right before my vacation and I just don't feel like getting immersed into it again. But when I come back, I plan on posting some of my notes on how to build an advanced web test plan in JMeter. I plan on using a more advanced scenario than the one with basic Get requests, probably something similar to my previous post about using an MD5 library to create a LoadRunner script. If I can't find something else, I'll use that very example. I also plan on exploring some webservices and pure XML over HTTP kind of scripts. Stay tuned...

Oh... and I should also mention my reasons for looking into JMeter. It has been increasingly frustrating to borrow time on LoadRunner Controller from an external team, partially because of having to justify to the project team the additional charge back incurred to the project. I can't believe how many emails I have to write to justify a few 100 dollars. But anyway, turning misery into a learning experience, I wanted to explore if JMeter provides enough capabilities and is robust enough to be used for production performance testing.

Random Virtual User Pacing in LoadRunner

I guess there's always a first time. I had never used LoadRunner's random virtual user (VU) pacing earlier, but am working on a project that will need to use this feature for the first time. And as I thought about it a little more, I may start using it more frequently now. Here's how it happened:

This is one of the rare projects that provided me with excellent documentation - not only system documentation like the system specs, API Guides etc but also performance requirements like actual production volume reports and capacity models that estimated the projected volumes.

The capacity models estimated the maximum transaction load by hour as well as by minute (max TPM). What I needed to do was take maximum hourly load, divide it by 60 to get a per minute transactional load and use this as the average TPM. The idea was to vary the VU pacing so that over the whole duration of test, average load stays at this Average TPM but it also reaches the Max TPM randomly.

For example, if the maximum hourly transaction rate is 720 requests and maximum TPM is 20, the average TPM will be 720/60 = 12 and I will need to vary the pacing so that the load varies between 4TPM and 20TPM and averages to around 12TPM.

The Calculation:

To vary the transactional load, I knew I had to vary the VU Pacing randomly. Taking above example, I had to achieve 12TPM and I knew the transactions were taking around 1-2 seconds to complete. So I could have the pacing of around 120 seconds if I needed to generate a fixed load of 12TPM with a 5 second Ramp-up and 24 users.

Script	TPM	Number of VUs	Pacing (sec)	Ramp Up
Script 1	12	24	120	1 VU/5sec

So now to vary the TPM to x with the same 24 virtual users, I will need to have a pacing of 24*60/x. I got this from an old-fashioned logic which goes in my head this way:

24 users with a pacing of 60 seconds generate a TPM of 24
24 users with a pacing of 120 seconds generate a TPM of 24 * 60/120
24 users with a pacing of x seconds generate a TPM of 24 * 60/x

So using above formula, to vary the load from 20 to 4TPM I will need to vary the VU pacing from 72 to 360. So now we have:

Script	TPM	Number of VUs	Pacing (sec)	Ramp Up
Script 1	4 to 20	24	Random (72 to 360)	1 VU/5sec

Of course, there's a caveat. The range of 72 to 360 seconds has an arithmetic mean of 216. 120 is actually the harmonic mean of the 2 numbers. So the actual variation in TPM will depend on the distribution of random numbers that LoadRunner generates within the given range. If it generates the numbers with a uniform distribution around the arithmetic mean of the range, then we have a problem.

I ran a quick test to find this out. I created an LR script and used the rand() function to generate 1000 numbers between the range with the assumption that LR uses a similar function to generate the random pacing values.

int i;
srand(time(NULL));
for (i=0;i<1000;i++){
lr_output_message("%d\n", rand() % 289 + 72);
}

And of course, the average came out close to the arithmetic mean of 72 and 360, which is 216.

So with the assumption that the function used by LoadRunner for generating random pacing values generates numbers that are uniformly distributed around the arithmetic mean of the range, we'll need to modify the range of pacing values so that the arithmetic mean of the range gives us the arithmetic mean of the TPM that we want...phew. What it means is that the above pacing values need to be modified from 72 to 360 (arithmetic mean = 216) to 72 to 168 (arithmetic mean = 120). However, this gives us the TPM range of 20 to 8.6 TPM with a harmonic mean of 12TPM.

But I'll live with it. I would rather have the average load stay around 12TPM. So here are the new values. Note the asterisk on TPM. I need to mention in the test plan that the actual TPM will vary from 8.6 to 20TPM with an average of 12TPM.

Script	TPM*	Number of VUs	Pacing (sec)	Ramp Up
Script 1	4 to 20	24	Random (72 to 168)	1 VU/5sec

Why I'll continue to recommend and use LoadRunner instead of eLoad

I've raised this concern in meetings with Empirix and haven't heard this as being a priority. However, this is a major reason why I'll continue to recommend and use LoadRunner instead of eLoad:

Lack of ability to generate specified transactional load in eLoad and why it's very important

Background:
Web based (3 and n-tier) applications are different from client-server types of applications in terms of how they deal with user load (specifically how the system performs when users use the system) as explained below:

Load in web-based applications (3 or n-tier) is generally represented in terms of number of transactions expected per unit time (seconds/minutes etc) and not in terms of users per unit time. This is because in web/application systems, a user using the system can generate varying amount of load on the system depending on how active the user is. For example, a single user who submits 10 transactions every second will use much more system resources than 100 users who submit 1 transaction every minute (given the 'transaction' as defined is equal in both cases, and let's rule out caching as well). Even though those 100 users in latter case are using some fixed amount of resources on the system (for maintaining session information, other objects etc) that is 100 times more than that being use by the single user, significant resources are only consumed when the users are actively interacting with the system or waiting for response from the system.

So if while planning performance tests for an application, I get business requirements stating that X users are expected to use the system in a day, I have to work with them to further refine these requirements to specify what is the user activity profile (or scenario profile) and what is the frequency of their actions. For example, how many users will logon every hour/minute, how many will navigate to certain web pages or consume a service every hour/minute and how many will logoff every hour/minute. For simple scenarios, a transaction can be 1 to 3 steps - logon, stepA and logoff. Based on answers from previous questions, I will then refine the requirements to state how many transactions are expected to be executed every minute (or TPM). For complex scenarios, multiple transaction types may have to be defined - usertype1(stepA, stepB...stepX), usertype2(stepB...stepY) etc. In this case as well, I will have to use same questions to refine the requirements for each transaction type, i.e., for each transaction type, what is the expected transactional load in TPM or TPS (transactions per second). Then I'll go ahead and create the load test scenarios based on these transactional loads.

This kind of transaction based load specification is even more important in XML over HTTP/web services applications because there is no concept of user. Rather the system deals with requests (or transactions) that can come from a third-party web application, a custom client etc. that are generally referred to as service consumers. There is rarely a need to maintain session information and each transaction is idempotent (at least from the web/application server perspective). Note that the whole transaction may be non-idempotent but still be made up of a series of idempotent transactions. For example, a credit check service may be non-idempotent because it writes the number of inquiries to the user's credit profile and gives it a weight in calculating the user's credit score. But from the web and application server perspective and especially in test environments where limited test data is available, each transaction can be considered idempotent since we don't care about test user's credit score and need to generate a production load using limited data.

How the 2 tools handle (or do not handle) this:
Coming back to the point of why I prefer to use LoadRunner over eLoad in these scenarios (there are other reasons too but let's just focus on this one for now)...
Both eLoad and LoadRunner let me specify the iteration delay (or VU pacing in LoadRunner terminology, don't confuse it with VU pacing in eLoad which actually means think time!) that controls how long a Virtual User (VU) waits before starting the next iteration. However there is only one way to specify this in eLoad - from the time previous iteration ends (see Figure 1). This creates a problem because the time when next iteration will be started depends on how long the previous transaction took. For example, if you specify this delay to be 30 seconds and the previous transaction took 30 seconds, next iteration will be started after the end of 30 + 30 = 60th second. However, if the previous transaction took 5 seconds to complete, next transaction will start after 5 + 30 = 35th second. Now suppose you want to generate a transactional load of 10 TPM. You use 10 virtual users and specify the delay to be 55 seconds expecting each transaction to take around 5 seconds. This way you can have each of the 10 VUs submitting 1 transaction every minute, thus generating a load of 10 TPM. But when you run the load, server (or the application under test) gets busy and takes 30 seconds or more to return a response. eLoad's virtual users are still going to wait 30 + 55 seconds before starting subsequent iterations, thus reducing the overall transactional load by almost 30% (or 10 * (1/85) * 60 = 7 TPM as compared to required 10 * (1/60) * 60 = 10 TPM). But you really need to find out how the system behaves at production load of 10 TPM even at busy periods...do the requests keep queuing up and ultimately cause the system to become unresponsive or the system returns to stable state soon after! Well...hard luck, because eLoad is going to decrease the load if the system starts taking longer to return the responses. You can possibly add more virtual users to the scenario to increase the load when this happens but I don't want to have to sit and watch the load test for this to happen when I'm running a 12 hour test and am only allowed to run the tests in non-business hours. And I'm not even sure if I'll be able to calculate that fast how many users to increase/decrease everytime this happens.

Figure 1: eLoad VU Settings

LoadRunner gives 3 options in setting the iteration delay/pacing (see Figure 2):
a) As soon as the previous iteration ends
b) After the previous iteration ends: with a fixed/random delay of XXX seconds (In case of random delay, it lets you specify a range)
c) At fixed/random intervals, every XXX seconds.

Figure 2: LoadRunner VU Settings

So the above situation is handled very easily by selecting the 3rd option and choosing a delay of 60 seconds. In the above example, if the previous iteration took 5 seconds, it'll wait for 55 seconds and if it took 30 second, it'll wait another 30 seconds before starting next iteration. No matter how long the previous iterations take (as long as they are less than 60 seconds), it will always start subsequent transactions at specified intervals of 60 seconds from the start of previous transactions. thus keeping the load stable at 10 TPM. If I expect the transactions to take longer than 60 seconds, I can start with more VUs and increase the delay. For example, I can use 20 VUs and set the delay to 120 seconds. This will still generate a load of 10 TPM if I specify the ramp-up time correctly.

Conclusion:
So my conclusion is that I will use LoadRunner as much as I'm able to. In case you're wondering why my company has 2 load testing tools when buying 1 is costly enough, my team is under a different business unit that owns eLoad licenses because LoadRunner was considered too expensive. However, there is another business unit that has LoadRunner licenses and even though I had to go through a lengthy procedure, I got them to agree on letting us use LoadRunner and charge us for the usage.

---------------------------------------

Note 1: You can argue that load in real-life production scenario is never stable. But when it comes to defining the system's performance, I prefer to use multiple scenarios with increasingly different loads. For example, if the business expects about 10,000 transactions per day, considering a 10 hour business day this comes to 16.67 TPM. I will run load tests at 16 TPM (1x), around 40 TPM (2.5x) and around 80 TPM (5x) to give them numbers on how the system can be expected to perform if the transactional load varies from 16 to 80 TPM. I will probably also run some stress tests by running a background load of stable 16 TPM and then submitting a batch of multiple requests (100/200 etc.) to see how the system recovers in this case. Again, this will depend on business requirements and expectations. Also, If I really need to vary the load, I would rather use the random option in LoadRunner and vary the load but still keep the overall load stable.

Note 2: I am not implying that defining load in terms of number of concurrent users is not important. For some applications (e.g., citrix or remote desktop applications) it is the most important load defining criteria. Even for web based applications, you may want to find out how many users you can concurrently support before the server runs out of memory. This will help you determine when you'll need to buy extra hardware. But any commercial load testing tool has to support the ability to generate transactional load as well now that XML and webservices are becoming more and more common.

Note 3: Current eLoad version that I'm using is 8.10 and LoadRunner is 8.1.4

WebScarab

There have been times during LoadRunner scripting that I needed to see the low-level HTTP request that is being sent from my client (which I am using to record, e.g., a browser, or a custom client) to the server. Earlier, I used Ethereal (http://www.ethereal.com/) successfully but the problem is that it doesn't support SSL directly. So if the communication is over SSL, all I would see is encrypted data and there was no way to see the headers/data being transferred. This made me look for alternatives. I recently came across WebScarab and I wouldn't say it's free of bugs but I'm sticking with this tool for as far as I can see in future. Here's how I was able to solve some of the problems in LoadRunner scripting using this tool.

Solution 1: (SSL Intercept)
First things first, WebScarab proxy

is able to observe both HTTP and encrypted HTTPS traffic, by negotiating an SSL connection between WebScarab and the browser instead of simply connecting the browser to the server and allowing an encrypted stream to pass through it.

This is a major advantage. So I no longer have to hope that one of the test environments will not have SSL implemented and will let me observe the non-SSL HTTP traffic. Watching browser traffic was just as easy as starting the WebScrarab proxy and pointing the browser to the local proxy. It gives the options to intercept request and/or responses and lets you modify the requests in any way before passing them on the server. For a custom client, I can capture the exact headers being sent and add them to my web_custom_request:

    web_add_header("Cache-Control", "no-cache");
    web_add_header("SOAPAction", "\"\"");
    web_add_header("Accept-Encoding", "gzip, deflate");

And I can also copy the body if it's an HTTP XML post for example, to get the exact XML data being sent and put that in the body of the request:

    web_custom_request("SampleService",
        "URL={URL}",
        "Method=POST",
        "EncType=text/xml; charset=utf-8",
        "TargetFrame=",
        "Resource=0",
        "RecContentType=text/xml",
        "Mode=HTTP",
        "Body="…

and so on. See screenshot.


Solution 2: (Reverse Proxy/Act as a web server)
As it happened, the client I was using (for more details on this, see my previous post) had pre-configured options of selecting the URL. So I didn't have any way to point the client to the local WebScarab proxy. I looked through the help contents and found this:

WebScarab allows you to specify a "base address" for a Listener. The base address instructs the Listener to operate as a reverse proxy, and should be formatted as a HTTP or HTTPS URL. In this mode, it will act as a web server, rather than as a proxy server, and will construct the URL by concatenating the base URL and the path that appears in the request line. If the base URL is an HTTPS URL, it will immediately negotiate an SSL tunnel prior to trying to read the request from the browser. This is useful for the situation where you are using a custom HTTP-based client that does not support configuring an upstream proxy. Simply change the hosts file on the computer on which the custom client is running to point the site in question to the computer on which WebScarab is running on, and WebScarab will receive requests for the targeted website.

This meant that I could use the hosts file to point to the proxy and specify the base address in the proxy listener to intercept the requests. In a few tries, I was able to intercept the SSL requests over a non-local base address. Again, I could get the headers and body and use it in the web_custom_request. See screenshot.





Solution 3: (SSL Server Certificate)
Another client that I was recording my script against had a similar issue. The client didn't support pointing to an upstream proxy so I configured the hosts file to point to the listener proxy. However when running the client, it threw this exception:

{http://xml.apache.org/axis/}stackTrace:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

I tried adding the WebScarab certificate to the keystore through Java Control Panel but no luck. After googling a little more, I came across this forum thread: http://forum.java.sun.com/thread.jspa?threadID=220329&tstart=165
Apparently, Java uses its default keystore and to use another keystore, it has to be created and provided in one of the arguments. So after importing the WebScarab certificates into IE trusted certificates and then exporting it into .cer, creating a keystore with the certificates and modifying the batch run file to add
-Djavax.net.ssl.trustStore= -Djavax.net.ssl.trustStorePassword=

I had my fingers crossed when running the client again. But fortunately, this time it worked as expected and intercepted the HTTPS requests without any errors. Once again, I used the custom headers and the XML body to create LR web_custom_header request.

LoadRunner function: web_convert_param

After spending about 6 hours trying to figure out a solution to the problem of parameterizing URL encoded strings in my LoadRunner script, I finally found the solution: a built-in function provided by LR.

The problem:
I recently recorded a script against an Java application that ultimately sends a HTTP POST request to the server being tested with an XML in the Body of the request. The script came out fine, except that the body (XML content) was URL encoded. It would have been fine if I didn't need to parameterize the data in the body but since I had to, I realized in a few seconds that I would have to do some extra work to convert the data from parameter file to URL encoded format. The data in parameter file is of the form:

Fname,Mname,Lname,Gen,SSN,A1,A2,A3,A4,A5,City,State,Zip
John,,Smith,,123456789,123,Main,St,#,88,TestCity,TestState,987654321

However, in the script, address is one field that is a concatenated string of A1 - A5 parameters with a space between each of them.

Further Research:
A1 - A5 are the components that make up the street address that is supposed to be concatenated to one field in the script. No problem.
sprintf(as,"%s %s %s %s %s", lr_eval_string("{a1}"), lr_eval_string("{a2}"), lr_eval_string("{a3}"), lr_eval_string("{a4}"), lr_eval_string("{a5}"));
lr_save_string(as, "addressStreet");

…but the problem is there are spaces in between this components, which we all know get converted to a '+' character in URL encoding. Ok, so I could've just used sprintf(as,"%s+%s+%s+%s+%s",…
But some of the addresses also have '#' signs that get converted into '%23'. Now I had 2 options:
1. remove all the addresses that have a '#' sign
2. write a URLEncode(char *) function that does the obvious.

1st option wasn't very tempting because however unlikely it may be, the addresses can have other characters that need to get URL encoded as well. And if a sizeable chunk of the data given to me has these characters, I could lose a lot of records. Also I didn't want to have to modify the data every time I get a new 10,000 record file.
2nd option seemed the way to go. But it was late Friday and I had to start the tests in a short time so I was lacking the much needed patience. Anyways, I ended up running my tests by removing the records with '#' character. Luckily, there were few.

Solution:
On Monday after running all the tests, I got back to writing the URLEncode function. After researching online on what the characters should get converted to and trying to find an already published function in C that I could tweak to my purpose, and stumbling on PHP, Java, JavaScript functions, I started getting a sense that LR may have some kind of built-in function that does that or something similar. I don't know why I hadn't thought of using Mercury Knowledgebase before that. So a simple search of "url encoding" brought up 3 articles - none of which seemed helpful. But reading through the 2nd one (KB Problem ID: 18880), I couldn't believe what I saw:
Pass the 'XMLSource' parameter as an input to the web_convert_param function, and store the result as 'TargetXML'
web_convert_param("TargetXML", "SourceString={XMLSource}", "SourceEncoding=HTML", "TargetEncoding=URL", LAST);

So this web_convert_param function seemed to do exactly what I needed. I tried it out with a couple of different strings and it did as promised. So my new script had:
sprintf(as,"%s %s %s %s %s", lr_eval_string("{a1}"), lr_eval_string("{a2}"), lr_eval_string("{a3}"), lr_eval_string("{a4}"), lr_eval_string("{a5}"));
lr_save_string(as, "addressStreet");

web_convert_param("encAddressStreet", "SourceString={addressStreet}", "SourceEncoding=PLAIN", "TargetEncoding=URL", LAST);
lr_output_message("%s", lr_eval_string("{encAddressStreet}"));

It took me only a few minutes to make that change. And I didn't need to worry about the '#' characters or any other unsafe characters in the address string. Phew…

Morale:
Use the Mercury Knowledgebase more often. If you think there should be any other, let me know.