Use Validators and Load Generators to Test Your Web Applications continued...
Use a Load Generator to Test Your Server Software
Stress test your web applications to find performance problems. Links to load generators for testing HTTP, HTTPS and other protocols.
Michael D. Crawford
crawford@goingware.com
Copyright © 2001 Michael D. Crawford
This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.
Use a Load Generator to Test Your Server Software
I got my first look at a load generator when I was working at Octel Communications, a company that built multiprocessor computers with a custom-developed real-time operating system for use in voice mail. These were impressive machines; the larger models basically consisted of refrigerator-sized card cages with dozens and dozens of 80386 chips with half a cubic meter of hard disks in the base for storage of the digitized voice messages.
I often worked through the night in my job as a network administrator to take down the Sun servers for maintainence, and I noticed that in the test lab outside the computer room at night there was a machine in a rack mount with hundreds of "silver satin" telephone patch cords coming out of it making a cacophony of touch-tone beeps.
I asked my manager, Karen Coates, what this strange device was and she told me "It's a load generator". It had the job of testing our messaging software and hardware by placing an endlessly repeating series of simultaneous phone calls to a system under test and leaving machine-generated voice mail.
Wanna do lunch today?
I think load generators are likely in more widespread use in the real-time and embedded software areas, but they are available for testing web and other internet applications.
Note that I entitled this section "Use a Load Generator to Test Your Server Software".
You might say that what we need to stress test is the hardware, for example to predict how powerful a server one needs to buy to handle a certain number of simultaneous users.
That's a valid use for a load generator, but I suggest that what you really need to concentrate on is the performance of your software. Performance problems can be insidious and hard to fix in a completed application; it is important for each programmer to do load testing on a daily basis during development, and for extensive stress testing during each step of a formal quality assurance process.
If you don't, you might find that your half-million dollar server falls over after you take your site live and just a few dozen people log in.
Let's say we get extra payoff in hardware performance out of proportion to the extra cash we spend; suppose paying twice as much for a server, hundred of megabytes more memory or faster RAID storage for your database will increase performance by a factor of four.
That's not a bad investment. What is a bad investment is that one lazy, inexperienced or uneducated programmer, or an otherwise good programmer who is ill-informed, working with a poorly designed overall architecture or not communicating effectively with his coworkers can take that two-fold investment in server hardware and reduce the performance of your server a hundredfold in just a few lines of bad code - code that will be devilishly hard to find once your application is completed.
Performance problems may not be contained in one small section of code but be spread throughout your program, and they could be caused by the way disparate parts of your program interact with each other - or by interactions between different server processes in a system.
If you are running more than one process (as is typically the case) or using a multithreaded implementation, you need to worry about race conditions, deadlocks, or performance degradation caused by simply holding a lock longer than necessary. You may not notice any of these problems at all unless you stress test.
Another significant problem is memory leaks. Having a memory leak in your web application may slow your response due to violent paging and eventually run your server out of virtual memory backing store and crash your application, but you may not find out that you even have a memory leak in your application because manual testing of your pages does not result in memory allocation growing fast or far enough to be noticed.
It is popular to write web applications these days in garbage-collected languages like Java, Python or Perl, and you might object that a garbage collected application is inherently free of memory leaks. While it is true that this is a common claim of marketing literature for some of these languages, it is simply not true. I've experienced plenty of examples of memory leaks in garbage-collected applications written in these and other languages such as Smalltalk in the course of my career, and I've heard of far more.
Under garbage collection, an object will not be freed if some other object holds a reference to it. All you need to do to have a memory leak in a garbage collected application is to be careless about keeping unwanted references around.
A single reference to the root node of a DOMTree in a Document Object Model application can hold megabytes of XML data in memory - permanently so. You may not be able to find where this reference is kept no matter how hard you search your source code for it. You may not even know it's there because you never bother to look.
It's as simple as that. Actually fixing or preventing performance problems or memory leaks in your web applications is beyond the scope of this article; the first step is to detect them, and for that we can use a load generator.
Suggestion for Validation During Stress Testing
I'd like to suggest that someone with the inclination to do the work should combine one of the available validators with a load generator, so that all (or perhaps just a sample) of the documents downloaded from the machine under test are validated.
Because of the extra work required to validate, your load generator / validator would need to run on a more powerful machine than the server under test, or you would need to run the attack with several load generators working simultaneously.
This would serve an important purpose. While you can use a validator to check that your pages are correct in basic use, and you can use a load generator to ensure your server system can withstand the expected user load, it is hard to tell that the pages being generated by a stressed server are actually what you want.
If bugs due to stressful conditions cause corrupt documents, then such a validating load generator is likely to find it.
You can leave your tip in the jar.
The Available Load Generators
First, an important warning:
Using a load generator can cause signicantly large amounts of network traffic or CPU load on your server. You are advised to avoid using a load generator on a live site over the Internet, as you may incur costly bills for bandwidth usage. If your server is shared with other sites, as is the case with database hosting services, the administrators may kick you off for hogging the CPU.
I suggest that even on an intranet you isolate the load generation machine and the system under test by using an ethernet switch or router to put them on a private network segment, otherwise your coworkers may curse you for interfering with their ping times whenever you turn on your test tool. Or just unplug their connection to the rest of the office.
For day-to-day use by an individual developer, it will probably do to put the load generation software and the application under test on a single workstation, so you don't generate network traffic at all or have to spend a lot of money on testing hardware.
You can also try putting a load generator on your laptop as I suggest in my discussion of installing the W3C validator source code . That way maybe you can grab a latte and listen to some nice jazz in a cafe while you develop your web application.
I list the available load generators for which source code is available at the time of this writing. I found the load generators by using the following search on Freshmeat:
- Deluge - Download
Web site testing tool with included proxy server for recording playback scripts, and log evaluator for generating result statistics.
- Hammerhead 2 - Download
Highly configurable multithreaded web site stress testing tool. It can generate traffic from multiple IP aliases.
- ISIC (IP Stack Integrity Checker) - Download
Network stack tester. It sends pseudo-random packets to the selected protocol to test the stability of the drivers and to check for leaks in firewalls
- Ptester - Download
Simple HTTP benchmarking tool
- Rjobd - Download
Framework for reliably controlling the execution of processes on a remote or local system. See author Mike Werner's paper Evolving the Resilience of an Operating System.
From Rjobd's page:
The Rjobd Toolkit is the enabling technology for the Chaos Resilience Testing technique. This technique exploits elements of complexity theory to uncover an addional class of reliability problems that are otherwise delivered to the customer base.
But note from the Freshmeat page:
Take note that installing the current version will more or less completely compromise system security.
- Siege - Download - Manual
HTTP and HTTPS regression testing and benchmarking utility. It can hit a single URL repeatedly or hit multiple URLs. HTTPS encryption is provided by OpenSSL
- smpop - download on same page
POP (Post Office Protocol) stress tester that simply pulls out a lot of email from a server given a list of usernames and passwords
- SSL Stress Client - download on same page
Small program for stress testing SSL servers. It is built using OpenSSL. It is also capable of testing static HTTP web page testing
