Performance testing is arguably the most important part of the software development life cycle, yet it’s usually the first part to get cut off the tail end of a project or sprint. This is if performance testing is even being done at all. With experience from working with dozens of the biggest names in high traffic online applications, I can tell you that the percentage of companies out there doing performance testing is staggeringly low. Of the companies that are doing performance testing, many are missing the mark in terms of testing the right things, and they’re usually not testing enough. Web performance is critical; it’s customer response and retention that proves that. Amazon states that for every 100ms of latency, they lose 1% of their sales. Google found that an extra 500ms in latency cost them 20% of their search traffic. These are a few of the reasons why a down-to-earth performance strategy with measurable results is so important.
Performance engineering has existed as a discipline since client server applications ruled the world, yet the industry still doesn’t know how to define it. People are confused, and rightfully so. What exactly is performance engineering anyways? If you search online or poll a handful of performance engineers for a definition, you’re guaranteed to get varying responses with minimal overlap. Answers will include testing and engineering of the following: page speed, scalability of the underlying architecture, failover of infrastructure components, transaction time, and more. Even more disparate will be the metrics that people are looking at to define and measure performance, things like: average response time, time to first byte, top 98th percentile request times, transaction completion time, and more.
I think you’re starting to understand the problem. You might even have first-hand experience with some of the pain points that come from confusion in this space. We have all of the data to show that customers define what performance means with their words and actions, so why aren’t we paying attention? I would submit to you that I’ve never heard an end user define performance on a web site they visit by mandating a 1 second page response time SLA with all requests meeting that criteria being in the 98th percentile. That’s ridiculous. They just want a site to be up when they want to visit, and they want to have a “good” experience.
Real world performance testing begins with the “customer’s happiness” view of performance. As much as we are all engineers and love our performance data, reporting on performance and the customer experience is a much higher presentation level. Imagine that your CEO or General Manager has a dashboard on their desk relaying the performance of your online application to them. It should probably have 3 or 4 key metrics on it with a green, yellow or red light next to them. They care about customer experience and revenue. Customer experience means uptime and response times.
So why don’t teams create performance testing strategies around these types of metrics and outcomes? If teams spend all of the valuable time in a testing cycle going too low or too high without achieving the good customer experience results, the value proposition of their work is questionable.
The heart of a successful performance strategy is rooted in simplicity, not complexity. Performance engineering can be deeper than any other type of software development. But it should only be complicated when it needs to be. Facebook serves more ‘Like’ buttons in one second than most web sites serve pages in an entire month (about 2.1 million per second or so at last check). If you have those types of performance concerns, you can go as deep as you want. However, there are only a few web sites in that traffic category and they’re in their own little world*. Elsewhere, I find that a lot of teams have a hard time answering the question “How does average response time look as load increases on the site?” Almost every application out there can benefit from a top-down performance strategy that begins with the customer.
This chart shows users ramping up and the average response time of pages on the web site. A simple metric that is important to a customer. Do you know what this chart looks like for your application?
Here is a model I like to use when I think about what my customers care about in terms of the performance priorities I’m setting for my team.
A model showing the importance of performance engineering efforts to the customer
Capacity is number one. Whether the customer knows it or not, capacity is the most important focus because if you crash under typical or peak load, you can’t move on to the next step which is experiencing the site. Then a user judges the reliability of their experience. Studies have shown that website visitors would rather have a consistently slower than ideal page load time instead of it being fast sometimes, slow some other times, and down the rest of the time. Then we get deeper into some of the more engineering centric topics such as the scalability of the site, or its ability to increase capacity easily as needed, and transaction and code level metrics.
My challenge to all performance engineers, engineering leaders, and business leaders, is to look at the performance of their online applications and ask the question “is my team doing real-world performance testing?” If all of the lights aren’t green on the customer experience dashboard, there is an opportunity to align the performance strategy towards what matters most to customers. Keep it “real”… meaning only test what is necessary to achieve the right outcomes around performance, scalability, and reliability.
Discover performance testing by attending the Software Test Professionals Conference & Expo 2010 in Las Vegas, Nevada. Learn:
• actual performance testing skills tools and techniques
• develop test objectives and scripts
• use business and technical metrics to quantify target load and acceptance criteria
• design test scenarios
• measure and correlate system resources with response time and load
• diagnose bottlenecks
• communicate conclusions and recommendations to stakeholders.
Mr. Bartow will present Architecting Applications for Performance and Scalability. Learn more about his
presentation and the other
performance testing speakers.
* Source: http://royal.pingdom.com/2010/07/05/what-it-takes-to-be-a-top-100-website charts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+RoyalPingdom+%28Royal+Pingdom%29