If you’re a software tester or quality assurance (QA) professional for any business with a heavy stake in the holiday season – i.e. a retail, hotel, airline, car rental or parcel shipping company – you know that early, thorough planning is essential to ensuring your web operations can handle heavier than normal traffic volumes. High quality websites, web applications and mobile sites – those that remain fast, reliable and consistent under load – can have a big impact on your holiday season revenues. Research has shown that the majority of consumers expect superior online experiences no matter how much traffic a site may be experiencing at any given point in time. In fact, more than three out of four consumers will switch to a competitor’s website if they encounter slowdowns, errors and transaction problems during peak traffic times.
If we consider the retail industry as a litmus test for overall holiday season web performance trends, a look back at last year’s Black Friday/Cyber Monday shopping period – typically a period of extremely heavy online traffic – revealed the following:
- Large retailers performed consistently well, validating a year-round focus on optimizing site design and infrastructure for this peak buying period.
- Smaller retailers also performed well but experienced a dip mid-morning Cyber Monday, which retrospectively proved to be the heaviest e-commerce shopping day on record. While smaller retailers clearly recognized the impact of web performance on user satisfaction and made concerted efforts to support increases in online volume, they may not have had the same level of resources concentrated on performance in comparison to their larger industry counterparts.
- The mobile web proved to be a challenge for all retailers, both small and large, particularly on Cyber Monday and into the following Tuesday. During this time, a majority of mobile sites – even for the most prominent retailers – experienced moderate or more significant decreases in performance.
Based on last year’s observations, there are several strategies software testers and QA professionals can adopt to take full advantage of the revenue opportunities presented by the upcoming online holiday shopping season. Many of these strategies are free or low-cost and can help even smaller organizations achieve a more even playing field with larger competitors.
1. Never Underestimate the Business Impact of Even the Slightest Performance Improvement
As a software tester or quality assurance professional, it’s your job to be the web performance ambassador within your organization. This means clearly understanding and communicating how even subtle web performance improvements can have a major business impact, in order to secure buy-in from other cross-functional groups for the necessary changes and investments that will help drive stronger web performance during peak traffic periods.
As an example, a recent study analyzing millions of page views on websites around the world found that conversion rates increase 74 percent when page load time improves from eight to two seconds. Another study found that page abandonment rates increase steeply as page load times increase. According to Aberdeen Group’s report, “The Performance of Web Applications: Customers are Won or Lost in One Second,” for every one-second delay in website download time there’s an approximate seven percent drop in conversions. The takeaway here is that even small performance improvements can pay off handsomely, taking online customer behavior in a more positive, revenue-generating direction.
2. Involve All the Relevant Cross-functional Teams in the Load Testing Effort
Superior web performance should not be considered the sole responsibility of any single group within an organization. Rather, all groups with a vested interest in holiday season web performance – including line of business, IT and marketing – bear responsibility. When these groups share knowledge and work together, the end-result is often a strategic, well-thought out load test, the results of which drive the actions and decisions needed to promote a specific web performance goal. Software testers and QA professionals can take the lead in defining this goal based on industry research like Forrester Consulting’s report, “E-Commerce Website Performance Today,” which states two seconds or less as the threshold most consumers are willing to wait before abandoning a website and going to a competitor. There are also benchmarks available to demonstrate the bar being set by web performance leaders in various industries.
Business executives often come to the table with a strong understanding of growth projections and revenue and profitability targets. Marketing executives know what campaigns are being carried out when and where, which helps dictate which website areas and applications are highest priority, for which geographic segments of end-users, during which dates and even what times of day. IT executives possess a strong understanding of the capabilities and limitations of the existing environment.
A load test executed within these parameters and with a clear performance goal in mind can demonstrate to line of business executives if extra investments are needed to offset potential losses from poor web performance. It can also show IT how sufficient their infrastructure is, if idle resources can be re-deployed elsewhere, or if extra resources may be justified. Additionally, IT can directly show marketing the performance impact of certain content management decisions during peak periods – like adding extra graphics or video – while marketing can help IT better plan for anticipated campaign-driven traffic surges. Test Early and Often Rarely do businesses meet their performance goals in the first load test. If you know you are going live on Tuesday but leave testing to Monday, the only answer you’re going to get is a ‘yes’ or ‘no’ – as in, we’re going to likely succeed, or fail. If you test well in advance of application deployment, you’re still apt to get a ‘yes’ or ‘no’ answer, but you have time to address it – and that’s the key. Testing early also enables you to include any third-party services adding extra website and application features in your test. Consider online retail applications, which often include features like product demos, shopping carts and check-outs provided by external third-parties. Retail and travel websites in particular tend to be quite feature-rich, incorporating a lot of advanced third-party functionality. Including these third-parties in your load test is important, because your ability to deliver performance under load is increasingly a by-product of their abilities.
Gauge Performance Under Various Load Sizes from the Real World Perspective of End-users
The importance of this approach cannot be emphasized enough. It’s impossible to get an accurate read on the end-user experience under load if you load test only the systems and application components based in your data center, from within the data center. That’s because there are so many elements beyond your own firewall, including internet service providers (ISPs), content delivery networks (CDNs), cloud service providers, third-party services, mobile carriers, devices and browsers, that can impact the end-user’s experience. This is known as the web application delivery chain, and one single “offending” element can degrade an entire experience. In fact, it’s estimated that two out of three end-user performance issues are the result of an element beyond the firewall.
The only way to gain control over the complex web application delivery chain is to begin with a true view of the end-user experience, available through worldwide testing networks comprising real user desktops and devices. The most effective means of gauging how this experience is impacted by various load sizes is to simulate traffic loads generated from the cloud with load generated from these real desktops and devices. From there, you can trace back through the chain to identify and address any faulty elements. For example, if you determine a third-party service to be the problem, you can alert them to the problem and re-visit the terms of your service level agreement (SLA).
Optimize Your Mobile Site for Speed
A recent survey demonstrated that 60 percent of mobile web users expect site downloads in three seconds or less – not too far off from Forrester’s twoseconds- or-less threshold for traditional PC sites. Given the performance demands of mobile web users, when higher than usual mobile web traffic volumes lead to unbearably slow load times, revenues, brand image and customer relationships can be jeopardized. Like PC websites, businesses must now carefully ensure their mobile sites and infrastructure can handle peak traffic conditions when user expectations are at their highest, such as the holidays or when you’re leveraging a mobile ad network to drive increased traffic to your site. This may mean load testing, which again must be based on an understanding of the true mobile user experience under load.
Given the sheer volume of mobile browser and device combinations in use today, it’s important for businesses to prioritize and ensure strong performance under load across the mobile devices most prevalent among their customers. Mobile benchmarks can show you how other companies are performing on the mobile web in aggregate, as well as across individual device platforms.
Conclusion
The Internet continues to account for an increasing percentage of the U.S. consumer’s disposable income, and over the past several years, as overall holiday season revenues have remained flat or tepid for many companies, e-commerce revenues – as a proportion of overall consumer holiday spend – have continually accelerated. With the slowly recovering economy still on shaky footing, businesses that depend heavily on holiday season revenues will look to the web as an engine for revenues and profitability, helping to offset flat sales or losses occurring through other channels. The focus on a high-performance Internet reaches a fever pitch in the lead-up to the holiday season, and businesses simply cannot afford for websites, web applications and mobile sites to buckle under load.
The upcoming holiday season presents software testers and QA professionals with a valuable opportunity to elevate their strategic roles within an organization. By serving as web performance ambassadors; involving the right organizational stakeholders; testing early and often; focusing on the true end-user perspective and equally emphasizing the PC and mobile web, they can help ensure a web presence that enhances customer satisfaction and revenues this holiday season.
About the Author
Colin Mason Colin Mason is Product Manager for the Reality Load performance testing solution at Gomez, the Web Performance Division of Compuware. Colin has 10 years of performance testing experience and has conducted over 500 load tests for 100 companies. He has presented at numerous industry conferences, including STAR, and has authored industry-acclaimed papers on performance testing web applications.