As more and more software test engineers and developers focus on application performance, they need to figure out career paths around performance testing and engineering that will help them grow their skills in performance optimization, coding for performance, scalability solutions, and load testing. If you have been a performance tester for some time now, you too might be wondering about next steps. Every week I talk with performance testers who tell me stories about how they found new ways to influence their peers and leadership with their talents. Like me, they have found an answer to the question: What’s next after performance testing?
When I started in performance testing in the early 1990’s I had very little formal training. Early on I was employed as a functional tester and the opportunity arose to work on automated performance testing for branch office systems in mortgage processing. I was young and inexperienced but luckily I was surrounded by senior engineers who shared with me the basics of performance. I could outline the primary objective of the test (End-user Response Time) and they guided me through a few steps on how to measure that accurately and repeatedly. They helped me to remember from my college statistics class the importance of sample size in my data collection, calculations, and extrapolations. Their mentoring became the primary means of learning about performance testing and performance engineering, but only once I had determined my objectives. Good mentoring requires that you (as the mentee) bring your own ambition and goals to the equation.
And I have found that when you spend time bugging experts for guidance and to help you evaluate your outcomes and learning, inevitably they point you towards other mentors and experts to help you go farther. Often that happens in the form of reading suggestions. During one project working the night-shift on a batch processing performance test, I had very little distraction for hours-on-end, and I was able to read two recommended books on performance and testing: O’Reilly’s System Performance Tuning and Software System Testing and Quality Assurance by Boris Beizer. I also read a few books on Oracle, C++ and IBM AIX Unix. The great thing was I could apply what I read to the testing I was doing each night. By the end of my project, I had re-invented how the organization approached batch performance testing and optimization.
Like many testers I have an enormous appetite for learning. That means that after a period of time working in a specific role or project at an organization I naturally hit a limit on the things that I can learn — which means it’s time to make the leap to another team or project in that organization, or even make a move to an new company. You should never be afraid to take that leap, but also try to never burn a bridge with a mentor. Ask them to respect your decision to move on to new things and suggest new ways to collaborate in the future. If you can, find a way to connect back to your mentors as you grow. Share with them what you are learning and see if there is a way to help them in return.
Yet even with the great mentors and leadership I had over the years, it was hard to know the landscape of the performance testing world and just where to begin. What options should you consider as you take your next step in developing performance skills?
Here are a few tips that can help you to choose a path in your performance career:
Tip #1: Learn performance monitoring and root-cause analysis. When you find resource overutilization on the server or slow response times on the client end, do you know why? This is the easiest skill to develop and will definitely make you a more valuable performance tester on your team. A good place to start is with the basics of CPU, Memory, Disk, and Network resources on all components of the system under test. You can learn a lot from system administrators and network administrators as mentors, particularly on system resource measuring and root-cause diagnosis. And don’t just stop at the simple measures of resource utilization — go deeper into how each resource works and how the components work together and depend on one another. CPU has L2 and L3 caching and different processor architectures. Disk resources consist of diverse types of cache memory, solid-state and classic spindles. Which one are you using? The different roles or jobs you could consider using these skills would be as a performance engineer in production, working on performance triage or incidents — that is if you like being in the hot seat!
Tip #2: Focus on real-world performance design. Most load testers and developers are good at hitting the system under test with lots and lots of load. Who wouldn’t like that? Stress testing is exciting and you can really see the system working at maximum capacity, or in some cases completely crashing under the load. But most often that kind of overload testing doesn’t really help your project to succeed at proving real value to the end-user or the customer. You can set yourself apart from other performance testers by really studying what the end-user performance requirements are for response time. Almost half of the customers I talk to don’t even have response time objectives for performance tests. You can also develop skills in understanding real-world production workloads, and learn profiles for traffic and utilization from the production logs. Business analysts, technical architects and system administrators or DBA’s would be a good choice as mentors on these subjects. You could even volunteer to lead a collaborative, cross-group team for performance.
Tip #3: Use application diagnostics and profiling. Start to implement a diagnostics profiler into your performance tests and uncover the secrets to resource over utilization right down to the code level. Consider using a trace on the database to determine what the top worst performing database queries might be, and what those query plans are. And why not learn what a query plan actually is, inside the database engine? You might be surprised when you learn about blocking, locking, schedulers, buffers and process threading. When it comes to digging into the application code, don’t be surprised if you find performance bottlenecks in the code that the developers never even dreamed they could find. Your mentors in these areas will be developers, architects, database developers and admins — and you might find a mentor with a 3rd-party vendor or even one of their developers.
Tip #4: Bring performance into other testing types across the lifecycle. With new application architectures, the test requirements for functionality and performance are now more intertwined than ever before. Functional testers are finding that their pass/fail test criteria are greatly expanded due to performance-related states in the underlying dependencies in the architecture. These guys need your help! Working up-stream with developers doing unit-level performance testing can help you to improve performance before it’s too late — before their code gets buried beneath the application architecture. Developers and functional testers are your best mentors here. They might be resistant at first, but be politely persistent about the importance of performance for the project success. And study some of the new books on web performance optimization and agile testing, especially if your work as a performance tester is being “left out” of the agile pairing or stand-ups.
Tip #5: Become a performance analysis and reporting pro. If there is one thing most performance testers are bad at, I’d say it’s reporting the results of the performance test to anyone who is not also a performance tester. Working on your analysis and reporting skills can make you stand out from the crowd; project leaders asking for you by name to work on their projects because you can “walk the walk, and talk the talk” on performance. The best mentors here are project managers and project sponsors who understand the business objectives for the performance testing efforts. But remember, they don’t know performance results. Your challenge would be to find a way to communicate the performance results into a story that even they could understand.
Tip #6: Expand your platform for performance. If you are early in your career, you might be trying to master a specific technology (Java, .NET, JMS, MySQL, Javascript or Ruby, etc.) before you try to move on to other technical platforms, architectures, etc. My advice would be exactly the opposite, and here’s why: when you are just starting out, it’s expected that you are learning and are not the expert in any one type of technology or platform so the pressure to have mastery is much less. Take advantage of this and keep in mind that the core concepts for performance root-cause analysis (see tips #1 and #3) are roughly the same for every platform out there. While you might be on a project doing SAP testing for performance, you could find some work doing performance testing on Java, especially considering that the SAP platform often includes Java processing in Netweaver. If you are doing performance testing while modernizing a web application, ask about mobile performance — and try to find a mentor to help you learn all you can about mobile platform performance.
As performance testers and engineers, ours is a richly complex and wonderfully deep discipline. It might take you 10 years to reach your full potential as a performance expert, but it is worth the effort. For me, looking back over my 20 years in this field, I can tell you that I have been incredibly lucky to have the mentoring I have had over the years. You should seek out people who will help you to learn about performance, expand your view of technology and shape your professional acumen. And someday, you can become the mentor, or write the book, or be the person who hires a new performance engineer to your team. In the end, the most important piece of advice I can give any performance tester is this: never let anyone else tell you how to be a performance tester. Use the tips I have shared, but remember, you must choose how to proceed on your own path.
About the Author
Mark Tomlinson Mark Tomlinson is a software tester and test engineer. His career began in 1992 with a comprehensive two-year test for a life-critical transportation system, a project which piqued his interest for software testing, quality assurance, and test automation. That first test project sought to prevent trains from running into each other — and Mark has metaphorically been preventing “train wrecks” for his customers for the past 20 years. He has broad experience with real-world scenario testing of large and complex systems and is regarded as a leading expert in software testing automation with a specific emphasis on performance.
Mark worked for six years at Microsoft Corporation as a performance consultant and engineer in the Microsoft Services Labs, in the Enterprise Engineering Center and in the SQL Server labs. His efforts to foster the success of Microsoft’s top-tier Enterprise customers was focused on their early adoption of Microsoft products as part of mission-critical operations. In 2008, as the LoadRunner Product Manager at HP Software Mark led the team to deliver leading innovations for performance testing and engineering as part of HP’s suite of performance validation and management products.
Mark now offers coaching, training and consulting to help customers adopt modern performance testing and engineering strategies, practices and behaviors for better performing technology systems.