#36 Verify Before Deploy: Load Testing for Production Readiness
Knowing how your application behaves under expected traffic before production helps you find problems and set proper resource limits. This prevents a lot of issues when real users start using app.
One of the common problems in building systems of any kind is expected traffic. Often, we do not know what to expect—especially when we are building a greenfield application and about to release it for the first time.
Then, we release it and observe the growing number of users. We monitor how the application works, and suddenly it stops working due to too much traffic. We are stressed as now we have to react fast to this situation to make the application available.
Instead, you could perform load and stress tests before you even think about releasing the application to the public, and thanks to it you know the numbers up in front. Today, I would like to focus on load testing which helps you check how your application performs under defined traffic for a given period.
Let’s assume you have developed an MVP that hasn’t been released yet. Based on your analysis, you expect around 50,000 concurrent users in the first month.
It is much better to discover problems now than when real users are hitting your system. No one wants to be firefighting issues while customers are complaining and executives are breathing down your neck. Load testing lets you find and fix bottlenecks before they become real problems.
What should you test? Focus on what matters most.
Think about it - users will not constantly update their profile picture or reset passwords. But they will hammer certain features hard. If you are running a ticketing platform, the checkout flow will get slammed. Payment processing might become your bottleneck. User default address updates? Not so much.
Put pressure where it counts. Identify your high-traffic features and load test them properly. Don't waste time load-testing everything equally - it is not how your app will be used in real life.
First, figure out which parts of your app need load testing. Focus on features you expect users to use the most. Sounds simple, right? Well, it depends.
If your app is not live yet, you are making educated guesses. Sometimes these guesses are good, sometimes they are shots in the dark.
Let me give you an example:
In a large hardware company, you might know exactly what is coming: 25,000 devices syncing their data exactly at 3 pm every day. That is a clear target to test for.
It would be trickier in a startup. Your users might surprise you by using features you didn't expect, or your whole business model might pivot next month and this will completely change your expected numbers.
If you already have a live application, check your analytics. The data will show you exactly where users are piling up.
The key is being realistic about how much you can predict. Sometimes you know exactly what is coming. Other times, you are making your best guess and need to stay flexible.
Once you have spotted your high-traffic areas, you will need a solid test plan. Here is what should be in it:
The scenarios you want to test. Thanks to the discovery, you have a list of areas you want to subject to load testing. Now, you need to develop specific scenarios for these areas. These scenarios will outline the actions and conditions you want to test. I recommend preparing scenarios that imitate the regular user flow. For example, in the case of buying tickets, your test should mirror the user’s journey from accessing the home page, searching for the event, opening the event page, selecting the number of tickets, proceeding to the checkout page, and completing the payment.
The number of concurrent users. It defines how many users you want to imitate to generate the expected traffic. This number may vary between different areas; for example, one area might typically handle 25,000 concurrent users while another might handle 100,000. I also recommend adding 20-30% for each load test because it gives you a certain reserve in case the traffic increases fast in the production.
The time limitation to perform load tests. Load tests should not run indefinitely, as this can incur costs and delay feedback. It is advisable to set a time limit, such as running each load test for approximately 30 seconds, with all tests completed within 10-15 minutes.
One common pitfall is assuming that users will follow a predictable pattern. For example, a 70% discount on a popular item may not be spread evenly over 15 minutes for 15,000 users (1,000 users per minute). Instead, you might see most of the users in the first few seconds. Be cautious of such scenarios!
The last aspect to consider is where and when to run load tests. This is a nuanced question with no single answer; it depends on your specific context.
Load testing should not be a one-time action but a repeatable process. Ideally, it should be conducted for every change made to the production code. However, this may not always be feasible (as discussed in previous steps). Therefore, there are various approaches to scheduling load tests.
If you deploy the application multiple times per day (like in continuous deployment), it makes sense to run load tests each time the commit is pushed to production. This way, load tests will be a part of your deployment pipeline.
Some companies can't just push code whenever they want. Maybe you are dealing with banking software or medical systems - places where every release needs serious vetting. In these cases, your load tests happen before your scheduled releases - think quarterly or bi-annual deployments. You will run them in staging or UAT environments, giving yourself plenty of time to catch and fix issues before the big release day.
It is less frequent than continuous testing, but that is fine - it matches how you actually ship code. The key is having a testing rhythm that fits your release cycle.
How do you handle load tests in your application?
Join 100,000 architects and engineers building apps and transforming workflows with Budibase.