Introduction to Testing in Production & Tips to do it the Right Way

Testing in production (TiP) can be described as the process of performing several types of software tests in a production environment where it is accessible to the end-users. It is quite rare to find a test environment that is exactly the copy of the production environment, so the scale is not the same. With most of the processes in software development, experts suggest considering the risks, rewards and best practices before moving to test in production.

Why TiP?

Testing in production introduces risks but it also provides several benefits, which is why a tester should consider it. The major benefit of testing in production is that it allows you to see how the application will work in the live environment in which it runs.

Testing in production becomes necessary in the case of complex services, as it would become difficult to imitate a production environment and know how the service will be utilized by the users. The staging and the real environment are never the same. Some organizations might not have the budget to create this type of environment again that closely resembles the production environment, which is why they switch to testing in production.

Test environments do have certain limitations, such as smaller capacity and smaller data volume, being different from the production environment. These limitations signify that the results are not always extendable to the production environment. It can be roughly said that it puts the software through the same distress as it will face in the “real world”. Some of the key aspects that one should have in mind while designing testing in the production environment are as follows:

# The data used for the testing should be encrypted as it is ‘real’.

# The load injectors should behave like real users.

# Different environments involve different technologies and it is quite expensive to have experts in all the areas.

# Time having a low impact (generally at night) should be found out.

# There is a need for a data set for testing.

# It is helpful to create a map with the testing phases along with the risks for each phase.

Tips for Testing in Production

Following are some of the tips that can be implemented in order to develop robust procedures for testing in your production environment without creating a mess for the users.

1. Divide the production testing into layers: While we talk about the idea of ‘testing in production, it includes testing the applications running separately on the production platform, running live tests against 100% deployed code, and testing the whole list of test servers within the production data centre. Therefore, production testing should be layered in order to test different aspects of the production environment in different ways.

2. Schedule the time with low usage: Performance testing can have a great impact on the entire user base. It can make the server environment inactive which is something no one wants. One should study analytics and determine when is the best time to schedule the tests. For example, look for the following:

# Count of users on-site.

# Revenue generation across the site.

# Resource-intensive processes within the environment.

3. Collect original traffic data for testing: One should collect and use actual traffic data in production (such as user workflows, resources and user behaviour) to drive the load generation for test cases. When you exercise the tests within the production environment, you will have the confidence that the simulated behaviour is real.

[ Related Read: How To Improve Test Case Efficiency? ]

4. Focused monitoring: While running a production test, always keep an eye on user performance metrics so that the user knows if the test is having an unacceptable impact on the user experience. Prepare yourself to shut the test down if that is the case.

5. Create an “Opt-in” experience: A great way to test how the application is going to perform with real users is to have some “opt-in” to new feature releases. This will allow the user to monitor and collect data from the real-time users and make the adjustments accordingly to your testing strategy without concerning about impacting their experience.