Writing a software usually happens on a machine with a prepared testing set of data. When you write another feature, you just slightly modify the testing set. Then you just go along by modification of the existing set again and again. Does it sound familiar?
How about deployment of the software? If you run continuous delivery pipeline, you most likely deploy directly to production with each new approved commit. That means you are deploying the software to an environment with pre-existing data.
Do you remember the last time you tested your cloud-deployed software with no data at all? How about 3rd party services and their state? Would you be able to bet your salary it works without issues?
Testing a software with no data or in the initial deployment state seems to be slightly forgotten use-case with the continuous delivery pipeline. You might then be surprised how the software behaves when a new user signs up, or when you just clear your statistics database.
It is very important to test a software in an initial state with no data including 3rd party services due to three main points:
- Discovery of trivial bugs like expectation of data in list of items instead as oppose to an empty list
- Disaster recovery – what if you need to suddenly re-deploy your database or change your provider?
- User experience – what a user sees when they have no data in their profile?
I know software engineers must work with constraints like expensive 3rd party environment or expensive resources. On the other hand we cannot have a stable software without appropriate investment. Any shortcut during the testing would be unfortunately “paid off” in production.
I would like to recommend to all team leaders and stakeholders – be curious and ask your software development team how your software behaves in an initial state and how you can support them in testing this use-case.