Recently I spoke about automated testing with my colleagues who are developing our off-the-shelf supply chain apps.
How and how often do you run the automated tests?
Each time we commit AIMMS code to our code repository in Gitlab, an automated pipeline
runs our entire test collection. Around 15 minutes later we will know whether any tests failed as a result of the latest code changes.
How many tests are in that test collection?
We have unit tests and end-to-end tests. For the unit tests we use the AIMMS UnitTest library. For the end-to-end tests we are using a proprietary framework that has been developed in-house and that is being used by several AIMMS teams. For Network Design, for example, we currently have over 450 unit tests and over 400 end-to-end tests, offering us a coverage of about 85%.
That is quite an investment, all those tests. What do you get in return?
Oh, we get a lot in return. First of all, it makes us feel very confident when we are changing or refactoring code. We cannot always foresee all implications of a change, but we know the tests are there to protect us, for example from introducing regressions. Trust me, the tests often catch issues! And it is so nice to get almost instant feedback. It helps you to go faster. And we do not have to trouble the manual testers. They can focus on testing the new features.
It is so nice to get almost instant feedback.
What else do you get in return?
We find we are improving our code by writing test scenarios because you have a different mindset when writing test scenarios. You might think of situations or combinations that you forgot about when writing your code. You can directly adapt the code then. Sometimes while writing the test, it does not give the expected result, so you already spotted another bug there. By writing tests (scenarios) we go over the model again and sometimes find issues because of that. And it makes it much safer for new people to work on the code, so onboarding is quicker.
I can imagine it takes a lot of time, writing and maintaining all these tests. How do you experience that?
To be honest, it takes quite some discipline to keep the test coverage up. Building new features is more fun. We are keen to get the new features to the users, so we are tempted to postpone writing the tests until later. But in the end, we know we gain time as we don’t have to spend a lot of time on fixing bugs reported by customers.
Onboarding new people becomes easier and safer.
Any tips or tricks for other app developers who want to start using automated tests?
Start small. Start by creating a few unit tests. You can use the AIMMS Unit Test Library for that. Just to get a feel for the work and the feedback it provides. No need for pipelines initially. Just run the tests manually to gain some experience and hopefully reap some benefits. Then gradually expand.
Start by creating a few unit tests. No need for pipelines initially.
What is the next step for the team on this topic?
We want to get closer to a 100% test code coverage. There are still some areas where we need to write additional tests, so that is one step we focus on. Another step is that we want all our app developers to be able to write end-to-end tests themselves ánd run these tests locally before committing their code changes. That would allow for even quicker feedback on our code changes.
Note: In this interview the team mentions the proprietary end-to-end test framework. That is an in-house development, not suitable for distribution. We are of course considering if and how to make that available for our customers, but we have no specific plan or timing for that yet. Our plans will also depend on demand and other priorities.