How the AIMMS SC app developers use automated testing

  • 3 December 2021
  • 5 replies
  • 151 views
How the AIMMS SC app developers use automated testing
Userlevel 3
Badge +4

Recently I spoke about automated testing with my colleagues who are developing our off-the-shelf supply chain apps.

How and how often do you run the automated tests?

Each time we commit AIMMS code to our code repository in Gitlab, an automated pipeline

runs our entire test collection. Around 15 minutes later we will know whether any tests failed as a result of the latest code changes.

How many tests are in that test collection?

We have unit tests and end-to-end tests. For the unit tests we use the AIMMS UnitTest library. For the end-to-end tests we are using a proprietary framework that has been developed in-house and that is being used by several AIMMS teams. For Network Design, for example, we currently have over 450 unit tests and over 400 end-to-end tests, offering us a coverage of about 85%.

That is quite an investment, all those tests. What do you get in return?

Oh, we get a lot in return. First of all, it makes us feel very confident when we are changing or refactoring code. We cannot always foresee all implications of a change, but we know the tests are there to protect us, for example from introducing regressions. Trust me, the tests often catch issues! And it is so nice to get almost instant feedback. It helps you to go faster. And we do not have to trouble the manual testers. They can focus on testing the new features.

It is so nice to get almost instant feedback.

What else do you get in return?

We find we are improving our code by writing test scenarios because you have a different mindset when writing test scenarios. You might think of situations or combinations that you forgot about when writing your code. You can directly adapt the code then. Sometimes while writing the test, it does not give the expected result, so you already spotted another bug there. By writing tests (scenarios) we go over the model again and sometimes find issues because of that. And it makes it much safer for new people to work on the code, so onboarding is quicker.

I can imagine it takes a lot of time, writing and maintaining all these tests. How do you experience that?

To be honest, it takes quite some discipline to keep the test coverage up. Building new features is more fun. We are keen to get the new features to the users, so we are tempted to postpone writing the tests until later. But in the end, we know we gain time as we don’t have to spend a lot of time on fixing bugs reported by customers.

Onboarding new people becomes easier and safer.

Any tips or tricks for other app developers who want to start using automated tests?

Start small. Start by creating a few unit tests. You can use the AIMMS Unit Test Library for that. Just to get a feel for the work and the feedback it provides. No need for pipelines initially. Just run the tests manually to gain some experience and hopefully reap some benefits. Then gradually expand.

Start by creating a few unit tests. No need for pipelines initially.

What is the next step for the team on this topic?

We want to get closer to a 100% test code coverage. There are still some areas where we need to write additional tests, so that is one step we focus on. Another step is that we want all our app developers to be able to write end-to-end tests themselves ánd run these tests locally before committing their code changes. That would allow for even quicker feedback on our code changes.  

Note: In this interview the team mentions the proprietary end-to-end test framework. That is an in-house development, not suitable for distribution. We are of course considering if and how to make that available for our customers, but we have no specific plan or timing for that yet. Our plans will also depend on demand and other priorities.


5 replies

Any exposure to the thought process behind the end-to-end test framework would be super helpful!

Userlevel 3
Badge +4

Hi Benjamin, thanks. Any chance you could be more specific on what would help you? I am asking because there are many aspects that I can think of. For example:

  • Strategy how to combine E2E tests and Unit Tests for maximum effectiveness
  • How to write effective E2E tests for your AIMMS application
  • Tooling used for creating E2E tests (ours is an in-house development, not so suited for handing it to others)

Looking forward to your response.

Jan-Willem, AIMMS

Hi Jan-Willem,

 

All of these would be be great. I am more used to the pythonic unit testing / code coverage flow. I’d also be interested in understanding the framework of E2E testing within an AIMMS project (let’s say bullet’s 1 and 2).

My org would also be very interested in CI style automation thoughts. For example, we’ve started creating an approach with CLI & AIMMS unit tests, but would benefit from code coverage tools, and potential examples of how these tests could be automated to the cloud.


Thanks for the response,

Ben 

 

Userlevel 4
Badge +4

Dear @benjaminleroy 

I would like to share some information regarding this topic.

  1. Unit testing

We have the following articles available that can assist you on setting up unit testing.

Using the unit testing library (how to):

https://how-to.aimms.com/Articles/216/216-effective-use-unit-test-library.html

 

Library documentation (language reference):

https://documentation.aimms.com/unit-test/index.html

 

  1. Code coverage

We do not have a way to completely track code coverage. As of now, you are able to track code that is accessed, but cannot list code that is not covered.

https://documentation.aimms.com/functionreference/development-support/profiler-and-debugger/profilercollectalldata.html

If you feel you need additional functionality, such listing all executable code, feel free to share the idea in the ideation group.

 

  1. E2E

Some of our customers do implement E2E Testing. Even internally, for our SC Apps, we have implemented this kind of solution. I will ask my colleagues to share strategies on this.

 

  1. CI

We have also seen customers using CI automation with AIMMS. I would like to point out that in our roadmap we have excellent functionality coming up to help with this.

To quote @Gertjan :

PRO REST services

Providing programmatic access to our cloud deployment platform for publishing apps, running apps, managing users/groups/permissions, etcetera.

 

Feel free to read more on this with the link below:

 

I hope this has helped clarify your questions, but some still remain, feel free to reach out!

Userlevel 3
Badge +4

And I can confirm that the ‘Providing programmatic access to our cloud deployment platform for publishing apps, running apps, managing users/groups/permissions, etcetera.’ is due for release in Q2. This will take the form of a collection of REST API services to manage users, groups, apps and sessions on the AIMMS Cloud Platform.

Reply


Didn't find what you were looking for? Try searching on our documentation pages:

AIMMS Developer & PRO | AIMMS How-To | AIMMS SC Navigator