Blog

About the 3rd 'wave' in AIMMS data management technology

Related products: AIMMS Developer Integrations and Data Exchange AIMMS PRO and Cloud Platform

 

A collection of new technologies much improves how you can manage, import and export data in, into and from AIMMS. 

This matters because it:

  • can improve end-user experience
  • reduces the effort to integrate with external data systems
  • allows leverage of emerging corporate data repositories 

Therefore I invite you to learn about these new technologies and explore their relevance for your job. You can read a bit more in this update.

 

 

Without data there is no optimization. Without data AIMMS models cannot do their magic. Without data management tooling AIMMS models will not have data. 

For 30+ years we have been developing data management features into our products. First we focused on files and file storage, starting with files on your local computer. Later on centrally in PRO Storage, a first step in enabling multi-user data sharing. 

Next we added support for relational databases to manage data for AIMMS models. Many app developers adopted this method. It enabled powerful querying, improved multi-user experiences and improved data exchange with external platforms. 

AIMMS data management technology is now entering a 3rd 'wave’ with a return to files, a move to lower-code data exchange features and use of 3rd party Azure technology. In our eyes this 3rd wave is opening an ‘opportunity candy store’ to improve user experience, app developer productivity and data exchange with external systems. 

Why are we making this move?

  • We noticed customers increasingly consolidating their data in centralized data repositories such as data lakes, often using Parquet files.
  • We saw a growing preference for using REST API services for data exchange
  • We realized the majority of AIMMS models work with what-we-call ‘immutable data sets’. A complete set of data values that is loaded/saved as one entity. For that a file is more logical container than a relational database. 
  • Most of our customers are adopting cloud vendors such as Azure and AWS as their primary IT platforms

What is the move?

A combination of technologies:

  • Data Exchange Library, a new AIMMS library that offers low-code tools for (a) converting model identifiers into files and vice versa, to (b) generating REST Clients and REST Servers for data import and export.
  • We are adding an Azure Data Lake Storage service to each cloud account. This enables app developers to build powerful data management solutions, for example storing and retrieving AIMMS model data for access by external systems, possibly with the growing collection of off-the-shelf connectors for Azure Data Lake Storage, including PowerBI, Databricks, Snowflake, SQL Server, etc. 
  • A growing collection of ready-to-use connectors between AIMMS models and applications such as Snowflake and PowerBI.
  • Features to deploy Python scripts inside your AIMMS cloud account, running in an exclusive Docker container with a choice of Python images. 

So why am I telling you this?

I am excited to share this '3rd wave’ with you, because of all the opportunities it offers. I believe it will empower you, AIMMS app developer, to achieve more with less effort. I believe it will help you further increase your impact. And ultimately I am confident this will lead to even better decisions, our AIMMS purpose. 

And how about CDM?

We have come to see that CDM, Collaborative Data Management, is a very powerful tool for building AIMMS applications that support concurrent data editing by multiple users. And we also realize it is a complex tool that requires considerable learning. So we recommend only to use it for that 'sweet spot'. 

Finally

My intention with this update is to make more AIMMS app developers explore and adopt these important innovations, to trigger also their curiosity. I am not the right person to give you detailed explanations. Our documentation and my colleagues in User Support will take care of that.

And I want you to know that these developments are also new for us. We are learning about them everyday, mostly from early adopters among our app developer community. Their feedback and the feedback from our SC Navigator colleagues is flowing back into our development work. 

Acknowledgement

I would like to acknowledge Marcel Roelofs, our CTO, for his compelling and credible vision on this topic. Because this 3rd wave of AIMMS data management is mostly his brain-child, like so many AIMMS innovations.  

Hi Team! Amazing innovations here, thanks for the updates! Is the Snowflake ready-to-use connector already available, or is this something which is still in the works and part of next year’s roadmap?


Can this technology and the Azure Data Studio be used to manage data for AIMMS SC Navigator and to load data into SC Navigator?


Hi Diego, thanks for asking. The Snowflake ready-to-use connector is still ‘in the works’ and we have no release timing yet. But we might still be able to help you now with the example we built ourselves. Please contact my colleagues at support@aimms.com and ask for @RoxannaBindenga. She can support you there.


Hi Jonathan, you also thanks for asking. The short answer: Yes!

In a few weeks we will release a new version of SC Navigator that uses the Azure Data Lake Storage instead of a relational database. And with that we will release a comprehensive collection of documentation how to manage data for SC Navigator using Parquet files and REST services. And a bonus: data set loading and saving is much faster using this new technology!


@Diego Perez-Urruti, all,

The initial pubiic release of the Snowflake library is available now via the Library Manager; see also:
https://documentation.aimms.com/snowflake/library.html

 


Hi Jonathan, you also thanks for asking. The short answer: Yes!

In a few weeks we will release a new version of SC Navigator that uses the Azure Data Lake Storage instead of a relational database. And with that we will release a comprehensive collection of documentation how to manage data for SC Navigator using Parquet files and REST services. And a bonus: data set loading and saving is much faster using this new technology!

Exciting @Jan-Willem!


@Diego Perez-Urruti, all,

The initial pubiic release of the Snowflake library is available now via the Library Manager; see also:
https://documentation.aimms.com/snowflake/library.html

 

Thanks @Gertjan I’m asusming this library is only available for cloud customers and not on-premise ones. Is this correct?


Hi @Diego Perez-Urruti,

You can use the Snowflake library just fine as an on-premise customer. The only requirement is having access to a Snowflake account, and to an Azure Blob Storage account. The latter is used as intermediate storage for communicating parquet files between AIMMS and Snowflake. This could also have been AWS S3, or some other cloud storage solutions supported by Snowflake, but we chose Azure Blob Storage because of our AIMMS Cloud platform which runs in Azure. 

For development of the Snowflake library, we just used some Azure Blob Storage account, the credentials of which were communicated through the config file of the library. For authorizing Snowflake you can use any external OAuth provider supported by Snowflake, or you can use public-private key authentication, which will make it completely independent of an external identity provider. 

Setting all of this up is described extensively in the Snowflake documentation, which I'm certain your IT department will be very capable of helping you out with.

 


Didn't find what you were looking for? Try searching on our documentation pages:

AIMMS Developer & PRO | AIMMS How-To | AIMMS SC Navigator