Can there be a Free-Threading built for the aimmspy library on pypi.org?

Dear Guido,

This is a great idea - I have shared it with our team and we will evaluate on when/how this could be implemented. I'll get back to you, when I have an answer.

Best regards,

Dear ​@gdiepen ,

We looked into this with our engineering team, and I want to give you an answer.

The short version: free-threaded (cp313t/cp314t) wheels for aimmspy wouldn't actually get you what you're after. The AIMMS engine itself only supports one active instance per process, regardless of how the Python bridge is built or how many threads you use. So even with a free-threaded build, you'd still be limited to a single AIMMS engine running at a time within one process, which means the threads wouldn't be able to run independent solves in parallel the way you're hoping.

The good news is that multiprocessing gets you there today, with the standard builds you already have. Since your pricing problems are independent, you can run them across separate processes (each with its own AIMMS engine instance) using Python's multiprocessing module, and that will scale across your available cores exactly the way you're describing for the column generation loop. The overhead of spinning up separate processes is generally much smaller than the overhead of the ILP formulation you mentioned wanting to avoid, so this should still get you a solid speedup.

Happy to help think through the multiprocessing setup if useful, for example how to structure the worker pool so each process initializes its own AIMMS instance efficiently.

Best regards, Luis

Hi Luis,

I understand that there are limitations with regards to the aimms engine being tied to just a single thread and that this potentially could cause problems if you use the multi threading within python and have multiple threads connect to the same aimms engine.

My idea for the approach would be slightly different

• From AIMMS call a function solve_pricing_problems that is in a python module
• This function will first request all dual multiplier information from AIMMS using the python bridge functions (Since there are quite some constraints, this will take a bit of time)
• Then, the function will trigger all of the actual separate pricing problems. Each of these problems is independent from the rest and does NOT need any communication with the AIMMS engine anymore because the main-thread of the function got all of the data initially
• The function will wait till all pricing problems are solved and when this is done will gather all results, combine all of them and communicate that back to the AIMMS engine

The third step is where the advantage of using threads vs multiprocessing happens: there is no need to start a new process (which compared to starting a thread is still very expensive). Also, there is no need for the multiprocessing to serialize the dual-multiplier information anymore to make it available to the newly spawned processes (which uses pickle internally to provide the data to the child processes that are started). Also the results from each of the sub processes that was run to solve a pricing problem instance need to be serialized again to make it available in the main process.

There might be some possibilities of using the multiprocessing.SharedMemory functionality provided by Python in combination with numpy using the buffer of such a SharedMemory object to try and prevent as much of the large data serialization as possible

So to be clear, it is not my goal to connect to the AIMMS engine from multiple threads: only one thread will be connecting with the aimms engine and this threads coordinates the solving of the actual pricing problems using additional threads.