Efficient Handling of Long Lines

During the initial development of AIMMS, it seemed reasonable to limit output file line lengths to around 32 KB.  However, in today's context, it is not uncommon to encounter single-line data files exceeding 100 MB.

Recently, I had the opportunity to work with a customer on a service that involved handling input files. 
These files were single-line files of approximately 100 MB in size, and although the format was similar to JSON, it had some slight differences.
 

Here's how it works:

1. dex::ImportStreamContent(streamName, initialContents): This function creates a stream called streamName and initializes its contents with initialContents.
2. dex::AppendStream(streamName, additionalContents): You can use this function to append additionalContents to the existing stream named streamName.
3. dex::WriteStreamToFile(streamName, fileName): This function allows you to write all the contents of the specified stream to the file specified by fileName.

More about memory streams can be found here.

To evaluate the performance of these methods, I conducted a test on my laptop. I attempted to write out a single-line file with a size of 130 MB using two different approaches:

a. dex::WriteToFile: This method took approximately 12 seconds to complete.
b. The three dex procedures mentioned above: In this case, the execution time was around 70 seconds.

Please find the example project used attached.