Ok, the below is from my limited knowledge so may need correcting.

The base file format is simple, the complication is the compression methods (which includes encoding) used and making sure that is all supported as it is part of the actual file format specification.

I have issues with some of the speed claims as they seem to stem from statements that the column based, as opposed to row based layout, means it doesn't have to read the whole file but that is pretty limited, as long as you need to look at a single record and the last field then you still have to read through the whole file.  It also ignores that for most of the usage I have had of CSV files I have needed to ingest the whole file anyway.

Any other speed improvements are likely due to the code and database ingesting/searching the data not the format itself.

That doesn't mean there aren't advantages.  It may be easier to write more efficient code and the built in compression can reduce time and costs.

So the question is how popular is this format.  Well it's pretty new and so hasn't made great inroads as yet, however, given Intersystems interest in "Big Data", ML, and it's integration with Spark etc...  it does make sense as it is part of Hadoop and Spark.

One thing to note though is that python already supports this through the PyArrow package and so this might be the best solution, simply use embedded python to process the files.

As far as I recall this isn't logged at the point the status becomes "dead" only when cleaned up.

We once wrote a scheduled task that looks for and logs these processes using IsGhost() as they can sit there a while waiting to be cleaned up at times.

There is a mention in the IRIS documentation that says these are logged to the Event Log but I am not sure what happens if you aren't using Interoperability or Productions in IRIS, I suspect there is no logging then as it is part of Production Monitoring.

As Eduard suggests, you need to know the class that the pInput object is derived from as it doesn't seem to be a valid Stream if .Size fails.

It also looks like you might be expecting a file as you retrieve the name so why not use the %File class if the file already exists.

In fact, noting how you get the Filename using the Attributes method this looks like the pInput object is a response with content which means the actual stream will be in pInput.Content

It is far too long since I worked on Hospital systems to give an authoritative answer given extra restrictions you may have but for https the connection and data are already secure.

For outbound infrastructure can ensure it is routed outside appropriately and you could even limit traffic to specific external endpoints.

For inbound a load balancer or reverse proxy in the DMZ can keep your system away from the outside world and limit traffic both between the two and only allow specific external endpoints to have access.

A dashboard for monitoring when SAM or a third party tool is not available.
Make sure there is the ability to add custom elements for the application and to select what elements are displayed to a user.
Also provide access to screens such as locks and processes with granular control over what they are able to do.

Essentially you don't want such users to have full management portal access just selected screens which are useful fronted by a monitor/dashboard that allows them to keep an eye on the database and application.

Just as a note, where you are manually building up your global in gbl you could use the $name function instead, for example...

set gbl="^NigelGlobal(""Subscript1"""_","_"""Subscript2"")"

can be

set gbl=$name(^NigelGlobal("Subscript1","Subscript2"))

which makes it easier to handle quotes etc...

update: ignore the very odd formatting this editor applied to the objectscript.

All the current suggestions are possible, however, usually if it is a Cache/Ensemble start issue the exe will not crash but will report the database failed to start.

In this case the error is that the EXE itself crashed and this is likely why there is nothing in alerts.log or cconsole.log as it did not progress that far.

This implies either an issue with the EXE or with the config details it will read from the registry.

If you check Windows Event Viewer there may be more information there to help resolve.

Should you be unable to resolve then the simplest option would be to install a fresh copy of Cache/Ensemble, first try the repair option with the same version then try upgrading with a newer version, over the top.  This should not lose anything and will either highlight an installation issue or rebuild the problem configuration.

If that fails, keep a copy of the database files, wipe the existing installation and then rebuild, extreme but it depends on how long you want to spend trying to find the problem.