Cookie monster and other troubles (and some workarounds too) we ran into while doing Django on IRIS WSGI

As a part of the IRIS Python 2024 contest, my colleague Damir and I went with an idea to build a platform called ShelterShare for connecting victims and volunteers for shelter requests . To do so we chose django as a framework and proceeded to build the first version with 3 different docker containers, django, iris and nginx which would then utilize IRIS as a pure Database engine via the beautifly composed django_iris (cudos to Dimitry). As we were progressing fast, we decided to explore the option of running it within the same container as IRIS by utilizing WSGI added in 2024.1. We knew ahead of time that we won't be able to entirely rely on WSGI as we were utilizing WebSockets for instanenous updates and communication in the tool, but we figured we can always run uvicorn in the container in parallel to iris and hook-up websocket to it on a different port.

And, that's when we started running into problems...

Our first issue was that we were using an older version of django-iris which was relying on a package called iris and which was conflicting with the inbuilt iris.py (i.e. part of IRIS WSGI). We realized that the issue was resolved in a later django-iris package by renaming iris to intersystems_iris, so we updated django-iris, resolved the issue and moved on.

Our second issue came up when we wanted to utilize ipm to install the package from the module. For whatever reason it kept failing to do the migration with strange ConnectionReset errors...

sheltershare    | Waited 3 seconds for InterSystems IRIS to reach state 'running'
sheltershare    | 
sheltershare    | Load started on 07/26/2024 14:39:06
sheltershare    | Loading file /usr/irissys/csp/sheltershare/module.xml as xml
sheltershare    | Imported document: sheltershare.ZPM
sheltershare    | Load finished successfully.
sheltershare    | 
sheltershare    | Skipping preload - directory does not exist.
sheltershare    | Load started on 07/26/2024 14:39:07
sheltershare    | Loading file /usr/irissys/csp/sheltershare/module.xml as xml
sheltershare    | Imported document: sheltershare.ZPM
sheltershare    | Load finished successfully.
sheltershare    | 
sheltershare    | Loading sheltershare in process 716
sheltershare    | [%SYS|sheltershare]	Reload START (/usr/irissys/csp/sheltershare/)
sheltershare    | [%SYS|sheltershare]	requirements.txt START
sheltershare    | Collecting Django==5.0.7
sheltershare    |   Using cached Django-5.0.7-py3-none-any.whl (8.2 MB)
sheltershare    | Collecting uvicorn[standard]
sheltershare    |   Using cached uvicorn-0.30.3-py3-none-any.whl (62 kB)
sheltershare    | Collecting channels
sheltershare    |   Using cached channels-4.1.0-py3-none-any.whl (30 kB)
sheltershare    | Collecting Faker
sheltershare    |   Using cached Faker-26.0.0-py3-none-any.whl (1.8 MB)
sheltershare    | Collecting django-iris
sheltershare    |   Using cached django_iris-0.2.4-py3-none-any.whl (134 kB)
sheltershare    | Collecting tzdata
####LOG TRIMMED FOR BREVITY####
sheltershare    | [%SYS|sheltershare]	requirements.txt SUCCESS
sheltershare    | Skipping preload - directory does not exist.
sheltershare    | [%SYS|sheltershare]	Reload SUCCESS
sheltershare    | [sheltershare]	Module object refreshed.
sheltershare    | [%SYS|sheltershare]	Validate START
sheltershare    | [%SYS|sheltershare]	Validate SUCCESS
sheltershare    | [%SYS|sheltershare]	Compile START
sheltershare    | [%SYS|sheltershare]	Compile SUCCESS
sheltershare    | [%SYS|sheltershare]	Activate START
sheltershare    | [%SYS|sheltershare]	Configure START
sheltershare    | [%SYS|sheltershare]	Configure SUCCESS
sheltershare    | Studio project created/updated: sheltershare.PRJ
sheltershare    | [%SYS|sheltershare]	Activate SUCCESS
sheltershare    | ShelterShare installed successfully!
sheltershare    | 
sheltershare    | 126 static files copied to '/usr/irissys/csp/sheltershare/static', 3 unmodified.
sheltershare    | /usr/irissys/mgr/python/django/core/management/commands/makemigrations.py:160: RuntimeWarning: Got an error checking a consistent migration history performed for database connection 'default': [Errno 104] Connection reset by peer
sheltershare    |   warnings.warn(
sheltershare    | No changes detected
sheltershare    | Traceback (most recent call last):
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/dbapi/_DBAPI.py", line 47, in connect
sheltershare    |     return native_connect(
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/_IRISNative.py", line 183, in connect
sheltershare    |     connection._connect(hostname, port, namespace, username, password, timeout, sharedmemory, logfile, sslcontext, autoCommit, isolationLevel, featureOptions, application_name)
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/_IRISConnection.py", line 304, in _connect
sheltershare    |     raise e
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/_IRISConnection.py", line 212, in _connect
sheltershare    |     self._in_message._read_message_sql(sequence_number)
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/_InStream.py", line 46, in _read_message_sql
sheltershare    |     is_for_gateway = self.__read_message_internal(expected_message_id, expected_statement_id, type)
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/_InStream.py", line 59, in __read_message_internal
sheltershare    |     self.__read_buffer(header.buffer, 0, _MessageHeader.HEADER_SIZE)
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/_InStream.py", line 138, in __read_buffer
sheltershare    |     data = self._device.recv(length)
sheltershare    |   File "/usr/irissys/mgr/python/intersystems_iris/_Device.py", line 40, in recv
sheltershare    |     return self._socket.recv(len)
sheltershare    | ConnectionResetError: [Errno 104] Connection reset by peer

which we weren't able to resolve, so we fell back to using Dockerfile, entrypoint.sh and docker-compose to fully setup the django app in the /usr/irisys/csp folder, then relied on ipm to load our app.xml into Security.Applications.

import iris
iris.system.Process.SetNamespace("%SYS")
imported_status=iris.cls("Security.Applications").Import("/usr/irissys/csp/sheltershare/app.xml", num_imported,0)

This works well and is a reliable way of deploying. You can check out an example of setting it up here: ShelterShare-SingleDocker

Note though that without the merge.cpf, the irissetup.py script will fail to run iris due to Authorization issues... i.e.
 

RUN \
  cd /usr/irissys/csp/sheltershare && \
  iris start IRIS && \
  iris merge IRIS merge.cpf

Now we ran into a big problem, well big for us because it was really difficult to understand what was going on, but in the end fairly easy to workaround...

You see, we had relied on django authentication to handle our user accounts and groups etc. which we've utilized by simply doing:

user_obj = authenticate(username=request.POST['username'],
                        password=request.POST['password'])
if user_obj:
    login(request, user_obj)
    return redirect("index")

Which works great in gunicorn, uvicorn, even simple python manage.py runserver... but in IRIS it kept silently failing and throwing us back to login screen.

After much digging, browser console debugging and inspecting, we realized that in normal circumstances, the WSGI/ASGI server in question (e.g. uvicorn, gunicorn) will on successful login return 2 Set-Cookie headers, one with CSRF token and some other info, while the other contains the sessionid.

The sessionid then gets saved to browser storage and is utilized whenever accessing a page under the same domain.

However, it seems that IRIS WSGI, for whatever reason, combines the two Set-Cookie headers into one. See difference below:

UVICORN:

 
IRIS WSGI:
 and the associated Cookie store:

 
So we started figuring out how to circumvent this problem as Browser was obviously ignoring the sessionid from the single Set-Cookie header, so we attempted the following:

WORKAROUND:

user_obj = authenticate(username=request.POST['username'],
                        password=request.POST['password'])
if user_obj:
    login(request, user_obj)
    response = redirect("index")
    # Add your custom header
    # Extract the sessionid from the Set-Cookie header
    sessionid = request.session.session_key

    if sessionid:
        # Add the sessionid as a separate Set-Cookie header
        response.set_cookie('sessionid', sessionid, httponly=True)

    return response

and now the sessionid and CSRF were still in the same Set-Cookie header, but as it happens, sessionId was at the very beginnign of the Set-Cookie so the browser picked it up without any issue...
AFTER WORKAROUND IRIS WSGI:

 and the result in the browser Cookie storage:

So, with this Workaround, the django-authentication begun working properly and we could use the application for most of the non Async content. However, we then ran into the issue of POSTs not working correctly, possibly because the CSRF token was now pushed to the back of the Set-Cookie (but could be some other reason) and we didn't have enough time to research further workarounds. In the end, with POSTs issues and our need for ASGI, we went back to the initial three docker container solution and plan to investigate WSGI (and perhaps ASGI) some more in the future...