Question
· 3 hr ago

XDBC memory leaks

I have a business service that actively reads data from a remote Postgres database. OnProcessInput opens a XDBC (actually JDBC) connection, executes an SQL query, fetches several thousand rows, iterates the resultset, and closes the connection. On each iteration I also need to update each source row in the remote database using PreparedStatement.

In other words, in every OnProcessInput call I have a long running SELECT statement and several thousands small UPDATE statements.

The problems I'm facing are:

  1. I cannot execute an UPDATE statement via the same connection that was used before to select rows. IRIS raises exceptions of kind " 34 %PreInvoke+1^%Net.Remote.Object.1". Obviously, a separate connection instance is needed for updates.
  2. In the Java/.NET world, a database connection is a reusable resource that is managed by a pool. But if I start creating and destroying connections frequently, I see a drop in the performance of UPDATE operations (about 10x):
Method ReadRows()
{
  Set cnn = ##class(%XDBC.Gateway.Connection).GetConnection("postgres")
  Set st = cnn.CreateStatement()
  Set rs = st.ExecuteQuery("select * from my_postgres_table limit 10000")
  While (rs.Next()) {
     Set id = rs.Get("id")
     Do ..UpdateRow(id, "S")
  }
  Do rs.Close()
  Do st.Close()
  Do cnn.Close()
}

/// DOES IRIS LEVERAGE JAVA/JDBC connection pooling?
Method UpdateRow(id as %String, status as %String)
{
   Set cnn = ##class(%XDBC.Gateway.Connection).GetConnection("postgres")
   Set st = cnn.PrepareStatement("update my_postgres_table set status = ? where id = ?")
   ...
   // Set parameters, execute UPDATE
   Do st.Close()
   Do cnn.Close()
   Set cnn = ""
}
  1. This approach also results in a rapid increase in memory consumption by the IRIS server. Pretty soon, IRIS eats up all the memory and I have to restart the Language Server.
  2. If I create longer-lived connections, the UPDATE performance is much better, but memory leakage still occurs, although not that quickly.

What am I doing wrong? How can I avoid memory leaks while achieving maximum performance for data update operations in the remote database?

Product version: IRIS 2025.3
Discussion (4)3
Log in or sign up to continue

I'm afraid I don't have an answer, I use JDBC regularly (admittedly , not Postgres and NOT using %XDBC classes) and I've never seen this behavior.
If anyone wants to dive deeper in this issue, more information are required to analyze and maybe try to reproduce it, like:

"....rapid increase in memory consumption by the IRIS server" What process is consuming memory? 
What OS, Windows or Linux? If Linux, what flavor?
What Java version are you using?
What version of Postgres  JDBC driver are you using?

Last but to least, note that class %XDBC.Gateway says:
"FOR INTERNAL USE - do not invoke directly"

Hi Enrico,

  1. I'm aware of that warning "FOR INTERNAL USE" in the source code, but the way I create XDBC connections is officially documented

  2. The OS is Ubuntu, IRIS is containerized, the memory is consumed by the IRIS Java language server

  3. Regarding memory leaks, I suspect that ResultSets are not closed properly - method %XDBC.Gateway.JDBC.ResultSet::Close() is empty and does nothing but it seems to be it should call ..%externalResultSet.close(). But I'm not sure it is the only reason

  4. UPD: calling Java GC really helps:

Set gateway = $system.external.getGateway("%Java Server")
Do gateway.invoke("java.lang.System", "gc")

```