Why do I still see old messages after running the purge task?

To manage the accumulation of production data, InterSystems IRIS enables users to manage the database size by periodically purging the data. This purge can apply to messages, logs, business processes, and managed alerts.

Please check the documentation for more details on the settings of the purge task:
https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=EGMG_purge#EGMG_purge_settings

An issue that many users ran into is still finding old messages after running the purge task for messages. For example, a user has a purge task for messages that has NumberOfDaysToKeep=45. This means that messages generated in the last 45 days are retained, and messages generated before 45 days should be deleted. After the purge task completes successfully, most of the messages before the 45-day retention period are deleted, but there are still some old messages that were created before 45 days. Why aren't these messages purged?

In this article, I will discuss the most commonly seen root causes and how to handle the situation. I will assume that the user has a purge task using Ens.Util.Tasks.Purge. 

1) Check the purge task
First, we want to check the purge task's settings. We want to confirm the NumberOfDaysToKeep setting and make sure that the task is set to purge messages in the namespace we are examining, but the setting we want to check the most is KeepIntegrity.

 If KeepIntegrity is enabled, the purge task will only purge complete sessions. By definition, a complete session contains only messages with the status of Complete, Error, Aborted, or Discarded. Notice that if any message in the session is not of one of these four statuses (e.g., the message is Queued or Suspended), all messages in the session will not be purged.

2) Check the message status of all messages in that session
Knowing that KeepIntegrity can make the purge task skip some messages in incomplete sessions, we can now check if this is the current issue by checking the status of the messages in the session. In the Message Viewer, search for messages that should have been deleted based on the NumberOfDaysToKeep criteria by applying the time filter. Check the status of all of the messages in one of these sessions by using the Session ID. Are there any messages with a status other than Complete, Error, Aborted, or Discarded?

In addition to the Message Viewer, you can verify this information using SQL by querying the Ens.MessageHeader table.

3) Manage the incomplete sessions
To resolve the issue, you need to change the status of these messages to they can be purged. For example, some messages might still be in a queue and need to be aborted.

Another way to resolve it is to create a purge task with KeepIntegrity disabled to purge the messages even if the sessions are incomplete. You should choose an appropriate NumberOfDaysToKeep.