Open Exchange App DeleteHelper - A Class to Help with Deleting Referenced Persistent Classes

Primary tabs

Following up on the topic of making sure that referenced instances are deleted when the referrer is deleted –

This topic was discussed in this "Ensemble Orphaned Messages" post, and was also touched in my post with a utility that can help verify all related data was purged.

What is the DeleteHelper

In the "Ensemble Orphaned Messages" one, Suriya suggests to use an %OnDelete() method in your related classes (also as documented), and provides a sample for one.

In this post I want to share a class that could help with this.

You can find it here on GitHub.

This is based on some code that evolved through being used by several sites, and then adapted for this public share. You are of course welcome to fix any issues you find, or add any enhancements.

The class includes a Code Generator %OnDelete method, which generates code at compile-time that should take care of deleting persistent objects referenced by the instance being deleted.

[Note in this case the %OnDelete approach was taken, but in general there are various ways of "telling" a class to delete "other objects" when an instance is deleted – there are some "cascading" mechanisms, for Relationships and for Foreign Keys, as well as using an SQL Trigger (which nowadays also will be called upon object events as well)].

What you'd need to do is add this class as another Super class for your class.

There is also an accompanying class that should help with adding this as a Super class for several classes at once (useful for example for classes that were generated by the SOAP Wizard, which might be quite a few).

Note the use-case in which this class is brought here is Ensemble Message (and Business Process Context) purging, but this is relevant also for "pure" Caché scenarios.

Example

There is more information in the class reference of the 2 classes involved, including usage examples and sample cases. From there:

Here is an example of a class and what it's generated %OnDelete will look like -

A snippet from the Class Definition:

Property Name As %String(MAXLEN = "", XMLNAME = "Name") [ Required ];

Property SSN As %String(MAXLEN = "", XMLNAME = "SSN") [ Required ];

Property DOB As %Date(XMLNAME = "DOB");

Property Home As ITest.Proxy.s0.Address(XMLNAME = "Home");

Property Office As ITest.Proxy.s0.Address(XMLNAME = "Office");

Property Spouse As ITest.Proxy.s0.Person(XMLNAME = "Spouse");

(Assuming the Address and Person classes are Persistent)

And this is the generated %OnDelete (from the generated INT routine):

%OnDelete(oid) public { 
Set status = 1 
Set obj = ..%Open(oid,,.status) 
If ('status) { 
 Set errorCode = $System.Status.GetErrorCodes(status) 
  If errorCode [ 5809 { 
   Quit 1 
  } Else { 
   Quit status 
  } 
} 
If $IsObject(obj.Home) { 
 Set delStatus = obj.Home.%DeleteId(obj.Home.%Id()) 
  If ('delStatus) { 
   Set errorCode = $System.Status.GetErrorCodes(delStatus) 
   If errorCode [ 5810 { 
  } Else { 
   Set status=$select(+status:delStatus,1:$$AppendStatus^%occSystem(status,delStatus)) 
  } 
 } 
} 
If $IsObject(obj.Office) { 
 Set delStatus = obj.Office.%DeleteId(obj.Office.%Id()) 
 If ('delStatus) { 
  Set errorCode = $System.Status.GetErrorCodes(delStatus) 
  If errorCode [ 5810 { 
  } Else { 
   Set status=$select(+status:delStatus,1:$$AppendStatus^%occSystem(status,delStatus)) 
  } 
 } 
} 
If $IsObject(obj.Spouse) { 
 Set delStatus = obj.Spouse.%DeleteId(obj.Spouse.%Id()) 
 If ('delStatus) { 
  Set errorCode = $System.Status.GetErrorCodes(delStatus) 
  If errorCode [ 5810 { 
  } Else { 
   Set status=$select(+status:delStatus,1:$$AppendStatus^%occSystem(status,delStatus)) 
  } 
 } 
} 
Quit status }

(The code uses some macros, primarily for Status handling, that are displayed above, as this is INT code, in their translated manner)

Here is another sample, this time with a list collection -

The class definition snippet:

Property GetListByNameResult As list Of ITest.Proxy.s0.PersonIdentification;

And the generated code snippet:

If $IsObject(obj.GetListByNameResult) { 
 Set key="" 
 Set item = obj.GetListByNameResult.GetNext(.key) 
 While key'="" { 
  If $IsObject(item) { 
   Set delStatus = item.%DeleteId(item.%Id()) 
   If ('delStatus) { 
    Set errorCode = $System.Status.GetErrorCodes(delStatus) 
    If errorCode [ 5810 { 
    } Else { 
     Set status=$select(+status:delStatus,1:$$AppendStatus^%occSystem(status,delStatus)) 
    } 
   } 
  } 
 Set item = obj.GetListByNameResult.GetNext(.key) 
} 

 

Upcoming New 2017.1 Planned Related Feature

Version 2017.1 (available now as a Field Test) includes a new feature (internal reference – MXT2018) that allows you to choose, as part of the XML Schema Wizard, or the SOAP Wizard, that your generated classes will include %OnDelete methods to do exactly what was discussed above:

And this is a sample method that was created in the Wizard-generated class (not via the DeleteHelper):

/// The %OnDelete method is generated in order to cascade deletes of an XML tree.
/// If this class is modified, then the %OnDelete method will need to change to accommodate changed persistent object references.
ClassMethod %OnDelete(deleteOid As %ObjectIdentity) As %Status [ Private, ProcedureBlock = 1, ServerOnly = 1 ]
{
 Set oref=..%Open(deleteOid,,.sc) If $$$ISERR(sc) Quit sc
 Set oid=oref.HomeGetObject()
 If oid'="" Set sc=$$$ADDSC(sc,##class(WSTest.Proxy.s0.Address).%Delete(oid))
 Set oid=oref.OfficeGetObject()
 If oid'="" Set sc=$$$ADDSC(sc,##class(WSTest.Proxy.s0.Address).%Delete(oid))
 Set oid=oref.SpouseGetObject()
 If oid'="" Set sc=$$$ADDSC(sc,##class(WSTest.Proxy.s0.Person).%Delete(oid))
 Quit sc
}

[The method generated by the DeleteHelper is similar, yet slightly different. See sample in Class Reference and above]

So when 2017.1 is released you can definitely use this approach.

A few notes though:

  • See the comment in the generated method: "If this class is modified then the %OnDelete method will need to change to accommodate changed persistent object references", with the DeleteHelper as it is an inherited generator-method and changes should be reflected automatically in the re-generated method. Of course if you run the Wizard (XML Schema or SOAP) then the classes will also be regenerated with "new" appropriate methods.
  • This will add this to the Wizard XML/WSDL-generated data-related classes, but not for example for the Message classes (even if the Request/Response classes were created by the same Wizard, at least as it is now). The DeleteHelper could help with that.
  • And obviously this new feature will not cover manually developed classes, but the DeleteHelper could.

A quick disclaimer regarding the availability of this upcoming feature - as is always true with regards to Field Tests and planned features - changes to the contents of the 2017.1 version, or to it's release itself, could occur (possibly without notice). So eventually you might not find this feature in 2017.1, or it might behave different than mentioned above.

Replies

FYI

The latest version available in the Open Exchange supports now also ZPM (Package Mapper) installation.

Hello Netanel, 

Nice framework you've provided, thanks.

I am wondering if it would be of any help to my case: I have used the Studio SOAP wizard to generate some webservice client classes and amongst those classes:

Class RMH.SOAP.s0.Output Extends (%Persistent, %XML.Adaptor) [ ProcedureBlock, SqlTableName = _Output ] {
...
Property Value As %GlobalCharacterStream(XMLNAME = "Value");

And the caller looks like

Class RMH.SOAP.SoapTreeSoap Extends %SOAP.WebClient [ ProcedureBlock ] {
...
Method Run...(Tree As %String, Inputs As %String, Debug As %Integer) As RMH.SOAP.s0.Output 

We do run a daily purge in all of our productions, but I noticed that some globals aren't being taken into consideration. The global related to the class above is occupying 8GB worth of data and its size isn't reduced after the daily purge take place (it uses the ensemble base class Ens.Util.Tasks.PurgeActivityData)

When I checked the global contents, I could definitely spot very old data... so that reinforces my assumption. Could the delete helper take care of this scenario? Or should I develop custom classes for the purge process?

Any help is appreciated.

Many thanks!

Hi Murillo,

Happy you found interest in this utility.

The situation you described is in fact the classic scenario I had in mind when I built this utility, so it could definitely help.

Here are a few clarifications though –

  1. Ensemble Version

As I mentioned in my original post above, in “relatively” newer versions of Ensemble (since 2017.1), and definitely in InterSystems IRIS, within the SOAP Wizard there is a checkbox that if you check, the auto-generated classes will include a similar %OnDelete method to the one generated by my utility. And therefore the built-in Purge Task will take care of deleting this data (going ahead that is… see more about this in the next comment).

So in these versions, for this specific scenario (SOAP Wizard generated classes, as well the XML Schema Wizard) you don’t have to use my utility. For other cases, where you have custom Persistent classes with inter-relations, my utility would still be helpful.

 

  1. Looking Ahead vs. Looking Back

This %OnDelete method (generated by my utility, or by checking the checkbox in the Wizard mentioned above) takes care of deleting these objects when the Message Body is deleted. But if the Message Body has already been deleted (which seems like your case), this would not help (directly) retroactively.

It would still be recommended to add it, for Purges happening going ahead (and for another consideration mentioned soon), but just by adding this, the old data accumulated will not get auto-magically deleted.

If indeed you have this old data (of objects pointed to by older, previously purged Message Bodies) accumulated then you’ll have to take care of deleting it programmatically.

I won’t get into too many details about how to do this, but here are a few general words of background –

The general structure of Ensemble Messages (in this context) is:

Message Header -> Message Body [ -> Possible other Persistent objects referenced at; and potentially other levels of more object hierarchy]

The built-in Purge Task will delete the Message Header along with the Message Bodies (assuming you checked the “Include Bodies” checkbox in the Purge Task definition, which in 99.99% cases I’ve seen should be checked). But it will not delete other Persistent objects the Message Body was referencing, at least not just “out-of-the-box” (that’s where the %OnDelete method comes into play).

So if your Message Headers and Bodies referring to the “other objects” were already deleted, in order to delete the “not referenced anymore” objects, you’d need to find the last ID of these objects (at the top most level of the object hierarchy) that still has a reference, and delete those under that ID (assuming an accumulative running integer ID for these objects).

I believe the WRC (InterSystems Support – Worldwide Response Center) have a utility built for these kinds of cases (at least scenarios), so I recommend reaching out to the WRC for their assistance, if this is your situation.

Note that if you added my %OnDelete to your classes, then finding just the top-level object in the hierarchy, the one(s) that the Message Body referenced, would be enough to delete, since the %OnDelete will take care of deleting the rest of “the tree” of objects. That’s why I said above that it won’t help the situation of this “lingering” data all by itself, but it could help.

 

  1. Future Proofing

As you can see finding yourself in the situation of having accumulated old data that is not too straightforward to delete, is something you’d very much want to avoid. That is why I also built another utility that helps in validating, during development/testing stages, that there are no such “Purge leaks” in your interfaces.

In addition this tool also helps at providing an estimate as to how much diskspace (for database growth and journaling) your interface will require.

You can check this out here.

 

Hope this helps.

Hi Netanel, thanks for your insightful and detailed response. It  did help a lot :)

Important Note -

Some classes - for example the built-in Business Process class (Ens.BusinessProcessBPL) - have already specific implementation of an %OnDelete method, and hence adding the DeleteHelper as a SuperClass would override the "default" %OnDelete that class might have (depending on the order of Inheritance defined in the class).

So for example a BPL-based Business Process would have this %OnDelete -

ClassMethod %OnDelete(oid As %ObjectIdentity) As %Status
{
    Set tId=$$$oidPrimary(oid)
    &sql(SELECT %Context INTO :tContext FROM Ens.BusinessProcessBPL WHERE %ID = :tId)
    If 'SQLCODE {
        &sql(DELETE from Ens_BP.Context where %ID = :tContext)
        &sql(DELETE from Ens_BP.Thread where %Process = :tId)
    }
    Quit $$$OK
}

Note this takes care of deleting the related Context and Thread data when deleting the Business Process instance. This is the desired behavior. Without this lots of "left-over" data could remain.

Note also that in fact the "AddHelper" class that assists in adding the DeleteHelper as a Super Class to various classes - does not add the DeleteHelper to Business Process classes -

// Will ignore classes that are Business Hosts (i.e. Business Service, Business Process or Business Operation)
// As they do not require this handling
If  $ClassMethod(className,"%IsA","Ens.Host")||$ClassMethod(className,"%IsA","Ens.Rule.Definition")  {
Continue
}


But if you add the DeleteHelper manually to your classes please take caution to see if the class already has an %OnDelete() method and how you'd like to handle that.

A good option is to simply add a call to the original %OnDelete in the newly generated %OnDelete() method calling ##super().

Thanks to @Suriya Narayanan Vadivel Murugan for helping a customer diagnose this situation!