Preventing Globals From Getting Journaled (Continued from How do I Minimize My Journals)

Coming back to the topic of how to minimize journals.

(It took me longer than I hoped since my last post on this topic...)

As I concluded in my last post -

To avoid a global from being journaled there are several options:

    • Turning off journaling system wide
    • Mapping globals to CACHETEMP
    • Mapping globals to non-journaled databases
    • Turning off journaling for a process
    • Turning off transactions for object filing

    In this post I will cover these options:

    a. Stop journaling on a system-wide level

    For obvious reasons (detailed in my previous post), this is not recommended and is quite an "aggressive" move to solve a specific problem.

    Programmatically you can do this via calling the %SYS.Journal.System:Stop() method. (or you can use the ^JRNSTOP utility).

    For example:

    Then if I try the same example shown in the previous post of saving a Person object, in SAMPLES:

    This does not get into the Journal file…

    Note I am using the same SSN value that already exists in the database, though I know there is a Unique index defined on that field, so this Save should fail, and a rollback should be performed…

    But journaling is off so we get a ROLLFAIL error:

    (In theory if one would want to take this approach of totally disabling journaling, you'd also need to take care of system startup, since the system always comes up with journaling on, so you need to make sure journaling is to be stopped for every system startup (you'd put this into the SYSTEM label of the %ZSTART routine you'd define. See here for more details)

    b. Map the relevant globals to CACHETEMP

    CACHETEMP is a special database for temporary data, and data in it is not journalled even in transactions.

    Assuming you know what globals are the problematic ones (see a previous post re finding them) you can map the specific ones to CACHETEMP

    A few considerations before you go ahead with this:

    • You'd need to take special care if you'd consider mapping the same global from different namespaces to CACHETEMP, unintended "clashes" can occur and this should be prevented (for example by using a different identifier subscript).
    • Another consideration is security. You might not want some Roles/Users having access to different data coming from various namespaces.
    • The third one is quite obvious but still worth pointing out - CACHETEMP is for temporary data. As such it's contents gets deleted upon restart. In some cases you might have opted not to journal the data as in case of disaster you can recover the data by some other means (rebuilding from some external source, or by some logic) but you might not want to have to rebuild this data every system startup.

    Here's an example how this plays out -  say I map the global ^myMapped from the ENSEMBLE namespace to CACHETEMP database:

    [By the way if you prefix your global name by CacheTemp (e.g. ^CacheTempMyGlobal) it will be automagically mapped to CACHETEMP, without explicitly defining the mapping]

    I then set it (together with another global for comparison - ^notMapped):

    In the journal we just see the ^notMapped global:

    And since this global is mapped to CACHETEMP this is true even in a transaction:

    Again, the ^myMapped global is not journaled:

    Note rolling back such a transaction will not generate an error, but the CACHTEMP-mapped global will not get rolled-back.

    Say I start off with both (mapped and not mapped) globals like this:

    Then in the transaction I change their values, but eventually roll it back:

    So the not-mapped will roll back its value:

    But the mapped will not…

    You can see here more about mapping globals (as well as the link from the Tutorial provided above). Note you can also use wildcards to define a global prefix.

     

    Regarding defining this programmatically you can use the Config.MapGlobals related class, or use the related %Installer manifest tag <GlobalMapping>. For example from the Sample.Installer class in the SAMPLES namespace:

    c. Map the relevant globals to a non-journaled database

    This is similar to the CACHETEMP mapping approach - only one needs to keep in mind, as previously mentioned, that if the global gets set within a transaction the global will still be journaled even if the database it's in is marked as not journaled. Therefore this approach is appropriate only if you are sure the data is being manipulated not within transactions. Note that by default, any object saving, is done within transactions (see below a note about that)

    d. Turn off journaling for the specific process

    Turning off journaling all-together for a specific process (and for a specific time-frame) would prevent journaling for any globals within the process (even in transactions).

    You can do this by calling the DISABLE^%NOJRN routine (at some stage also referred to as %SYS.NOJRN).

    For example I did the following (in two different Terminals to illustrate that while one process has journals disabled, the other still journals):

    In the right-hand side (Process #8852) you can see I set a global before disabling, one during, and again one after, and on the left side (process #11516) I set one global while the journaling was disabled on the right side.

    This is what we can see in the Journal:

    You can see the ^beforeDisable from 8852 (you can ignore the Audit entry as it audits the journal state change), but not the SET while journal was disabled for process 8852, while you still see the SET that was performed in 11516 while it was disabled in 8852, and then finally once it was enabled again you can see the 8852 got the SET journaled again.

    (The reverse by the way, as seen in the example above, is ENABLE^%NOJRN)

    e. Turning off transactions for object filing

    Per above even if a database is marked to be not journaled, within a transaction, and an object Save is by default in a transaction, globals are still journaled. So you can turn off transactions for object saving by calling the %SYSTEM.OBJ.SetTransactionMode() method.

    For example, the same from above, while saving a Person object in SAMPLES which is not journaled, though in %Save(), this will not journal due to turning off the transaction mode:

    Again this would help only for a database that is anyway not journaled.

    • + 4
    • 0
    • 706
    • 9

    Comments

    c. Map the relevant globals to a non-journaled database

    Sometimes it's being done just to conceal some typical app level drawbacks (such as missing the cases when  temporary globals can be used, (excessive) looped rewrites of persistent data, etc) although it may lead to more serious administrative level drawbacks.

    Normally all app's globals (whether they are journaled or not) should by coordinated with each other; if a global should not (or may not), it is a good candidate to be mapped to CACHETEMP DB. Any fail-over scenario you may imagine includes a step of journal restoration. Believe me, it can be a great problem for admin to decide what to do with each non-journaled global after the server fault: which one can be fully (or partially) KILLed, and which one need to be ^REPAIRed. Mostly the right decision is impossible without the developer's intervention. So, the simple dev level solution can introduce much more serious admin level complications.

    IMHO, a global should be considered of one of two types basing on its content type:
    1. "normal" persistent data: such global should be placed in journaled DB without any exceptions;
    2. temporary data which is need only during process run: map it to CACHETEMP or make it private ^||global.

    Sometimes I was told by developers that a third type exists which comprises some pre-generated data that is stored to improve performance of some heavy jobs (e.g. reporting). The pre-generation process can be time (and resource) consuming, so it looks like that the best place for this 3d type globals is a non-journaled DB. After several database reparations I'd answer "No!". Depending on pre-generation process details, each of these globals can (and should) be put in one of two categories taking in account that reboots of modern servers are relatively rare events:
    - if it's not too hard to regenerate the data: just automate it in your code and map the global to CACHETEMP;
    - if not, consider your global as operational one and place it into journaled DB; to reduce excessive journaling during its generation, just choose between approaches "e" or "d" of Tani's article.

    Regarding using non-journaled databases I just wanted to point out that  this approach was taken by us (InterSystems) internally for certain Ensemble globals.

    See this from the 2015.1 Ensemble Release Notes. For each Ensemble database another non-journaled database is created, with a naming convention of <myDatabase>ENSTEMP.

    For example:

    This database will hold specific data (that used to be in CACHETEMP).

    Indeed the release note also mentions the mirroring implication.

    This was done mainly for security considerations (as mentioned in the post above).

    [Internally one can see JGM092 and JGM097 for details]

    A few comments:

    1. Similar to what Alexey said, any time you're using a mix of data that is journaled and non-jounaled but also not temporary (will survive a restart), you have to remain keenly aware of recovery semantics.  After a crash and restart, the journaled data will be at a later point in time than the non-journaled data.  It's only pretty special cases where data is meant to persist across restarts, but doesn't really have to be as up to date as the rest for the integrity of the application.  This needs to be considered in the development cycle.

    2. If using non-journaled databases, be aware of their recovery semantics; it can be a bit non-intuitive. Transactions are journaled there for satisfying rollback at runtime, but that journal information is not used during journal recovery or rollback at startup so transaction there are not atomic or durable (even if in synchronous commit mode) across restarts.  What this does get you is that all data in all the journaled databases are recovered to the same moment in time after a crash, regardless of whether they were in transaction or not.

    3. Mirrored databases ignore the process ^%NOJRN flag discussed in e. (though it is honored for non-mirrored databases on mirror members).   

    See comment to Alexey re <database>ENSTEMP databases.

    Tani, please change all terminal samples and samples of code from pictures to plain formatted text.
     

    I can do this Dmitry, but I am wondering why this is needed.

    If this is in order to be able to copy & paste code, then in this post's case, the code pasted from terminal is either simple SETs, which have no meaning outside the context of the post's basic example, or if they are sample commands, then they appear elsewhere in the post where they can be copied from.

    If there is another reason please let me know.

    Thanks.

    Well, there are some different reasons to do it:

    • optimization network traffic for readers. You should remember, that you wrote the public article for people all around the world, with different quality of their network connection. Some people may even disable images for loading.
    • in slow internet connection, you may see empty spaces for your images, while it will be loaded.
    • it is a bit difficult to recognize your terminal images due to the white background, the same as for page. If you put it as a formatted text, it will have the different background.

    Thanks for clarifying Dmitry.

    In this case, if people can't view the images then just replacing the Terminal images with text, will not be enough as their meaning and relevance is accompanied by the related Journal contents screenshots. And I don't think it makes sense to start replacing all of that with text as well. So for now I will leave this post as is. The essence anyway is written up in the body of the post, and the images are just for illustration and/or emphasis.

    I will though take this into consideration and for future posts prefer text over Terminal images.