Importing globals limitations and errors

Hi,

I exported selected globals from a Cache 2017 database into a single 4 Gb gof file. Now I tried to import from this file via Management Portal on a different machine . Only about half of the globals was imported and my attempts to select additional globals led to nothing, no new globals have been imported. Well, obviously I am mildly curious what's going on and how can I see the corresponding error which did not appear in the Import window but I can also shrug it off and consider what should I do next.

I am now considering exporting and importing globals programmatically in a loop, say, via $System.Obj Export and Load functions. This way I can retrieve and hopefully bypass a specific global error. I am not putting my hopes on GBLOCKCOPY  due to its limitations (https://cedocs.intersystems.com/latest/csp/docbook/Doc.View.cls?KEY=GSTU_remote#GSTU_remote_gblockcopy) Any other worthwhile approach?  

An additional but related question whether any of several existing export and import functions accept wildcards in the list of corresponding globals like $System.Obj.Load("L*"), all globals that start with L. Not that I saw in system routines code but maybe I missed it.

Thanks in advance,
Anna

  • 00
  • 0
  • 138
  • 23

Replies

4GB is a significant amount.
A different approach could be to have just an additional DB  (not journaled)
where instead of a loop you use MERGE to copy the global
MERGE ^|"newdb"| myGLOBAL = ^myGLOBAL
once your collection is done you dismount the deb and just copy it  wherever you need it. 
"newdb" is the extended Global reference. Either just the DB or a helping namespace.

So you can check exactly what data you move.

Of course, this works also across ECP if it is available.

It's not a single global it's about 1500 separate globals so you approach would involve a loop as well but yes, it is different.

So ECP might be even more interesting as you can copy directly to the target DB.
Instead of copy local, copy DB or file, load global.

At that moment what is the file system of your installation? 
The 4GB limit did ring a bell in my head. 

Windows 10, i.e. NTFS, if that is the question. Not Fat32. What is ECP in this context?

I'm not aware of a file size limit with NTFS.
ECP is the Enterprise Caché Protocol that can connect Caché / Ensemble / IRIS instances over the network.
To use it you require a multiserver license

I didn't see a wildcard for %Library.Global but $SYSTEM.OBJ.Export does appear to support wildcards.

https://cedocs.intersystems.com/latest/csp/documatic/%25CSP.Documatic.cls?APP=1&CLASSNAME=%25SYSTEM.OBJ

That being said, I would be interested in why the global export/import is failing in the first place.  One pitfall is "Block format ignores mapped globals and mapped global subscripts", but other than that we'd probably need more details on your process and what globals are succeeding or failing. This is probably appropriate to contact the WRC with.

What is WRC? It looks like the import process goes alphabetically, then stops and refuses to process any more globals. I would assume that the GOF file contains all globals alphabetically as well. I will look if the first not processed global is somehow different from previous ones.

Half of the globals were imported alright. Yes, I thought too it might be a failure to properly export, like an export related error. The 4GB file does contain all intended globals, I can see it in the Management Portal's Import utility. I tried to check just a couple on the list and they failed with no error.

I am not putting my hopes on GBLOCKCOPY  due to its limitations

This precaution sounds strange in your context: if you are simply copying a set of globals that are actively being modified, you will get an unpredictable result nevertheless the utility you choose. To make a consistent copy, one should apply journal records or (if copying to another Cache instance) use some cross-system technique, such as Mirroring or Shadowing, while both require Multi-Server cache.key.

If you tell us more about the task you are trying to solve, we'd advise you better.

Apparently GBLOCKCOPY is simply not available on my Cache 2017 version since search for it in system files returns nothing. I am trying to transfer a large number of globals, not all, from one computer to another, not just between namespaces within the same database. The system does not have actively modified globals. I got an old-timer advice to use %GOF which does accept ranges both for exclusion and inclusion so will work on that for a while.

That's interesting: 
How did you look for GBLOCKCOPY.int  ?  there never was a  .mac!
And it is even on the news IRIS.

Instance: IRIS
USER>zn "%SYS"
%SYS>zl GBLOCKCOPY
%SYS>p
GBLOCKCOPY ; Fast global copy from database to namespace
         ; Copyright (c) 2020 by InterSystems Corporation.
         ; Cambridge, Massachusetts, U.S.A.  All rights reserved.
         ; Confidential property of InterSystems Corporation.
         n
         i $p($g(^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")),"~",1)'="1.0" k ^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")
         i '$d(^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")) s ^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")="1.0~"
         s POP=0
         w !,"This routine will do a fast global copy from a database to another database or"
         w !,"to a namespace. If a namespace is the destination, the global will follow any"
         w !,"mappings set up for the namespace."
         ;
MENU     ;

If this is missing you may also miss other important pieces:
Check your installation or reinstall it

I searched for it in Studio in all files but yes, I see it your way. I'll let know here how %GOF does the job, tests seem reasonable.

OK. there is a bunch of system utilities that are protected from studio.  
Typically the real powerful and sometimes dangerous old stuff.

No, no, no. I already edited my previous reply, it's there. Still going the %GOF way for now unless it does not work

HI

I have seen this behaviour before. Basically it only loads the globals that are visible in the 'select globals' form. The trick is to check the flag 'Run in the background' and then select all of the globals and it works fine

Nigel

Everybody, thanks for your help and advice but the old-timer's advice to go the d ^%GOF route in Cache Terminal proved to be it. It was actually amazing to see how much superior this route turned out to be comparing with the much more modern Export/Import Management Portal route. Specifically:

  1. Preparation to exclude certain globals, few dozens of them: Terminal - about 10 minutes because you can use ranges (see below and here, https://docs.intersystems.com/priordocexcerpts/prgglobalchui-41.pdf), portal - much longer because you have to uncheck each global individually
  2. Export speed: Terminal - 10 minutes, portal 3 hours (no kidding)
  3. Export/Import log per global: Terminal - log provided, complete with the number of blocks per global, portal - no log
  4. Import speed: Terminal - less than 10 minutes, portal - stuck in the middle with no error shown

Sample %GOF usage in terminal, hit Enter if no special requests, exporting:
d ^%GOF
Device: tmp.gof
           file format: ("UNW*") =>
          Maximum media size (bytes): (No maximum)
Enter a short description of the contents of this tape or file
Description:
All Globals? No => No
Global ^S-SZ         [All globals in this range, case-sensitive - AG]
Global ^'STPL*     [Exclude all globals that start with STPL - AG]

Real life output from my real life usage:
...............
^ZPG                                           182 data blocks written
^ZTEMP                                           1 data block written
491,337 blocks written in 10 minutes,  51 seconds
--------------------
Importing was done via d ^%GIF, via terminal as well. At some point the %GIF utility asked me:

Globals with a preceding asterisk ('*') already exist in this directory
What would you like to do with these globals that exist?
 
1. Skip inputting globals that exist in this directory
2. Merge input globals with the existing globals
3. Specify skip/merge for individual globals
 
Your choice: 1=>

Thanks again,
Anna

Hey Anna,

Glad you were able to get this resolved. I would be curious, as Nigel mentioned, if you had been importing/exporting in the background from the management portal, as that level of speed seems surprisingly slow. Also, I assume there have been improvements to the newer utilities in even more recent versions of Caché and IRIS, so perhaps you would see better performance if you were able to upgrade.

I think this case study proves that newer utilities are not always better so I would not bet on the latest IRIS version. I did see the Background run advice but would it cut three hours to 10 minutes? I doubt it. And if it would I'd say it's a portal implementation deficiency.

I can't speak definitively as I haven't tested but if you were able to show the management portal underperforming on the latest version, you could report it to InterSystems.

We tend to forget that %SYS contains a whole load of utilities that have been around for years. Some of the utilities were wrapped into classes that could be invoked through $system (i.e. the %SYSTEM Package).

Some of the utilities are still invoked by the Management Portal. 

I answered another issue that came up in the Dev Community about how you can determine which classes and routines in %SYS are InterSystems code and those that may have been written by a developer many years ago when it was commonplace for some applications to write %Routines or %Classes especially in the MUMPS and very early Cache days where things like Package Mapping didn't exist.

The global utilities ^%G, ^%GIF and %GOF I still use periodically bcause they have such useful masking features as pointed out in other messages in this thread. Likewise ^%RO, ^%RI, %RD, ^ROUTINE, ^%RDELETE can be useful paerticularly if you want to export the .obj compiled routine code rather than the .int version. 

The commands ZLOAD, ZPRINT, ZINSERT, ZREMOVE, ZSAVE are useful for writing small little bits of routine code and once i a while I will use zload and zprint to see what code is in the routine buffer especially if I am testing some code and I get an "<UNDEF> zMyClass.int.1 zMyMethod+10" error status. Normally you open the class in studio and then select View -> Other Code and then fine the label and offset which is fine but sometines it is just quicker to execute the commands ZLOAD MyRourine.int.1  ZPRINT zMyMethod+10

Utilities like ^LOCKTAB can be useful if you want to see whats in the Lock Table and maybe release a lock. I have used this when the Management Portal fails to load properly bcause there is a rougue cache process that is consuming all available resources

^RESJOB is useful for killing off unwanted processes e.g. Rogue Processes  

The $system.Security api's can be very useful if you want to manipulate Cache Users programatically. For example your application may maintain its own User/Role classes and UI which will be referenced by the UI of the application but you also want the application Roles and Users to be created or updated as Cache Roles and Users. 

Oe of the issues with the management portal is that it will comply with the CSP session timeout limit and the example of attempting to import a large global export file for two reasons

1) It only displays N number of global names in the GOF file so even though you have clicked the "select all" check box it seems to only select those globals that have been displayed and there is no way of increasing the number of rows that it displays

2) If the load takes too long the csp session will expire and the page becomes unresponsive and you will find that only the globals that were 'checked' have been loaded whereas if you select 'run in background' and then check the 'select all' globals when it creates the background task it passes in 'true' for the parameter 'select all globals' so gets around the issue of selecting only those that are visible in the select globals form and secondly the background task has no implicit timeout associated with it so it will run for as long as it needs to. In reality it is ultimately invoking an entry point in ^%GIF in order to do the import.

I seem to remember that the Cache and Ensemble documentation used to have a section on "legacy" utilities which are all of the ones I have listed here along with ^DATABASE, ^JOURNAL and so on but I don't think that has carried over in the IRIS documentation

Nigel