New post
Question
· Sep 1, 2020

Importing globals limitations and errors

#Data Import and Export #Caché

Hi,

I exported selected globals from a Cache 2017 database into a single 4 Gb gof file. Now I tried to import from this file via Management Portal on a different machine . Only about half of the globals was imported and my attempts to select additional globals led to nothing, no new globals have been imported. Well, obviously I am mildly curious what's going on and how can I see the corresponding error which did not appear in the Import window but I can also shrug it off and consider what should I do next.

I am now considering exporting and importing globals programmatically in a loop, say, via $System.Obj Export and Load functions. This way I can retrieve and hopefully bypass a specific global error. I am not putting my hopes on GBLOCKCOPY  due to its limitations (https://cedocs.intersystems.com/latest/csp/docbook/Doc.View.cls?KEY=GSTU_remote#GSTU_remote_gblockcopy) Any other worthwhile approach?  

An additional but related question whether any of several existing export and import functions accept wildcards in the list of corresponding globals like $System.Obj.Load("L*"), all globals that start with L. Not that I saw in system routines code but maybe I missed it.

Thanks in advance,
Anna

Discussion (23)2
Log in or sign up to continue
Robert Cemper · Sep 1, 2020

4GB is a significant amount.
A different approach could be to have just an additional DB  (not journaled)
where instead of a loop you use MERGE to copy the global
MERGE ^|"newdb"| myGLOBAL = ^myGLOBAL
once your collection is done you dismount the deb and just copy it  wherever you need it. 
"newdb" is the extended Global reference. Either just the DB or a helping namespace.

So you can check exactly what data you move.

Of course, this works also across ECP if it is available.

2 0
Vic Sun · Sep 1, 2020

I didn't see a wildcard for %Library.Global but $SYSTEM.OBJ.Export does appear to support wildcards.

https://cedocs.intersystems.com/latest/csp/documatic/%25CSP.Documatic.cls?APP=1&CLASSNAME=%25SYSTEM.OBJ

That being said, I would be interested in why the global export/import is failing in the first place.  One pitfall is "Block format ignores mapped globals and mapped global subscripts", but other than that we'd probably need more details on your process and what globals are succeeding or failing. This is probably appropriate to contact the WRC with.

1 0
Alexey Maslov · Sep 2, 2020

I am not putting my hopes on GBLOCKCOPY  due to its limitations

This precaution sounds strange in your context: if you are simply copying a set of globals that are actively being modified, you will get an unpredictable result nevertheless the utility you choose. To make a consistent copy, one should apply journal records or (if copying to another Cache instance) use some cross-system technique, such as Mirroring or Shadowing, while both require Multi-Server cache.key.

If you tell us more about the task you are trying to solve, we'd advise you better.

1 0
Anna Golitsyna · Sep 2, 2020

Apparently GBLOCKCOPY is simply not available on my Cache 2017 version since search for it in system files returns nothing. I am trying to transfer a large number of globals, not all, from one computer to another, not just between namespaces within the same database. The system does not have actively modified globals. I got an old-timer advice to use %GOF which does accept ranges both for exclusion and inclusion so will work on that for a while.

0 0
Robert Cemper · Sep 2, 2020

That's interesting: 
How did you look for GBLOCKCOPY.int  ?  there never was a  .mac!
And it is even on the news IRIS.

Instance: IRIS
USER>zn "%SYS"
%SYS>zl GBLOCKCOPY
%SYS>p
GBLOCKCOPY ; Fast global copy from database to namespace
         ; Copyright (c) 2020 by InterSystems Corporation.
         ; Cambridge, Massachusetts, U.S.A.  All rights reserved.
         ; Confidential property of InterSystems Corporation.
         n
         i $p($g(^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")),"~",1)'="1.0" k ^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")
         i '$d(^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")) s ^|"^^"_$zu(12)|%SYS("GBLOCKCOPY")="1.0~"
         s POP=0
         w !,"This routine will do a fast global copy from a database to another database or"
         w !,"to a namespace. If a namespace is the destination, the global will follow any"
         w !,"mappings set up for the namespace."
         ;
MENU     ;

If this is missing you may also miss other important pieces:
Check your installation or reinstall it

2 0
Anna Golitsyna · Sep 2, 2020

Everybody, thanks for your help and advice but the old-timer's advice to go the d ^%GOF route in Cache Terminal proved to be it. It was actually amazing to see how much superior this route turned out to be comparing with the much more modern Export/Import Management Portal route. Specifically:

  1. Preparation to exclude certain globals, few dozens of them: Terminal - about 10 minutes because you can use ranges (see below and here, https://docs.intersystems.com/priordocexcerpts/prgglobalchui-41.pdf), portal - much longer because you have to uncheck each global individually
  2. Export speed: Terminal - 10 minutes, portal 3 hours (no kidding)
  3. Export/Import log per global: Terminal - log provided, complete with the number of blocks per global, portal - no log
  4. Import speed: Terminal - less than 10 minutes, portal - stuck in the middle with no error shown

Sample %GOF usage in terminal, hit Enter if no special requests, exporting:
d ^%GOF
Device: tmp.gof
           file format: ("UNW*") =>
          Maximum media size (bytes): (No maximum)
Enter a short description of the contents of this tape or file
Description:
All Globals? No => No
Global ^S-SZ         [All globals in this range, case-sensitive - AG]
Global ^'STPL*     [Exclude all globals that start with STPL - AG]

Real life output from my real life usage:
...............
^ZPG                                           182 data blocks written
^ZTEMP                                           1 data block written
491,337 blocks written in 10 minutes,  51 seconds
--------------------
Importing was done via d ^%GIF, via terminal as well. At some point the %GIF utility asked me:

Globals with a preceding asterisk ('*') already exist in this directory
What would you like to do with these globals that exist?
 
1. Skip inputting globals that exist in this directory
2. Merge input globals with the existing globals
3. Specify skip/merge for individual globals
 
Your choice: 1=>

Thanks again,
Anna

0 0
Vic Sun · Sep 2, 2020

Hey Anna,

Glad you were able to get this resolved. I would be curious, as Nigel mentioned, if you had been importing/exporting in the background from the management portal, as that level of speed seems surprisingly slow. Also, I assume there have been improvements to the newer utilities in even more recent versions of Caché and IRIS, so perhaps you would see better performance if you were able to upgrade.

1 0
Nigel Salm · Sep 3, 2020

We tend to forget that %SYS contains a whole load of utilities that have been around for years. Some of the utilities were wrapped into classes that could be invoked through $system (i.e. the %SYSTEM Package).

Some of the utilities are still invoked by the Management Portal. 

I answered another issue that came up in the Dev Community about how you can determine which classes and routines in %SYS are InterSystems code and those that may have been written by a developer many years ago when it was commonplace for some applications to write %Routines or %Classes especially in the MUMPS and very early Cache days where things like Package Mapping didn't exist.

The global utilities ^%G, ^%GIF and %GOF I still use periodically bcause they have such useful masking features as pointed out in other messages in this thread. Likewise ^%RO, ^%RI, %RD, ^ROUTINE, ^%RDELETE can be useful paerticularly if you want to export the .obj compiled routine code rather than the .int version. 

The commands ZLOAD, ZPRINT, ZINSERT, ZREMOVE, ZSAVE are useful for writing small little bits of routine code and once i a while I will use zload and zprint to see what code is in the routine buffer especially if I am testing some code and I get an "<UNDEF> zMyClass.int.1 zMyMethod+10" error status. Normally you open the class in studio and then select View -> Other Code and then fine the label and offset which is fine but sometines it is just quicker to execute the commands ZLOAD MyRourine.int.1  ZPRINT zMyMethod+10

Utilities like ^LOCKTAB can be useful if you want to see whats in the Lock Table and maybe release a lock. I have used this when the Management Portal fails to load properly bcause there is a rougue cache process that is consuming all available resources

^RESJOB is useful for killing off unwanted processes e.g. Rogue Processes  

The $system.Security api's can be very useful if you want to manipulate Cache Users programatically. For example your application may maintain its own User/Role classes and UI which will be referenced by the UI of the application but you also want the application Roles and Users to be created or updated as Cache Roles and Users. 

Oe of the issues with the management portal is that it will comply with the CSP session timeout limit and the example of attempting to import a large global export file for two reasons

1) It only displays N number of global names in the GOF file so even though you have clicked the "select all" check box it seems to only select those globals that have been displayed and there is no way of increasing the number of rows that it displays

2) If the load takes too long the csp session will expire and the page becomes unresponsive and you will find that only the globals that were 'checked' have been loaded whereas if you select 'run in background' and then check the 'select all' globals when it creates the background task it passes in 'true' for the parameter 'select all globals' so gets around the issue of selecting only those that are visible in the select globals form and secondly the background task has no implicit timeout associated with it so it will run for as long as it needs to. In reality it is ultimately invoking an entry point in ^%GIF in order to do the import.

I seem to remember that the Cache and Ensemble documentation used to have a section on "legacy" utilities which are all of the ones I have listed here along with ^DATABASE, ^JOURNAL and so on but I don't think that has carried over in the IRIS documentation

Nigel

2 0