What is a core file? and When are they use­ful?

Core file ba­sics

Caché, Ensemble, HealthShare, and InterSystems IRIS data plat­form are very re­li­able. The vast ma­jor­ity of our cus­tomers never ex­pe­ri­ence any kind of fail­ure. However, un­der rare con­di­tions, processes have failed, and in do­ing so have pro­duced a core file (called a process dump file on Windows and OpenVMS). The core file con­tains a de­tailed copy of the state of the process at the in­stant of its fail­ure, in­clud­ing the processes reg­is­ters, and memory (in­clud­ing or ex­clud­ing shared memory de­pend­ing upon con­fig­u­ra­tion de­tails).

The core file is, in essence, an in­stan­ta­neous pic­ture of a fail­ing process at the mo­ment it at­tempts to do some­thing very wrong. From this pic­ture, we cam ex­trap­olate back­ward in time to find the ini­tial mis­take that led to the fail­ure. As we look back in time, our pic­ture of the process be­comes fuzzier. With more de­tailed cores, we can look far­ther back in time be­fore the pic­ture be­comes too fuzzy.

With prop­erly col­lected core files and as­so­ci­ated in­for­ma­tion, we can of­ten solve, and oth­er­wise ex­tract valu­able in­for­ma­tion about the fail­ing process. With an ar­ti­fi­cially in­duced core file, usu­ally all we can say (of­ten af­ter hours of analy­sis) is “I see what hap­pened to this process, some­one ar­ti­fi­cially forced a core of the process.” An ar­ti­fi­cially in­duced core of a mis­be­hav­ing but the ex­tant process can be use­ful as a sec­ondary source of in­for­ma­tion to fill in de­tails of an analy­sis gath­ered from in­for­ma­tion not avail­able in the core.

InterSystems prod­ucts can be con­fig­ured to record full cores on any process fail­ure. This has no im­pact on performance on your day-to-day op­er­a­tion. All you need is to keep a sig­nif­i­cant amount of disk space free for any po­ten­tial, al­beit un­likely, fail­ure. InterSystems has a good record of solv­ing prob­lems when a full core is avail­able. Sometimes we dis­cover it was an ob­scure hard­ware fail­ure that is never go­ing to oc­cur again.

InterSystems prod­ucts can also be con­fig­ured to record lit­tle or no in­for­ma­tion for process fail­ures. While there is no performance ad­van­tage to dis­abling cores, you might find an op­er­a­tional ad­van­tage. Cores files can con­tain sen­si­tive in­for­ma­tion. If you don’t want to have a pol­icy for se­cur­ing core files, you can en­able core files only af­ter re­peated fail­ures.

Out of the box, InterSystems prod­ucts in­stall with an in­ter­me­di­ate ap­proach. That be­ing lim­ited size cores. With these small cores, InterSystems can nor­mally iden­tify a pre­vi­ously solved prob­lem, and maybe solve sim­ple prob­lems. We can’t solve all prob­lems with the de­fault lim­ited cores.

The pri­mary con­trol for de­ter­min­ing the size and type of core you will get is DumpStyle. This is a pa­ra­me­ter in your cache.cpf or iris.cpf file. There are sev­eral other Operating System spe­cific con­trols.

DumpStyle is ex­plained here: http://docs.intersystems.com/iris20241/csp/docbook/DocBook.UI.Page.cls?KEY=RCPF_Dumpstyle. DumpStyle takes an in­te­ger value from 0 to 8 that ap­plies to every process in a Caché, Ensemble, HealthShare, or InterSystems IRIS data plat­form in­stance, and de­fines what kind of core (or process dump) file is saved should a process en­counter a se­ri­ous er­ror. The de­fined val­ues are:

So, for this con­trol, set it as fol­lows:

There are three ways to change the value of DumpStyle. They are:

[Debug]
dumpstyle=1

SET old=$SYSTEM.Config.ModifyDumpStyle(1)

VIEW $ZUTIL(40,2,165):-2:4:1

VIEW $ZUTIL(40,1,48):-1:4:1

An of­ten asked ques­tion is how large my cores will be? The an­swer is the amount of [dirty] memory used by the process at the time of fail­ure, plus a lit­tle more to de­scribe that memory’s lay­out. Unfortunately, there are no sim­ple formulæ to com­pute that size ac­cu­rately. The best es­ti­mate de­pends upon whether or not you will be in­clud­ing shared memory.

Start with this:

Note: As a prac­ti­cal mat­ter, On most large pro­duc­tion de­ploy­ments, global is large enough that it dwarfs all other fac­tors. To save core files with shared memory in a typ­i­cal large pro­duc­tion de­ploy­ment, size = 1.25 × global is a rea­son­able es­ti­mate.

If you are con­cerned about the amount of disk space needed to store a core file, con­sider:

• If you will be run­ning on cloud ser­vice, and you don’t want to pay to keep a large amount of disk space re­served, you may have to ac­cept lim­ited core files. That is, core files with­out shared memory.
• If you will be run­ning on a server that you con­trol but don’t want to re­serve a large amount of disk space on your ex­pen­sive disk ar­ray, you can pur­chase a cheap USB disk. You don’t need a fast or re­dun­dant disk for core files. You may want to at­tach a hasp sta­ple onto the cheap USB disk with epoxy and pad­lock it to some­thing sub­stan­tial.

Most op­er­at­ing sys­tems have con­trols to redi­rect cores to a com­mon di­rec­tory and con­trol the amount of in­for­ma­tion in cores. These also should be set, and you should con­sider the ram­i­fi­ca­tions for do­ing so, es­pe­cially from a data pri­vacy per­spec­tive. The following sec­tions cover the de­tails for in­di­vid­ual op­er­at­ing sys­tems.

Moving cores to a com­mon di­rec­tory is very use­ful for ca­pac­ity plan­ning, but may also make the cores more ac­ces­si­ble to any­one wish­ing to ex­fil­trate data from your site.

Many types of prob­lems sim­ply can­not be solved with­out in­clud­ing shared memory in the core. Cores that in­clude shared memory tend to be much larger than cores that do not. Most of the difference is the size of your global and rou­tine buffers.

If you are pro­cess­ing sen­si­tive in­for­ma­tion, a core file with­out shared memory will only con­tain the sen­si­tive in­for­ma­tion be­ing processed by the one process that failed. A core file with shared memory will also con­tain all the global vari­ables re­cently ac­cessed by every process. Recently might rep­re­sent min­utes, or con­sid­er­ably longer.

AIX

Full (and mod­ern style) cores should be en­abled with smit:

System Environments
> Change / Show Characteristics of Operating System
> > Enable full CORE dump                               true
> > Use pre-430 style CORE dump                         false

This can also be seen from the com­mand line with:

# lsattr -E -l sys0 | egrep 'fullcore|pre430core'↩
fullcore     true             Enable full CORE dump                 True
pre430core   false            Use pre-430 style CORE dump           True

And set with:

# chdev -l sys0 -a fullcore=true -a pre430core=false -P↩

The -P makes the change per­ma­nent.

◆

By de­fault core files are writ­ten to the de­fault di­rec­tory of the process at the time of process fail­ure. Typically that is the same di­rec­tory as one of your main CACHE.DAT or IRIS.DAT file. This can be changed with smit:

Problem Determination
> Change/Show/Reset Core File Copying Directory

or from the com­mand line with:

# chcore -p on -l /cores -n on -d↩

◆

Insure the file /etc/security/limits, has a sec­tion with the line:

default:
core = -1

Finally, en­sure that by what­ever means you set up en­vi­ron­ment vari­ables for user processes each user has CORE_NOSHM de­fined or not de­fined as de­sired. If CORE_NOSHM=1 is de­fined, core files ex­clude shared memory. If CORE_NOSHM=0 or not de­fined at all, core files in­clude shared memory. The easy way to do this for all users is to edit /etc/environment to in­clude the line:

CORE_NOSHM=1

To as­sign what users have and do not have shared memory cores sup­pressed on an in­di­vid­ual ba­sis, edit one of these files based upon the user and the shell they use:

CORE_NOSHM=1;export CORE_NOSHM    # sh in /etc/profile or $HOME/.profile
export CORE_NOSHM=1               # ksh in /etc/.kshrc or $HOME/.kshrc
export CORE_NOSHM=1               # bash in /etc/bashrc or ~/.bashrc
setenv CORE_NOSHM 1               # csh in ~/.cshrc

Docker

Docker Core file cre­ation for an InterSystems IRIS data plat­form docker con­tainer is con­trolled by the host Linux sys­tem (with a few caveats). You must plan to send core files di­rectly to an op­er­at­ing sys­tem file. That file can be in­side the docker con­tainer, or to a di­rec­tory mapped onto the host Linux sys­tem. The ad­van­tage to send­ing the core file to a di­rec­tory mapped onto the host Linux sys­tem is that it will sur­vive a com­plete fail­ure of the con­tainer.

Since the core file must go to an op­er­at­ing sys­tem file, you must dis­able any ad­vanced core cap­tur­ing soft­ware on the host plat­form. You will want to set /proc/sys/kernel/core_pattern with an ap­pro­pri­ate value for both the host and con­tainer sys­tem. You should choose a rel­a­tively sim­ple di­rec­tory that you know will ex­ist on both the host and con­tainer (/tmp or /cores are the ob­vi­ous best choices). You may also want to in­clude vari­ables to in­sure that cores from mul­ti­ple docker con­tain­ers don’t over­write each other. Thus /cores/core.%p.%e is a good choice.

When you launch the con­tainer you may want to in­clude the op­tion to map the di­rec­tory you will be us­ing for cores to the host op­er­at­ing sys­tem. Thus:

# docker run ⋯ -v /cores:/cores ⋯ ↩

If you don’t in­clude -v /cores:/cores any core files cre­ated by a process fail­ure in­side the docker con­tainer, will sur­vive only as long as the docker con­tainer is run­ning. If the map­ping given by the -v op­tion is not sym­met­ri­cal, that is the value to the left and right of the colon are dif­fer­ent, you may fail to cap­ture some cores.

Set the corefile size ulimit. Since this is a runtime de­ci­sion, add the following to the docker run com­mand:

# docker run ⋯ --ulimit core=-1 ⋯ ↩

A full com­mand to start a docker con­tainer might look like:

# docker run --init --detach \↩ 
        --volume /cores:/cores \↩
        --ulimit core=-1 \↩
        --volume /home/salzer/volume_host:/volume_con­tainer \↩
        --publish 52773:52773 \↩
        --publish 52443:52443 \↩
        --publish 51773:51773 \↩
        --publish 1972:1972 \↩
        --env ISC_DATA_DIRECTORY=/volume_con­tainer/config_2024 \↩
        --env ICM_SENTINEL_DIR=/volume_con­tainer \↩
        --name iris2024 \↩
        intersystems/iris:2024.1.0.267.2 \↩
        --key /volume_con­tainer/iris_con­tainer2024.key↩

HP–UX

Enable plac­ing cores in a com­mon di­rec­tory with ex­tended nam­ing with:

# coreadm -e global -g /cores/core.%p.%f↩

%p places the pid in the path­name, %f places the name of the ex­e­cutable (such as cache or iris) in the path­name. See:

% man 1m coreadm↩

for more op­tions.

◆

Review if shared memory has been en­abled in core files with:

# /usr/sbin/kctune core_addshmem_read↩
# /usr/sbin/kctune core_addshmem_write↩

Change with:

# /usr/sbin/kctune core_addshmem_read=1↩
# /usr/sbin/kctune core_addshmem_write=1↩

1 means en­able, 0 means dis­able. HP–UX di­vides shared memory into two types. In gen­eral InterSystems only uses write shared memory, but we rec­om­mend set­ting both types the same.

◆

On HP–UX the core size is lim­ited by the maxdsiz_64bit ker­nel pa­ra­me­ter. Make sure that it is set high enough that a full core can be gen­er­ated.

Review with:

# /usr/sbin/kctune maxdsiz_64bit↩

Set with:

# /usr/sbin/kctune maxdsiz_64bit=4294967296↩

A user can fur­ther limit their core with a ulimit -c com­mand. This com­mand should be re­moved from /etc/profile, $HOME/.profile, and sim­i­lar files for other shells un­less it is your in­ten­tion to limit core files.

RedHat Linux

If you are run­ning Rhel 6.0 or later (also CentOS), RedHat has added their Automatic Bug Reporting Tool (ABRT). As in­stalled this is not com­pat­i­ble with Caché, Ensemble, HealthShare, or InterSystems IRIS data plat­form. You need to de­cide if you wish to con­fig­ure ABRT to sup­port Caché, Ensemble, HealthShare, InterSystems IRIS data plat­form, or dis­able ABRT.

◆

ABRT To make InterSystems prod­ucts com­pat­i­ble with ABRT, de­ter­mine the ver­sion of ABRT you are run­ning:

# abrt-cli --ver­sion↩

Edit the ABRT con­fig­u­ra­tion file. The name varies de­pend­ing upon the ver­sion of ABRT:

ABRT 1.x: /etc/abrt/abrt.conf
ABRT 2.x: /etc/abrt/abrt-action-save-package-data.conf

If you in­stalled Caché, Ensemble, or HealthShare with a cinstall com­mand (most com­mon), or InterSystems IRIS data plat­form with an irisinstall com­mand, find the ProcessUnpackaged= line, and change the value to yes.

ProcessUnpackaged = yes

Otherwise, if you in­stalled Caché, Ensemble, HealthShare, or InterSystems IRIS data plat­form from an RPM mod­ule, find the OpenGPGCheck= line, and change the value to no.

OpenGPGCheck = no

Regardless of how you in­stalled Caché, Ensemble, HealthShare, or InterSystems IRIS data plat­form, find the BlackListedPaths= line, and add a ref­er­ence to cstat or irisstat in the in­stal­la­tion/bin di­rec­tory. If the BlackListedPaths= line does not ex­ist, add it at the end with just the cstat or irisstat ref­er­ence.

BlackListedPaths=[retain_ex­ist­ing_list,]in­stal­la­tion_directory/bin/cstat

Save your ed­its, and restart abrtd:

# ser­vice abrtd restart↩

Configured as such, ABRT cre­ates a new di­rec­tory (un­der /var/spool/abrt or /var/tmp/abrt) for each process fail­ure, and in that di­rec­tory, place the core, and as­so­ci­ated in­for­ma­tion.

When a process fail­ure oc­curs, is­sue the com­mand:

# abrt-cli --list↩        # for ABRT 1.x
# abrt-cli list↩          # for ABRT 2.x

This will show a list of re­cent process fail­ures, and for each will give a di­rec­tory spec­i­fi­ca­tion. In each di­rec­tory will be a coredump file, along with many other small files that col­lec­tively can be quite use­ful in de­ter­min­ing the cause of the process fail­ure.

From some other di­rec­tory, en­ter the com­mand:

% tar -cvzf wrcnumber-core.tar.gz /var/spool/abrt/directory/*↩

Where wrcnumber is the num­ber InterSystems as­signs to in­ves­ti­gate your case. You can send us the com­pressed wrcnumber-core.tar.gz file.

◆

AB/RT Alternatively, you can dis­able ABRT with:

# ser­vice abrtd stop↩
# ser­vice abrt-ccpp stop↩       # ABRT 2.x only.

To per­ma­nently dis­able ABRT:

# chkconfig abrtd off↩
# chkconfig abrt-ccpp off↩     # ABRT 2.x only.

Finally you need to up­date /proc/sys/kernel/core_pattern, see the next sec­tion.

◆

AB/RT You can con­trol where cores are de­posited (un­less you are us­ing ABRT).

① If you are us­ing ABRT, you must skip this step.
② If you have dis­abled ABRT, you must per­form this step.
③ If you never had ABRT, this step is op­tional.

Edit the file /proc/sys/kernel/core_pattern

In the sim­ple case, just use:

core

It is gen­er­ally use­ful to add the pid, and name of the pro­gram gen­er­at­ing the core with:

core.%p.%e

You might also place the cores in a com­mon directory with:

/cores/core.%p.%e

Verify that all users have write ac­cess to di­rec­tory cho­sen. See man core for more op­tions. You should make this change per­ma­nent by cre­at­ing a file in the di­rec­tory /etc/sysctl.d with a name end­ing with .conf, and con­tain­ing:

kernel.core_pattern=/cores/core.%p.%e

◆

ABRT  AB/RT You should set the /proc/self/coredump_filter to con­trol the amount of memory dumped to the core. This can be in an ap­pro­pri­ate /etc/profile.d/some­thing.sh file. The com­mand is:

# echo 0x33 >/proc/self/coredump_filter↩

The exact bitmap used de­pends upon the level of data you wish to col­lect. The mean­ings of the bits can be found in man core, sam­ples that make sense for InterSystems prod­ucts are:

As an al­ter­na­tive to plac­ing this in a shell spe­cific script, you can mod­ify this dur­ing boot. These in­struc­tions only ap­ply if you boot with grub2. You can test this with:

# grub2-install --ver­sion↩
grub2-install (GRUB) 2.02~beta2

Edit /etc/default/grub. Change the line that be­gins GRUB_CMDLINE_LINUX_DEFAULT=. If the line doesn’t al­ready ex­ist in the file, just add it at the end. It should con­tain:

GRUB_CMDLINE_LINUX_DEFAULT="oldcmd coredump_filter=newval"

Note: The oldcmd is the old value of GRUB_CMDLINE_LINUX_DEFAULT (omit, if the line didn’t pre­vi­ously ex­ist). newval is the new value for coredump\_filter in hexa­dec­i­mal with a lead­ing “0x”.

Run:

# grub2-mkconfig -o /boot/grub2/grub.cfg↩

◆

ABRT  AB/RT You should set your ulimit -c for all processes to un­lim­ited. This can be set glob­ally in the file /etc/security/limits.conf. Add these two lines:

*       soft    core        unlimited
*       hard    core        unlimited

SuSE Linux

If your are run­ning SuSE Linux Enterprise Server 12 or later, SuSE now stores all cores in the systemd jour­nal. Core files stored in the systemd jour­nal are tran­si­tory. They do not sur­vive a sys­tem re­boot. Cores, if needed, must be ex­tracted from the systemd jour­nal be­fore any sys­tem re­boot.

To list the core files cur­rently in the systemd jour­nal:

# [systemd-]coredumpctl list

To ex­tract a core se­lected by the pid that cre­ated the core:

# [systemd-]coredumpctl -o core.morename dump
pid↩

Note: the systemd- pre­fix was re­moved from the com­mand name ef­fec­tive with SuSE 12-SP2.) It is rec­om­mended that you leave this systemd be­hav­iour in place, and not at­tempt to de­feat it.

If you are run­ning an older ver­sion of SuSE Linux Enterprise (11 or ear­lier), you can con­trol where cores are de­posited by edit­ing the file /proc/sys/kernel/core_pattern

In the sim­ple case, just use:

core

It is gen­er­ally use­ful to add the pid, and name of the pro­gram gen­er­at­ing the core with:

core.%p.%e

You might also place the cores in a com­mon di­rec­tory with:

/cores/core.%p.%e

Verify that all users have write ac­cess to di­rec­tory cho­sen. See man core for more op­tions.

You can make this change per­ma­nent by ap­pend­ing these lines to the file /etc/sysctl.conf:

# Make this core pat­tern per­ma­nent (SuSE 12 breaks this, don’t use):
kernel.core_pattern=/cores/core.%p.%e

◆

You should set the /proc/self/coredump\_filter to con­trol the amount of memory dumped to the core. This can be in an ap­pro­pri­ate /etc/profile.d/some­thing.sh file. The com­mand is:

# echo 0x33 >/proc/self/coredump_filter↩

The exact bitmap used de­pends upon the level of data you wish to col­lect. The mean­ings of the bits can be found in man core, sam­ples that make sense for InterSystems prod­ucts are:

As an al­ter­na­tive to plac­ing this in a shell spe­cific script, you can mod­ify this dur­ing boot. To do this use yast2. The user in­ter­face for yast2 will vary de­pend­ing upon whether you are con­nected with a ter­mi­nal in­ter­face (it will use a curses in­ter­face), or a GUI in­ter­face. These in­struc­tions try to be in­ter­face ag&sny;nos­tic.

◆

You should set your ulimit -c for all processes to un­lim­ited. This can be set glob­ally in the file /etc/security/limits.conf. Add these two lines:

*       soft    core        unlimited
*       hard    core        unlimited

◆

Note: It may benecessary to dis­able AppArmor, which blocks ap­pli­ca­tion be­hav­iour which it con­sid­ers un­usual, and writ­ing to a core file may be con­sid­ered un­usual.

# rcapparmor stop↩

Ubuntu Linux

Ubuntu LINUX Ubuntu uses apport to trap all process fail­ures, and for pack­ages added with its in­stal­la­tion pack­age, cre­ate apport re­ports which con­tain en­coded and com­pressed cores with ad­di­tional in­for­ma­tion. It is pos­si­ble ask apport to process unpackaged code, that is ap­pli­ca­tions not in­stalled with Ubuntu’s pack­age man­ager. Unfortunately, in do­ing so, Canonical treats the apport re­ports cre­ated for unpackaged code as some­thing it can ex­am­ine for the im­prove­ment of Ubuntu.

Since it is pos­si­ble to ex­tract your data from an apport re­port, you al­most cer­tainly do not want to en­able apport pro­cess­ing of unpackaged code. Your only choice is to dis­able apport.

Use these commands:

# systemctl stop apport.ser­vice↩ 
# systemctl dis­able apport.ser­vice↩
# systemctl mask apport.ser­vice↩

◆

Create a file /etc/sysctl.d/30-core-pattern.conf (or any sim­i­lar name in that di­rec­tory). In that file place:

kernel.core_pattern=/cores/core.%p.%e

Insure that the di­rec­tory you spec­ify for sav­ing cores is pub­licly writable, and has suf­fi­cient disk space. See man core for more op­tions.

◆

You should set the /proc/self/coredump\_filter to con­trol the amount of memory dumped to the core. This can be in an ap­pro­pri­ate /etc/profile.d/some­thing.sh file. The com­mand is:

# echo 0x33 >/proc/self/coredump_filter↩

The exact bitmap used de­pends upon the level of data you wish to col­lect. The mean­ings of the bits can be found in man core, sam­ples that make sense for Caché are:

As an al­ter­na­tive to plac­ing this in a shell spe­cific script, you can mod­ify this dur­ing boot. These in­struc­tions only ap­ply if you boot with grub2. You can test this with:

# grub-install --ver­sion↩
grub-install (GRUB) 2.02-2ubuntu8.12

Edit /etc/default/grub. Change the line that be­gins GRUB\_CMDLINE\_LINUX\_DEFAULT=. If the line doesn’t al­ready ex­ist in the file, just add it at the end. It should con­tain:

GRUB_CMDLINE_LINUX_DEFAULT="oldcmd coredump_filter=newval"

Note: The oldcmd is the old value of GRUB\_CMDLINE\_LINUX\_DEFAULT (omit, if the line didn’t pre­vi­ously ex­ist). newval is the new value for coredump\_filter in hexa­dec­i­mal with a lead­ing “0x”.

Run:

# grub-mkconfig -o /boot/grub2/grub.cfg↩

◆

You should set your ulimit -c for all processes to un­lim­ited. This can be set glob­ally in the file /etc/security/limits.conf. Add these two lines:

*                soft    core            unlimited
*                hard    core            unlimited

macOS (OS X, Darwin)

Mac OS X was re­named OS X, and it was later re­named macOS. All these op­er­at­ing sys­tems are Apple’s pro­pri­etary user in­ter­face lay­ered upon Darwin, an op­er­at­ing sys­tem that Apple de­rived from BSD Unix and the­o­ret­i­cally, re­leased to the pub­lic do­main. Nevertheless, Apple re­leases Darwin in such a way, that as a prac­ti­cal mat­ter no one will ever run just Darwin.

InterSystems prod­ucts only re­quire Darwin, but since Darwin isn’t prac­ti­cally avail­able, all in­struc­tions are based upon the full Apple Mac OS X, OS X, or macOS.

macOS in­cludes CrashReporter. A tool that au­to­mat­i­cally in­ter­cepts process fail­ures, pack­ages the fail­ure de­tails as text logs, and sends the data to Apple for Analysis. CrashReporter will cap­ture process fail­ure de­tails for third-party soft­ware, such as Caché, Ensemble, HealthShare, and InterSystems IRIS data plat­form. Which, in the­ory, Apple might for­ward to InterSystems.

InterSystems does not re­ceive CrashReporter logs form Apple, nor have we de­vel­oped the abil­ity to an­a­lyze them. InterSystems works strictly form core files. Fortunately, CrashReporter works in­de­pen­dently from core file cre­ation. That is, it is pos­si­ble to to process a process fail­ure through nei­ther, ei­ther, or both CrashReporter, and core file cre­ation.

CrashReporter preferences can be set in System Preferences → Security & Privacy, Privacy tab. The panel name and se­lec­tion of boxes varies from ver­sion to ver­sion. In Mac OS X 10.4, the panel was called just Security, and there were no rel­e­vant check boxes. In those older ver­sion the user was al­ways pre­sented with a di­a­log box on any process fail­ure, and asked if they wanted to send the data to Apple for analy­sis.

Depending upon the sen­si­tiv­ity of the data you processes, you may want to untick all the op­tions re­lated to CrashReprter.

◆

The method for en­abling cores in macOS has un­der­gone sig­nif­i­cant changes from ver­sion to ver­sion. See the following chart, and use the ap­pro­pri­ate method for your ver­sion.

Method 1: For ver­sions OS X 10.3 (Cheetah), and prior un­sup­ported ver­sions: Edit the file /hostconfig. Find the line COREDUMPS=, and change the value to -YES-.

◆

Method 2: For ver­sions OS X 10.4 (Tiger) to OS X 10.9 (Mavericks), edit the file /etc/launchd.conf, and add the line:

limit core unlimited

And re­boot.

◆

Method 3: For ver­sions OS X 10.10 (Yosemite) and newer, /etc/launchd.conf is elim­i­nated. Core file gen­er­a­tion is now half dis­abled. Either users must en­able cores for each process with:

% ulimit -c unlimited↩

Prior to run­ning their ap­pli­ca­tion, a priv­i­leged user must run:

# launchctl limit core unlimited↩

Then lo­gout, and lo­gin again prior to start­ing Caché. Apple specif­i­cally does not pro­vide a good way to au­to­mate this, as they con­sider the de­fault gen­er­a­tion of a core file to be a po­ten­tial se­cu­rity vul­ner­a­bil­ity.

Apple does pro­vide a way to to­tally dis­abling core file gen­er­a­tion. This is done by edit­ing the file /etc/sysctl.conf, and adding the line:

kern.coredump=0

It can be re-en­abled, by re­mov­ing the line, or chang­ing the value to 1.

OpenVMS

OpenVMS By de­fault Caché, and Ensemble will only pro­duce CACCVIO-pid.LOG files for fail­ing processes. With these only rel­a­tively sim­ple prob­lems can be solved. These CACCVIO-pid.LOG files will al­ways be placed in the processes de­fault di­rec­tory (typ­i­cally the di­rec­tory of a CACHE.DAT file), and can only be redi­rected by chang­ing the processes de­fault di­rec­tory.

Caché, and Ensemble may also pro­duce CERRSAVE-pid.LOG files. These are sim­i­lar to CACCVIO-pid.LOG files. Usually, you do not need to con­cern your­self with the difference. In some cases Caché, and Ensemble will pro­duce both files in re­sponse to a fail­ure. In all cases seen so far, the CACCVIO-pid.LOG file is pro­duced first with the full con­text of the er­ror, while the CERRSAVE-pid.LOG file is pro­duced dur­ing fi­nal run­down of the process, and con­tains com­par­a­tively lit­tle in­for­ma­tion of value.

If ex­tended process dumps (FULL dumps) are en­abled, they too will be placed in the process de­fault di­rec­tory. However they can be redi­rected, by defin­ing the log­i­cal name SYS$PROCDMP to point to a di­rec­tory in which to store the process dump. This log­i­cal name can be de­fined at the /SYSTEM level. The file name will be CACHE.DMP or CSESSION.DMP.

OpenVMS also pro­vides the log­i­cal name SYS$PROTECTED_PROCDMP. You should also de­fine that log­i­cal name with both /EXECUTIVE_MODE and /SYSTEM. This ap­plies to process fail­ures of priv­i­leged im­ages, and parts of Caché are priv­i­leged. The OpenVMS doc­u­men­ta­tion will ad­vise you to de­fine the two log­i­cal names to dif­fer­ent di­rec­to­ries, and place higher se­cu­rity on di­rec­tory cor­re­spond­ing to SYS$PROTECTED_PROCDMP. This is based upon the as­sump­tion that that the data processed by priv­i­leged im­ages is more sen­si­tive than that processed by non-priv­i­leged im­ages. If both are sen­si­tive, it is ok to point both log­i­cal names to the same di­rec­tory.

◆

There is a his­tory of de­fects ef­fect­ing the cre­ation of CACCVIO-pid.LOG and CERRSAVE-pid.LOG files as well as full process dumps. These are the most im­por­tant changes.

Solaris

Solaris You can en­able plac­ing cores in a com­mon di­rec­tory with ex­tended nam­ing with:

# coreadm -e global -g /cores/core.%p.%f -G all↩

all in­cludes all types of memory, default in­cludes all but the last two. If you want sig­nif­i­cantly smaller cores (to save space at the ex­pense of mak­ing fewer prob­lems solv­able), the most space is saved by re­mov­ing dism shared memory. Do this with:

# coareadm -e Global -g /cores/core.%p.%f -G (default-dism)↩

See:

% man 1m coreadm↩

for more op­tions.

◆

By de­fault users have

% ulimit -c unlimited↩

You may use the ulimit (or limit com­mand in csh) to dis­able cores, but coreadm is gen­er­ally more flex­i­ble. So you should in­sure ulimit commands don’t ap­pear in /etc/profile or $HOME/.profile, or cor­re­spond­ing files for other shells.

Windows

The in­for­ma­tion to be in­cluded in a dumpfile for Windows is fully con­trolled by the DumpStyle pa­ra­me­ter in the cache.cpf file (or other in­ter­face to chang­ing DumpStyle de­fined above.

Testing

Local se­cu­rity setup among other prob­lems can pre­vent a core from ac­tu­ally be­ing writ­ten. It can be very use­ful to test if a core will ac­tu­ally be cre­ated un­der real-world con­di­tions. To do that, en­ter the com­mand:

USER>DO $ZUTIL(150,"DebugException")↩

To be cer­tain, you should test this state­ment in­ter­ac­tively, in­side JOBs (as­sum­ing your ap­pli­ca­tion uses the JOB com­mand), and even hid­ing in­side an op­tion of your ap­pli­ca­tion that your users will not ac­ci­den­tally se­lect. Verify that you get a core file, and fol­low the san­ity check in the next sec­tion to ver­ify that it is a good core file.

Sanity Test

Core files (and process dumps) can be quite large, and they can con­tain sen­si­tive in­for­ma­tion. Before trans­mit­ting a core file to InterSystems for analy­sis, it is best to per­form a san­ity test of core file on the sys­tem that gen­er­ated it, or a very sim­i­lar sys­tem. Based upon your op­er­at­ing sys­tem, please per­form the following san­ity test:

# dbx cache core↩
(dbx) set $stack_de­tails↩
(dbx) where↩
(dbx) quit↩

# gdb cache core↩ 
(gdb) frame 0↩ 
(gdb) while 1↩
 > info frame↩
 > up↩ 
 > end↩ 
(gdb) quit↩

# adb cache core↩
adb> $c↩
adb> $q↩

# gdb cache core↩
(gdb) frame 0↩
(gdb) while 1↩
 > info frame↩
 > up↩
 > end↩
(gdb) quit↩

# lldb↩
(lldb) target create -c core↩
(lldb) thread backtrace all↩
(lldb) quit↩

# gdb cache core↩
(gdb) frame 0↩
(gdb) while 1↩
 > info frame↩
 > up↩
 > end↩
(gdb) quit↩

$ ANALYZE/CRASH dumpfile.DMP↩
SDA> SHOW CALL_FRAME/ALL↩
If you are still run­ning OpenVMS v7.x (or ear­lier),
the pre­vi­ous com­mand will not work, in­stead use:
SDA> SHOW CALL_FRAME↩
SDA> SHOW CALL_FRAME/NEXT↩
Repeat the prior com­mand un­til you get an er­ror.
SDA> QUIT↩

$ ANALYZE/PROCESS dumpfile.DMP↩
DBG> SHOW CALL/IMAGE↩
DBG> QUIT↩

# mdb cache core↩
> ::stackregs↩
> ::quit↩

# dbx cache core.DMP↩
(dbx) where↩
(dbx) quit↩

At­tach the de­tails of the san­ity test to your WRC case, or e-mail to: support@intersystems.com).

Transmission

Be pre­pared to send us the full core along with sup­port files that may be needed for your par­tic­u­lar op­er­at­ing sys­tem. We need to know the exact ver­sion of Caché, Ensemble, HealthShare, or InterSystems IRIS data plat­form gen­er­ated the core file. If you have re­linked the soft­ware to in­clude cus­tom $ZF() func­tions, please send the ex­e­cutable. (Actually, it is more con­ve­nient, if you al­ways send the ex­e­cutable.)

On most Unix sys­tems, it is also best to send the li­braries, used by the ex­e­cutable. The like­li­hood we will need li­braries for any given plat­form varies. Consult this ta­ble:

Explanation of Support lev­els

Issue an ldd com­mand to list the needed li­braries:

# ldd install_directory/bin/im­age↩
        linux-vdso.so.1 => (0x00007fffd1320000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f23e5002000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f23e4dfa000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f23e4af0000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f23e47ee000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f23e45d8000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f23e4216000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f23e521a000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f23e3ffa000)

The above con­tains sam­ple out­put for Rhel 7. The out­put of all Unix sys­tems are sim­i­lar. install_directory refers to the di­rec­tory in which Caché, Ensemble, HealthShare, or InterSystems IRIS data plat­form is in­stalled. im­age is cache for all prod­ucts, prior to 2018, and irisdb for prod­ucts since 2019.

If you are send­ing mul­ti­ple files, it is best to place them in a com­pressed con­tainer file. In gen­eral .ZIP is best. .tar.gz is also rea­son­able. For OpenVMS cre­at­ing a backup file with

$ BACKUP *.* [-]saveset.BCK/SAVE/DATA=COMPRESS↩

It can be help­ful to in­clude a man­i­fest that ex­plains the files be­ing sent. Please pre­pare the man­i­fest as a plain text file.

If you will be send­ing the data elec­tron­i­cally, please do not en­crypt the file, use an en­crypted trans­mis­sion method in­stead.

You can send core files to InterSystems by any of these meth­ods:

It is im­por­tant to re­mem­ber that some of the files we want are bi­nary files, while oth­ers are text. For some file trans­fer meth­ods (es­pe­cially be­tween un­like op­er­at­ing sys­tems), it is im­por­tant to spec­ify if the file is bi­nary or text to pre­vent the file from be­ing cor­rupted.

Index