$zf call out to Window DIR command not working when running concurrent process

Question

Question

David Sterngast · Apr 4, 2021

I have an application that is called by terminal in local cache environment. The application basically has two processes that runs and does the below call to retrieve files and sub-directories in separate folders. When the main process starts, it jobs off and calls the second process that gathers all files in a directory by using a window dir and file command that pipes the results to a file. The main process continues on and does the same thing but from a different folder. When checking out what was retrieved by the secondprocess, there were directories and files missing. When I run this as just one process it works properly. I am running this on a 2017 license cache instance. I have also tried this on IRIS instance and got the same results. Could you please advise me what is causing the window directory to not work properly when running the processes at the same time?

Main process one

S FILE=SDIR_"DIRLIST.TXT"

I $$FSIZECK(FILE) S FSIZE=1 Q ""

S X=$ZF(-1,"DEL """_FILE_"""")

S X=$ZF(-1,"DIR """_SDIR_""" >> """_FILE_"""")

Second process one

S FILE=SDIR_"DIRLIST.TXT"

I $$FSIZECK(FILE) S FSIZE=1 Q ""

S X=$ZF(-1,"DEL """_FILE_"""")

S X=$ZF(-1,"DIR """_SDIR_""" >> """_FILE_"""")

Product version: IRIS 2020.4

$ZV: cache 2017 and IRIS

Discussion (8)2

Log in or sign up to continue

Vitaliy Serdtsev · Apr 12, 2021

Important:
Jobbed Process Permissions are Platform-dependent
Running Programs or System Commands with $ZF(-100)

So that we speak the same language, I made a simple example using Using the Work Queue Manager

Class dc.test [ Abstract ]
{

ClassMethod MyJob(SDIR As %String)
{
  s FILE=##class(%File).NormalizeFilename("DIRLIST.TXT",SDIR)

  q:##class(%File).GetFileSize(FILE)>0

  d ##class(%File).Delete(FILE)

  s X=$ZF(-1,$$$FormatText("DIR %1 >> %2",$$$quote(##class(%File).NormalizeDirectory(SDIR)),$$$quote(FILE)))
}

/// d ##class(dc.test).test()
ClassMethod test()
{
  s N=4
  s queue=$system.WorkMgr.Initialize(,.sc,N)
  f i=1:1:N d queue.Queue("##class(dc.test).MyJob","C:\Temp\test "_i)
  d queue.WaitForComplete()
}

}

I copied different files to the following directories:

C:\Temp\test 1
C:\Temp\test 2
C:\Temp\test 3
C:\Temp\test 4

After calling d ##class(dc.test).test() , everything worked out as expected: DIRLIST.TXT were created in the corresponding directories each with its own content.

0 0

score 1 · Answer 1 · 2021-04-04T12:19:00-04:00

Let's analyze this: [leaving aside that $Zf(-1 .. is Deprecated)

You run the same sequence in 2 Caché/IRIS processes in parallel.
There is no synchronization between them
Both run in the same namespace ==> therefore the same default directory where SDIR_"DIRLIST.TXT" is located ???? " .. but from a different folder... " is SDIR different ????? if not :
Both processes spawn 2 times a sub-process for each $ZF(-1)
let's name the sub processes fg1, fg2 and bg1,bg2
The sub_process from foreground is not synchronized to the sub-process of background and your construct has no control on the sequence they are running.
If the run sequence fg, fg1,fg2,bg,bg1,bg2 there should be no problem. But that's rater unlikely
it looks like fg,bg,fg1,fg2,bg1 (partially deleting results of fg2), bg2

To verify what's really happening run fg,fg1,fg1,bg (JOB ), bg1,bg2
So they both can't influence each other.

score 3 · Answer 2 · 2021-04-04T13:30:06-04:00

Instead of the double $ZF() I'd suggest using the query in Library class %File.

 
set rs=##class(%ResultSet).%New("%File:FileSet")
do rs.Execute(SDIR,"*",1)
while rs.Next() { write !,rs.GetData(1) }

score 0 · Answer 3 · 2021-04-06T15:02:39-04:00

Thanks for your response.

There is a main process that jobs off another process in parallel.

There is no communication/ synchronization between the processes.

Both run in the same namespace and the same main folder.

The process are different because there sub-folders are different SDIR_"DIRLIST.TXT" is located.

SDIR is different

The main processes spawn $zf(-1) and does $ZF(-1)

When running in parallel.

The first process completes without any issue.

The child process the dir.txt file created from the dir is missing files and sub-folders.

score 0 · Answer 4 · 2021-04-07T03:10:37-04:00

Robert Cemper · Apr 7, 2021

Sorry! That doesn't indicate any reason. $zf(-1) is just a spawn.

0 0

score 0 · Answer 5 · 2021-04-08T13:47:15-04:00

To answer your question that they can't influence each other, that is correct. I can run each process independently from each other they work. I can also run it as one process and it works properly. When I run it in parallel it fails. Might be due to that it is a tied terminal process, the privileges are different when spawning a job or window process when I use $ZF. The zf was used to call window command functions.

score 0 · Answer 6 · 2024-08-16T11:52:23-04:00

We have a situation that looks suspiciously similar:

job that runs an external program via $ZF(-100,...) to perform a task from a business process runs perfectly when the Pool Size = 1
not all of the external tasks complete successfully when Pool Size > 1

More detail:

Production takes incoming stream of HL7 ORU_R01 messages, and for each one produces a PDF
this is done by converting each HL7 to an XML representation, then calling Apache FOP (the one pre-installed in Ensemble) with a stylesheet and the XML to build the PDF. A business process takes care of this step.
with Pool Size = 1 runs correctly
with Pool Size = 2 all the XML files are generated (via a call to a class method) but only a small subset of the PDF files are generated - maybe 4 out of 20?
no error messages that we've been able to find yet

Here's an illustrative screenshot - yellow are first HL7->XML->PDF, green are second HL7->XML->PDF. Yellow produces a PDF, green doesn't. As far as we can tell the FOP commands should be independent (no shared files - unless stylesheets can't be opened by multiple processes simultaneously?)

Only thing we've seen in documentation that gives us pause is the line: "On a Windows system you should never omit both the /ASYNC and /STDIN flags." (from $ZF(-100) | InterSystems IRIS Data Platform 2024.2) - but when only one copy is running it appears to be fine with "" as the flags argument.

Is this a $ZF/Ensemble issue, or is it something about FOP specifically?

score 0 · Answer 7 · 2024-08-16T15:31:24-04:00

A simpler solution than using SET rs=##CLASS(%ResultSet).%New("%File:FileSet"), is the $ZSEARCH() function:

    SET f=$ZSEARCH(SDIR)
    WHILE f'="" {
        // Do something with f
        SET f=$ZSEARCH(f)
    }

There is no need to create a child process or use classes.