Interprocess comunication (main< = >job)

Hello.

I would like to know if there's a way to batch a certain amount of writes done inside a job and display it sequentially on the main process.

My idea is to prevent the main process from freezing while displaying the batched data. The main process's device could be

the terminal or the Studio output.

 

%Studio.Debugger does something close to that when printing the output from the debug target process.

$System.Event.Signal($zparent) doesn't signals the parent if a device is open. So I can't create the batch buffer along, because I can keep the buffering device open and notify the main process.

 

I tried using  an IJC device, but I couldn't get the buffered data back from the writer device using the reader device.

Also what does ChangePrincipal ($zu(132)) do?

Vote up!
Vote down!

Rating: 1

Comments: 7 Views: 207

4 answers

Answer

I tried making a simple example, no Event or IJC for now, just to understand how devices works. I understand the concept, but there are many types of devices and types of configuration that always lead me to a dead end.

I'll escalate the example (using Events and IJC) as I keep understanding the concepts.

So, what am I doing wrong here? read x:3 always times out, regardless of my doings.
 

Note: Just to make myself clear, I'm planning on using it along with this project. Well, use it with big projects and you'll understand.

Maybe I could even create an utility that makes it easier to print a job's outputs as long as it's possible.

 

Class Log.Test [ Abstract ]
{

ClassMethod Write(
device As %String,
start As %Integer = 1) As %Boolean
{
  open device:(:33568::"\n"):3
  quit:'$test
  
  use device
  
  for i=start:1:1000000000 {
    write "This is a test #"_i_$c(13,10)
    if i#1000 = 0 {
      write "\n"
    }
  }
}

ClassMethod Start()
{
   set device = "|TCP|1"
   
   job ..Write(device)::3
   
   while 1 {
     open "|TCP|2":(:33568):3
     quit:'$test
     
     use "|TCP|2"
     read x:3
     
     close "|TCP|2"
     
     if '= "" {
       use 0
       write x
     }
   }
}

}
 

Vote up!
Vote down!

Rating: 0

The problem is both OPEN statements are for TCP/IP servers. One OPEN has to be a client. You distinguish between the two by giving a DNS or IP address in the OPEN. If everything is running locally, use "127.0.0.1", So

     OPEN dev:("127.0.0.1":33568):3 

Also on one OPEN us are using "\n" for the terminators argument. That makes reads break on backslash and lowercase n. Is that really what you want? This isn't "C". Generally you should specify a mode for TCP/IP I/O. For text based messaging over TCP/IP "PTSE" ≣ "M" works best. Add an "A", that is "MA", for a multi-server. For a binary channel, using mode "S" is still a good idea. Thus:

     OPEN dev:(:33568:"M"):3 ; Open the server.

     OPEN dev:("127.0.0.1":33568:"M"):3 ; Open the client.

Vote up!
Vote down!

Rating: 1

Wow, it worked! But I don't want it to send the data back to the other process for each iteration, so I tried buffering it using a variable. Now, If you run this code you will notice the following error:

 [CRLF] class '%Studio.General', method 'Execute': <WRITE> 41 zExecute+26^%Studio.General.1

The routine might according to the principal device, of course.

There're two points I noticed here:

1 - The caller process must wait for the job for it's first write. Or a <READ> error will happen.
2 - The caller process must know when stop iterating if the job is finished.
3 - Sometime the line break works, sometimes it doesn't.

That's why I tried introducing the Event API, however it doesn't seems to be working, since the Signal is being ignored by that WaitMsg. Is there anything I'm missing here?

Class Log.Test [ Abstract ]
{

ClassMethod Write(
device As %String,
start As %Integer = 1) As %Boolean
{
  set startEventTriggered = 0
  open device:("127.0.0.1":33568:"M"):3
  quit:'$test
  
  use device
  
  set buf = ""
  set msg = "This is a test #"
  
  for i=start:1:10460 {
  set composedMessage = msg_i_$c(13, 10)_" [CRLF] "
  set expectedSize = $length(buf) + $length(composedMessage)
  if expectedSize > $$$MaxStringLength {
  //set offset = expectedSize - $$$MaxStringLength
  //set subbuff = $e
  if 'startEventTriggered {
  set startEventTriggered = 1
  use 0
  do $System.Event.Signal($zparent, "1")
  use device
  }
   write buf
   set buf = ""
  else {
   set buf = buf_composedMessage
  
  }
  
  if buf '= "" write buf
  write $c(0)
  
  close device
}

ClassMethod Start()
{
   set server = "|TCP|1"
   set client = "|TCP|2"    
   
   job ..Write(server)::3
   
   do $System.Event.WaitMsg("", 3)
   
   open client:(:33568:"M"):3 // Moved open outside the loop
   quit:'$test 
   
   while 1 {
     use "|TCP|2"
     read x:3
     
     if '= "" {
       use 0
       write x
     }     
     quit:x=$c(0)
   }   
   close client
}

}
 

Vote up!
Vote down!

Rating: 0

Answer

Depending upon your version of Caché you may also have the %SYSTEM.WorkMgr and %SYSTEM.WorkMgrIPQ classes available to use.  In WorkMgr child jobs on their own can write back to the parent processes main device so it isn't quite IPC in the sense that you're likely used to.  However, it does make multithreading a job a bit easier than spawning children and then monitoring them.  I've not used the WorkMgrIPQ class myself (at least, not yet) but the class documentation is closer to IPC in that structured data can be sent back to the parent/host process.

Vote up!
Vote down!

Rating: 1

This indeed made the communication much easier!
Now I only have to create a strategy for message batching.

It seems that I can't simply use WaitForComplete since it puts the caller process into sleep.

My next step is do an experiment with the IPQ variation.

Anyway, here's my experiment using the WorkMgr.

 

Class Log.Test2 [ Abstract ]
{

ClassMethod Log()
{
  set buf = ""
  set msg = "This is a test #"
  
  for i=1:1:1046000 {
    set composedMessage = msg_i_$c(13, 10)
    set expectedSize = $length(buf) + $length(composedMessage)
    if expectedSize > $$$MaxStringLength {
      write buf
      set buf = ""
    else {
      set buf = buf_composedMessage
    
  }
  
  if buf '= "" write buf
  quit $$$OK
}

ClassMethod Start()
{
  write "This is from "_$job, !!
  set queue = $System.WorkMgr.Initialize("d", .sc, .sc)
  set sc = queue.Queue("##class(Log.Test2).Log")
  if $$$ISERR(sc) do $System.OBJ.DisplayError(sc) quit sc
  set sc = queue.WaitForComplete()
  if $$$ISERR(sc) do $System.OBJ.DisplayError(scquit sc
  quit $$$OK
}

}

Vote up!
Vote down!

Rating: 0

Answer

Fast and easy way:

1. Set temporary global:

Set ^temp($zparent, $job) = current status

2. From a parent, iterate over ^temp and display current status.

 

Other ideas.

1. Try to switch devices before calling $System.Event.Signal($zparent)

2. Can you provide a simple code sample where $System.Event.Signal($zparent) doesn't signals the parent if a device is open?

Vote up!
Vote down!

Rating: 2

1. Set temporary global:

I though about using that, but I'm targeting the fastest batched feedback as possible.

About $System.Event, I had the code, but switched to IJC, so I'll have to re-implement it. No problem though, I'll redo that asap.

Vote up!
Vote down!

Rating: 0

Answer

IJC devices work! They come in pairs, and all may not be defined on your system. Start at the beginning and try writing to 225 and reading from 224. Each process must OPEN their devices before reading or writing. You get to write a certain amount to 225 before the device blocks. The reader can read with a zero second time-out, if you don't want the reader waiting. 

In the system management portal goto
System Administration → Configuration → Additional Settings → Advanced Memory. ijcbuff controls the amount of memory per IJC device. The bigger the more you can write to an IJC device with an lagging reader, before the device blocks. ijcnum is the number of defined IJC device pairs.

Calling $ZUTIL(132) with no extra arguments makes the current device the principal device. It was documented in the Caché ObjectScript Language Reference up until Caché 2009.1. It has been replaced with ##CLASS(%Device).ChangePrincipal(), which does the same thing. This isn't typically very useful.

Often more useful is the former $ZUTIL(82,12,bool), now ##CLASS(%Device).ReDirectIO(bool) which lets you redirect I/O through routines that can filter, redirect, record, though routines. Unfortunately, while the workings of $ZUTIL(82,12,bool) did eventually make it into the documentation, the workings were removed from the ##CLASS(%Device).ReDirectIO(bool) documentation. The details are the same, look at the old documentation on-line.

Vote up!
Vote down!

Rating: 2

Hmm, about I'll have to do something more elaborated then. I had done a quick draft before.

Calling $ZUTIL(132) with no extra arguments makes the current device the principal device.

And how does this affects the parent process? (if it does affect).

Often more useful is the former $ZUTIL(82,12,bool), now ##CLASS(%Device).ReDirectIO(bool) which lets you redirect I/O through routines that can filter, redirect, record, though routines.

I remember reading about that here and here

Vote up!
Vote down!

Rating: 0

$ZUTIL(132) or ##CLASS(%Device).ChangePrincipal() have no effect on the parent process. Actually, what do you mean by parent process? If a process is created with a JOB command, while the job ID of of the created process does appear in $ZCHILD, the two processes don't have any lasting parent/child relationship.

Vote up!
Vote down!

Rating: 0

Oh, sorry for my misleading words.

I said "parent process" or "main process" due to the $zparent, $zchild and $job. But I didn't mean something close to process forks, since as you said, they haven't a parent-child relationship, but more liking siblings as they can run independently.

Which situtation would be interesting to change the principal device? This method's description is quite 'dry' to understand what is it's usage.

I noticed that devices are my weak spot, maybe my questions are looking dumb for some people here. :P

Thanks for your help so far.

Vote up!
Vote down!

Rating: 0

Log in to answer