New post
Question
· May 12, 2017

Interprocess comunication (main< = >job)

#Object Data Model #Studio #Caché

Hello.

I would like to know if there's a way to batch a certain amount of writes done inside a job and display it sequentially on the main process.

My idea is to prevent the main process from freezing while displaying the batched data. The main process's device could be

the terminal or the Studio output.

%Studio.Debugger does something close to that when printing the output from the debug target process.

$System.Event.Signal($zparent) doesn't signals the parent if a device is open. So I can't create the batch buffer along, because I can keep the buffering device open and notify the main process.

I tried using  an IJC device, but I couldn't get the buffered data back from the writer device using the reader device.

Also what does ChangePrincipal ($zu(132)) do?

Discussion (11)0
Log in or sign up to continue
Rubens Silva · May 12, 2017

Hmm, about I'll have to do something more elaborated then. I had done a quick draft before.

Calling $ZUTIL(132) with no extra arguments makes the current device the principal device.

And how does this affects the parent process? (if it does affect).

Often more useful is the former $ZUTIL(82,12,bool), now ##CLASS(%Device).ReDirectIO(bool) which lets you redirect I/O through routines that can filter, redirect, record, though routines.

I remember reading about that here and here

0 0
Stuart Salzer · May 17, 2017

The problem is both OPEN statements are for TCP/IP servers. One OPEN has to be a client. You distinguish between the two by giving a DNS or IP address in the OPEN. If everything is running locally, use "127.0.0.1", So

     OPEN dev:("127.0.0.1":33568):3 

Also on one OPEN us are using "\n" for the terminators argument. That makes reads break on backslash and lowercase n. Is that really what you want? This isn't "C". Generally you should specify a mode for TCP/IP I/O. For text based messaging over TCP/IP "PTSE" ≣ "M" works best. Add an "A", that is "MA", for a multi-server. For a binary channel, using mode "S" is still a good idea. Thus:

     OPEN dev:(:33568:"M"):3 ; Open the server.

     OPEN dev:("127.0.0.1":33568:"M"):3 ; Open the client.

1 0
Rubens Silva · May 23, 2017

This indeed made the communication much easier!
Now I only have to create a strategy for message batching.

It seems that I can't simply use WaitForComplete since it puts the caller process into sleep.

My next step is do an experiment with the IPQ variation.

Anyway, here's my experiment using the WorkMgr.

 

Class Log.Test2 [ Abstract ]
{

ClassMethod Log()
{
  set buf = ""
  set msg = "This is a test #"
  
  for i=1:1:1046000 {
    set composedMessage = msg_i_$c(13, 10)
    set expectedSize = $length(buf) + $length(composedMessage)
    if expectedSize > $$$MaxStringLength {
      write buf
      set buf = ""
    } else {
      set buf = buf_composedMessage
    } 
  }
  
  if buf '= "" write buf
  quit $$$OK
}

ClassMethod Start()
{
  write "This is from "_$job, !!
  set queue = $System.WorkMgr.Initialize("d", .sc, .sc)
  set sc = queue.Queue("##class(Log.Test2).Log")
  if $$$ISERR(sc) do $System.OBJ.DisplayError(sc) quit sc
  set sc = queue.WaitForComplete()
  if $$$ISERR(sc) do $System.OBJ.DisplayError(sc) quit sc
  quit $$$OK
}

}

0 0
Stuart Salzer · May 12, 2017

IJC devices work! They come in pairs, and all may not be defined on your system. Start at the beginning and try writing to 225 and reading from 224. Each process must OPEN their devices before reading or writing. You get to write a certain amount to 225 before the device blocks. The reader can read with a zero second time-out, if you don't want the reader waiting. 

In the system management portal goto
System Administration → Configuration → Additional Settings → Advanced Memory. ijcbuff controls the amount of memory per IJC device. The bigger the more you can write to an IJC device with an lagging reader, before the device blocks. ijcnum is the number of defined IJC device pairs.

Calling $ZUTIL(132) with no extra arguments makes the current device the principal device. It was documented in the Caché ObjectScript Language Reference up until Caché 2009.1. It has been replaced with ##CLASS(%Device).ChangePrincipal(), which does the same thing. This isn't typically very useful.

Often more useful is the former $ZUTIL(82,12,bool), now ##CLASS(%Device).ReDirectIO(bool) which lets you redirect I/O through routines that can filter, redirect, record, though routines. Unfortunately, while the workings of $ZUTIL(82,12,bool) did eventually make it into the documentation, the workings were removed from the ##CLASS(%Device).ReDirectIO(bool) documentation. The details are the same, look at the old documentation on-line.

2 0
Rubens Silva · May 17, 2017

Oh, sorry for my misleading words.

I said "parent process" or "main process" due to the $zparent, $zchild and $job. But I didn't mean something close to process forks, since as you said, they haven't a parent-child relationship, but more liking siblings as they can run independently.

Which situtation would be interesting to change the principal device? This method's description is quite 'dry' to understand what is it's usage.

I noticed that devices are my weak spot, maybe my questions are looking dumb for some people here. :P

Thanks for your help so far.

0 0
Rubens Silva · May 19, 2017

Wow, it worked! But I don't want it to send the data back to the other process for each iteration, so I tried buffering it using a variable. Now, If you run this code you will notice the following error:

 [CRLF] class '%Studio.General', method 'Execute': <WRITE> 41 zExecute+26^%Studio.General.1

The routine might according to the principal device, of course.

There're two points I noticed here:

1 - The caller process must wait for the job for it's first write. Or a <READ> error will happen.
2 - The caller process must know when stop iterating if the job is finished.
3 - Sometime the line break works, sometimes it doesn't.

That's why I tried introducing the Event API, however it doesn't seems to be working, since the Signal is being ignored by that WaitMsg. Is there anything I'm missing here?

Class Log.Test [ Abstract ]
{

ClassMethod Write(
device As %String,
start As %Integer = 1) As %Boolean
{
  set startEventTriggered = 0
  open device:("127.0.0.1":33568:"M"):3
  quit:'$test
  
  use device
  
  set buf = ""
  set msg = "This is a test #"
  
  for i=start:1:10460 {
  set composedMessage = msg_i_$c(13, 10)_" [CRLF] "
  set expectedSize = $length(buf) + $length(composedMessage)
  if expectedSize > $$$MaxStringLength {
  //set offset = expectedSize - $$$MaxStringLength
  //set subbuff = $e
  if 'startEventTriggered {
  set startEventTriggered = 1
  use 0
  do $System.Event.Signal($zparent, "1")
  use device
  }
   write buf
   set buf = ""
  } else {
   set buf = buf_composedMessage
  } 
  }
  
  if buf '= "" write buf
  write $c(0)
  
  close device
}

ClassMethod Start()
{
   set server = "|TCP|1"
   set client = "|TCP|2"    
   
   job ..Write(server)::3
   
   do $System.Event.WaitMsg("", 3)
   
   open client:(:33568:"M"):3 // Moved open outside the loop
   quit:'$test 
   
   while 1 {
     use "|TCP|2"
     read x:3
     
     if x '= "" {
       use 0
       write x
     }     
     quit:x=$c(0)
   }   
   close client
}

}
 

0 0
Rubens Silva · May 17, 2017

I tried making a simple example, no Event or IJC for now, just to understand how devices works. I understand the concept, but there are many types of devices and types of configuration that always lead me to a dead end.

I'll escalate the example (using Events and IJC) as I keep understanding the concepts.

So, what am I doing wrong here? read x:3 always times out, regardless of my doings.
 

Note: Just to make myself clear, I'm planning on using it along with this project. Well, use it with big projects and you'll understand.

Maybe I could even create an utility that makes it easier to print a job's outputs as long as it's possible.

Class Log.Test [ Abstract ]
{

ClassMethod Write(
device As %String,
start As %Integer = 1) As %Boolean
{
  open device:(:33568::"\n"):3
  quit:'$test
  
  use device
  
  for i=start:1:1000000000 {
    write "This is a test #"_i_$c(13,10)
    if i#1000 = 0 {
      write "\n"
    }
  }
}

ClassMethod Start()
{
   set device = "|TCP|1"
   
   job ..Write(device)::3
   
   while 1 {
     open "|TCP|2":(:33568):3
     quit:'$test
     
     use "|TCP|2"
     read x:3
     
     close "|TCP|2"
     
     if x '= "" {
       use 0
       write x
     }
   }
}

}
 

0 0
John Matson · May 22, 2017

Depending upon your version of Caché you may also have the %SYSTEM.WorkMgr and %SYSTEM.WorkMgrIPQ classes available to use.  In WorkMgr child jobs on their own can write back to the parent processes main device so it isn't quite IPC in the sense that you're likely used to.  However, it does make multithreading a job a bit easier than spawning children and then monitoring them.  I've not used the WorkMgrIPQ class myself (at least, not yet) but the class documentation is closer to IPC in that structured data can be sent back to the parent/host process.

1 0