The "write daemon" is a process that is responsible to write all new/changed data to the disk where the WIJ (write image journal) file is located. Only then, actual databases are being updated.

Sometimes, when the system is very busy with a lot of pending writes this error appears, and then after few minutes it is cleared automatically. (e.g. when you rebuild a huge index, or do some data migration process).

I would monitor the disk activity for the disk that the WIJ file is located on (by default its on the same disk you installed Cache).

One solution is to move the WIJ to a different disk, less occupied. This will give the "write daemon" more writing capabilities (you will have to restart Cache).    

As you probably know the Intersystems IRIS is capable of doing calculations on a numbers in length of up to 19 positions. This is due to the fact that it is stored as a signed 64–bit integer.

If you are using one of the latest version of IRIS which has an "embedded Python" then you will have the Python support for "bignum" or "long".
So you may have a ClassMethod like this:

Which will give you the output:

SAM is executing a REST API call to http://[your-server]/api/monitor/metrics for any server included in your cluster. I'm not sure where the interval is being configured.

If your own metric is needed to be run once a day, you can schedule it with the 'task manager" have the result stored in a global, and let the "user defined" sensor read this global which will not caused any performance issue.

BTW - one thing I forgot to mention in the previous post is:
In order to have SAM run your "user defined" metrics, you need to add it to SAM:

First create a class:

Add a parameter that will indicate the prefi name for al your used defined metrics.

Create a wrap method GetSensors() for all your user defined sensors (which can be ClassMethods):

Now you will get by the API REST call for the /api/mertics your "user defined" sensors at names:
Prefix_sensor1 and Prefix_sensor2

Remarks:
- Make sure that your GetSensors() and all your "used defined" sensors (classmethods) have a proper error handling so they are fail safe (you may use a try/catch or any other error trap like $ZT="something")
- Make sue all your "user defined" sensors are preforming fast. This will enable the SAM metrics API REST call to get the data quickly without delays. In case some calculations are "heavy" it is better to have a separate process (task manager) to do those calculations, and store them in a global for fast data retrieval by the sensor   

When you install SAM, it is usually installed with a container. (we use docker, so I don't have experience with podman).  

We have this on a seperate machine (Linux) when our IRIS servers are Windows, but I don't see any limitation (except memory & CPU = performance) to run a container with SAM on the same IRIS machine.

 Grafana, and Promethus are part of the "bundle" (container) for SAM do you do not need to install them separately.

Hello,

I have done a similar thing in the past with Cache. As long as at the end of the process the IRIS will be at a new partition with the same letter, all the keys in registry that were created during the install, will still be valid.
The risk with the procedure you mentioned is minimal, and all should work  as expectged.

get the mirror name:

ZN "%sys"
Set mirrorName=$system.Mirror.GetMirrorNames()
Set result = ##class(%ResultSet).%New("SYS.Mirror:MemberStatusList")
Set sc = result.Execute(mirrorName)
while result.Next() {
   Set transfer=result.GetData(6)
   // you may filer the check for a specific machine on GetData(1)
   // Do any check on "transfer" to see if behind and calculate the threshold time e.g.
  // For i=1:1:$l(transfer," ") {  
  //    If $f($p(transfer," ",i),"hour") { W !,"hour(s) behind" }
  //      Elseif $f($p(transfer," ",i),"minute") { Set minutes=$p(transfer," ",i-1) W !,minutes_" minutes behind" }
  //   }
}

To get any component status, you may use:

SELECT Name, Enabled FROM Ens_Config.Item where Name['Yourname'

To check queues you may use the following sql :

select Name,PoolSize from ENS_Config.Item where Production='YourProductionName'

Then, iterate on the result set and get the queue depth by:

Set QueueCount=##class(Ens.Queue).GetCount(Name)

To check latest activiy on a component, I would go to the:

SELECT * FROM Ens.MessageHeader where TargetQueueName='yourComponentName'

and then, to check the TimeProcessed

There is an option to get a long time token (refresh token), if tokens are "cached" locally, you may have a scheduled task to refresh them.
Another approach I would try here, is to use embedded python with this library:
https://pypi.org/project/O365/#authentication

On IRIS 2022.1 the class (%Net.POP3).Connect(...) has a 4th parameter: AccessToken 

I di not try, but maybe it will allow a connection with OAuth

You may do it programmatically:

set mirrorName=$lg(##class(%SYSTEM.Mirror).GetMirrorNames(),1)
zn "%SYS"
Set result = ##class(%ResultSet).%New("SYS.Mirror:MemberStatusList")    
Set sc = result.Execute(mirrorName)
while result.Next() {
   s status=result.GetData(9)   // get all other info you need from GetData(nnn)
   }

You have to distinguish between "journals" and "mirror-journals" files. the 1st are to ensure any instance DB integrity (DB corruption) in case of a failure. The 2nd are you ensure proper mirror "failover" and any A-Sync members.

When LIVETC01 (as a Backup) is "catch up" its a good source to copy .DAT files to the LIVEDR.
It is also safe to delete its mirror-journals.
The steps you did to catch up the LIVEDR are correct. (I assume you did "activate" & "catch up" after that in LIVEDR)

After the IRIS.DAT copy (of all DBs in mirror) from LIVETC01 to LIVEDR and both are "catch up" - It is safe to delete mirror-journals up to the point of the copy from your primary LIVETC02

Hello,

I would try to contact the author of the GIT you mentioned, to find out which version of Cache he used and get some help.
looks like he was updating the GIT at Apr 10, 2022, so this might indicate that he is active.
I saw that he is using a Unix driver "libcacheodbcur6435.so" but I'm not sure which version is this.