#Mirroring

Hi All,

Our mirrored HealthShare environment has failed over a few times recently due to underlying infrastructure issues (that are being worked on and resolved).

In the HealthShare logs we are seeing:

10/06/22-00:54:35:925 (4736) 1 Journal Daemon has been inactive with I/O pending for 10 seconds:
gjrnoff=524741316,iocomplete=523852600,filecnt=1011,fail=0
10/06/22-00:54:55:086 (4736) 3 CP: Pausing users because Journal Daemon has not shown
    signs of activity for 30 seconds.

Basically the title, I couldn't find anything concrete in the documentation regarding this. The documentation states 'The user-defined ^ZMIRROR routine allows you to implement your own custom, configuration-specific logic and mechanisms for specific mirroring events, such as a failover member becoming primary.' I didn't see any more details on what exactly those mirroring events are besides a failover member becoming primary, which I wasn't sure if that included a DR async becoming primary.

History

Hi Community,

Have you already set up a mirrored environment? Does it have a private network, virtual IP address, and SSL configuration? After doing this a couple of times, I realized that it is long, and there are a lot of manual actions required to generate certificates and configure each IRIS instance. It is a pain in the neck for people who often have to do this.

Hi

This  morning I noticed that the ISCAgent.log for one of my high availabilty healthconnect servers has many entries that say:

"Mirror remote control command requested access to instance directory not found in registry."
Requested:d:\intersystems\healthconnect\

In addition, Messages.log has many of the following entries:

ValidateFailoverPartner: Failed: Server at SERVER:2188 (d:\intersystems\healthconnect\) sent /NE/ not /OK/...(repeated 20 times)

As far as I can see, nothing has changed on either server (01 and 02) and the arbiter they use is working correctly.

Any idea?

Andy

How do I detect that sync mirror backup is caught up to primary? Or at least close to caught up.

I'm executing the code on a backup.

Hi all

I have a pair of servers configured as a mirror, each server sits in a separate data centre.

I have noticed that occasionally, the primary will report that the backup is disconnected in the Mirror Monitor, and I believe this to be due to connection conditions between the two data centres.

What I want to do is log the status of both servers from the primary on a regular basis using a service. I know %System.Mirror can return the status of the current member, but how can I log the get the status of the other member as well?

Cheers

This might be a dumb question but I didn't see anything about it in the Mirroring documentation. Are processes that are currently running when the primary fails over to the secondary automatically switched over and continue processing? Or are they terminated and would need to be restarted on the secondary?

Hi All,

When I log into Backup  mirror member it becomes too slow to load and navigate, I tried to check message log and I saw the error message about Database mirror latency and database disk issue which when I check it looks fine to me. Please have a look at the below screenshots and advise what the issue could be.

When I run df -h through SSH :

200G is the volume size, 194G is used space, 6.5G is available space and 97% IS %Use

Datababase view from management portal for the same db volume

Disk space error message from message log:

The below error shows sometimes when the system tries to load

I am receiving Mirroring updates to an IRIS for Health async reporting server and need to pause the mirroring journal processing to periodically quiesce the database to rebuild cubes and custom staging tables.

What API or web service can I use to quiesce the reporting server and then later initiate catch up?

Is it possible to mirror the same database on Iris and an older Ensemble instance?

Specifically to try a zero downtime upgrade

Hey Developers,

We're pleased to invite you to join the next InterSystems IRIS 2020.1 Tech Talk: DevOps on June 2nd  at 10:00 AM EDT! 

In this InterSystems IRIS 2020.1 Tech Talk, we focus on DevOps. We'll talk about InterSystems System Alerting and Monitoring, which offers unified cluster monitoring in a single pane for all your InterSystems IRIS instances. It is built on Prometheus and Grafana, two of the most respected open source offerings available.

Next, we'll dive into the InterSystems Kubernetes Operator, a special controller for Kubernetes that streamlines InterSystems IRIS deployments and management. It's the easiest way to deploy an InterSystems IRIS cluster on-prem or in the Cloud, and we'll show how you can configure mirroring, ECP, sharding and compute nodes, and automate it all.

Finally, we'll discuss how to speed test InterSystems IRIS using the open source Ingestion Speed Test. This tool is available on InterSystems Open Exchange for your own testing and benchmarking. 

Hello IRIS Community,

InterSystems Certification is developing a certification exam for IRIS system administrators and, if you match the exam candidate description given below, we would like you to beta test the exam. The exam will be available for beta testing on June 20-23, 2022 at InterSystems Global Summit 2022, but only for Summit registrants (visit this page to learn more about Certification at GS22) . Beta testing will open for all other interested beta testers on July 18, 2022. However, interested beta testers should sign up now by emailing certification@intersystems.

Hi:
The task Mirror Monitor Launch Task recently failed and is placed in Suspend leave. Although it says USER it is not known as a task for us. 

The code runs is HSSYS which is not our main system or namespace for production items. 

The task name is HS.Util.Mirror.Task runs 

If $system.Mirror.IsMember() Do ##class(HS.Util.MirrorMonitorAgent).StartAgent() and just says the description 

/// This is the task definition that Launches the mirror monitor on any mirrored instance

I don't understand why this task runs or what it does. I assume it has just suspeneded as the startAgent() failed.

Hi,

What's the best way to get list of members attached to a Mirror? 

In my scenario, in a Mirror set there are 5 members. 2 Failover, 2 DR Async and 1 R/W reporting.

On Primary, in the code I want to identify Backup member and then do my stuff.

And on Backup, in the code I want to identify DR Async and then do my stuff.

Basically the idea is to push file through code from Primary > Backup and then later files will be pushed from Backup > DR Async.

Regards

Kamal Suri

I want to write some code differently to run on Primary, Backup and DR depending upon their status.

If its Primary then do something, if Backup then something else and if DR then do something different from both Primary and Backup.

This can identified whether a system is Primary or Not Primary by calling below method

w $SYSTEM.Mirror.IsPrimary()

1 means Primary and 0 means Not Primary. But a Not Primary member may be a backup or a Async DR or a Async Reporting.

How to identify if a system is a DR or Backup or Reporting?

I have described my efforts to optimize IRIS Mirror deployment in AWS ElasticContainer Service (ECS) in my prior article.

IRIS Mirror in the cloud (AWS) | InterSystems Developer Community | AWS
 

I have come to the opinion that IRIS Mirror is not as reliable as needed when deployed in ECS. The root of the problem is the fact that ECS randomly assigns one of the available IP addresses to each EC2 host or Fargate task it starts.

These get stored in iris.cpf file in MapMirrors section as shown here:

[MapMirrors.IRISMIRROR]

FAILOVER1=10.2ab.1cd.146,2188,,10.2ab.1cd.

I have been working on redesigning a Health Connect production which runs on a mirrored instance of Healthshare 2019. We were told to take advantage of containers. We got to work on IRIS 2020.1 and split the database part from the Interoperability part. We had the IRIS mirror running on EC2 instances and used containers to run IRIS interoperability application. Eventually we decided to run the data tier in containers as well. Later we switched from using EC2 instances to Fargate “server-less” compute.

Hi,

I have a DR member and in this cache server the database "cachetemp" start to getting bigger without any reason (50GB that was all the free disk space we have)

In the members of the mirrors the cachetemp its ok and the size is 31MB.

I restarted the server because I read that the cachetemp database purge when restarting, but didnt happend.

Any recommendation to clean this database? can I just deleate the CACHE.DAT from this database?

Thanks!!!

Hi, 

I'm trying to add a DR async member to a mirror but when I add this member I get this messages in the mirror monitor (on the DR member):

The message im talking about is the "missing Mirrored Databases report".

The only step I did was the " System Administration -> Configuration -> Mirror Settings -> Join as Async" an fill the blanks. Maybe Im skipping a step? 

I dont get how to make this DBs start getting synchronized, the cconsole log dont show nothing 

Thanks a lot!!!

Purpose

Most CloudFormation articles are Linux-based (no wonder), but there seems to be a demand for automation for Windows as well. Based on this original article by Anton, I implemented an example of deploying a mirror cluster to Windows servers using CloudFormation.I also added a simple walk through.
The complete source code can be found here.

Update: 2021 March 1 I added a way to connect to Windows shell by public key authentication via a bastion host as a one-liner.

When I create a TASK on the Primary mirror member and Try to create the same TASK in the backup mirror member,  It Gives me CSP Application error.

The TASK gets created in the Primary node without any errors and When creating in the Backup node it prompts the error.

It happens is IRIS and I didn't face this in CACHE.

Finally what I did is export the TASK in Primary and Imported in Backup.

Is this a feature of IRIS?

In the context of IKO (Iris Kubernetes Operator) the question of Service not redirecting dynamically to the correct Pod is still pending.
In production this can be dangerous since an overload (or any other simpler problem) can cause you to change the main Pod and leave the application inoperable until we intervene.

Intersystems support warned that this is still an issue of IKO, but there are some possibilities that I am studying.

Hey everyone.

I have noticed that my backup mirror is warning that the MirrorDatabaseLatencyTime is having a bad time (time in ms is 3000, and warnvalue is 3000). While I look into what may be causing this latency between the two servers, I was considering if reducing the size of the journal files would improve this value in any way.

My assumption is that reducing the file size would mean that the frequency of the journal files being created would be increased, but the reduced size would mean that the transfer and application of each file would be reduced.

Has anyone had experience in this area?

Good day! Is there an opportunity to debug the ISCAgent behaviour (in Linux)? ISCAgent uses /etc/iscagent/iscagent.conf as configuration file, but in docs I've found a description only for two parameters (port number and interface - http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=GHA_mirror#GHA_mirror_set_agent). Other parameters I can see by running /usr/local/etc/cachesys/ISCAgentCtrl status:

application_server.interface_address=*
application_server.port=2188
daemonized=yes
mirroring=on
pid=41850
running=yes
version=2015.2.0.664.

Hello!

First of all, let me state that I am no senior InterSystems expert. 

In my organization, we have a HealthShare Health Connect setup where each namespace has one code database and one data database, which are both actively mirrored. We have two nodes in the mirror.

We had a controlled failover last night to make sure that the backup node works as intended, which it didn't. It turned out that we had only deployed code onto the primary node in several namespaces causing errors with missing classes after the failover.

Hello,

I work on deploying IRIS inside Docker container. I really like %Installer class can automate many steps. I want to establish an ECP connection to a mirror database and then define a remote database on the application server. I have already seen we can create local database and namespace in %Installer. What code is needed to establish ECP connection?

I have a Read-Only Reporting Async with a Remote Database defined to a non-mirrored database on the primary failover member. I would like to update the non-mirrored database on the primary from the async; so far all my attempts fail with *protect* errors.

I found this page 

https://community.intersystems.com/post/execute-method-failover-member-…

and tried the SqlProc method. The procedure executed, no errors, but the change to the global was never made (or least not retained) either directly or through a JOBbed process. I wonder if I'm missing something in my configuration.

Is there a way to force a failover from the primary mirror member to the backup member without forcing down the primary (which happens with ##class(SYS.Mirror).BecomePrimary())? What I'm trying to simulate is a network isolated condition on the primary.

Hi, we have mirrored databases, now we have three nodes. Primary, backup, async. They are sync by journals and with some TB each node (expensive storage).

I would like to have a scenario where the two nodes (Primary and backup) have the same amount of messages (purge task of x days) , both as failover, but the asynchronous node should have more messaging, as much as the storage allows us. We want to use this node so provider can give support(search old messages), but not have databases in production that are so large that they are hardly consulted.

Any ideas?

Thanks in advance!

Hi,

I want to know if there is a way to manually change Primary to secondary  and vice versa?

I need this becasue we backup the two VMs (primary and secondary) and my idea is to make a script to when the primary VM is going to backup with Veeam change to "backup" on the mirror.

Thanks!!!