Ivan Wang view profile on InterSystems Developer Community

Ivan Wang · May 12, 2022

Yes, there are primary server, backup server and an arbiter. Recently servers experienced some network connection issue, lots of primary/backup switch happened. In the log, I found lots of above log content.
"

[Generic.Event] ECP: Mirror Connection request from 'SM25SDLMIIRROR:LOCALHOST.LOCALDOMAIN:IRIS' (10.xxx.xxx.27:33040)

05/05/22-01:05:11:376 (16166) 1 [Generic.Event] ECP: Active server not defined, cannot redirect mirror connection

Also, there are lots of similar content like following:

05/07/22-18:06:11:800 (24254) 1 [Generic.Event] MirrorServer: Timed out (66 sec) waiting for ack from 10.160.14.27/IRIS. Expected seq #1783871. ReadDmn Exiting

05/07/22-18:06:16:462 (11727) 1 [Generic.Event] MirrorServer: Mirror id #0 set trouble state. Failed to send journal data through 345779884 for file #15237 (cur = 15237/343868564)within timeout period. Disconnected from journal system

05/07/22-18:06:39:269 (12039) 0 [Utility.Event] Available buffers have been recovered

05/07/22-18:07:19:980 (24252) 1 [Generic.Event] MirrorServer: Failed sending journal data. Disconnecting from journal daemon

05/07/22-18:07:19:980 (24252) 2 [Generic.Event] MirrorServer: Connection to 10.160.14.27/IRIS (backup) terminated

05/07/22-18:07:19:987 (29982) 0 [Utility.Event] Backup has disconnected from Arbiter - waiting for reconnection.

05/07/22-18:07:19:988 (29982) 0 [Utility.Event] Primary switched to agent contact mode: Data channel lost, backup lost arbiter connection

05/07/22-18:07:39:358 (12039) 1 [Utility.Event] Available buffers are getting low (25% above the threshold for suspending updates)

05/07/22-18:07:46:186 (11727) 1 [Generic.Event] MirrorServer: Mirror entered trouble state (backupjrnend = 343868564)

05/07/22-18:07:46:187 (11727) 1 [Generic.Event] MirrorServer: Mirror id #0 set trouble state. Failed to send journal data through 361794312 for file #15237 (cur = 15237/343868564)within timeout period. Disconnected from journal system

05/07/22-18:09:46:212 (11727) 0 [Generic.Event] MirrorServer: Mirror trouble cleared, Agent Controlled failover, trouble timeout expired (120025280) v. 120000000

05/07/22-18:09:46:240 (12039) 2 [Utility.Event] [SYSTEM MONITOR] WriteDaemon Alert: Write Daemon still on pass 5514

05/07/22-18:09:48:175 (19792) 0 [Generic.Event] MirrorServer: Starting connection for: 10.160.14.27/IRIS (failover member) (ver 4) at offset 0x14830158 (344129880) in file #15237

05/07/22-18:10:16:301 (12039) 0 [Utility.Event] [SYSTEM MONITOR] WriteDaemon Alert: Write Daemon completed a pass.

05/07/22-18:10:46:309 (12039) 0 [Utility.Event] Available buffers have been recovered

05/07/22-18:10:56:604 (19792) 0 [Generic.Event] MirrorServer: Client up to date, linking with journal daemon @ 0x16f40748

05/07/22-18:10:57:337 (19792) 0 [Generic.Event] MirrorServer: Switched from Agent Controlled to Arbiter Controlled failover