Cache failes to switch JOURNALs to the Primary directory if the instance started on Secondary directory?

Caché

Sounds odd - but I encountered it - twice now. Cache was restarted. in that period the primary journal directory was full and it was writing to the secondary journal directory. So upon start-up, it was quite obvious that it was not going to write to the primary. Cache reported this but was able to continue with the start-up but writing to the secondary. All of a sudden it doesn't recognize the 'primary' config any more and started saying 'Alternate and primary journal directories are the same'. The problem with this is that when Secondary was full, it did not switch back to 'Primary' and obviously this created problems for us.

So cache fails to switch to Primary if it started in Secondary???

%SYS>w $ZVERSION
Cache for UNIX (Red Hat Enterprise Linux for x86-64) 2015.1.1 (Build 505U) Wed Apr 29 2015 12:02:38 EDT
%SYS>!cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)

Below is a hindsight of the cconsole.log:

------------------First messages at startup----------------------------------------------------------------------------------------------

03/24/18-11:18:29:438 (5035) 0 START: /tcl_prod_int/ensjrnl/20180324.125
03/24/18-11:18:29:442 (5026) 2 JRNCREATE in jrnstart() - Error 28=(No space left on device)
03/24/18-11:18:29:443 (5035) 0 START: /tcl_prod_int/ensjrnlalt/20180324.125
03/24/18-11:18:29:443 (5035) 1 Warning: Alternate and primary journal directories are the same ***At this stage it was confused already***
03/24/18-11:18:29:511 (5035) 0 CACHE JOURNALING SYSTEM MESSAGE
Journaling started to: /tcl_prod_int/ensjrnlalt/20180324.125
03/24/18-11:18:29:512 (5035) 0 Journaling to /tcl_prod_int/ensjrnlalt/20180324.125 started.

------------------------------Later on messages showing it failed to Switch to Primary - The Primary by this date was empty after old journal files were deleted--------------------------------------- 

03/28/18-19:32:26:435 (5080) 1 [SYSTEM MONITOR] JournalSpace Warning: Journal Space is below 243.22MB
03/28/18-19:44:00:559 (5080) 2 [SYSTEM MONITOR] JournalSpace Alert: Journal Space is below 44.41MB
03/28/18-19:45:47:673 (5026) 2 JRNCREATE in jrnswitch() - Error 28=(No space left on device)
 Reoccurrences of the error will be suppressed for the next 60 seconds
03/28/18-19:45:47:673 (5026) 2 CACHE JOURNALING SYSTEM: Journal switch failed
03/28/18-19:45:47:704 (5026) 2 CACHE JOURNALING SYSTEM: Journal updates are being blocked due to the ongoing journal I/O errors
03/28/18-19:45:48:705 (5026) 2 CACHE JOURNALING SYSTEM: Write to journal file has failed

-----------------------------------After rescuing the condition- Had to manually trick it by removing an old file in the Secondary directory for Cache to resume as it was in hung state - then re-saving journal settings to force a switch------------------------------------------------------------------------

03/28/18-22:54:49:361 (5026) 2 CACHE JOURNALING SYSTEM: Write to journal file has failed
03/28/18-22:55:36:514 (5026) 2 CACHE JOURNALING SYSTEM: Write to journal file has failed...(repeated 46 times)
03/28/18-22:55:36:514 (5026) 0 CACHE JOURNALING SYSTEM: Journal output completed - Journal I/O operation resumed
03/28/18-22:55:36:517 (5026) 0 CACHE JOURNALING SYSTEM: Journal updates are no longer blocked
03/28/18-22:55:37:026 (17472) 0 CACHE JOURNALING SYSTEM MESSAGE
Journaling switched to: /tcl_prod_int/ensjrnlalt/20180328.106
03/28/18-22:55:37:162 (5080) 2 [SYSTEM MONITOR] TransOpenSecs Alert: One or more transactions open longer than 20 minutes. Process id(s): 5144 5207 5391 5398 5537 6158 6335 6426 6791 6802 6828 6853 8659
03/28/18-22:56:07:620 (5080) 0 [SYSTEM MONITOR] JournalSpace Alert: Journal Space ok.
03/28/18-22:59:46:539 (17511) 0 CACHE JOURNALING SYSTEM MESSAGE
Journaling switched to: /tcl_prod_int/ensjrnl/20180328.107
03/28/18-22:59:46:539 (17511) 0 SWITCH: /tcl_prod_int/ensjrnl/20180328.107

------------------------------------------------------------------------------------------------------------

Regards;

Anzelem.

Comments

In case this behaviour is version-specific, please tell us your $ZVERSION string.

%SYS>w $ZVERSION
Cache for UNIX (Red Hat Enterprise Linux for x86-64) 2015.1.1 (Build 505U) Wed Apr 29 2015 12:02:38 EDT
%SYS>!cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)

I think this is a better question for support. I recommend opening a WRC issue for this.