Check carefully. Note -- in ^%ISCLOG (with percent) you enable the log. Then you read ^ISCLOG (without percent) in %SYS namespace.
When I repeated steps that I suggested to you, I saw the following error in ^ISCLOG

/* ERROR #5002: ObjectScript error: <PROTECT>^%CSP.Login.1 ^|^^c:\intersystems\iris2023x1\mgr\|dc.CustomLogin.1 */

Then I enabled auditing of Protect events, reproduced the eror and got more details:

Description:  Attempt to access a protected resource
Timestamp:    2023-03-19 16:01:38.000   Username:     CSPSystem
UTCTimestamp: 2023-03-19 15:01:38.000   Pid:          10896
Event Source: %System                   JobId/JobNum: 131089/17
Event Type:   %Security                 Session ID:   eK95W6SMCS
Event:        Protect                   IPAddress:    127.0.0.1
System ID:    DEP5570AKOBLOV:IRIS2023X1 Executable:   CSPa24.dll
Namespace:    %SYS                      Index:        196
Roles:
User Info:                              O/S User:     CSP Gateway
Routine:      ^%CSP.Login.1 |"^^c:\intersystems\iris2023x1\mgr\irislib\"|
Authentication: Password
Event Data:
<PROTECT>^%CSP.Login.1 *^|^^c:\intersystems\iris2023x1\mgr\|dc.CustomLogin.1

Indeed, user CSPSystem does not have READ permission on irissys database, where custom login class is located. Rather I should have created a new role that has only READ permission on %DB_IRISSYS resource, not RW.

I added role %DB_IRISSYS to user CSPSystem, closed connections from Apache to IRIS, so that the role is added on new connection, then login page began to work

404 is also returned when there are some permissions issues.
Try to give %All to the user that you calling this API with and see if it helps. If yes -- then this is permissions issue.
Or enable Audit and auditing for Protect event and see if it is logged in Audit when the problem happens

Other idea -- enable ISCLOG, reproduce the problem, disable ISCLOG and see if there are any errors there. ISCLOG is very verbose, so just do one HTTP request with ISCLOG enabled

enable:

%SYS>kill ^ISCLOG,^%ISCLOG
%SYS>set ^%ISCLOG = 3

disable:

%SYS>set ^%ISCLOG = 0

analyze:

%SYS>zw ^ISCLOG (yes, without %)

What error did you get?

Are you sure, that the error is inside %CSP.Broker?

%CSP.Broker is a dispatch class to call server-side methods.

In most cases the error happens in the application method that is called via %CSP.Broker.

Enable logging for CSP Broker and see if any additional information is logged in ^ISCLOG:

USER>zn "%SYS"

%SYS>kill ^ISCLOG, ^%ISCLOG

%SYS>set ^%ISCLOG("Category","CSPBroker")=3

%SYS>set ^%ISCLOG=3

To disable logging:

%SYS>kill ^%ISCLOG

If %CSP.Broker is used to call ZEN Methods then enable also ZEN logs:

%SYS>do ##class(%ZEN.Utils).%StartLog()

%SYS>do ##class(%ZEN.Utils).%ShowLog()
1  13:40:51.606 OnPreHTTP               /csp/sys/%CSP.Portal.Home.zen
2  13:40:51.809 InvokeInstanceMethod    138@%ZEN.Auxiliary.jsonProvider:RefreshFromServer
3  13:40:51.830 InvokeClassMethod       %CSP.Portal.Home.GetNamespaceList
4  13:40:51.845 InvokeInstanceMethod    169@%ZEN.Component.html:ReallyRefreshContents

%SYS>do ##class(%ZEN.Utils).%StopLog()