Changing the Registry Endpoint to the "/services" URL after Upgrading

After upgrading an Information Exchange/HealthShare instance that has a registry namespace, you may notice that the Caché audit log gets filled with frequent login failures by the HS_Services user (check this by going to Management Portal > System Administration > Security > Auditing > View Audit Database). Clicking the “details” shows event data like:

Error message: Access Denied
CSP Application: /csp/healthshare/hsregistry
Authentication: Delegated

Problem Background

Every time a remote HealthShare instance makes a call to the registry (“hsregistry” in this example), it first fails delegated authentication, and then succeeds at password authentication (if the remote instance is failing to sync or communicate at all, address that problem first by confirming the correct credentials are being used). This issue appears after upgrading the registry instance to HealthShare Core 13 or higher from an earlier version, and is caused by changes in the new HealthShare and Caché versions. New installations of Core 13 or higher are not affected.

The goal this enhancement was to separate HealthShare user and web service calls into two different web applications that are able to handle authentication differently. Prior to Core 13, a web app named like "/csp/healthshare/hsregistry" handled authentication for both users and calls by web services. After upgrading, a new web app is created for each HealthShare namespace with a name ending in "/services", like "/csp/healthshare/hsregistry/services", which is configured to handle the web service calls. On a Core 13 instance installed from scratch, the logins by HS_Services from remote edges and access gateways would be directed here, while the non-"/services" web application ("/csp/healthshare/hsregistry") would use delegated authentication to handle logins by users who need registry access.

However, this new approach is more difficult for the installer to implement on systems that are already operating. After upgrading from pre-Core 13, activating the HealthShare namespaces creates the "/services" web apps but doesn't automatically start using this version of the URL as the registry's endpoint. This cautious approach to upgrading the registry instance is meant to prevent it from suddenly being inaccessible to external systems that are still using the non-"/services" URL. Instead, the calls from these systems are allow to first fail delegate authentication and then succeed at password authentication. This situation can persist on systems long after the upgrade, since it doesn’t cause failures or Ensemble event log entries.

Motivations for making this change

Even though the system will continue to work without taking any action, there are good reasons to follow these steps after upgrading. The flood of irrelevant messages in the audit log wastes space and can make it difficult to see important patterns of login errors. Also, having to first fail at delegated authentication causes these calls to take more time and waste system and network resources. Finally, separating services and users into different web applications opens up the possibility of improving security by using two-factor authentication (available on Core 15 and higher) for the users' web app.

Steps to fix this problem

1. Change registry endpoint in the Service Registry.

Go to Management Portal > HealthShare > (then navigate to your registry namespace) > Registry Management > Service Registry. Select the entry with the same name as the registry namespace (HSREGISTRY in this example). In the "URL" field, add "/services" after the namespace. For example, change

/csp/healthshare/hsregistry/HS.Hub.HSWS.WebServices.cls

to

/csp/healthshare/hsregistry/services/HS.Hub.HSWS.WebServices.cls

Click "Save". This change should propagate to remote HealthShare instances within a few minutes, provided they are started and syncing with the registry. To confirm, you can check the SQL table “HS_Registry_Service.SOAP” on each remote namespace. Start or manually sync any namespaces that are missing "/services" in the EndPoint column of the registry’s entry in this table.

2. Update any other references to the old URL
If any external systems that communicate with the registry have the non-"/service" URL hard-coded (rather than pulling it from the Service Registry), update the URL there too. It is important to be sure no more web service calls will be made to the old URL before the next step. If you continue to see audit log login errors in the format shown at the top of this article, then some web service client hasn't been switched to the new URL yet.

3. Stop allowing web service calls on the non-"/services" web application.
This final step isn't necessary to eliminate the audit login failures, but is a good step to prepare for possible using two-factor authentication in the future. Go to Management Portal > System Administration > Security > Applications > Web Applications. Click the non-"/services" entry for the registry (for example, "/csp/healthshare/hsregistry"). Uncheck "Inbound Web Services" and click "Save". Watch the audit log to confirm there are no login failures caused by any remaining web service calls to this application. Only the checkbox for inbound web service calls should need to be changed after the upgrade, but for reference, here are screenshots of the non-"/services" and "/services" app settings, followed by text descriptions of the settings  needed for the two apps to work as intended. The settings that aren’t highlighted or mentioned might vary according to your needs:
 

In the non-"/services" web application:
"CSP/ZEN" is checked
"Inbound Web Services" is unchecked
"Delegate" is checked
 

 
In the "/services" web application:
"CSP/ZEN" is unchecked
"Inbound Web Services" is checked
"Password" is checked
"Delegate" is unchecked

Other Considerations

After step 3, be aware that any remaining pre-Core 13 remote instances still won't know that the default hub endpoint contains "/services". For instance, if you create a new edge or access gateway on an old system, you will have to add this to the URL that the Installer Wizard generates. These productions may also take several minutes to sync for the first time. Upgrading the remote instances eliminates these minor issues.

These steps can be taken for other namespaces for which you are seeing login failures or plan to setup two-factor authentication. Note that buses, access gateways, and edges might have “/services” automatically added to their URLs when their productions are started for the first time after an upgrade. If you are running an IHE instance, the Service Registry entries for IHE components on the registry and bus namespaces won’t be changed automatically, and calls to these endpoints may be a continued source of login errors.