Are you trying to add TLS (aka SSL) to your connection, or use WS-Security?  Which steps to take next depend on which of those you're trying to set up.

If you're trying to add TLS, you'll want to create an SSL/TLS configuration, and use that in the operation.   You can find information about these configurations in the documentation.  You'll likely also want to read up on certificates in order to understand which files to put in the configuration.  It sounds like you already have some certificates (PFX is a certificate format), but I'm not certain whether that is your certificate or the CA certificate for the other side.

The TLS configuration information above assumes your operation is one end of the connection.  That's likely since you're sending the data instead of receiving it.  If you were receiving it, you'd want to understand whether your webserver was the endpoint.

There are two different connections here - one from the browser to the webserver, and one from the CSP gateway to Cache. Either or both can use SSL and they are configured separately.

You said you want HTTPS. This would be used on the connection between the browser and the webserver. It does not involve Cache or the CSP gateway at all. It is configured entirely in the web server configuration. For example, if you are using Apache, it is configured in the httpd.conf and related Apache conf files.

The settings you've shown above are for the CSP gateway to Cache connection. If the gateway and Cache are on separate machines, you may also want to configure SSL for this connection. The gateway will be connecting to the SuperServer on Cache, so you will want to follow the instructions for configuring SuperServer SSL if you want to get this part working. The instructions for that are here:

https://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY...

You might be interested in this page:

https://docs.intersystems.com/ens20181/csp/docbook/DocBook.UI.Page.cls?K...

Or potentially some of the documents on this one:

https://www.intersystems.com/gt/

Database encryption uses AES. You select the key size when creating the key; 128, 192, and 256 bits are all options.

If you have a specific question about standards not covered there, I would recommend contacting the WRC.

The answers you've gotten so far are for the management portal, but you also mentioned Studio, and I wanted to follow up on that part, as it's configured quite differently.

Studio connects directly to the SuperServer, so you will need to configure SuperServer TLS on the server side if you haven't already. The documentation discusses how to do this here:

https://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY...

Currently, I don't know of a way to enforce TLS for just the %Service_Bindings, so you will need to enforce it on all connections to the SuperServer if you want to require it. This means you'll need to configure TLS for any other types of connections to the SuperServer you use, including between the CSP gateway and the SuperServer.

Each machine which has Studio installed will also need to be configured to use TLS. I've written up how to do that here:

https://community.intersystems.com/post/configuring-cach%C3%A9-client-ap...

There are a lot of details not included here which could be necessary.  For example:

Are you using a custom login page?  The "invalid password" message you state should never be returned by default Cache pages.  This message would leak information to an attacker by letting them know that they had found a valid username.   "Access denied" is the standard message returned by Cache when a login fails, for any reason.

Have you checked the audit log for login and/or loginfailure events?  You may need to enable auditing, and then the individual event types, then reproduce the problem.  The loginfailure event should give a reason for the failure to log in.  Depending on what's happening here, it may not be the same as the error returned to the user.   

The format on that public key doesn't look right.  OpenSSH format keys usually start with ssh-rsa, then the encoded key.  They do not have ascii armor (the -----BEGIN SSH2 PUBLIC KEY----- part.)

Whenever you see a message in the log which makes you think the system may be hung or frozen, such as 'Freezing system', please run the CacheHung command before restarting the instance or OS.  This will collect information on how and why the system is hung, and may direct you to take further steps to collect information.

CacheHung is an OS script located in the bin directory under your install directory.  On a Linux system, you can run it with a command such as:

bash$ cd /InterSystems/Cache/bin

bash$ ./CacheHung.sh

You'll need to use your own path to the bin directory, of course.  There will be several prompts about what to collect, and the script will take a minute or two to run.  Depending on your configuration, you may need to be root to run it.

Once the data has been collected, you can open the generated file with a web browser and go to the "self-diagnosis" section to see whether more steps are suggested.

There isn't currently a way to avoid running ZAUTHENTICATE before password logins.  Running the routine is deliberate behavior because it's used by some sites as part of their login process.  You could ask for a change, such as a configuration option to control the behaviour, to be added but that won't fix your issue right now.

Have you considered changing the existing password users to being delegated users?  This would remove the extra login failure event for them.  You would have to update your ZAUTHENTICATE code to handle the accounts, or create matching accounts on your AD server.

You can change user account types by editing the user objects.  Here's an example of how you can do it:

%SYS>s status=##class(Security.Users).Get("Admin",.prop)

%SYS>w prop("Flags")  ; This is the property which states this is a password user account                                   
1
%SYS>s prop("Flags")=4   ; This value says this is a delegated user                 

%SYS>s status=##class(Security.Users).Modify("Admin",.prop)
 

This isn't one of the errors that easy to diagnose based on the client log, unfortunately.  Here are a couple of the most likely things it could be:

- This error can happen when the client and server don't support compatible versions of TLS.  For example, if you're configured to use only TLS v1.0 and the server wants TLS v1.2.  This is more likely if you're using an older version of Ensemble which doesn't support TLS v1.2.  Which versions of SSL/TLS have you enabled? (Note: trying to fix this by enabling all possible versions is a bad idea.  SSLv3 should not be used.) 

- This could also mean that the server requires server name indication (SNI) and the SNI info is not being sent by your client.  SNI is supported in Ensemble 2017.2. 

This isn't causing your failure, but I would recommend that you change the value of peer certificate verification from 'none' to 'required'.  As a client, you want to check that you're connecting to the server you think you are, not another server pretending to be that one.   You will need to set a certificate authority file for this to work.

That error is the generic failure, so it doesn't tell us what's wrong.  You can get more information by enabling the network debug flags:

%SYS>d INTSET^REDEBUG("FFFFFFFF")

and then trying again.  Detailed errors will be printed in the cconsole.log.  This level of debug info can fill the log quickly, so remember to set the level back to "FF" after you're done testing.

Also note that the test button in a configuration can't handle any protocols where there are messages before the TLS handshake.  This shouldn't be a problem for https, but it may be an issue for SMTP, FTPS, etc.

The 'troubled' state you're referring to is the state listed in the portal's system dashboard, correct?

This is likely to be due to the write daemon not completing a pass in a timely fashion.  I'd start by checking the cconsole.log for any messages about the write daemon.

With delegated logins, you need to return an array from ZAUTHENTICATE with all the properties you want the user to have, including their roles.   This happens during login; you can't change the roles once login is over.  The array used is Properties.  To set the roles in this array, you would do something similar to:

set Properties("Roles")="ACustomRole" 

This would set this user to have the role named ACustomRole.  

Your code gets to decide what that roles are, and can do it based on the group you retrieved.    For example, If you have a group named CacheAdmin and you want all members of that to get the %Manager role, you could do something like

if group="CacheAdmin" {set Properties("Roles")="%Manager"}

(Note that this code makes many assumptions that may not be true for you, such as that you want to overwrite Properties("Roles") instead of adding to it, and the name of the variable you're holding the group name in.)

If you want to add more than one role, you can use a comma separated string, such as "%Manager,ACustomRole"

The %Net.SSH.Session.SetTraceMask method tells a process to collect information about an SSH handshake and connection in a text file, which is useful for debugging SSH connections.  It is information about what the local process is doing and does not include network trace information, such as would be in a WireShark pcap file.  

If you're trying to collect information about an SFTP connection problem, you can use the same SetTraceMethod for the SSH session associated with your FTP connection to collect information.

If you're trying to collect information about an FTPS connection (meaning SSL is in use, not SSH), you can use the ^REDEBUG utility to log information about the SSL handshake to the cconsole.log.  You can enable the REDEBUG flags by running:

Do INTSET^REDEBUG("FFFFFFFF")

in the %SYS namespace.  Remember to set the flags back to "FF" when you're done testing, as the higher log level can generate a lot of data.

If you're trying to debug an un-secured FTP connection or these don't provide the information you want, you may need to use a network trace tool such as Wireshark.

I would suggest you contact the WRC about these questions.  

I usually find that connections may be delayed but do succeed after the "Failure Event: -37 - Unable to exchange encryption keys" messages, so I suspect there may be another issue here.

Using StartTLS (explicit mode) is the default for %Net.FtpSession.   (If you wanted to, you could turn explicit mode off by setting the LegacySSL parameter.)

Have you looked at the connection attempts with a network trace to see what's actually happening?  You might also find useful information by setting the network debug flags to FFFFFFFF via ^REDEBUG in the %SYS namespace, trying a connection, and checking the cconsole.log for information about the SSL configuration which was invoked.  

How would you secure and authenticate network access to the key?  This seems like a question you would want to have a reasonable answer for before considering sending a key over the network.

You might also want to look at the features in InterSystems IRIS data platform.  KMIP support was announced as one of the features in it.

There aren't enough details here for me to guess exactly what's going wrong, but I can suggest some debugging steps that might help make things clearer.

First, if you haven't already, enable auditing and the Protect,  Login and LoginFailure events.  Then try reproducing this and see whether any audit events were generated.  The data in these may make it clearer what's wrong.

Second, I'm not sure how you checked the ^ERRORS global, but if you haven't looked at the "application error log" in the portal (under System Operation > System Logs) I would recommend doing that.  That page lets you see all the namespaces with errors logged.

What you're describing is moving the location of an instance after installation.  It's not supported.  I've seen people do some pretty bad things to their systems by trying it, and would strongly recommend against it.  

You will need to either run the installer in the new path, or keep the paths the same.

I agree - that SOP sounds like you have an outside vendor acting as a CA.  If you're using a CA vendor that still only supports SHA-1, it may be worth considering looking for a new vendor.  Whether an internal or external CA makes sense for you long term can be a complicated question.  It's worth having a detailed internal discussion about what your security requirements are and what the risks and benefits of each case are before making that decision.  

If you want to look at a PEM-format certificate in order to see who it's signed by, you can run:

openssl x509 -in certificate-to-display.pem -noout -text

where certificate-to-display.pem is the name of the file you want to read.  This will display the certificate in a human-readable format.  The CA who signed the certificate will be listed in the Issuer field.  

I'm worried that you may be conflating several ideas here.   You said you need to sign your partner's certificate to do mutual authentication.  That's not required technically.  (Maybe it is required by policy for your particular organizations, but I didn't see you say that so I suspect it's not.)

A CA is a trusted party for verifying identities, usually a third party.  You don't need to sign your partner's certificates yourself to do mutual authentication.  That's the power of using a CA - you don't have to do identity verification and issue certificates for every possible connection partner, you trust the CA to do it for you.   

To use an analogy, if a certificate is a driver's license, then the CA is the state agency that issues licenses.  If you operate a service where you need to check identities of people, you don't need to start issuing your own licenses, you can just trust ones issued by states that you think 1) do a good job checking identities before giving out licenses and 2) you recognize and know how to tell if they're real.

To do mutual authentication, you need to have a certificate, and the other side needs to have a certificate.  Both certificates need to be signed by CAs the other side trusts - meaning your certificate needs to be signed by a CA that your partner trusts, and their certificate needs to be signed by a CA you trust.  

You can (and probably should) get your certificates for use in TLS connections with Ensemble/HealthShare through whatever processes you normally use for generating certificates.