The two places that I would look for more information on a <PROTECT> error are the audit log and the line of code the error is coming from.   The audit entry for a <PROTECT> error sometimes has more information about why the error was generated.  The line of code can also help as it can show you whether there was an explicit permission check, a reference to a particular global, etc.

Yes, you must enable an authentication type system-wide before you can use it in an application.  An authentication type must be enabled both system-wide and in the individual application or service the login is using to be used.

You can turn LDAP off in all other applications if you only want it in the one application.   The one which is using LDAP authentication will need to look at the shared system-wide settings about domains, servers, attributes, etc, to know what to do with users logging in. 

If the users and employees are part of different domains, you might want to look into the multiple domain support.  You may be able to use the multiple domain support to let both sets of logins work separately.

It sounds like you're trying to configure a webgateway on Linux to authenticate to an IRIS server on another machine using Kerberos.  Is that right?

If so, are you already using Kerberos authentication on the IRIS server for other connections?    I would start by making sure Kerberos is working for other connection types first, which will help sort out whether this is a configuration problem on the server side or the Webgateway side.

Do you already use Kerberos elsewhere on the Linux server?

Yes.  There's a property in the %Net.FtpSession class which says which one to use: LegacySSL.  If it's true, you'll get implicit FTPS.  If it's false, you get the default of explicit FTPS.  There's a note in the documentation of that property you might try out if you haven't already:

"Depending on the configuration of the server you are talking to it may be needed to also send 'PBSZ 0' and 'PROT P' before you can communicate, this can be done with 'Set rc=ftp.sendCommand("PBSZ 0"),rc2=ftp.sendCommand("PROT P")'."

If that doesn't help, you might need to give more details about what error it's giving.

Cache 2017.1 supports Kerberos and OpenAM, which are both SSO methods.  You can also implement your own authentication methods using delegated authentication, or use LDAP to do logins with domain accounts (which is not full SSO since you have to type the login again, but sometimes people call it that because it's just one account.)   If you want to set up SSO via SAML, you'll need to write some code to handle that, there isn't anything that can be enabled by just configuring it.

I am assuming you mean Cache database encryption keys, which are stored in files.  If the HSM supports transparently giving the file to Cache when asked, I don't see why this wouldn't work, but I doubt it's been tested. 

If the HSM requires you to enter a passphrase, use a private key to decrypt the file, or other interactive step, you might or might not have a problem.  How are you planning to do the key activation?  Will you be present and able to do those steps?  

I assume your HSM does not support KMIP, which is a communication protocol for sending keys to a central key server.  Cache does support KMIP.

First, are you trying to set up stunnel on the Studio machine to encrypt the connection, stunnel on the Cache server to decrypt the connection, or both?  If you're trying to do stunnel on the Studio machine, you would configure it to listen on a local port, then configure Studio to think that is the host and port of the Cache server.  Studio connects to the local port and stunnel forwards the data to Cache after encrypting it.  Using stunnel with Studio shouldn't be different than using it with other protocols; Studio isn't doing anything unusual with the connection.

Second, is there a reason you're using stunnel instead of the built-in TLS support for Studio?  It's been available since Cache 2015.1.  Here's a writeup on setting it up: https://community.intersystems.com/post/configuring-cach%C3%A9-client-applications-ssltls