Securing IRIS Integrations with Mutual TLS (mTLS): A Practical Guide

In today’s enterprise environments, secure communication between systems is not optional—it’s essential. Whether you're integrating InterSystems IRIS with cloud APIs, internal microservices, or third-party platforms, Mutual TLS (mTLS) offers a powerful way to ensure both ends of the connection are authenticated and encrypted.

This post walks through how to configure IRIS for mTLS and how to validate your certificates to avoid common pitfalls.

🔐 What is Mutual TLS (mTLS)?

TLS (Transport Layer Security) is the standard protocol for securing data in transit. In traditional TLS, only the server presents a certificate. Mutual TLS goes a step further: both the client and server present certificates to authenticate each other.

This bidirectional trust is ideal for:
- Internal service-to-service communication
- API integrations with sensitive data
- Zero-trust architectures

🧰 Prerequisites

Before you begin, make sure you have:

✅ A server certificate and private key for IRIS
✅ A CA certificate to validate client certificates
✅ A client certificate and private key for the external system
✅ IRIS version 202X.X, which provides support for TLS 1.2 and higher

⚙️ Configuring IRIS for mTLS

1. IRIS as a Server (Accepting mTLS Connections)

🔸 Import Certificates

Use the System Management Portal or command line to import:
- Server certificate
- Server private key
- CA certificate (to validate clients)

🔸 Create TLS Configuration

Go to:

System Administration > Security > SSL/TLS Configurations

Create a new configuration
Enable “Require client certificate”

🔸 Assign TLS to Listener

Apply the TLS configuration to the relevant service (e.g., web server, REST endpoint).

2. IRIS as a Client (Connecting to External Systems)

This section also applies to external client systems connecting to IRIS servers.

🔸 Import Client Certificates

Import the client certificate and private key into IRIS.

🔸 Configure Outbound TLS

Use ObjectScript to set up the connection:

set http = ##class(%Net.HttpRequest).%New()
set http.SSLConfiguration = "MyClientTLSConfig"
set http.Server = "api.external-system.com"
set http.Port = 443
set status = http.Get("/endpoint")

🧪 Testing Your Certificates for mTLS

Before deploying, validate your certificates to ensure they meet mTLS requirements.

✅ 1. Check Certificate Validity

openssl x509 -in client.crt -noout -text

Look for:
- Validity dates
- Subject and Issuer fields
- Extended Key Usage (should include TLS Web Client Authentication as shown below)

X509v3 Extended Key Usage:
    TLS Web Client Authentication

✅ 2. Verify Private Key Matches Certificate

openssl x509 -noout -modulus -in client.crt | openssl md5
openssl rsa -noout -modulus -in client.key | openssl md5

The hashes should match.

✅ 3. Test mTLS Handshake with OpenSSL

openssl s_client -connect server.example.com:443   -cert client.crt -key client.key -CAfile ca.crt

This simulates a full mTLS handshake. Look for:
- Verify return code: 0 (ok)
- Successful certificate exchange

✅ 4. Validate Certificate Chain

openssl verify -CAfile ca.crt client.crt

Ensures the client certificate is trusted by the CA.

📊 mTLS Handshake Diagram

🧯 Troubleshooting Tips

🔍 Certificate chain incomplete? Ensure intermediate certs are included.
🔍 CN/SAN mismatch? Match certificate fields with expected hostnames.
🔍 Permission errors? Check file access rights on certs and keys.
🔍 Handshake failures? Enable verbose logging in IRIS and OpenSSL.

✅ Conclusion

Mutual TLS is a cornerstone of secure system integration. With IRIS, configuring mTLS is straightforward—but validating your certificates is just as important. By following these steps, you’ll ensure your connections are encrypted, authenticated, and enterprise-grade secure.