Secure NonStop OSM Communication via Custom SSL Configuration

[#1]

In my last blog, I wrote about various OSMCONF settings that you can use to customize OSM to your specific NonStop environment. In this blog, I will discuss OSMCONF settings specific to OSM client-server communication using Secure Sockets Layer (SSL). In this context, OSM client means not only OSM Service Connection and OSM Event Viewer, but also HP Systems Insight Manager (SIM), HP Insight Remote Support Advanced, HP Insight Control Power Management, or any NonStop Essentials product.

Configure SSL support

T0682 H02 AAM and later versions of OSM provide SSL support for a secure connection between OSM server and its clients. SSL is optional for OSM Service Connection and OSM Event Viewer, but required for HP SIM, HP Insight Remote Support Advanced, HP Insight Control Power Management and NonStop Essentials products.

For HP Integrity NonStop BladeSystems, SSL is always enabled, by default, for OSM client-server communication. T0682 H02 ADD and later versions of OSM have SSL enabled by default for all NonStop systems running J-Series or H-Series software. But if you are running older versions of OSM on a NonStop NS-Series server, use the following in OSMCONF to enable SSL:

UseSSL = On

Configure cipher suites used by SSL

To configure the cipher suites used by SSL for OSM client-server communication, use the following in OSMCONF:

CIPHERSUITES = value

Here, value is a comma-separated list of three different values. The default is 0.4,0.10,0.5. You can choose from the following values in any order:

• 0.4 - RSA-key-exchange + RC4-128-bit encryption and MD5 (RC4-MD5)
• 0.5 - RSA-key-exchange + RC4-128-bit encryption and SHA (RC4-SHA)
• 0.10 - RSA-key-exchange + 3-DES encryption and SHA (DES-CBC3-SHA)

Create your own private SSL certificate

SSL encryption and authentication depend on X.509 certificates to provide keys and identify the parties involved in the communication. The SSL support shipped with OSM relies on a shared SSL certificate. But for more security, you should generate and use a private SSL certificate. OpenSSL toolkits can help you generate your own SSL certificate.

Requirements for a private SSL certificate

Before you create a private SSL certificate, let’s look at the requirements for such a certificate from OSM perspective:

• The certificate’s Subject Common Name (CN) must include the DNS name or IP address configured for OSM. If you have more than one, select the most outward facing or most used address.
• A Subject Alternative Name (SAN) is required. If used, it overrides anything in the CN; so ensure to repeat the name you used in the CN, in the SAN.

The Subject Alternative Name (SAN), subjectAltName, is included in a sample config.txt file below. You can use an ASCII text editor to create the custom configuration file config.txt, which is used by OpenSSL.

[ req ]

default_bits= 512

default_md= md5

string_mask= utf8only

distinguished_name= req_DN

[ req_DN ]

C= Country Name

C_min= 2

C_max= 2

ST= State or Province Name

L= Locality Name

O= Organization Name

OU= Organization Unit Name

CN= Common Name

emailAddress = Email Address

[ verisign_CA ]

keyUsage=keyCertSign

basicConstraints= critical, CA:true , pathlen:20

subjectKeyIdentifier= hash

[ v3_req_ext_SERVER ]

subjectAltName = @alt_names_SERVER

[ alt_names_SERVER ]

DNS.1 = mynsk.mydomain.com

IP.1 = 123.1.1.23

IP.2 = 123.1.1.24

IP.3 = 123.1.1.25

IP.4 = 192.168.36.11

IP.5 = 2620:0:a05:e014:a00:111f:f001:0001

[ ca ]

default_ca= CA_default

[ CA_default ]

database= index.txt

serial= serial.txt

default_days= 365

default_crl_days= 30

default_md= md5

email_in_dn= no

name_opt= ca_default

cert_opt= ca_default

copy_extensions= copy

new_certs_dir = C:/OpenSSL-Win32/bin

[ policy_anything ]

countryName= optional

stateOrProvinceName= optional

organizationName= optional

organizationalUnitName= optional

commonName= optional

emailAddress= optional

[ verisign_SERVER ]

basicConstraints= critical, CA:false

keyUsage = keyEncipherment, dataEncipherment

extendedKeyUsage= clientAuth,serverAuth

subjectKeyIdentifier= hash

authorityKeyIdentifier= keyid:always

nsCertType= client, server

The IP addresses for the SAN field are the IP addresses that the OSM client uses to communicate with the OSM server on the NonStop system as well as any additional IP addresses available for OSM Event Viewer. For example, OSM Service Connection may use any of the IP addresses available for the default TCPIP processes ($ZTCP0 and $ZTCP1), and also IP addresses for the user-defined TCPIP processes specified in OSMCONF with stack settings.

Any IP addresses available for OSM server or OSM Event Viewer use which are not listed in the SAN field of a customer-supplied SSL certificate causes security warning dialogs for OSM Service Connection or OSM Event Viewer clients. The certificate SAN field, subjectAltName, must include both numerical IP addresses and any corresponding DNS names to be used for OSM Service Connection or OSM Event Viewer sessions.

OSM Service Connection uses not only the IP address or DNS names visible in the original URL for the web browser session, it also uses multiple IP addresses in the background for fault tolerance. All IP addresses for $ZTCP0 and $ZTCP1, as well as those specified by stack settings in the OSMCONF file, are tried in the background of the user session. If OSM Service Connection is invoked in the maintenance LAN, it uses connections on both the $ZTCP0 and $ZTCP1 TCP/IP stacks, even though only one IP address is included in the original URL of the web browser session. When OSM Service Connection is invoked from outside the maintenance LAN, the IP addresses corresponding to $ZTCP0 and $ZTCP1 are probably inaccessible, but multiple other IP addresses corresponding to OSMCONF stack settings are accessible. The certificate SAN field, subjectAltName, must include all IP addresses which could be used on any LAN, because only one SSL certificate is available for the different LANs used for OSM Service Connection.

OSM Event Viewer uses only the IP address specified in the original URL of the web browser session. It uses the same SSL certificate as OSM Service Connection. 

• Wildcards are allowed in DNS names only.
• RSA key lengths can be up to 2048 bits.
• Although MD5 hashing is supported, SHA1 is recommended as it is much more secure.

Use your own private SSL certificate

Once a private SSL certificate is generated, it needs to be placed on the NonStop system. OSM needs to know the certificate used for SSL and also the Certificate Authority (CA) certificate that signed the server’s certificate. Additionally, the server’s private key file is needed to encrypt the communications. The requirements for these files are:

• The server certificate and the CA certificate must be in binary DER format.
• The server key file must also be in binary DER format, but must also be in PKCS#8 format.
• The server certificate, the CA certificate, and the server key file must be binary code 0 files. This means, you should FTP to NonStop system in binary mode.
• The server key file must be encrypted with a password.

Now, add the following in OSMCONF:

SERVCERT = <server certificate filename>

CACERTS = <CA certificate filename>

SERVKEY = <server private key filename>

SERVKEYPASS = <key password>

On the client side, the certificate will be trusted by default if all of the following conditions are satisfied:

• The certificate is not self-signed.
• The certificate is valid (and not expired).
• The certificate is signed by a trusted CA.
• If the certificate has a SAN, the server’s address matches one of the entries in the SAN.
• If the certificate does not have a SAN, the server’s address matches the Subject CN.

If any of these conditions is not satisfied, OSM Service Connection displays a warning dialog detailing issues with the certificate, with the option to exit or continue. You also have an option to trust this certificate or this CA in the future and skip past the warning dialog.

In OSM Event Viewer, if any of these conditions is not satisfied, various different warnings are possible depending on Internet Explorer’s version and its security settings.

I highly recommend that you create your own more secure private SSL certificate for OSM client-server communication and pass your audits for this communication.