Enhancing the security of the SMTP connection

SMTP servers and clients communicate over the internet in plain text. Communication often occurs through one or more routers which are neither controlled nor trusted by either communicating party. Such an untrusted router may allow a third party to eavesdrop on the connection between the server and client or modify it.

In addition, two SMTP agents often need to authenticate each other. For this purpose, the client and server exchange certificates in unencrypted form when the SMTP connection is being established.

If the client does not want to show its certificate, it can prompt the server to use anonymous ciphers. In general, the encryption settings of the remote party cannot be controlled, but message delivery must be guaranteed. In such cases, relaxed settings for sending and receiving email are applied. In the KSMG web interface, the TLS encryption settings are located in the Settings → Built-in MTA → TLS Encryption section.

When receiving messages from remote servers, the connection security level is determined by the selection in the Server TLS security level drop-down list. The default setting is Attempt TLS Encryption. In this case, the client prompts the server to establish an encrypted connection with the STARTTLS command. If the server is unable to establish an encrypted connection, the connection is established without TLS encryption. A stricter setting, for example Require TLS Encryption, at this stage terminates the connection, and email messages are not delivered.

When creating a cluster, the KSMG application automatically creates a self-signed certificate. This certificate is displayed in the TLS certificate table in the Settings → Built-in MTA → TLS Encryption section with the Default Cert name; the certificate has an RSA key length of 4096 bits with a SHA-256 signature. If you are using relaxed settings, this certificate is sufficient for TLS encryption of the connection.

When sending messages to remote servers, the connection security level is determined by the selection in the Client TLS security level drop-down list. The default setting is Attempt TLS Encryption, which means that KSMG prompts the remote server to establish a connection with TLS encryption, and in case of refusal, sends the message in unencrypted form. Stricter settings, for example Require TLS Encryption and don't verify certificate, require the remote server to support TLS encryption regardless of the results of the TLS certificate verification. If Require TLS Encryption and verify certificate is selected, the server must additionally produce the matching TLS certificate. A mismatch of the settings of the remote server with the configured values results in the connection being terminated, and email messages are not delivered.

In KSMG, you can configure message sending using TLS encryption for each domain in the list. You can configure the sending of messages for an individual domain in the Settings → Built-in MTA → Domains section.

You can use the following settings to enhance the security of an SMTP connection when exchanging messages between trusted agents, for example, within the same company, with the same strict settings applied for TLS encryption of the agents, with certificates certified by certification authorities issued and uploaded, and with mail from unknown sources never accepted. To edit the settings, go to the Settings → Built-in MTA → TLS Encryption section and set the following values:

• Server TLS security level – Require TLS Encryption.
• Client TLS security level – Require TLS Encryption and don't verify certificate or Require TLS Encryption and verify certificate.

Applying strict TLS encryption settings increases the load on the server computational resources and restricts the throughput of the gateway.