How to improve SSL/TLS performance with kTLS offload

kTLS is an in-kernel offload mechanism for streamlining SSL/TLS operations. In this blog we will discuss how we can improve TLS performance with kTLS and measure the same using the well known iperf tool.

To use kTLS feature, one needs to setup kTLS ULP among other things. Most of the linux applications use OpenSSL as the prefered library, but until recently OpenSSL did not support kTLS directly. So, many developers needed to use certain hacks to get OpenSSL context information to program kTLS ULP and utilize kTLS. However, these hacks were only possible with OpenSSL 1.0 because with OpenSSL 1.1 onwards, it is not possible to access the internal OpenSSL context info. Luckily, beginning with OpenSSL 3.0.0, the support for kTLS has been added into OpenSSL mainline.

Currently, all that is required by an application to use kTLS using OpenSSL is adding the following line :

SSL_set_options(fd, SSL_OP_ENABLE_KTLS); 

However as of Ubuntu 20.04, only OpenSSL 1.1 comes as the default. So, we would need to download and build OpenSSL 3.0.0 to use kTLS feature-set. We can use the following steps to do the same :

# wget https://www.openssl.org/source/openssl-3.0.0.tar.gz
# tar -xvzf openssl-3.0.0.tar.gz
# cd openssl-3.0.0
# ./configure --prefix=/home/netlox/source/ssl --openssldir=/home/netlox/source/ssl enable-ktls '-Wl,-rpath,$(LIBRPATH)'
# make
# make install

Note — We are using a custom directory to install OpenSSL 3.0.0 so it does not overwrite the currently installed openssl version.

Next, we will build a custom iperf application to measure SSL/TLS performance with kTLS.

# git clone git@github.com:netlox-dev/iperf_ssl.git
# cd iperf_ssl
# ./autogen.sh
# ./build.sh /home/netlox/source/ssl

To run iperf, we now follow the usual method to run iperf but with some additional options:

Server Mode
# src/iperf  iperf --tls=v1.2 --ktls -s
Client Mode 
# src/iperf --tls=v1.2 --ktls -c <ip> -t 60 -i 1

We can also enable kTLS hardware offload in supported SmartNICs (e.g Mellanox Bluefield 2 ) using the following commands :

# ethtool -K <interface-name> tls-hw-tx-offload on
# ethtool -K <interface-name> tls-hw-rx-offload on

kTLS stats can also be checked :

Kernel kTLS stats :
# cat /proc/net/tls_stat
Offloaded kTLS stats :
# ethtool -S <interface-name> | grep tls

With Mellanox Bluefield 2, we were able to achieve the following performance numbers :

Conclusion — There is huge benefit (2x) in terms of TLS performance boost when we use kTLS offload in HW. However, in a cloud-native architecture, we have to carefully design our micro-services not to over-burden the platform by using SSL/TLS/mTLS across every micro-service tier. There are several factors to consider like whether use-case is for private-cloud or public-cloud and also the type of applications. There is a need to walk the fine line between having optimal security and performance all the while scaling your K8s cluster.