How to implement hybrid encryption with AES and asymmetric keys

Introduction

A few years ago, while working on a project that required a high level of security for data transmission, I found myself needing to implement a hybrid encryption system. This approach combines the speed of symmetric encryption with the security of asymmetric encryption, and in this post I'll share how I achieved it using AES for symmetric encryption and asymmetric keys using RSA for secure information exchange.

At that time, the approach was implemented using the Java & Spring Boot duo to achieve it, however, to share this experience I used Nest.js (An excellent framework!) as my backend, which was my preferred tool to perform the proof of concept for the challenge I had in mind.

Key Concepts

Before diving into the implementation, let's briefly review the key concepts:

• Symmetric Encryption: Uses the same key to encrypt and decrypt data. It's fast and efficient, ideal for large volumes of data, but relies on the security of the shared key.

• AES (Advanced Encryption Standard): is a widely used symmetric encryption algorithm due to its speed and security.

• Asymmetric Encryption: Uses a pair of keys (public and private). The public key encrypts the data, while the private key decrypts it. It's more secure for key exchange, but slower.

• RSA: is a commonly used asymmetric encryption algorithm for secure key exchange.

Now that we know the basic concepts, it's important to understand that in practice combining both methods is ideal to achieve the goal of security and efficiency. This is where hybrid encryption comes into play.

Hybrid Encryption Implementation

Prerequisites

1. An RSA key pair (public and private) is generated from the server.

(Optional) If you want to generate an RSA key pair, you can use openssl from the terminal. For Linux it's straightforward. However for Windows, you can install dependencies like Openssl for Windows.

After that, you can run the following commands:

To generate the private key:

1openssl genrsa -out {private-location.pem}

To generate the public key from the private key:

1openssl rsa -in {private-location.pem} -pubout -out {public-location.pem}

Encryption Flow

The hybrid encryption flow consists of the following steps which will be commented in the following code:

1class EncryptUseCase {
2  constructor(
3    /**
4     * Handles symmetric AES encryption operations.
5     */
6    private readonly aesEncryptor: AesEncryptor,
7    /**
8     * Handles asymmetric RSA encryption operations.
9     */
10    private readonly rsaKeyManager: RsaKeyManager,
11    /**
12     * Handles HMAC generation operations
13     * (Optional - Integrity validation).
14     */
15    private readonly hmacGenerator: HmacGenerator,
16    /**
17     * Handles data compression operations
18     * (Optional - Performance improvement).
19     */
20    private readonly compression: CompressionService,
21  ) {}
22
23  public encrypt(data: string): string {
24    /**
25     * 1) A unique AES key is generated for each encryption session.
26     */
27    const aesKey = this.aesEncryptor.generateKey();
28
29    /**
30     * 2) Additionally, a random initialization vector (IV) is 
31     * generated. This ensures that each encrypted message is unique.
32     */
33    const iv = this.aesEncryptor.generateIv();
34
35    /**
36     * 3) The data is encrypted using the AES key.
37     */
38    const encryptedPayload = this.aesEncryptor
39      .encrypt(data, aesKey, iv);
40
41    const encryptedIv = iv.toString('base64');
42
43    /**
44     * 4) The AES key is encrypted using the server's RSA public key.
45     */
46    const encryptedAesKey = this.rsaKeyManager
47      .encryptKey(aesKey)
48      .toString('base64');
49
50    /**
51     * 5) (Optional) HMAC generation to validate data integrity.
52     */
53    const hmacMessage = StringUtils.concat(
54      encryptedPayload, 
55      encryptedAesKey, 
56      encryptedIv
57    );
58
59    const hmac = this.hmacGenerator.generate(
60      aesKey, 
61      hmacMessage
62    );
63
64    /**
65     * 6) The client gathers both the encrypted data 
66     * and the encrypted AES key to send to the server.
67     */
68    const instance = plainToInstance(EncryptedData, {
69      d: encryptedPayload,
70      k: encryptedAesKey,
71      i: encryptedIv,
72      h: hmac,
73    });
74
75    /**
76     * 7) (Optional) Encrypted data compression.
77     * 8) Finally, the encrypted and compressed information
78     * is returned to the client.
79     */
80    return this.compression.compress(JSON.stringify(instance));
81  }
82}

If you're more visual, the following diagram illustrates the described hybrid encryption flow.

Now, as seen in the snippet, additional steps are included such as HMAC generation, which allows verifying data integrity, ensuring that data hasn't been altered during transmission, and data compression, which reduces the size of encrypted data, optimizing network performance respectively.

An important feature to highlight is that every buffer generated in the encryption process is converted to a base64 string to facilitate transmission supporting different mediums (HTTP, WebSockets, etc.).

Decryption Flow

The reverse decryption flow consists of the following steps which will be commented in the following code:

1class DecryptUseCase {
2  constructor(
3    /**
4     * Handles symmetric AES encryption operations.
5     */
6    private readonly aesEncryptor: AesEncryptor,
7    /**
8     * Handles asymmetric RSA encryption operations.
9     */
10    private readonly rsaKeyManager: RsaKeyManager,
11    /**
12     * Handles HMAC generation operations
13     * (Optional - Integrity validation).
14     */
15    private readonly hmacGenerator: HmacGenerator,
16    /**
17     * Handles data compression operations
18     * (Optional - Performance improvement).
19     */
20    private readonly compression: CompressionService,
21  ) {}
22
23  public decrypt(encryptedData: string): string {
24    /**
25     * 1) (Optional) Decompression of received data, using the inverse
26     *    strategy of the compression applied during encryption.
27     */
28    const unzipped = this.compression.decompress(encryptedData);
29
30    /**
31     * 2) The decompressed data is parsed to extract
32     *    the encrypted payload, the encrypted AES key, the IV, and the HMAC.
33     */
34    const data = JSON.parse(unzipped) as object;
35    const encryptedOutput = plainToInstance(EncryptedData, data);
36
37    /**
38     * 3) The AES key is decrypted using the server's RSA private key.
39     */
40    const encryptedAesKey = Buffer.from(encryptedOutput.k, 'base64');
41    const decryptedKey = this.rsaKeyManager
42      .decryptKey(encryptedAesKey);
43
44    /**
45     * 4) (Optional) Data integrity verification using HMAC.
46     *    If the HMAC doesn't match, it's assumed that the data has been altered,
47     *    therefore an error exception is thrown.
48     */
49    const hmacMessage = StringUtils.concat(
50      encryptedOutput.d, 
51      encryptedOutput.k, 
52      encryptedOutput.i
53    );
54
55    if (!this.hmacGenerator.verify(decryptedKey, hmacMessage, encryptedOutput.h)) {
56      throw new DecryptException('Data integrity check failed.');
57    }
58
59    const iv = Buffer.from(encryptedOutput.i, 'base64');
60
61    /**
62     * 5) Finally, the data is decrypted using the decrypted AES key and the IV.
63     */
64    return this.aesEncryptor.decrypt(encryptedOutput.d, decryptedKey, iv);
65  }
66}

If you're more visual, the following diagram illustrates the described hybrid decryption flow.

Again, optional steps for integrity verification via HMAC and data decompression are included to optimize performance.

Although the implementation is oriented towards a backend environment with Nest.js, the principles and techniques described are applicable to any environment that supports the necessary cryptographic operations.

Additional Resources

Now, if you want to dive deeper into the implementation of each of the dependencies mentioned above (AES, RSA, HMAC, Compression), in the following repository, you can find a fully functional proof of concept of this hybrid encryption approach:

Mandatory

• AESEncryptor: Contains the implementation of AES encryption/decryption and methods to generate keys and IVs.
• RSAKeyManager: Contains the implementation of RSA encryption/decryption and methods to load public and private keys.

Optional

• HMACGenerator: Contains the implementation to generate and verify HMACs.
• CompressionService: Contains the implementation to compress and decompress data using different algorithms. It's built under the Strategy pattern, allowing selection of the compression algorithm at runtime, for our practical case gzip.

Summary

If you want to implement a hybrid encryption system:

1. Generate a unique AES key for each encryption session.
2. Generate a random initialization vector (IV).
3. Encrypt the data using the AES key.
4. Encrypt the AES key using the server's RSA public key.
5. Gather both the encrypted data and the encrypted AES key for the server.
6. Return the encrypted and compressed information to the client.

If you want to implement the reverse decryption flow:

1. Parse the decompressed data to extract the encrypted payload, the encrypted AES key, and the IV.
2. Decrypt the AES key using the server's RSA private key.
3. Decrypt the data using the decrypted AES key and the IV.

Conclusion

That's how I implemented the hybrid encryption system in the project I was developing. If you have any questions or suggestions, don't hesitate to contact me through my social networks. Thanks for reading!