HTTPS Refresher (5) —— Key Calculation in TLS

Protocol , HTTPS
,

In this article, we’ll compare key derivation in TLS 1.2 and TLS 1.3.

1. Keys in TLS 1.2

In TLS 1.2, there are three types of keys: the pre-master secret, the master secret, and the session keys (key block). These keys are all related to one another.

         struct {
             uint32 gmt_unix_time;
             opaque random_bytes[28];
         } Random;
         
        struct {
             ProtocolVersion client_version;
             opaque random[46];
         } PreMasterSecret;  

        struct {
             uint8 major;
             uint8 minor;
         } ProtocolVersion;

For the RSA handshake key exchange algorithm, the client generates a 48-byte pre-master secret, where the first 2 bytes are the ProtocolVersion and the remaining 46 bytes are random data. It is encrypted with the server’s public key and sent to the server in the Client Key Exchange submessage; the server decrypts it with its private key. For (EC)DHE, the pre-master secret is generated by both parties using an elliptic-curve algorithm. Each side generates an ephemeral public/private key pair, keeps the private key, and sends the public key to the peer. Each side can then use its own private key and the peer’s public key to derive the pre-master secret via the elliptic-curve algorithm. The length of the pre-master secret depends on the DH/ECDH public key. The pre-master secret is 48 bytes or X bytes long.

The master secret is generated from the pre-master secret, the ClientHello random, and the ServerHello random via the PRF function. The master secret is 48 bytes long. As you can see, as long as we know the pre-master secret or the master secret, we can decrypt captured packet data. Therefore, in TLS 1.2, decrypting packet captures for debugging requires only the master secret. SSLKEYLOG exports the master secret; after importing it into Wireshark, you can decrypt the corresponding captured packet data.

The session keys (key block) are generated from the master secret, SecurityParameters.server_random, and SecurityParameters.client_random via the PRF function. The session keys contain the symmetric encryption keys, message authentication keys, and the initialization vectors for CBC mode. For non-CBC-mode cipher algorithms, this initialization vector is not used.

What is stored in the Session ID cache and Session Ticket is also the master secret, not the session keys. This way, whenever a session is resumed, the session keys are derived again from the two parties’ random values and the master secret, ensuring that each encrypted communication session uses different session keys. Even if the master secret of one session is leaked or cracked, it will not affect another session.

2. HMAC and Pseudorandom Functions in TLS 1.2

The TLS record layer uses a keyed Message Authentication Code (MAC) to protect message integrity. Cipher suites use a MAC algorithm called HMAC (described in [HMAC]), which is based on a hash function. If necessary, other cipher suites may define their own MAC algorithms.

In addition, for key generation or validation, a MAC algorithm is needed to expand data blocks to increase confidentiality. This pseudorandom function (PRF) takes secret information (secret), a seed, and an identity label as input, and produces output of arbitrary length.

In TLS 1.2, a PRF function is defined based on HMAC. This PRF function, which uses the SHA-256 hash function, is used for all cipher suites. New cipher suites must explicitly specify a PRF, and should typically use SHA-256 or a stronger standard hash algorithm together with the TLS PRF.

First, we define a data expansion function, P_hash(secret, data), which uses a hash function to expand a secret and seed into output of arbitrary size:

      P_hash(secret, seed) = HMAC_hash(secret, A(1) + seed) +
                             HMAC_hash(secret, A(2) + seed) +
                             HMAC_hash(secret, A(3) + seed) + ...

Here, “+” means concatenation.

A() is defined as:

              A(0) = seed
              A(i) = HMAC_hash(secret, A(i-1))

When necessary, P_hash can be iterated multiple times to produce the required amount of data. For example, if P_SHA256 is used to generate 80 bytes of data, it should be iterated 3 times (through A(3)). SHA_256 outputs 32 bytes (256 bits) each time, so 3 iterations are needed to produce 96 bytes of output data. The final 16 bytes produced by the last iteration are discarded, leaving 80 bytes as the output data.

The TLS PRF can be implemented by applying P_hash to the secret:

             PRF(secret, label, seed) = P_<hash>(secret, label + seed)

label is an ASCII string. It should be processed exactly as given, without a length byte or a terminating null character. For example, the label “slithy toves” should be processed by hashing the following bytes:

             73 6C 69 74 68 79 20 74 6F 76 65 73

The data above is the hexadecimal representation of the string “slithy toves”.

The hash algorithm used by the PRF depends on the cipher suite and the TLS version, as shown below:

PRF AlgorithmHash Algorithm
prf_tls10For TLS 1.0 and TLS 1.1, the PRF algorithm combines the MD5 and SHA_1 algorithms
prf_tls12_sha256For TLS 1.2, the default is the SHA_256 algorithm (the algorithm that meets the minimum security requirements)
prf_tls12_sha384For TLS 1.2, if the HMAC algorithm specified by the cipher suite has a higher security level than SHA_256, the SHA_384 cryptographic primitive is used

In TLS 1.0 and TLS 1.1, P_HASH is invoked twice: once with MD5 and once with SHA1. The results of the two invocations are XORed to produce the final result.

r1 = P_MD5(...);
r2 = P_SHA1(...);
r  = r1 xor r2

In TLS 1.2, the PRF algorithm is essentially a direct invocation of the P_HASH algorithm, with SHA_256 as the default.

III. Key Calculation in TLS 1.2

The keying algorithm in TLS 1.2 is mainly the PRF discussed in the previous chapter. The PRF is primarily used to derive the master secret and the session keys (key block).

1. Computing the Master Secret

To begin protecting the connection, the TLS record protocol requires that a cipher suite, a master secret, and random values from both the Client and Server be specified. The authentication, encryption, and message authentication code algorithms are determined by cipher_suite, which is selected by the Server and indicated in the ServerHello message. The compression algorithm is negotiated in the hello messages, and the random values are also exchanged in the hello messages. All of these are used to compute the master secret.

For all key exchange algorithms, the same algorithm is used to transform the pre_master_secret into the master_secret. Once the master_secret has been computed, the pre_master_secret should be deleted from memory. This prevents an attacker from obtaining the premaster secret. If an attacker obtains the premaster secret, then because ClientHello.random and ServerHello.random are transmitted unencrypted and are therefore also easy to obtain, the attacker can synthesize the master secret and further derive the session keys, completely breaking the entire encryption process.

        master_secret = PRF(pre_master_secret, "master secret",
                            ClientHello.random + ServerHello.random)
                            [0..47];

The master secret is always 48 bytes long. The length of the pre-master secret varies depending on the key exchange algorithm.

RSA

When RSA is used for authentication and key exchange, the Client generates a 48-byte pre_master_secret, encrypts it with the Server’s public key, and sends it to the Server. The Server decrypts the pre_master_secret using its own private key. Both parties then convert the pre_master_secret into the master_secret using the method described above.

        struct {
             ProtocolVersion client_version;
             opaque random[46];
         } PreMasterSecret;

Diffie-Hellman

A conventional Diffie-Hellman computation must be performed. The negotiated key (Z) is used as the pre_master_secret and converted into the master_secret using the method described earlier. Before being used as the pre_master_secret, all leading zero bits in Z are compressed.

Note: The Diffie-Hellman parameters are specified by the Server; they may be ephemeral or included in the Server’s certificate.

2. Computing the Extended Master Secret

In the previous article, we saw that the ClientHello extensions carried the extended_master_secret extension, which indicates that the Client and Server use the extended master secret computation method.

The Server responds to this extension in ServerHello by returning an empty extended_master_secret extension, indicating that the extended master secret computation method will be used.

So how is the extended master secret computed? The computation is as follows:

        master_secret = PRF(pre_master_secret, "extended master secret",
                            session_hash)
                            [0..47];

The calculation above differs from the standard way of computing the master secret in the following ways:

  • “extended master secret” replaces “master secret”
  • session_hash replaces ClientHello.random + ServerHello.random

In addition to the cipher suite from the Client and Server, the key exchange information, and the certificate (if any), “session_hash” also depends on the handshake transcript, which includes “ClientHello.random” and “ServerHello.random”. Therefore, the extended master secret depends on the selection of all these session parameters.

This design reflects the recommendation that keys should be bound to the security context in which they are computed SP800-108. The technique of mixing the hash of the key exchange messages into master secret derivation is already used in other well-known protocols, such as Secure Shell (SSH) RFC4251. The Client and Server should not accept handshakes that do not use the extended master secret, especially if they rely on features such as compound authentication.

Readers interested in this attack can refer to this article: Triple Handshake Preconditions and Impact

3. Computing Session Keys

Session keys (the key block) are used for encryption in the TLS record layer. The record protocol requires an algorithm to generate the keys needed for the current connection state from the security parameters provided by the handshake protocol.

   enum { null(0), (255) } CompressionMethod;

   enum { server, client } ConnectionEnd;

   enum { tls_prf_sha256 } PRFAlgorithm;

   enum { null, rc4, 3des, aes } BulkCipherAlgorithm;

   enum { stream, block, aead } CipherType;

   enum { null, hmac_md5, hmac_sha1, hmac_sha256, hmac_sha384,
     hmac_sha512} MACAlgorithm;

   /* Other values may be added to the algorithms specified in
   CompressionMethod, PRFAlgorithm, BulkCipherAlgorithm, and
   MACAlgorithm. */

   struct {
       ConnectionEnd          entity;
       PRFAlgorithm           prf_algorithm;
       BulkCipherAlgorithm    bulk_cipher_algorithm;
       CipherType             cipher_type;
       uint8                  enc_key_length;
       uint8                  block_length;
       uint8                  fixed_iv_length;
       uint8                  record_iv_length;
       MACAlgorithm           mac_algorithm;
       uint8                  mac_length;
       uint8                  mac_key_length;
       CompressionMethod      compression_algorithm;
       opaque                 master_secret[48];
       opaque                 client_random[32];
       opaque                 server_random[32];
   } SecurityParameters;

The master secret is expanded into a secure byte sequence, which is split into a client_write_MAC_key, a server_write_MAC_key, a client_write_key, and a server_write_key. Each of these is generated from the byte sequence in the order described above. Unused values are empty. Some AEAD ciphers may additionally require a client_write_IV and a server_write_IV. When generating encryption keys and MAC keys, the master secret is used as a source of entropy. Therefore, the length and number of session keys (the key block) depend on the negotiated cipher suite; more precisely, they depend on the encryption parameters SecurityParameters. The PRF function must be used to expand a key block of sufficient length, computed as follows:

            key_block = PRF(SecurityParameters.master_secret,
                      "key expansion",
                      SecurityParameters.server_random +
                      SecurityParameters.client_random);

Note: The three inputs to the PRF used to compute the session keys and the master secret are all different: PRF(secret, label, seed). For the master secret, they are (pre_master_secret, "master secret", ClientHello.random + ServerHello.random); for the session keys, they are (SecurityParameters.master_secret, "key expansion", SecurityParameters.server_random + SecurityParameters.client_random). The seed order changes: the order in which the Client and Server random values are combined is swapped.

This continues until enough output has been produced. Then, the key_block is split as follows:

      client_write_MAC_key[SecurityParameters.mac_key_length]
      server_write_MAC_key[SecurityParameters.mac_key_length]
      client_write_key[SecurityParameters.enc_key_length]
      server_write_key[SecurityParameters.enc_key_length]
      client_write_IV[SecurityParameters.fixed_iv_length]
      server_write_IV[SecurityParameters.fixed_iv_length]

client_write_key, server_write_key, client_write_MAC_key, and server_write_MAC_key are the keys required for encryption and message authentication codes. The Client and Server each have their own set of keys, and the keys they use are different. If a block cipher mode is used, initialization vectors client_write_IV and server_write_IV are also required. If AEAD mode is used, client_write_MAC_key and server_write_MAC_key may not be needed; client_write_IV and server_write_IV are used as the nonce (random value).

At present, client_write_IV and server_write_IV can only be generated by AEAD’s implicit nonce technique.

Among the currently defined cipher suites, AES_256_CBC_SHA256 is the most widely used. It requires 2 x 32-byte keys and 2 x 32-byte MAC keys, which are generated from 128 bytes of keying material.

The TLS 1.2 key calculation process is summarized as follows:

IV. TLS 1.2 Finished Verification

At the end of the TLS 1.2 handshake, a Finished submessage is sent. This message is the first encrypted message, and the recipient of the Finished message must verify that its contents are correct. The content to be verified is computed using the PRF algorithm.

      verify_data = PRF(master_secret, 
      					finished_label, 
      					Hash(handshake_messages))
            			[0..verify_data_length-1];

When computing verify_data, the secret in PRF(secret, label, seed) is the master secret, the label is the finished_label: for the Client it is “client finished”, and for the Server it is “server finished”. The seed is the hash value of all handshake messages. For the Client, the contents of handshake_messages include all messages sent and received, but do not include the Finished message sent by the Client itself. For the Server, the contents of handshake_messages include all messages from the ClientHello message up to, but not including, the Server’s Finished message, and also include the Client’s Finished submessage.

handshake_messages contains only handshake submessages; it does not include ChangeCipherSpec submessages, Alert submessages, or HelloRequest messages.

In early TLS protocols, the length of verify_data is 12 bytes. For TLS 1.2, the length of verify_data depends on the cipher suite. If the cipher suite does not specify verify_data_length, the default length is also 12 bytes.

5. Keyless TLS 1.2

If a CDN vendor wants to support HTTPS, what changes are required? The approach used by domestic vendors is to upload the private key of their HTTPS website to a server provided by the CDN vendor. Some customers with very high security requirements (for example, banks) want to use a third-party CDN to accelerate access to their websites, but for security reasons they cannot hand over their private keys to the CDN provider. If you have understood the TLS key calculation method described above, there is absolutely no need to upload the private key to a third-party CDN server. CloudFlare offered a Keyless service long ago: you can host your site on their CDN and use TLS/SSL-encrypted connections without providing the private key for your certificate.

During the handshake phase, the main result is the negotiation of three random values. These three random values produce the session keys (key block) required by the TLS record layer. After the handshake completes, all subsequent encryption is symmetric encryption. The private key in asymmetric cryptography is needed only once. If RSA key negotiation is used, the role of the private key is to decrypt the premaster secret sent by the Client. The public key in asymmetric cryptography is used to encrypt the key negotiation parameters sent to the Client. However, the Server’s public key can be obtained from the certificate. Therefore, the only problem the CDN cannot solve is decrypting the premaster secret sent by the Client. If ECDHE key negotiation is used, the role of the private key is to sign the DH parameters.

The solution is relatively simple:

If RSA key negotiation is used, when the CDN vendor’s server receives the premaster secret sent by the Client, it sends this encrypted premaster secret to the user’s own key server. The user decrypts the premaster secret with their own private key and sends it back to the CDN vendor’s server. In this way, the CDN vendor obtains the decrypted premaster secret and can then continue computing the master secret and the session keys (key block). The flow is as follows:

If a DH key negotiation algorithm is used, the premaster secret can be computed jointly by the Server and the Client, but the DH-related parameters need to be negotiated by both sides. When the Server sends the DH-related parameters to the Client, it needs to use the private key corresponding to the certificate. The CDN vendor sends the hash of the Client random, the Server random, and the DH parameters to the user’s key server. The key server signs them and sends the signature back to the CDN vendor’s server. The CDN vendor then sends the signed message to the Client. This completes key negotiation. The CDN and the Client compute the premaster secret, master secret, and session keys. The flow is as follows:

6. Keys in TLS 1.3

In TLS 1.3, the PRF algorithm is no longer used; instead, the more standard HKDF algorithm is used for key derivation. TLS 1.3 also applies finer-grained optimization to keys: encryption in each phase or direction does not use the same key. In TLS 1.3, all data after the ServerHello message is encrypted. Messages sent by the Server to the Client during the handshake are encrypted with keys derived from server_handshake_traffic_secret via HKDF, while handshake messages sent by the Client to the Server are encrypted with keys derived from client_handshake_traffic_secret via HKDF. These two keys are derived from the Handshake Secret, which in turn is derived from the PreMasterSecret and Early Secret. The Handshake Secret is then used to derive the master secret, Master Secret.

The Master Secret is then used to derive the following keys:
client_application_traffic_secret: used to derive the symmetric encryption key for application data sent by the client to the server.
server_application_traffic_secret: used to derive the symmetric encryption key for application data sent by the server to the client.
resumption_master_secret: used to generate the PSK.

Finally, the four keys server_handshake_traffic_secret, client_handshake_traffic_secret, client_application_traffic_secret, and server_application_traffic_secret each generate one set of write_key and write_IV for symmetric encryption, for a total of four sets.

If early_data is used, client_early_traffic_secret is also required; it generates one set of write_key and write_IV for encrypting and decrypting 0-RTT data.

7. HMAC and Pseudorandom Functions in TLS 1.3

A Key Derivation Function (KDF) is an essential component in cryptographic systems. Its purpose is to expand one key into multiple keys that are cryptographically secure. TLS 1.3 uses the HMAC-based Extract-and-Expand Key Derivation Function (HKDF). HKDF follows the extract-then-expand design pattern, meaning that the KDF has two main modules. In the first phase, the input key material is “extracted” to obtain a fixed-length key. In the second phase, this key is “expanded” into multiple additional pseudorandom keys. The length and number of output keys depend on the specified encryption algorithm. Because the extract process is not mandatory, the expand process can be used independently.

HMAC has two parameters: the first is key, and the second is data. data can consist of several elements, which we generally denote with |, for example:

   HMAC(K, elem1 | elem2 | elem3)

1. Extract

   HKDF-Extract(salt, IKM) -> PRK

  • Variables:
    Hash is the hash function; HashLen denotes the output size, in bytes, of this hash function.

  • Input:
    salt is an optional value. If it is not specified, HashLen zeros are used instead.
    IKM is the input keying material; IKM stands for Input Keying Material.

  • Output:
    PRK is a pseudorandom key (HashLen bytes in size); PSK stands for PseudoRandom Key.

PRK is computed as follows:

   PRK = HMAC-Hash(salt, IKM)

The definition of HKDF allows operation both with a random salt value and without a salt value. This is intended for compatibility with applications that do not have a salt. However, using a salt is strongly recommended, as it can significantly strengthen the HKDF algorithm. It also ensures independence among different uses of the hash function, supports “source-independent” extraction, and strengthens the analytical results underpinning HKDF’s design.

A random salt differs fundamentally from the initial keying material (IKM) in two ways: the random salt is non-secret and can be reused. Therefore, a random salt value can be used in many applications. For example, a pseudorandom number generator (PRNG) that continuously produces output by applying HKDF to a renewable entropy pool (e.g., sampling system events) can determine a salt value and use it across multiple applications of HKDF without needing to protect the secrecy of the salt. In a different application domain, a key-agreement protocol that derives cryptographic keys from a Diffie-Hellman exchange can obtain the salt value from public nonces exchanged and authenticated between the communicating parties, and make this practice part of the key agreement protocol (this is the approach used in IKEv2).

Ideally, the salt value is a random (or pseudorandom) string of length HashLen. However, even lower-quality salt values (shorter sizes or limited entropy) may still make a significant contribution to the security of the output keying material; therefore, if such values are available to an application, application designers are encouraged to provide salt values to HKDF.

It is worth noting that, although this is not the typical case, some applications may even have secret salt values available for use. In such cases, HKDF provides stronger security guarantees. One example of such an application is IKEv1 in its “public-key encryption mode”, where the extractor’s salt is computed from encrypted nonces. Similarly, IKEv1’s pre-shared mode uses a secret salt derived from the pre-shared key.

2. Expand

   HKDF-Expand(PRK, info, L) -> OKM

  • Variables:
    Hash is a hash function; HashLen denotes the number of output bytes of this hash function.

  • Inputs:
    PRK is a pseudorandom key of at least HashLen bytes (typically derived by the extract step).
    info is an optional value and may be “”.
    L is the desired number of output bytes (length <= 255 * HashLen).

  • Output:
    OKM is the output keying material (L bytes); OKM is short for Output Keying Material.

OKM is computed as follows:

   N = ceil(L/HashLen)
   T = T(1) | T(2) | T(3) | ... | T(N)
   OKM = first L octets of T

   where:
   T(0) = empty string (zero length)
   T(1) = HMAC-Hash(PRK, T(0) | info | 0x01)
   T(2) = HMAC-Hash(PRK, T(1) | info | 0x02)
   T(3) = HMAC-Hash(PRK, T(2) | info | 0x03)
   ...

Although the info value is optional in the HKDF definition, it is often very important in applications. Its primary purpose is to bind the derived keying material to application- and context-specific information. For example, info can include a protocol number, algorithm identifier, user identity, and so on. In particular, it can prevent the same keying material from being derived for different contexts (when the same input keying material (IKM) is used in different settings). If needed, it can also carry additional input for the key-expansion phase (for example, an application might want to bind the keying material to its length L, thereby extending the info field to length L). info has one technical requirement: it should be independent of the value of the input keying material, IKM.

Compare this with the PRF computation method in TLS 1.2:

   PRF(secret, label, seed) = P_<hash>(secret, label + seed)
   P_hash(secret, seed) = HMAC_hash(secret, A(1) + seed) +
                          HMAC_hash(secret, A(2) + seed) +
                          HMAC_hash(secret, A(3) + seed) + ...
                             
   where:                            
   A(0) = seed
   A(i) = HMAC_hash(secret, A(i-1))
   ...

The difference between these two algorithms can be seen.

In some applications, the input keying material (IKM) may already exist as a cryptographically strong key (for example, the premaster secret in a TLS RSA cipher suite is a pseudorandom string, except for the first two bytes). In this case, the extract step can be skipped and the IKM can be used directly as the input to HMAC in the expand step. On the other hand, for compatibility with the general case, applications may still use the extract step. In particular, if the IKM is random (or pseudorandom) but longer than the HMAC key, the extract step can be used to output a suitable HMAC key (in the case of HMAC, shortening via the extractor is not strictly necessary, because HMAC also works with keys up to a certain length). However, note that if the IKM is a Diffie-Hellman value, as in TLS when Diffie-Hellman is used, the extract step should not be skipped. Doing so would result in using the Diffie-Hellman value g ^ {xy} itself (which is not a uniformly random or pseudorandom string) as the key to HMAC, namely the PRK. Instead, HKDF should first apply the extract step to g ^ {xy} (preferably with a salt value), and use the resulting PRK as the key for the HMAC expansion part.

When the required number of key bits L is no greater than HashLen, the PRK can be used directly as the OKM. However, this is not recommended, especially because it omits the use of info as part of the derivation process (and adding info as an input to the extract step is not recommended—see HKDF-paper).

The TLS 1.3 key schedule uses the HKDF-Extract and HKDF-Expand functions defined by HMAC-based Extract-and-Expand Key Derivation Function (HKDF) [RFC5869], as well as the functions defined below:

       HKDF-Expand-Label(Secret, Label, Context, Length) =
            HKDF-Expand(Secret, HkdfLabel, Length)

       Where HkdfLabel is specified as:

       struct {
           uint16 length = Length;
           opaque label<7..255> = "tls13 " + Label;
           opaque context<0..255> = Context;
       } HkdfLabel;

       Derive-Secret(Secret, Label, Messages) =
            HKDF-Expand-Label(Secret, Label,
                              Transcript-Hash(Messages), Hash.length)

The Hash function used by Transcript-Hash and HKDF is the cipher suite’s hash algorithm. Hash.length is its output length (in bytes). The messages are the concatenation of the represented handshake messages, including the handshake message type and length fields, but excluding the record layer header. Note that in some cases, a zero-length context (represented by “") is passed to HKDF-Expand-Label. Labels are ASCII strings and do not include a trailing NUL byte.

From the function call relationships above, we can draw the following conclusions:

		Derive-Secret(Secret, Label, Messages) = 
			  HKDF-Expand(Secret, HkdfLabel, Length)

HKDF-Extract(salt, IKM) is the Extract step of HKDF in TLS 1.3; Derive-Secret(Secret, Label, Messages) is the Expand step of HKDF in TLS 1.3.

3. Transcript-Hash

Finally, let’s discuss the Transcript-Hash function. Many cryptographic computations in TLS use a transcript hash. This value is computed by hashing the concatenation of each included handshake message. It includes the handshake message type and length fields carried in the handshake message header, but excludes the record-layer header. For example:

   Transcript-Hash(M1, M2, ... Mn) = Hash(M1 || M2 || ... || Mn)

As an exception to this general rule, when the Server responds to a ClientHello message with a HelloRetryRequest message, the value of ClientHello1 is replaced with a special synthetic handshake message whose handshake type is “message_hash” and that contains Hash(ClientHello1). For example:

   Transcript-Hash(ClientHello1, HelloRetryRequest, ... Mn) =
      Hash(message_hash ||        /* Handshake type */
           00 00 Hash.length  ||  /* Handshake message length (bytes) */
           Hash(ClientHello1) ||  /* Hash of ClientHello1 */
           HelloRetryRequest  || ... || Mn)

The reason for designing this structure is to allow the Server to perform a stateless HelloRetryRequest by storing only the hash of ClientHello1 in the cookie, rather than requiring it to export the entire intermediate hash state.

Specifically, the hash copy is always taken over the following sequence of handshake messages, starting with the first ClientHello and including only messages that have been sent: ClientHello, HelloRetryRequest, ClientHello, ServerHello, EncryptedExtensions, server CertificateRequest, server Certificate, server CertificateVerify, server Finished, EndOfEarlyData, client Certificate, client CertificateVerify, client Finished.

In general, implementations can maintain the hash copy as follows: keep a running hash copy based on the negotiated hash. Note that subsequent post-handshake authentication messages do not include each other; they only include the messages up to the end of the main handshake.

8. Key Computation in TLS 1.3

The randomness of the keying material derived from key negotiation may be insufficient, and the negotiation process may be observable by an attacker. A key derivation function is therefore needed to derive stronger keys from the initial keying material (a PSK or a key computed from DH key agreement). HKDF is exactly the algorithm used for this purpose in TLS 1.3. It takes the negotiated keying material and the hash of the handshake messages as input, and outputs new keys with stronger security properties.

From the previous chapter, we know that HKDF consists of a two-stage extract_then_expand process. The extract stage increases the randomness of the keying material. The key derivation function PRF used in TLS 1.2 actually implements only the expand portion of HKDF; it does not perform extract, and instead directly assumes that the randomness of the keying material already meets the requirements.

In this chapter, let’s look at how TLS 1.3 performs extract_then_expand on keying material. This chapter also shows why TLS 1.3 raises the security bar beyond TLS 1.2.

All keys in TLS 1.3 are derived jointly by HKDF-Extract(salt, IKM) and Derive-Secret(Secret, Label, Messages). Here, Salt is the current secret state, and the input keying material (IKM) is the new secret to be added. In TLS 1.3, the two IKM inputs are:

  • PSK (an externally established pre-shared key, or one derived from the resumption_master_secret value of a previous connection)
  • (EC)DHE shared secret

The complete key derivation flow in TLS 1.3 is as follows:

             0
             |
             v
   PSK ->  HKDF-Extract = Early Secret
             |
             +-----> Derive-Secret(., "ext binder" | "res binder", "")
             |                     = binder_key
             |
             +-----> Derive-Secret(., "c e traffic", ClientHello)
             |                     = client_early_traffic_secret
             |
             +-----> Derive-Secret(., "e exp master", ClientHello)
             |                     = early_exporter_master_secret
             v
       Derive-Secret(., "derived", "")
             |
             v
   (EC)DHE -> HKDF-Extract = Handshake Secret
             |
             +-----> Derive-Secret(., "c hs traffic",
             |                     ClientHello...ServerHello)
             |                     = client_handshake_traffic_secret
             |
             +-----> Derive-Secret(., "s hs traffic",
             |                     ClientHello...ServerHello)
             |                     = server_handshake_traffic_secret
             v
       Derive-Secret(., "derived", "")
             |
             v
   0 -> HKDF-Extract = Master Secret
             |
             +-----> Derive-Secret(., "c ap traffic",
             |                     ClientHello...server Finished)
             |                     = client_application_traffic_secret_0
             |
             +-----> Derive-Secret(., "s ap traffic",
             |                     ClientHello...server Finished)
             |                     = server_application_traffic_secret_0
             |
             +-----> Derive-Secret(., "exp master",
             |                     ClientHello...server Finished)
             |                     = exporter_master_secret
             |
             +-----> Derive-Secret(., "res master",
                                   ClientHello...client Finished)
                                   = resumption_master_secret

A few notes:

  1. HKDF-Extract is shown in the diagram as taking the Salt parameter from the top and the IKM parameter from the left; its output is shown at the bottom and with the name on the right.
  2. The Secret parameter of Derive-Secret is indicated by the incoming arrow. For example, Early Secret is the Secret used to generate client_early_traffic_secret.
  3. “0” means a string of Hash.length bytes set to zero.

If the given secret is not available, a 0 value consisting of a Hash.length-byte string set to zero is used. Note that this does not mean skipping the round; therefore, if PSK is not used, Early Secret is still HKDF-Extract(0,0). For the computation of binder_key, the label is “ext binder” for external PSKs (those provided outside TLS) and “res binder” for resumption PSKs (those provided as the resumption master secret from a previous handshake). Different labels prevent one type of PSK from being substituted for the other.

This means there are multiple potential Early Secret values, depending on which PSK the server ultimately selects. The client needs to compute a value for each potential PSK; if no PSK is selected, it needs to compute the Early Secret corresponding to the zero PSK.

Once all values derived from a given secret have been computed, that secret should be deleted.

TLS 1.3 involves three Secret computation methods, as follows:

        Early Secret = HKDF-Extract(salt, IKM) = HKDF-Extract(0, PSK)
    Handshake Secret = HKDF-Extract(salt, IKM) = HKDF-Extract(Derive-Secret(Early Secret, "derived", ""), (EC)DHE)
       Master Secret = HKDF-Extract(salt, IKM) = HKDF-Extract(Derive-Secret(Handshake Secret, "derived", ""), 0)

TLS 1.3 involves eight key computation methods, as follows:

        client_early_traffic_secret = Derive-Secret(Early Secret, "c e traffic", ClientHello)
       early_exporter_master_secret = Derive-Secret(Early Secret, "e exp master", ClientHello)
       
    client_handshake_traffic_secret = Derive-Secret(Handshake Secret, "c hs traffic", ClientHello...ServerHello)
    server_handshake_traffic_secret = Derive-Secret(Handshake Secret, "s hs traffic", ClientHello...ServerHello)
    
client_application_traffic_secret_0 = Derive-Secret(Master Secret, "c ap traffic", ClientHello...server Finished)
server_application_traffic_secret_0 = Derive-Secret(Master Secret, "s ap traffic", ClientHello...server Finished)
			 exporter_master_secret = Derive-Secret(Master Secret, "exp master", ClientHello...server Finished)
		   resumption_master_secret = Derive-Secret(Master Secret, "res master", ClientHello...client Finished)

For example:

CLIENT_EARLY_TRAFFIC_SECRET edb6c73462794c0fe79296853fd17b06cd30e63e87e69c8864eba6996e5d9434 5a0d40c3afa57cbb5aa427456f8dc21b9c4c17bfb731600f93e35358f5b581cb
EARLY_EXPORTER_SECRET edb6c73462794c0fe79296853fd17b06cd30e63e87e69c8864eba6996e5d9434 274e61024f88d0952898889a54211200a76456434d8e546cd6450f8313412df5
CLIENT_HANDSHAKE_TRAFFIC_SECRET edb6c73462794c0fe79296853fd17b06cd30e63e87e69c8864eba6996e5d9434 c041776dc29543e87e3442111be79f289062eef7603ec566f28f5b05b15c9718
SERVER_HANDSHAKE_TRAFFIC_SECRET edb6c73462794c0fe79296853fd17b06cd30e63e87e69c8864eba6996e5d9434 68e19a5d69dfdf8ca701a370cfd7c21e98b1bd933c03ee9dd72738e60147e8db
CLIENT_TRAFFIC_SECRET_0 edb6c73462794c0fe79296853fd17b06cd30e63e87e69c8864eba6996e5d9434 b866b25bc12f5272dbc6d27471edce47d04f496362b56800d5f95e0760d044ee
SERVER_TRAFFIC_SECRET_0 edb6c73462794c0fe79296853fd17b06cd30e63e87e69c8864eba6996e5d9434 8f07b32b6191019bac664d5071dd961e92ff2060db629d4e3eb3689a43cc71d3
EXPORTER_SECRET edb6c73462794c0fe79296853fd17b06cd30e63e87e69c8864eba6996e5d9434 c7a1fb9092f245a8b92cd7a481eb0bd6d255b4d06c6d05096ef8a8bf3face22e

EXPORTER_SECRET is an exporter secret, used for other user-defined purposes.

Of the eight secrets derived above, excluding the two exporter secrets required for user-defined purposes and the resumption_master_secret used for session resumption, the remaining five secrets have already gone through one HKDF Expand step. However, these five secrets are still only “intermediate variables”; deriving the final encryption parameters requires one more Expand step:

  [sender]_write_key = HKDF-Expand-Label(Secret, "key", "", key_length)
  [sender]_write_iv  = HKDF-Expand-Label(Secret, "iv", "", iv_length)

[sender] indicates the sender. The Secret value for each record type is shown in the table below:

       +-------------------+---------------------------------------+
       | Record Type       | Secret                                |
       +-------------------+---------------------------------------+
       | 0-RTT Application | client_early_traffic_secret           |
       |                   |                                       |
       | Handshake         | [sender]_handshake_traffic_secret     |
       |                   |                                       |
       | Application Data  | [sender]_application_traffic_secret_N |
       +-------------------+---------------------------------------+

Whenever the underlying Secret changes (for example, when switching from the handshake to application data keys, or during a key update), all traffic key material is recomputed.

The resumption_master_secret key is used to derive the PSK for session resumption, and is computed as follows:

     PskIdentity.identity = ticket 
                          = HKDF-Expand-Label(resumption_master_secret, "resumption", ticket_nonce, Hash.length)

The Server sends the ticket to the Client in NewSessionTicket, and the Client uses the ticket to generate PskIdentity. It then computes PskBinderEntry:

    PskBinderEntry = HMAC(binder_key, Transcript-Hash(Truncate(ClientHello1)))
                   = HMAC(Derive-Secret(HKDF-Extract(0, PSK), "ext binder" | "res binder", ""), Transcript-Hash(Truncate(ClientHello1)))
                   
where     binder_key = Derive-Secret(HKDF-Extract(0, PSK), "ext binder" | "res binder", "")

The client combines PskIdentity and PskBinderEntry into a PSK, and sends the PSK to the server as a ClientHello extension when session resumption is needed. The PSK serves as the input keying material (IKM) for the Early Secret.

        Early Secret = HKDF-Extract(salt, IKM) = HKDF-Extract(0, PSK)
        client_early_traffic_secret = Derive-Secret(Early Secret, "c e traffic", ClientHello)

The write_key and write_iv generated from client_early_traffic_secret are ultimately used for 0-RTT encryption and decryption.

The TLS 1.3 0-RTT key calculation process is as follows:

IX. TLS 1.3 Finished Verification

In TLS 1.3, Finished is not the first encrypted message in the entire handshake, but its role is the same as in TLS 1.2: it is critical for authenticating the handshake and the key calculation.

In TLS 1.3, the computation of Authentication messages uniformly uses the following inputs:

  • The certificate and signing key to use
  • The handshake context, consisting of a set of messages in the hash transcript
  • The Base key used to compute the MAC key

The Finished submessage contains a MAC computed from the value of Transcript-Hash(Handshake Context, Certificate, CertificateVerify). The MAC value is computed using the MAC key derived from the Base key.

For each scenario, the following table defines the handshake context and the MAC Base Key.

   +-----------+-------------------------+-----------------------------+
   | Mode      | Handshake Context       | Base Key                    |
   +-----------+-------------------------+-----------------------------+
   | Server    | ClientHello ... later   | server_handshake_traffic_   |
   |           | of EncryptedExtensions/ | secret                      |
   |           | CertificateRequest      |                             |
   |           |                         |                             |
   | Client    | ClientHello ... later   | client_handshake_traffic_   |
   |           | of server               | secret                      |
   |           | Finished/EndOfEarlyData |                             |
   |           |                         |                             |
   | Post-     | ClientHello ... client  | client_application_traffic_ |
   | Handshake | Finished +              | secret_N                    |
   |           | CertificateRequest      |                             |
   +-----------+-------------------------+-----------------------------+

The key used to compute the Finished message is derived using HKDF, with the Base Key being server_handshake_traffic_secret and client_handshake_traffic_secret. Specifically:

   finished_key =
       HKDF-Expand-Label(BaseKey, "finished", "", Hash.length)

The data structure of this message is:

      struct {
          opaque verify_data[Hash.length];
      } Finished;

verify_data is calculated as follows:

      verify_data =
          HMAC(finished_key,
               Transcript-Hash(Handshake Context,
                               Certificate*, CertificateVerify*))

      * Only included if present.

HMAC [RFC2104] uses a hash algorithm for the handshake. As described above, the HMAC input is typically produced by the running hash implementation; at this point, it is simply the hash of the handshake.

In earlier versions of TLS, the length of verify_data was always 12 octets. In TLS 1.3, it is the size of the HMAC output of the hash used to represent the handshake.

Note: alerts and any other non-handshake record types are not handshake messages and are not included in the hash calculation.

Any Post-Handshake records after the Finished message must be encrypted under the appropriate client_application_traffic_secret_N. In particular, this includes any alert sent by the Server in response to the Client’s Certificate and CertificateVerify messages.

10. TLS 1.3 KeyUpdate

At this point, readers may wonder why the TLS 1.3 KeyUpdate message is discussed again at the end of the article. The reason is that this message triggers TLS 1.3 to recompute keys, so it deserves closer examination.

Research has shown that if the same key is used to encrypt a large amount of data, an attacker may have some chance of recovering the symmetric encryption key by recording all ciphertexts and identifying patterns. Therefore, a mechanism for synchronously updating keys needs to be introduced. This mechanism also uses the HKDF algorithm to derive the next generation of keys from the old keys.

After encrypted messages reach a certain length, both sides also need to send KeyUpdate messages to recompute the encryption keys.

The KeyUpdate handshake message indicates that the sender is updating its own sending encryption keys. Either peer may send this message after sending its Finished message. If an implementation receives a KeyUpdate message before receiving the Finished message, it must terminate the connection with an “unexpected_message” alert. After sending a KeyUpdate message, the sender should use the new generation of keys for all of its traffic. After receiving a KeyUpdate, the receiver must update its receiving keys.

      enum {
          update_not_requested(0), update_requested(1), (255)
      } KeyUpdateRequest;

      struct {
          KeyUpdateRequest request_update;
      } KeyUpdate;
  • request_update:
    This field indicates whether the recipient of the KeyUpdate should respond with its own KeyUpdate. If an implementation receives any other value, it MUST terminate the connection with an “illegal_parameter” alert message.

If the request_update field is set to “update_requested”, the recipient MUST send its own KeyUpdate, with request_update set to “update_not_requested”, before sending its next application data record. This mechanism allows either side to force an update of the entire connection, but it results in an implementation receiving multiple KeyUpdates while still silently responding to a single update. Note that an implementation may receive any number of messages between sending a KeyUpdate (with request_update set to “update_requested”) and receiving the peer’s KeyUpdate, because those messages may already have been in transit. However, because sending and receiving keys are derived from independent traffic keys, retaining the receiving traffic keys does not affect the forward secrecy of data sent before the sender changed keys.

If implementations independently send their own KeyUpdates with request_update set to “update_requested” and their messages are both in transit, the result is that both sides respond and both sides update keys.

Both the sender and the receiver MUST encrypt their KeyUpdate messages with the old keys. In addition, before accepting any message encrypted with the new keys, both sides MUST enforce receipt of a KeyUpdate with the old keys. Failing to do so could enable a message truncation attack.

The next-generation traffic keys are computed by generating client_ / server_application_traffic_secret_N + 1 from client_ / server_application_traffic_secret_N, and then re-deriving the traffic keys as described in the previous section.

The next-generation application_traffic_secret is computed as follows:

       application_traffic_secret_N+1 =
           HKDF-Expand-Label(application_traffic_secret_N,
                             "traffic upd", "", Hash.length)

Once client_ / server_application_traffic_secret_N + 1 and its associated traffic keys have been computed, implementations should delete client_ / server_application_traffic_secret_N and its associated traffic keys.

XI. Key Export in TLS 1.3

In TLS 1.3, there are two exported keys, exporter:

       early_exporter_master_secret = Derive-Secret(Early Secret, "e exp master", ClientHello)
		     exporter_master_secret = Derive-Secret(Master Secret, "exp master", ClientHello...server Finished)

RFC5705 defines the TLS keying material exporter in terms of the TLS pseudorandom function (PRF). TLS 1.3 replaces the PRF with HKDF, so a new structure is required. The exporter interface remains unchanged.

The exporter value is computed as follows:

   TLS-Exporter(label, context_value, key_length) =
       HKDF-Expand-Label(Derive-Secret(Secret, label, ""),
                         "exporter", Hash(context_value), key_length)

Secret can be either early_exporter_master_secret or exporter_master_secret. Unless explicitly specified by the application, the implementation MUST use exporter_master_secret. early_exporter_master_secret is defined for use in cases where 0-RTT data requires exporter settings. It is recommended to provide a separate interface for the early exporter; this prevents exporter users from accidentally using the early exporter when they need the regular exporter, and vice versa.

If no context is provided, context_value is zero length. Therefore, computing without providing a context yields the same result as providing an empty context. This is a change from earlier versions of TLS, where an empty context produced output different from that produced when no context was provided. As of TLS 1.3, allocated exporter labels are not used regardless of whether a context is used. Future specifications MUST NOT define exporter usages that allow both an empty context and no context with the same label. New exporter usages should provide a context in all exporter computations, even if the value may be empty.

The requirements for the exporter label format are defined in [RFC5705] Section 4.

References:

RFC 5246
RFC 8466
Keyless SSL: The Nitty Gritty Technical Details
Cryptographic Extraction and Key Derivation: The HKDF Scheme

GitHub Repo: Halfrost-Field

Follow: halfrost · GitHub

Source: https://halfrost.com/HTTPS-key-cipher/