HTTPS Revisited (1) —— Introduction

1. Why HTTPS Is Needed

HTTP/1.1 has the following security issues:

1. It communicates in plaintext (unencrypted), so content may be eavesdropped on;
2. It does not verify the identity of the communicating party, so that party’s identity may be spoofed;
3. It cannot prove message integrity, so messages may be tampered with.

Because these points were not considered when HTTP was originally designed, applications built on HTTP all have these security issues.

1. Data Is Not Encrypted

On TCP/IP-based networks, there is a risk of traffic being monitored at every point in the network. Moreover, when communicating over the HTTP protocol, HTTP itself has no encryption capability, so it cannot encrypt the communication as a whole (the contents of requests and responses transmitted using HTTP). In other words, HTTP messages are sent in plaintext (that is, unencrypted messages).

As shown in the figure above, monitoring can occur at every stage of the Internet. Even encrypted communication can be monitored; it is just that the eavesdropper sees ciphertext. To solve the three major security issues with HTTP, the first step is to use encrypted communication. Therefore, an SSL (Secure Sockets Layer) / TLS (Transport Layer Security) layer was added at the transport layer to encrypt HTTP communication content.

HTTPS (HTTP Secure) is not a new protocol. Instead, HTTP first communicates with SSL (Secure Sockets Layer) / TLS (Transport Layer Security), and SSL/TLS then communicates with TCP. In other words, HTTPS communicates through a tunnel.

At this point, some readers may wonder: why not encrypt HTTP messages directly, so that the SSL/TLS layer is unnecessary? Indeed, directly encrypting HTTP messages can also achieve encrypted communication. However, although that solves the first issue, the latter two issues are much harder to solve.

Even if HTTP is encrypted directly, the HTTP headers are not encrypted, and header information can also make information insecure.

2. The Identity of the Peer Cannot Be Verified

Although TCP-based HTTP can ensure that data is delivered completely to the peer, it cannot verify the identity of that peer. HTTP is also very widely used because of the flexibility of the protocol. Neither side of the communication needs to verify identity. As long as the server receives a request it can recognize, it returns a response; every request necessarily has a response. Because the peer is not authenticated, this creates several risks:

• The server cannot verify who the request comes from or whether it is a legitimate client.
• After receiving a response, the client also cannot verify whether it came from a legitimate server.
• It cannot prevent DoS attacks (Denial of Service) caused by massive numbers of requests.

3. Data Tampering Cannot Be Prevented

The HTTP protocol cannot guarantee data integrity. Integrity refers to the accuracy of information. If information integrity cannot be proven, then it is impossible to determine whether the information is accurate.

When clients and servers receive responses and requests, they can only accept them unconditionally. HTTP also has no way to know whether a request or response has been tampered with during transmission, for example by a Man-in-the-Middle attack (MITM).

HTTP does have ways to verify message integrity, but they are still unreliable. For example, hash-based verification methods such as MD5 and SHA-1 can be used to verify a file’s digital signature. (Low-bit-strength MD5 and SHA-1 are no longer secure and are vulnerable to collision attacks; this will be analyzed in detail in a later article.)

Websites that provide download services may also provide digital signatures created with PGP (Pretty Good Privacy) and hash values generated by the MD5 algorithm. PGP is used to prove the digital signature of the file, and MD5 is a hash value generated by a one-way function. Under the HTTP protocol, the browser cannot know whether the data has been tampered with; the user still has to check manually. But if the PGP signature and MD5 value were already tampered with before transmission, the user may receive them and find that verification matches, in which case data integrity and correctness still cannot be guaranteed.

By using SSL, HTTPS not only guarantees ciphertext transmission, but more importantly can also verify the identity of the communicating party and ensure message integrity. It perfectly solves HTTP’s three major security flaws.

2. What Are the Benefits of Deploying HTTPS?

Some readers may wonder: aside from e-commerce, finance, and websites that deal with money, where HTTPS must be deployed, it does not matter whether other websites use HTTPS. I used to have a similar view, but that view is wrong.

There is no doubt that e-commerce, finance, and other money-related websites must deploy HTTPS to prevent users from suffering financial losses. But what about other websites? If HTTPS is not deployed and bare HTTP is used, the site can easily be hijacked, including having small ads injected by an ISP. Such ads severely affect user experience; if they are adult ads, they can also affect users’ impression of the website. In addition, which pages a user has browsed and the user’s behavior can also be easily analyzed, which counts as a privacy leak.

Deploying HTTPS has the following benefits:

1. Use HTTP/2 for Higher Performance

Content delivery networks and web hosting providers are beginning to promote HTTP/2. At a Velocity conference, Load Impact and Mozilla reported that Internet users can achieve 50–70% better website performance with HTTP/2 optimizations than with HTTP/1.1. However, to use HTTP/2’s performance advantages, HTTPS must first be deployed. This requirement can also be considered a way to promote HTTPS.

2. Improve SEO Ranking

Google announced in 2014 that websites supporting HTTPS would receive greater weight for higher rankings.

3. Better Referral Data

If you use Google’s Analytics library, it currently runs mandatorily over HTTPS. If you still use HTTP, Analytics will not obtain Referral information for the HTTP site, resulting in inaccurate data.

4. Higher Security

Mainstream browsers now add a small green lock indicator for HTTPS websites. A website without the green lock will not make a good first impression on users.

5. Improve Website Trust and Credibility

Starting with Chrome 62, if a webpage has an input field, any page without HTTPS is displayed as insecure.

6. HTML5 New Features

Starting with Chrome 50, geolocation and audio/video APIs must run over HTTPS, with the goal of ensuring secure data transmission.

7. iOS ATS Requirement

To promote HTTPS, Apple also announced at WWDC 2017 that new apps must enable the APS (App Transport Security) security feature.

3. Cryptography in HTTPS

1. Symmetric-Key Encryption

Symmetric-Key Encryption uses the same key for encryption and decryption.

• Advantage: fast computation;
• Disadvantage: the key is easy to obtain.

For more detailed information about symmetric encryption, see my previous article “A Tour of Symmetric Encryption Algorithms”

2. Public-Key Encryption

Public-Key Encryption, also known as asymmetric-key encryption, uses a pair of keys for encryption and decryption: a public key and a private key. Anyone can obtain the public key. After the sender obtains the receiver’s public key, it can use the public key to encrypt the data. After receiving the communication content, the receiver decrypts it with the private key.

• Advantage: more secure;
• Disadvantage: slower computation;

For more detailed information about public-key encryption, see my previous article “A Tour of Public-Key Cryptography Algorithms”

3. The Encryption Method Used by HTTPS

HTTPS uses a hybrid encryption mechanism: public-key encryption is used to transmit the symmetric key, and symmetric-key encryption is then used for communication. (The Session Key in the figure below is the symmetric key.)

4. Authentication

HTTPS uses certificates to authenticate the communicating party.

A digital certificate authority (CA, Certificate Authority) is a trusted third party for both the client and the server. The server operator applies to the CA for a public key. After the CA verifies the applicant’s identity, it digitally signs the requested public key, then distributes the signed public key and binds it into a public-key certificate.

During HTTPS communication, the server sends the certificate to the client. After the client obtains the public key from it, it first verifies the certificate. If verification succeeds, communication can begin.

For more detailed information about certificates, see my previous article “Ubiquitous Public-Key Certificates”

Using the open-source OpenSSL suite, anyone can build their own certificate authority and issue themselves a server certificate. When a browser accesses that server, it will display warning messages such as “Unable to confirm the security of the connection” or “There is a problem with this website’s security certificate.”

5. Integrity

TLS / SSL provides a message digest feature to verify integrity.

4. The TLS / SSL Protocol in HTTPS

What makes HTTPS secure is the TLS protocol behind it. It originated from a protocol called Secure Sockets Layer (SSL), developed at Netscape in the mid-1990s. By the late 1990s, Netscape handed SSL over to the IETF, which renamed it TLS and has managed the protocol ever since. Many people still refer to Web encryption as SSL, even though the vast majority of services have switched to supporting only TLS.

• 1995: SSL 2.0. Proposed by Netscape. Due to design flaws, this version was not secure; serious vulnerabilities were quickly discovered, and it has been deprecated.

• 1996: SSL 3.0. Written up as an RFC and began to gain popularity. It is now (as of 2015) insecure and must be disabled.

• 1999: TLS 1.0. The Internet standards organization ISOC took over from NetScape and released TLS 1.0, an upgraded version of SSL.

• 2006: TLS 1.1. Published as RFC 4346. It mainly fixed vulnerabilities related to CBC mode, such as BEAST attacks.

• 2008: TLS 1.2. Published as RFC 5246. Improved security. This is currently (as of 2015) the version that should be deployed in most cases; make sure you are using this version.

• 2018: On August 10, RFC8446, the TLS 1.3 protocol, was officially published. It removed insecure factors from TLS 1.2 and greatly improved the protocol’s security and performance.

In the IETF, protocols are referred to as RFCs. TLS 1.0 is RFC 2246, TLS 1.1 is RFC 4346, and TLS 1.2 is RFC 5246. Now, TLS 1.3 is RFC 8446. It took nearly 10 years to go from TLS 1.2 to TLS 1.3. RFCs are usually published sequentially, so the fact that the official TLS specifications all use 46 as part of their RFC numbers seems more planned than coincidental.

The TLS/SSL protocol sits between the application layer and the TCP transport-layer protocol. At a high level, TLS can be divided into two layers:

• The TLS Handshaking Protocols, closer to the application layer
• The TLS Record Protocol, closer to TCP

The TLS handshake protocols can be further divided into five subprotocols:

• change_cipher_spec (this protocol has been removed in TLS 1.3, but may still be present for compatibility with older TLS versions)
• alert
• handshake
• application_data
• heartbeat (this was newly added in TLS 1.3; versions before TLS 1.3 did not have this protocol)

The relationships among these subprotocols can be represented by the following diagram:

1. TLS Record Protocol

The record layer fragments upper-layer information blocks into TLSPlaintext records. A TLSPlaintext record contains a block of data of 2^14 bytes or less. Depending on the underlying ContentType, message boundaries are handled differently. The rules in TLS 1.3 are stricter than those enforced in TLS 1.2.

Handshake messages can be coalesced into a single TLSPlaintext record, or fragmented across several records, provided that:

• Handshake messages must not be interleaved with other record types. That is, if a handshake message is split across two or more records, there must not be any other records between them.

• Handshake messages must never span a key change. Implementations must verify that all messages before a key change are aligned with record boundaries; if they are not, they must terminate the connection with an “unexpected_message” alert. Because ClientHello, EndOfEarlyData, ServerHello, Finished, and KeyUpdate messages can occur immediately before a key change, implementations must align these messages with record boundaries.

Implementations must never send zero-length fragments of handshake type, even if those fragments contain padding.

In addition, Alert messages must not be fragmented across records, and multiple alert messages must not be coalesced into a single TLSPlaintext record. In other words, a record with alert type must contain exactly one message.

Application data messages contain data that is opaque to TLS. Application data messages should always be protected. Zero-length fragments of application data may be sent, because they can be used as a countermeasure against traffic analysis. Application data fragments may be split across multiple records or coalesced into a single record.

      struct {
          ContentType type;
          ProtocolVersion legacy_record_version;
          uint16 length;
          opaque fragment[TLSPlaintext.length];
      } TLSPlaintext;

• type:
  Used to handle high-level protocols in the TLS handshake layer.

      enum {
          invalid(0),
          change_cipher_spec(20),
          alert(21),
          handshake(22),
          application_data(23),
          heartbeat(24),  /* RFC 6520 */
          (255)
      } ContentType;

ContentType is a wrapper for the handshake protocol. The correspondence between message header types and handshake-layer subprotocol numbers is as follows:

• legacy_record_version:
  For all records generated by a TLS 1.3 implementation except the initial ClientHello (i.e., records not generated after a HelloRetryRequest), this must be set to 0x0303; for compatibility purposes, it may also be 0x0301. This field is deprecated in TLS 1.3 and must be ignored. In some cases, earlier versions of TLS used other values in this field.

In TLS 1.3, version is 0x0304. The mapping between earlier protocol versions and version is as follows:

• length:
  The length of TLSPlaintext.fragment, in bytes. The length must not exceed 2 ^ 14 bytes. An endpoint that receives a record exceeding this length must terminate the connection with a “record_overflow” alert message.

• fragment:
  The data being transmitted. The value of this field is opaque and is treated as an independent block to be processed by the higher-level protocol specified by the type field.

When cryptographic protection has not yet been applied, the TLSPlaintext structure is written directly to the wire. Once record protection begins, TLSPlaintext records are cryptographically protected. Note that application data records must not be written to an unprotected connection. Therefore, application data cannot be sent before the handshake succeeds.

The position of the TLS record layer protocol within the overall TLS protocol is as follows:

• It encapsulates the parallel subprotocols in the upper TLS layer (the handshake layer)—5 subprotocols in TLS 1.3, and 4 subprotocols in TLS 1.2 and earlier versions—adds a message header, and packages them downward for TCP processing.

• It applies cryptographic protection to the upper-layer application data protocol, while only performing simple encapsulation for the other subprotocols (i.e., without encryption).

More details about the TLS record layer protocol will be analyzed in the upcoming articles. TLS 1.2 and TLS 1.3 will also be compared.

2. TLS Cipher Change Protocol

Note: This protocol has been removed from the TLS 1.3 standard specification, but in real-world use it may still appear on the wire for compatibility with older TLS versions and some message intermediaries.

The change_cipher_spec protocol (hereinafter referred to as the CCS protocol) is the boundary in the TLS record layer that determines whether application data is encrypted. Once the client or server receives the CCS protocol from its peer, it indicates that application data can be encrypted during subsequent data transmission.

When the TLS record layer processes the five upper-layer protocols (cipher change protocol, alert protocol, handshake protocol, heartbeat protocol, and application data protocol), different TLS versions encrypt different protocols differently. The details are as follows:

More details about the TLS CCS protocol will be analyzed in the upcoming handshake articles. TLS 1.2 and TLS 1.3 will also be compared.

The protocol data structure is as follows:

   struct {
       enum { change_cipher_spec(1), (255) } type;
   } ChangeCipherSpec;

After being wrapped by the TLS record layer, the structure is as follows:

3. TLS Alert Protocol

TLS provides the alert content type to indicate closure information and errors. As with other messages, alert messages are encrypted according to the current connection state. In TLS 1.3, the severity of an error is implicit in the type of alert being sent, and the “level” field can be safely ignored. The “close_notify” alert is used to indicate an orderly shutdown of the connection in one direction. After receiving such an alert, a TLS implementation should signal to the application that the application data has ended.

After receiving an error alert, a TLS implementation should signal the error to the application and must not allow any further data to be sent or received on the connection.

The protocol data structure is as follows:

      enum { warning(1), fatal(2), (255) } AlertLevel;
      
      struct {
          AlertLevel level;
          AlertDescription description;
      } Alert;

After being wrapped by the TLS record layer, the structure is as follows:

TLS 1.3 makes very few changes to this protocol compared with TLS 1.2; it only adds a few new enum types.

All alert description messages in TLS 1.2:

enum {
       close_notify(0),
       unexpected_message(10),
       bad_record_mac(20),
       decryption_failed_RESERVED(21),
       record_overflow(22),
       decompression_failure(30),
       handshake_failure(40),
       no_certificate_RESERVED(41),
       bad_certificate(42),
       unsupported_certificate(43),
       certificate_revoked(44),
       certificate_expired(45),
       certificate_unknown(46),
       illegal_parameter(47),
       unknown_ca(48),
       access_denied(49),
       decode_error(50),
       decrypt_error(51),
       export_restriction_RESERVED(60),
       protocol_version(70),
       insufficient_security(71),
       internal_error(80),
       user_canceled(90),
       no_renegotiation(100),
       unsupported_extension(110),           /* new */
       (255)
   } AlertDescription;

All TLS 1.3 alert description messages:

      enum {
          close_notify(0),
          unexpected_message(10),
          bad_record_mac(20),
          decryption_failed_RESERVED(21),
          record_overflow(22),
          decompression_failure_RESERVED(30),
          handshake_failure(40),
          no_certificate_RESERVED(41),
          bad_certificate(42),
          unsupported_certificate(43),
          certificate_revoked(44),
          certificate_expired(45),
          certificate_unknown(46),
          illegal_parameter(47),
          unknown_ca(48),
          access_denied(49),
          decode_error(50),
          decrypt_error(51),
          export_restriction_RESERVED(60),
          protocol_version(70),
          insufficient_security(71),
          internal_error(80),
          inappropriate_fallback(86),
          user_canceled(90),
          no_renegotiation_RESERVED(100),
          missing_extension(109),
          unsupported_extension(110),
          certificate_unobtainable_RESERVED(111),
          unrecognized_name(112),
          bad_certificate_status_response(113),
          bad_certificate_hash_value_RESERVED(114),
          unknown_psk_identity(115),
          certificate_required(116),
          no_application_protocol(120),
          (255)
      } AlertDescription;

TLS 1.3 adds 9 alert description messages compared with TLS 1.2:

          inappropriate_fallback(86),
          missing_extension(109),
          certificate_unobtainable_RESERVED(111),
          unrecognized_name(112),
          bad_certificate_status_response(113),
          bad_certificate_hash_value_RESERVED(114),
          unknown_psk_identity(115),
          certificate_required(116),
          no_application_protocol(120),

4. TLS Handshake Protocol

The handshake protocol is the most central protocol in the entire TLS protocol suite, and HTTPS can provide security thanks to it.

The handshake protocol consists of multiple sub-messages. The first time the server and client complete a handshake, it requires 2 RTTs.

The purpose of the handshake protocol is for both parties to negotiate a cipher block, which is then handed to the TLS record layer for key encryption. In other words, the “consensus” (cipher block) reached by the handshake protocol is the foundation of security for all of TLS and HTTPS.

The handshake protocol changed significantly between TLS 1.2 and TLS 1.3. TLS 1.3’s 0-RTT is an entirely new concept. The two versions differ greatly in key negotiation and cipher suite selection.

The TLS 1.2 protocol data structure is as follows:

   enum {
       hello_request(0), 
       client_hello(1), 
       server_hello(2),
       certificate(11), 
       server_key_exchange (12),
       certificate_request(13), 
       server_hello_done(14),
       certificate_verify(15), 
       client_key_exchange(16),
       finished(20)
       (255)
   } HandshakeType;

   struct {
       HandshakeType msg_type;
       uint24 length;
       select (HandshakeType) {
           case hello_request:       HelloRequest;
           case client_hello:        ClientHello;
           case server_hello:        ServerHello;
           case certificate:         Certificate;
           case server_key_exchange: ServerKeyExchange;
           case certificate_request: CertificateRequest;
           case server_hello_done:   ServerHelloDone;
           case certificate_verify:  CertificateVerify;
           case client_key_exchange: ClientKeyExchange;
           case finished:            Finished;
       } body;
   } Handshake;

The TLS 1.3 protocol data structures are as follows:

      enum {
          hello_request_RESERVED(0),
          client_hello(1),
          server_hello(2),
          hello_verify_request_RESERVED(3),
          new_session_ticket(4),
          end_of_early_data(5),
          hello_retry_request_RESERVED(6),
          encrypted_extensions(8),
          certificate(11),
          server_key_exchange_RESERVED(12),
          certificate_request(13),
          server_hello_done_RESERVED(14),
          certificate_verify(15),
          client_key_exchange_RESERVED(16),
          finished(20),
          certificate_url_RESERVED(21),
          certificate_status_RESERVED(22),
          supplemental_data_RESERVED(23),
          key_update(24),
          message_hash(254),
          (255)
      } HandshakeType;

      struct {
          HandshakeType msg_type;    /* handshake type */
          uint24 length;             /* bytes in message */
          select (Handshake.msg_type) {
              case client_hello:          ClientHello;
              case server_hello:          ServerHello;
              case end_of_early_data:     EndOfEarlyData;
              case encrypted_extensions:  EncryptedExtensions;
              case certificate_request:   CertificateRequest;
              case certificate:           Certificate;
              case certificate_verify:    CertificateVerify;
              case finished:              Finished;
              case new_session_ticket:    NewSessionTicket;
              case key_update:            KeyUpdate;
          };
      } Handshake;

After being wrapped by the TLS record layer, the structure is as follows:

Although there are many types of handshake messages, by the time they are ultimately passed to the TLS record layer, some of them may be combined into a single message.

More details about the TLS handshake protocol will be analyzed in depth in the upcoming articles. TLS 1.2 and TLS 1.3 will also be compared.

5. TLS Application Data Protocol

The application data protocol refers to the various upper-layer protocols above TLS. The data that TLS primarily protects is the data of the application data protocol.

After being wrapped by the TLS record layer, the structure is as follows:

Depending on the encryption mode, the TLS record layer appends MAC verification data to the end of the application data.

6. TLS Heartbeat Protocol

This protocol was added in TLS 1.3. For more details, see the article “TLS & DTLS Heartbeat Extension”, which the author translated based on [RFC 6520]. If you are interested, you can take a look. That article also covers DTLS and PMTU discovery.

The protocol data structure is as follows:

   enum {
      heartbeat_request(1),
      heartbeat_response(2),
      (255)
   } HeartbeatMessageType;
   
   struct {
      HeartbeatMessageType type;
      uint16 payload_length;
      opaque payload[HeartbeatMessage.payload_length];
      opaque padding[padding_length];
   } HeartbeatMessage;

After being wrapped by the TLS record layer, the structure is as follows:

According to the definition in [RFC6066], during negotiation, the total length of a HeartbeatMessage must not exceed 2 ^ 14 or max_fragment_length.

The length of a HeartbeatMessage is TLSPlaintext.length for TLS and DTLSPlaintext.length for DTLS. In addition, the type field is 1 byte long, and payload_length is 2 bytes long. Therefore, padding_length is TLSPlaintext.length - payload_length - 3 for TLS, and DTLSPlaintext.length - payload_length - 3 for DTLS. padding_length must be at least 16.

The sender of a HeartbeatMessage must use at least 16 bytes of random padding. The padding of a received HeartbeatMessage message must be ignored.

V. What’s Next

As the opening article in this HTTPS series, this article analyzed why the HTTPS protocol is necessary, the benefits HTTPS brings, the essence of HTTPS security, and the relationships and roles of the various TLS subprotocols.

The next few articles will provide a detailed comparison of TLS 1.2 and TLS 1.3 in terms of differences in the handshake protocol, differences at the record layer, differences in key derivation, and what the new 0-RTT feature in TLS 1.3 is really about.

References:

HTTP Explained with Diagrams
HTTPS: From Basics to Depth
TLS 1.3 Specification [RFC 8446]
TLS 1.2 Specification [RFC 5246]

GitHub Repo：Halfrost-Field

Follow: halfrost · GitHub

Source: https://halfrost.com/HTTPS_begin/