This week I encountered the most bizzare support request: "Is it possible that our website would not load on iPhones?". Don't be silly, "that's not a thing" I told the customer ... It appears I was wrong.
TL;DR
- Every browser on iOS behaved the same and gave vague feedback
- Curl gave the same error, but provided far better feedback
- The error was caused by a web server behind a reverse proxy sending out an
Upgrade: h2header which is not permitted in the HTTP/2 protocol (RFC7540) - To fix the problem, remove
Protocols h2from the Apache config
The Error
Every browser (Safari, Chrome & Firefox) on iOS (iPhone and iPad) completely failed when loading the website. At the same time the site was accessible and functionting normaly on all browsers on Mac, Linux, Windows and Android. The error on iOS presented slightly differently depending on which browser was being used, some gave vague feedback, others gave none.
Safari and Firefox (iOS)
When the URL was input into the address bar the browser indicated it was loading the site, then quickly failed and closed the tab. There was no error feedback whatsoever.
Chrome (iOS)
When the URL was input into the address bar the browser indicates it is loading the site, then quickly fails and shows the following error page with the vague error description "ERR_FAILED":
Investigation
The website in question was a WordPress site running on a LAMP stack in an LXC container, all requests to the site were being routed through an NGINX reverse proxy. The configutation was managed by Ansible and the same roles had been applied to a number of other hosts. These were also exhibiting the same error which meant I had the same error on over 20 websites and applications. There was a lot of diversity in the group of containers affected, so this quickly allowed me to focus on a small set of commonanlities amongst the sites that were being affected.
It was very difficult to know where to start becuase there was such little feedback on the error, so I started looking at the raw HTTP conversations using Firefox, this yielded nothing. So I then went to the command line and ran curl. BINGO, I got the the following error: curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1). When I appended -v, I got the following:
* Trying 116.202.000.000:443...
* TCP_NODELAY set
* Connected to site.example.com (116.202.000.000) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=site.example.com
* start date: May 23 09:25:56 2020 GMT
* expire date: Aug 21 09:25:56 2020 GMT
* subjectAltName: host "site.example.com" matched cert's "site.example.com"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ea761971d0)
> GET / HTTP/2
> Host: site.example.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [h2]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host site.example.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
So it turns out the error was caused by an invalid header being sent to the client. The server advertises that the connection can be upgraded to HTTP/2 despite the fact the connection has already been established using HTTP/2. The protocol as defined in RFC7540 does not permit this:
8.1.2.2. Connection-Specific Header Fields
HTTP/2 does not use the Connection header field to indicate connection-specific header fields; in this protocol, connection- specific metadata is conveyed by other means. An endpoint MUST NOT generate an HTTP/2 message containing connection-specific header fields; any message containing connection-specific header fields MUST be treated as malformed (Section 8.1.2.6).
The only exception to this is the TE header field, which MAY be present in an HTTP/2 request; when it is, it MUST NOT contain any value other than "trailers".
This means that an intermediary transforming an HTTP/1.x message to HTTP/2 will need to remove any header fields nominated by the Connection header field, along with the Connection header field itself. Such intermediaries SHOULD also remove other connection- specific header fields, such as Keep-Alive, Proxy-Connection, Transfer-Encoding, and Upgrade, even if they are not nominated by the Connection header field.
Note: HTTP/2 purposefully does not support upgrade to another protocol. The handshake methods described in Section 3 are believed sufficient to negotiate the use of alternative protocols.
The root cause, "How did this invalid header get sent to the client?"
In my setup an NGINX reverse proxy recevied requests from the public internet and then forwarded them to Apache instances running in LXC containers on the same host. While the connection between the clients browser and the NGINX reverse proxy was over HTTPS and used HTTP/2, the connection onwards to the Apache server used neither. So when an Ansible role, which assumed that TLS should be setup on all Apache servers, inserted Protocols h2 into the Apache config it created the scenario described above and fell fowl of RFC7540.
The fix
Don't advertised HTTP/2 twice, remove Protocols h2 from the Apache config.
Further work
Understand why the three web browsers from different vendors had exactly the same error on iOS and why the shared their approach to the protocol error with curl.
Update 13/09/2021
While working on a similar host I encountered the issue again and it was caused by a bug in the configuration being managed by Ansible. However this time there was a more descriptive, although still unintuitive error: "NSPOSIXErrorDomain:100". This lead me to another article on the issue by Sebastian N. (megamorf):
