We keep a close eye on tweets that mention CloudFlare because sometimes we get early warning about odd errors that we are not seeing ourselves through our monitoring systems.
Towards the end of August we saw a small number of tweets like this one:
indicating that trying to browse to a CloudFlare customer web site using the Twitter in-app browser was resulting in an error page. Which was very odd because it was clearly only happening occasionally: very occasionally.
Luckily, the person who tweeted that was in the same timezone as me and able to help debug together (thanks James White!); we discovered that the following sequence of events was necessary to reproduce the bug:
Click on a link in a tweet to a web site that is using an https URL and open in the Twitter in-app browser (not mobile Safari). This site may or may not be a CloudFlare customer.
Then click on a link on that page to a site over an http URL. This site must be on CloudFlare.
BOOM
That explained why this happened very rarely, but the question became... why did it happen at all? After some debugging it appeared to happen in recent versions of both iOS and the Twitter app (including the iOS 9 beta).
To figure out what was going on I turned to Charles Proxy and used it to intercept the communication between my iPhone and CloudFlare. Happily, Charles Proxy can intercept SSL connections by installing a custom certificate on a phone. I ran Charles Proxy on my laptop, pointed my iPhone it at (to use it as a proxy) and clicked on a tweet to a site that I had set up specially for testing.
Charles Proxy showed that the request that generated an error looked like this:
GET /test HTTP/1.1\r\n
Host: www.example.com\r\n
Referer: \r\n
Accept-Encoding: gzip, deflate\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n
Accept-Language: en-us\r\n
Connection: keep-alive\r\n
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Mobile/12H321 Twitter for iPhone\r\n
\r\nAt first glance this looked pretty normal but on a second look something really stuck out:
Referer: \r\nRFC7231 clearly states that if the Referer header is present it must contain a URI:
Referer = absolute-URI / partial-URIThe RFC also gives a clue is to why the header is in this state when jumping from HTTPS to HTTP:
A user agent MUST NOT send a Referer header field in anunsecured HTTP request if the referring page was received with asecure protocol.
CloudFlare's Browser Integrity Check was verifying that the Referer header was well formed and generating the error (since there's a Referer, but it's empty) and it looks like the Twitter app in-app browser (and, we later discovered, the Facebook app in-app browser) are, instead of removing the header, blanking it out.
This is a good example of how it's not always easy to be "strictly" RFC-compliant. Postel's Law tells us that in this case CloudFlare needs to relax its check.
We reported this problem to both Twitter and Facebook and rolled out a fix that allows this behavior on the part of these clients and will turn the strict checking back on when it makes sense.
But watch your software if you validate the Referer header. You might be running into this oddness.