Cloudflare uses a vendor called Verkada for cameras in our offices in San Francisco, Austin, New York, London and Singapore. These cameras are used at the entrances, exits and main thoroughfares of our offices and have been part of maintaining the security of offices that have been closed for almost a year.
Yesterday, we were notified of a breach of Verkada that allowed a hacker to access Verkada’s internal support tools to manage those cameras remotely, as well as access them through a remote root shell. As soon as we were notified of the breach, we proceeded to shut down the cameras in all our office locations to prevent further access.
To be clear: this hack affected the cameras and nothing else. No customer data was accessed, no production systems, no databases, no encryption keys, nothing. Some press reports indicate that we use a facial recognition feature available in Verkada. This is not true. We do not.
Our internal systems follow the same Zero Trust model that we provide to our customers, and as such our corporate office networks are not implicitly trusted by our other locations or data centers. From a security point of view connecting from one of our corporate locations is no different than connecting from a non-Cloudflare location.
While these camera devices were breached, it’s important to understand that inherently every employee has something like a root shell on a corporate network. It is simply not enough to rely on background checks, training on phishing and other threats or policies about not visiting questionable websites to keep malware off your network. This incident emphasizes the importance of the Zero Trust model that Cloudflare follows and provides to customers, which ensures that if any one system or vendor is compromised, it does not compromise the entire organization.
For example, a claim was made on Twitter that it might have been possible to access Cloudflare CEO Matthew Prince’s laptop.
This claim is spurious for a few reasons. Firstly, Matthew isn’t in the San Francisco office because of COVID-19. None of us are. So being on the corporate San Francisco network doesn’t get you closer to his or my machine.
Secondly, even if, somehow, an attacker were to get into his laptop they’d still have to get past the per-application, hard-key protected access control through our Zero Trust product Cloudflare Access.
And another claim was that the attackers had a “root shell inside Cloudflare’s network”.
That sounds scary but we don’t trust our corporate network; we use our products, such as Cloudflare Access, to control access to resources. The fact that the attacker had access to a machine inside the corporate network is no better than the kind of access they’d have had if they’d connected to our corporate WiFi network. The network isn't important, it's the access control that matters.
Of course, if we had been using the old castle-and-moat style of corporate networking (where anything and anyone on the corporate network are inherently trusted) the outcome could have been different.
This is why Zero Trust is so powerful. It allowed us all to work from home because of COVID-19 and it means that an attacker who got into the office network doesn’t get any further.
Internal Monitoring
Our monitoring and network intrusion detection system were able to capture the Verkada compromise and show exactly what network connections were made from the cameras. Additionally Verkada was able to provide us with a log of commands executed on our cameras.
Some press stories contain an image of the “root shell” on one of the cameras. It shows the attacker having executed commands like whoami, ifconfig and curl ifconfig.me.
Here are the commands executed by the intruder on March 8 (we are waiting for Verkada to supply us with the commands executed on March 9 but our other monitoring and logging tools give us a good sense of what those commands will be):
2358 Singapore curl ifconfig.me
2357 New York curl ifconfig.me
2352 San Francisco curl ifconfig.me
2352 San Francisco ifconfig
2352 San Francisco whoami
1451 San Francisco ifconfig
1451 San Francisco whoamiHere’s a log of the traffic from the cameras. Let’s start with unusual DNS queries. This shows DNS lookups from two cameras in San Francisco, one in London and one in Singapore. Mostly the attacker was using ifconfig.me to find their external IP address.
In San Francisco they attempted to access Cloudflare’s main website, an admin panel that doesn’t exist (there is no admin.cloudflare.com) and a specific customer that I have replaced with example.com (the hacker also tried admin.example.com).
2021-03-09T00:01:15.373343Z London ifconfig.me
2021-03-09T00:01:15.373343Z London ifconfig.me
2021-03-09T00:01:15.372302Z London ifconfig.me
2021-03-09T00:01:15.371451Z London ifconfig.me
2021-03-09T00:01:00.580199Z San Francisco admin.example.com
2021-03-09T00:00:56.818201Z San Francisco admin.cloudflare.com.as13335.com
2021-03-09T00:00:56.768506Z San Francisco admin.cloudflare.com
2021-03-09T00:00:50.285755Z San Francisco ff.as13335.com
2021-03-09T00:00:45.014855Z San Francisco example.com
2021-03-09T00:00:45.011649Z San Francisco example.com
2021-03-09T00:00:33.418232Z San Francisco cloudflare.com
2021-03-08T23:59:49.677219Z San Francisco ifconfig.me
2021-03-08T23:59:49.677216Z San Francisco ifconfig.me
2021-03-08T23:59:27.261902Z Singapore cloudflare.com
2021-03-08T23:59:27.25919Z Singapore cloudflare.com
2021-03-08T23:58:30.274265Z Singapore ifconfig.me
2021-03-08T23:58:30.274155Z Singapore ifconfig.me
2021-03-08T23:52:23.145321Z San Francisco ifconfig.meThey made some of their requests using HTTP and we captured those requests
2021-03-09T00:01:15.375176Z London
ifconfig.me GET /
curl/7.71.1
200 OK
2021-03-09T00:01:00.630747Z San Francisco
admin.example.com GET /
curl/7.64.0
301 Moved Permanently
2021-03-09T00:00:45.041609Z San Francisco
example.com GET /
curl/7.64.0
301 Moved Permanently
2021-03-09T00:00:33.476684Z San Francisco
cloudflare.com GET /
curl/7.64.0
301 Moved Permanently
2021-03-08T23:59:49.708081Z San Francisco
ifconfig.me GET /
curl/7.64.0
200 OK
2021-03-08T23:58:30.289352Z Singapore
ifconfig.me GET /
curl/7.64.0
200 OK
2021-03-08T23:52:23.183201Z San Francisco
ifconfig.me GET /
curl/7.64.0
200 OKWe also have complete connection logging but in the interests of space I won’t put it all here.
The downloaded archived video
The attacker used the Verkada tools to download two archived recordings of our San Francisco office lobby. We only keep 30 days of video but when there’s a formal investigation we keep specific video for up to one year. The downloaded videos date back to July 13, 2020; we had kept them to support investigation into a theft. Here are some stills from each of those videos:
Timeline (UTC)
March 81451 - Unauthorized access to Verkada security cameras2352 - Unauthorized access is used to run curl ifconfig.me from a Cloudflare camera in San Francisco
March 90029 - Unauthorized download of an archived video from our front door camera in San Francisco0033 - Unauthorized download of an archived video from our elevator camera in San Francisco0040 - A second unauthorized download of an archived video from our elevator camera1750 - Contacted by a reporter about security incident involving cameras2107 - First internal report of a Twitter user claiming to have accessed our cameras with confirmation one of our cameras had been compromised2122 - Cloudflare disables all network connectivity of our security cameras2126 - Internal review of our corporate network security controls is completed2149 - Article is released about compromise of Verkada cameras