Last Friday, Tavis Ormandy from Google’s Project Zero contacted Cloudflare to report a security problem with our edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare.
It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.
Dan Goodin / Ars Technica:
Cloudflare fixes bug from September 2016 that leaked private user info, says search engine caches have now been cleared of sensitive info — Service used by 5.5 million websites may have leaked passwords and authentication tokens. — Cloudflare, a service that helps optimize the security …
Ryan Lackey / octal:
Cloudflare’s Cloudbleed may have exposed user authentication info and credentials; sites hosted by Cloudflare should consider prompting users to reset passwords
Tavis Ormandy (Tavis Ormandy) of Google’s Project Zero uncovered a major vulnerability in the Cloudflare Internet infrastructure service. Essentially, web requests to Cloudflare-backed sites received answers which included random information from other Cloudflare-backed sites! This information could potentially include confidential information
Both Project Zero and Cloudflare acted promptly. The bug was reported on 2017–02–17 and a mitigation was in place within an hour. Public notification was given on 2017–02–23.
The duration (2016–09–22 to 2017–02–20) and potential breadth of information exposed is huge — Cloudflare has over 2 million websites on its network, and data from any of these is potentially exposed. Cloudflare has said the actual impact is relatively minor, so I believe only limited amounts of information were actually disseminated. Essentially, broad range of data was potentially at risk, but the risk to any individual piece of data was very low. Regardless, unless it can be shown conclusively that your data was NOT compromised, it would be prudent to act as if it were.
While Cloudflare’s service was rapidly patched to eliminate this bug, data was leaking constantly before this point — for months. Some of this data was cached publicly in search engines such as Google, and is being removed.
The most sensitive information leaked is authentication information and credentials. A compromise of this data can have lasting and ongoing consequences until credentials are revoked and replaced.
From an individual perspective, this is straightforward —the most effective mitigation is to change your passwords. While this might not be necessary (it is unlikely your passwords were exposed in this incident), it will absolutely improve your security
For site operators using Cloudflare: while it may be disruptive, you should seriously consider forcing a password change for all of your users. Be careful not to “scare” them, but it may be prudent to push such a change.
Additionally, any sites not on Cloudflare but with a lot of users who use Cloudflare-hosted sites (essentially any large or consumer sites on the Internet) should consider forcing user password changes in case their users had re-used the same passwords on each site.
Sites should invalidate authentication credentials for mobile applications and other machine to machine communications (IoT devices, etc.), forcing users to re-enroll apps and devices, if they used Cloudflare as an infrastructure provider.
Subjectively, I believe this is a serious security issue and should be evaluated by any major service using Cloudflare. It is likely not an “end of the world” in severity, as unless there was concerted malicious exploitation
CloudFlare has been working around the clock in the past few days to address a critical security problem that led to sensitive customer data getting leaked and cached by search engines.
The uninitialized memory leak was discovered by Google Project Zero researcher Tavis Ormandy, who jokingly said he considered the idea of calling it “Cloudbleed” due to similarities to the OpenSSL bug known as HeartBleed.
Ormandy noticed the leakage on February 17, while working on a fuzzing-related project. He immediately notified CloudFlare and the CDN had an initial mitigation in place within an hour. However, the cleanup effort took several days since Google, Yahoo, Bing and other search engines had cached at least 770 URIs across 161 unique domains containing leaked memory.
In case you are still wondering about the SHA-1 being broken and if someone is going to be spending hundreds of thousands of dollars to create a fake Certificate Authority and sniff your OkCupid credentials, don’t worry. Why spend so much money when your credentials are being cached by search engines?… Wait, what?
A serious combination of bugs, dubbed Cloudbleed by [Tavis Ormandy], lead to uninitialized memory being present in the response generated by the reverse proxies and leaked to the requester. Since these reverse proxies are shared between Cloudflare clients, this makes the problem even worst, since random data from random clients was leaking. It’s sort of like Heartbleed for HTTP requests.
The seriousness of the issue can be fully appreciated in [Tavis] words:
“The examples we’re finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I’ve informed cloudflare what I’m working on. I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”
According to Cloudflare, the leakage can include HTTP headers, chunks of POST data (perhaps containing passwords), JSON for API calls, URI parameters, cookies and other sensitive information used for authentication (such as API keys and OAuth tokens). An HTTP request to a Cloudflare web site that was vulnerable could reveal information from other unrelated Cloudflare sites.
Just to reassure the readers and not be alarmist, there is no evidence of anyone having exploiting what happened. Before public exposure, Cloudflare worked in proximity with search engines companies to ensure memory was scrubbed from search engine caches from a list of 161 domains they had identified.
Cloudflare informed customers on Wednesday that it has found no evidence of the recently discovered memory leak being exploited for malicious purposes before it was patched.
The bug was discovered on February 17 by Google Project Zero researcher Tavis Ormandy. The expert jokingly considered the idea of calling it “Cloudbleed” due to some similarities to HeartBleed and the name stuck.
Cloudflare determined that the bug caused its edge servers to run past the end of a buffer and return memory that contained potentially sensitive information, including cookies and authentication tokens. Ormandy also found that the leaked data included passwords, encryption keys, private messages from dating sites, chat messages, IP addresses and HTTPS requests.
The flaw was introduced in September 2016, but it had the greatest impact between February 13 and February 18, when one in every 3.3 million requests going through Cloudflare’s systems may have resulted in memory leakage.
In a lengthy blog post published on Wednesday, Cloudflare co-founder and CEO Matthew Prince said that while this was “an extremely serious bug” with a potentially massive impact, an analysis of the logs had turned up no evidence of malicious exploitation. Prince also pointed out that a vast majority of customers were not impacted.
The summary is that, while the bug was very bad and had the potential to be much worse, based on our analysis so far: 1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched; 2) the vast majority of Cloudflare customers had no data leaked; 3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and 4) our review is ongoing.
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.
We are a professional review site that has advertisement and can receive compensation from the companies whose products we review. We use affiliate links in the post so if you use them to buy products through those links we can get compensation at no additional cost to you.OkDecline
10 Comments
Tomi Engdahl says:
https://techcrunch.com/2017/02/23/major-cloudflare-bug-leaked-sensitive-data-from-customers-websites/
Tomi Engdahl says:
List of Sites possibly affected by Cloudflare’s #Cloudbleed HTTPS Traffic Leak
https://github.com/pirate/sites-using-cloudflare/blob/master/README.md
Tomi Engdahl says:
Incident report on memory leak caused by Cloudflare parser bug
https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
Last Friday, Tavis Ormandy from Google’s Project Zero contacted Cloudflare to report a security problem with our edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare.
It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.
Tomi Engdahl says:
Dan Goodin / Ars Technica:
Cloudflare fixes bug from September 2016 that leaked private user info, says search engine caches have now been cleared of sensitive info — Service used by 5.5 million websites may have leaked passwords and authentication tokens. — Cloudflare, a service that helps optimize the security …
Serious Cloudflare bug exposed a potpourri of secret customer data
Service used by 5.5 million websites may have leaked passwords and authentication tokens
https://arstechnica.com/security/2017/02/serious-cloudflare-bug-exposed-a-potpourri-of-secret-customer-data/
Tomi Engdahl says:
Ryan Lackey / octal:
Cloudflare’s Cloudbleed may have exposed user authentication info and credentials; sites hosted by Cloudflare should consider prompting users to reset passwords
Cloudbleed: How to deal with it
https://medium.com/@octal/cloudbleed-how-to-deal-with-it-150e907fd165#.pl89h6avs
Tavis Ormandy (Tavis Ormandy) of Google’s Project Zero uncovered a major vulnerability in the Cloudflare Internet infrastructure service. Essentially, web requests to Cloudflare-backed sites received answers which included random information from other Cloudflare-backed sites! This information could potentially include confidential information
Both Project Zero and Cloudflare acted promptly. The bug was reported on 2017–02–17 and a mitigation was in place within an hour. Public notification was given on 2017–02–23.
The duration (2016–09–22 to 2017–02–20) and potential breadth of information exposed is huge — Cloudflare has over 2 million websites on its network, and data from any of these is potentially exposed. Cloudflare has said the actual impact is relatively minor, so I believe only limited amounts of information were actually disseminated. Essentially, broad range of data was potentially at risk, but the risk to any individual piece of data was very low. Regardless, unless it can be shown conclusively that your data was NOT compromised, it would be prudent to act as if it were.
While Cloudflare’s service was rapidly patched to eliminate this bug, data was leaking constantly before this point — for months. Some of this data was cached publicly in search engines such as Google, and is being removed.
The most sensitive information leaked is authentication information and credentials. A compromise of this data can have lasting and ongoing consequences until credentials are revoked and replaced.
From an individual perspective, this is straightforward —the most effective mitigation is to change your passwords. While this might not be necessary (it is unlikely your passwords were exposed in this incident), it will absolutely improve your security
For site operators using Cloudflare: while it may be disruptive, you should seriously consider forcing a password change for all of your users. Be careful not to “scare” them, but it may be prudent to push such a change.
Additionally, any sites not on Cloudflare but with a lot of users who use Cloudflare-hosted sites (essentially any large or consumer sites on the Internet) should consider forcing user password changes in case their users had re-used the same passwords on each site.
Sites should invalidate authentication credentials for mobile applications and other machine to machine communications (IoT devices, etc.), forcing users to re-enroll apps and devices, if they used Cloudflare as an infrastructure provider.
Subjectively, I believe this is a serious security issue and should be evaluated by any major service using Cloudflare. It is likely not an “end of the world” in severity, as unless there was concerted malicious exploitation
Tomi Engdahl says:
https://www.wired.com/2017/02/crazy-cloudflare-bug-jeopardized-millions-sites/
Tomi Engdahl says:
Password management made easy as news of CloudFlare leak surfaces
https://opensource.com/article/17/2/password-management?sc_cid=701600000011jJaAAI
Tomi Engdahl says:
CloudFlare Leaked Sensitive Customer Data
http://www.securityweek.com/cloudflare-leaked-sensitive-customer-data
CloudFlare has been working around the clock in the past few days to address a critical security problem that led to sensitive customer data getting leaked and cached by search engines.
The uninitialized memory leak was discovered by Google Project Zero researcher Tavis Ormandy, who jokingly said he considered the idea of calling it “Cloudbleed” due to similarities to the OpenSSL bug known as HeartBleed.
Ormandy noticed the leakage on February 17, while working on a fuzzing-related project. He immediately notified CloudFlare and the CDN had an initial mitigation in place within an hour. However, the cleanup effort took several days since Google, Yahoo, Bing and other search engines had cached at least 770 URIs across 161 unique domains containing leaked memory.
Tomi Engdahl says:
Cloudbleed — Your Credentials Cached in Search Engines
http://hackaday.com/2017/02/24/cloudbleed-your-credentials-cached-in-search-engines/
In case you are still wondering about the SHA-1 being broken and if someone is going to be spending hundreds of thousands of dollars to create a fake Certificate Authority and sniff your OkCupid credentials, don’t worry. Why spend so much money when your credentials are being cached by search engines?… Wait, what?
A serious combination of bugs, dubbed Cloudbleed by [Tavis Ormandy], lead to uninitialized memory being present in the response generated by the reverse proxies and leaked to the requester. Since these reverse proxies are shared between Cloudflare clients, this makes the problem even worst, since random data from random clients was leaking. It’s sort of like Heartbleed for HTTP requests.
The seriousness of the issue can be fully appreciated in [Tavis] words:
“The examples we’re finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I’ve informed cloudflare what I’m working on. I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”
According to Cloudflare, the leakage can include HTTP headers, chunks of POST data (perhaps containing passwords), JSON for API calls, URI parameters, cookies and other sensitive information used for authentication (such as API keys and OAuth tokens). An HTTP request to a Cloudflare web site that was vulnerable could reveal information from other unrelated Cloudflare sites.
Just to reassure the readers and not be alarmist, there is no evidence of anyone having exploiting what happened. Before public exposure, Cloudflare worked in proximity with search engines companies to ensure memory was scrubbed from search engine caches from a list of 161 domains they had identified.
cloudflare: Cloudflare Reverse Proxies are Dumping Uninitialized Memory
https://bugs.chromium.org/p/project-zero/issues/detail?id=1139
Tomi Engdahl says:
Cloudflare Finds No Evidence of “Cloudbleed” Exploitation
http://www.securityweek.com/cloudflare-finds-no-evidence-cloudbleed-exploitation
Cloudflare informed customers on Wednesday that it has found no evidence of the recently discovered memory leak being exploited for malicious purposes before it was patched.
The bug was discovered on February 17 by Google Project Zero researcher Tavis Ormandy. The expert jokingly considered the idea of calling it “Cloudbleed” due to some similarities to HeartBleed and the name stuck.
Cloudflare determined that the bug caused its edge servers to run past the end of a buffer and return memory that contained potentially sensitive information, including cookies and authentication tokens. Ormandy also found that the leaked data included passwords, encryption keys, private messages from dating sites, chat messages, IP addresses and HTTPS requests.
The flaw was introduced in September 2016, but it had the greatest impact between February 13 and February 18, when one in every 3.3 million requests going through Cloudflare’s systems may have resulted in memory leakage.
In a lengthy blog post published on Wednesday, Cloudflare co-founder and CEO Matthew Prince said that while this was “an extremely serious bug” with a potentially massive impact, an analysis of the logs had turned up no evidence of malicious exploitation. Prince also pointed out that a vast majority of customers were not impacted.
Quantifying the Impact of “Cloudbleed”
https://blog.cloudflare.com/quantifying-the-impact-of-cloudbleed/
The summary is that, while the bug was very bad and had the potential to be much worse, based on our analysis so far: 1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched; 2) the vast majority of Cloudflare customers had no data leaked; 3) after a review of tens of thousands of pages of leaked data from search engine caches, we have found a large number of instances of leaked internal Cloudflare headers and customer cookies, but we have not found any instances of passwords, credit card numbers, or health records; and 4) our review is ongoing.