2025 年 3 月 21 日发生的 Cloudflare 事件
2025 年 3 月 21 日 ，包括 R2 对象存储 在内的多项 Cloudflare 服务经历了持续 1 小时 7 分钟（UTC 时间 21:38 开始，22:45 结束）的错误率升高情况。在事件发生期间，全球范围内 R2 的 100% 写入操作失败，大约 35% 的读取操作失败。尽管该事件是从 R2 开始的，但它也影响了其他 Cloudflare 服务，包括Cache Reserve、Images、Log Delivery、Stream 和 Vectorize。
在为 R2 Gateway 服务（R2 的 API 前端）轮换用于验证存储基础设施的凭据时，R2 工程团队无意中将新凭据（ID 和密钥对）部署到了服务的开发实例而非生产环境。当旧凭据从我们的存储基础设施中删除时（作为密钥轮换过程的一部分），生产 R2 Gateway 服务无法访问新凭据。这最终导致 R2 的 Gateway 服务无法与我们的存储后端进行身份验证。本事件中没有发生数据丢失或损坏：任何返回成功 HTTP 状态代码的进行中上传或修改操作均已被持久化保存。
一旦确定了根本原因并意识到我们未能将新凭据部署到生产环境的 R2 Gateway 服务，我们立即部署了更新凭据，服务可用性随即恢复。
本事件发生是由于人为错误，而持续时间过长是因为我们对 Gateway Worker 用于验证存储基础设施的凭据缺乏适当的可见性。
对此事件及其给您或您的用户造成的中断，我们深表歉意。我们以高标准要求自己，这样的事件是不可接受的。本文详细说明了此次事件的影响、发生的经过和时间，以及我们为确保此类故障（及类似故障）不再发生而正在采取的措施。
什么受到影响？
主要事件时段为 21:38 UTC 到 22:45 UTC 之间。
下表详细说明了 R2 和依赖于 R2 或与 R2 交互的 Cloudflare 服务受到的具体影响：
| Product/Service | Impact |
|---|---|
| R2 | All customers using Cloudflare R2 would have experienced an elevated error rate during the primary incident window. Specifically: * Object write operations had a 100% error rate. * Object reads had an approximate error rate of 35% globally. Individual customer error rate varied during this window depending on access patterns. Customers accessing public assets through custom domains would have seen a reduced error rate as cached object reads were not impacted. * Operations involving metadata only (e.g., head and list operations) were not impacted. There was no data loss or risk to data integrity within R2's storage subsystem. This incident was limited to a temporary authentication issue between R2's API frontend and our storage infrastructure. |
| Billing | Billing uses R2 to store customer invoices. During the primary incident window, customers may have experienced errors when attempting to download/access past Cloudflare invoices. |
| Cache Reserve | Cache Reserve customers observed an increase in requests to their origin during the incident window as an increased percentage of reads to R2 failed. This resulted in an increase in requests to origins to fetch assets unavailable in Cache Reserve during this period. User-facing requests for assets to sites with Cache Reserve did not observe failures as cache misses failed over to the origin. |
| Email Security | Email Security depends on R2 for customer-facing metrics. During the primary incident window, customer-facing metrics would not have updated. |
| Images | All (100% of) uploads failed during the primary incident window. Successful delivery of stored images dropped to approximately 25%. |
| Key Transparency Auditor | All (100% of) operations failed during the primary incident window due to dependence on R2 writes and/or reads. Once the incident was resolved, service returned to normal operation immediately. |
| Log Delivery | Log delivery (for Logpush and Logpull) was delayed during the primary incident window, resulting in significant delays (up to 70 minutes) in log processing. All logs were delivered after incident resolution. |
| Stream | All (100% of) uploads failed during the primary incident window. Successful Stream video segment delivery dropped to 94%. Viewers may have seen video stalls every minute or so, although actual impact would have varied. Stream Live was down during the primary incident window as it depends on object writes. |
| Vectorize | Queries and operations against Vectorize indexes were impacted during the incident window. During the incident window, Vectorize customers would have seen an increased error rate for read queries to indexes and all (100% of) insert and upsert operation failed as Vectorize depends on R2 for persistent storage. |
事件时间线
所有时间戳均为协调世界时 (UTC)。
| Time | Event |
|---|---|
| Mar 21, 2025 - 19:49 UTC | The R2 engineering team started the credential rotation process. A new set of credentials (ID and key pair) for storage infrastructure was created. Old credentials were maintained to avoid downtime during credential change over. |
| Mar 21, 2025 - 20:19 UTC | Set updated production secret (wrangler secret put) and executed wrangler deploy command to deploy R2 Gateway service with updated credentials. Note: We later discovered the --env parameter was inadvertently omitted for both Wrangler commands. This resulted in credentials being deployed to the Worker assigned to the default environment instead of the Worker assigned to the production environment. |
| Mar 21, 2025 - 20:20 UTC | The R2 Gateway service Worker assigned to the default environment is now using the updated storage infrastructure credentials. Note: This was the wrong Worker, the production environment should have been explicitly set. But, at this point, we incorrectly believed the credentials were updated on the correct production Worker. |
| Mar 21, 2025 - 20:37 UTC | Old credentials were removed from our storage infrastructure to complete the credential rotation process. |
| Mar 21, 2025 - 21:38 UTC | – IMPACT BEGINS – R2 availability metrics begin to show signs of service degradation. The impact to R2 availability metrics was gradual and not immediately obvious because there was a delay in the propagation of the previous credential deletion to storage infrastructure. |
| Mar 21, 2025 - 21:45 UTC | R2 global availability alerts are triggered (indicating 2% of error budget burn rate). The R2 engineering team began looking at operational dashboards and logs to understand impact. |
| Mar 21, 2025 - 21:50 UTC | Internal incident declared. |
| Mar 21, 2025 - 21:51 UTC | R2 engineering team observes gradual but consistent decline in R2 availability metrics for both read and write operations. Operations involving metadata only (e.g., head and list operations) were not impacted. Given gradual decline in availability metrics, R2 engineering team suspected a potential regression in propagation of new credentials in storage infrastructure. |
| Mar 21, 2025 - 22:05 UTC | Public incident status page published. |
| Mar 21, 2025 - 22:15 UTC | R2 engineering team created a new set of credentials (ID and key pair) for storage infrastructure in an attempt to force re-propagation. Continued monitoring operational dashboards and logs. |
| Mar 21, 2025 - 22:20 UTC | R2 engineering team saw no improvement in availability metrics. Continued investigating other potential root causes. |
| Mar 21, 2025 - 22:30 UTC | R2 engineering team deployed a new set of credentials (ID and key pair) to R2 Gateway service Worker. This was to validate whether there was an issue with the credentials we had pushed to gateway service. Environment parameter was still omitted in the deploy and secret put commands, so this deployment was still to the wrong non-production Worker. |
| Mar 21, 2025 - 22:36 UTC | – ROOT CAUSE IDENTIFIED – The R2 engineering team discovered that credentials had been deployed to a non-production Worker by reviewing production Worker release history. |
| Mar 21, 2025 - 22:45 UTC | – IMPACT ENDS – Deployed credentials to correct production Worker. R2 availability recovered. |
| Mar 21, 2025 - 22:54 UTC | The incident is considered resolved. |
分析
R2 的架构主要由三部分组成：R2 生产 Gateway Worker（处理来自 S3 API、REST API、Workers API 的请求）、元数据服务和存储基础设施（存储加密的对象数据）。
%22%2F%3E%3C%2Fsvg%3E)
R2 Gateway Worker 使用凭据 （ID 和密钥对）为我们的分布式存储基础设施进行安全的身份验证。作为最佳安全预防实践，我们会定期轮换这些凭据。
我们的密钥轮换流程涉及以下主要步骤：
- 为我们的存储基础设施创建一组新的凭据（ID 和密钥对）。在此阶段，旧凭据被保留，以避免在凭据切换期间出现停机。
- 使用
wrangler secret put命令设置 R2 生产 Gateway Worker 的新凭据密钥。 - 使用
wrangler deploy命令将新的更新凭据 ID 设置为 R2 生产 Gateway Worker 中的环境变量。此时，Gateway Worker 开始使用新的存储凭据。 - 从我们的存储基础设施中移除以前的凭据，以完成凭据轮换过程。
- 监视运行仪表板和日志以验证切换。
R2 工程团队使用Workers 环境来分隔 R2 Gateway Worker 的生产和开发环境。每个环境都使用单独的环境变量和密钥定义一个单独的隔离 Cloudflare Worker。
关键是，如果不包含 --env 命令行参数，wrangler secret put 和 wrangler deploy 命令都默认指向默认环境。在本例中，由于人为错误，我们无意中忽略了 --env 参数，并将新的存储凭据部署到了错误的 Worker（即 默认环境，而不是生产）。要将存储凭据正确部署到生产 R2 Gateway Worker，我们需要指定 --env production 。
我们在上述第 4 步中执行从存储基础设施中删除旧凭据的操作，导致身份验证错误，因为 R2 Gateway 生产 Worker 仍然拥有旧凭据。这就是最终导致可用性下降的原因。
R2 可用性指标的下降是渐进的，并没有立即显现出来，因为将上一个凭据删除传播到存储基础设施时存在延迟。这是导致我们首次发现问题推迟的原因。在更新旧凭据后，我们不应依赖可用性指标，而是应该显式验证 R2Gateway 服务使用哪个令牌来对 R2 的存储基础设施进行身份验证。
总体而言，由于位于存储前的中间缓存及其继续为请求提供服务，对读取可用性的影响显著降低。
解决方案
在确定根本原因之后，我们通过将新凭据部署到生产 R2 Gateway Worker 来快速解决事件。这导致 R2 可用性立即恢复。
后续步骤
本事件发生是由于人为错误，而持续时间过长是因为我们对 Gateway Worker 用于验证存储基础设施的凭据缺乏适当的可见性。
我们已立即采取措施防止这一故障（及其他类似故障）再次发生：
- 添加了日志记录标记，其中包含 R2 Gateway Worker 用于与我们的存储基础设施进行身份认证的凭据 ID 后缀。通过此变更，我们可以显式确认正在使用的凭证。
- 与上述步骤相关，我们的内部流程现在要求在删除先前的令牌之前，显式确认新令牌 ID 的后缀与我们存储基础设施的日志匹配。
- 要求密钥轮换通过我们的热修补发布工具进行，而不是依赖可能引入人为错误的手动 wrangler 命令输入。我们的热修补发布部署工具显式强制使用环境配置并包含其他安全检查。
- 虽然按照隐含标准，这个过程需要至少两个人来验证将进行的更改，但我们已经更新了相关的 SOP（标准操作程序） 以显式包含这一点。
- 进行中的工作：扩展我们现有的闭环健康检查系统（负责监控端点以测试新密钥），通过我们的告警平台自动报告其状态，并确保在发布 Gateway Worker 之前完成全球传播。
- 进行中的工作：加快对分布式存储端点未来问题的排查，我们正在更新可观测性平台，加入绕过缓存的上游成功率视图，以更清晰地指示任何原因导致的请求服务问题。
上述列表并非详尽无遗：在我们处理上述项目的过程中，除常规工作外，我们很可能会发现需要对系统、控制和流程进行其他改进，并将应用以提升 R2 的韧性。我们相信这些变化将防止此类故障及相关的凭据轮换故障模式再次发生。我们再次为此事件真诚道歉，并对由此给您或您的用户造成的中断深表歉意。
