
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/">
    <channel>
        <title><![CDATA[ The Cloudflare Blog ]]></title>
        <description><![CDATA[ Get the latest news on how products at Cloudflare are built, technologies used, and join the teams helping to build a better Internet. ]]></description>
        <link>https://blog.cloudflare.com</link>
        <atom:link href="https://blog.cloudflare.com/" rel="self" type="application/rss+xml"/>
        <language>en-us</language>
        <image>
            <url>https://blog.cloudflare.com/favicon.png</url>
            <title>The Cloudflare Blog</title>
            <link>https://blog.cloudflare.com</link>
        </image>
        <lastBuildDate>Fri, 03 Apr 2026 17:14:36 GMT</lastBuildDate>
        <item>
            <title><![CDATA[Investigating multi-vector attacks in Log Explorer]]></title>
            <link>https://blog.cloudflare.com/investigating-multi-vector-attacks-in-log-explorer/</link>
            <pubDate>Tue, 10 Mar 2026 13:00:00 GMT</pubDate>
            <description><![CDATA[ Log Explorer customers can now identify and investigate multi-vector attacks. Log Explorer supports 14 additional Cloudflare datasets, enabling users to have a 360-degree view of their network. ]]></description>
            <content:encoded><![CDATA[ <p>In the world of cybersecurity, a single data point is rarely the whole story. Modern attackers don’t just knock on the front door; they probe your APIs, flood your network with "noise" to distract your team, and attempt to slide through applications and servers using stolen credentials.</p><p>To stop these multi-vector attacks, you need the full picture. By using Cloudflare Log Explorer to conduct security forensics, you get 360-degree visibility through the integration of 14 new datasets, covering the full surface of Cloudflare’s Application Services and Cloudflare One product portfolios. By correlating telemetry from application-layer HTTP requests, network-layer DDoS and Firewall logs, and Zero Trust Access events, security analysts can significantly reduce Mean Time to Detect (MTTD) and effectively unmask sophisticated, multi-layered attacks.</p><p>Read on to learn more about how Log Explorer gives security teams the ultimate landscape for rapid, deep-dive forensics.</p>
    <div>
      <h2>The flight recorder for your entire stack</h2>
      <a href="#the-flight-recorder-for-your-entire-stack">
        
      </a>
    </div>
    <p>The contemporary digital landscape requires deep, correlated telemetry to defend against adversaries using multiple attack vectors. Raw logs serve as the "flight recorder" for an application, capturing every single interaction, attack attempt, and performance bottleneck. And because Cloudflare sits at the edge, between your users and your servers, all of these events are logged before the requests even reach your infrastructure. </p><p>Cloudflare Log Explorer centralizes these logs into a unified interface for rapid investigation.</p>
    <div>
      <h3>Log Types Supported</h3>
      <a href="#log-types-supported">
        
      </a>
    </div>
    
    <div>
      <h4>Zone-Scoped Logs</h4>
      <a href="#zone-scoped-logs">
        
      </a>
    </div>
    <p><i>Focus: Website traffic, security events, and edge performance.</i></p><table><tr><td><p><b>HTTP Requests</b></p></td><td><p>As the most comprehensive dataset, it serves as the "primary record" of all application-layer traffic, enabling the reconstruction of session activity, exploit attempts, and bot patterns.</p></td></tr><tr><td><p><b>Firewall Events</b></p></td><td><p>Provides critical evidence of blocked or challenged threats, allowing analysts to identify the specific WAF rules, IP reputations, or custom filters that intercepted an attack.</p></td></tr><tr><td><p><b>DNS Logs</b></p></td><td><p>Identify cache poisoning attempts, domain hijacking, and infrastructure-level reconnaissance by tracking every query resolved at the authoritative edge.</p></td></tr><tr><td><p><b>NEL (Network Error Logging) Reports</b></p></td><td><p>Distinguish between a coordinated Layer 7 DDoS attack and legitimate network connectivity issues by tracking client-side browser errors.</p></td></tr><tr><td><p><b>Spectrum Events</b></p></td><td><p>For non-web applications, these logs provide visibility into L4 traffic (TCP/UDP), helping to identify anomalies or brute-force attacks against protocols like SSH, RDP, or custom gaming traffic.</p></td></tr><tr><td><p><b>Page Shield</b></p></td><td><p>Track and audit unauthorized changes to your site's client-side environment such as JavaScript, outbound connections.</p></td></tr><tr><td><p><b>Zaraz Events</b></p></td><td><p>Examine how third-party tools and trackers are interacting with user data, which is vital for auditing privacy compliance and detecting unauthorized script behaviors.</p></td></tr></table>
    <div>
      <h4>Account-Scoped Logs</h4>
      <a href="#account-scoped-logs">
        
      </a>
    </div>
    <p><i>Focus: Internal security, Zero Trust, administrative changes, and network activity.</i></p><table><tr><td><p><b>Access Requests</b></p></td><td><p>Tracks identity-based authentication events to determine which users accessed specific internal applications and whether those attempts were authorized.</p></td></tr><tr><td><p><b>Audit Logs</b></p></td><td><p>Provides a trail of configuration changes within the Cloudflare dashboard to identify unauthorized administrative actions or modifications.</p></td></tr><tr><td><p><b>CASB Findings</b></p></td><td><p>Identifies security misconfigurations and data risks within SaaS applications (like Google Drive or Microsoft 365) to prevent unauthorized data exposure.</p></td></tr><tr><td><p><b>Magic Transit / IPSec Logs</b></p></td><td><p>Helps network engineers perform network-level (L3) monitoring such as reviewing tunnel health and view BGP routing changes.</p></td></tr><tr><td><p><b>Browser Isolation Logs</b></p></td><td><p>Tracks user actions <i>inside</i> an isolated browser session (e.g., copy-paste, print, or file uploads) to prevent data leaks on untrusted sites </p></td></tr><tr><td><p><b>Device Posture Results </b></p></td><td><p>Details the security health and compliance status of devices connecting to your network, helping to identify compromised or non-compliant endpoints.</p></td></tr><tr><td><p><b>DEX Application Tests </b></p></td><td><p>Monitors application performance from the user's perspective, which can help distinguish between a security-related outage and a standard performance degradation.</p></td></tr><tr><td><p><b>DEX Device State Events</b></p></td><td><p>Provides telemetry on the physical state of user devices, useful for correlating hardware or OS-level anomalies with potential security incidents.</p></td></tr><tr><td><p><b>DNS Firewall Logs</b></p></td><td><p>Tracks DNS queries filtered through the DNS Firewall to identify communication with known malicious domains or command-and-control (C2) servers.</p></td></tr><tr><td><p><b>Email Security Alerts</b></p></td><td><p>Logs malicious email activity and phishing attempts detected at the gateway to trace the origin of email-based entry vectors.</p></td></tr><tr><td><p><b>Gateway DNS</b></p></td><td><p>Monitors every DNS query made by users on your network to identify shadow IT, malware callbacks, or domain-generation algorithms (DGAs).</p></td></tr><tr><td><p><b>Gateway HTTP</b></p></td><td><p>Provides full visibility into encrypted and unencrypted web traffic to detect hidden payloads, malicious file downloads, or unauthorized SaaS usage.</p></td></tr><tr><td><p><b>Gateway Network</b></p></td><td><p>Tracks L3/L4 network traffic (non-HTTP) to identify unauthorized port usage, protocol anomalies, or lateral movement within the network.</p></td></tr><tr><td><p><b>IPSec Logs</b></p></td><td><p>Monitors the status and traffic of encrypted site-to-site tunnels to ensure the integrity and availability of secure network connections.</p></td></tr><tr><td><p><b>Magic IDS Detections</b></p></td><td><p>Surfaces matches against intrusion detection signatures to alert investigators to known exploit patterns or malware behavior traversing the network.</p></td></tr><tr><td><p><b>Network Analytics Logs</b></p></td><td><p>Provides high-level visibility into packet-level data to identify volumetric DDoS attacks or unusual traffic spikes targeting specific infrastructure.</p></td></tr><tr><td><p><b>Sinkhole HTTP Logs</b></p></td><td><p>Captures traffic directed to "sinkholed" IP addresses to confirm which internal devices are attempting to communicate with known botnet infrastructure.</p></td></tr><tr><td><p><b>WARP Config Changes</b></p></td><td><p>Tracks modifications to the WARP client settings on end-user devices to ensure that security agents haven't been tampered with or disabled.</p></td></tr><tr><td><p><b>WARP Toggle Changes</b></p></td><td><p>Specifically logs when users enable or disable their secure connectivity, helping to identify periods where a device may have been unprotected.</p></td></tr><tr><td><p><b>Zero Trust Network Session Logs</b></p></td><td><p>Logs the duration and status of authenticated user sessions to map out the complete lifecycle of a user's access within the protected perimeter.</p></td></tr></table>
    <div>
      <h2>Log Explorer can identify malicious activity at every stage</h2>
      <a href="#log-explorer-can-identify-malicious-activity-at-every-stage">
        
      </a>
    </div>
    <p>Get granular application layer visibility with <b>HTTP Requests</b>, <b>Firewall Events</b>, and <b>DNS logs</b> to see exactly how traffic is hitting your public-facing properties.<b> </b>Track internal movement with <b>Access Requests</b>, <b>Gateway logs</b>, and <b>Audit logs</b>. If a credential is compromised, you’ll see where they went. Use <b>Magic IDS</b> and <b>Network Analytics logs</b> to spot volumetric attacks and "East-West" lateral movement within your private network.</p>
    <div>
      <h3>Identify the reconnaissance</h3>
      <a href="#identify-the-reconnaissance">
        
      </a>
    </div>
    <p>Attackers use scanners and other tools to look for entry points, hidden directories, or software vulnerabilities. To identify this, using Log Explorer, you can query <code>http_requests</code> for any <code>EdgeResponseStatus</code> codes of 401, 403, or 404 coming from a single IP, or requests to sensitive paths (e.g. <code>/.env</code>, <code>/.git</code>, <code>/wp-admin</code>). </p><p>Additionally, <code>magic_ids_detections</code> logs can also be used to identify scanning at the network layer. These logs provide packet-level visibility into threats targeting your network. Unlike standard HTTP logs, these logs focus on <b>signature-based detections</b> at the network and transport layers (IP, TCP, UDP). Query to discover cases where a single <code>SourceIP</code> is triggering multiple unique detections across a wide range of <code>DestinationPort</code> values in a short timeframe. Magic IDS signatures can specifically flag activities like Nmap scans or SYN stealth scans.</p>
    <div>
      <h3>Check for diversions</h3>
      <a href="#check-for-diversions">
        
      </a>
    </div>
    <p>While the attacker is conducting reconnaissance, they may attempt to disguise this with a simultaneous network flood. Pivot to <code>network_analytics_logs</code> to see if a volumetric attack is being used as a smokescreen.</p>
    <div>
      <h3>Identify the approach </h3>
      <a href="#identify-the-approach">
        
      </a>
    </div>
    <p>Once attackers identify a potential vulnerability, they begin to craft their weapon. The attacker sends malicious payloads (e.g. SQL injection or large/corrupt file uploads) to confirm the vulnerability. Review <code>http_requests</code> and/or <code>fw_events</code> to identify any Cloudflare detection tools that have triggered. Cloudflare logs security signals in these datasets to easily identify requests with malicious payloads using fields such as <code>WAFAttackScore</code>, <code>WAFSQLiAttackScore</code>, <code>FraudAttack</code>, <code>ContentScanJobResults</code>, and several more. Review <a href="https://developers.cloudflare.com/logs/logpush/logpush-job/datasets/zone/http_requests/"><u>our documentation</u></a> to get a full understanding of these fields. The <code>fw_events</code> logs can be used to determine whether these requests made it past Cloudflare’s defenses by examining the <code>action</code>, <code>source</code>, and <code>ruleID</code> fields. Cloudflare’s managed rules by default blocks many of these payloads by default. Review Application Security Overview to know if your application is protected.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1zpFguYrnbOPwyASGQCqZK/63f398acce2226e453a5eea1cc749241/image3.png" />
          </figure><p><sup><i>Showing the Managed rules Insight that displays on Security Overview if the current zone does not have Managed Rules enabled</i></sup></p>
    <div>
      <h3>Audit the identity</h3>
      <a href="#audit-the-identity">
        
      </a>
    </div>
    <p>Did that suspicious IP manage to log in? Use the <code>ClientIP</code> to search <code>access_requests</code>. If you see a "<code>Decision: Allow</code>" for a sensitive internal app, you know you have a compromised account.</p>
    <div>
      <h3>Stop the leak (data exfiltration)</h3>
      <a href="#stop-the-leak-data-exfiltration">
        
      </a>
    </div>
    <p>Attackers sometimes use DNS tunneling to bypass firewalls by encoding sensitive data (like passwords or SSH keys) into DNS queries. Instead of a normal request like <code>google.com</code>, the logs will show long, encoded strings. Look for an unusually high volume of queries for unique, long, and high-entropy subdomains by examining the fields: <code>QueryName</code>: Look for strings like <a href="http://h3ldo293js92.example.com"><code><u>h3ldo293js92.example.com</u></code></a>, <code>QueryType</code>: Often uses <code>TXT</code>, <code>CNAME</code>, or <code>NULL</code> records to carry the payload, and <code>ClientIP</code>: Identify if a single internal host is generating thousands of these unique requests.</p><p>Additionally, attackers may attempt to leak sensitive data by hiding it within non-standard protocols or by using common protocols (like DNS or ICMP) in unusual ways to bypass standard firewalls. Discover this by querying the <code>magic_ids_detections</code> logs to look for signatures that flag protocol anomalies, such as "ICMP tunneling" or "DNS tunneling" detections in the <code>SignatureMessage</code>.</p><p>Whether you are investigating a zero-day vulnerability or tracking a sophisticated botnet, the data you need is now at your fingertips.</p>
    <div>
      <h2>Correlate across datasets</h2>
      <a href="#correlate-across-datasets">
        
      </a>
    </div>
    <p>Investigate malicious activity across multiple datasets by pivoting between multiple concurrent searches. With Log Explorer, you can now work with multiple queries simultaneously with the new Tabs feature. Switch between tabs to query different datasets or Pivot and adjust queries using filtering via your query results.</p><div>
  
</div>
<p></p><p>When you correlate data across multiple Cloudflare log sources, you can detect sophisticated multi-stage attacks that appear benign when viewed in isolation. This cross-dataset analysis allows you to see the full attack chain from reconnaissance to exfiltration.</p>
    <div>
      <h3>Session hijacking (token theft)</h3>
      <a href="#session-hijacking-token-theft">
        
      </a>
    </div>
    <p><b>Scenario:</b> A user authenticates via Cloudflare Access, but their subsequent HTTP_request traffic looks like a bot.</p><p><b>Step 1:</b> Identify high-risk sessions in <code>http_requests</code>.</p>
            <pre><code>SELECT RayID, ClientIP, ClientRequestUserAgent, BotScore
FROM http_requests
WHERE date = '2026-02-22' 
  AND BotScore &lt; 20 
LIMIT 100</code></pre>
            <p><b>Step 2:</b> Copy the <code>RayID</code> and search <code>access_requests</code> to see which user account is associated with that suspicious bot activity.</p>
            <pre><code>
SELECT Email, IPAddress, Allowed
FROM access_requests
WHERE date = '2026-02-22' 
  AND RayID = 'INSERT_RAY_ID_HERE'</code></pre>
            
    <div>
      <h3>Post-phishing C2 beaconing</h3>
      <a href="#post-phishing-c2-beaconing">
        
      </a>
    </div>
    <p><b>Scenario:</b> An employee clicked a link in a phishing email which resulted in compromising their workstation. This workstation sends a DNS query for a known malicious domain, then immediately triggers an IDS alert.</p><p><b>Step 1:</b> Find phishing attacks by examining email_security_alerts for violations. </p>
            <pre><code>SELECT Timestamp, Threatcategories, To, Alertreason
FROM email_security_alerts
WHERE date = '2026-02-22' 
  AND Threatcategories LIKE 'phishing'</code></pre>
            <p><b>Step 2:</b> Use Access logs to correlate the user’s email (To) to their IP Address.</p>
            <pre><code>SELECT Email, IPAddress
FROM access_requests
WHERE date = '2026-02-22' </code></pre>
            <p><b>Step 3:</b> Find internal IPs querying a specific malicious domain in <code>gateway_dns</code> logs.</p>
            <pre><code>
SELECT SrcIP, QueryName, DstIP, 
FROM gateway_dns
WHERE date = '2026-02-22' 
  AND SrcIP = 'INSERT_IP_FROM_PREVIOUS_QUERY'
  AND QueryName LIKE '%malicious_domain_name%'</code></pre>
            
    <div>
      <h3>Lateral movement (Access → network probing)</h3>
      <a href="#lateral-movement-access-network-probing">
        
      </a>
    </div>
    <p><b>Scenario:</b> A user logs in via Zero Trust and then tries to scan the internal network.</p><p><b>Step 1:</b> Find successful logins from unexpected locations in <code>access_requests</code>.</p>
            <pre><code>SELECT IPAddress, Email, Country
FROM access_requests
WHERE date = '2026-02-22' 
  AND Allowed = true 
  AND Country != 'US' -- Replace with your HQ country</code></pre>
            <p><b>Step 2:</b> Check if that <code>IPAddress</code> is triggering network-level signatures in <code>magic_ids_detections</code>.</p>
            <pre><code>SELECT SignatureMessage, DestinationIP, Protocol
FROM magic_ids_detections
WHERE date = '2026-02-22' 
  AND SourceIP = 'INSERT_IP_ADDRESS_HERE'</code></pre>
            
    <div>
      <h3>Opening doors for more data </h3>
      <a href="#opening-doors-for-more-data">
        
      </a>
    </div>
    <p>From the beginning, Log Explorer was designed with extensibility in mind. Every dataset schema is defined using JSON Schema, a widely-adopted standard for describing the structure and types of JSON data. This design decision has enabled us to easily expand beyond HTTP Requests and Firewall Events to the full breadth of Cloudflare's telemetry. The same schema-driven approach that powered our initial datasets scaled naturally to accommodate Zero Trust logs, network analytics, email security alerts, and everything in between.</p><p>More importantly, this standardization opens the door to ingesting data beyond Cloudflare's native telemetry. Because our ingestion pipeline is schema-driven rather than hard-coded, we're positioned to accept any structured data that can be expressed in JSON format. For security teams managing hybrid environments, this means Log Explorer could eventually serve as a single pane of glass, correlating Cloudflare's edge telemetry with logs from third-party sources, all queryable through the same SQL interface. While today's release focuses on completing coverage of Cloudflare's product portfolio, the architectural groundwork is laid for a future where customers can bring their own data sources with custom schemas.</p>
    <div>
      <h3>Faster data, faster response: architectural upgrades</h3>
      <a href="#faster-data-faster-response-architectural-upgrades">
        
      </a>
    </div>
    <p>To investigate a multi-vector attack effectively, timing is everything. A delay of even a few minutes in the log availability can be the difference between proactive defense and reactive damage control.</p><p>That is why we have optimized our ingestion for better speed and resilience. By increasing concurrency in one part of our ingestion path, we have eliminated bottlenecks that could cause “noisy neighbor” issues, ensuring that one client’s data surge doesn’t slow down another’s visibility. This architectural work has reduced our P99 ingestion latency by approximately 55%, and our P50 by 25%, cutting the time it takes for an event at the edge to become available for your SQL queries.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/41M2eWP0BwrQFSZW4GzZbV/7a6139354abb561aba17e77d83beb17a/image4.png" />
          </figure><p><sup><i>Grafana chart displaying the drop in ingest latency after architectural upgrades</i></sup></p>
    <div>
      <h2>Follow along for more updates</h2>
      <a href="#follow-along-for-more-updates">
        
      </a>
    </div>
    <p>We're just getting started. We're actively working on even more powerful features to further enhance your experience with Log Explorer, including the ability to run these detection queries on a custom defined schedule. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2JIOu9PXDwVAVcmbgq456q/1eace4b5d38bb705e82442a4ee8045dc/Scheduled_Queries_List.png" />
          </figure><p><sup><i>Design mockup of upcoming Log Explorer Scheduled Queries feature</i></sup></p><p><a href="https://blog.cloudflare.com/"><u>Subscribe to the blog</u></a> and keep an eye out for more Log Explorer updates soon in our <a href="https://developers.cloudflare.com/changelog/product/log-explorer/"><u>Change Log</u></a>. </p>
    <div>
      <h2>Get access to Log Explorer</h2>
      <a href="#get-access-to-log-explorer">
        
      </a>
    </div>
    <p>To get access to Log Explorer, you can purchase self-serve directly from the dash or for contract customers, reach out for a <a href="https://www.cloudflare.com/application-services/products/log-explorer/"><u>consultation</u></a> or contact your account manager. Additionally, you can read more in our <a href="https://developers.cloudflare.com/logs/log-explorer/"><u>Developer Documentation</u></a>.</p> ]]></content:encoded>
            <category><![CDATA[Analytics]]></category>
            <category><![CDATA[Logs]]></category>
            <category><![CDATA[Security]]></category>
            <category><![CDATA[R2]]></category>
            <category><![CDATA[Storage]]></category>
            <category><![CDATA[SIEM]]></category>
            <category><![CDATA[Product News]]></category>
            <category><![CDATA[Connectivity Cloud]]></category>
            <guid isPermaLink="false">1hirraqs3droftHovXp1G6</guid>
            <dc:creator>Jen Sells</dc:creator>
            <dc:creator>Claudio Jolowicz</dc:creator>
            <dc:creator>Nico Gutierrez</dc:creator>
        </item>
        <item>
            <title><![CDATA[Cloudflare Log Explorer is now GA, providing native observability and forensics]]></title>
            <link>https://blog.cloudflare.com/logexplorer-ga/</link>
            <pubDate>Wed, 18 Jun 2025 13:00:00 GMT</pubDate>
            <description><![CDATA[ We are happy to announce the General Availability of Cloudflare Log Explorer, a powerful product designed to bring observability and forensics capabilities directly into your Cloudflare dashboard. ]]></description>
            <content:encoded><![CDATA[ <p>We are thrilled to announce the General Availability of <a href="http://cloudflare.com/application-services/products/log-explorer/"><u>Cloudflare Log Explorer</u></a>, a powerful new product designed to bring <a href="https://www.cloudflare.com/learning/performance/what-is-observability/">observability and forensics capabilities</a> directly into your Cloudflare dashboard. Built on the foundation of Cloudflare's vast <a href="https://www.cloudflare.com/network/"><u>global network</u></a>, Log Explorer leverages the unique position of our platform to provide a comprehensive and contextualized view of your environment.</p><p>Security teams and developers use Cloudflare to detect and mitigate threats in real-time and to optimize application performance. Over the years, users have asked for additional telemetry with full context to investigate security incidents or troubleshoot application performance issues without having to forward data to third party log analytics and Security Information and Event Management (SIEM) tools. Besides avoidable costs, forwarding data externally comes with other drawbacks such as: complex setups, delayed access to crucial data, and a frustrating lack of context that complicates quick mitigation. </p><p>Log Explorer has been previewed by several hundred customers over the last year, and they attest to its benefits: </p><blockquote><p><i>“Having WAF logs (firewall events) instantly available in Log Explorer with full context — no waiting, no external tools — has completely changed how we manage our firewall rules. I can spot an issue, adjust the rule with a single click, and immediately see the effect. It’s made tuning for false positives faster, cheaper, and far more effective.” </i></p></blockquote><blockquote><p><i>“While we use Logpush to ingest Cloudflare logs into our SIEM, when our development team needs to analyze logs, it can be more effective to utilize </i><b><i>Log Explorer</i></b><i>. SIEMs make it difficult for development teams to write their own queries and manipulate the console to see the logs they need. Cloudflare's Log Explorer, on the other hand, makes it much </i><b><i>easier</i></b><i> for dev teams to look at logs and directly search for the information they need.”</i></p></blockquote><p>With Log Explorer, customers have access to Cloudflare logs with all the context available within the Cloudflare platform. Compared to external tools, customers benefit from: </p><ul><li><p><b>Reduced cost and complexity:</b> Drastically reduce the expense and operational overhead associated with forwarding, storing, and analyzing terabytes of log data in external tools.</p></li><li><p><b>Faster detection and triage:</b> Access Cloudflare-native logs directly, eliminating cumbersome data pipelines and the ingest lags that delay critical security insights.</p></li><li><p><b>Accelerated investigations with full context:</b> Investigate incidents with Cloudflare's unparalleled contextual data, accelerating your analysis and understanding of "What exactly happened?" and "How did it happen?"</p></li><li><p><b>Minimal recovery time:</b> Seamlessly transition from investigation to action with direct mitigation capabilities via the Cloudflare platform.</p></li></ul><p>Log Explorer is available as an add-on product for customers on our self serve or Enterprise plans. Read on to learn how each of the capabilities of Log Explorer can help you detect and diagnose issues more quickly.</p>
    <div>
      <h3>Monitor security and performance issues with custom dashboards</h3>
      <a href="#monitor-security-and-performance-issues-with-custom-dashboards">
        
      </a>
    </div>
    <p>Custom dashboards allow you to define the specific metrics you need in order to monitor unusual or unexpected activity in your environment.</p><p>Getting started is easy, with the ability to create a chart using natural language. A natural language interface is integrated into the chart create/edit experience, enabling you to describe in your own words the chart you want to create. Similar to the <a href="https://blog.cloudflare.com/security-analytics-ai-assistant/"><u>AI Assistant we announced during Security Week 2024</u></a>, the prompt translates your language to the appropriate chart configuration, which can then be added to a new or existing custom dashboard.</p><p>As an example, you can create a dashboard for monitoring for the presence of Remote Code Execution (RCE) attacks happening in your environment. An RCE attack is where an attacker is able to compromise a machine in your environment and execute commands. The good news is that RCE is a detection available in Cloudflare WAF.  In the dashboard example below, you can not only watch for RCE attacks, but also correlate them with other security events such as malicious content uploads, source IP addresses, and JA3/JA4 fingerprints. Such a scenario could mean one or more machines in your environment are compromised and being used to spread malware — surely, a very high risk incident!</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1UWOHhIaIFiBTtnohdvbAx/40eeac0b52bc278d0687f7d48cd875fd/BLOG-2838_2.png" />
          </figure><p>A reliability engineer might want to create a dashboard for monitoring errors. They could use the natural language prompt to enter a query like “Compare HTTP status code ranges over time.” The AI model then decides the most appropriate visualization and constructs their chart configuration.</p><p>While you can create custom dashboards from scratch, you could also use an expert-curated dashboard template to jumpstart your security and performance monitoring. </p><p>Available templates include: </p><ul><li><p><b>Bot monitoring:</b> Identify automated traffic accessing your website</p></li><li><p><b>API Security:</b> Monitor the data transfer and exceptions of API endpoints within your application</p></li><li><p><b>API Performance:</b> See timing data for API endpoints in your application, along with error rates</p></li><li><p><b>Account Takeover: </b>View login attempts, usage of leaked credentials, and identify account takeover attacks</p></li><li><p><b>Performance Monitoring:</b> Identify slow hosts and paths on your origin server, and view <a href="https://blog.cloudflare.com/ttfb-is-not-what-it-used-to-be/">time to first byte (TTFB)</a> metrics over time</p></li><li><p><b>Security Monitoring:</b> monitor attack distribution across top hosts and paths, correlate DDoS traffic with origin Response time to understand the impact of DDoS attacks.</p></li></ul>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3PO726Rhjol9khGOdMMnQJ/55462052782974b0fc5b0c885e42e41b/BLOG-2838_3.png" />
          </figure>
    <div>
      <h3>Investigate and troubleshoot issues with Log Search </h3>
      <a href="#investigate-and-troubleshoot-issues-with-log-search">
        
      </a>
    </div>
    <p>Continuing with the example from the prior section, after successfully diagnosing that some machines were compromised through the RCE issue, analysts can pivot over to Log Search in order to investigate whether the attacker was able to access and compromise other internal systems. To do that, the analyst could search logs from Zero Trust services, using context, such as compromised IP addresses from the custom dashboard, shown in the screenshot below: </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4iPrTc1ZtLU4ZxQWojvmje/d09bb0bf25bd17cea1d2f955371d991e/BLOG-2838_4.png" />
          </figure><p>Log Search is a streamlined experience including data type-aware search filters, or the ability to switch to a custom SQL interface for more powerful queries. Log searches are also available via a <a href="https://developers.cloudflare.com/logs/log-explorer/"><u>public API</u></a>. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4AytV9wASU5kUuThnhl0CQ/de8c9f4b829e1ccfebdb33bd9522ae5b/BLOG-2838_5.png" />
          </figure>
    <div>
      <h3>Save time and collaborate with saved queries</h3>
      <a href="#save-time-and-collaborate-with-saved-queries">
        
      </a>
    </div>
    <p>Queries built in Log Search can now be saved for repeated use and are accessible to other Log Explorer users in your account. This makes it easier than ever to investigate issues together. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4ouInu3nk7iZnAcJAs39F8/cc7ca6a61d19d3d9c1371ad2ca87e913/BLOG-2838_6.png" />
          </figure>
    <div>
      <h3>Monitor proactively with Custom Alerting (coming soon)</h3>
      <a href="#monitor-proactively-with-custom-alerting-coming-soon">
        
      </a>
    </div>
    <p>With custom alerting, you can configure custom alert policies in order to proactively monitor the indicators that are important to your business. </p><p>Starting from Log Search, define and test your query. From here you can opt to save and configure a schedule interval and alerting policy. The query will run automatically on the schedule you define.</p>
    <div>
      <h4>Tracking error rate for a custom hostname</h4>
      <a href="#tracking-error-rate-for-a-custom-hostname">
        
      </a>
    </div>
    <p>If you want to monitor the error rate for a particular host, you can use this Log Search query to calculate the error rate per time interval:</p>
            <pre><code>SELECT SUBSTRING(EdgeStartTimeStamp, 1, 14) || '00:00' AS time_interval,
       COUNT() AS total_requests,
       COUNT(CASE WHEN EdgeResponseStatus &gt;= 500 THEN 1 ELSE NULL END) AS error_requests,
       COUNT(CASE WHEN EdgeResponseStatus &gt;= 500 THEN 1 ELSE NULL END) * 100.0 / COUNT() AS error_rate_percentage
 FROM http_requests
WHERE EdgeStartTimestamp &gt;= '2025-06-09T20:56:58Z'
  AND EdgeStartTimestamp &lt;= '2025-06-10T21:26:58Z'
  AND ClientRequestHost = 'customhostname.com'
GROUP BY time_interval
ORDER BY time_interval ASC;
</code></pre>
            <p>Running the above query returns the following results. You can see the overall error rate percentage in the far right column of the query results.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5v8SNmHt4OJrLSkiM2EKtJ/182c7f5709eef1fbb9e93c5423fc1bae/BLOG-2838_7.png" />
          </figure>
    <div>
      <h4>Proactively detect malware</h4>
      <a href="#proactively-detect-malware">
        
      </a>
    </div>
    <p>We can identify malware in the environment by monitoring logs from <a href="https://developers.cloudflare.com/cloudflare-one/policies/gateway/">Cloudflare Secure Web Gateway</a>. As an example, <a href="https://www.broadcom.com/support/security-center/protection-bulletin/new-katz-stealer-malware-as-a-service-compromises-web-browsers"><u>Katz Stealer</u></a> is malware-as-a-service designed for stealing credentials. We can monitor DNS queries and HTTP requests from users within the company in order to identify any machines that may be infected with Katz Stealer malware. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7jgBTCWYpnWoNrFh8xe6ki/306e644ec3753976315c16c9d1560eec/BLOG-2838_8.png" />
          </figure>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7jFwBxsk8rfD3VLAYkG2iA/ebd2ebcd95d12b40978f22bf1bc7be39/BLOG-2838_9.png" />
          </figure><p>And with custom alerts, you can configure an alert policy so that you can be notified via webhook or PagerDuty.</p>
    <div>
      <h3>Maintain audit &amp; compliance with flexible retention (coming soon)</h3>
      <a href="#maintain-audit-compliance-with-flexible-retention-coming-soon">
        
      </a>
    </div>
    <p>With flexible retention, you can set the precise length of time you want to store your logs, allowing you to meet specific compliance and audit requirements with ease. Other providers require archiving or hot and cold storage, making it difficult to query older logs. Log Explorer is built on top of our R2 storage tier, so historical logs can be queried as easily as current logs. </p>
    <div>
      <h3>How we built Log Explorer to run at Cloudflare scale</h3>
      <a href="#how-we-built-log-explorer-to-run-at-cloudflare-scale">
        
      </a>
    </div>
    <p>With Log Explorer, we have built a scalable log storage platform on top of <a href="https://www.cloudflare.com/developer-platform/products/r2/"><u>Cloudflare R2</u></a> that lets you efficiently search your Cloudflare logs using familiar SQL queries. In this section, we’ll look into how we did this and how we solved some technical challenges along the way.

Log Explorer consists of three components: ingestors, compactors, and queriers. Ingestors are responsible for writing logs from Cloudflare’s data pipeline to R2. Compactors optimize storage files, so they can be queried more efficiently. Queriers execute SQL queries from users by fetching, transforming, and aggregating matching logs from R2.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1qEH0futV2are5GnT6vjta/e50c0ec4bbb1cacada117d31b71502e2/BLOG-2838_10.png" />
          </figure><p>During ingestion, Log Explorer writes each batch of log records to a Parquet file in R2. <a href="https://parquet.apache.org/"><u>Apache Parquet</u></a> is an open-source columnar storage file format, and it was an obvious choice for us: it’s optimized for efficient data storage and retrieval, such as by embedding metadata like the minimum and maximum values of each column across the file which enables the queriers to quickly locate the data needed to serve the query.</p><p>Log Explorer stores logs on a per-customer level, just like Cloudflare D1, so that your data isn't mixed with that of other customers. In Q3 2025, per-customer logs will allow you the flexibility to create your own retention policies and decide in which regions you want to store your data.

But how does Log Explorer find those Parquet files when you query your logs? Log Explorer leverages the <a href="https://databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf"><u>Delta Lake</u></a> open table format to provide a database table abstraction atop R2 object storage. A table in Delta Lake pairs data files in Parquet format with a transaction log. The transaction log registers every addition, removal, or modification of a data file for the table – it’s stored right next to the data files in R2.</p><p>Given a SQL query for a particular log dataset such as <a href="https://developers.cloudflare.com/logs/reference/log-fields/zone/http_requests/"><u>HTTP Requests</u></a> or <a href="https://developers.cloudflare.com/logs/reference/log-fields/account/gateway_dns/"><u>Gateway DNS</u></a>, Log Explorer first has to load the transaction log of the corresponding Delta table from R2. Transaction logs are checkpointed periodically to avoid having to read the entire table history every time a user queries their logs.</p><p>Besides listing Parquet files for a table, the transaction log also includes per-column min/max statistics for each Parquet file. This has the benefit that Log Explorer only needs to fetch files from R2 that can possibly satisfy a user query. Finally, queriers use the min/max statistics embedded in each Parquet file to decide which row groups to fetch from the file.</p><p>Log Explorer processes SQL queries using <a href="https://arrow.apache.org/datafusion/"><u>Apache DataFusion</u></a>, a fast, extensible query engine written in Rust, and <a href="https://github.com/delta-io/delta-rs"><u>delta-rs</u></a>, a community-driven Rust implementation of the Delta Lake protocol. While standing on the shoulders of giants, our team had to solve some unique problems to provide log search at Cloudflare scale.</p><p>Log Explorer ingests logs from across Cloudflare’s vast global network, <a href="https://www.cloudflare.com/network"><u>spanning more than 330 cities in over 125 countries</u></a>. If Log Explorer were to write logs from our servers straight to R2, its storage would quickly fragment into a myriad of small files, rendering log queries prohibitively expensive.</p><p>Log Explorer’s strategy to avoid this fragmentation is threefold. First, it leverages Cloudflare’s data pipeline, which collects and batches logs from the edge, ultimately buffering each stream of logs in an internal system named <a href="https://blog.cloudflare.com/cloudflare-incident-on-november-14-2024-resulting-in-lost-logs/"><u>Buftee</u></a>. Second, log batches ingested from Buftee aren’t immediately committed to the transaction log; rather, Log Explorer stages commits for multiple batches in an intermediate area and “squashes” these commits before they’re written to the transaction log. Third, once log batches have been committed, a process called compaction merges them into larger files in the background.</p><p>While the open-source implementation of Delta Lake provides compaction out of the box, we soon encountered an issue when using it for our workloads. Stock compaction merges data files to a desired target size S by sorting the files in reverse order of their size and greedily filling bins of size S with them. By merging logs irrespective of their timestamps, this process distributed ingested batches randomly across merged files, destroying data locality. Despite compaction, a user querying for a specific time frame would still end up fetching hundreds or thousands of files from R2.</p><p>For this reason, we wrote a custom compaction algorithm that merges ingested batches in order of their minimum log timestamp, leveraging the min/max statistics mentioned previously. This algorithm reduced the number of overlaps between merged files by two orders of magnitude. As a result, we saw a significant improvement in query performance, with some large queries that had previously taken over a minute completing in just a few seconds.</p>
    <div>
      <h3>Follow along for more updates</h3>
      <a href="#follow-along-for-more-updates">
        
      </a>
    </div>
    <p>We're just getting started! We're actively working on even more powerful features to further enhance your experience with Log Explorer. <a href="https://blog.cloudflare.com/"><u>Subscribe to the blog</u></a> and keep an eye out for more updates in our <a href="https://developers.cloudflare.com/changelog/"><u>Change Log</u></a> to our observability and forensics offering soon.</p>
    <div>
      <h3>Get access to Log Explorer</h3>
      <a href="#get-access-to-log-explorer">
        
      </a>
    </div>
    <p>To get started with Log Explorer, <a href="https://www.cloudflare.com/application-services/products/log-explorer/">sign up here</a> or contact your account manager. You can also read more in our  <a href="https://developers.cloudflare.com/logs/log-explorer/"><u>Developer Documentation</u></a>.</p> ]]></content:encoded>
            <category><![CDATA[undefined]]></category>
            <category><![CDATA[SIEM]]></category>
            <category><![CDATA[Product News]]></category>
            <category><![CDATA[Connectivity Cloud]]></category>
            <category><![CDATA[Analytics]]></category>
            <category><![CDATA[Developer Platform]]></category>
            <category><![CDATA[Developers]]></category>
            <category><![CDATA[Security]]></category>
            <guid isPermaLink="false">kg7dxMzYcRnJdVFrxQmCw</guid>
            <dc:creator>Jen Sells</dc:creator>
            <dc:creator>Claudio Jolowicz</dc:creator>
        </item>
        <item>
            <title><![CDATA[Log Explorer: monitor security events without third-party storage]]></title>
            <link>https://blog.cloudflare.com/log-explorer/</link>
            <pubDate>Fri, 08 Mar 2024 14:05:00 GMT</pubDate>
            <description><![CDATA[ With the combined power of Security Analytics + Log Explorer, security teams can analyze, investigate, and monitor for security attacks natively within Cloudflare, reducing time to resolution and overall cost of ownership for customers by eliminating the need to forward logs to third-party SIEMs ]]></description>
            <content:encoded><![CDATA[ <p></p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1GhVBYNZAsGZtOfgo8C3VY/42fc180d060574162071cbdd13ad6a88/image6-6.png" />
            
            </figure><p>Today, we are excited to announce beta availability of <a href="https://developers.cloudflare.com/logs/log-explorer/">Log Explorer</a>, which allows you to investigate your HTTP and Security Event logs directly from the Cloudflare Dashboard. Log Explorer is an extension of <a href="/security-analytics">Security Analytics</a>, giving you the ability to review related raw logs. You can analyze, investigate, and monitor for security attacks natively within the Cloudflare Dashboard, reducing time to resolution and overall cost of ownership by eliminating the need to forward logs to third party security analysis tools.</p>
    <div>
      <h3>Background</h3>
      <a href="#background">
        
      </a>
    </div>
    <p>Security Analytics enables you to analyze all of your HTTP traffic in one place, giving you the security lens you need to identify and act upon what matters most: potentially malicious traffic that has not been mitigated. Security Analytics includes built-in views such as top statistics and in-context quick filters on an intuitive page layout that enables rapid exploration and validation.</p><p>In order to power our rich analytics dashboards with fast query performance, we implemented <a href="https://developers.cloudflare.com/analytics/graphql-api/sampling/">data sampling</a> using <a href="/explaining-cloudflares-abr-analytics">Adaptive Bit Rate</a> (ABR) analytics. This is a great fit for providing high level aggregate views of the data. However, we received feedback from many Security Analytics power users that sometimes they need access to a more granular view of the data — they need logs.</p><p>Logs provide critical visibility into the operations of today's computer systems. Engineers and SOC analysts rely on logs every day to troubleshoot issues, identify and investigate security incidents, and tune the performance, reliability, and <a href="https://www.cloudflare.com/application-services/solutions/">security</a> of their applications and infrastructure. Traditional metrics or monitoring solutions provide aggregated or statistical data that can be used to identify trends. Metrics are wonderful at identifying THAT an issue happened, but lack the detailed events to help engineers uncover WHY it happened. Engineers and SOC Analysts rely on raw log data to answer questions such as:</p><ul><li><p>What is causing this increase in 403 errors?</p></li><li><p>What data was accessed by this IP address?</p></li><li><p>What was the user experience of this particular user’s session?</p></li></ul><p>Traditionally, these engineers and analysts would stand up a collection of various monitoring tools in order to capture logs and get this visibility. With more organizations using multiple clouds, or a hybrid environment with both cloud and on-premise tools and architecture, it is crucial to have a unified platform to regain visibility into this increasingly complex environment.  As more and more companies are moving towards a cloud native architecture, we see Cloudflare’s <a href="https://www.cloudflare.com/en-gb/learning/cloud/what-is-a-connectivity-cloud/">connectivity cloud</a> as an integral part of their performance and security strategy.</p><p>Log Explorer provides a lower cost option for storing and exploring log data within Cloudflare. Until today, we have offered the ability to export logs to expensive third party tools, and now with Log Explorer, you can quickly and easily explore your log data without leaving the Cloudflare Dashboard.</p>
    <div>
      <h3>Log Explorer Features</h3>
      <a href="#log-explorer-features">
        
      </a>
    </div>
    <p>Whether you're a SOC Engineer investigating potential incidents, or a Compliance Officer with specific log retention requirements, Log Explorer has you covered. It stores your Cloudflare logs for an uncapped and customizable period of time, making them accessible natively within the Cloudflare Dashboard. The supported features include:</p><ul><li><p>Searching through your HTTP Request or Security Event logs</p></li><li><p>Filtering based on any field and a number of standard operators</p></li><li><p>Switching between basic filter mode or SQL query interface</p></li><li><p>Selecting fields to display</p></li><li><p>Viewing log events in tabular format</p></li><li><p>Finding the HTTP request records associated with a Ray ID</p></li></ul>
    <div>
      <h3>Narrow in on unmitigated traffic</h3>
      <a href="#narrow-in-on-unmitigated-traffic">
        
      </a>
    </div>
    <p>As a SOC analyst, your job is to monitor and respond to threats and incidents within your organization’s network. Using Security Analytics, and now with Log Explorer, you can identify anomalies and conduct a forensic investigation all in one place.</p><p>Let’s walk through an example to see this in action:</p><p>On the Security Analytics dashboard, you can see in the Insights panel that there is some traffic that has been tagged as a likely attack, but not mitigated.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5Oq3oqY8JXMigK8OJKPFZ4/5d3a8751a56f06f58e96538f1d46a480/Screenshot-2024-03-07-at-20.20.41.png" />
            
            </figure><p>Clicking the filter button narrows in on these requests for further investigation.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7sWkjUYz1J0So4nphy4FSs/769d5ebb0b706a073a616b706783030c/image11.jpg" />
            
            </figure><p>In the sampled logs view, you can see that most of these requests are coming from a common client IP address.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5gtrP14GKbnB0YeV0ySgL1/6b476ec9d19e255912315eac9730604d/Sampled-logs.png" />
            
            </figure><p>You can also see that Cloudflare has flagged all of these requests as bot traffic. With this information, you can craft a WAF rule to either block all traffic from this IP address, or block all traffic with a bot score lower than 10.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2YgFTRD7u3KYh0bbInLylK/f076a70bc09f41d8ace0569bca172b39/Screenshot-2024-03-07-at-20.22.04.png" />
            
            </figure><p>Let’s say that the Compliance Team would like to gather documentation on the scope and impact of this attack. We can dig further into the logs during this time period to see everything that this attacker attempted to access.</p><p>First, we can use Log Explorer to query HTTP requests from the suspect IP address during the time range of the spike seen in Security Analytics.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/qsi7UnxjtygCQHnMCIx02/cda0aacf6d783b05c15196f27907c611/Log-Explorer.png" />
            
            </figure>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/YyewPADkPzjofXHXSihyi/e494ca5d4b9a3ad7071d5d3e27f57887/Query-results.png" />
            
            </figure><p>We can also review whether the attacker was able to <a href="https://www.cloudflare.com/learning/security/what-is-data-exfiltration/">exfiltrate</a> data by adding the OriginResponseBytes field and updating the query to show requests with OriginResponseBytes &gt; 0. The results show that no data was exfiltrated.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3TgN2aPwWTDo5niA95TGQx/71143be92550dfea1ce284507fa688ac/No-logs-found.png" />
            
            </figure>
    <div>
      <h3>Find and investigate false positives</h3>
      <a href="#find-and-investigate-false-positives">
        
      </a>
    </div>
    <p>With access to the full logs via Log Explorer, you can now perform a search to find specific requests.</p><p>A 403 error occurs when a user’s request to a particular site is blocked. Cloudflare’s security products use things like <a href="/introducing-ip-lists/">IP reputation</a> and <a href="/stop-attacks-before-they-are-known-making-the-cloudflare-waf-smarter/">WAF attack scores</a> based on ML technologies in order to assess whether a given HTTP request is malicious. This is extremely effective, but sometimes requests are mistakenly flagged as malicious and blocked.</p><p>In these situations, we can now use Log Explorer to identify these requests and why they were blocked, and then adjust the relevant WAF rules accordingly.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5ecKfF4lRDQSyLCoOgDjTw/6f0872962caff8d719958e6fdfcc8dbc/Log-Explorer-2.png" />
            
            </figure><p>Or, if you are interested in tracking down a specific request by Ray ID, an identifier given to every request that goes through Cloudflare, you can do that via Log Explorer with one query.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3NCDOP0axFC3wZ4qs03Jei/86286da2fd9fca3cdb68214c1d0f472a/Log-Explorer-3.png" />
            
            </figure><p>Note that the LIMIT clause is included in the query by default, but has no impact on RayID queries as RayID is unique and only one record would be returned when using the RayID filter field.</p>
    <div>
      <h3>How we built Log Explorer</h3>
      <a href="#how-we-built-log-explorer">
        
      </a>
    </div>
    
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3pNBd5iSyVW7aqfJ0JfYDP/89cd0341cc35ad9d86fd8687ba7a9147/How-we-built-Log-Explorer.png" />
            
            </figure><p>With Log Explorer, we have built a long-term, append-only log storage platform on top of <a href="https://www.cloudflare.com/developer-platform/r2/">Cloudflare R2</a>. Log Explorer leverages the <a href="https://databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf">Delta Lake</a> protocol, an open-source storage framework for building highly performant, <a href="https://en.wikipedia.org/wiki/ACID">ACID</a>-compliant databases atop a cloud object store. In other words, Log Explorer combines a large and cost-effective storage system – <a href="www.cloudflare.com/developer-platform/r2/">Cloudflare R2</a> – with the benefits of strong consistency and high performance. Additionally, Log Explorer gives you a SQL interface to your Cloudflare logs.</p><p>Each Log Explorer dataset is stored on a per-customer level, just like Cloudflare D1, so that your data isn't placed with that of other customers. In the future, this single-tenant storage model will give you the flexibility to create your own retention policies and decide in which regions you want to store your data.</p><p>Under the hood, the datasets for each customer are stored as Delta tables in R2 buckets. A <i>Delta table</i> is a storage format that organizes Apache Parquet objects into directories using Hive's partitioning naming convention. Crucially, Delta tables pair these storage objects with an append-only, checkpointed transaction log. This design allows Log Explorer to support multiple writers with optimistic concurrency.</p><p>Many of the products Cloudflare builds are a direct result of the challenges our own team is looking to address. Log Explorer is a perfect example of this <a href="/tag/dogfooding">culture of dogfooding</a>. Optimistic concurrent writes require atomic updates in the underlying object store, and as a result of our needs, R2 added a PutIfAbsent operation with strong consistency. Thanks, R2! The atomic operation sets Log Explorer apart from Delta Lake solutions based on Amazon Web Services’ S3, which incur the operational burden of using an <a href="https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/">external store</a> for synchronizing writes.</p><p>Log Explorer is written in the Rust programming language using open-source libraries, such as <a href="https://github.com/delta-io/delta-rs">delta-rs</a>, a native Rust implementation of the Delta Lake protocol, and <a href="https://arrow.apache.org/datafusion/">Apache Arrow DataFusion</a>, a very fast, extensible query engine. At Cloudflare, Rust has emerged as a popular choice for new product development due to its safety and performance benefits.</p>
    <div>
      <h3>What’s next</h3>
      <a href="#whats-next">
        
      </a>
    </div>
    <p>We know that application security logs are only part of the puzzle in understanding what’s going on in your environment. Stay tuned for future developments including tighter, more seamless integration between Analytics and Log Explorer, the addition of more datasets including Zero Trust logs, the ability to define custom retention periods, and integrated custom alerting.</p><p>Please use the <a href="https://forms.gle/tvKQDdXmCk98zyV9A">feedback link</a> to let us know how Log Explorer is working for you and what else would help make your job easier.</p>
    <div>
      <h3>How to get it</h3>
      <a href="#how-to-get-it">
        
      </a>
    </div>
    <p>We’d love to hear from you! Let us know if you are interested in joining our Beta program by completing <a href="https://cloudflare.com/lp/log-explorer/">this form</a> and a member of our team will contact you.</p><p>Pricing will be finalized prior to a General Availability (GA) launch.</p><div>
  
</div><p>Tune in for more news, announcements and thought-provoking discussions! Don't miss the full <a href="https://cloudflare.tv/shows/security-week">Security Week hub page</a>.</p> ]]></content:encoded>
            <category><![CDATA[Security Week]]></category>
            <category><![CDATA[Analytics]]></category>
            <category><![CDATA[Logs]]></category>
            <category><![CDATA[Security]]></category>
            <category><![CDATA[undefined]]></category>
            <category><![CDATA[SIEM]]></category>
            <category><![CDATA[Product News]]></category>
            <category><![CDATA[Connectivity Cloud]]></category>
            <guid isPermaLink="false">3K5UjFarMC09kkM507HshK</guid>
            <dc:creator>Jen Sells</dc:creator>
            <dc:creator>Claudio Jolowicz</dc:creator>
            <dc:creator>Cole MacKenzie</dc:creator>
        </item>
    </channel>
</rss>