Introducing Logpush
Today, we’re excited to announce a new way to get your logs: Logpush, a tool for uploading your logs to your cloud storage provider, such as Amazon S3 or Google Cloud Storage. It’s now available in Early Access for Enterprise domains.
We first explained Cloudflare’s logging functionality almost six years ago. Since then, the number of domains on our network has grown by ten times. We’ve continued giving our Enterprise customers the ability to download logs using a REST API, which has gotten a large number of functional and technical updates. We’ve also been paying attention to how our customers’ needs have evolved, especially as we protect and accelerate increasingly larger domains. This led to the development of Logpush.
The Value of Logs
Cloudflare works by being an intermediary between our customers’ websites, applications, and devices, and their end-users or potential attackers. As part of providing our service, we create a record of each request that goes through our network. These records (or request logs) have detailed information regarding the connecting client, our actions—including whether the request was served by the cache or blocked by our firewall—and the response from the origin web server. For Enterprise customers who request these logs, we save them for up to a week and make them available for download.
Although some of our customers download their logs only when they need to investigate a problem or question, others download them regularly by writing scripts using our “Logpull” API. They may then combine their logs with data from other parts of their infrastructure, such as their application servers or marketing tracking tools. This process allows them to create analytics to see what is happening across all of their platforms; debug issues with their Cloudflare configuration or their own systems; and monitor traffic and make adjustments to improve security or performance. In fact, many download the logs and then upload them to a few common cloud services that have become popular for storage and performing analysis.
We were glad to learn that our logs were proving so useful, but having each customer write their own script simply to download and then upload them to the same few places seemed really inefficient. Couldn’t we just do that on their behalf?
Pushing Instead of Pulling
So that’s the basic idea behind Logpush: rather than writing a script to repeatedly download logs using our Logpull API, simply tell us once where to send them, and we’ll push them there for you. You’ll get the exact same logs either way. If you’re already using Logpull, we’ve made transitioning to Logpush easy by keeping all the functionality the same: you can select the fields you want to receive; change the time format for timestamp fields; and even get a randomly-sampled percentage of logs. When setting up a push job using the Logpush API, you can directly copy all of the previous options you had set in Logpull using the `logpull_options` parameter. We also provide a Logpush UI in our Analytics tab that walks you through the setup for a domain. Full documentation on all of our logging products is available in our new developer documentation section.
Logpush currently works with Amazon S3 and Google Cloud Storage, two of the most popular cloud storage providers. As you may already know, we’re big proponents of working with many cloud services, so more popular destinations are coming soon. Want to help us decide where to push next? Take this survey. Interested in helping us build? We’re hiring Systems Engineers with an interest in data in San Francisco, London, and Austin.