Subscribe to receive notifications of new posts:

The S3 to R2 Super Slurper is now Generally Available

2023-05-16

3 min read

R2 is Cloudflare’s zero egress fee object storage platform. One of the things that developers love about R2 is how easy it is to get started. With R2’s S3-compatible API, integrating R2 into existing applications only requires changing a couple of lines of code.

However, migrating data from other object storage providers into R2 can still be a challenge. To address this issue, we introduced the beta of R2 Super Slurper late last year. During the beta period, we’ve been able to partner with early adopters on hundreds of successful migrations from S3 to Cloudflare R2. We’ve made many improvements during the beta including speed (up to 5x increase in the number of objects copied per second!), reliability, and the ability to copy data between R2 buckets. Today, we’re proud to announce the general availability of Super Slurper for one-time migration, making data migration a breeze!

Data migration that’s fast, reliable, and easy to use

R2 Super Slurper one-time migration allows you to quickly and easily copy objects from S3 to an R2 bucket of your choice.

Fast

Super Slurper copies objects from your S3 buckets in parallel and uses Cloudflare’s global network to tap into vast amounts of bandwidth to ensure migrations finish fast.

This migration tool is impressively fast! We expected our migration to take a day to complete, but we were able to move all of our data in less than half an hour. - Nick Inhofe, Engineering Manager at PDQ

Reliable

Sending objects through the Internet can sometimes fail. R2 Super Slurper accounts for that, and is capable of driving multi-terabyte migrations to completion with robust retries of failed transfers. Additionally, larger objects are transferred in chunks, so if something goes wrong, it only retries the portion of the object that’s needed. This means faster migrations and lower cost. And if for some reason an object just won’t transfer, it gets logged, so you can keep track and sort it out later.

Easy to use

R2 Super Slurper simplifies the process of copying objects and their associated metadata from S3 to your R2 buckets. Point Super Slurper to your S3 buckets and an asynchronous task will handle the rest. While the migration is taking place, you can follow along from the dashboard.

R2 has saved us both time and money. We migrated millions of images in a short period of time. It wouldn't have been possible for us to build a tool to migrate our data in this amount of time in a cost-effective way. - Damien Capocchi, Backend Engineering Manager at Reelgood

Migrate your S3 data into R2

  1. From the Cloudflare dashboard, expand R2 and select Data Migration.

  2. Select Migrate files.

  3. Enter your Amazon S3 bucket name, optional bucket prefix, and associated credentials and select Next.

  4. Enter your R2 bucket name and associated credentials and select Next.

  5. After you finish reviewing the details of your migration, select Migrate files.

You can view the status of your migration job at any time on the dashboard. If you want to copy data from one R2 bucket to another R2 bucket, you can select Cloudflare R2 as the source bucket provider and follow the same process. For more information on how to use Super Slurper, please see the documentation here.

Next up: Incremental migration

For the majority of cases, a one-time migration of data from your previous object storage bucket to R2 is sufficient; complete the switch from S3 to R2 and immediately watch egress fees go to zero.

However, in some cases you may want to migrate data to R2 incrementally over time (sip by sip if you will). Enter incremental migration, allowing you to do just that.

The goal of incremental migration is to copy files from your origin bucket to R2 as they are requested. When a requested object is not already in the R2 bucket, it is downloaded - one last time - from your origin bucket then copied to R2. From now on, every request for this object will be served by R2, which means less egress fees!

Since data is migrated all within the flow of normal data access and application logic, this means zero cost overhead of unnecessary egress fees! Previously complicated migrations become as easy as replacing your S3 endpoint in your application.

Join the private beta waitlist for incremental migration

We’re excited about our progress making data migration easier, but we’re just getting started. If you’re interested in participating in the private beta for Super Slurper incremental migration, let us know by joining the waitlist here.

We encourage you to join our Discord community to share your R2 experiences, questions, and feedback!

Watch on Cloudflare TV

Cloudflare's connectivity cloud protects entire corporate networks, helps customers build Internet-scale applications efficiently, accelerates any website or Internet application, wards off DDoS attacks, keeps hackers at bay, and can help you on your journey to Zero Trust.

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.

To learn more about our mission to help build a better Internet, start here. If you're looking for a new career direction, check out our open positions.
Developer Week

Follow on X

Phillip Jones|@akaphill
Jérôme Schneider|@jeromeschneider
Cloudflare|@cloudflare

Related posts