_{One of the several alerts fogos.pt received related to the DDoS attack}

What started in 2015 as a late-night side project with friends around a dinner table in Aveiro has grown into a critical public resource. During wildfires, the site is where firefighters, journalists, citizens, and even government agencies go to understand what’s happening on the ground. Over the years, fogos.pt has evolved from parsing PDFs into visual maps to a full-featured app and website with historical data, weather overlays, and more. It’s also part of Project Galileo, Cloudflare’s initiative to protect vulnerable but important public interest sites at no cost.

Wildfires are not just a Portuguese challenge. They are frequent across southern Europe (Spain, Greece, currently also under alert), California, Australia, and in Canada, which in 2023 faced record-setting fires. In all these cases, reliable information can be crucial, sometimes life-saving. Other organizations offering similar public services can also apply to join Project Galileo to receive protection and handle heavy traffic.

Fogos.pt began with a simple question: why was fire data only available in hard-to-read PDF documents? João and a group of friends, including volunteer firefighters, decided to build something better. They pulled the data, geolocated the fire reports, and visualized them on a map.

Soon, thousands of people were using it. Then tens of thousands. Today, fogos.pt is integrated into official communications, including mentions from the Portuguese government on social media and direct links from the national wildfire information portal (SGIFR.gov.pt).

In 2018, fogos.pt formally joined forces with VOST Portugal, a digital volunteer organization that was early on also part of our Project Galileo — whose story was also featured in an earlier case study. João Pina is also a co-founder of VOSTPT. Together, they created a complementary model: fogos.pt provides data and the platform; VOSTPT validates and communicates it to the public in real-time during emergencies.

It’s an operation run entirely by volunteers, with no funding, no formal team — just passion, and the help of partners.

_{Homepage of fogos.pt on August 20, 2025, highlighting a major wildfire near Piódão in central Portugal.}

On July 31 and August 1, 2025, two Distributed Denial of Service (DDoS) attacks targeted fogos.pt. Cloudflare automatically detected and mitigated both attacks.

July 31 attack: • Duration: 7 minutes • Peak: 33,000 requests per second at 11:27 UTC • Bandwidth: 1.7 Gbps (Max) How the attack looks like in requests per second:

August 1 attack: • Duration: 5 minutes • Peak: 31,000 requests per second at 10:24 UTC • Bandwidth: 849 Mbps (Max) How the attack looks like in requests per second from our perspective:

By Cloudflare’s standards, these were small. For comparison, last year we mitigated an attack exceeding 700,000 requests per second against a high-profile US election campaign site. But for an civic project like fogos.pt, even tens of thousands of requests per second — if unprotected — can be enough to take services offline at the worst possible time.

Attackers typically use three main methods for DDoS attacks:

• IoT devices: hacked cameras, routers, or smart gadgets sending traffic.

• Proxies: open or misconfigured servers, residential proxy networks, or anonymity tools that hide attackers’ IPs.

• Cloud machines: compromised or rented servers from cloud providers.

The July 31 attack likely relied on open proxies, with much of the traffic arriving unencrypted (a common sign of proxy-based attacks). The August 1 attack, in contrast, came largely from cloud machines, matching patterns we see from botnets that exploit cloud infrastructure.

These attacks were blocked without disruption. Cloudflare’s autonomous mitigation systems kicked in, and email alerts were automatically sent to João and the team. No downtime, no manual intervention required.

Fogos.pt has used Cloudflare’s free services since the beginning, starting with DNS and gradually expanding to DDoS mitigation, caching, rate limiting, and more. The site joined Project Galileo, which protects journalists, human rights defenders, and public service projects, to get stronger, upgraded features and service at no cost.

“Without Cloudflare, the site would have gone down many times during fire season,” says João Pina. “We use almost every product — but protection against attacks is critical.”

_{August 11, 2025, detail the area of interest of a wildfire in central Portugal.} 

Traffic to fogos.pt surges when wildfires hit the news or get mentioned by authorities. These spikes can bring tens of thousands of visitors per day. And as attention grows, so does the risk. Attacks can be used to silence or disrupt critical services, or simply as distractions for more malicious activity. In August 2025, the site often had close to 60,000 people browsing at the same time, with around 40,000 being the norm across the web and app services.

In just two weeks (with an August 15 peak of almost 70 million requests), fogos.pt handled over 550 million requests (more than 25 million per day) 9 TB of data transfer, nearly 100 million page views, 15 million visits, and 240 million API calls. A massive load for a volunteer-run project, as the next screenshot from the fogos.pt team shows:

In a time when timely wildfire updates can mean the difference between safety and danger, keeping the site online is essential. 

Fogos.pt is a reminder of what’s possible when public service meets technology, and why we launched Project Galileo: to protect the digital infrastructure that keeps people informed and safe. Built with no formal funding or full-time team, it runs on volunteers, partners, and a shared sense of purpose, an authenticity that João Pina believes is why it works, and why it matters.

And while this story is about Portugal, wildfires are a global challenge. Other organizations providing critical public services can also apply to join Project Galileo and receive this protection.

From a dinner-table idea by an engineer to critical national infrastructure, fogos.pt shows the Internet at its best. Cloudflare is proud to help protect it.

These wave machines are a new source of entropy, joining lava lamps in San Francisco, suspended rainbows in Austin, and double chaotic pendulums in London. The entropy they generate contributes to securing the Internet through LavaRand.

^{The new waves wall at Cloudflare’s Lisbon office sits beside the Radar Display of global Internet insights, with the 25th of April Bridge overlooking the Tagus River in the background.}

It’s exciting to see waves in Portugal now playing a role in keeping the Internet secure, especially given Portugal’s deep maritime history.

The installation honors Portugal’s passion for the sea and exploration of the unknown, famously beginning over 600 years ago, in 1415, with pioneering vessels like caravels and naus/carracks, precursors to galleons and other ships. Portuguese sea exploration was driven by navigation schools and historic voyages “through seas never sailed before” (“Por mares nunca dantes navegados” in Portuguese), as described by Portugal’s famous poet, Luís Vaz de Camões, born 500 years ago (1524).

Anyone familiar with Portugal knows the sea is central to its identity. The small country has 980 km of coastline, where most of its main cities are located. Maritime areas make up 90% of its territory, including the mid-Atlantic Azores. In 1998, Lisbon’s Expo 98 celebrated the oceans and this maritime heritage. Since 2011, the small town of Nazaré also became globally famous among the surfing community for its giant waves.

^{Nazaré’s waves, famous since Garrett McNamara’s 23.8 m (78 ft) ride in 2011, hold Guinness World Records for the biggest waves ever surfed. Photos: Sam Khawasé & Beatriz Paula, from Cloudflare.}

Portugal’s maritime culture also inspired literature and music, including poet Fernando Pessoa, who referenced it in his 1934 book Mensagem, and musician Rui Veloso, who dedicated his 1990s album Auto da Pimenta to Portugal’s historic connection to the sea.

As Cloudflare’s CEO, Matthew Prince, said recently, this new wall of entropy began with an idea back in 2023: “What could we use for randomness that was like our lava lamp wall in San Francisco but represented our team in Portugal?”

The original inspiration came from wave motion machine desk toys, which were popular among some of our team members. Waves and the ocean not only provide a source of movement and randomness, but also align with Portugal’s maritime history and the office’s scenic view.

However, this was easier said than done. It turns out that making a wave machine wall is a real challenge, given that these toys are not as popular as they were in the past,  and aren’t being manufactured in the size we needed any more. We scoured eBay and other sources but couldn't find enough, consistent in style and in working order wave machines. We also discovered that off-the-shelf models weren’t designed to run 24/7, which was a critical requirement for our use.

Undaunted, Cloudflare’s Places team, which ensures our offices reflect our values and culture, found a U.S.-based artisan that specializes in ocean wave displays to create the wave machines for us. Since 2009, his one-person business, Hughes Wave Motion Machines, has blended artistry, engineering, and research, following his transition from Lockheed Martin Space Systems, where he designed military and commercial satellites.

^{Timelapse of the mesmerizing office waves, set to the tune of an AI-generated song.}

Collaborating closely, we developed a custom rectangular wave machine (18 inches/45 cm long) that runs nonstop — not an easy task — which required hundreds of hours of testing and many iterations. Featuring rotating wheels, continuous motors, and a unique fluid formula, these machines create realistic ocean-like waves in green, blue, and Cloudflare’s signature orange. 

Here’s a quote from the artist himself about these wave machines:

“The machine’s design is a balancing act of matching components and their placement to how the fluid responds in a given configuration. There is a complex yet delicate relationship between viscosity, specific gravity, the size and design of the vessel, and the placement of each mechanical interface. Everything must be precisely aligned, centered around the fluid like a mathematical function. I like to say it’s akin to ’balancing a checkerboard on a beach ball in the wind.’”

^{The Cloudflare Places Team with Lisbon office architects and contractor testing wave machine placement, shelves, lighting, and mirrors to enhance movement and reflection, March 2024.}

Despite delays, the Lisbon wave machines finally debuted on March 10, 2025 — an incredibly exciting moment for the Places team.

Some numbers about our wave-machine entropy wall:

• 50 wave machines, 50 motion wheels & motors, 50 acrylic containers filled with Hughes Wave Fluid Formula (two immiscible liquids)

• 3 liquid colors: blue, green, and orange

• 15 months from concept to completion

• 14 flips (side-to-side balancing movements) per minute — over 20,000 per day

• Over 15 waves per minute

• ~0.5 liters of liquid per machine

Cloudflare’s servers handle 71 million HTTP requests per second on average, with 100 million HTTP requests per second at peak. Most of these requests are secured via TLS, which relies on secure randomness for cryptographic integrity. A Cryptographically Secure Pseudorandom Number Generator (CSPRNG) ensures unpredictability, but only when seeded with high-quality entropy. Since chaotic movement in the real world is truly random, Cloudflare designed a system to harness it. Our 2024 blog post expands on this topic in a more technical way, but here’s a quick summary.

In 2017, Cloudflare launched LavaRand, inspired by Silicon Graphics’ 1997 concept However, the need for randomness in security was already a hot topic on our blog before that, such as in our discussions of securing systems and cryptography. Originally, LavaRand collected entropy from a wall of lava lamps in our San Francisco office, feeding an internal API that servers periodically query to include in their entropy pools. Over time, we expanded LavaRand beyond lava lamps, incorporating new sources of office chaos while maintaining the same core method.

A camera captures images of dynamic, unpredictable randomness displays. Shadows, lighting changes, and even sensor noise contribute entropy. Each image is then processed into a compact hash, converting it into a sequence of random bytes. These, combined with the previous seed and local system entropy, serve as input for a Key Derivation Function (KDF), which generates a new seed for a CSPRNG — capable of producing virtually unlimited random bytes upon request. The waves in our Lisbon office are now contributing to this pool of randomness.

Cloudflare’s LavaRand API makes this randomness accessible internally, strengthening cryptographic security across our global infrastructure. For example, when you use Math.random() in Cloudflare Workers, part of that randomness comes from LavaRand. Similarly, querying our drand API taps into LavaRand as well. Cloudflare offers this API to enable anyone to generate random numbers and even seed their own systems.

^{Photo of the view from our Lisbon office, featuring ceiling lights arranged in a wave-like pattern.}

Entropy also inspired the design ethos of our new Lisbon office, given that the wall of waves and the office are part of the same project. As soon as you enter, you're greeted not only by the motion of the entropy wall but also by the constant movement of planet Earth on our Cloudflare Radar Display screen that stands next to it. But the waves don’t stop there — more elements throughout the space mimic the dynamic flow of the Internet itself. Unlike ocean tides, however, Internet traffic ebbs and flows with the motion of the Sun, not the Moon.

As you walk through the office, waves are everywhere — in the ceiling lights, the architectural contours, and even the floor plan, thoughtfully designed by our architect to reflect the fluid movement of water. The visual elements create a cohesive experience, reinforcing a sense of motion. Each meeting room embraces this maritime theme, named after famous Portuguese beaches — including, naturally, Nazaré.

We partnered with an incredible group of local Portuguese vendors for this construction project, where all the leads were women — something incredibly rare for the industry. The local teams worked with passion, proudly wore Cloudflare t-shirts, and fostered a warm, family-like atmosphere. They openly expressed pride in the project, sharing how it stood out from anything they had worked on before.

^{Our amazing third-party team and internal Places team, proudly rocking Cloudflare shirts after bringing this project to life.}

Next, we have several name options for this new wall of entropy. Help us decide the best one, and register your vote using this form.

The Surf Board

Chaos Reef

Waves of Entropy

Wall of Waves

Whirling Wave Wall

Chaotic Wave Wall

Waves of Chaos

If you’re interested in working in Cloudflare’s Lisbon office, we’re hiring! Our career page lists our open roles in Lisbon, as well as our other locations in the U.S., Mexico, Europe and Asia.

^{Acknowledgements: This project was only possible with the effort, vision and help of John Graham-Cumming, Caroline Quick, Jen Preston, Laura Atwall, Carolina Beja, Hughes Wave Motion Machines, P4 Planning and Project Management, Gensler Europe, Openbook Architecture, and Vector Mais.}

Justina Wong, Technical Support Team Lead in Lisbon, talks about what it’s like working at Cloudflare, and everything you need to know if you want to join us.

Justina joined Cloudflare about three years ago in London as a Technical Support Engineer. Currently, she’s part of their Customer Support team working in Lisbon as a team lead.

I can’t speak for others, but I love the things you can learn from the others. There are so many talented individuals who are willing and ready to teach/share. They are my inspiration and I want to become them!

Justina’s favourite Cloudflare products are firewall-related ones. The company’s primary care is for the customers and they want to make attack mitigation as easy as possible. As she puts it, “the fact that these protections are on multiple layers, like L7, L3/4, is very important, and I’m proud to be someone who can help our customers when they face certain attacks.”.

Cloudflare is constantly releasing new products to help build a better Internet, so product managers are always on top of tool updates to facilitate that. The company believes that it’s not only important to help customers from the product side, but it’s also as important to teach them how to help themselves so that they can fix their issues promptly without having to wait for an answer.

According to Justina, one of the amazing things about Cloudflare is the unified company culture. As their SVP of Engineering, Usman, said in a recent meeting with the team, “Be helpful, look around for problems and help find solutions”.

Every Cloudflare office has its own little “flare”: London’s love of mince pies; Singapore’s super fun cultural richness in one location (they have four new years in one year, officially); and Lisbon’s forever love (and fight) for pastéis de nata.

Each office also has its own function or focus, so people working at Cloudflare get to meet very diverse individuals. For Justina, the things that she'd loved the most are learning from all of the engineers in London, picking up new customer service skills in Singapore and helping to build the new Lisbon office. She says that every time she goes to a different office, they have grown at least 50% in headcount compared to when she was last there. Talk about growth!

As a hiring manager, she also says that the company is mindful of diversity.

Like everywhere else, remote work has become the current normal at Cloudflare. As someone who enjoyed being in the office, Justina says “all the countless times I just walked over to someone to ask a question, now all turned into a chat message; or the random coffee chat when we waited for our coffee to be done.”

Funnily enough, the EMEA CSUP team is working closer than before the pandemic. Previously, each office was somewhat in its own communication bubble, now it has turned into a collective conversation. This is great for getting to know colleagues during and beyond work hours.

For Cloudflare, growing the team is a continuous challenge, and Justina has never needed to do as many interviews as she has done in the Lisbon office. Although it’s a huge challenge for her, it’s also fun. Since the company is hiring aggressively despite the pandemic, their teams are eager to welcome anyone who’s ready to be part of Lisbon Cloudflare.

One of the things you can expect if you work at Cloudflare is for your manager to care and for your feedback to be heard. We know these are valuable things when considering where to work. So if you’re someone who’s willing to learn and is excited about their technologies, this call is for you. The company is expanding in different markets, so they’re looking for tech candidates who can speak multiple languages.

Currently, Cloudflare has over 25 open positions for their offices in Lisbon. Categories include Security Engineers, Full-Stack Developers, Data Scientists, and more.

It has now been more than 90 days since I joined Cloudflare’s EMEA Recruiting Team as a Recruiting Coordinator based in Lisbon. In a year filled with hardships for so many people around the world, I wanted to share my journey. I hope people will relate and feel encouraged to pursue their dreams, even during these challenging times.

When 2020 started, it was not in my plans to change jobs and start working at a new company, completely remote, without ever meeting my colleagues in person or visiting the office. However, that is exactly what happened, and I am so glad I did.

The number of interviews in the hiring process at Cloudflare may feel overwhelming for some - in my case, I met 11 people during this process. For me, I was glad to have so many chances to get to know the people I would be working with. I believe I got as much out of the conversations as the interviewers did, which is great — a recruitment process should be as much about the company getting to know you, as you getting to know the company.

A great thing about interviewing remotely is that I got the chance to talk to people all around the globe, which enriched the process and my idea of Cloudflare as a company. I started to picture myself as an actual member of the team, definitely interested in working towards a better and safer Internet. Even though there were many interviews to get through, the constant communication with the team made me feel engaged and excited. In the end, the process went by quickly, even quicker than I expected.

The best thing was the outpouring of support I received from what would be my future teammates once I accepted the offer. I felt welcomed way before my actual start date!

In all my previous companies, onboarding was done in person and small groups. I was not prepared for a fully remote experience with a class of more than 20 people, yet it was so smooth and well-coordinated that you wouldn’t believe it had been run virtually for only a few months!

My onboarding class included people from all over the world — Lisbon, Austin, Miami, Washington, London, Munich, Singapore… And not only that, but we were all starting different roles, from Customer Success to Engineering, and even Legal Counsel! This gave me the opportunity to know people I otherwise wouldn’t have had the chance to meet, and it allowed me to establish bonds early on with my colleagues. Given the current situation, knowing that people were in the same boat with me felt reassuring. I felt that we were in it together, in a way. Not only that, but I got everything I needed for work (and more — like a pair of Cloudflare socks!) delivered to my home, making the whole experience very comfortable for me.

Starting in a new role can be a daunting experience — it’s a new environment, a new team, a new project, and lots of things that could go sideways. However, there are also a lot of things that can go right!

At Cloudflare, I found an extremely welcoming, supportive team that helped me ramp up and take ownership of my work quickly and effectively. I felt so supported that I took ownership of a big project right away — Cloudflare Careers Day. Right from the start, it was clear to me that Cloudflare has ambitious goals for the growth of our Lisbon office. I thought about the ways I could help with that, and a virtual careers day seemed like a great first step to drive brand awareness and let people know we are hiring and that we are hiring! The Recruitment Team set in motion a plan to turn this idea into reality in less than three months, resulting in a successful and fun first edition of the Cloudflare Careers Day in November 2020.

Of course, there were times when I felt unsure of myself and my abilities. But this is why it is so important to be able to rely on your team. In the end, I feel I have grown a lot in just three months — not only professionally, but personally as well!

I look forward to working on more projects. I’m excited to write with this blog post, which I hope will inspire more people to take a chance, believe in themselves and just go for it! Even in these strange, stressful times, good things can and do happen, especially when you are surrounded by talented, inspiring people.

Lisbon! I am excited to help grow our Lisbon office, recruiting talented people that feel as strongly as I do about helping build a better Internet. We have many different open roles at the moment so, if you see one that suits you, take a chance and reach out. Maybe you’ll embark on a new journey, just like me.

Our Lisbon story is just beginning. I can’t wait to see all the amazing things we will accomplish in 2021, both as a team and as a company.

It's the end of the year, so we thought it would be a great time to give you an update on how we're doing and what we're planning for 2021. If you're reading this, you know we like to share everything we do at Cloudflare, including how the organization is evolving.

In July, John Graham-Cumming wrote a blog post entitled Cloudflare's first year in Lisbon. and showed how we went from an announcement, just a few months before, to an entirely bootstrapped and fully functional office. At the time, despite a ramping pandemic, the team was already hard at work doing a fantastic job scaling up and solidifying our presence here.

A few weeks later, in August, I proudly joined the team.

Cloudflare is, by any standard, a big company. There's a lot you need to learn, many people you need to get to know first, and a lot of setup steps you need to get through before you're in a position to do actual real productive work.

Joining the company during COVID was challenging. I felt just as excited as I was scared. We were (and still are) fully working from home, I didn't have a team to work with in person. A setup like this surely looks daunting, even for experienced people.

But here's the thing. Cloudflare isn’t just any company. We're unparalleled because we masterfully combine scale, ambition, talent, product, vision, values, and culture in a way that's very difficult to replicate and maintain at any other company.

We're big, but we move fast. We're over 1,600 working together, but it feels like a cohesive group. We're distributed across multiple offices and continents, often working in teams with members from different time zones, but we don't notice it. We have tools, documentation, and methodologies, but they don't get in the way of our “shipping products'' mantra. There are product owners, teams for specific features, but we all hold ownership for everyone's work.

I felt all of this right after my orientation week. The warm welcome, the regular check-ins to say hello and see how I was doing, and everyone's urge to make sure I was adjusting and getting all the help I needed, giving me advice, introducing me to other colleagues. Cloudflarians take genuine pride in making sure everyone feels at home. You can learn more about this experience from a Story Time segment John did with me.

Cloudflare Lisbon has come a long way. We now have 74 incredibly talented people working or joining in areas such as Engineering, Security, Infrastructure, Customer Support, People, Places, Product Management, Emerging Technologies or Accounting, and growing fast.

Although the pandemic didn't help our plans, especially those related to growing and physically working in our brand new office on Praça Marquês de Pombal, it didn't slow us down either. November and December alone, 15 people joined the team. We're gaining momentum.

More interestingly, we have a super diverse team in Lisbon, and we couldn't be prouder of it. We're putting action ahead of words and actively contributing to create more opportunities for women in technology and to attract people to work in Portugal regardless of their country of origin.

Our discussions on whether "Pastéis de Nata" is best served with or without cinnamon, our holiday traditions, Portuguese music, coffee, our frequent virtual Pub Quizzes, escape room events, and of course, the comments on shirtless Marcelo are now routine. They are evidence that we feel like a group working together, having fun while growing.

We live in unusual and contemplative times. Many of our emigrants living outside the country are considering returning home to Portugal and our office in Lisbon is proof of this growing movement. Portuguese returnees represent roughly 10% of our team.

The Portuguese Government has an initiative called "Programa Regressar," where they provide tax benefits and financial assistance to support emigrants and their families returning to Portugal.

While this is great, we think it's not enough. Moving you and your family to another country is a life-changing event. Although things like patriotism, cost of living, and tax incentives play an essential role in the personal decision process, skilled and talented people will also be looking for a great workplace and a meaningful, ambitious company to join.

This is where Cloudflare can help you. We can provide you the best of the two worlds. Living in a beautiful country, your home, while working in a world-class company, solving big problems at scale on a mission to help build a better Internet with a unique culture. Furthermore, we support your return, and we're ready to help you in any way we can.

Cloudflare is serious about its presence in Portugal. We're going to continue growing and investing in highly skilled talent for our Lisbon office and making it one of Cloudflare's top locations, alongside San Francisco, Austin, Singapore, and London.

Currently, we have 28 open positions in Lisbon, and you can expect new ones to open over the upcoming weeks. Some are for teams based in Lisbon, like Data Insights and Cloudflare Radar (we're doubling in 2021), while others will join different projects, some of which have teams distributed across multiple offices.

If you decide to apply, there are many resources you can use to learn about Cloudflare and improve your chances of snatching your dream job. Here are a few:

• Cloudflare Careers Day: Meet the Engineering Team with Isabel Rodrigues, Jen Langdon, and me.

• Our Blog. We share an unusual amount of information about our infrastructure and products, our technical decisions, architecture, and our approach to solving complex, large-scale technical challenges.

• Our Official GitHub Page. We have open-source encoded all over in our DNA, and we like to give back to the community whenever possible. Cloudflare has over 300 public projects that you can explore, try them yourself, or fork.

• Our Developers Website, where you can learn about our products, the way they work, their features, and APIs. Speaking of APIs, take a look at cloudflare-go and flarectl.

• Check our community Forum, ask us questions; we're always there for you, you'll be surprised. Follow us on Twitter.

• Cloudflare TV airs excellent content all the time. You can check our schedule for numerous live segments with the team and guests or re-run past segments. We also have a "Best of" archive.

• Finally, you can try our products. As part of our mission and values, we offer very generous free tiers to individual users and small startups. You can try our CDN features, DDoS, Workers (100,000 requests per day, with Workers KV included), and even Access for Teams (with Argo tunnel included, for companies or households under 50 seats), at no cost.

We're a highly ambitious, large-scale technology company with a soul. Fundamental to our mission to help build a better Internet is protecting the free and open Internet. Cloudflare powers Internet requests for ~16% of the Fortune 1,000 and serves 20 million HTTP requests per second on average.

A year ago I wrote about the opening of Cloudflare’s office in Lisbon, it’s hard to believe that a year has flown by. At the time I wrote:

Lisbon’s combination of a large and growing existing tech ecosystem, attractive immigration policy, political stability, high standard of living, as well as logistical factors like time zone (the same as the UK) and direct flights to San Francisco made it the clear winner.

We landed in Lisbon with a small team of transplants from other Cloudflare offices. Twelve of us moved from the UK, US and Singapore to bootstrap here. Today we are 35 people with another 10 having accepted offers; we’ve almost quadrupled in a year and we intend to keep growing to around 80 by the end of 2020.

If you read back to my description of why we chose Lisbon only one item hasn’t turned out quite as we expected. Sure enough TAP Portugal does have direct flights to San Francisco but the pandemic put an end to all business flying worldwide for Cloudflare. We all look forward to getting back to being able to visit our colleagues in other locations.

The pandemic also put us in the odd position of needing to move from one empty office to another. Back in January the Cloudflare Lisbon office was in the Chiado and only had capacity for about 14 people. With our rapid growth we moved, in February, to a larger, temporary location on Avenida da Liberdade which had room for about 25 people.

Leaving the Chiado‌‌

And in early April, we moved to our longer term office on Praça Marquês de Pombal. Of course, by that time the State of Emergency had been declared in Portugal and the office move took place in our absence. But it sits waiting for our return sometime in early 2021.

The team that landed in Lisbon covered Customer Support, Security, IT, Technology, and  Emerging Technology and Incubation, but, as we suspected, we’ve grown in many other departments and the rest of Cloudflare is realizing how much Lisbon and Portugal have to offer. In addition to the original team we now have people in SRE, Payroll, Accounting, Trust and Safety, People and Places, Product Management and Infrastructure.

View from the Cloudflare Lisbon office‌‌

Despite the pandemic we’re continuing to invest in Lisbon with 24 open roles in Customer Support, Infrastructure, People and Places, Engineering, Accounting and Finance, Security, Business Intelligence, Product Management and Emerging Technology and Incubation.

As I said in an interview with AICEP earlier this year “É nosso objetivo construir em Lisboa um dos maiores escritórios da Cloudflare” (“It’s our objective to build in Lisbon one of the major Cloudflare offices”). You can read the full Portuguese-language interview here. We continue to believe that Lisbon is a vital part of Cloudflare’s growth.

I’ve spent a huge amount of my career on aircraft and the last few months have felt very odd, but I couldn’t have been happier to find myself temporarily stuck in Lisbon. No doubt we’ll all be traveling again but this last year has confirmed my impression that Lisbon is a great place to live.

I asked our team what they’d found they love about living in Lisbon and Portugal. They came back with pasteis de nata, sunshine every day, the jacaranda trees, feijoada, empada de galinha, Joker, Super Bock, chocolate mousse being an everyday staple, Maria biscuits, quality fresh produce, dolphins, lizards in the gardens, MB Way, ovos moles de Aveiro, so great that only ~30/40min from here you get such nice beaches like the ones in Setubal, Sintra, Cascais, Sesimbra, bica, sardines, the Alentejo coastline, the chicken from Bonjardim, family friendliness and how nice it is to raise children here, fast, reliable and cheap Internet access, and so much more.

If you’d like to join us please visit our careers page for Lisbon.

Image courtesy of Unbabel

This is the text I used for a talk at artificial intelligence powered translation platform, Unbabel, in Lisbon on September 25, 2019.

Bom dia. Eu sou John Graham-Cumming o CTO do Cloudflare. E agora eu vou falar em inglês.

Thanks for inviting me to talk about Cloudflare and how we think about security. I’m about to move to Portugal permanently so I hope I’ll be able to do this talk in Portuguese in a few months.

I know that most of you don’t have English as a first language so I’m going to speak a little more deliberately than usual. And I’ll make the text of this talk available for you to read.

But there are no slides today.

I’m going to talk about how Cloudflare thinks about internal security, how we protect ourselves and how we secure our day to day work. This isn’t a talk about Cloudflare’s products.

Let’s begin with culture.

Many companies have culture statements. I think almost 100% of these are pure nonsense. Culture is how you act every day, not words written in the wall.

One significant piece of company culture is the internal Security Incident mailing list which anyone in the company can send a message to. And they do! So far this month there have been 55 separate emails to that list reporting a security problem.

These mails come from all over the company, from every department. Two to three per day. And each mail is investigated by the internal security team. Each mail is assigned a Security Incident issue in our internal Atlassian Jira instance.

People send: reports that their laptop or phone has been stolen (their credentials get immediately invalidated), suspicions about a weird email that they’ve received (it might be phishing or malware in an attachment), a concern about physical security (for example, someone wanders into the office and starts asking odd questions), that they clicked on a bad link, that they lost their access card, and, occasionally, a security concern about our product.

Things like stolen or lost laptops and phones happen way more often than you’d imagine. We seem to lose about two per month. For that reason and many others we use full disk encryption on devices, complex passwords and two factor auth on every service employees need to access. And we discourage anyone storing anything on their laptop and ask them to primarily use cloud apps for work. Plus we centrally manage machines and can remote wipe.

We have a 100% blame free culture. You clicked on a weird link? We’ll help you. Lost your phone? We’ll help you. Think you might have been phished? We’ll help you.

This has led to a culture of reporting problems, however minor, when they occur. It’s our first line of internal defense.

Just this month I clicked on a link that sent my web browser crazy hopping through redirects until I ended up at a bad place. I reported that to the mailing list.

I’ve never worked anywhere with such a strong culture of reporting security problems big and small.

We also use HackerOne to let people report security problems from the outside. This month we’ve received 14 reports of security problems. To be honest, most of what we receive through HackerOne is very low priority. People run automated scanning tools and report the smallest of configuration problems, or, quite often, things that they don’t understand but that look like security problems to them. But we triage and handle them all.

And people do on occasion report things that we need to fix.

We also have a private paid bug bounty program where we work with a group of individual hackers (around 150 right now) who get paid for the vulnerabilities that they’ve found.

We’ve found that this combination of a public responsible disclosure program and then a private paid program is working well. We invite the best hackers who come in through the public program to work with us closely in the private program.

So, that’s all about people, internal and external, reporting problems, vulnerabilities, or attacks. A very short step from that is knowing who the people are.

And that’s where identity and authentication become critical. In fact, as an industry trend identity management and authentication are one of the biggest areas of spending by CSOs and CISOs. And Cloudflare is no different.

OK, well it is different, instead of spending a lot of identity and authentication we’ve built our own solutions.

We did not always have good identity practices. In fact, for many years our systems had different logins and passwords and it was a complete mess. When a new employee started accounts had to be made on Google for email and calendar, on Atlassian for Jira and Wiki, on the VPN, on the WiFi network and then on a myriad of other systems for the blog, HR, SSH, build systems, etc. etc.

And when someone left all that had to be undone. And frequently this was done incorrectly. People would leave and accounts would still be left running for a period of time. This was a huge headache for us and is a huge headache for literally every company.

If I could tell companies one thing they can do to improve their security it would be: sort out identity and authentication. We did and it made things so much better.

This makes the process of bringing someone on board much smoother and the same when they leave. We can control who accesses what systems from a single control panel.

I have one login via a product we built called Cloudflare Access and I can get access to pretty much everything. I looked in my LastPass Vault while writing this talk and there are a total of just five username and password combination and two of those needed deleting because we’ve migrated those systems to Access.

So, yes, we use password managers. And we lock down everything with high quality passwords and two factor authentication. Everyone at Cloudflare has a Yubikey and access to TOTP (such as Google Authenticator). There are three golden rules: all passwords should be created by the password manager, all authentication has to have a second factor and the second factor cannot be SMS.

We had great fun rolling out Yubikeys to the company because we did it during our annual retreat in a single company wide sitting. Each year Cloudflare gets the entire company together (now over 1,000 people) in a hotel for two to three days of working together, learning from outside experts and physical and cultural activities.

Last year the security team gave everyone a pair of physical security tokens (a Yubikey and a Titan Key from Google for Bluetooth) and in an epic session configured everyone’s accounts to use them.

Note: do not attempt to get 500 people to sync Bluetooth devices in the same room at the same time. Bluetooth cannot cope.

Another important thing we implemented is automatic timeout of access to a system. If you don’t use access to a system you lose it. That way we don’t have accounts that might have access to sensitive systems that could potentially be exploited.

To return to the subject of Culture for a moment an important Cloudflare trait is openness.

Some of you may know that back in 2017 Cloudflare had a horrible bug in our software that became called Cloudbleed. This bug leaked memory from inside our servers into people’s web browsing. Some of that web browsing was being done by search engine crawlers and ended up in the caches of search engines like Google.

We had to do two things: stop the actual bug (this was relatively easy and was done in under an hour) and then clean up the equivalent of an oil spill of data. That took longer (about a week to ten days) and was very complicated.

But from the very first night when we were informed of the problem we began documenting what had happened and what were doing. I opened an EMACS buffer in the dead of night and started keeping a record.

That record turned into a giant disclosure blog post that contained the gory details of the error we made, its consequences and how we reacted once the error was known.

We followed up a few days later with a further long blog post assessing the impact and risk associated with the problem.

This approach to being totally open ended up being a huge success for us. It increased trust in our product and made people want to work with us more.

I was on my way to Berlin to give a talk to a large retailer about Cloudbleed when I suddenly realized that the company I was giving the talk at was NOT a customer. And I asked the salesperson I was with what I was doing.

I walked in to their 1,000 person engineering team all assembled to hear my talk. Afterwards the VP of Engineering thanked me saying that our transparency had made them want to work with us rather than their current vendor. My talk was really a sales pitch.

Similarly, at RSA last year I gave a talk about Cloudbleed and a very large company’s CSO came up and asked to use my talk internally to try to encourage their company to be so open.

When on July 2 this year we had an outage, which wasn’t security related, we once again blogged in incredible detail about what happened. And once again we heard from people about how our transparency mattered to them.

The lesson is that being open about mistakes increases trust. And if people trust you then they’ll tend to tell you when there are problems. I get a ton of reports of potential security problems via Twitter or email.

After Cloudbleed we started changing how we write software. Cloudbleed was caused, in part, by the use of memory-unsafe languages. In that case it was C code that could run past the end of a buffer.

We didn’t want that to happen again and so we’ve prioritized languages where that simply cannot happen. Such as Go and Rust. We were very well known for using Go. If you’ve ever visited a Cloudflare website, or used an app (and you have because of our scale) that uses us for its API then you’ve first done a DNS query to one of our servers.

That DNS query will have been responded to by a Go program called RRDNS.

There’s also a lot of Rust being written at Cloudflare and some of our newer products are being created using it. For example, Firewall Rules which do arbitrary filtering of requests to our customers are handled by a Rust program that needs to be low latency, stable and secure.

The other post-Cloudbleed change was that any crashes on our machines came under the spotlight from the very top. If a process crashes I personally get emailed about it. And if the team doesn’t take those crashes seriously they get me poking at them until they do.

We missed the fact that Cloudbleed was crashing our machines and we won’t let that happen again. We use Sentry to correlate information about crashes and the Sentry output is one of the first things I look at in the morning.

Which, I think, brings up an important point. I spoke earlier about our culture of “If you see something weird, say something” but it’s equally important that security comes from the top down.

Our CSO, Joe Sullivan, doesn’t report to me, he reports to the CEO. That sends a clear message about where security sits in the company. But, also, the security team itself isn’t sitting quietly in the corner securing everything.

They are setting standards, acting as trusted advisors, and helping deal with incidents. But their biggest role is to be a source of knowledge for the rest of the company. Everyone at Cloudflare plays a role in keeping us secure.

You might expect me to have access to our all our systems, a passcard that gets me into any room, a login for any service. But the opposite is true: I don’t have access to most things. I don’t need it to get my job done and so I don’t have it.

This makes me a less attractive target for hackers, and we apply the same rule to everyone. If you don’t need access for your job you don’t get it. That’s made a lot easier by the identity and authentication systems and by our rule about timing out access if you don’t use a service. You probably didn’t need it in the first place.

The flip side of all of us owning security is that deliberately doing the wrong thing has severe consequences.

Making a mistake is just fine. The person who wrote the bad line of code that caused Cloudbleed didn’t get fired, the person who wrote the bad regex that brought our service to a halt on July 2 is still with us.‌‌

Naturally, things do go wrong internally. Things that didn’t get reported. To do with them we need to detect problems quickly. This is an area where the security team does have real expertise and data.‌‌

We do this by collecting data about how our endpoints (my laptop, a company phone, servers on the edge of our network) are behaving. And this is fed into a homebuilt data platform that allows the security team to alert on anomalies.‌‌

It also allows them to look at historical data in case of a problem that occurred in the past, or to understand when a problem started. ‌‌

Initially the team was going to use a commercial data platform or SIEM but they quickly realized that these platforms are incredibly expensive and they could build their own at a considerably lower price.‌‌

Also, Cloudflare handles a huge amount of data. When you’re looking at operating system level events on machines in 194 cities plus every employee you’re dealing with a huge stream. And the commercial data platforms love to charge by the size of that stream.‌‌

We are integrating internal DNS data, activity on individual machines, network netflow information, badge reader logs and operating system level events to get a complete picture of what’s happening on any machine we own.‌‌

When someone joins Cloudflare they travel to our head office in San Francisco for a week of training. Part of that training involves getting their laptop and setting it up and getting familiar with our internal systems and security.‌‌

During one of these orientation weeks a new employee managed to download malware while setting up their laptop. Our internal detection systems spotted this happening and the security team popped over to the orientation room and helped the employee get a fresh laptop.‌‌

The time between the malware being downloaded and detected was about 40 minutes.‌‌

If you don’t want to build something like this yourself, take a look at Google’s Chronicle product. It’s very cool. ‌‌

One really rich source of data about your organization is DNS. For example, you can often spot malware just by the DNS queries it makes from a machine. If you do one thing then make sure all your machines use a single DNS resolver and get its logs.‌‌‌‌

In some ways the most interesting part of Cloudflare is the least interesting from a security perspective. Not because there aren’t great technical challenges to securing machines in 194 cities but because some of the more apparently mundane things I’ve talked about how such huge impact.‌‌

Identity, Authentication, Culture, Detection and Response.‌‌

But, of course, the edge needs securing. And it’s a combination of physical data center security and software. ‌‌

To give you one example let’s talk about SSL private keys. Those keys need to be distributed to our machines so that when an SSL connection is made to one of our servers we can respond. But SSL private keys are… private!‌‌

And we have a lot of them. So we have to distribute private key material securely. This is a hard problem. We encrypt the private keys while at rest and in transport with a separate key that is distributed to our edge machines securely. ‌‌

Access to that key is tightly controlled so that no one can start decrypting keys in our database. And if our database leaked then the keys couldn’t be decrypted since the key needed is stored separately.‌‌

And that key is itself GPG encrypted.‌‌

But wait… there’s more!‌‌

We don’t actually want to have decrypted keys stored in any process that accessible from the Internet. So we use a technology called Keyless SSL where the keys are kept by a separate process and accessed only when needed to perform operations.‌‌

And Keyless SSL can run anywhere. For example, it doesn’t have to be on the same machine as the machine handling an SSL connection. It doesn’t even have to be in the same country. Some of our customers make use of that to specify where their keys are distributed to).

One key strategy of Cloudflare is to eat our own dogfood. If you’ve not heard that term before it’s quite common in the US. The idea is that if you’re making food for dogs you should be so confident in its quality that you’d eat it yourself.

Cloudflare does the same for security. We use our own products to secure ourselves. But more than that if we see that there’s a product we don’t currently have in our security toolkit then we’ll go and build it.

Since Cloudflare is a cybersecurity company we face the same challenges as our customers, but we can also build our way out of those challenges. In  this way, our internal security team is also a product team. They help to build or influence the direction of our own products.

The team is also a Cloudflare customer using our products to secure us and we get feedback internally on how well our products work. That makes us more secure and our products better.

The data that passes through Cloudflare’s network is private and often very personal. Just think of your web browsing or app use. So we take great care of it.‌‌

We’re handling that data on behalf of our customers. They are trusting us to handle it with care and so we think of it as more precious than our own internal data.‌‌

Of course, we secure both because the security of one is related to the security of the other. But it’s worth thinking about the data you have that, in a way, belongs to your customer and is only in your care.‌‌‌‌

I hope this talk has been useful. I’ve tried to give you a sense of how Cloudflare thinks about security and operates. We don’t claim to be the ultimate geniuses of security and would love to hear your thoughts, ideas and experiences so we can improve.‌‌

Security is not static and requires constant attention and part of that attention is listening to what’s worked for others.‌‌

Thank you.‌‌‌‌‌‌‌‌‌‌‌‌

CC BY-SA 2.0 image by Sridhar Saraf

And Cloudflare didn’t stop at London. We now have people in Munich, Singapore, Beijing, Austin, TX, Chicago and Champaign, IL, New York, Washington, DC, San Jose, CA, Miami, FL, and Sydney, Australia, as well as San Francisco and London. And today we’re announcing the establishment of a new technical hub in Lisbon, Portugal. As part of that office opening I will be relocating to Lisbon this summer along with a small number of technical folks from other Cloudflare offices.

We’re recruiting in Lisbon starting today. Go here to see all the current opportunities. We’re looking for people to fill roles in Engineering, Security, Product, Product Strategy, Technology Research, and Customer Support.

CC BY-SA 2.0 Image by Rustam Aliyev

My first real idea of Lisbon dates to 30 years ago with the 1989 publication of John Le Carré’s The Russia House. As real, of course, as any Le Carré view of the world:

[...] ten years ago on a whim Barley Blair, having inherited a stray couple of thousand from a remote aunt, bought himself a scruffy pied-a-terre in Lisbon, where he was accustomed to take periodic rests from the burden of his many-sided soul. It could have been Cornwall, it could have been Provence or Timbuktu. But Lisbon by an accident had got him [...]

Cloudflare’s choice of Lisbon, however, came not by way of an accident but a careful search for a new continental European city in which to locate a technical office. I had been invited to Lisbon back in 2014 to speak at SAPO Codebits and been impressed by the size and range of technical talent present at the event. Subsequently, we looked at 45 cities across 29 countries, narrowing down to a final list of three.

Lisbon’s combination of a large and growing existing tech ecosystem, attractive immigration policy, political stability, high standard of living, as well as logistical factors like time zone (the same as the UK) and direct flights to San Francisco made it the clear winner.

Eu comecei a aprender Português há três meses... and I’m looking forward to discovering a country and a culture, and building a new technical hub for Cloudflare. We have found a thriving local technology ecosystem, supported both by the government and a myriad of exciting startups, and we look forward to collaborating with them to continue to raise Lisbon's profile.