Key Points
Anyone with public-facing web properties is likely to have bot traffic on their website.
One type of bot that commonly targets eCommerce and online portals is a ‘scraper bot’.
Some scraper bots are good (such as those used by search engines to assess your website’s content to inform search results, or price comparison sites to help inform consumer decisions), however many are malicious, and will work to scrape not only images but also pricing data from your site for use by a competitor.
Many Bot Management providers will need to divert your traffic to a dedicated data centre to analyse your traffic and ‘scrub’ it clean from malicious bot traffic before sending it on to your site. While effective this will almost certainly add latency to the traffic’s ‘journey’ resulting in degraded user experience. Look for a technology partner with an expansive network who can scan your traffic in real time as it passes through any data centre on their network.
Good and bad scraper bots behave in largely the same way, making it difficult for bot protection systems to differentiate between the two. A common challenge with Bot Management solutions is that they can return a high number of false positives (legitimate bot or customer traffic blocked as though it were malicious). This can result in legitimate customers being challenged to various and repeated authentication challenges, or in extreme cases, blocked altogether. Look for a technology partner who can consistently return low rates of false positives on your traffic.
The Success Story
I often joke that the key to understanding what that role of Customer Success is all about is to say: repeat the sentence again slowly… it’s there, in the name. Customer Success is, at its core, about making customers successful. And this is what makes our day, and makes us happy. Allow me to share a short story on how Cloudflare made a successful customer even more so with our Bot Management solution.
Once upon a time, an online property portal, let’s call them Property Portal, came to Cloudflare for our DDoS and WAF solution. We worked well together. The customer liked our ease of use and we delivered on our promise to provide performance and security to them. As Property Portal’s brand and digital footprint grew, so did the instances of malicious bot traffic, in particular ‘scraper bots’.
They say that imitation is the sincerest form of flattery. But when that imitation turns into someone else profiting off your IP, the shine starts to wear off, and that ‘imitation’ starts to become something more akin to outright theft. This is a challenge common to market leaders in the eCommerce and online portal space, where market leading organizations who pride themselves on presenting a solid portfolio of quality product offerings often find that competing sites seem to be not only replicating their content but matching or undercutting their prices in near real time.
Cloudflare wasn’t working with Property Portal when they first started facing these challenges, and as such, Property Portal engaged another party - at that time, one of the market leaders in the space - to provide a Bot Management solution for them.
At first pass, this seemed to solve the problem, however it wasn’t long before additional challenges became apparent:
performance was impacted slightly as this solution required traffic to be re-routed to a scrubbing centre to be ‘scrubbed’ of requests from bad bots before coming to their site;
a small percentage of malicious bots were still getting through and scraping valuable content of their website, and most damagingly;
Property Portal discovered that they were seeing a significant number of ‘false positives’, resulting in rising frustration for legitimate visitors (buyers, sellers and renters) repeatedly being asked to complete challenges in order to validate that they were human and not bots as they tried to navigate through the site.
During this time, Cloudflare released and matured our own Bot Management offering. Being aware that Property Portal weren’t seeing success from their existing solution, the account team began discussing the value of consolidating their bot solution with their DDoS and WAF offering from Cloudflare. We were given very clear success criteria which in technical terms, translated to the following:
Don’t mess anything up, deprecate our user experience, or make us change our domains if we switch to you;
Stop more of the bad traffic.;
And most importantly, let more of the good users in, and stop challenging them as much as our current provider does (reduce the number of false positives).
We passed with flying colours. In addition, Property Portal was happy that they were able to consolidate additional services under one vendor.
For our side, Cloudflare now has the privilege of knowing that we are helping to improve the experience for many of Property Portal’s end customers, while at the same time working to protect their IP and hard work by keeping the ‘imitators’ and their scraper bots at a safe distance.
Happy Customer = Happy Customer Success Managers & Account Teams. Day made.
Would you like to know more?
Does any of the above feel familiar to you? Do your competitors have the uncanny ability to present near identical inventory, images or pricing to yours just as soon as you publish changes to your site? Or, are you keen to learn more in order to stay ahead of the bot armies?
Well, you’ve come to the right place, friend:
If you’re the self-serve type then take a look at our learning centre here and here, or our product pages. (And yes, we have a (good) chat bot there waiting to help you).
If ‘tuning in and geeking out’ is your preferred method of learning, then tune into the next episode of ‘Customers + Success’ on Cloudflare TV where I’ll be interviewing some of the people involved in this case, and hearing more about the challenges that this customer faced first-hand. The live segment will air at 4PM PST October 21st / 7AM SGT October 22nd / 10AM AEST October 22nd.
Alternatively, if you consume your knowledge in old-fashioned human style feel free to contact us here. Someone from our team will be in touch to get you the answers you are looking for.