Fighting malicious bot traffic has become a daily chore for many companies. Let’s dive into the problem of bots, botnets, and how to become more efficient at bot mitigation.
What Is Bot Mitigation?
Bot mitigation is the process of filtering bot traffic on your site. Its key goal can be to accept good bots and block out malicious bots. Bot mitigation is an integral process of modern cybersecurity, fraud prevention, and risk management.
What Are Bots and Botnets?
A bot, or Internet bot, is a software application programmed to perform online tasks automatically.
A botnet is a network of bots, or numerous computers connected together to perform the same repetitive tasks together.
According to research by Imperva, bots accounted for over half of all web traffic in 2020.
Common Types of Bots
While many people think of bots as nefarious programs, it’s actually important to distinguish between good and bad bots. Here are various kinds you may encounter on the Internet.
Examples of Good Bots
In the first category, we’ll look at bots that are useful and should, in most cases, be allowed on your site. These include:
- Crawler bots: also known as robot or spider bots, are used to automatically discover website content by following links. The most famous are from Google, such as GoogleBot, StoreBot, AdsBot, and Googlebot News, amongst others.
- Site monitoring bots: used to monitor website metrics, such as uptime or page speed.
- Copyright bots: these crawl platforms or websites looking for content that may infringe on copyright laws.
- Feed bots: destined to collect large volumes of newsworthy content from the Internet. Content aggregators and social media networks use feed bots to populate their sites.
Examples of Bad Bots
- Scalper / ticketing bots: these bots attempt to purchase large amounts of events tickets so that scalpers may resell them at a higher price.
- DDoS attack bots: designed to overload an application, network or website in order to shut it down.
- Credential stuffing bots: when fraudsters acquire a large amount of leaked login details, they don’t have the time or patience to try them all manually. That’s where credential stuffing is used, to take over other people’s online accounts.
- Inventory denial attack bots: if you’re a shady eCommerce trying to take down a competitor, you could use bots to deplete their inventory by automatically loading a shopping cart with bots. This is also known as inventory hoarding or shopping bots attack.
- Spam bots: we’re all familiar with suspicious internet comments claiming that you can work from home and make fortunes. More often than not, these spam comments are posted by bots, while attempting to lure victims towards a phishing website.
- Account farming bots: bots that build up credibility of one or more user accounts by scripted activity, like posting, reviewing, following. These are often re-sold as various services (such as reviews) or coordinated for a given purpose.
- Fraudulent checkout bots: it’s easy to write a script to select and automatically order given products, and fraudsters can combine this with stolen credit card details to scale their attacks.
- Bookmaker or arbitrage bots: similar to checkout bots, these are scripted to place bets instantly on betting sites if an advantageous opportunity arises.
Combining machine learning, sophisticated algorithms and human intelligence, advanced technology can detect patterns that fraudsters aren’t aware of.
Fight Fraud Faster
How Does Bot Mitigation Work?
Regardless of what kind of bot traffic you attempt to mitigate, there are three key steps in the process.
- Identifying bot traffic: you must first be able to detect a bot from non-bot traffic in order to filter it.
- Assessing bot nature: this is where understanding the difference between useful versus nefarious bots comes in handy.
- Blocking damaging bots: finally, you should be able to stop the bad bots from getting anywhere near your website.
Techniques for Bot Mitigation
Bot mitigation techniques vary based on the kind of threat you face.
- Challenge-based: challenging a bot to replicate human behavior is a good way to filter out the least sophisticated of them. See Captcha, for instance.
- Signature-based: in the world of bot mitigation, a signature is a pattern of data that instantly points to well-known botnets.
- Anomaly and behavior-based detection: similar to signature-based detection, but you’ll be looking at suspicious data points that differ from how legitimate users behave on your site.
- IP blocking & IP reputation: most bot traffic tends to use the same connection methods. This is useful as you can easily identify residential connections versus, say, proxies and VPNs.
- Allow lists & block lists: there are of course online blacklists you can refer to in order to confirm whether or not you’re dealing with a bot.
- Rate limiting & TPS: you can control how much traffic passes through your site with rate limiting, and how many transactions you process with TPS (transactions per second).
- Device fingerprinting: a modern technology designed to look at the configuration of software and hardware of your visitors. This helps spot connections between bots to take down botnets and mitigate fresh bot attacks in real-time. Some tools, such as SEON can even directly detect emulator use like Selenium or other headless Chrome connection attempts, which should immediately raise red flags.
- Browser profiling (cookie and browser hashes): used in combination with device fingerprinting, this allows you to create IDs for each visitor. Once again, very helpful for spotting connections between users in a variety of settings, from multi-accounting for bonus abuse to online poker bots.
3 Steps to Mitigate Bot Traffic
Mitigating bot traffic can be a complex affair. This is why it’s always good to take a multi-layered approach, by enabling more than one mitigation solution.
1. Enable CAPTCHA
Captcha is one of the most famous challenge-based bot mitigation solutions. It’s also one with the most hit-and-miss results. While Forbes, for instance, claims CAPTCHAs reduced bot-driven submissions to website forms by 88%, many companies (such as ours) have stopped using it on account of its poor accuracy results.
On the plus side, Google’s latest Captcha v3 introduces the idea of risk scores, so it’s not so much a clear-cut case of accepting or rejecting traffic based on the outcome of the challenge.
Captcha Pros
- Affordable: Captcha is one of the least expensive forms of bot mitigation solution you can enable today.
- Relatively frictionless: Captcha can be frustrating, but it’s also not as disruptive as, say, a face ID check.
Captcha Cons
- AI bots can solve CAPTCHA: the most sophisticated bots will have no problems bypassing it.
- Captcha farms: there is a whole cottage industry of low-paid workers who manually solve captchas for fraudsters.
- Slower website: Captcha can damage your page speed, which is bad both for user experience and SEO.
2. Use a Web Application Firewall (WAF)
WAFs can protect web applications from attacks such as cross-site scripting (XSS), SQL injection and session hijacking. A Web Application Firewall analyzes incoming traffic, checks GET and POST-based HTTP requests, and feeds the data through static rules.
WAF Pros
- Good at identifying familiar attack signatures: WAF works great when the botnets are well known.
- Good for patching cybersecurity vulnerabilities: you can test your application or server for other cybersecurity vulnerabilities.
WAF Cons
- Not great at flagging sophisticated bots: any kind of advanced bot that can replicate human behavior will easily bypass a Web Application Firewall.
- Too reliant on IP-based rules: botnets, zombie networks, IoT deployments and IPv6 allow fraudsters and cybercriminals to rotate through millions of different IPs to bypass IP-based filtering.
- Slower UX: a Web Application Firewall can also low down your users’ online experience.
Deploy Bot Detection Software
Last but not least, you could look at a complete all-in-one solution for bot mitigation, provided by a third-party company. Most bot detection software is available under a SaaS, pay monthly model, and the good news is that it’s easier than ever to integrate into your website via API.
You can read more about the best bot detection software.
Bot Detection Software Pros
- Combine multiple bot detection methods: bot detection software will let you use a multi-layered approach via different analysis and filtering tools.
- Specific software for specific bot challenges: you can find bot detection software designed to reduce ad click fraud, general fraud, or even just DDoS.
- Easy to integrate: a good bot detection software will come with well-documented APIs so the integration can be fast and seamless.
Bot Detection Software Cons
- Extra expense: most work under the SaaS model so you’ll need to budget an extra expense. It can also be hard to get a good idea of the ROI unless the solution offers a free trial.
How SEON Does Bot Mitigation
SEON is a fraud detection specialist, with all the right risk-scoring tools needed for bot mitigation.
This includes powerful velocity rules (to understand how often a user does something), complete device fingerprinting and IP analysis (to understand how they connect to your site), and browser and cookie hash (to spot connections between users)
Best of all, you can integrate via API, try the tool for free for 30 days, and get complete control over your risk strategy thanks to custom rules and machine-learning rule suggestions.
SEON’s anti-fraud tools are designed to detect suspicious usage and uncover hidden fraudsters
Ask an Expert
Frequent asked questions
Yes. Bots can be detected and filtered out with the right rules, challenges, or other detection methods.
You can spot bots and botnets via firewall filters, challenges, or signature detection, amongst others. Advanced bot mitigation software offers risk scoring so you can control bots with more flexibility.
Some bots have nefarious purposes, such as buying an entire shop’s inventory, breaking into user accounts, or a DDOS bot attack. Others are useful to index your site with search engines or check for copyright infringement.
A good bot mitigation strategy should include multiple layers of protection such as Captcha, WAF, and the deployment of specific bot detection software with risk scoring. Device fingerprinting and IP analysis are increasingly popular solutions for filtering out bot traffic.
Sources
