Suspicious bot activity

Bots are programs that visit websites automatically. Some are useful, such as search-engine crawlers and uptime monitors. Others copy your content, inflate your traffic, hunt for weaknesses or run attack tools such as vulnerability scanners.

The aim of bot detection is to notice the clearly automated and unwanted visitors early, and let you decide what to do about them.

1. A scraper requests your pages in order, often thousands of them, to copy prices, products or articles.
2. Attack tools such as vulnerability scanners send requests that announce themselves, or hide behind a fake browser name.
3. Simple scripts often leave the user agent empty or use the default name of their library.
4. The traffic wastes server resources, distorts your analytics, and in the case of attack tools often precedes more serious attempts.

• Requests with no user agent at all.
• User agents naming tools such as curl, python-requests, sqlmap, nmap, nikto, masscan or wpscan.
• A single address walking through pages in perfect order at a steady pace.
• Traffic that ignores your robots.txt.
• Analytics numbers that rise without any extra sales or sign-ups.

• Content theft and copied product data.
• Higher hosting costs and slower pages for real visitors.
• Misleading analytics that hide how real people use your site.
• Attack tools finding weaknesses that are exploited later.

• Publish a robots.txt, and treat visitors that ignore it as suspicious.
• Rate-limit requests by IP address.
• Add a light challenge such as CAPTCHA on sign-up, search and contact forms.
• Cache pages so scraping costs you little.
• Allow known good crawlers explicitly and review the rest.
• Use a bot-management feature from your CDN or hosting provider if you are a frequent target.

Not sure how your own site looks from outside? Run the free website security scanner.

Rakshak flags a request as bot activity when it has no user agent, or when the user agent names a known tool: sqlmap, nmap, nikto, masscan, wpscan, curl or python-requests.

• Every detection creates an alert in your Rakshak dashboard with the IP address, the request that triggered it, the severity and the time.
• If you have connected Telegram, Rakshak also sends you a message. You choose which severities you want to be told about.
• Repeated alerts of the same type from the same IP address are grouped for ten minutes, so you are not flooded with duplicates.
• Suspicious bot activity is a medium-severity detection. It raises an alert but does not block anyone automatically, because it can also come from harmless visitors. You review it and, if you agree, add the IP address to your blocklist on the IP management page.
• Blocking needs the Rakshak server middleware on your website. The lightweight browser snippet can detect and alert, but a browser script cannot block a visitor.

An online shop notices heavy traffic on its product pages. A script is requesting product after product and has not changed its default library user agent.

Rakshak raises a medium-severity bot alert showing the user agent and the IP address. The owner confirms it is not a partner or a monitor and blocks the address.

A determined scraper could return with a browser-style user agent, so the owner also adds rate limits and a challenge on the pages that matter most.

User-Agent: python-requests/2.31.0

Does Rakshak stop every bot automatically?

No. It flags requests with no user agent or with the name of a known tool, and alerts you. Bots that pretend to be ordinary browsers are not detected this way.

Will it flag Googlebot or my uptime monitor?

Search-engine crawlers use their own names and are not on the list. A monitor that uses curl or python-requests can be flagged, so check an alert before you block an address.

Why is bot detection medium severity?

A bot visit is often harmless or even useful, so Rakshak asks you to look rather than blocking automatically.

How do I stop scrapers properly?

Combine rate limits, caching, challenges on sensitive pages and a bot-management feature from your CDN. No single measure stops a determined scraper.

Is blocking bots bad for SEO?

Blocking search-engine crawlers is. Block only addresses that you have checked and that clearly misbehave, and keep your robots.txt in order.