How to block bad bots using nginx configuration

Question

Some bad bots, such as ClaudeBot, AmazonBot, are hammering my server. I tried to block them using robots.txt, but it seems that they do not follow robots.txt. Also, there are 100s of IP that they are using to crawl my webpage. How can I block them by modifying nginx configuration file?

Answer 1

To block bot bots, you can modify the Nginx configuration file specific to your website. Open the configuration file and make the following changes:

• At the top before the server statement, add the following lines:

map $http_user_agent $block_bot {

    default 0;

    ~*(ClaudeBot|Amazonbot) 1;

}

Here, you can add all the bad bots that you want to block. 

• Inside the location statement, add the following lines:

if ($block_bot) {

    return 403;

The above code returns 403 status if any of those bad bots tried to access your webpage.