Crawlers (semrush, gtpbot, meta) block

[Ariel]

Hi. I see in the security log that meta, GPTbot, semrush, etc, crawlers are being blocked, see log:

[403 GET Request: julio 21, 2026 - 9:18 am]
BPS Pro: 17.6
WP: 7.0.2
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 185.191.171.12
Host Name: 12.bl.bot.semrush.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /es/family-resort-heated-pool-water-slides-and-fun/
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

I added in 9. CUSTOM CODE REQUEST METHODS FILTERED: this

RewriteCond %{HTTP_USER_AGENT} !meta-externalagent [NC]
RewriteCond %{HTTP_USER_AGENT} !SemrushBot [NC]
RewriteCond %{HTTP_USER_AGENT} !AhrefsBot [NC]
RewriteCond %{HTTP_USER_AGENT} !Applebot[NC]
RewriteCond %{HTTP_USER_AGENT} !GPTBot[NC]
RewriteCond %{HTTP_USER_AGENT} !OAI-SearchBot[NC]

Then encryped, activated, and ran the setup wizard, but they are still being blocked. I’m obviously doing something wrong, so can you help me with this? Thanks!

[AITpro Admin]

It is standard for bots to make a HEAD Request. Use this solution > https://forum.ait-pro.com/forums/topic/allow-uptime-robot/#post-43771 to allow all Bots to make HEAD Requests. Blocking HEAD Requests is only a nuisance filter and not a security feature. So it is safe to allow all Bots to make HEAD Requests to your website.

When SemrushBot sends a HEAD request to your server, it is checking HTTP response headers (such as status codes, content type, or file size) without downloading the actual page body.

Or this .htaccess code should work to only allow Bots that you want to allow to make HEAD Requests to your website.

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteCond %{HTTP_USER_AGENT} !^meta-externalagent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^SemrushBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^Applebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^OAI-SearchBot [NC]
RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

[Ariel]

Thanks for your prompt reply. Actually, I was using the uptimerobot tweak already, so this is what I have now in place:

# REQUEST METHODS FILTERED
RewriteEngine On
# 1. ALLOW UptimeRobot and approved crawlers to make HEAD requests
# If HEAD *AND* UA matches an allowed bot, stop processing here and let the request through normally.
RewriteCond %{REQUEST_METHOD} ^HEAD [NC]
RewriteCond %{HTTP_USER_AGENT} UptimeRobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} uptimerobot\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} meta-externalagent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SemrushBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Applebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} OAI-SearchBot [NC]
RewriteRule ^ - [L]
# 2. BLOCK EVERYONE ELSE
# If we are still here and it is a HEAD request, kill it immediately (F = Forbidden).
RewriteCond %{REQUEST_METHOD} ^HEAD [NC]
RewriteRule ^ - [F]
# Standard BPS Block
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]

But, the crawlers are still blocked, as they use GET. So, is it safe to enable it? And if yes, do I have to enable it at the same place as I’m allowing HEAD, or should I just whitelist by hostname? Thanks a lot for your advice on this.

[AITpro Admin]

All Bots make HEAD Requests first and then may use a GET Request after the HEAD Request. Sometimes HEAD Request are seen as GET Requests.

Note: SemrushBot.* would match: SemrushBot/7~bl; +http://www.semrush.com/bot.html)

Try this code:

# REQUEST METHODS FILTERED
# If you want to allow HEAD Requests use BPS Custom Code and copy
# this entire REQUEST METHODS FILTERED section of code to this BPS Custom Code
# text box: CUSTOM CODE REQUEST METHODS FILTERED.
# See the CUSTOM CODE REQUEST METHODS FILTERED help text for additional steps.
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]
RewriteCond %{REQUEST_METHOD} ^(HEAD) [NC]
RewriteCond %{HTTP_USER_AGENT} !^UptimeRobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^uptimerobot\.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^meta-externalagent.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^SemrushBot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^AhrefsBot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^Applebot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^GPTBot.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^OAI-SearchBot.* [NC]
RewriteRule ^(.*)$ /wp-content/plugins/bulletproof-security/405.php [L]

[AITpro Admin]

Since you have modified your custom htaccess code to use the BPS GET Request error template then that is why it is being logged as a GET Request. At least as far as the template label goes. The BPS 405 error template has this template label: 405 HEAD Request:

RewriteCond %{REQUEST_METHOD} ^HEAD [NC]
RewriteRule ^ - [F]

[Ariel]

Hi..I implemented the code you sent me, and now uptimerobot is not working, nor the crawlers

[405 HEAD Request: July 21, 2026 - 2:06 pm]
BPS Pro: 17.6
WP: 7.0.2
Event Code: BFHS-HEAD - HEAD Request Blocked
Solution: https://forum.ait-pro.com/forums/topic/security-log-event-codes/
REMOTE_ADDR: 52.22.236.30
Host Name: checker-nvi5.uptimerobot.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: HEAD
HTTP_REFERER: https://regalmanagers.com
REQUEST_URI: /
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)

[403 GET Request: julio 21, 2026 - 2:10 pm]
BPS Pro: 17.6
WP: 7.0.2
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 185.191.171.2
Host Name: 2.bl.bot.semrush.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /es/enjoy-a-resort-stay-4m-to-disney-free-waterslides/
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

[Ariel]

I implemented this change, allow GET and HEAD for crawlers, allow HEAD for uptimerobot, disallow everything else, and it seems to be working fine. Please let me know if this ok, security-wise.

# CRAWLER USER-AGENT WHITELIST — bypass all further BPS rewrite rules for approved bots
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(GET|HEAD)$ [NC]
RewriteCond %{HTTP_USER_AGENT} meta-externalagent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SemrushBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Applebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} OAI-SearchBot [NC]
RewriteRule ^ - [L]

# 1. ALLOW UPTIMEROBOT
# If it is HEAD *AND* matches UptimeRobot user agent, allow it (L = Last rule, stop processing).
RewriteCond %{REQUEST_METHOD} ^HEAD [NC]
RewriteCond %{HTTP_USER_AGENT} (UptimeRobot|uptimerobot\.com) [NC]
RewriteRule ^ - [L]

# 2. BLOCK EVERYONE ELSE
# If we are still here and it is a HEAD request, kill it immediately (F = Forbidden).
RewriteCond %{REQUEST_METHOD} ^HEAD [NC]
RewriteRule ^ - [F]

# Standard BPS Block
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ - [F]

[AITpro Admin]

The HEAD Request htaccess code is not website security code. It’s nuisance protection against Bots. Not sure why the code I posted is not working. Your code looks fine. That could be because I added the ^ character, which means start checking here, but did not add the closing $ character.

RewriteCond %{HTTP_USER_AGENT} !^UptimeRobot.*$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} !^uptimerobot\.com.*$ [NC,OR]

[Ariel]

Thanks..Now, a (somehow) related question, is there a way with BPS pro to make sure a hacker can not impersonate a crawler, by a reserve DNS or PHP-level check?

[AITpro Admin]

It’s very quick and simple to change the User Agent. All browsers have tools that let you do that. When someone is automating Bots from a server the Bot code is designed to automatically change User Agents if the Bot is blocked. If blocked, then try X User Agent. That leaves you with IP blocking, which is a waste of time since hackers and spammers typically have blocks of IP addresses or VPN’s. That would then leave you with CIDR IP blocking, which can cause your website to lose traffic. So the only effective method that ensures that you don’t shoot yourself in the foot is to block by bad actions. If a bad action is blocked then it doesn’t matter who or what made the request. The BPS htaccess security filters are designed like that. So if you know what bad action is occurring then you can specifically block that bad action. An example of a bad action would be a Query string added to the end of a request => https://example.com/search?q=<script>alert('Hacked')</script>

[AITpro Admin]

Jeff Starr at Perishable Press created an extensive htaccess file here > https://perishablepress.com/6g/

Warning: You will need to test everything on your website to make sure nothing important is being blocked.

[Ariel]

Thanks for your reply. Actually, what I want to achieve is only a reverse dns, to make sure that e.g., when a meta crawler HEAD or GET from 57.141.0.17 arrives, something checks the IP belongs to Meta and if not, drop it. There is a plugin that does that, BotBlocker Security, but I’m reluctant to have it along BPS PRO.

[Ariel]

thanks for the htaccess file, I will take a look at it

[AITpro Admin]

https://stackoverflow.com/questions/45836233/ptr-record-is-confusing

Yes, it is relatively simple to create reverse DNS checking code using PHP code, but here are the pitfalls. The IP address you are using to check with could belong to an innocent web host who is hosting a malicious person or what is more common is the IP address you are using to check with points to a hacked victim website on an innocent web host. The other scenario that is less common is an individual hacker server that is setup to launch automated Bot attacks. Hackers and spammers usually use victim websites that are hacked to launch their attacks. There is a lot less risk in using victim websites vs hosting your own hacker servers.

 

[Ariel]

Got it. So, I will rephrase my question. If I get a visit like this

[403 GET Request: Juli 21, 2026 - 2:23 p.m.]
BPS Pro: 17.6
WP: 7.0.2
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 57.141.0.17
Host Name: 57.141.0.17
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /de/universalorlando/
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

and the useragent is one of the ones I allow via the useragent script I sent before (meta-externalagent), check that 57.141.0.17 belongs to Meta..this way, any pre-authorized crawler (via useragent) will be further allowed if the IP belongs to them. Does it make sense?

[Author]

Posts