Crawlers (semrush, gtpbot, meta) block

Home Forums BulletProof Security Pro Crawlers (semrush, gtpbot, meta) block

Viewing 4 posts - 16 through 19 (of 19 total)
  • Author
    Posts
  • #46140
    AITpro Admin
    Keymaster

    My honest opinion – That would take significant effort for very little in return. Or in other words, not really worth the effort.

    In order for your checks to work you would have to grab a list of valid IP addresses for Meta, put them into an array and check against that array.

    Something like this generally…

    If User Agent = X and IP address = Y then check IP addresses in an IP address array to verify that Y = a User Agent IP address in your array.
    Then one of your final allow/block PHP code conditions to handle the 403 Access Denied block.

    if (!in_array($client_ip, $whitelist)) {
        http_response_code(403); // Forbidden
        exit('Access Denied: Your IP is blocked.');
    }

    Or I guess you could just check if the IP address matches the domain name that it’s supposed to match. That’s assuming that the IP address was not spoofed.

    #46143
    Ariel
    Participant

    Thanks a lot for your replies and insight! it seems that my effort to allow SEO crawlers, like AhrefsBot or SemrushBot, even OAI-SearchBot (which is OpenAI indexer, not for training) , are not something other webmasters do? I say this because allowing them seems to have some risks that are difficult to mitigage. I was under the impression that having my site crawled by them would be beneficial…it is not?

    #46144
    AITpro Admin
    Keymaster

    Most bad Bots are just annoying and are not malicious. Bad Bots were a bigger issue many years ago because they caused excessive server load. These days the server load is insignificant because PHP, web servers, etc. have improved so much. So since you are blocking Bots in general it is important to allow the specific good Bots that you want to allow. Or you could just allow all Bots by not blocking HEAD Requests. The choice is entirely up to you.

    #46145
    Ariel
    Participant

    Thanks!

Viewing 4 posts - 16 through 19 (of 19 total)
  • You must be logged in to reply to this topic.