WordPress Robots.txt – WordPress Virtual Robots.txt, WordPress Virtual Robots function

Home Forums BulletProof Security Free WordPress Robots.txt – WordPress Virtual Robots.txt, WordPress Virtual Robots function

This topic contains 3 replies, has 2 voices, and was last updated by  AITpro Admin 5 years ago.

Viewing 4 posts - 1 through 4 (of 4 total)
  • Author
    Posts
  • #6519

    PT
    Participant

    Thank you so much.  I have a security questions unrelated to BP but i feel as important, i didn’t see BP include a robots.txt option, why is that? I have a robots.txt in the root like below, should i be including the /mywp/ for each of the directory/file specific block or leave as is? Hope you can add this to future version.

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: */trackback/
    Disallow: */feed/
    Disallow: /*/feed/rss/$
    Disallow: /category/*
    Disallow: /wp-login.php
    Disallow: /wp-config.php
    Disallow: /wp-register.php
    Disallow: /wp-content/
    Disallow: /archives/
    Disallow: /*?*
    Disallow: /category/
    Disallow: /tag/
    Disallow: /search
    Disallow: /index.php
    Disallow: /xmlrpc.php
    Disallow: *?replytocom
    Disallow: /wp-*
    Disallow: /author
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /comments/feed/
    Disallow: /page/
    Disallow: /comments/
    
    User-agent: Mediapartners-Google*
    Allow: /
    
    User-agent: Googlebot-Image
    Allow: /wp-content/uploads/
    
    User-agent: Adsbot-Google
    Allow: /
    
    User-agent: Googlebot-Mobile
    Allow: /
    #6523

    AITpro Admin
    Keymaster

    A robots.txt file is ONLY supposed to go in a website root folder.  It will not work correctly in a subdirectory/subfolder.  So if you add a robots.txt file to your /mywp folder it will be ignored/not looked at/not seen.

    A WordPress robots.txt file or a WordPress virtual robots.txt function is for SEO and proper indexing of files and folders and is not a security measure since only good bots will follow the rules in a robots.txt file.  Bad bots will ignore the robots.txt rules and do whatever they want to do.

    Since a robots.txt file is not a security measure then we may decide to add something like this at a later point/later version of BPS.  For now we are concentrating on only website security measures/features.

    You can add either a robots.txt file or add WordPress virtual robots.txt function to your Theme’s functions.php file or you can add both.

    WordPress Virtual robots.txt function

    // WordPress Virtual robots.txt additions
    add_filter( 'robots_txt', 'v_robots', 10, 2 );
    
    function v_robots( $output, $public ) {
    $output .= "Disallow: /members/" . "\n";
    $output .= "Disallow: /groups/" . "\n";
    $output .= "Disallow: /wp-login.php" . "\n";
    $output .= "Disallow: /activity/p/" . "\n";
    return $output;
    }

    Adding a sitemap to your Virtual robots.txt function

    // WordPress Virtual robots.txt additions
    add_filter( 'robots_txt', 'v_robots', 10, 2 );
    
    function v_robots( $output, $public ) {
    $output .= "SITEMAP: http://www.example.com/sitemap.xml" . "\n";
    $output .= "Disallow: /members/" . "\n";
    $output .= "Disallow: /groups/" . "\n";
    return $output;
    }
    #6527

    PT
    Participant

    I think you misunderstood me. I know robots.txt should be in the root dir, but are the files suppose to be pointing to the subdir? as in should this

    Disallow: /wp-admin 
    be this?
    Disallow: /mywp/wp-admin
    and so on...
    
    #6528

    AITpro Admin
    Keymaster

    The path that you do not want bots to crawl or index is “literal” (actually technically relative – literal is used loosely here) . If you do not want a bot to crawl or index a folder named “foobar” that is located here – /some-folder/another-folder/foobar then your Disallow rule is this

    Disallow: /some-folder/another-folder/foobar

    And you can also specify specific files in a folder instead of the entire folder (all files in that folder).

    Disallow: /some-folder/another-folder/foobar/foobar-example-file.php
    Disallow: /some-folder/another-folder/foobar/foobar-another-example-file.php
Viewing 4 posts - 1 through 4 (of 4 total)

You must be logged in to reply to this topic.