Home › Forums › BulletProof Security Free › WordPress Robots.txt – WordPress Virtual Robots.txt, WordPress Virtual Robots function
- This topic has 5 replies, 3 voices, and was last updated 2 years, 1 month ago by AITpro Admin.
-
AuthorPosts
-
PTParticipant
Thank you so much. I have a security questions unrelated to BP but i feel as important, i didn’t see BP include a robots.txt option, why is that? I have a robots.txt in the root like below, should i be including the /mywp/ for each of the directory/file specific block or leave as is? Hope you can add this to future version.
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: */trackback/ Disallow: */feed/ Disallow: /*/feed/rss/$ Disallow: /category/* Disallow: /wp-login.php Disallow: /wp-config.php Disallow: /wp-register.php Disallow: /wp-content/ Disallow: /archives/ Disallow: /*?* Disallow: /category/ Disallow: /tag/ Disallow: /search Disallow: /index.php Disallow: /xmlrpc.php Disallow: *?replytocom Disallow: /wp-* Disallow: /author Disallow: /feed/ Disallow: /trackback/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /comments/ User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Mobile Allow: /
AITpro AdminKeymasterA robots.txt file is ONLY supposed to go in a website root folder. It will not work correctly in a subdirectory/subfolder. So if you add a robots.txt file to your /mywp folder it will be ignored/not looked at/not seen.
A WordPress robots.txt file or a WordPress virtual robots.txt function is for SEO and proper indexing of files and folders and is not a security measure since only good bots will follow the rules in a robots.txt file. Bad bots will ignore the robots.txt rules and do whatever they want to do.
Since a robots.txt file is not a security measure then we may decide to add something like this at a later point/later version of BPS. For now we are concentrating on only website security measures/features.
You can add either a robots.txt file or add WordPress virtual robots.txt function to your Theme’s functions.php file or you can add both.
WordPress Virtual robots.txt function
// WordPress Virtual robots.txt additions add_filter( 'robots_txt', 'v_robots', 10, 2 ); function v_robots( $output, $public ) { $output .= "Disallow: /members/" . "\n"; $output .= "Disallow: /groups/" . "\n"; $output .= "Disallow: /wp-login.php" . "\n"; $output .= "Disallow: /activity/p/" . "\n"; return $output; }
Adding a sitemap to your Virtual robots.txt function
// WordPress Virtual robots.txt additions add_filter( 'robots_txt', 'v_robots', 10, 2 ); function v_robots( $output, $public ) { $output .= "SITEMAP: http://www.example.com/sitemap.xml" . "\n"; $output .= "Disallow: /members/" . "\n"; $output .= "Disallow: /groups/" . "\n"; return $output; }
PTParticipantI think you misunderstood me. I know robots.txt should be in the root dir, but are the files suppose to be pointing to the subdir? as in should this
Disallow: /wp-admin be this? Disallow: /mywp/wp-admin and so on...
AITpro AdminKeymasterThe path that you do not want bots to crawl or index is “literal” (actually technically relative – literal is used loosely here) . If you do not want a bot to crawl or index a folder named “foobar” that is located here – /some-folder/another-folder/foobar then your Disallow rule is this
Disallow: /some-folder/another-folder/foobar
And you can also specify specific files in a folder instead of the entire folder (all files in that folder).
Disallow: /some-folder/another-folder/foobar/foobar-example-file.php Disallow: /some-folder/another-folder/foobar/foobar-another-example-file.php
keeweeParticipantAre the virtual robots.txt file needed in wordpress when using BPS security pro?
AITpro AdminKeymaster@ keewee – Nope, BPS does not require a virtual robots txt file. You can use any method you want for your robots.txt file, but WP recommends using the virtual robots method.
-
AuthorPosts
- You must be logged in to reply to this topic.