googlebot wp-login.php blocked – block googlebot from indexing wp-login.php

[David G]

Google alerted me to the fact it cant crawl my site. Here is 1 of over 100 log entries. please help.

>>>>>>>>>>> 403 GET or Other Request Error Logged - June 27, 2013 - 5:27 pm <<<<<<<<<<<
REMOTE_ADDR: 66.249.72.216
Host Name: crawl-66-249-72-216.googlebot.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /wp-login.php?redirect_to=http%3A%2F%2Fquestasmasonry.com%2F%3Fattachment_id%3D338
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

[AITpro Admin]

Why would you want Google or any bot to crawl your login page.  You should be blocking all bots from crawling or indexing your login page.

See this Forum topic link on how to add a WordPress virtual robots.txt function to your theme’s functions.php file to block it from being crawled or indexed:

http://forum.ait-pro.com/forums/topic/wordpress-robots-txt-wordpress-virtual-robots-txt/#post-6523

[David G]

My robot file disappeared… hmmm. Still doesn’t explain why all attempts by the googlebot gets redirected to the login.

[AITpro Admin]

If you do not have a robots.txt file or you are not telling the googlebot to not index or crawl your wp-login.php/login page then the googlebot and all other bots will crawl and index your login page.  Do this Google search – “what is a robots.txt” file.  Also disallowing folders or files to crawl in a robots.txt file does not necessarily guarantee that they will see or follow those disallow rules.

[David G]

I have replaced the robot file. I had one there before. I do know what the limitations are with robot.txt files. I have restored the robot file and restarted BPS. We’ll see what happens.

[Author]

Posts