Custom XML feed 403 error – Kyero Feed 403 error

[Nicki]

I am creating a custom xml feed for a property site that is being validated for kyero.com. The support at kyero are telling me that when they try to grab the feed it is returning a 403 error. I am not 100% sure that bulletproof is to blame, but I have seen in the forums here similar problems however I have not been able to find a solution for this problem.

The feed url is: http://www.hiltonrealestates.com/hre34111253hiltonkyero/

This is what the support at kyero say:

“However, our downloader does have a slight problem – as the url is a directory http://www.hiltonrealestates.com/hre34111253hiltonkyero/ – we’re receiving a 403 forbidden response when we try to grab it (even though it resolves with no problem in a browser). Is it possible to serve the file with an xml file extension?”

I am wondering if I can allow access to the feed in the .htaccess by unblocking something, but am not sure what. As the feed is dynamic I am not sure how I would serve it as an .xml file.

Can you point me in the right direction for fixing this issue?

Thanks.

[AITpro Admin]

Check your BPS Security Log and post the 403 log entry for what is being blocked.  BPS logs all 403 errors whether or not BPS or something else is blocking something.

[Nicki]

That’s another odd thing because I only have one 403 error logged for yesterday and I’m not sure it is for the feed 403 error.

[403 GET / HEAD Request: November 5, 2015 - 4:18 pm]
Event Code: BFHS - Blocked/Forbidden Hacker or Spammer
Solution: N/A - Hacker/Spammer Blocked/Forbidden
REMOTE_ADDR: 92.63.87.10
Host Name: ip87-10.mwtv.lv
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /xmlrpc.php
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.8.0.8) Gecko/20061213 Firefox/1.5.0.8

[AITpro Admin]

I checked the Feed and I do not see any 403 errors.  I do see this error and the Feed is not displaying correctly:  “This XML file does not appear to have any style information associated with it. The document tree is shown below.”  You can do BPS troubleshooting steps #1 and #2 to eliminate that BPS is causing any problems.

http://forum.ait-pro.com/forums/topic/read-me-first-free/#bps-free-general-troubleshooting

1. On the Security Modes page, click the Root Folder BulletProof Mode Deactivate button. See Custom Code Note if doing this step works.
2. On the Security Modes page, click the wp-admin Folder BulletProof Mode Deactivate button.  See Custom Code Note if doing this step works.

[Nicki]

I have been talking to kyero support and have discovered that they are accessing the feed using wget or curl. This seems similar to the mailchimp 403 error, so I have removed the ‘curl’ part from the BPSQSE BPS QUERY STRING EXPLOITS. I am waiting to see if that solves the problem or if I need to remove wget as well.

[Nicki]

Update: It seems Kyero have several ways of validating/checking a feed and then a completely different way of actually accessing it with their own software. So they check it manually using wget or curl, but their software is still returning a 403 error. They have asked if I can whitelist their user agent, I am not exactly sure how to do this.
Their user agent is “Kyero Downloader v1.0”. Would something like the below work, or would removing the head block for all bots solve this problem?

RewriteCond %{HTTP_USER_AGENT} !^(Kyero Downloader) [NC]

[AITpro Admin]

Excellent clue.  Here is the solution below.  Not sure why you are not seeing a Security Log entry for this, but maybe there is something installed on your host server that is blocking the 403 error from being logged by BPS or the 403 error is occurring in some way externally that cannot be captured/logged.

What is being blocked is “loader” in the Kyero Downloader v1.0 User Agent String by the 2 User Agent nuisance filters.  I have removed “loader” from the User Agent filters below.

1.  Copy the modified BPS Query String Exploits code below (“loader” has been removed from the code below) to this BPS Root Custom Code text box:  CUSTOM CODE BPSQSE BPS QUERY STRING EXPLOITS: Modify Query String Exploit code here 
2.  Click the Save Root Custom Code button.
3. Go to the BPS Security Modes page and click the Root Folder BulletProof Mode Activate button.

# BEGIN BPSQSE BPS QUERY STRING EXPLOITS
# The libwww-perl User Agent is forbidden - Many bad bots use libwww-perl modules, but some good bots use it too.
# Good sites such as W3C use it for their W3C-LinkChecker. 
# Use BPS Custom Code to add or remove user agents temporarily or permanently from the 
# User Agent filters directly below or to modify/edit/change any of the other security code rules below.
RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|email|harvest|extract|grab|miner) [NC,OR]
RewriteCond %{THE_REQUEST} (\?|\*|%2a)+(%20+|\\s+|%20+\\s+|\\s+%20+|\\s+%20+\\s+)(http|https)(:/|/) [NC,OR]
RewriteCond %{THE_REQUEST} etc/passwd [NC,OR]
RewriteCond %{THE_REQUEST} cgi-bin [NC,OR]
RewriteCond %{THE_REQUEST} (%0A|%0D|\\r|\\n) [NC,OR]
RewriteCond %{REQUEST_URI} owssvr\.dll [NC,OR]
RewriteCond %{HTTP_REFERER} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
RewriteCond %{HTTP_REFERER} \.opendirviewer\. [NC,OR]
RewriteCond %{HTTP_REFERER} users\.skynet\.be.* [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(http|https):// [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [NC,OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC,OR]
RewriteCond %{QUERY_STRING} \=PHP[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} [NC,OR]
RewriteCond %{QUERY_STRING} (\.\./|%2e%2e%2f|%2e%2e/|\.\.%2f|%2e\.%2f|%2e\./|\.%2e%2f|\.%2e/) [NC,OR]
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} (http|https)\: [NC,OR] 
RewriteCond %{QUERY_STRING} \=\|w\| [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)/self/(.*)$ [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)cPath=(http|https)://(.*)$ [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*embed.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^e]*e)+mbed.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*object.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^o]*o)+bject.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (\<|%3C).*iframe.*(\>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^i]*i)+frame.*(>|%3E) [NC,OR] 
RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [NC,OR]
RewriteCond %{QUERY_STRING} base64_(en|de)code[^(]*\([^)]*\) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} ^.*(\(|\)|<|>|%3c|%3e).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(\x00|\x04|\x08|\x0d|\x1b|\x20|\x3c|\x3e|\x7f).* [NC,OR]
RewriteCond %{QUERY_STRING} (NULL|OUTFILE|LOAD_FILE) [OR]
RewriteCond %{QUERY_STRING} (\.{1,}/)+(motd|etc|bin) [NC,OR]
RewriteCond %{QUERY_STRING} (localhost|loopback|127\.0\.0\.1) [NC,OR]
RewriteCond %{QUERY_STRING} (<|>|'|%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
RewriteCond %{QUERY_STRING} concat[^\(]*\( [NC,OR]
RewriteCond %{QUERY_STRING} union([^s]*s)+elect [NC,OR]
RewriteCond %{QUERY_STRING} union([^a]*a)+ll([^s]*s)+elect [NC,OR]
RewriteCond %{QUERY_STRING} \-[sdcr].*(allow_url_include|allow_url_fopen|safe_mode|disable_functions|auto_prepend_file) [NC,OR]
RewriteCond %{QUERY_STRING} (;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).*(/\*|union|select|insert|drop|delete|update|cast|create|char|convert|alter|declare|order|script|set|md5|benchmark|encode) [NC,OR]
RewriteCond %{QUERY_STRING} (sp_executesql) [NC]
RewriteRule ^(.*)$ - [F]
# END BPSQSE BPS QUERY STRING EXPLOITS

[Nicki]

That’s brilliant, the new code has worked and Kyero have successfully accessed the feed. They are saying they are having trouble downloading the images and wonder if something is limiting or blocking it. I can’t see anything in the htaccess that is blocking images (although not entirely sure what I’m looking for) so maybe it’s something on the hosting.
Thanks for your help with this.

[AITpro Admin]

I have to second this:  “although not entirely sure what I’m looking for” since I would not know that either.  You would have to ask Kyero questions like:  “how do you download images”, etc. to be able to know what to check.

[Author]

Posts