LinkedIn bot blocked by UA 'scan'

[DBR]

Hi, when posting a link to share in LinkedIn, the featured image was not being displayed. After some testing, I discovered that there were two blocked user agents in the hackrepair blacklist in my .htaccess custom code that were partly the cause –
‘Link’ and ‘Jakarta\ Commons-HttpClient’. The latter makes sense as that is listed as LinkedIn bot’s UA.

The other reason that LinkedIn wasn’t able to scrape the featured images, was ‘scan’ in these two lines from BPSQSE BPS QUERY STRING EXPLOITS (wget was previously removed to allow cron jobs):

RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
....
RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]

Deleting ‘scan’ has now allowed LinkedIn to display the featured image when sharing a link. Now that I have removed ‘scan’from these two lines of code, what kind of security risk (if any) does that represent?

Many thanks

[AITpro Admin]

Those 2 HTTP_USER_AGENT rules are bad bot/scanner anti-nuisance rules.  So any changes you make to them would not be a security risk.

[DBR]

That makes sense. Thanks for confirming that.

[Author]

Posts