LinkedIn bot blocked by UA 'scan'

Home Forums BulletProof Security Pro LinkedIn bot blocked by UA 'scan'

This topic contains 2 replies, has 2 voices, and was last updated by  DBR 6 months ago.

Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #34580

    DBR
    Participant

    Hi, when posting a link to share in LinkedIn, the featured image was not being displayed. After some testing, I discovered that there were two blocked user agents in the hackrepair blacklist in my .htaccess custom code that were partly the cause –
    ‘Link’ and ‘Jakarta\ Commons-HttpClient’. The latter makes sense as that is listed as LinkedIn bot’s UA.

    The other reason that LinkedIn wasn’t able to scrape the featured images, was ‘scan’ in these two lines from BPSQSE BPS QUERY STRING EXPLOITS (wget was previously removed to allow cron jobs):

    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
    ....
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]

    Deleting ‘scan’ has now allowed LinkedIn to display the featured image when sharing a link. Now that I have removed ‘scan’from these two lines of code, what kind of security risk (if any) does that represent?

    Many thanks

    #34581

    AITpro Admin
    Keymaster

    Those 2 HTTP_USER_AGENT rules are bad bot/scanner anti-nuisance rules.  So any changes you make to them would not be a security risk.

    #34582

    DBR
    Participant

    That makes sense. Thanks for confirming that.

Viewing 3 posts - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.