vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Black Snow 12-10-2014 10:29 AM

Quote:

Originally Posted by Max Taxable (Post 2526554)
No sir we have been trying to solve the mystery of why Baidu gets through on some v4 installations, but not all and never a v3, and my hook conflict idea opened a new can of worms for investigation, and Ozz found something very interesting.

I never looked before but it is also getting through on my site.

CAG CheechDogg 12-10-2014 10:54 AM

Baidu has kissed my "gluteus maximus" for almost 2 years and change if not more ... So has Yandex and a handful of others as well ... I must have a "magic" forum :)

ozzy47 12-10-2014 11:04 AM

I am re working a couple of things, and then need to test further, I can then share my findings with Simon. :)

Gadget_Guy 12-10-2014 01:14 PM

I would be happy to test on my site if it helps the community.

D.

Max Taxable 12-10-2014 03:55 PM

Quote:

Originally Posted by ForceHSS (Post 2526565)
What interesting thing was found

I don't wanna flap it, already flapped too much. But I am pretty sure the problem is solved.

Gadget_Guy 12-11-2014 02:27 AM

If this helps anyone.... this is a list of what I am seeing in terms of spiders on my site with this installed.

Bing Spiders (6),
Google Favicon Spiders (9),
Proximic Spiders (135),
Baidu Spiders (175),
WinHTTP Spiders (12),
Facebook Spiders (20),
Google AdSense Spiders (7),
Magpie Spiders (9),
linkdexbot/2.0 Spiders (7),
AhrefsBot Spiders (14),
Coccoc Spiders (2),
Google AppEngine Spiders (6),
Google Spiders (40),
Sucuri Spiders (3),
Twitterbot Spiders (4),
Google FeedFetcher Spiders (3),
Apple RSS Spiders (1),
WordPress.com mShots Spiders (1),
Google Web Preview Spiders (3),
Grapeshot Spiders (2),
James BOT WebCrawler Spiders (5),
Netseer crawler/2.0 Spiders (2),
Google Images Spiders (3),
Galaxy Spiders (2),
Feedly Spiders (2),
DotBot Spiders (1),
Yahoo! Slurp Spiders (1),
360Spider Spiders (4),
Netcraft Web Server Survey Spiders (1),
NerdyBot Spiders (2),
Exabot Spiders (1),
Integrity Bot Spiders (1),
ContextAd Bot Spiders (2),
Twitturls.com (Python-urllib) Spiders (1)

I am happy to supply any information that you may find useful to assist in the work you are doing.

D.

Simon Lloyd 12-11-2014 05:34 AM

I need a snapshot of your settings for the mod as there is no way all those being entered in the mod would get past the mod!

CAG CheechDogg 12-11-2014 08:09 AM

1 Attachment(s)
This is a snapshot of the spiders that are showing up in the whos online:

https://vborg.vbsupport.ru/external/2014/12/30.jpg

What exactly do you need a snapshot in the settings Simon?

This is my list of spiders I have banned with your mod:

almaden
Anarchie
Artabus
ASPSeek
attach
autoemailspider
BackWeb
Baidu
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
GNIP
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Ichiro
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Magpie
magpie
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
omgili
Openfind
PageGrabber
Papa\Foto
PaperLiBot
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
speedy
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus
baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
Nesotebot
DCPbot
AOL Advertising R&D
DataCha0s
aiHitBot
Apache-HttpClient
Zend_Http_Client
ReverseGet
XXX bot Content
vBSEO
spbot
OffByOne
thyroidbuzz
AcoonBot
coccoc
xpymep
proxyproxy2884
AppEngine
start.exe
Semiocast HTTP client
Firefox/3.6.23
TurnitinBot
curl
SwpLc/1.6
GrepNetstat.com
news bot
AskTbPTV
checks
panopta
App3le
PhantomJS
AlwaysOnline
SISTRIX
proximic
CRAWL-E/0.6.4
WebMoney
Maxthon
HTMLParser
oBot
UnisterBot
ERACrawler
Butterfly
Topsy
Butterfly Topsy Crawler
Ezooms
Deepnet
Alexa
Bitlybot
Seznam
Fulltext
Facebook
Sunrise Communications AG
crawl
Crawl
MJ12bot
Bimbot
Snapbot
thunderstone
Thunderstone
grub-client
Bing
MSN
OOZBOT
Wayback Machine
Crowsnest Spider
FlipboardProxy
Feedly

Gadget_Guy 12-11-2014 10:48 AM

1 Attachment(s)
Here is my stuff:

Simon Lloyd 12-11-2014 05:06 PM

Hi Gadget Guy, remove the second picture as it has your email address in it. I see the settings are ok, now can you just copy the list as you have it (copy straight out of the textbox in the mod) sitck it in a wordpad document, zip it and attach it here so i can check that please.


All times are GMT. The time now is 05:08 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01973 seconds
  • Memory Usage 1,750KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (2)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (4)pagenav_pagelink
  • (3)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (10)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete