vb.org Archive

vb.org Archive (https://vborg.vbsupport.ru/index.php)
-   vBulletin 4.x Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=245)
-   -   Miscellaneous Hacks - Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)

Simon Lloyd 06-05-2016 11:30 PM

Its not normal to have high CPU usage because of this mod, it hardly uses any resources at all, it redirects right at page call so doesn't even allow your site to load for the banned list, however, if you have banned many things then they will be constantly redirecting, after a short while they will stop visiting your site.

I think you may have other issues going on here, it might be a point to hire Max or Ozzy to take a look at your site and get it optimized properly :)

Max Taxable 06-05-2016 11:41 PM

Quote:

Originally Posted by Simon Lloyd (Post 2571691)
I think you may have other issues going on here, it might be a point to hire Max or Ozzy to take a look at your site and get it optimized properly :)

Thanks for the reference but neither of us hires out, as policy. Ever. Made a exception and tried it once last summer though, on the insistence of the person we were helping. Didn't end well.

To marko - either of us would be happy to take a look at the situation, for free.

Simon Lloyd 06-05-2016 11:43 PM

No problem, just thought you two now did it as a business, I haven't been on here in such a long while I'm out of touch :(

Max Taxable 06-05-2016 11:47 PM

Quote:

Originally Posted by Simon Lloyd (Post 2571693)
No problem, just thought you two now did it as a business, I haven't been on here in such a long while I'm out of touch :(

Since our negative experience last year, neither of us does much at all anymore. Ozzy even less than I unfortunately.

BGN64 06-08-2016 09:01 AM

I have a user agent showing up in a regular basis, often dozens at a time.

Code:

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
The version is often 34, 29, etc (there are several). I have yet to find a legit user. What string would you suggest to send them on their way?

Simon Lloyd 06-08-2016 11:25 AM

You can enter the entire string, that way you will only ban users with that exact useragent

Max Taxable 06-08-2016 04:12 PM

Quote:

Originally Posted by BGN64 (Post 2571776)
I have a user agent showing up in a regular basis, often dozens at a time.

Code:

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
The version is often 34, 29, etc (there are several). I have yet to find a legit user. What string would you suggest to send them on their way?

If they're not doing anything malicious, what's the problem? Did you trace out IP addresses?

Blocking that UA string WILL block legitimate people. It's nothing exotic, it's a Windows 8 computer with only about a year or so old version of Firefox. It's not unusual for people to not upgrade FF.

Just something to think over.

BGN64 06-11-2016 01:21 PM

Thanks for your reply.

All IPs resolve to China, Iran, Venezuela, Russia etc (at least the several dozen I have checked). I have yet to find one from the US, CA or AU (which are my primary audience). I often have 10-15 on the site at any given time.

markoroots 06-29-2016 01:47 PM

Thanks for the tips guys and many thanks Max for your availability, I will write you a pm.

grafbyte 07-13-2016 12:36 PM

hi

after the install ...
PHP Code:

Warnung (PHP Warning): stristr(): Empty needle in ..../includes/class_bootstrap.php(103) : eval()'d code (Zeile 100)

Warnung (PHP Warning): stristr(): Empty needle in ..../includes/class_bootstrap.php(103) : eval()'
d code (Zeile 110

mit dis list

Quote:

baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6
w3m

almaden
Anarchie
Artabus
ASPSeek
attach
autoemailspider
BackWeb
Baidu
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
GNIP
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Ichiro
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Magpie
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
omgili
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
speedy
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus


Quote:

Importiere Sprache, Bitte warten...

VB 4.2.3 (i have change admin foldername)

2.4. If you need German, Spanish or Romanian translation, import
vblang-glowhostspamomatic-210_XX.xml, where XX is the language locale code
(Located at AdminCP -> Languages & Phrases > Download / Upload Languages).

[:]

Warnung (PHP Warning): Illegal string offset 'value' in ..../includes/adminfunctions_language.php (Zeile 651)

Warnung (PHP Warning): Illegal string offset 'value' in ..../includes/adminfunctions_language.php (Zeile 651)

Warnung (PHP Warning): Illegal string offset 'value' in ..../includes/adminfunctions_language.php (Zeile 651)

Warnung (PHP Warning): Illegal string offset 'value' in ..../includes/adminfunctions_language.php (Zeile 651)

Warnung (PHP Warning): Illegal string offset 'value' in ..../includes/adminfunctions_language.php (Zeile 651)

Simon Lloyd 11-01-2016 02:26 PM

That isn't an issue with this modification as it doesn't use glowhostspamomatic. You also have a blank in your list, this could effectively ban everyone and everything!!

RichieBoy67 11-01-2016 02:33 PM

Quote:

Originally Posted by grafbyte (Post 2573178)
hi

after the install ...
PHP Code:

Warnung (PHP Warning): stristr(): Empty needle in ..../includes/class_bootstrap.php(103) : eval()'d code (Zeile 100)Warnung (PHP Warning): stristr(): Empty needle in ..../includes/class_bootstrap.php(103) : eval()'d code (Zeile 110

mit dis list

Add this to your config under <?php:
PHP Code:

define('SKIP_ALL_ERRORS'true); 


imported_dboogie2288 12-28-2016 07:37 PM

I have a dumb question. I'm up on running a forum and such, but not on SEO and what these spiders and bots are actually doing. Since installing this product, I continually notice that the blocked spiders number keeps going up...at an alarming rate. First it was a few hundred, then it was a thousand, and now its about 8100 blocked spiders at a given moment. That was about 3-4 weeks ago. What's going on? Is this something along the lines of, "block them and they will come?" I fear the day that this plug in gets disabled by accident or whatever because the floodgates might come bursting open.

Simon Lloyd 01-03-2017 06:53 PM

Which bots are showing as increasing in numbers? it's more than likely the same bot that keeps changing IP address, don't worry it will settle down.

w00fy 01-05-2017 05:06 PM

I just wanted to thank you for this. My site had slowed to a crawl with page load times of 15 seconds and I did not know why. I installed this and my page loads are now mostly under a second.

I had no idea these bots and spiders were doing this to me..after a week I see this.

Spiders/Bots Activity Blocked From Site: 102,285

Thanks again, much appreciated.

Simon Lloyd 01-08-2017 02:25 PM

I suspect that you have some other process issues that are affecting load times but I'm glad this has helped.

Do take a look at your images that your site has, try to use a CDN to deliver them or maybe create a sprite to contain them all and deliver the images from sprite coordinates, also look at which scripts are called on page load, sometimes its best to try and minimize these and minify your JS scripts, this might help https://hostway.com/blog/10-ways-to-...e-load-faster/

Its a bit of work but well worth the effort.

conaero 04-25-2017 07:14 AM

I also have been using this product for over year and initially it wiped 90% of the traffic, as they were bots but over time the number has gone back up again.

I would assume what we need is a new list of bots?

Have had a search around but cant find one???

I am using this updated list as per a few posts back:

Quote:

baiduspider
beta.statsit.com
statsit
SiteIntel
Yandex
GomezAgent
FunWebProducts
MSIE 1
MSIE 2
MSIE 3
MSIE 4
MSIE 5
MSIE 6
w3m

almaden
Anarchie
Artabus
ASPSeek
attach
autoemailspider
BackWeb
Baidu
Bandit
BatchFTP
BlackWidow
Bot\mailto:craftbot@yahoo.com
Buddy
bumblebee
CherryPicker
ChinaClaw
CICC
Collector
Copier
Copyscape
Crescent
DIIbot
DISCo
DISCo\Pump
dotbot
Download\Demon
Download\Wonder
Downloader
Drip
DSurf15a
eCatch
EasyDL/2.99
EirGrabber
email
EmailCollector
EmailSiphon
EmailWolf
Express\WebPictures
ExtractorPro
EyeNetIE
FileHound
FlashGet
FrontPage
GetRight
GetSmart
GetWeb!
gigabaz
GNIP
Go\!Zilla
Go!Zilla
Go-Ahead-Got-It
gotit
Grabber
GrabNet
Grafula
grub-client
HMView
HTTrack
httpdown
.*httrack.*
ia_archiver
Ichiro
Image\Stripper
Image\Sucker
Indy*Library
Indy\Library
InterGET
InternetLinkagent
Internet\Ninja
InternetSeer.com
Iria
JBH*agent
JetCar
JOC\Web\Spider
JustView
larbin
LeechFTP
LexiBot
lftp
Link*Sleuth
likse
//Link
LinkWalker
Mag-Net
Magnet
Magpie
Mass\Downloader
Memo
Microsoft.URL
MIDown\tool
Mirror
Mister\PiX
Mozilla.*Indy
Mozilla.*NEWT
Mozilla*MSIECrawler
MS\FrontPage*
MSFrontPage
MSIECrawler
MSProxy
Navroad
NearSite
NetAnts
NetMechanic
NetSpider
Net\Vampire
NetZIP
NICErsPRO
Ninja
Nutch
Octopus
Offline\Explorer
Offline\Navigator
omgili
Openfind
PageGrabber
Papa\Foto
pavuk
pcBrowser
Ping
PingALink
Pockey
psbot
Pump
QRVA
RealDownload
Reaper
Recorder
ReGet
Scooter
Seeker
Siphon
sitecheck.internetseer.com
SiteSnagger
SlySearch
SmartDownload
Snake
sogou
Soso
SpaceBison
speedy
Spinn3r
sproose
Stripper
Sucker
SuperBot
SuperHTTP
Surfbot
Szukacz
tAkeOut
Teleport\Pro
URLSpiderPro
Vacuum
VoidEYE
Web\Image\Collector
Web\Sucker
WebAuto
[Ww]eb[Bb]andit
webcollage
WebCopier
Web\Downloader
WebEMailExtrac.*
WebFetch
WebGo\IS
WebHook
WebLeacher
WebMiner
WebMirror
WebReaper
WebSauger
Website
Website\eXtractor
Website\Quester
Webster
WebStripper
WebWhacker
WebZIP
Wget
Whacker
Widow
WWWOFFLE
x-Tractor
Xaldon\WebSpider
Xenu
Yandex
Yeti
YOUDAOBOT
Zeus.*Webster
Zeus

WOLF-G4MIN-X 10-11-2017 07:54 AM

I just installed this on 4.2.5, and as soon as I have it installed, this is showing up on the left upper side of my header where the logo is.

Code:

PHP Warning: stristr(): Empty needle in ..../includes/class_bootstrap.php(103) : eval()'d code on line 110
Could anyone tell me if that can be fixed, and if, how ?

Thanks in advance.

Wolf

furnival 08-23-2019 08:39 AM

Seems to work fine for me on VBulletin 4.2.5 so the error above in @WOLF-G4MIN-X's post must be the Glowhost Spamomatic mod incompatibility mentioned elsewhere in the thread.

Simon Lloyd 08-24-2019 10:50 AM

Quote:

Originally Posted by WOLF-G4MIN-X (Post 2590390)
I just installed this on 4.2.5, and as soon as I have it installed, this is showing up on the left upper side of my header where the logo is.

Code:

PHP Warning: stristr(): Empty needle in ..../includes/class_bootstrap.php(103) : eval()'d code on line 110
Could anyone tell me if that can be fixed, and if, how ?

Thanks in advance.

Wolf

I havent seen this before and it doesn't use the bootstrap, This loads at every page call which prevents the spiders actually loading a page, so it could be a third party hack that doesn't like it being loaded, you could change the load order in the products page of admincp, normally most hacks are set at 5.

martin1 01-20-2020 11:00 AM

I installed latest version of hack but it not stops huawei crawler service I see multiple entries that are online for example
ecs-159-138-101-77.compute.hwclouds-dns.com

ecs-159-138-109-55.compute.hwclouds-dns.com

Simon Lloyd 01-20-2020 12:55 PM

Just use this in your ban list,
compute.hwclouds-dns.com

martin1 01-21-2020 12:10 PM

Quote:

Originally Posted by Simon Lloyd (Post 2601820)
Just use this in your ban list,
compute.hwclouds-dns.com

I did, it's not blocked.Still showing in online list
Is the latest hack compatibile with 4.2.3pl1 version of the forum?

Simon Lloyd 01-21-2020 01:20 PM

When you say "online list" are you using another hack like Paul Ms Who's online....etc?

martin1 01-21-2020 08:38 PM

No, thats the only hack I'm using to ban spiders.
User agents appear under forum user online list , it can be up to 20 entries in the same time.

Forum is behind cloudflare.

Simon Lloyd 01-22-2020 08:36 AM

Can you PM me a link to your forum?

martin1 01-22-2020 06:18 PM

sure

martin1 01-25-2020 07:52 AM

Simon did you checked the forum ?

hwcloud is still browsing

by_online 05-02-2020 11:48 PM

Hello,
i have also blocked the bots but still seeing them in forum online list.
Anyone with the same issue ?

Simon Lloyd 05-03-2020 09:47 AM

What other hacks do you have installed, like PaulM's Who's Online....etc?

by_online 05-03-2020 11:21 AM

1 Attachment(s)
Hello,
I have below attached plugins at the moment :

Simon Lloyd 05-03-2020 04:25 PM

It's possible it's either a bot where you havent actually got the right user agent string or if it is a plugin clash it will only possibly be VSA advanced forum statistics......my money is on the UA string :)

oldfan 05-18-2020 07:30 PM

Sorry, its been years. But has the bot list been updated?
I'm still getting 100's http://www.metallifukinca.com/online.php

thanks

Simon Lloyd 05-18-2020 07:35 PM

Sorry, I've never kept an updated list as it's a personal thing, there will be users of this mod that may be able to share theirs with you, but you'd be better of customising your own for which bots are allowed information from your site or not.

oldfan 05-22-2023 01:39 AM

anyone have a update list please

Simon Lloyd 05-22-2023 02:18 AM

Even if someone provided a list it would be what's important to them to block and might not suit you or your users as some of the user agent strings of your users may contain something being blocked.in that list.

oldfan 05-23-2023 12:04 AM

btw the way it works on 4.2.6. pho 7.2 no problem


All times are GMT. The time now is 05:59 PM.

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.

X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.01764 seconds
  • Memory Usage 1,841KB
  • Queries Executed 10 (?)
More Information
Template Usage:
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (4)bbcode_code_printable
  • (3)bbcode_php_printable
  • (9)bbcode_quote_printable
  • (1)footer
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (6)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (1)pagenav_pagelinkrel
  • (1)post_thanks_navbar_search
  • (1)printthread
  • (37)printthreadbit
  • (1)spacer_close
  • (1)spacer_open 

Phrase Groups Available:
  • global
  • postbit
  • showthread
Included Files:
  • ./printthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/class_bbcode_alt.php
  • ./includes/class_bbcode.php
  • ./includes/functions_bigthree.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • printthread_start
  • pagenav_page
  • pagenav_complete
  • bbcode_fetch_tags
  • bbcode_create
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • printthread_post
  • printthread_complete