Go Back   vb.org Archive > vBulletin 4 Discussion > vB4 General Discussions

Reply
 
Thread Tools Display Modes
  #1  
Old 07-09-2014, 08:30 AM
dacho's Avatar
dacho dacho is offline
 
Join Date: Dec 2004
Location: Tell-Aviv
Posts: 224
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default How can I stop Unknown robot, spider, crawler

Hello mate

How can we stop the Unknown robot, spider, crawler ?
They probably skipped and not refer to robot.txt
In robot.txt I made some bots are enabled and blocking all others bots, and still without permission they scan me
Reply With Quote
  #2  
Old 07-09-2014, 08:36 AM
Dave Dave is offline
 
Join Date: May 2010
Posts: 2,583
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

I believe it usually takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side.
By the way you named it robots.txt right? With the s after robot?
Reply With Quote
  #3  
Old 07-09-2014, 02:08 PM
dacho's Avatar
dacho dacho is offline
 
Join Date: Dec 2004
Location: Tell-Aviv
Posts: 224
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Yes my friends, it is clear that file name robots.txt and sits on Root.
It's not about takes a few days up to a week before those robots "look" at the robots.txt file and update it on their side

It probably really are a lot of bots, spiders and crawlers pass through the robots.txt and go inside without identified

The Solution probably need to combine the robots.txt and .htaccess

See the following guide :
http://michael.langley.id.au/blog/posts/28
Reply With Quote
  #4  
Old 07-09-2014, 02:14 PM
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Posts: 3,134
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Robots.txt isn't a law. Bots aren't required to follow it or even index it. Only the legitimate SE bots will. Bad bots simply ignore it. It's just a sign basically, politely saying "please don't crawl these areas of my site." It's alot like gun laws in the US - criminals simply ignore them.

For the others, you want the "Ban Spiders by User Agent" Mod.
Reply With Quote
2 благодарности(ей) от:
ForceHSS, tbworld
  #5  
Old 07-09-2014, 02:15 PM
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
Posts: 6,357
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

<a href="https://vborg.vbsupport.ru/showthread.php?t=268208" target="_blank">https://vborg.vbsupport.ru/showthread.php?t=268208</a>
This will stop them as soon as you install it
Reply With Quote
Благодарность от:
Max Taxable
  #6  
Old 07-09-2014, 10:17 PM
Lee G Lee G is offline
 
Join Date: Jun 2006
Location: Costa Blanca
Posts: 143
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

This mod by Paul M will help you track them
https://vborg.vbsupport.ru/showthread.php?t=232182

There used to be an updated spider list for vbulletin on the net. No idea what or if that's still shared

You can then track the heavy hitters and block them
Reply With Quote
  #7  
Old 07-09-2014, 10:28 PM
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Posts: 3,134
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Lee G View Post
There used to be an updated spider list for vbulletin on the net. No idea what or if that's still shared[/B]
It is:

Updated vBulletin spiders list


But that is NOT the block list for the ban spiders mod, there's never been a "official" list for that as the Mod is blank when first installed..
Reply With Quote
Благодарность от:
ozzy47
  #8  
Old 07-09-2014, 10:57 PM
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
Posts: 6,357
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

When was the last time that list was updated Max
Reply With Quote
  #9  
Old 07-09-2014, 11:00 PM
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Jul 2009
Location: USA
Posts: 10,929
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

It was updated in January of this year. It had over 200 spiders added.
Reply With Quote
  #10  
Old 07-09-2014, 11:03 PM
ForceHSS ForceHSS is offline
 
Join Date: Apr 2008
Posts: 6,357
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

So the last few in the last few posts have not been added? Can they be added
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 01:20 PM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.05988 seconds
  • Memory Usage 2,252KB
  • Queries Executed 11 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (10)post_thanks_box
  • (4)post_thanks_box_bit
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (3)post_thanks_postbit
  • (10)post_thanks_postbit_info
  • (10)postbit
  • (10)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • fetch_musername
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • post_thanks_function_fetch_thanks_bit_start
  • post_thanks_function_show_thanks_date_start
  • post_thanks_function_show_thanks_date_end
  • post_thanks_function_fetch_thanks_bit_end
  • post_thanks_function_fetch_post_thanks_template_start
  • post_thanks_function_fetch_post_thanks_template_end
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete