Go Back   vb.org Archive > vBulletin 4 Discussion > vB4 General Discussions
Reload this Page About Robotx txt
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Display Modes
  #1  
Old 09-20-2015, 11:34 PM
tanzeelniazi tanzeelniazi is offline
 
Join Date: Apr 2012
Posts: 375
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default About Robotx txt
Hello all
I want to know why robots.txt file is necessary ?
Is this for SEO ? or use in security ?
When i open my robots.txt file i see this
PHP Code:
#Begin Attracta SEO Tools Sitemap. Do not remove
sitemap: http://cdn.attracta.com/sitemap/2*****0.xml.gz
#End Attracta SEO Tools Sitemap. Do not remove 
i dont know what is this and dont know what before add
When i search vbulletin robots.txt i see this

PHP Code:
User-agent: Slurp
Crawl-delay: 5 
i want to add good things in my robots.txt file but i dont know what is the uses
Please tell me in simple words or make a robots file for me
thanks

--------------- Added [DATE]1442799540[/DATE] at [TIME]1442799540[/TIME] ---------------

Before i have good guests and bots like alexa bot google bot many other and before i have good alexa rank and traffic now bad
please some one make a good robots file for me pplease
Reply With Quote
  #2  
Old 09-21-2015, 12:30 AM
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Posts: 3,134
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
It's not "necessary" and also does nothing at all for security. It's just a signpost that says "please bots don't index any thing on this list" and few search engines pay any mind to it.

It's not a blocker, it's not a security measure, it's just a .txt file that "noble" search engine spiders do check and try to follow the guide of it.
Reply With Quote
  #3  
Old 09-21-2015, 12:35 AM
RichieBoy67's Avatar
RichieBoy67 RichieBoy67 is offline
 
Join Date: Apr 2004
Location: CT - Down in a hole..
Posts: 3,057
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
It is always good to use it though many seedy search engines ignore it. The main searches though like Google and Bing honor it. Those are the ones you care about really. Others you can use htaccess to block if you are concerned with them.

You can use it to block pages you do not want indexed in the Google, etc. For example if you have a really spammy advertising section or something. You also add your sitemap there to keep google crawling it though you should have your sitemap submitted using webmaster tools anyways.

On my sites I use it to keep the focus on my topic content. SEO is about quality, not quantity so I do not index my off topic forums.

Also, as done in yours you can use robots.txt to put a delay on search bots if your resources are limited.

As for Attractaca.. I would not waste time using that.
Reply With Quote
  #4  
Old 09-21-2015, 09:20 AM
tanzeelniazi tanzeelniazi is offline
 
Join Date: Apr 2012
Posts: 375
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
which sitemap is good for me ?? how to find more sitemap in my site ?
if i upload another sitemap any loss in my site ? or decreasing rank ?
Reply With Quote
  #5  
Old 09-21-2015, 09:30 AM
Seven Skins's Avatar
Seven Skins Seven Skins is offline
 
Join Date: Sep 2008
Location: London, UK
Posts: 1,481
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Sitemaps don't usually have any affect on site ranking ... If you want to add sitemap to your site with vbulletin then add something like this to robots.txt.

Code:
sitemap: http://sitemap_1.xml
sitemap: http://sitemap_2.xml
Reply With Quote
Reply

« Previous Thread | Next Thread »

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 08:38 AM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.04080 seconds
  • Memory Usage 2,201KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)bbcode_code
  • (2)bbcode_php
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (5)post_thanks_box
  • (5)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (5)post_thanks_postbit_info
  • (5)postbit
  • (5)postbit_onlinestatus
  • (5)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete