Prevent folders being crawled

[DemOnstar]

I did an online robots.txt generating script, it came up with this..

Sitemap: https://www.xxxxxxxxxxx.com/xxxxxxxx/vbulletin_sitemap_blog_0.xml.gz


User-agent: Baiduspider
Disallow: /
User-agent: *

Somebody advised me to add Baiduspider and the rest is pretty much the same as you previously posted.

It now looks like this.

Sitemap: https://www.xxxxxxxxxxx.com/xxxxxxxx/vbulletin_sitemap_blog_0.xml.gz


User-agent: Baiduspider
Disallow: /
User-agent: *
Disallow: /cgi-bin/
Disallow: /activity.php
Disallow: /admincp/
Disallow: /announcement.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /subscriptions.php
Disallow: /threadrate.php
Disallow: /usercp.php

Which now has me totally baffled.

Cheers for that.