if you use that robots.txt. is that the spider can crawl the newthread and post or not if i put this robots. txt? my forum is new not so many index page. thanks
They will still crawl the showthread page which has all the posts on it. The showpost page just shows a single post and the newthread page is used to make new pages - there is no content on it.