Mini Mods - HTTP Error Response Codes by BOP5 (Soft 404 Solution)

Brought to you by BirdOPrey5 / Qapla.com

This issue was brought to my attention by Sherrie at TAZ in this thread.

The issue is that when Google crawls a site and runs into the vBulletin No Permission message or other standard vBulletin error pages vBulletin gives the standard "200" HTTP response code which means everything is OK, even though there is no real content on these pages. Therefore Google considers these "soft errors" and they can damage overall SEO of your site if there are too many of them.

What this does is make vBulletin return standard HTTP "Error" responses- by default 403 (Not Authorized) for No Permission pages and 404 (Not Found) for general error messages. When Google sees these responses (as opposed to 200) it knows not to bother indexing the page and everyone is happy.

As of version 2.2 new options have been added to return custom response codes when the forum is closed or when the forum is busy. It will also send a retry header telling the crawler they can try again later. As of version 2.0 there is also a separate response code for search.php errors.

A limitation of this mod is it works best with PHP 5.4.0 or higher. As of version 2.0 If you are on an older version of PHP it may not work, but it should not cause any errors, so it is worth trying.

There are no instructions, once installed it is enabled by default. In Admin CP -> Settings -> Options -> HTTP Error Responses by BOP5 you can enable/disable the mod and choose the response codes you want for the different error types. You can use any standard HTTP response code available from this list at wikipedia.

The mod works fine on VB 3.x or 4.x including VB 3.8.8 (still in Beta) and VB 4.2.2.

To test that it is working use a free online status checker such as: http://httpstatus.io/
(Note the site will cache your result for a while so test with a unique URL each time.)

Before you install the mod any URL that will show a no-permission or other vBulletin error will show a status code of 200. After installing the same page will show a 403, 404, or other code that you have configured.

Added Boothby as a "co-author" as thanks for his contribution of code and ideas. :up:

------------------------------------------------------

Please "Mark as Installed" if you use this.
Donations always appreciated. :up:
[s]Nominate MOTM if you LOVE it! [/s] (You did, Thanks!)

[karnevil]

After looking through which template was returning the 404 error

STANDARD_ERROR

I replaced the 404 Not Found text to read 403 Forbidden

I checked that a genuine 404 page still came up as such so I know it hasn't affected that.

Not ideal but will see what results webmaster crawl errors comes back with over the next couple of weeks and let you know.

[BirdOPrey5]

I think there may be some confusion... Standard Error doesn't return a 404 error... If it did this mod wouldn't be necessary- the response code on standard error is 200 (OK) regardless of the actual text displayed in the browser- the text in the browser doesn't determine the response code Google gets. The response code isn't a visible element.

Am I right you have VBSEO installed? I don't think there is any other means vBulletin would display a 404 error message?

[Mickie D]

Thank you

Perfect Mod, should be standard built into vbulletin!!

Mick

[Mecho]

Awesome

it works without any issue on 4.2.2 PL1 - Vbseo installed.

First error number was 200
after installing this, it is 403

thanks BirdOPrey5

[synseal]

Hi and thanks for this!

In my webmasters tools I get a load of errors like this all related to tags



Shouldn't this modification fix this with the 403 no permission message?

Crawlers are getting to these pages (Tags) and because I don't allow guests to use the search function they are getting these type of errors and marking it as a soft 404.





Any idea TIA.

[BirdOPrey5]

I can confirm http://www.techkings.org/tags/worked.html is now getting a REAL 404 response code. So those soft errors should go away soon.

[synseal]

Originally Posted by BirdOPrey5

I can confirm http://www.techkings.org/tags/worked.html is now getting a REAL 404 response code. So those soft errors should go away soon.

These are errors I am still getting after installing this modification.

I cleared all the errors in webmasters tools (and I had loads) and these tag errors have come back after installing this, see the "detected dates" above in the image :up:

[BirdOPrey5]

if you put any of your "tag" URLs into this site- http://httpstatus.io - it confirms a 404 is being returned. It isn't 403 because by default the mod sends 403 for "No Permission" errors only and 404 for all others.

The dates on the image are all several or more days ago so I assume you won't see any more past today assuming the mod remains active.

However it is pretty clear you should add your entire /tags/ path to your robots.txt file, that would be the right way to prevent the crawl issues.

[Mecho]

installed and result is great. check soft.png (attached)

just one question , after installing this Access Denied urls increased (access.png) . All these urls are for the threads which has been moved to Trash and those redirect to Trashcan url. is it ok or bad for SEO?

[BirdOPrey5]

From what I've read (I'm not claiming to be an expert) a 403/404 won't hurt overall SEO although the actual page causing the error will eventually be de-indexed. (Which is exactly what we want to happen.)