HTTP Error Response Codes by BOP5 (Soft 404 Solution)
 

Brought to you by BirdOPrey5 / Qapla.com

This issue was brought to my attention by Sherrie at TAZ in this thread.

The issue is that when Google crawls a site and runs into the vBulletin No Permission message or other standard vBulletin error pages vBulletin gives the standard "200" HTTP response code which means everything is OK, even though there is no real content on these pages. Therefore Google considers these "soft errors" and they can damage overall SEO of your site if there are too many of them.

What this does is make vBulletin return standard HTTP "Error" responses- by default 403 (Not Authorized) for No Permission pages and 404 (Not Found) for general error messages. When Google sees these responses (as opposed to 200) it knows not to bother indexing the page and everyone is happy.

As of version 2.2 new options have been added to return custom response codes when the forum is closed or when the forum is busy. It will also send a retry header telling the crawler they can try again later. As of version 2.0 there is also a separate response code for search.php errors.

A limitation of this mod is it works best with PHP 5.4.0 or higher. As of version 2.0 If you are on an older version of PHP it may not work, but it should not cause any errors, so it is worth trying.

There are no instructions, once installed it is enabled by default. In Admin CP -> Settings -> Options -> HTTP Error Responses by BOP5 you can enable/disable the mod and choose the response codes you want for the different error types. You can use any standard HTTP response code available from this list at wikipedia.

The mod works fine on VB 3.x or 4.x including VB 3.8.8 (still in Beta) and VB 4.2.2.

To test that it is working use a free online status checker such as: http://httpstatus.io/
(Note the site will cache your result for a while so test with a unique URL each time.)

Before you install the mod any URL that will show a no-permission or other vBulletin error will show a status code of 200. After installing the same page will show a 403, 404, or other code that you have configured.

Added Boothby as a "co-author" as thanks for his contribution of code and ideas. :up:

------------------------------------------------------

Please "Mark as Installed" if you use this. :)
Donations always appreciated. :up:
[s]Nominate MOTM if you LOVE it! ;)[/s] (You did, Thanks!)

Reserved.

Good lad BOP5, on the ball as ever....

Excellent. Thanks and works like a charm.

This so sad. Today I was looking to solve those error and find your thread, but my forum is in Banahosting and the PHP version on the server is 5.3.28... There is a way I can solve this without migrate to other hosting?

See if your host will update the PHP, if not then you would need to either find a different host, or not use this particular mod. :(

Sorry, been looking for ways to do it in older PHP versions but the only work-around uses the header() function, the problem is that function must be called before any output is created by the script and the hooks for the error messages aren't called until after script produces some output- so the header function calls are basically ignored. I'm not seeing any alternatives.

I am using vbulletin 4.2.2 but version PHP 5.3, i very want use this mod, you can modify this product

That was answered in the post above yours. :)

So if i want to use this i have to spare my VBSEO right ?

I don't know how it would work with VBSEO or not, I don't have VBSEO so I can't and don't test against it.

No reason why it shouldn't work with vBSEO.

I have VBSEO and its working like a charm.

And no way to make it work with lower php version no way at all

if i edit the min php on the xml
<dependency dependencytype="php" minversion="5.4.0" maxversion="" />
Should it not work

Update got it working for a lower php

i just installed the mod it still show a status code of 200. the status did not change to 404

I just tried a random bad URL from the site in your signature

http://www.questlot.com/forums/1998

On the website - http://httpstatus.io

And it is showing a 404 response code indicating the mod is working. What URL are you trying that you are getting a 200?

Also- Please "Mark as Installed"

Just received this in an email from from Google Webmaster Tools:

I'd say that's pretty good evidence that this add-on is working perfectly. :)

nice mod. should i install this along side dbseo?

Would probably just suggest DBSEO add the feature if they haven't already, but I see no reason it should be a problem.

Are your sure brother Bird

Using your mod along with latest dragonbyte wont get any conflicts right ?

Regards,

I'm not sure about using it with the dragon byte mod, no- as I don't have DBSEO installed and never intend to install it- so I will never be testing my mods against it- just like I never tested them against VBSEO. It is most certainly use at your own risk.

I don't believe there will be a conflict. I also don't know when it will be added to DBSEO- but I was in communication with them and was told they do intend to add the feature. I suppose plans can always change but it is such an easy feature I doubt they would change their mind.

There is an error show " The following dependencies were not met:
This product is not compatible with version 5.3.28 of PHP. (Compatible starting with 5.4.0)"

That means that your server is running PHP 5.3.28 but I made it clear you need at least PHP 5.4.0 in the description of this mod. It will not work on your server.

Hi BOP5 and all,

Question for you if I may

I regularly check google webmaster tools for crawl errors such as this, on our forum www.legalbeagles.info/forums/ we have a VIP area http://www.legalbeagles.info/forums/...-MEMBERS-FORUM which google obviously does not have crawl access to.

If a member posts(publicly and crawlable) a link to a thread/post contained in the VIP area google crawls that post, tries to follow the link and comes up with the no permissions page.

This is then logged as a soft 404 error in webmaster tools. What is the best way to either return a different code that will not affect the crawl errors or prevent it from returning an error?

Would this mod fix this?

This is exactly the mod to fix this. Under the default setup Google would instead get a 403 (not authorized) code on such pages.

Excellent thank you,

We are running vB 4.2.1, not sure of the php version as yet, how would I check this?

Found it, PHP Version 5.2.10

So reading back this mod will not work for us :(

Check with your hosting company, they may have a simple line code to add to your .htaccess to access 5.4

Will do thanks.

Just looking through the templates to see where nopermission generates a 404 error to see if this can be modified

If you do upgrade to PHP 5.4.x you must upgrade to VB 4.2.2 as well. You might as well upgrade to VB 4.2.2 first.

Fairly sure it isn't in the templates. If it is expressly set at all it would be in the PHP code.

After looking through which template was returning the 404 error

STANDARD_ERROR

I replaced the 404 Not Found text to read 403 Forbidden

I checked that a genuine 404 page still came up as such so I know it hasn't affected that.

Not ideal but will see what results webmaster crawl errors comes back with over the next couple of weeks and let you know.

I think there may be some confusion... Standard Error doesn't return a 404 error... If it did this mod wouldn't be necessary- the response code on standard error is 200 (OK) regardless of the actual text displayed in the browser- the text in the browser doesn't determine the response code Google gets. The response code isn't a visible element.

Am I right you have VBSEO installed? I don't think there is any other means vBulletin would display a 404 error message?

Thank you :)

Perfect Mod, should be standard built into vbulletin!!

Mick

Awesome

it works without any issue on 4.2.2 PL1 - Vbseo installed.

First error number was 200
after installing this, it is 403

thanks BirdOPrey5

Hi and thanks for this!

In my webmasters tools I get a load of errors like this all related to tags

https://vborg.vbsupport.ru/external/2014/04/15.jpg

Shouldn't this modification fix this with the 403 no permission message?

Crawlers are getting to these pages (Tags) and because I don't allow guests to use the search function they are getting these type of errors and marking it as a soft 404.

https://vborg.vbsupport.ru/external/2014/04/16.jpg

https://vborg.vbsupport.ru/external/2014/04/17.jpg

Any idea TIA. :cool:

I can confirm http://www.techkings.org/tags/worked.html is now getting a REAL 404 response code. So those soft errors should go away soon.

These are errors I am still getting after installing this modification.

I cleared all the errors in webmasters tools (and I had loads) and these tag errors have come back after installing this, see the "detected dates" above in the image :up:

if you put any of your "tag" URLs into this site- http://httpstatus.io - it confirms a 404 is being returned. It isn't 403 because by default the mod sends 403 for "No Permission" errors only and 404 for all others.

The dates on the image are all several or more days ago so I assume you won't see any more past today assuming the mod remains active.

However it is pretty clear you should add your entire /tags/ path to your robots.txt file, that would be the right way to prevent the crawl issues.

installed and result is great. check soft.png (attached)

just one question , after installing this Access Denied urls increased (access.png) . All these urls are for the threads which has been moved to Trash and those redirect to Trashcan url. is it ok or bad for SEO?

From what I've read (I'm not claiming to be an expert) a 403/404 won't hurt overall SEO although the actual page causing the error will eventually be de-indexed. (Which is exactly what we want to happen.)