PubSubHubBub for vBulletin 4.x - Realtime Indexation of New Threads!

[MOGmartin]

PubSubHubBub for vB 4.x.x

IF YOU LIKE THIS PLUGIN, OR ARE A FAN OF SEO, PLEASE FOLLOW ME ON TWITTER! (http://twitter.com/seoforumsorg)

About PubSubHubBub (PuSH):
PubSubHubBub is a new protocol for realtime indexation of webpages. It works by pushing new content to a distributed network of PuSH compatible servers, rather than relying on a crawler hitting the board to find new content.


Is it similar to Ping-o-Matic?:
Similar yes, but PuSH works in a slightly different way, and is getting some heavyweight support from big players like wordpress (http://ma.tt/2010/03/pubsubhubbub/) who have included it on every wp.com blog.


Where can I learn More about PubSubHubBub?:
Try the official google documentation (http://code.google.com/p/pubsubhubbub/) for full information.
Alternatively, watch this video by google. (http://www.youtube.com/watch?v=B5kHx0rGkec)


How do I install it?
Simple, upload the files in the upload.zip to your forum root, and import the enclosed XML in the adminCP where you would normally import products.


A Word about Copyright:
The file push.php is NOT my copyright, it is infact copyright of InternetBrands Inc (the owners of vBulletin). It is mentioned in the file itself that it is not my copyright, and it is ONLY FOR USE BY ACTIVE LICENSE HOLDERS. I am distributing this with consent of both vBulletin and InternetBrands (see the next section for more details)


The History of This Plugin:
Its a plugin I was never going to release publicly, I use it on my seo forum (http://seoforums.org) and was quite happy leaving it at that. I experienced a measurable increase in spider activity, and time to indexing after getting it working for the first time.

However - D.Bradley @ Internet Brands got in touch and asked me to release it for the community.

I would also like to thank Fabian Schonholz of vBulletin.com for reviewing it and allowing the re-publication of the copyrighted material within push.php
- without those two people evangelising this plugin, I would not have released it publicly.


How do I tell if its working?
Thats proving to be a bit of a problem, the only testing tool that Im aware of for PuSH hasnt been giving stable results for the last few weeks, and other PuSH clients like the ones within wordpress.com blogs are also giving inconsistant results. I am however in touch with google who are reviewing the plugin.

In the meantime, I suggest you install the plugin, and keep an eye on two factors, 1) spider activity in google webmaster tools, and 2) time to indexation of new threads (by manual checks for now). As soon as I get some joy back from them I will update this post.


Can I remove the powered by.... accreditation link?
No. Doing so invalidates the license by which I am distributing this software.

If you must use a version without the accreditation link, I am selling this plugin in a GOLD version for just ?10 over at my plugin shop: http://seoforums.org/cart.php.

Please note, if you are using the GOLD version, you still need an active vb license to download the push.php file, which is only available on this page at vb.org.


A Word About Support:
I sometimes go a few weeks or more without coming back to vb.org so the best place to get in touch is either on http://twitter.com/seoforumsorg or on http://seoforums.org - and PM me on "MOGmartin".

[MOGmartin]

In response to Brandon's question below (I guess I should have put this in the post above!):


This plugin uses a specific pubsubhubbub "hub" which is part of the wider PuSH network.

I have setup this hub SPECIFICALLY for vBulletin users, and its available here: http://vbulletin.superfeedr.com


thanks

[Brandon Sheley]

interesting.. thx, will check it out

/me tags

I may add this, but it would be the gold, aka no copyright links
I'll wait a few days to see how it works for others, maybe add the free version for now.

thx for the mod, looking forward to seeing how this works.

ps.. what's this site for?
vbulletin.superfeedr.com

[MOGmartin]

interesting.. thx, will check it out

* Loco.M tags

I may add this, but it would be the gold, aka no copyright links
I'll wait a few days to see how it works for others, maybe add the free version for now.

thx for the mod, looking forward to seeing how this works.

Sure thing Brandon - it is really a beta release I guess anyway, (as mentioned in the post, I never intended to release it publicly until IB asked for it).

[Brandon Sheley]

well I'm glad you did, but I'm a bit of a nut for anything that has to do with seo/spiders/serp's :D

[Kolbi]

I?ll have a look at it :)

[MOGmartin]

well I'm glad you did, but I'm a bit of a nut for anything that has to do with seo/spiders/serp's :D

:D you and me both mate!

I was going to release this plugin in October, as I was going to talk about it in my presentation at the SEOmoz Conference (http://www.seomoz.org/ugc/pro-seo-seminar-presentation-on-the-mayday-update), but my seminar lead away from that particular topic. Will have to do it next time!

[puertoblack2003]

i thought you were missing in action :D, I'm a fan of your product, Another good addition to seo,Are you going to update the other products you have here?

[MOGmartin]

i thought you were missing in action :D, I'm a fan of your product, Another good addition to seo,Are you going to update the other products you have here?

I'll take a look at the other threads and see whats going on mate ;) any future releases might be gold only though.

Again, I've only released this one because I was asked to (nicely!) :)

[XiTCLUB]

Hubs (one per line) ? what to enter here?

[saadessa]

installed iam going to test on my site thank you

[BadgerDog]

Installed for testing ... thank you .. :)

Questions:

As a Google Adsense publisher for both content and custom search engine (searches only our site internall), I assume this doesn't conflict with or violate any of our agreements with Google that could jeopardize our account with them and get us canceled?

Also, will see any improvement in how quickly new threads posted on our site appear in our Google custom search engine?

Finally, do I need to do this action below to get the benefits of this add-on and if so, sorry for my ignorance, but I don't understand this vernacular to know what to fill in for the answers?

Send an POST request to http://vbulletin.superfeedr.com, with the following params :

* hub.mode : subscribe or unsubscribe
* hub.verify : sync or async
* hub.callback : http://domain.tld/your/callback
* hub.topic : http//feed.you.want.to/subscribe/to

Regards,
Doug

[SuperTaz]

Will this conflict with vBSEO software?

[MOGmartin]

installed iam going to test on my site thank you

That setting is there to manually add new hubs to be used when threads are published. You dont need to change anything though as its pre-configured to work.

[MOGmartin]

installed iam going to test on my site thank you

I would REALLY appreciate people's feedback on this plugin, yours and anyone else who installs it. Personally Ive seen great results, but I really want other people's evidence as well.

thanks!

[MOGmartin]

Installed for testing ... thank you .. :)

Questions:

As a Google Adsense publisher for both content and custom search engine (searches only our site internall), I assume this doesn't conflict with or violate any of our agreements with Google that could jeopardize our account with them and get us canceled?


Absolutely not, PubSubHubBub is a google supported indexation protocol, nothing more - it has zero impact on adsense etc.


Also, will see any improvement in how quickly new threads posted on our site appear in our Google custom search engine?


Hmmmmm.... that is a fantastic question.

I dont know if google CSE is rigged up to PuSH yet, a lot of google services are - but not all (this is a pretty new protocol after all). I will do some research of my own and see what the status is. Thanks for asking, its a valid point!


Finally, do I need to do this action below to get the benefits of this add-on and if so, sorry for my ignorance, but I don't understand this vernacular to know what to fill in for the answers?

Send an POST request to http://vbulletin.superfeedr.com, with the following params :

* hub.mode : subscribe or unsubscribe
* hub.verify : sync or async
* hub.callback : http://domain.tld/your/callback
* hub.topic : http//feed.you.want.to/subscribe/to

Regards,
Doug

No, you dont need to worry about any of those instructions, thats handled internally.

cheers
martin

[BadgerDog]

Thank you Martin ... :)

You've been very helpful... :up:

I'm up live and running with it, so I'll provide feedback (pro or con) once I have some valid observations to give you.

Regards,
Doug

[Alfa1]

Could you please release a vb3.8 version?

[MOGmartin]

Could you please release a vb3.8 version?

Im about to release a new revised version in 20 minutes, but I dont have a 3.x board (I moved onto 4 immediately after its gold release).

When Ive got the new version uploaded (v0.2) give it a shot on your board and see if it installs.
If its working correctly you will see the rel hub vbulletin.superfeedr.com in your header meta tags.

thanks

MOGmartin

[MOGmartin]

Could you please release a vb3.8 version?

Ive just uploaded a new version, but havent got a 3.x board to test it on.

If you are confident enough with vB, please give it a try, but I cant guarantee it will work, nor if it will install - its been over a year since I used vb3....

Please let me know how it goes, if it installs I will release it for vb3 as well.

thanks

Martin

[BadgerDog]

Now updated to v0.2 and testing ..

Regards,
Doug

[LuisManson]

do i need something special to add https://pubsubhubbub.appspot.com ?

[LbR]

What a sexy name "PubSubHubBub"

[FReeSTER]

I was getting an Fatal Error after this Mod got Installed on my forum, only appear to happen when New Threads or New Post were posted in forum

Fatal error: Call to undefined function curl_init() in /home/myfreene/public_html/forums/global.php(29) : eval()'d code on line 88

So I Disable the Mod and the forum is back to Normal.
Any Ideas how to fix this issue?

By the way im on vb4.0.8 pl2

[COL NIL SATIS]

tagged

[argothiusz]

Installed but there is no setting except for filling out the hub url o.O

[XiTCLUB]

What's New in Version 0.2?

[ForenOpfer]

Fatal error: Call to undefined function curl_init() in /home/myfreene/public_html/forums/global.php(29) : eval()'d code on line 88

Same here... What do i need to add in the only field in the settings to push an information about new posts and threads? O.o

Best Regards from Germany
ForenOpfer

[Alfa1]

Which search engines use this protocol? Only Google or more?

[Taurus1]

Looks interesting. Can I use this with vBSEO?

[MOGmartin]

Same here... What do i need to add in the only field in the settings to push an information about new posts and threads? O.o

Best Regards from Germany
ForenOpfer

I was getting an Fatal Error after this Mod got Installed on my forum, only appear to happen when New Threads or New Post were posted in forum

Fatal error: Call to undefined function curl_init() in /home/myfreene/public_html/forums/global.php(29) : eval()'d code on line 88

So I Disable the Mod and the forum is back to Normal.
Any Ideas how to fix this issue?

By the way im on vb4.0.8 pl2


Hmmmmm... Crap. This is exactly the kind of error that prevented me from releasing this mod in the first place!

So, most other people dont have the problem, do you two have any other mod's installed that function at a posting level (ie. things that are triggered when a new post is submitted)?

[MOGmartin]

Which search engines use this protocol? Only Google or more?

Sure, lots of companies are using it not just google, and its important to point out that Google are also using it in lots of different ways, and are only just implementing it in many "divisions".

You can check how google are doing on their pubsubhubbub progress page here:
http://code.google.com/p/pubsubhubbub/wiki/HubbubAtGoogle

Furthermore, you can see a brief list of other companies also hooked up to the PuSH infrastructure (both publishers and readers):
http://code.google.com/p/pubsubhubbub/wiki/Companies

[MOGmartin]

Installed but there is no setting except for filling out the hub url o.O

correct. There arent really any settings, its an "on or off" deal.

Check out the official wordpress plugin, you will find that they dont even have a hub setting ;)

[MOGmartin]

Looks interesting. Can I use this with vBSEO?

Yes. Im using both together.

VBSEO ofcourse uses ping o matic to a similar effect, so its arguably less useful on a VBSEO enabled board, but.... Pubsubhubbub is gaining a much wider reach into the search engines (see my post a few above this one) so I think its probably worth leaving on.

[BadgerDog]

Ok, need some advice ... :)

Running Version 0.2 and I'm not sure what I'm supposed to be seeing as a result. :confused:

Perhaps a jump in the number of "spiders" accessing our site? If so, I think there was an improvement in that area last night, but I'm not sure.

Sorry to be so vague ... :o

Regards,
Doug

[MOGmartin]

Ok, need some advice ... :)

Running Version 0.2 and I'm not sure what I'm supposed to be seeing as a result. :confused:

Perhaps a jump in the number of "spiders" accessing our site? If so, I think there was an improvement in that area last night, but I'm not sure.

Sorry to be so vague ... :o

Regards,
Doug

Hi Doug,

see my paragraph in the original description titled: "How do I tell if its working?" - as its something that works entirely transparantly, I myself am finding it tough to give any colour to that bit!

So, over the space of a few weeks, I saw a noticeable upshift in spider activity on google webmaster tools, and I also ran lots of tests with either PuSH/PingOmatic/nothing active and measured their inclusion time in the main google index. Of the three test groups PuSH and PingOmatic were both in the <minutes range, which the "nothing" group was significantly longer (hours).

Incidently, Im waiting on google helping out on this point (as developers of wordpress PuSH plugins).

cheers

Martin

[MOGmartin]

What a sexy name "PubSubHubBub"

I agree. You just cant ignore a new protocol when its so snappily named.

;)

[BadgerDog]

Hi Doug,

see my paragraph in the original description titled: "How do I tell if its working?" - as its something that works entirely transparantly, I myself am finding it tough to give any colour to that bit!


Hello Martin ... :)

Ok, I'll keep an eye on these

1) spider activity in google webmaster tools

2) time to indexation of new threads (by manual checks for now)

Just anecdotal feedback, but I think it's possible new posts from our site are appearing faster in our Custom Google Search engine and the spider count seems to be higher as well.

Will advise when I see a definitive pattern ... ;)

I have not had any SQL errors as reported by others here. We're running v4.04pl1 of vBulletin with a LOT of add-on mods.

Regards,
Doug

[MOGmartin]

Hello Martin ... :)

.....Will advise when I see a definitive pattern ... ;)



Im playing about with the google CSE as well now, I dislike the native vb search functions - but am not convinced yet by the google offering. Might try out the digitalpoint sphinx search as well.


I have not had any SQL errors as reported by others here. We're running v4.04pl1 of vBulletin with a LOT of add-on mods.

Regards,
Doug

Me neither, Im running 4.06 with dozens of plugins, 4.1 on a test board, and 4.07 on another, with no sql problems at all. (sorry you two that did have problems, diagnosing it is a bit difficult ;))

[BadgerDog]

Im playing about with the google CSE as well now, I dislike the native vb search functions - but am not convinced yet by the google offering. Might try out the digitalpoint sphinx search as well.

I agree .. hate vBulletin's search.... :up:

I like Google's CSE we just started testing, but I do think there is a lag in how quickly content gets updated after new posts are placed. I also don't like the fact that content that's restricted to only certain user groups displays in Google's CSE. I do realize that when an unauthorized user group member clicks on that type of link, vBulletin does present the user with the crappy "unauthorized" screen, but I wish we had a way of stopping Google's CSE from displaying it in the first place. ;)

Anyway, Digitalpoint Sphinx sounds interesting, but I'm just a 62 year old Fortran programmer out of the 1960's era, struggling to support our not-for-profit collectors site where the average demographic age is 60+ .. :D

So, I doubt I could handle the install of that product unless it loaded as simply and elegantly as yours. :)

Do you know any contractors who could quote a price for us to install and test this product?

Thanks Martin.. :)

Regards,
Doug

[MOGmartin]

Anyway, Digitalpoint Sphinx sounds interesting, but I'm just a 62 year old Fortran programmer out of the 1960's era, struggling to support our not-for-profit collectors site where the average demographic age is 60+ .. :D

LOL - well, Im a just a 31 year old BASIC programmer out of the commodore64/Amiga generation!

On a more serious note, Ive read the documentation that Shawn Hogan wrote up (he is "digitalpoint" here on vb.org, and owns forums.digitalpoint.com) and it looks a bit tougher than the average mod installation.

Im sure it would be relatively easy, just pretty time consuming - never having done it I dont know though sorry...



Do you know any contractors who could quote a price for us to install and test this product?



sorry, not off the top of my head - I have a few coders I use to build bits n pieces for me, but I dont know anyone other than shawn that has installed sphinx on vB.

Having said that, if your board is huuuuuuugge..... Id give it a shot. Take a look how it works on his board, with tens of millions of posts and it works a charm.

[abdelali]

Im using vb 4.1 PL2 and i dont have any error.ive seen a jump spider activity but new threads didnt get in google faster.

Regards.I like the idea of this mod still working on it you ll get results.

[FReeSTER]

Hmmmmm... Crap. This is exactly the kind of error that prevented me from releasing this mod in the first place!

So, most other people dont have the problem, do you two have any other mod's installed that function at a posting level (ie. things that are triggered when a new post is submitted)?
No not that i know, at least anything related to new posting templates no. :o

[OldSchoolDSL]

Working (I assume) on 4.0.8

[XiTCLUB]

Hubs (one per line) ? what to enter here?

Hubs (one per line) ? what to enter here?

[OldSchoolDSL]

Hubs (one per line) ? what to enter here?

I'll 2nd or 3rd the question. The only thing that seems to be there at this moment is

http://vbulletin.superfeedr.com

[ForenOpfer]

No not that i know, at least anything related to new posting templates no. :o

Me neither :/

[MOGmartin]

Hubs (one per line) ? what to enter here?

nothing. leave it as it is....

.....unless you build your own hub, then you can add that.

[OldSchoolDSL]

nothing. leave it as it is....

.....unless you build your own hub, then you can add that.

DONE :)

Go to superfeedr.com and register your own free account.

[MOGmartin]

DONE :)

Go to superfeedr.com and register your own free account.

;)

when I said "build your own", I really meant, install pubsubhubbub server, onto your own server - that is something I could see real value in doing, which is why the option is there!

[OldSchoolDSL]

;)

when I said "build your own", I really meant, install pubsubhubbub server, onto your own server - that is something I could see real value in doing, which is why the option is there!

Ah. I see.

Not sure if that was something I could do with "shared hosting", but I'll settle for the off site push for now.

So now I've got the original line and my own;

http://vbulletin.superfeedr.com
http://sociallyuncensored.superfeedr.com

[MOGmartin]

Me neither :/

sorry, Ive no clue then....

anyone?

[BadgerDog]

Update on testing of Version 0.2 ...

I do believe there has been a strong jump in the number of "spiders" accessing our site...

I don't believe there's any improvement in our Google CSE, in fact it seems to me that it's actually taking longer now for threads to show up after they're published. :confused:

Does that make any sense and if so, why?

Regards,
Doug

[MOGmartin]

Does that make any sense and if so, why?



nope, it certainly shouldnt have any negative effect, but not having tested CSE I can't really comment on it.

Been looking through the net to see if CSE uses PuSH yet, and cant seem to find that either. In the case (likely) that it doesnt, it certainly wouldnt have a negative effect though.

[MOGmartin]

Does anyone have any initial feedback on this plugin for me? I would love to hear your comments by PM please - any stats you can work out will be great!

[BadgerDog]

Does anyone have any initial feedback on this plugin for me? I would love to hear your comments by PM please - any stats you can work out will be great!

Anecdotal only ... I believe our spiders have increased but I see no perceivable affect on our Google CES ....

Regards,
Doug

[MOGmartin]

... but I see no perceivable affect on our Google CES ....


HI Doug,

as covered above, you shouldn't see any affect on your CSE ;)

[CRDeveloper]

I just installed and tested!!

No errors, working fine 4.1 PL2!!!

[SuperTaz]

Thanks.

[combs]

Slows down the quite a bit, had to uninstall, submitting posts took an extra 2 seconds each

[BadgerDog]

Slows down the quite a bit, had to uninstall, submitting posts took an extra 2 seconds each

Been wondering about this ....

Can anyone else confirm it?

Thanks .. :)

Regards,
Doug

[Videx]

Is that what the delay is? I don't really mind the delay, but why would this mod do that? Is it happening for all my members?

[The Rocketeer]

how well does this work on the latest vb and vbseo?
can anyone confirm its SEO capability or an increase

[DNN]

I'll hold off on this one until I learn more about it

[The Rocketeer]

seriously looking for some good input from the author.

How does this plugin help my content in google search engine. lets say I create an article based on a specific subject and title. Lets say its a recent news of some sort, but it takes hours for google to index it, sometimes even a day till I can find my site on google using the keywords.... So if I use this, will that get improve drastically..

like moments after I make the post I go on to google and search the keywords and select "latest"

[Mukashi]

Some time recently (if not today) the copyright footer of this addon appears to have been replaced with the following lines.

bring roger to Spain for a Mozcation.
C'mon Mozzers, you know spain is full of "increible!"
The word "Mozcation" was set to a specific URL.

I'm assuming this is something the author has done for personal SEO purposes (I understand there was some SEO competition related to this), but quite frankly I find it unacceptable. It's not only a breach of trust, it seems the way you've gone about this is a huge XSS security vulnerability for any site using this. It's pulling code from an external source without validation. You could've shoved practically anything at our users.

[BadgerDog]

Uninstalled. ...

Regards,
Doug

[Videx]

FYI: To uninstall you also want to delete the two files in upload.zip from your forum root - push.php and pushapi.php .

[BadgerDog]

FYI: To uninstall you also want to delete the two files in upload.zip from your forum root - push.php and pushapi.php .

Thanks ... :up:

Regards,
Doug

[sportsfroma2]

Some time recently (if not today) the copyright footer of this addon appears to have been replaced with the following lines.


The word "Mozcation" was set to a specific URL.

I'm assuming this is something the author has done for personal SEO purposes (I understand there was some SEO competition related to this), but quite frankly I find it unacceptable. It's not only a breach of trust, it seems the way you've gone about this is a huge XSS security vulnerability for any site using this. It's pulling code from an external source without validation. You could've shoved practically anything at our users.

Yeah- if this was included as part of some update, I guess I could understand to some degree... But this is kind of like remotely change our footer- not cool in my opinion..

So although I like the mod, I just don't know what you're going to do next- starting tomorrow you might be linking our sites to some adult content or stuff like that.. Can't take that risk.

Will have to uninstall, I guess
FYI: To uninstall you also want to delete the two files in upload.zip from your forum root - push.php and pushapi.php .
Thanks- would have totally forgot about those files

[ngocol]

sorry for asking im new for the Vbulletin could you explain to me ..:
do i have to insert this line ??

hub.mode : subscribe or unsubscribe
hub.verify : sync or async
hub.callback : http://domain.tld/your/callback
hub.topic : http//feed.you.want.to/subscribe/to

and if yes should i change the word "your" and you to my site ? or can you give me an example please . thank

[8thos]

I am just now seeing this thread. Wow.

Last february I imported my threads to digg.com through use of a feature they got rid of long time ago.

I had nothing but superfeedr spiders. They made my traffic jump. That digg.com feature was more potent than anything I've ever used.

I don't think this modification is related to that digg.com feature but I can vouch for the awesomeness of superfeedr.

[pete_brady]

Disabling this took ~3 seconds off my page load times. No choice but to uninstall.

[Supramania]

Disabling this took ~3 seconds off my page load times. No choice but to uninstall.

Same here... I've been optimizing everything else, I never thought to look at this plugin.

Forums have gone from "sluggish" to "fast" in the words of my users. What a big difference.

Uninstalled.

[Successfulsteps]

OH my, so glad I continued reading .. I cannot afford my forum slowed down nor random links. I was actually reading on to see if it was compatible with 4.1.4, as I stopped updating due to all the customizations I have. But customizations also can slow down the forum and I don't want another one that does. Too bad, looked like a useful tool also. But still, the random link is not professional IMO. And that says a lot about what I would like our members to see. If I can't control the content to be up to the standards of my forum, this would cause for concern.

Thanks for those who posted these issues. Saved me a potential problem or two.

[alhidaya]

I have version 4.1.1.1

I just installed. I will give news in a while. For now, I see no inconvenience in my forum.

Thank you for this product

[Naijasite]

Please what am i suppose to add to this field Hubs (one per line) is it my rss feed or Doamin?

[tpearl5]

For anyone trying to use this, it looks like pushapi was trying to call a script for the sole purpose of inserting the copyright link. The script URL is no longer valid, which is probably why this is causing your pages to load so slowly.

It looks like the author no longer owns seoforums.org so I doubt he'll care if you disable the copyright at this point.

[vimarc]

Does this help in SE ranking

[ringnews24]

I had to disable this, it makes my forum run mega slow when its enabled.

[dizzynation]

Great idea, but its 2013 and google has spent alot of money on their bots and make them more efficient. Digg.com was one of the best instant indexing around, but since it has died down use facebook and a twitter account. Anyone remember when you would submit a link to Digg and before you could even open a new browers window it was already in google?!,,,,