vb.org Archive - New Posting Features - Automatic Thread Tagger (Project Terminated)

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- Modification Graveyard (https://vborg.vbsupport.ru/forumdisplay.php?f=224)

- - New Posting Features - Automatic Thread Tagger (Project Terminated) (https://vborg.vbsupport.ru/showthread.php?t=179927)

Automatic Thread Tagger (Project Terminated)

This modification has now been taken over by Phalynx updates/information etc can be found here:

https://vborg.vbsupport.ru/showthread.php?t=185495

Do not post on this thread, or mark this thread as installed.

Old Post:

Quote:

Unfortunately I find myself in a position where continued development of this modification is not possible. So as of today (29/05/08) this project is terminated. If you are a coder feel free to take this project on and take it from beta to gold.

Quote:

You will be please to learn that this modification has been taken over by Phalynx.

https://vborg.vbsupport.ru/member.php?u=49918

Automatic Thread Tagger v1.0
Beta 3

THIS IS A BETA MODIFICATION

Please read the entire post before diving in

If you report a bug please post the thread title that created it, without this I cannot test it and improve the language parsers

When a user submits a new thread this product will automatically take keywords from the thread title and use these as tags.

As an example, if a user submits a thread with a title of:

Quote:

Fish Food for Cats

The thread will be automatically tagged with:

Fish
Food
Cats

If the user also submits an actual tag of "Fish" this will not be duplicated. Additionally this product hooks in before vBulletin processes tags so any rules you have setup for this will be respected. It also checks if you have tagging enabled, if not the product does nothing.

If you choose to do so this product will also automatically tag threads created by incoming RSS feeds.

And if all that wasn't enough there is a whole load of different configuration options.

Installation

If you have a previous beta version of this modification install this MUST be uninstalled before proceed to install this version

Download the modification zip file to your local machine and extract the contents.
Upload the functions_autotagger.php to your includes folder (the same directory as your config.php)
Goto "ACP -> Plugins & Products -> Manage Products".
Click "Add/Import Product".
In the "Import Product" section, click the browse button and select the downloaded XML file (product-auto_thread_tagger.xml).
Click "Import"

After install, and by default the modification is disabled, this will allow you to play around with configuration before switching it on.

Configuration
Configuration for this product can be accessed via:

ACP -> vBulletin Options -> Automatic Thread Tagger

Each configuration entry has a description against it, so I won't rehash it all here. However the following configuration items are worth mentioning:

Disable Auto Tag if Tagged?

If this option is enabled and a user submits tags with their post then the title keywords will not be added.

Use Smart Quotes?

If this option is enabled (default) then the auto tagger will treat quoted terms as a single tag. For example if a user submits a title of:

Quote:

This is a great technical website

The tags will be:

great
technical
website

However if they submit a title of:

Quote:

This is a great "technical website"

The tags will be:

great
technical website

Filter / Replacement Characters

This is the heart of the modification, as we are dealing with human input it can be rather complicated for a machine to handle and parse meaningful keywords. This configuration area allows you to configure filter and replacements for incoming thread titles. For example, you might not want & in your tags, through the options here you can filter it out or replace it with and.

In addition to this you can also use filter/replacements to extend tags. So if you run a motorbike website you might want to extend the term "GSXR" to "Suzuki GSXR" or change it completly to "Fast Bike".

Filters/replacements do not change titles, they only effect the resulting auto tags.

This configuration area is pre-populated with a series of default rules, these rules are based on my experience during development and should cover most things. However as mentioned this is a beta modification and we are dealing with human input so with your help and input this list will be improved.

Composite Tags

With composite tags you can define a collection of words to use a single tags. So if for example you specify a composite tag of "brown dog", and the user submits a title of "I love my brown dog" the tags will be "love, brown dog" without this composite tag the tags would be "love, brown, dog"

RSS Feed Additional Tags

If you use this modification to tag RSS created threads, you can also specify additional tags to add for each RSS feed. So for example if you have a feed from BBC News, in addition to the title based tags, you could also tag each thread with "General News".

In summary the other configuration values allow you to:

Globally enable or disable the product (default is off).
Exclude forums by id
Exclude usergroups by id
Exclude user by id
Exclude words from the search words stop list (default is on). This excludes words like can, by, do, etc (open includes/searchwords.php for a full list). I would strongly recommend not disabling this, however you could and then use the following:
Define your own list of exclude words.
Exclude RSS feeds by feed ID.

Using this to tag RSS feeds

Unfortunately vBulletin does not include (that I can see) a hook that is called when an RSS thread is created. This means that it is not possible to hook into the code and perform auto tagging. All is not lost though, however it does require an edit to one of the default vBulletin files.

This is what you need to do:

Open includes/cron/rssposter.php
Find the following line:

Code:

$itemdata->set('ipaddress', '');
On a new line immediately after this paste in:

Code:

require_once(DIR . '/includes/functions_autotagger.php'); $autotags = GetAutoTags($itemdata->fetch_field('title'), "", true, $itemdata->fetch_field('forumid'), $feed['userid'], $item['rssfeedid']); $itemdata->set('taglist', $autotags);
Save the file and place on you server.

Once this is done, and you have enabled RSS tagging in the configuration then RSS threads will be tagged.

I will find a solution to this as I really don't like the idea of having to edit files in this fashion. So if anybody can suggest a better way of doing this please speak up as it will mean I can get the solution out quicker.

Other Information

At the moment this product is most definitely in beta testing, once I and the people using it are happy with the product then I may consider releasing a hack that will tag all existing threads. Releasing this right now would probably cause more trouble than it is worth.

Parsing the thread text for tags is an extremely unlikely addition as this would require some fairly heavy processing to ensure quality of tags.

At the moment I am primarily concentrating on the parsing of human language and refining the filters/replacements. So adding Ajax support (i.e. auto populate the tag box as user enter words into the thread title box) is on the todo list however this won't happen until we are happy that the parsing system is as good as it can be.

One thing the product does not do which you may want to consider is change the "Separate tags using a comma." phrase used on the new thread post screen. You might want to consider changing this to something like "Separate tags using a comma, do not include words from your title". This depends on your personal preference and how you have the product configured.

Disclaimers
Obviously the input this product uses comes from human beings, therefore it is almost impossible to code for every single possible scenario. As such there is always a possibility that a specifically crafted submission may cause problems. I strongly suggest running this through some tests before committing to installing this on a live forum.

Ahh yes, one final thing - this is the first time I have created a product XML file, I have tested and install this and all seems fine - but it would be great if somebody in "the know" could cast an eye over it for obvious errors/issues.

Actually, one more one final thing, if you install it please click install :p

Hold on, one more one more one final thing, if you really like this mod then why not nominate it for MOTM ;)

Release History

1.0 Beta 1
First Release

1.0 Beta 2

Added filters/replacements
Added RSS support
Added RSS inclusion/exclusion support
Added composite tags
Added ability to tag all RSS threads with a specific tags (per RSS id)

1.0 Beta 3
Bug fix release

Fixes issues with escaped characters
Removed option to process filters on full title rather than potential tags.

As it seems to be the thing to do, I reserve the right to reserve the first post ;)

EDIT: As usual after I submit the mod I find another one that does the same thing that was posted before mine :( Anyway, you might want to look at this one as well:

https://vborg.vbsupport.ru/showthread.php?t=179772

Woow ,, Nice Work !!

Installed

wow great great great mod

thanks

please how i can active auto tags to rss forums which get threads automatic

Fantastic work.

Installed. Keep up the great work :)

Quote:

Originally Posted by paramegsoft (Post 1525950)

please how i can active auto tags to rss forums which get threads automatic

hmmmm, I didn't think of that and it would be useful for me as well.

I will investigate and get back to you

Great I can't wait to try this out. If anyone has tested the exludsion method what about partial,

Example
if I say exclude the word box will that also stop people from using boxer?

so useful.

tnx 4 it

Quote:

Originally Posted by Zaiaku (Post 1526000)

Great I can't wait to try this out. If anyone has tested the exludsion method what about partial,

Example
if I say exclude the word box will that also stop people from using boxer?

If the mod proves to be popular and there is sufficient interest I will probably add support for partial exclusions in a similar way to how VB handles censor words.

For example "dog" would exclude "dog, dogma, diddlydog", and "{dog}" will only exclude "dog"

Can you create a tool for creating/adding tags for all old threads ?

Quote:

Originally Posted by MrEyes (Post 1525975)

hmmmm, I didn't think of that and it would be useful for me as well.

I will investigate and get back to you

thanks

and hope you do it

I posted this in the first auto tagged mod that was previously posted, but I'll post it here too.

It would be great if there was a way to have old "non-tagged" threads also be updated with auto tagging.

A list of german stoppwords, maybe it helps someone:

Quote:

ab,aber,aehnlich,?hnlich,alle,allein,allem,aller,a lles,allg,allgemein,als,also,am,an,and,andere,ande ren,anderes,auch,auf,aus,au?er,author,autor,been,b ei,beim,bereits,besonders,bevor,bietet,bin,bis,bis t,b?den,boeden,bzw,da,dabei,dadurch,dafuer,daher,d amit,dann,daran,darauf,daraus,darum,das,dass,davon ,davor,dazu,dein,deine,dem,den,denen,denn,dennoch, der,derem,deren,des,deshalb,dessen,die,dies,diese, diesem,diesen,dieser,dieses,dinge,doch,dort,du,dun klen,durch,eben,eher,eigenen,eigenes,ein,eine,eine m,einen,einer,eines,einfach,einig,einige,er,erst,e rste,erster,es,etc,etwa,etwas,euer,eure,fall,finde n,for,f?r,ganz,ganze,ganzem,ganzen,ganzer,ganzes,g ar,geben,gegen,geht,gibt,gleich,gute,guten,habe,ha ben,hat,hatte,hatten,hattest,hattet,hier,hinter,ic h,ihm,ihr,ihre,ihrem,ihren,ihrer,ihres,im,immer,in ,information,ist,ja,je,jede,jedem,jeden,jeder,jede s,jedoch,jene,jenem,jenen,jener,jenes,jetzt,kann,k annst,kein,keine,keinem,keinen,keiner,keines,koenn en,koennt,kommen,kommt,k?nnen,k?nnt,langsam,lassen ,leicht,leider,lichten,machen,man,mehr,mehrere,mei n,meine,meist,mich,mit,m?chte,moechte,moeglich,m?g lich,m?ssen,m??t,muss,musst,nach,nachdem,nat?rlich ,neben,nein,neu,neue,neuem,neuen,neuer,neues,nicht ,nichts,noch,nun,nur,nutzung,ob,oder,off,ohne,onli ne,per,schnell,schon,schwierig,sehen,sehr,sehrwohl ,seid,sein,seine,seinem,seinen,seiner,seines,seit, seite,seiten,selber,selbst,sich,sie,sind,so,soda?, solch,solche,solchem,solchen,solcher,solches,soll, sollen,sollst,sollt,sollte,sollten,solltest,sonst, soviel,soweit,sowie,sowohl,spielen,statt,steht,suc hen,?ber,um,und,uns,unser,unsere,unseren,unseres,u nter,version,viel,viele,vieles,vom,von,vor,wachen, wann,war,warum,was,weiter,weitere,welche,welchem,w elchen,welcher,welches,wenig,wenige,weniger,wenn,w er,werde,werden,werdet,weshalb,wie,wieder,wieso,wi eviel,wir,wird,wirklich,wirst,wo,woher,wohin,wohl, wurde,wurden,zu,zum,zur,zwischen

Quote:

Originally Posted by jasculs (Post 1526032)

I will probably release something that does this once the product is polished.

I would rather not do it now as there is potential for things to go extremely wrong if you batch update 1000's of threads.

Quote:

Originally Posted by swissknife (Post 1526038)

A list of german stoppwords, maybe it helps someone:

Would they not be included in the german version of searchwords.php?

ups... I see... Forget my posting!

Thanx .. you are viry smart :)

this one is so much better then the other one! If you submit tags on the other one, the auto tag feature doesn't work correctly!

Installed!

This works great. Thanks very much.

Thanks alot! Does not require file edits like the other one does. Installed!

Great :)

just a question :

I Tested this title : " This is a TEST for Auto Tagger v1.0 "
It generated these tags : Test,Auto Tagger,v\.1\.0 .

Any solution to fix this too ???

Quote:

Originally Posted by Mecho (Post 1526190)

Great :)

just a question :

I Tested this title : " This is a TEST for Auto Tagger v1.0 "
It generated these tags : Test,Auto Tagger,v\.1\.0 .

Any solution to fix this too ???

Yeah I was getting the same thing with titles using ('s or )'s

can anyone tell me what the difference is between this mod and https://vborg.vbsupport.ru/showthread.php?t=179772 ?

i have a problem with parenthesis :(

when i posted new thread, auto tagger converted parenthesis' to new parenthesis' with "\" characters.

ex: word is (cio)

result with auto tagger $cio$

can you give any idea to solve it?

Great MOD. Would it be possible for tags associated with a deleted thread to remove themselves also?

This is an example of a perfect mod. It's taking an area that vB is lacking in and making it work a heck of a lot better.

I'll install once all the reported bugs are sorted :)

I second this request. Thanks.

Quote:

Originally Posted by Mesohu (Post 1526023)

Can you create a tool for creating/adding tags for all old threads ?

And thank you for this cool mod in general.

I'm vietnamese
I Tested this title : " Đóng góp ý kiến "
It generated these tags : góp,Đóng,kiến .
but I need: "Đóng góp,ý kiến
Any solution to fix this too ???I think this mod need 1 custom libarary muitiply word

It doesn't create tags for our rss feeds. Anyone else have this problem?

how add old thread auto tag...

Quote:

Originally Posted by MrEyes (Post 1525903)

Quote:

Originally Posted by Madlike (Post 1525977)

Idea ;

https://vborg.vbsupport.ru/showthread.php?t=179772

Isn't it ??

Quote:

Originally Posted by jhamaican (Post 1526221)

can anyone tell me what the difference is between this mod and https://vborg.vbsupport.ru/showthread.php?t=179772 ?

That other mod requires file edits, which means you'll have to reapply them for every upgrade. This one is an imported product with no file edits.

This is a lot better than the other, no hacking yourself into the source code. :)

However, one question ;

What if a user submits tags, does this stop adding them automatically? Also, are there any extra queries ect. we should know about?

Quote:

Originally Posted by xuanhuy238 (Post 1526518)

I'm vietnamese
I Tested this title : " Đ?ng g?p ? kiến "
It generated these tags : g?p,Đ?ng,kiến .
but I need: "Đ?ng g?p,? kiến
Any solution to fix this too ???I think this mod need 1 custom libarary muitiply word

This will be added in the next release

Quote:

Originally Posted by Mecho (Post 1526190)

I Tested this title : " This is a TEST for Auto Tagger v1.0 "
It generated these tags : Test,Auto Tagger,v\.1\.0.

Quote:

Originally Posted by ecihanuysal (Post 1526235)

i have a problem with parenthesis :(

A solution of sorts will be created in the next release (see comments at the end of this post)

Quote:

Originally Posted by Junkie (Post 1526326)

Great MOD. Would it be possible for tags associated with a deleted thread to remove themselves also?

I believe the default vBulletin software will do this.

Quote:

Originally Posted by paramegsoft (Post 1525950)

please how i can active auto tags to rss forums which get threads automatic

Working on this, see comments at the end of this post

Quote:

Originally Posted by Jase2 (Post 1526812)

What if a user submits tags, does this stop adding them automatically? Also, are there any extra queries ect. we should know about?

See the following:

The product does of course add some additional processing when a user submits a thread but this primarily uses what VB already has in memory so the impact is fairly low. It does not add any additional SQL queries.

In response to a few questions posted so far:

1) At the moment it does not support posts created via incoming RSS feeds, however I do intend to add this feature and I am in fact working on this right now.

2) As mention in the opening posts the product will not "retro-tag" already existing posts, the reason for this is that at the moment this is in beta and retro-tagging 1000's of posts with the current version could feasibly create alot of problem. Once the rule sets are defined then I will release something that will auto tag all existing threads.

3) As reported by a few people there are a few issues with non alphanumeric characters in titles (thing like ( ) \ /). The next version will have a fairly complex replacement system built once released this will allow you to perform filtering and replacements. So for example:

'\'=>' and '

will replace all instances of '\' with ' and '

This can also be used to extend tags, so for example you can set the following rule:

'GSXR'=>'Suzuki GSXR'

So if a user submits a title of "GSXR Paintwork", the tags will be "Suzuki GSXR, Paintwork"

I imagine that this replacement system will need, with your feedback, considerable tweaking to make it as good as it can be when process human language input.

4) Under the default configuration if the user submits tags with their thread then keywords from the title are still calculated but it will not duplicate. So for example if a thread title of "Fish food for cats" and tags of "pets, fish" are submitted, the final tags will be "pets, fish, cats". However the if the "Disable Auto Tag if Tagged?" option is enabled then if a user submits tags then the auto tagger will not do anything.

I am working on all this at the moment once I will release it as soon as I am happy that the changes are as good as they can be with single user/site testing.

And finally, thanks for the support it makes putting this together worthwhile :D

Thanks for explanation. Looking forward to a more 'stable' release. :)

This is really cool and, while not disrespecting the other guy's addon, a better piece of work to administer then 'the other' addon that does the same.

I'll echo my support from the other addons thread for something though;

AJAXifying it, making the 'tags' field autoupdate based on the title input. Make it just do the calculations and populate the 'tags' field when focus is removed from the 'title' field.

(Not being a coder enables me to use 'just' in all sorts of places, especially those where I know there'll be a lot of work to get it going :p.)

Phrase Groups Available:

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01593 seconds Memory Usage 1,865KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (2)bbcode_code_printable (22)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (2)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (40)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: