Before I re-invent the wheel... does anybody have a little cleanup script to remove duplicate posts?

In the vB ACP there is the Remove Duplicate option to remove duplicate threads but it doesn't handle duplicate posts.

Anybody? :)

Donnacha:

I just upgraded to vb 3.6.4 and exported messages in debug mode. It showed the messages being exported to my newsgroup server. I also logged onto the newsgroup server with my newsgroup reader and confirmed that my messages were there and that nothing looked strange in the headers, etc. Hopefully, it was just a temporary glitch for you?

Regards. -- Rik

Never mind. I ended up just creating a new "Remove Duplicate Posts" function in the ACP and that is doing the job fine. Taking forever to loop 1MM+ posts but it is working.

Thanks for the feedback Rik, much appreciated.

Donnacha

Has anyone figured out a way to use regex or some other means to filter out certain spam? I'm getting a large amount of viruses and other crap coming through my feeds which is guming up the database. It takes a long time to delete these threads (1 - 5 min EACH)

Please help.

Ladies and Gents,

I am offering $50 for the person who can resolve an issue with this mod, and update for 3.6

Please see this thread - https://vborg.vbsupport.ru/showthrea...73#post1145373

Did you try the killfile settings? (ACP => NNTP Gateway Options => NNTP Gateway Settings => Global Killfile) I have a few entries in my and it seems to be working as expected.

Check your newgroup properties to see if the last message ID is getting correctly updated; if that that'll at least help narrow down where the problem is occuring. How many messages are you trying to download? There is a chance that if you're trying to bring in a new group that the cron job runs once, times out (which doesn't update the last message ID), run again, and tries to bring in all of the messages again.

I'm guessing that about 95% of the messages that we import with the same title get properly merged into the prior thread of the same title. So perhaps 5% get orphaned and I have to merge them manually.

Is anyone getting a 100% proper merging of threads of the same thread title over time?

Thanks. -- Rik

PS:
I'm asking because if others are getting a 100% success rate, it may be a problem with our server (Although I've played around with setting delays between messages which doesn't, however, seem to affect things) and so it may be upgrade time.

For the most part. The only problem I have yet is where if a new thread arrives with a subject of something like "RE: Whatever..." and "Whatever..." doesn't already exist then it seems like it keeps creating new "RE: Whatever..." threads with each import (as opposed to keeping them all together). I haven't looked at the code lately in an attempt to fix it.

I'm running vB3.6.4 and Mailman 2.1.9 and using this script to provide gateways between vB and email.

vB->email: Took a while to figure out all the Mailman idiosyncrasies but this part of the script is running perfectly. I'm certain my users will love it.

email->vB: Struggling with this. I can send an email to MailMan's pop account and it will a) get properly reflected to all users, and b) show up in the MailMan archive. Or I can "reply" to an email from the gateway and it will also a) get properly reflected to all users, and b) show up in the MailMan archive.

What isn't working is vB reading the POP3 account and posting the "email" or "reply" into the message board thread. I'm not certain what might be going on except that MailMan is immediately reading the POP3 account, then clearing it and vB has nothing to import? What else could be going on?

Any ideas out there?

(Noob here - How do I click INSTALL to indicate I'm running this script?)