YaBB Community and Support Forum
YaBB Home About YaBB Download YaBB YaBB Support Customize Your Forum Development Contribute to the Project
  Welcome, Guest. Please Login or Register


 
Page Index Toggle Pages: 1
Topic Tools
 
Removing thousands of spam posts (Read 1,752 times)
 Mar 21st, 2013 at 10:07pm
There are no actions to perform.  

mw1197 
YaBB Newbie
*
Offline
Posts: 5


YaBB 2.1
Removing thousands of spam posts
Helping out a friend with a forum that's been neglected for years, to the point where it has about 24,000 spam posts made by thousands of different users (bots). It's running on YaBB 2.1 btw.

Before finding out YaBB is flat file based, my plan was to just go into the database and run an SQL query to delete all posts and users after a certain date. So when I found out there's no database, I figured I'd just be able to remove all the files from a certain point in the Messages and Members directory. But the board index doesn't seem to auto update...topics still exist, etc etc.

Has anyone ever dealt with removing this much spam? Obviously deleting posts/users one by one isn't an option at this level...surely there must be a more efficient way? Thanks.
 
 
IP Logged  
 Reply #1 - Mar 22nd, 2013 at 1:28am
There are no actions to perform.  

Dandello 
Global Moderator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline
Posts: 1,853
Earth


YaBB 2.5
Re: Removing thousands of spam posts
We've had some people with issues like this. I'm not familiar with 2.1 but, once the spam in removed, on newer versions of YaBB you can go to the Admin Center and under Maintenance Controls, running Rebuild Message Index and Recount Board Totals usually does the trick of removing references to the deleted thread files.
 
WWW  
IP Logged  
 Reply #2 - Mar 22nd, 2013 at 3:53pm
There are no actions to perform.  

Elrick 
YaBB Moderators
Beta Testers
***
Offline
Posts: 147
Edge of the Abyss


YaBB 2.5
Re: Removing thousands of spam posts
See this post by JonB.

How to Quickly Clean up Ghost Registrations. It worked a treat for me!!

 
~ Elrick ~
There is no direct experience of reality without interpretation; and all interpretation is corrupted by the cultural and personal prejudices or prejudgments of the interpreter.
 
IP Logged  
 Reply #3 - Mar 22nd, 2013 at 7:11pm
There are no actions to perform.  

Bill Myers 
God Member
Beta Testers
*****
Offline
Posts: 1,482
Los Angeles


YaBB 2.4
Re: Removing thousands of spam posts
At least regarding the deletion of spammers ...

Elrick wrote on Mar 22nd, 2013 at 3:53pm:
See this post by JonB.

How to Quickly Clean up Ghost Registrations. It worked a treat for me!!


That's a good example of how to delete a lot of members, but it's a bad example of what mw1197 actually wants to do.

mw1197 wrote on Mar 21st, 2013 at 10:07pm:
my plan was to just go into the database and run an SQL query to delete all posts and users after a certain date.

Happily enough, even though YaBB doesn't have a database per se, it can nonetheless query its membership data quite effectively as Jon instructed.

Important
: Before deleting that many members, make sure you have a good backup of all the members in your forum; just in case the backup is needed.


To do exactly what mw1197 wants to do, specifically to delete a lot of members at the same time, the following instructions should work:

Go to Admin Center > View/Delete Members ~ bottom of the page.

Set "Greater Than" ~ "120" ~ "Days Registered" ~ click the checkbox to the right.

Note
: Obviously, you'll want to set the "Days Registered" to however many days that fits your criteria.

Hint
: With thousands of different spam-bots that have registered, you may want to temporarily adjust the "Number of Members displayed per page in Member List" to 500 or 1,000 or whatever number that works best for you. That way, there will be less pages to cycle through as you perform this task.

Food for thought
: You may want to consider clicking the box to the right of "Send Email to deleted members:" before deleting members. That way, if you inadvertenly delete members in good standing, you can make them whole again by uploading a backup of their files onto the server.

If you choose to have the forum send deleted members automated emails, you may want to edit the file for that process so that it better explains why their membership has been deleted (for any members who are otherwise in good standing). This can be easily done through your Admin Center.

If I was doing this in our forum, I don't think I'd have automated emails going to the members I've deleted. I might instead put up a notice in the forum about it, and keep a local copy of all the members I've deleted just in case any of those files were needed.

******************************************************

As for the deletion of those approximately 24,000 spam posts made by all of those spam-bots, I think I may have a workable solution for that as well.

After I clear my head about it a bit, and do some experiments, I'll let you know here in this topic.

 
Morning, noon, or night, have a great one! ...
WWW BillHMyers  
IP Logged  
 Reply #4 - Mar 22nd, 2013 at 7:49pm
There are no actions to perform.  

Bill Myers 
God Member
Beta Testers
*****
Offline
Posts: 1,482
Los Angeles


YaBB 2.4
Re: Removing thousands of spam posts
mw1197 wrote on Mar 21st, 2013 at 10:07pm:
Before finding out YaBB is flat file based, my plan was to just go into the database and run an SQL query to delete all posts and users after a certain date.

Regarding those approximately 24,000 spam posts, a relatively quick, and effective way to get rid of them all would be to start fresh by moving all topics in any given board to a newly created board that only you and your forum's staff members can see. Create as many boards as you need based on however many boards you currently have.

Maybe call your new board, "Posts to Review, Board 1", and so on and so forth.

Or if you choose to be transparent about the process of getting rid of all the spam posts, move those posts to another board that everyone can see. You might call that board, "Spam Posts".

In other words, you do not need to delete any posts; you can just move them to another part of your forum in case any of them are ever needed. After all, you may end up moving posts that aren't spam, so those posts can always be moved back to their original board as needed, or desired.

If you do it this way, your forum will be cleaned up, and ready to be user-friendly again.

******************************************************

By the way, even though YaBB is flat file based, it should be easy enough to find, and delete those approximately 24,000 spam posts (after you've moved them to a staff-only section of your forum).

I say this because, presumably, your forum has been so neglected that it's not likely there have been any posts from legitimate members for some time; or at least only a few.

As such, you can simply open up a board to show a list of topics, many of which are probably spam topics, and those can be deleted. Or go page by page in each topic as you have time over the next few months; whatever; and continue with this process of getting rid of all those spam posts.

But again, my recommendation would be to start off fresh by essentially moving the bad part of your forum into a staff-only section, or to a section that is clearly marked as such. You don't even need to make the effort to find those spam posts; just move them all, and forget about them.

Good luck!

 
Morning, noon, or night, have a great one! ...
WWW BillHMyers  
IP Logged  
 Reply #5 - Mar 23rd, 2013 at 11:16pm
There are no actions to perform.  

mw1197 
YaBB Newbie
*
Offline
Posts: 5


YaBB 2.1
Re: Removing thousands of spam posts
Dandello wrote on Mar 22nd, 2013 at 1:28am:
Re: Removing thousands of spam posts
We've had some people with issues like this. I'm not familiar with 2.1 but, once the spam in removed, on newer versions of YaBB you can go to the Admin Center and under Maintenance


Thanks. I tried this, but got an error message when rebuilding saying that it was missing a txt file (presumably one of the spam messages I deleted). Maybe it's to do with 2.1.



 
 
IP Logged  
 Reply #6 - Mar 23rd, 2013 at 11:29pm
There are no actions to perform.  

mw1197 
YaBB Newbie
*
Offline
Posts: 5


YaBB 2.1
Re: Removing thousands of spam posts
Bill Myers wrote on Mar 22nd, 2013 at 7:49pm:
mw1197 wrote on Mar 21st, 2013 at 10:07pm:
Before finding out YaBB is flat file based, my plan was to just go into the database and run an SQL query to delete all posts and users after a certain date.

Regarding those approximately 24,000 spam posts, a relatively quick, and effective way to get rid of them all would be to start fresh by moving all topics in any given board to a newly created board that only you and your forum's staff members can see. Create as many boards as you need based on however many boards you currently have.

Maybe call your new board, "Posts to Review, Board 1", and so on and so forth.

Or if you choose to be transparent about the process of getting rid of all the spam posts, move those posts to another board that everyone can see. You might call that board, "Spam Posts".

In other words, you do not need to delete any posts; you can just move them to another part of your forum in case any of them are ever needed. After all, you may end up moving posts that aren't spam, so those posts can always be moved back to their original board as needed, or desired.

If you do it this way, your forum will be cleaned up, and ready to be user-friendly again.

******************************************************

By the way, even though YaBB is flat file based, it should be easy enough to find, and delete those approximately 24,000 spam posts (after you've moved them to a staff-only section of your forum).

I say this because, presumably, your forum has been so neglected that it's not likely there have been any posts from legitimate members for some time; or at least only a few.

As such, you can simply open up a board to show a list of topics, many of which are probably spam topics, and those can be deleted. Or go page by page in each topic as you have time over the next few months; whatever; and continue with this process of getting rid of all those spam posts.

But again, my recommendation would be to start off fresh by essentially moving the bad part of your forum into a staff-only section, or to a section that is clearly marked as such. You don't even need to make the effort to find those spam posts; just move them all, and forget about them.

Good luck!



Thank you for the detailed response, Bill. This looks solid and I'm looking into following this method.

And yeah, you're spot on. The forum has been neglected for years, so I highly doubt there's much if any valuable posts after the spam started. Maybe a few mixed in at the early stages, but once the spam ramped up, nothing.

So the way I see of moving posts is by viewing the board and checking the select all checkbox at the bottom of the page, and then choosing the board to move the posts to. That's fine, but the problem is that there's around 15,000 actual threads to move. Is there a way to show more than 20 threads at once in the board view, in the same way that you can view all members at once? I've been looking but don't see an option on the board view or in the admin center.

Also, a few stickied threads have valuable information in the first post, but then several hundred posts of spam following that. Any way to select all those posts at once to move or delete rather than doing it all one by one?

Thanks man.

 
 
IP Logged  
 Reply #7 - Mar 24th, 2013 at 4:35am
There are no actions to perform.  

Bill Myers 
God Member
Beta Testers
*****
Offline
Posts: 1,482
Los Angeles


YaBB 2.4
Re: Removing thousands of spam posts
mw1197 wrote on Mar 23rd, 2013 at 11:29pm:
So the way I see of moving posts is by viewing the board and checking the select all checkbox at the bottom of the page, and then choosing the board to move the posts to.

Exactly!

******************************************************

mw1197 wrote on Mar 23rd, 2013 at 11:29pm:
That's fine, but the problem is that there's around 15,000 actual threads to move. Is there a way to show more than 20 threads at once in the board view, in the same way that you can view all members at once?

Yes, you can do this, and you'll of course want to adjust this number to best fit your needs.

Go to Admin Center > Forum Settings > and scroll down to "Maximum Topics to display in the Message Index" - that's probably set at 20. For your purposes, I might change that to 500, and if that works, then maybe change that much higher.

Additionally, you may want to look at the following:

Go to Admin Center > Forum Settings > and scroll down to "Maximum number of messages to display on one page of a Topic" - that's probably set at 15. Change that to whatever you want.

******************************************************

mw1197 wrote on Mar 23rd, 2013 at 11:29pm:
Also, a few stickied threads have valuable information in the first post, but then several hundred posts of spam following that. Any way to select all those posts at once to move or delete rather than doing it all one by one?

You can follow the same instructions above, but after you've selected all of the posts, just uncheck the stickied threads, and that should do it.

By the way, if you end up inadvertently moving any posts or threads you didn't mean to move, you can simply go to those posts or threads in the newly created board for that purpose, and move back what you want.

Nice thing to remember: This method of cleaning up the spam posts simply moves posts around versus deleting them. This way, you kind of can't make any mistakes. Or more accurately, any mistakes that are made can be reversed however you want.

However you manage it all, you might want to stick a notice at the top of your forum so that everyone will know what's going on, and for them to bear with you during your spam abatement process.

 
Morning, noon, or night, have a great one! ...
WWW BillHMyers  
IP Logged  
 Reply #8 - Mar 24th, 2013 at 11:30pm
There are no actions to perform.  

mw1197 
YaBB Newbie
*
Offline
Posts: 5


YaBB 2.1
Re: Removing thousands of spam posts
Bill Myers wrote on Mar 24th, 2013 at 4:35am:
mw1197 wrote on Mar 23rd, 2013 at 11:29pm:
So the way I see of moving posts is by viewing the board and checking the select all checkbox at the bottom of the page, and then choosing the board to move the posts to.

Exactly!

******************************************************

mw1197 wrote on Mar 23rd, 2013 at 11:29pm:
That's fine, but the problem is that there's around 15,000 actual threads to move. Is there a way to show more than 20 threads at once in the board view, in the same way that you can view all members at once?

Yes, you can do this, and you'll of course want to adjust this number to best fit your needs.

Go to Admin Center > Forum Settings > and scroll down to "Maximum Topics to display in the Message Index" - that's probably set at 20. For your purposes, I might change that to 500, and if that works, then maybe change that much higher.

Additionally, you may want to look at the following:

Go to Admin Center > Forum Settings > and scroll down to "Maximum number of messages to display on one page of a Topic" - that's probably set at 15. Change that to whatever you want.

******************************************************

mw1197 wrote on Mar 23rd, 2013 at 11:29pm:
Also, a few stickied threads have valuable information in the first post, but then several hundred posts of spam following that. Any way to select all those posts at once to move or delete rather than doing it all one by one?

You can follow the same instructions above, but after you've selected all of the posts, just uncheck the stickied threads, and that should do it.

By the way, if you end up inadvertently moving any posts or threads you didn't mean to move, you can simply go to those posts or threads in the newly created board for that purpose, and move back what you want.

Nice thing to remember: This method of cleaning up the spam posts simply moves posts around versus deleting them. This way, you kind of can't make any mistakes. Or more accurately, any mistakes that are made can be reversed however you want.

However you manage it all, you might want to stick a notice at the top of your forum so that everyone will know what's going on, and for them to bear with you during your spam abatement process.



Thanks! That did the trick. Was able to sequester all the spam into a new board. Not sure how I missed that option.

One thing I've run into now is that some member profiles seem to be corrupted. For example, the most active member shows as an Ex Member. But in the Members/_vti_cnf/ folder, I can find files for him. In the member list it says he registered in 2012... butin the member files it dates back to 2007 (which is accurate as it's when he was posting). Any idea what's up, maybe a way to restore him using those files?

Thanks again for all the help.
 
 
IP Logged  
 Reply #9 - Mar 25th, 2013 at 12:45am
There are no actions to perform.  

Bill Myers 
God Member
Beta Testers
*****
Offline
Posts: 1,482
Los Angeles


YaBB 2.4
Re: Removing thousands of spam posts
I'm glad that this worked for you. Congratulations! Smiley

As for the active member you mentioned who is showing up as an ex-member, with that person in particular, access their profile, then click Modify (use your admin password).

Next, click Admin Edits, and look at Date Registered: to see what date is showing. From what you posted, that person's registration will probably show up as a 2012 registraion, right? If so, change their registration date to the same date or maybe at least one day earlier than the date they first posted in your forum (evidently sometime in 2007). Remember to click Save Changes after you've changed the date, and see if that changes their status. It should.

Note
: If you change the date of that person's registration, you may very well change it to the wrong date. That's because it's not uncommon for people to register, and then read for a while before they ever post, if they ever post. So if that member says anything about what their date of registration should be, you can always change the date again on their behalf.

As for all of the other members who are showing up as Ex Members, hopefully you have a good local backup of those membership files. If so, uploading those backups to the server again will probably fix that problem for all of those files. Or just as I described above, fix them one at a time as members contact you, if they contact you.

For reasons I don't really understand, sometimes files get corrupted, and then they're perfectly fine again after their respective backups have been uploaded to the server to replace the corrupted files. Go figure.

By the way, even if you do have a good local backup of your membership files, before you upload any of them to the server, it'll be a good idea to first download any such files onto your computer as a 2nd local backup just in case any of those files are needed.

******************************************************

Not to confuse your situation, but I've also known this kind of thing to happen when the Forum Start Date has somehow gotten changed to a later date than it should be. You might want to check that in Admin Center > Forum Settings to see what's listed.

Good luck!

 
Morning, noon, or night, have a great one! ...
WWW BillHMyers  
IP Logged  
 Reply #10 - Mar 25th, 2013 at 1:06am
There are no actions to perform.  

Dandello 
Global Moderator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline
Posts: 1,853
Earth


YaBB 2.5
Re: Removing thousands of spam posts
Bill is correct - if the Forum start date is after the posting date, members become ex-members and if the registration date on their profile gets set to after their first post, they can end up listed as 'ex-members'.
 
WWW  
IP Logged  
 Reply #11 - Mar 25th, 2013 at 2:41am
There are no actions to perform.  

mw1197 
YaBB Newbie
*
Offline
Posts: 5


YaBB 2.1
Re: Removing thousands of spam posts
Hmm, maybe the member has actually been deleted after all then, because I have no way of accessing his profile view. No links to it, not appearing in the member list anywhere anymore, etc. Though if I try to reregister an account under that name, it won't let me, saying it already exists. Kind of weird. Also clocked the forum start date back a bit, as it was a few years ahead of the early posters.

Thanks guys.
 
 
IP Logged  
 Reply #12 - Mar 25th, 2013 at 5:52am
There are no actions to perform.  

Dandello 
Global Moderator
YaBB Next Team
Operations Team
Beta Testers
Support Team
*****
Offline
Posts: 1,853
Earth


YaBB 2.5
Re: Removing thousands of spam posts
It's probably safe to delete the files related to that (former) member. If you're not sure, rename all their files, rerun Rebuild memberlist and see if that helps.
« Last Edit: Mar 25th, 2013 at 5:56am by Dandello »  
WWW  
IP Logged  
 Reply #13 - Mar 25th, 2013 at 6:55am
There are no actions to perform.  

Bill Myers 
God Member
Beta Testers
*****
Offline
Posts: 1,482
Los Angeles


YaBB 2.4
Re: Removing thousands of spam posts
I agree ... it can actually be better to rename files instead of deleting them. That way, you can once again have access to the original files if they're needed. Or not.

And/or, particularly with forum files, I pretty much always make local copies of any files I've chosen to delete from the server just in case I make a mistake that needs to be reversed.

In any case, just as Dandello instructed, rebuilding your memberlist will most likley fix the problem of the missing member.

 
Morning, noon, or night, have a great one! ...
WWW BillHMyers  
IP Logged  
Page Index Toggle Pages: 1
Topic Tools
 

Get Yet another Bulletin Board at SourceForge.net. Fast, secure and Free Open Source software downloads Support This Project BoardMod - YaBB features and templates YaBB Codex - support on installation and usage YaBB Toolbar for your browser

YaBB Facebook Group Page

Vulnerability Scanner

Valid RSS Valid XHTML Valid CSS Powered by Perl
YaBB Chat and Support Community » Powered by YaBB 3.0 Beta!
YaBB Forum Software © 2000-2011. All Rights Reserved.