Page Index Toggle Pages: [1] 2 3 ... 15
Topic Tools
Very Hot Topic (More than 25 Replies) YaBB Universal Converter [Beta 4] (Read 146,309 times)
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
YaBB Universal Converter [Beta 4]
Jan 14th, 2006 at 10:52pm
Post Tools
I've recently became interested in writing converters for YaBB again. However, Matt's converter wasn't ready. I talked to him, and he said it was ok for me to take over the Universal Converter project and release my own.

I've mostly followed his spec, only changing a few things (mainly adding to it).

So, I'm going to post the updated spec attached to this post, and the converter in a later post.

Make sure to use this version of the converter!
** Beta 4: Attached to this post.
« Last Edit: Nov 15th, 2007 at 4:22am by AK108 »  

YaBB_Universal_Converter_Beta_4.zip (Attachment deleted)

Do not PM me for support.
Back to top
WWW  
IP Logged
 
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
Re: YaBB Universal Converter [Beta 1]
Reply #1 - Jan 14th, 2006 at 10:59pm
Post Tools
And here's the converter. I have a Y2 output driver and an Eblah input driver. I chose Eblah because it was flatfile and already documented.

Try it out, and tell me what you think. You need to put the Prefs, Messages, Boards, and Members directories from Eblah into the convert directory. Chmod everything 777 or as you did for YaBB.

I'll post how to make your own input driver later.

Oh, and ignore the "NewFile"s. They serve no purpose except to include those directories.
  

YaBB_Universal_Converter_Beta_1.zip (Attachment deleted)

Do not PM me for support.
Back to top
WWW  
IP Logged
 
old goat
YaBB Legends (Inactive)
*
Offline



Posts: 2,488
Location: York, UK
Re: YaBB Universal Converter [Beta 1]
Reply #2 - Jan 16th, 2006 at 4:16pm
Post Tools
Looks good. Can you upload the HOW-TO for the 'from' convertor? I'm nearly there with my Webboard8  one and it would be very nice to see what I need to change to make it work with this, and whether there are any bits that need some more work  Cheesy

NB: not really looked too close (yet) -  how does it handle higher volumes, where setup.pl does reloads to deal with > 4000 members/messages? I've worked out an easy way for database-driven forums to deal with this aspect.
  

Please don't PM me for support - I'll only go and ignore it!
Back to top
AIM  
IP Logged
 
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
Re: YaBB Universal Converter [Beta 1]
Reply #3 - Jan 16th, 2006 at 11:42pm
Post Tools
Ok, I'll answer the easy question first. The problem with having too much data doesn't really happen here. The user of the converter is allowed to set a maximum amount of time for each step to run. If it exceeds that time, the converter jumps out of the conversion loop and dumps anything that needs to be saved to a Perl file. When the user clicks "next", it's loaded into memory again. It doesn't auto-reload, but it should be perfect at saving and resuming, as long as it's within the same time period.

Now, for the howto. Your existing code will need to be separated and transfered into 3 different types of routines for this to work. I'll explain the basic format of the indriver here, as well as explain the Init routine.

This might seem really complicated. It probably is, compared to most of the programming I do. But it's fairly easy to understand if you understand the following points:
- There's 3 main parts of the file: The setup part, which determines what parts of the conversion you can do and what code is executed when; the initial/finish routines, which are generally simple and similar to other routines of the same type; and the conversion routines.
- The save/resume work in progress is very generic. For saving, you just need a list of what variables to store and what they're named. For saving, you just call the resume routine and it does the rest.
- Use localized variables and filehandles all the time. This works wonders to prevent unexpected results of variables that weren't intended to be global.

I recommend taking my Eblah input driver (or if you want a fictional board format, I can provide that input driver) to start with and using the basic framework, especially the Init/Finish routines. If you know what you are doing, you can write a second converter in less than 10 hours, as I did with the Eblah one.

For the input driver, here's the basic form:
Code
Select All
package indriver;

# strict + warnings is good
use strict;
use warnings;

our %CONFIG = (
'BOARD_TYPE', 'the board type it converts (e.g. phpBB 2)',
'NAME', 'name of the converter',
'VERSION', 'Alpha'
);

# What I can convert
our @CONVERT = qw(Boards Categories Messages Members);
our %SUBROUTINES = (
'Boards', 'BoardConvert',
'Categories', 'CatConvert',
'Messages', 'MessageConvert',
'Members', 'MemberConvert'
);
our %INITSUBROUTINES = (
'Boards', 'BoardConvertInit',
'Categories', 'CatConvertInit',
'Messages', 'MessageConvertInit',
'Members', 'MemberConvertInit'
);
our %FINISHSUBROUTINES = (
'Messages', 'MessageConvertFinish'
);
 



Using warnings and strict is optional, but recommended.

Setup

The %CONFIG hash does nothing at the moment, but it's good to have in there for when it might be used.

Now, the @CONVERT array is important. It specifies what items the converter can convert. So far, there's four that are supported by YaBB's output driver: Boards, Categories, Messages, and Members. The converter uses the @CONVERT arrays of the input and output driver to find out what to convert.

Each item in the @CONVERT array is matched up to a subroutine using the %SUBROUTINES, %INITSUBROUTINES, and %FINISHSUBROUTINES hashes. You can call the subroutines that match up anything you want.

We'll start with the easy subroutines, which are the init/finish ones.

The input driver's init routine for each step has a very important step: telling how many steps to run the loop for. It also sets up package globals needed throughout the conversion process, like matching what board goes to what category, or what topics are sticky.

Here's the BoardConvertInit routine from my Eblah indriver
Code
Select All
sub BoardConvertInit {
	my $fh; # Use a localized variable for a filehandle to prevent problems.

	open($fh, "<$convertdir/Boards/bdindex.db") || die "Can't open $convertdir/Boards/bdindex.db: $!";
	@boardlist = <$fh>;
	close($fh);
	$maxsteps = scalar @boardlist;

	open($fh, "<$convertdir/Boards/bdscats.db") || die "Can't open $convertdir/Boards/bdscat.db: $!";
	@catlist = <$fh>;
	close($fh);

	return 1;
} 



The first interesting thing is the $fh variable. It's used exactly the same as a filehandle. I consistently use $fh when I need an anonymous filehandle.

Second, I use "die". Die outputs HTML, but you don't need to worry about the templating area, which is handled in the converter. Same goes for "warn", which is also considered a fatal error.

I open up the board index file and count up the boards present. Since there's one board per line, I capture the whole list in the global @boardlist. I use it in forced scalar context to count up the maxsteps.

Something important is that maxsteps is actually the number of steps, plus one! It works this way due to how I made the for loop. It usually works out perfectly since you often count the lines in the file.

A change from the Beta 1.0 and Beta 1.1 versions is that you should return the number of steps. All you have to do is return it instead of store it in $maxsteps.

Resuming work in progress

If you run out of time, you can resume work in progress like this:

Code
Select All
	if($_[0] eq 'resumeworkinprogress') {&ResumeWorkInProgess();}
	else {
	# The rest of the init code goes here
	}
 



That's it. You just need to make sure you're saving work in the corresponding Finish routine. You'll notice that sometimes I don't save/resume in the input driver. That's due to the fact that I only need to open 1 or 2 files, and there's no real problem with counting it again.



I'm running out of characters in this post, so I'll continue typing in my next post.
« Last Edit: Apr 25th, 2006 at 11:20pm by AK108 »  

Do not PM me for support.
Back to top
WWW  
IP Logged
 
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
Re: YaBB Universal Converter [Beta 1]
Reply #4 - Jan 16th, 2006 at 11:50pm
Post Tools
The actual conversion routine

Here's where the universal part comes into play. The convert routine translates data from the board-specific format to an in-memory version that is passed directly to the output routine. (Whatever you return in this routine is sent as the arguments to the next routine).

For the conversion routine, you get one argument: The current loop step. This is all you really should need, as you've already build the list of items to convert in the Init step.

Let's look at my BoardConvert routine
Code
Select All
sub BoardConvert {
	my($fh, @boardinfo, @catinfo, @test, $cattest, $cat, $item, %board);
	@boardinfo = split(/\//, $boardlist[$_[0]]);

	# Since I don't know of an easy way to find the board, we'll split on all the cats, and find it that way
	foreach $cattest (@catlist) {
		@catinfo = split(/\|/, $cattest);
		@test = split(/\//, $catinfo[3]);
		foreach $item (@test) {
			if($item eq $boardinfo[0]) {
				$cat = $catinfo[1];
				last;
			}
		}
	}

	%board = (
	'cID', $cat,
	'ID', $boardinfo[0],
	'description', $boardinfo[1],
	'moderators', $boardinfo[2],
	'name', $boardinfo[3],
	);

	return %board;
} 



First things first: Declare local variables.

Second, Eblah doesn't have data about the board in board-specific files. So all I need is what was loaded in memory earlier. I have to figure out the category, but other then that, a simple split and assignment to a hash is all that's needed. I return the %board hash, which is sent to the corresponding YaBB routine.

Easy enough, right? Well, it is for the boards, categories, and members. Don't even think of converting messages or PMs yet; they require a multi-dimensional hash. Ugly, but it works fairly easily. Just follow the spec to find out what all you need to get from the input files.



The Finish routine

The finish routine is optional for the input driver. All you'll need to do is send data to the SaveWorkInProgress routine if the argument says to.

I only use a finish routine in the Message section for Eblah.
Code
Select All
sub MessageConvertFinish {
	# We've run out of time, now we need to save our work.
	if($_[0] eq 'saveworkinprogress') {
		&SaveWorkInProgress('threadlist', \@threadlist, 'maxsteps', \$maxsteps, 'stickytopics', \%stickytopics, 'displayname', \%displayname);
		return 1;
	}

} 



It's fairly straightforward, I think. The SaveWorkInProgress subroutine takes an argument list of the name of the variable (without the symbol in front), and a reference to it (\@threadlist for the @threadlist array, for instance). You can send any data format this way, as long as it doesn't include references (so no hashes of hashes). If you need to send references, I can figure it out, but only if you need it.

Cleanup

Last but not least, there's a cleanup entry. If the variable $FINALROUTINE is set, the subroutine named $FINALROUTINE will be executed. The indriver doesn't need this much here, but it's available if you want it. No data is passed, but you can always put a "Thanks for converting" note in there using $HTMLOUTPUT (see below).

Summary of the Variables / Subroutines that have specific meaning

I'll review some of the variables and subroutines here, as well as introduce a few new ones.

Variable name: $maxsteps
Purpose: This is a global set by the Init routine and it tells how many steps there are needed to convert a particular item (boards, categories, etc).

Variable name: $HTMLOUTPUT
Purpose: A global variable. The content of this added to the main output if it is set.

Variable name: @CONVERT
Purpose: A global variable that is compared to the corresponding array in the outdriver. We'll only try to convert items that appear in both, even if one driver can convert more.

Subroutine name: &SaveWorkInProgress
Purpose: It's a global subroutine, defined in the converter.pl file. It takes a list of the variable name (without the symbol) followed by the variable's reference, and stores it to a file.

Subroutine name: &ResumeWorkInProgress
Purpose: A global subroutine defined like the one above. It takes no arguments and automatically figures out what workinprogress file to load.
« Last Edit: Jan 17th, 2006 at 1:49pm by AK108 »  

Do not PM me for support.
Back to top
WWW  
IP Logged
 
old goat
YaBB Legends (Inactive)
*
Offline



Posts: 2,488
Location: York, UK
Re: YaBB Universal Converter [Beta 1]
Reply #5 - Jan 17th, 2006 at 2:01pm
Post Tools
( Shocked oh, that was a scary 12 hours !!!!)

Yes - can see what you're doing now. Eblah looks safely the easiest to pick as the baseline.

Any thoughts on :

Where the
Code
Select All
use DBI ;   


is going to be best placed for the db driven ones?

I've had to add an extra screen in on my code, to allow for confirming that PERL can see and access the input database, since your kind of #stuck# otherwise, and then (this is Webboard for you  Roll Eyes confirm which (board) to do the convert on.

An extra hash for the database connection details perhaps?
Code
Select All
our %DBASE= (
'DBASE_TYPE', 'mysql/oracle/sql server/access',
'DB_NAME', 'name of the database',
'SERVER_NAME', 'name of the database server (probably localhost, but you never know),
'LOGIN', 'sa login id',
'SAPWD', 'sa password'
'VERSION', 'Alpha'
); 


  

Please don't PM me for support - I'll only go and ignore it!
Back to top
AIM  
IP Logged
 
old goat
YaBB Legends (Inactive)
*
Offline



Posts: 2,488
Location: York, UK
Re: YaBB Universal Converter [Beta 1]
Reply #6 - Jan 17th, 2006 at 2:25pm
Post Tools
How about having a sub to take in any 'input' date format, convert it to a standard that can then be processed by the 'output' to whatever format needed?
Or would that be more work than directly converting ?
  

Please don't PM me for support - I'll only go and ignore it!
Back to top
AIM  
IP Logged
 
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
Re: YaBB Universal Converter [Beta 1]
Reply #7 - Jan 18th, 2006 at 12:05am
Post Tools
I'll see if I can add options to the screen right before it converts, where you input the maximum time and see what can be converted. Expect this in a week or so -- school's being evil to me now).

The "use DBI;" should be placed right by the "use warnings", I think. It'll error out before any conversion is attempted this way.

As for the generic time conversion routine, there's a slight problem. The converter doesn't know what type of time to expect. You can do like I did with the Eblah one if you have to, which is adding a string-to-time routine. I'd perfer the input and output drivers to be fairly self-contained. You'll need to best figure out a way to convert into the Unix time() format.

I'll write up how to handle the Message Conversion step later.
  

Do not PM me for support.
Back to top
WWW  
IP Logged
 
old goat
YaBB Legends (Inactive)
*
Offline



Posts: 2,488
Location: York, UK
Re: YaBB Universal Converter [Beta 1]
Reply #8 - Jan 18th, 2006 at 9:47am
Post Tools

Quote:
The "use DBI;" should be placed right by the "use warnings", I think. It'll error out before any conversion is attempted this way.

that's me misunderstanding the way 'use' works! I'd assumed it would add load to the code  - like 'require'.

Quote:
As for the generic time conversion routine, there's a slight problem. The converter doesn't know what type of time to expect. You can do like I did with the Eblah one if you have to, which is adding a string-to-time routine.

sorry - that was badly described. What I meant was, I've written a sub:

Code
Select All
sub convert_time_toyabb	{
	($conv_time_string,$conv_from)  = @_;#[0];
	#$conv_from = $_[1];
	$yytrace .=qq~<br />convert_time_toyabb ($conv_time_string)($conv_from)~;
	my ($convmonth,$convday,$convyear, $convtime, $convhour, $convmn, $convmin, $convsec, $convdate, $convap,$converted_time_string);
	if($conv_from eq "wb")	{
		#2002-05-08 15:55:00.000
		($convdate, $convtime) = split(/\s+/,$conv_time_string);
		$yytrace .= qq~conv_time_string --> ($convdate)($convtime)~;
		($convyear, $convmonth, $convday) = split(/\-/,$convdate);
		$yytrace .= qq~convdate-->($convyear)($convmonth)($convday)~;
		#($convmonth, $convday, $convyear, $convtime) = split(/\s+/,$conv_time_string);
		#my $convmnum = $convm->{$convmonth};
		($convhour, $convmin, $convsec) = split(/\:/,$convtime);
		($convsec, $convdummy) = split(/\./,$convsec);
		$yytrace .= qq~ convtime-->($convhour)($convmin)($convsec)~;
		#$convmin = substr($convmn,0,2);
		#$convap = substr($convmn,2,2);
		#if($convhour < 12) {$convhour = $convhour+12;}
		#$yytrace .= qq~ lastpost($convhour)($convmin)($convsec)~;
		$converted_time_string = qq~$convmonth\/$convday\/$convyear at $convhour\:$convmin\:$convsec~;
		$yytrace .= qq~ converted_time_string($converted_time_string)~;
	}
	$yytrace .= qq~ converted_time_string($converted_time_string)~;
	return $converted_time_string;
} 


(ignore the $yytrace - that's my debugging string!)
which takes the time from the input , in whatever format its done - sql datetime in the above case - with the $conv_from variable switching the exact algorithm in. This returns the date/time in 'yabb compliant' form here, but could just as easily switch it to a standard that can be fed to a second sub to switch it out to the output format. Similar to the format_timestring sub in setup.pl.


  

Please don't PM me for support - I'll only go and ignore it!
Back to top
AIM  
IP Logged
 
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
Re: YaBB Universal Converter [Beta 1]
Reply #9 - Jan 18th, 2006 at 1:59pm
Post Tools
Hm. I guess something like that might be worth having. I'll think about it for Beta 2.

Looks like you're doing good so far. I'll see about a Beta 2 with the changes I mentioned soon.
« Last Edit: Jan 18th, 2006 at 1:59pm by AK108 »  

Do not PM me for support.
Back to top
WWW  
IP Logged
 
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
Re: YaBB Universal Converter [Beta 1]
Reply #10 - Jan 23rd, 2006 at 2:05am
Post Tools
s_m_b: Can you post your version here? I want to see what you have in mind.
  

Do not PM me for support.
Back to top
WWW  
IP Logged
 
old goat
YaBB Legends (Inactive)
*
Offline



Posts: 2,488
Location: York, UK
Re: YaBB Universal Converter [Beta 1]
Reply #11 - Jan 23rd, 2006 at 6:46am
Post Tools
Sure.
PLease bear in mind the version I have is currently wrapped around a variation on setup.pl, so the forum-specific parts are not as yet separated.

This one is complete up to 'members', and I'm working through the business of mapping the 'messages' over.
  

converter.zip (Attachment deleted)

Please don't PM me for support - I'll only go and ignore it!
Back to top
AIM  
IP Logged
 
AK108
YaBB Legends (Inactive)
*
Offline



Posts: 3,321
Re: YaBB Universal Converter [Beta 1]
Reply #12 - Jan 23rd, 2006 at 1:59pm
Post Tools
I'll take a look later.
  

Do not PM me for support.
Back to top
WWW  
IP Logged
 
Joris T
YaBB Newcomer
*
Offline



Posts: 24
Location: Amsterdam
Re: YaBB Universal Converter [Beta 1]
Reply #13 - Jan 25th, 2006 at 7:26pm
Post Tools
I am trying out the converter, but can't get step 4 to work.

The following message is displayed:
Error! Use of uninitialized value in split at <root>/cgi_bin/converter/Output/YaBB2.pl line 236.

Unfortunately I am not experienced in Perl scripting.
It has something to do with the Prefs/Ranks2.txt file, but I can't figure out the relationship between that file and the user data files.

Any pointers?

By the way, YaBB works great!
  
Back to top
WWW  
IP Logged
 
old goat
YaBB Legends (Inactive)
*
Offline



Posts: 2,488
Location: York, UK
Re: YaBB Universal Converter [Beta 1]
Reply #14 - Jan 25th, 2006 at 9:50pm
Post Tools
hoping ak108 doesn't mind me replying on this...  Wink

if you're looking at the same file I am, that error is coming from the split function being fed a null variable from the hash (member) that's created from the data sent to the MemberConvert sub.

Can't see where its being called from but there must be some missing 'groups' data there.
Presumably the ranks2.txt has something of that nature in it?
  

Please don't PM me for support - I'll only go and ignore it!
Back to top
AIM  
IP Logged
 
Page Index Toggle Pages: [1] 2 3 ... 15
Topic Tools