In the beginning...

Many, many years ago (2002), the Codex was created using the phpBB (and then later, phpBB2) open source forum software. For many eons, this platform served us well. It was free and there were lots of mods we could add to it that let us do things. Calis pretty much ran things back then until I (DarkUnderlord) came back onboard when Calis didn't really have the time anymore.

The forums were pretty much a stand-alone package with Calis' code on the front-end simply inserting stuff into certain database tables when we made news and content items. However Calis, like me, was a self-taught PHP hack and over time, both of us got better skills (eventually I built was is currently running on Duck and Cover).

So for a long, long time we spoke about "upgrading" the Codex. There were a bunch of features that we wanted, like a better image gallery and things but sadly, we never quite got around to it. Along the way, I levelled up my mad PHP skillz a bit more and started hacking my own mods into the forums. We added an ignore feature, little games every April and other stuff which I forget... and yay, for times were good.

But then phpBB3 came out and upgrades for phpBB2 stopped. On one hand, that was pretty sweet because it meant I no longer had to do the usual "upgrade phpBB2 with new security update; then re-install all the god-damned mods; 8 hours later..." routine. But on the other, things started to get slow.

The Codex has been increasing in popularity (read: spammers registering and clogging up our forums) for a pretty solid period. At one time we were doubling our visitors every 6 months. On top of this, we also had 8 years of content (read: crap) we were carrying around and if there's something you need to know about search functions, it's that more crap == longer search times == heavier load on the server == bad things happen.

There are two options here. One, upgrade the server or two, disable search. We were pretty much at the point where we'd have to do some kind of cloud computing setup thing (one server just isn't enough any more) so we took the cheap option and disabled search. And for a while, we could handle the increase in people just fine.

Eventually though (particularly through-out 2011), things became noticeably slow and various other things got turned off or tweaked. We really, really had to upgrade the server. But if we were going to do that, we may as well upgrade the forum tech first and see if that helped.


Why not phpBB3??

Forum upgrade options obviously fell into two categories: Free or paid. On the free side we had the logical choice of phpBB2 -> phpBB3. However, there was a slight problem. One of the major pains in the ass for a long, long time, was that every-time we upgraded the old phpBB2, we had to re-install the quick reply mod. It wasn't an automated process (like most things are these days) and it was a real fucking pain in the ass that meant I had to spend a day re-doing it. Here's the clincher, phpBB3 did not have a quick reply feature available. In fact, the developers insisted it was not a "design feature" for deliberate reasons. They "didn't believe" in quick reply and insisted that clicking to open a new page was much, much better.

I disagreed. I also didn't have any interest in re-installing a quick reply mod every-time we had to upgrade (and hope the mod worked with the new version or wait for the mod to be updated for the new version or deal with any of the other little things that entails, all of which lead to problems solved by spending hours and hours poking through code or waiting for stuff).

There's also a point where if you don't agree with the philosophy behind a piece of software, then there's probably going to be something else you'll eventually find that annoys you too. So phpBB3 was off the table (Fun Fact™: After several years, phpBB3 finally comes with a default quick reply feature).

Other free options were SMF (Simple Machines Forum). SMF was at version 1.0 though and had for a long time, spoken of releasing version 2.0. Again, free software has a habit of taking a long time for stuff to get done (because you're relying on volunteers) and when it comes out, it works but it's just not "flash" for the most part (think GUI vs dealing with a command line interface). Either way, SMF2 was still on the drawing table so there was no point changing to SMF1 (old-tech) only to upgrade to SMF2 later.

We also had a bit of cash in the kitty thanks to a successful con-job of getting people to pay for adding anime avatars to the old phpBB2 gallery (suckers).


Why XenForo?

So we looked at the paid alternatives, with vBulletin being the most prominent. However, there's a long and sordid tale you need to be aware of. I also had the advantage of Taluntain's experience who uses vBulletin over at sorcerers.net... So we looked at XenForo instead. It was going to be the Latest Shit™, coded in the Whizz-Bang Object-Oriented™ way of The Future™. In was going to be Fully Pro (fully!) and had the upside that we would be able to put off any further upgrade for as long as humanely possible. Maybe decades (here's hoping)!!

The devs seemed to have their shit together (always a good thing) and there was decent support (though not much in the way of help docs at the time we were looking). Taluntain and I tried the demo out and it was pretty sweet. There was Facebook integration (which you could disable), twitter (which you could disable), likes (which you could disable), trophies (which you could disable) and a whole bunch of other fancy crap going on (all of which you could disable).

On the downside, there was some really, really, stupid shit as well. For starters there was no avatar gallery. Not a huge deal but given the amount of retards we get here, it's better to take toys away from them because you know someone's going to abused it eventually. Plus it's part of the whole theme thing we had going, forcing you to choose avatars from RPGs (and some not-so-RPG games).

There's also editing thread titles. Regular users just can't edit their own. It's not an option (short of giving them access to edit EVERYONE's thread titles within that forum). This is a pretty major issue for us given the number of Let's Plays we have, where the creator will often update the title with "Updated (DATE)" when they update.

And then there's editing actual posts themselves. XenForo works on a time limit. You can edit the post as much as you like within the time limit, only there's no little "this post has been edited" text (the kind that used to appear underneath posts in phpBB2). It's just an issue when someone edits their post to say something they didn't say earlier - and there's no indication they've changed anything at all.

Originally we thought this was ok, as we'd just set the time limit to 30 mins and that's it. But the problem again is the number of index threads we have where thread creators are constantly updating the first post - often months or years later. We have things like Minecraft Server details; The Big Indie RPG List; The RPG Codex Let's Play Database; not to mention all of the Let's Play threads themselves where the creator will often update the first post with an index linking to all the updates within that thread. So the time limit got removed for users who have been here for about a year.

Finally, there's the rich text editor. Personally, I prefer to turn it off but there are times when I'd like it on. Unfortunately, while there's a nifty "turn off / on" option when you have it enabled, there's no such option when you have it disabled by default. Sometimes I'd just like to be able to turn it on for the odd post or two, without having to dig into my settings, turn it on, refresh and then turn it off again.

On the upside to all of this is that XenForo is new, so we'll either get easy to install mods / plugins that will eventually add this functionality or XF will add that functionality in a future update themselves. Just so long as the forum worked for now, we'd be fine...


The Upgrade Fun Begins

The only downside was that the ENTIRE code for the Codex front-end had to be re-written. Calis' code was... less than perfect (but as a first attempt worked just fine) which meant changing things wasn't easy. We also knew some things now which meant we could build stuff a bit better and with more features. Like for example, you could only have one game associated with a news item (the old "More Info" links) and yet sometimes that item might be about two or even three games. I handled all of this in the code I made for Duck and Cover but that too was old code and in fact, heavily relied on phpBB2's templating engine. Meaning that had to be completely re-written as well (the code that runs Duck and Cover is also the code that ran our sister site, Tacticular Cancer). What was to become the Codex' 3rd attempt at a re-write began...

On the plus-side, we had some time. XF still didn't have any phpBB2 -> XF converters and until that happened, we weren't going anywhere. So my original idea was simple, all news would be posted in the news forum. I'd code in some little add-ons for staff that would give them access to insert tags (hook items up with games, people and so on) and turn a news item into a content piece if necessary. Only there was one teensy, tiny, itty, bitty problem. XenForo is the New Shit™ (object-oriented) and I code Old Shit™ (procedurally).

After some months of trying, I still don't understand object-oriented syntax. There's just shit there I stare at and have no idea what I'm looking at. And because I don't know what it's called, it's really hard to google. Plus XF calls shit in the Latest and Greatest™ ways.

I did make some progress and was confident enough to think that with another week or two of code, I'd get there and all would be well. About that time, XF 1.1 came out and it had phpBB3 -> XF converters. We could finally upgrade, only we had to go via way of phpBB2 -> phpBB3 -> XF.

So, with backup in hand, I started upgrading on my localhost.


Upgrading to phpBB3

This is the difference between pro (XF) and amateur (phpBB3). The upgrade from phpBB2 -> phpBB3 took three days. That is literally three days of leaving a computer on while it runs the conversion process. Oh yeah, and it would crash. Repeatedly. For reasons it didn't want to tell me... And we had about 2GB of database and almost 2 million posts to process.

The upgrade process literally involved me firing it up, walking away, coming back 20 minutes later, finding out it had crashed, going into phpMyAdmin to find the last post it converted, find the next post in the database where it crashed, fix whatever problem there was with that post, then continuing. Given our forum population, problems included massive quote pyramids (which crashed the bbcode converter), long strings of text (which timed out the bbcode converter) and large amounts of smilies (bbcode converter again).

Eventually though, it would get through it and finish. I would then run the converter from phpBB3 -> XF. This took a mere four hours and relied on me clicking "convert", before walking away and leaving it. I like to think it's because the conversion from phpBB2 -> 3 did most of the work but when XF did crash, it crashed with nice error reports saying these are the problems and it gave me options to fix them (IE: Duplicate usernames because it scrubbed all those invisible characters people registered with at one time). That's fucking pro right there.

This process ran on my localhost (running xampp), with a downloaded copy of the database. When it was done, it was all uploaded straight into the fresh install of XF sitting on the server. It was literally 5 mins before the first people noticed the forums were back and within an hour we had 200 people online. The front page still wasn't even up though.

It also turns out we had a rather major issue. The XF converters were designed to carry over people's passwords from phpBB3 (which in turn, was designed to carry paswords over from phpBB2) so in theory, no-one should have had to reset their password. Unfortunately, something, somewhere got fucked up (I have no idea what) and it seems every single user (apart from me) had to reset their password. Given a large number of people have bogus e-mails in their profile, this meant over 400 requests along the lines of "DU can you reset my password?". Needless to say, that wasn't much fun at all. At least XF make it pretty easy with the user search button sitting right in the main panel of the admin page (Though mind you, being able to go straight to editing a user directly from their profile would be mighty handy).


Styling XenForo

At that point, the priority became making the forums look like the old Codex style. I did have a working head-start on my localhost... which I accidentally wiped when I did the real conversion (derp) so we started again from scratch. I still had Old Codex™ (untouched) sitting on my localhost for reference so grabbing the colours was easy. The hard part was working out which colour went the fuck where.

See, XF have this genius styling tool at the back-end where you can easily change colours. The problem is, they (in my opinion) badly misuse colours. All the colours are named very descriptive things like "Primary Light", "Primary Lightish" and "Primary Lighter". There's obviously two problems with this. The first, our colours weren't "light", so "light" would become "dark" and the "lighter" would actually end up being "darker" or - depending on quite where "Primary Light" was used - a different colour altogether. The second is that the colours are used all over the shop in all sorts of places. For example, "Primary Light" might be a font colour in one area, the background for a page in another and the border of something else.

This obviously makes it hard to pin down where in the fuck that colour you're trying to change is set - only to then change it and find that something else is now fucked up because it's changed colour too, when you didn't want it to. Thankfully XF has a pretty powerful back-end feature to handle this - you simply edit the raw.css files yourself and set the specific colours you want for specific things right in there.

The downside, is that this all happens in the database. You're basically given a textarea within which to style your CSS / HTML as necessary. It means there's no IDE with nice syntax highlighting and all the other features of a modern IDE. It is a shitty little box that you have to copy-paste tabs into otherwise you end up with a wall of left-aligned text that makes it even harder to read.

Then there's the ultimate sin. With all the fancy colour choices everywhere and funny little XF specific codes you can use, they go ahead and make the background of a form field black. As in "background-color: white;" right in the CSS (actually I think it was a "background-color: black;" but I can't remember now). So while you go ahead and change the text colour to that nice white you wanted, you've now broken a bunch of forms by making them completely unreadable for anyone who wants to use them. You now get to hunt-down every form and fix it.

That, along with fighting CSS (which I just won't go into) meant that lovely two week window I had to fix the front page went bye-bye, as I instead spent it fighting CSS and trying to understand the quirkier elements of XF's styling. Plus there were all those stupid-ass gradient images they put behind everything which we had to find and kill with fire.


The Time Arrives

... and so here we are. I finally stopped playing Minecraft for long enough got another window to finish off the coding. To cut a longer story short, I abandoned my attempts at making a whizz-bang object-oriented, integrated plug-in and went for stand-alone procedural code for our front-end (officially making this re-write Codex 4.0). On the downside it may mean some future upgrade will involve yet another complete re-write. On the plus-side, if we ever change forums, we might have saved ourselves a lot of work because in theory, I don't rely on the forums at all and I can just change a few variables for those places where I need to know the forum structure.

So far though, XF has been handling itself superbly. Taluntain has tweaked a few of the advanced features so that we maximise our speed and the forum has gone from crawling like a slug with over 300 people online (phpBB2 + mods) to running as smooth as silk with over 400 online (XF). We'll now see if my front page code is up to the task and we'll see if that slows things down at all.

If it does, we'll be asking you for more money and upgrading the server this time. Hopefully we now have a bit of room to handle our continued growth though, without things falling apart and over-loading the server.