mySQL Database Search & Replace With Serialized PHP [Updated]

Ever needed to migrate a database to a new server or website (especially with WordPress and other PHP applications) and been stuck because when you do a search and replace some of the data seems to get corrupted?

Please note that a newer version of this code is now available from my Interconnect’s site over at https://interconnectit.com/search-and-replace-for-wordpress-databases/ 

Ever needed to migrate a database to a new server or website (especially with WordPress and other PHP applications) and been stuck because when you do a search and replace some of the data seems to get corrupted?

Serialized PHP Arrays Cause Problems

In PHP one of the easiest ways of storing an array in a database is to use the serialize function.  Works a treat, but the downside is that you’re not storing data with a cross platform method.  In many product development environments this would get you a stern talking to, but in the world of web development where deadlines are tight and betas are the norm, this seems to be overlooked somewhat.

So what we have are tables full of data that can’t be easily edited by hand.  For example:

;a:3:{s:5:"title";s:17:"This Week\'s Poll";s:18:"poll_multiplepolls";s:0:"";s:14:"multiple_polls";N;}

Say you had thousands of records like the one above, and the word ‘multiple’ needs to be changed to ‘happy’.  Two bits would change – poll_multiplepolls would now read poll_happypolls and multiple_polls would read happy_polls.  In both cases you would have three characters fewer to deal with.

Fine, you may think, but you can only do the change by hand because where it says s:18:"poll_multiplepolls" it now has to say s:15:"poll_happypolls" – see the difference?  S18 spells out the length of the following string, and it has to be changed to s:15

I’ll say right now, that that was a pain.  For simple arrays I wrote the straightforward PHP Serialization fixer code, which got me out of many a pickle – do the search and replace without worrying, and then run the script.  Fixed about 90% of problems.

Multidimensional Array Problem

Sadly those 10% of problems left were a real pain.  I needed something more robust.  Something more powerful.  And finally today it was a Bank Holiday in the UK – that means no phone calls… I could have a quiet day of coding and concentrate on the best solution to this problem.

What I’ve done is to write a database search and replace utility in PHP that scans through an entire database (so use with care!) which is designed for developers to use on database migrations.  It’s definitely not what you’d call an end-user tool, though I may sanitize it at some point and turn it into an easy to use WordPress plugin.  Thing is – this is dangerous code – sometimes I think it’s better to make it deliberately a bit tricky, don’t you?

It’s not that bad though – if you can manually install WordPress, you can easily configure the database connection settings.

What the code does is to look at the database, analyse the tables, columns and keys, and then starts reading through it.  It will attempt to unserialize any data it finds, and if it succeeds it will modify that data then reserialize it and pop it back in the database where it found it.  If it finds unserialized data it will still carry out the search and replace.

Use in WordPress

In most WordPress migrations you tend to have the primary problem of changing the domain name entries in content, settings and widgets – you simply need to put in the $search_for string the old domain address (including the http if it’s there) as seen on the database, and the new one into $replace_with.  Then put this script onto your server, and run it by visiting it in your browser or inputting the appropriate command line – depending on your server configuration.

Other things you may want to check are for plugins or themes that have made the mistake of storing the full server path to the installation – cFormsII does this, for example.  You will need to find out your old and new server paths and use those, in full, for another iteration of this script.

After less than a second of running, you should have a freshly edited database.  It may take a little longer on slow or share hosting, or if you have a very large database, but on my laptop I can manage around 60,000 items of data per second.

I’ve just used the script to migrate, in its entirety, with content, settings, 87 widgets (yes, really!) and hundreds of images to my localhost server.  It took moments, and the site is perfectly preserved.

Search and Replace Database download.
download file

Search and Replace Database download

BIG WARNING: I take no responsibility for what this code does to your data. Use it at your own risk. Test it. Be careful. OK? Here in the North we might describe the code as being as “Rough as a badger’s arse.” Never felt a badger’s arse, but I’ll take their word for it.

Handy Tip for CSS Reloading

CSS not refreshing in the browser can be a pain – Mark Jaquith has come up with an elegant solution to this for WordPress coders.

I saw this rather handy tip on CSS reloading on Mark Jaquith’s blog today.  Such a simple approach, but a great one for when you’re making CSS changes and need browsers to pick up on it immediately, rather than some hours down the line.

Some changes are critical.  Now you can version your css by hand, but you don’t have to.

The basis of the change is to use the code:

<link rel="stylesheet" href="<?php bloginfo('template_url'); ?>/style.css?v=<?php echo filemtime(TEMPLATEPATH . '/style.css'); ?>" type="text/css" media="screen, projection" />

But for the detail, go read Mark’s post.

PHP Serialization Fix for WordPress Migrations (& other applications like Expression Engine)

Serialization of data loaded into an SQL table is a dreadful thing and makes WordPress migrations harder than they should be, but it happens and so we must deal with it. I’ve knocked up a rough and ready bit of code which does its best to resolve the problem.

When you move a WordPress blog from one folder to another, or from one site to another, you normally use the export/import functionality.

This is fine for normal blogs, but say you’ve developed a new website and set it up on your local machine – the URL for the site may be something like http://localhost/devsite and the live URL will be something like https://davidcoveney.com – you won’t want to set up all the theme options, site options, plugin options and so on all over again.

A different kind of migration - public domain from Wikipedia Commons
A different kind of migration – public domain from Wikipedia Commons

Instead, a theoretically simple approach is to do a database dump, a search and replace for all references to server paths and URLs, and then reimport that data in the new location.

Should work, but it often falls apart.

What happens is that in WordPress, its themes and its plugins, a lot of data is stored using a method known as serialization.  Now, in my opinion this breaks all known good practice around data – it’s language specific, it’s not relational even though it often could be, and it’s hard to edit by hand.

One particular problem is that if you change the length of the data in a serialised string you have to change the length declared in the generated string.

That’s very painful when you have hundreds of the fields.

So, because I’d found this painful I decided to knock together a quick application to at least reduce the amount of editing I had to do.  You just do your search and replace, forget about the serialized string lengths, upload your data to the new database, and run this script.

Warning: I haven’t got it to work for widgets and cForms II yet, but the latter has some export functionality anyway, which takes that particular pain away if you plan ahead.  In the meantime, feel free to play with the attached file.  You use it at your own risk, of course.

To use it, download the file linked in this post, extract it, open the file, edit the connection settings, tell it the table you want to scan through, the column, and the unique key field.  If you somehow manage to have more than one unique key to deal with (you shouldn’t, but then it surprises me what people manage to code up), then you’ll have to modify the code accordingly.  Once done, make sure you have a backup of that table, and execute the php – either at the command line or through the browser. License is WTFPL, and if you’d like to improve the code, please do and I’ll host the new version.

Serialization-fixer.zip download.
download file

Serialization-fixer.zip download

BIG WARNING: I take no responsibility for what this code does to your data. Use it at your own risk. Test it. Be careful. OK? Here in the North we might describe the code as being as “Rough as a badger’s arse.” Never felt a badger’s arse, but I’ll take their word for it.

Wordcamp UK 2009 to be in Cardiff

20090224-mp54k2h4uu7ada1jtkycce81qiIf you’re following the various WordCamp lists, you’ll already know about this event.  But many won’t.

If you use WordPress professionally, or with a great deal of enthusiasm, WordCamps are a great way to meet with other users, developers and designers who really understand the system.  There are useful presentations, social events and activities based around the event.

I was there last year at the Birmingham WordCamp, with James, and our company Interconnect IT was one of the sponsors.  This year we’re waiting to see how finances work out before throwing in sponsorship money, but I’ll definitely be there again and I’m likely to be presenting on the issues surrounding bringing WordPress to the enterprise space.  Because corporates love WordPress too…

This year’s event will take place on the 18th to the 19th of July at the Future Inn Cardiff Bay, Cardiff, Wales.

For more information, you can visit the official WordCamp UK Site.

Barcamp Liverpool 2008

I attended Barcamp Liverpool 2008 to join fellow geeks in a spot of technology appreciation at this ‘unconference’ at the CUC. Great venue, and great to be able to give an ad-hoc presentation on WordPress for News Sites in the Café

So… I’ve always meant to be a bit more active within the geek community.  I tried a spell in the late eighties/early nineties with the British Computer Society, CompuServe and CIX but sometimes found it all a bit tiresome.  There were too many who’d earned their stripes in the seventies on heavy iron and, I felt, even at that time were being left behind.  Of course, I must say that that doesn’t apply to all or even a majority of the people back then.  But there wasn’t much of a sense of fun.  It was all a bit… serious.

Barcamp Liverpool 2008 Official Logo
Barcamp Liverpool 2008 Official Logo

But you know, technology is cool.  Especially today when almost everyone seems to be a geek these days.  So after a successful trip to Birmingham for the first WordCamp UK I thought it was time to get geeking up in my home town!  Barcamp Liverpool beckoned…

And it was worthwhile.  Sadly I’d been up at 3am in the morning in order to take Romana to the airport, so I was a bit out of it.  But that didn’t prevent me doing a few things anyway…

WordPress for News Sites

I was asked if I wanted to do a presentation on WordPress by one guy.  And I thought… why the heck not.  Now, I know a lot of people at Barcamp know WordPress, so I felt the best approach was to be a little more specific.  So… I decided to talk casually in the Café about how WordPress can be used for the purpose of building a news site.  I covered the basics before showing off the Caribou demo.  To be honest, it wasn’t a perfect talk – I’d had half an hour to prepare and most of that was spent getting the latest WP Trunk installed on my laptop, with everything configured and ready to roll, plus a quick list of key points.  If I hadn’t had a client meeting for half the morning I might have done better.  But still, at one point there seemed to be about 20 people paying good attention.  Frankly I was surprised – I rambled and soon realised that the best approach was to get folk asking questions.  I think people enjoyed it!

Video Interviews

I ended up giving a couple.  I rambled.  I was tired.  Lack of sleep doesn’t help me.  I kept my glasses on so as to disguise the bags under my eyes.

1 Minute Pitch

Instead of pitching something we’re doing I decided to pitch a new concept we’ve been floating around the office.  I was nervous, shakey and tired.  I can’t even remember much about it all to be honest.  The other guys did better.  They had stuff like preparation – I’d had about thirty seconds to think about mine!  But it’s good to practice public speaking, so why not?  It was a good crowd.

But Hey…

I had a good time.  But overslept dramatically as I recovered from Saturday, so didn’t make the Sunday.  I hope everyone had a great time!

So now I’m on Facebook I Want a Blog Connection…

It makes sense, if you think about it – why duplicate data in two places? So in went the Wordbook plugin for WordPress, and then the Facebook application onto my profile.

This post, really, is to see if it’s a success – does my blog here appear in my profile? And so it does! Woo! Another little click in the Web 2.0 thingy.

Now what I want to know is, does your feed show my posts come up? If so, then this all works very nicely indeed. I have ideas….

Badminton Near Warrington

I’ve been telling anyone who’ll listen that having a web presence is free and simple, and doesn’t need to be opaque to the search engines. It can be standards compliant, easy to find, and well structured. You can change the content as and when you like. You can add pictures and so on. And it’s all free.

A recent example of this is one of my preferred Badminton Clubs’ website – my company builds websites professionally, so we’re not in the habit of giving them away. But these are custom jobs, for firms that need reliable websites which often generate far more money than they cost. But a small club doesn’t need the fine control of a commercial organisation. So I pointed Halton Badminton‘s chief chap Bob Redmond, to wordpress.com and we spent a couple of hours together as I showed him the ropes.

And now the site’s there and the search engines will soon pick it up. Hopefully it’ll soon attract the traffic it deserves – these are great clubs for anyone interested in badminton around Widnes, Runcorn or St Helens – and a fair few folk from Warrington turn up as well. For more details head to http://haltonbadminton.wordpress.com

Resolving WordPress Migration Issue

I’m posting this for the benefit of anyone who experiences a similar problem to mine when moving WordPress from a directory into the web’s root directory.

What happened was that many links and pictures, some placed in there by plugins, others simply links that were typed in, had failed.

I realised that although I’d followed the migration instructions, a lot of things were left poorly sorted. A quick run through of all the tables showed where values were left incorrectly set. To fix this I wrote the following SQL statements and applied them to my database:

update wp_posts set guid = replace(guid,”/wordpress”,””);

update wp_postmeta set meta_value = replace(meta_value,”/wordpress”,””);

update wp_options set option_value = replace(option_value,”/wordpress”,””);

update wp_posts set post_content = replace(post_content,”/wordpress”,””);

These worked a treat.

To use this yourself I’ve done a version below that you can edit – simply replace $$$olddir$$$ with your old directory name (in my case wordpress) and replace $$$newdir$$$ with your new directory name. If your new directory is the root, remove the / at the beginning of each $$$newdir$$$ – see my example above.

A couple of warnings though – first take a backup of your database! Then think carefully – if you don’t know what you’re doing you may well be advised in seeking someone out who does.
update wp_posts set guid = replace(guid,”/$$$olddir$$$”,”/$$$newdir$$$”);

update wp_postmeta set meta_value = replace(meta_value,”/$$$olddir$$$”,”$$$newdir$$$”);

update wp_options set option_value = replace(option_value,”/$$$olddir$$$”,”$$$newdir$$$”);

update wp_posts set post_content = replace(post_content,”/$$$olddir$$$”,”$$$newdir$$$”);

Good luck and have fun!