Category: Technical

Technical issues around social media. For the geekier-minded.

Blogsome: a WordPress Weblog Farm [en]

A hosted WordPress weblog provider just opened. I’m testing.

[fr] Blogsome est une solution hébergée fournissant à  qui le désire un weblog WordPress en deux clics (moins d'une minute).

Check out Blogsome. It looks exciting. I’ve set up a test blog: enter a username, an e-mail address, your blog title, and there you are. 1 minute.

I’m very curious to see how this will scale, especially given my experiences with my school blogs. I’m also curious to see if they will release the code used.

If you’re trying it, please let me know what you think!

[via Ollie]

Chosen Spam Tidbits [en]

Some fun things spammers tried to leave in my comments.

[fr] Quelques commentaires, drôles à  mon goût, que les spammeurs ont tenté de laisser sur ce blog.

Now that Spam Karma has been efficiently killing nearly 600 spams during the last 48 hours, here are a few chosen tidbits from the deletion log.

Your website is beautifully decorated and easily navigated. I have enjoyed visiting this site today and hope to visit many more times in the future.
Easily navigated! Wow. Now that exceeds my expectations. I’m not certain I want you to visit many more times in future, though.
Thank you for opening a wonderfully new sight..I wish you the best of luck with your new venture.
Er… thanks to you!
Good morning, I am new to this site. I have just learned about this site. I am going to read on and it’s very interesting to know
Yes, I’m very glad to know that!
I put my step in to your guestbook like I always.
Hope you didn’t break anything there.
I found your website after I have been surfing the internet to be useful.
That sounds like a great way to make oneself useful.
Just stopped by to visit and got the crunch on your stuff in here – bravo!
Did it taste nice?

Which one do you prefer? If you got fun spam lines too, post them in the comments.

Quick Comment Spam Tip for WordPress [en]

Here are the two plugins you need to fight comment spam using WordPress.

[fr] Deux plugins pour combattre le spam de commentaires sous WordPress: Kitten's Spaminator et Kitten's Spam Words. [Edit 03.12.2004] Ou plus simple, Spam Karma, que j'utilise à  présent.

If you want a quick tip to protect your WordPress installation from comment spam, the killer plugin combination is Kitten’s Spaminator and Kitten’s Spam Words.

The second allows you to “delete as spam”, adding keywords and IPs to the filter, and the second gives “spam points” to new comments (ie, for words matching the filter, previous comment posted within the last x minutes, IP matching the filter), deleting those over a certain threshold directly, and queuing the others for moderation. It’s also smart enough to give bonus points if the commentor has already had comments posted on the blog.

Edit 04.12.2004: See the comment, I’m now using Spam Karma rather than the two plugins presented above.

Scripts for a WordPress Weblog Farm [en]

A first step to WordPress-farming: a shell script and a PHP script which allow you to easily install a whole lot of WordPress weblogs in only a few minutes (I installed over 30 in less than 5 minutes). Scripts require adapting to your environment, of course.

Update 03.11.06: Batiste made me realise I should point the many people landing here in the search of multi-user WordPress to WordPress MU. All that I describe in this post is very pretty, but nowadays completely obsolete.

Here is the best solution I’ve managed to come up with in half a day to finally install over 30 WordPress weblogs in under 5 minutes (once the preparation work was done).

A shell script copies the image of a WordPress install to multiple directories and installs them. A PHP script then changes a certain number of options and settings in each weblog. It can be used later to run as a “patch” on all installed weblog if a setting needs modifying globally.

Here are the details of what I did.

I first downloaded and unzipped WordPress into a directory.

wget http://wordpress.org/latest.tar.gz
tar -xzvf latest.tar.gz
mv wordpress wp-farm-image

I cleaned up the install (removing wp-comments-popup.php and the import*.php files, for example), added a language directory (as I’m wp-farming in French) and modified index.php to my liking; in particular, I edited the import statement for the stylesheet so that it looked like this:

@import url( http://edublogs.net/styles/toni/style.css );

The styles directory is a directory in which I place a bunch of WordPress styles. I don’t need the style switcher capability, but I do need to styles. Later, users will be able to change styles simply by editing that line in their index.php (or I can do it for them).

Another very important thing I did was rename wp-config-sample.php to config-sample and fill in the database and language information. I replaced wp_ by xxx_ so that I had $table_prefix = 'xxx_';.

To make it easier to install plugins for everyone, correct the language files, and edit whatever may be in wp-images, I moved these three directories out of the image install and replaced them with symbolic links, taking inspiration from Shelley’s method for installing multiple WordPress weblogs.

mv image/wp-content common
mv image/wp-images common
mv image/wp-includes/languages common
ln -s common/wp-content image/wp-content
ln -s common/wp-images image/wp-images
ln -s common/languages image/wp-includes/languages

I also added an .htaccess file (after some painful tweaking on a test install).

Once my image was ready, I compiled a list of all the users I had to open weblogs for (one username per line) in a file named names.txt, which I placed in the root directory all the weblog subfolders were going to go in.

I then ran this shell script (many thanks to all those of you who helped me with it — you saved my life):

for x in `cat names.txt`
do
cp -rv /home/edublogs/wp-farm/image/ $x
cat $x/wp-config.php | sed "s/xxx/${x}/" > config.tmp
mv config.tmp $x/wp-config.php
wget http://st-prex.edublogs.net/$x/wp-admin/install.php?step=1
wget http://st-prex.edublogs.net/$x/wp-admin/install.php?step=2
wget http://st-prex.edublogs.net/$x/wp-admin/install.php?step=3
done

This assumes that my WordPress install image was located in /home/edublogs/wp-farm/image/ and that the weblog addresses were of the form http://st-prex.edublogs.net/username/.

This script copies the image to a directory named after the user, edits wp-config to set the table prefix to the username, and then successively wgets the install URLs to save me from loading them all in my browser.

After this step, I had a bunch of installed but badly configured weblogs (amongst other things, as I short-circuited the form before the third install step, they all think their siteurl is example.com).

Entered the PHP patch which tweaks settings directly in the database. I spent some time with a test install and PHPMyAdmin to figure out which fields I wanted to change and which values I wanted to give them, but overall it wasn’t too complicated to do. You’ll certainly need to heavily edit this file before using it if you try and duplicate what I did, but the basic structure and queries should remain the same.

I edited the user list at the top of the file, loaded it in my browser, and within less than a few seconds all my weblogs were correctly configured. I’ll use modified versions of this script later on when I need to change settings for all the weblogs in one go (for example, if I want to quickly install a plugin for everyone).

In summary:

  1. compile list of users
  2. prepare image install
  3. run shell script
  4. run PHP script

If you try to do this, I suggest you start by putting only two users in your user list, and checking thoroughly that everything installs and works correctly before doing it for 30 users. I had to tweak the PHP script quite a bit until I had all my settings correctly configured.

Hope this can be useful to some!

Update 29.09.2005: WARNING! Hacking WordPress installs to build a farm like this one is neat, but it gets much less neat when your weblog farm is spammed with animal porn comments. You then realise (oh, horror!) that none of the anti-spam plugins work on your beautiful construction, so you weed them out by hand as you can, armed with many a MySQL query. And then the journalist steps in — because, frankly, “sex with dogs” on a school website is just too good to be true. And then you can spend half a day writing an angry reaction to the shitty badly-researched article.

My apologies for the bad language. Think of how you’re going to deal with spam beforehand when you’re setting up a school blog project.

Requirements for a WordPress Installer Script [en]

To install 30+ WordPress blogs on a server, it would be nice to have an automatic installation script. Here is a list of what this script should do for me.

[fr] Pour installer plus de trente weblogs WordPress sur mon serveur, il serait utile d'avoir un script d'installation en PHP. Quelqu'un a offert d'en écrire un pour moi. Ce billet récapitule ce que devrait faire un tel script, de mon point de vue (installation et configuration de WordPress en fonction d'un nom d'utilisateur).

As you may know, I’m shortly going to install 30+ WordPress blogs on my server. Noderat on #wordpress kindly offered to have a go at writing a PHP script to automate WP installs. I sent him this list of what the ideal script should be able to do for me, but on second thoughts, I’m posting it here so that everybody may see it. Of course, if you know of an existing script which already does this, let me know!

  • take $username + $password as input
  • install wordpress in a subdir named “$username”, using table prefix “$username_” and with an extra user (on top of admin) named “$username” (password=”$password”), user level 3
  • mysql user should be “$username” too (password “$password” also), with grants only on the tables belonging to this weblog
  • set permalink scheme to
    /archives/%year%/%monthnum%/%day%/%postname%/ for monthlies and /categories for categories
  • generate .htaccess in directory, based on this template, with “blog” replaced by “$username” everywhere
  • in wp-config: define (‘WPLANG’, ‘fr’) and edit appropriate lines
  • wp-includes/languages and wp-content/plugins should be symlinked to directories I can specify in the script
  • blog admin password should be reset to something I can specify in the script

I should be able to edit the script config file to provide mysql root user + pass, wordpress database name.

It would also be interesting if the script was built in such a way that it could be further modified/developed to allow installation of blogs on separate subdomains rather than subfolders. From the point of view of the filesystem the blogs live in, this wouldn’t change much — but some extra WordPress options would need editing (e.g. blog address), as well as the Apache and Bind config files necessary to set up the subdomain. This is not mandatory, but could be useful at some point (if we’re thinking in the line of WordPress-farming).

Musings on a Multiblog WordPress [en]

Thinking about a solution to make WordPress MultiBlog. Comments, criticism and other ideas welcome — please join the fun. In particular, I bump into a hairy PHP include problem.

[fr] Je réfléchis à comment on pourrait donner à WordPress la capacité de gérer plusieurs blogs avec une installation. Je me heurte à un problème concernant les includes PHP. Feedback et autres idées bienvenues!

Update June 2007: Try WordPress Multi-User now.

I’ve used Shelley’s instructions using soft links. I tried Rubén’s proof-of-concept, but got stuck somewhere in the middle.

So I started thinking: how can we go about making WordPress MultiBlog-capable? Here is a rough transcript of my thoughts (I’ve removed some of the dead ends and hesitations) in the hope that they might contribute to the general resolution of the problem. I have to point out my position here: somebody with a dedicated server who’s thinking of setting up a “WordPress weblog-farm” (for my pupils, mainly). So I’m aware that I’m not the “standard user” and that my solution is going to be impractical to many. But hey, let’s see where it leads, all the same. Actually, I think I probably reconstructed most of Rubén’s strategy here — but I’m not sure to what extent what I suggest differs from what he has done.

From a system point of view, we want to have a unique installation of WordPress, and duplication of only the files which are different from one blog to another (index.php, wp-config.php, wp-comments.php, wp-layout.css, to name a few obvious ones). The whole point being that when the isntall needs to be upgraded, it only has to be upgraded in one place. When a plugin is downloaded and installed, it only has to be done once for all weblogs — though it can of course be activated individually for each weblog.

From the point of view of the weblogs themselves, they need to appear to be in different domains/subdomains/folders/whatever. What I’m most interested in is different subdomains, so I’ll stick to that in my thinking. (Then somebody can come and tell me that my “solution” doesn’t work for subfolders, and here’s one that works for subfolders and subdomains, and we’ll all be happy, thankyouverymuch.) So, when I’m working with blog1.example.com all the addresses need to refer to that subdomain (blog1.example.com/wp-admin/, etc); ditto for blog2.example.com, blog3.example.com, blogn.example.com (I used to like maths in High School a lot).

As Rubén puts it, the problem with symbolic links (“soft links”) is called “soft link hell”: think of a great number of rubber bands stretched all over your server. Ugh. So let’s try to go in his direction, for a while. First, map all the subdomains to the same folder on the server. Let’s say blog1.example.com, blog2.example.com (etc.) all point to /home/bunny/www/wordpress/. Neat, huh? Not so. They will all use the same wp-config.php file, and hence all be the same weblog.

This is where Rubén’s idea comes in: include a file at the top of wp-config.php which:

  1. identifies which blog we are working with (in my case, by parsing $HTTP_HOST, for example — there might be a more elegant solution)
  2. “replaces” the files in the master installation directory by the files in a special “blog” directory, if they exist

The second point is the tricky one, of course. We’d probably have a subfolder per blog in wordpress/blogs: wordpress/blogs/blog1, wordpress/blogs/blog2, etc. The included file would match the subdomain string with the equivalent folder, check if the page it’s trying to retrieve exists in the folder, and if it does, include that one and stop processing the initial script after that. Another (maybe more elegant) option would be to do some Apache magic (I’m dreaming, no idea if it’s possible) to systematically check if a file is available in the subdirectory matching the subdomain before using the one in the master directory. Anybody know if this is feasible?

The problem I see is with includes. We have (at least) three types of include calls:

  • include (ABSPATH . 'wp-comments.php');
  • require ('./wp-blog-header.php');
  • require_once(dirname(__FILE__).'/' . '/wp-config.php');

As far as I see it, they’ll all break if the calling include is in /home/bunny/www/wordpress/blogs/blog1 and the file to be called is in /home/bunny/www/wordpress. What is wrong with relative includes? Oh, they would break too. Dammit.

We would need some intelligence to determine if the file to be included or called exists in the subdirectory or not, and magically adapt the include call to point to the “right” file. I suspect this could be done, but would require modifying all (at least, a lot of) the include/requires in WordPress.

Maybe another path to explore would be to create a table in the database to keep track of existing blogs, and of the files that need to be “overridden” for each blog. But again, I suspect that would mean recoding all the includes in WordPress.

Another problem would be .htaccess. Apache would be retrieving the same .htaccess for all subdomains, and that happens before PHP comes into play, if I’m not mistaken.

Any bright ideas to get us out of this fix? Alternate solutions? Comments? Things I missed or got wrong? The comments and trackbacks are yours. Thanks for your attention.

Converting MySQL Database Contents to UTF-8 [en]

I finally managed to convert my WordPress database content to UTF-8. It’s easy to do, but it wasn’t easy to figure out.

[fr] Voici comment j'ai converti le contenu de ma base de données WordPress en UTF-8. C'est assez simple en soi, mais ça m'a pris longtemps pour comprendre comment le faire!

A few weeks ago, I discovered (to my horror) that my site was not UTF-8, as I thought it was. Checking my pages in the validator produced this error:

The character encoding specified in the HTTP header (iso-8859-1) is different from the value in the XML declaration (utf-8). I will use the value from the HTTP header (iso-8859-1).

In all likeliness, my server adds a default header (iso-8859-1) to the pages it serves. When I switched to WordPress, I was careful to save all my import files as UTF-8, and I honestly thought that everything I had imported into the database was UTF-8. Somewhere in the process, it got switched back to iso-8859-1 (latin-1).

The solution to make sure the pages served are indeed UTF-8, as specified in the meta tags of my HTML pages, is to add the following line to .htaccess:

AddDefaultCharset OFF

(If one wanted to force UTF-8, AddDefaultCharset UTF-8 would do it, but actually, it’s better to leave the possibility to serve pages with different encodings, isn’t it?)

Now, when I did that, of course, all the accented characters in my site went beserk — proof if it was needed that my database content was not UTF-8. Here is the story of what I went through (and it took many days to find the solution, believe me, although it takes only 2 minutes to do once everything is ready) to convert my database content from ISO-8859-1 to UTF-8. Thanks a lot to all those who helped me through this — and they are many!

First thing, dump the database. I always forget the command for dumps, so here it is:

mysqldump --opt -u root -p wordpress > wordpress.sql

As we’re going to be doing stuff, it might be wise to make a copy of the working wordpress database. I did that by creating a new database in PhpMyAdmin, and importing my freshly dumped database into it:

mysql -u root -p wordpress-backup < wordpress.sql

Then, conversion. I tried a PHP script, I tried BBEdit, and they seemed to mess up. (Though as I had other issues elsewhere, they may well have worked but I mistakenly thought the problem was coming from there.) Anyway, command-line conversion with iconv is much easier to do:

iconv -f iso-8859-15 -t utf8 wordpress.sql > wordpress-iconv.sql

Then, import into the database. I first imported it into another database, edited wp-config.php to point to the new database, and checked that everything was ok:

mysql -u root -p wordpress-utf8 < wordpress-iconv.sql

Once I was happy that it was working, I imported my converted dump into the WordPress production database:

mysql -u root -p wordpress < wordpress-iconv.sql

On the way there, I had some trouble with MySQL. The MySQL dump more or less put the content of all my weblog posts on one line. For some reason, it didn’t cause any problems when importing the dump before conversion, to create the backup database, but it didn’t play nice after conversion.

I got this error when trying to import:

ERROR 1153 at line 378: Got a packet bigger than 'max_allowed_packet'

Line 378 contained half my weblog posts… and was obviously bigger than the 1Mb limit for max_allowed_packet (the whole dump is around 2Mb).

I had to edit my.cnf (/etc/mysql/my.cnf on my system) and change the value for max_allowed_packet in the section titled [mysqld]. I set it to 8Mb. Then, I had to stop mysql and restart it: mysqladmin -u root -p shutdown to stop it, and mysqld_safe & to start it again (as root).

This is not necessarily the best way to do it, and it might not work like that on your system, but it’s what I did and the site is now back up again. Comments welcome, and hope this can be useful to others!

Batch Categories 0.9 [en]

Batch Categories for WordPress has been fixed and enhanced. If you have major category jobs to do, it can probably help you. Feedback and testers welcome.

[fr] Batch Categories pour WordPress a été corrigé et fonctionne à  présent. Le compagnon idéal si vous desirez changer les catégories de nombreux billets en même temps.

Batch Categories 0.9 is out! It’s the ideal companion for large-scale post-import messy category work. List all posts belonging to a category or matching a keyword, and edit their categories, easily visible at a glace with a collection of sexy drop-down lists. What’s new since the the first draft?

  • It now works, and does not “eat” categories without a warning. (Pretty nice of it, huh?)
  • It tells you what it did — which categories it added to which posts, and which ones it removed.
  • Add a whole bunch of posts to a category with one click.
  • Remove a whole bunch of posts from a category with one click.
  • Ensures that all the categories for a post are always listed, whatever the setting for the limit number of drop-down lists.
  • This is what it can look like.

You can still access it as a plugin or edit the Edit navigation menu, as described in my post introducing Batch Categories. If you’re in a hurry, just drop the PHP file into your wp-admin directory and send your browser straight on it.

Next steps?

  • Gather feedback from courageous testers (please don’t blame me if you haven’t backed up your post2cat table and things go wrong) for chasing the last bugs, improving interface and functionality.
  • Redo the code which generates the drop-down lists to take advantage of the category cache, and avoid flooding the database with useless queries.
  • Allow more subtle selection of posts: combinations of categories (AND/OR/NOT), categories without their subcategories…
  • Anything else you would want…?

Update 24.07.04: BB made me notice that “All” and “None” didn’t make much sense in the drop-down which allows one to select the categories to display. Replaced them in v. 0.91 by “Any category”.