I can has weblog?

The boss asked me to create a list of everyone subscribed to every discussion list we have. Ended up being like 3000 lines…luckly there is a python script that polls the mailman admin web interface. The below script makes it quite a bit easier to perform on all the lists.

First thing, you need a file with all of your list names – this works:

ls /usr/local/mailman/lists | cat > lists.txt ## This is where mailman is located on FreeBSD

Next, download mailman-subscribers.py to your home directory.

Finally, you need to create a shell script (i.e. getallsubscribers.sh) and paste the following (substitute the [[bracketed text]] with the info for your system):

while read line
do
 echo $line | cat >> listsAndMembers.csv
  ~/mailman-subscribers.py -c [[your.mailman.host]] $line [[listAdminPassword]] | cat >> listsAndMembers.csv
  echo | cat >> listsAndMembers.csv
  echo | cat >> listsAndMembers.csv
done < lists.txt

chmod +x getallsubscribers.sh
./getallsubscribers.sh

That's it. You should have a single file with all of your lists and their subscribers.

edit: It came up later that it would be nice to get a list of who from each domain was subscribed. This regex will turn up the listname and name/email for each subscriber in the .csv

(?:.*(?:gmail\.com)"|^[^"].*$)

So, despite having logged into my blog just now without doing this – I’m going to share with you the easy way that probably 90% of you can protect your passwords while using the public wifi:

Step 1:

ssh -D 9999 -C somebox.youcan.sshto

Step 2

Go get something like FoxyProxy for Firefox and MM3 for Thunderbird (especially if you are using any version of Thunderbird 3 – nothing else is compatible).

For Firefox, set up a SOCKS5 proxy (Prefs > Advanced > Network > Settings) on localhost port 9999. Start using the proxy. Stop using the proxy when you kill your ssh connection and make sure you open the connection if you are using the proxy.

For thunderbird it’s basically the same if you are using FoxyProxy – but if you are using MM3 like me, you need to create (click edit, oh btw you will probably have to add the button to the toolbar first) a proxy with this config:

[Arbitrary_name
  socks=127.0.0.1:9999
]

That’s it. Now I can’t steal your passwordsss! If you want to encrypt DNS requests you can set network.proxy.socks_remote_dns to true in about:config

I’ve been trying to write this for months now, but it always comes out cheesy sounding and not very well said. I’ve given up. Here is my best attempt at thanking a person.

So… this may seem a little bit odd if you have been to my site before. I’m not accustomed to writing about things other than geeky-computer-how-to type stuff, but I wanted to give a shout to a person who means a lot to myself and my family.

Margaret Waddell has been a wonderful friend of our family for a little over a year now. We first met in a natural foods store where she was, if memory serves(probably doesn’t), mixing herbs for a foot soaking cocktail. Of course, my very outgoing daughter Freya caught her attention and it quickly came to light that Margaret was a music teacher in need of marketing and web services. I could tell right away that we were a good match and offered to barter my web development services for her music classes. Best decision ever.

Freya absolutely loves Margaret. Freya likes just about everyone, but with Margaret it’s different. You can see it with Margaret too. A kinder person I’ve not met, but you can see and hear it when Margaret interacts with Freya…it’s love. Not only has Freya gained a lifelong friend, but her musical skills have gone through the roof. She loves listening to and making music all day long, it’s a real joy to see and hear.

You can find the website I built for Margaret at http://hummingbirdmusic.net if you have a child, and want them to have the benefits of an early childhood music program, do give her site a visit. Margaret teaches classes for little ones with singing, hand clapping, bounces, shakers, dancing, and so on…but also does piano/keyboard classes as well as Irish Tin Whistle. Check out the class listing here: http://hummingbirdmusic.net/classes.

Thanks Margaret, we all love you! (You too RavenWolf!)

Update: NATURALLY, after doing all of this I learned that I was given the wrong information. Turns out Mailman is more than happy to take a huge mbox file as input for the arch script. I did learn that running clean_arch on the mbox first is a good idea…

Are you like me? Do you get upset when you have to deal with an almost decade old problem that you had nothing to do with? Well then I’ve got a story for you…

So we’ve got these archives…

Our organization was using Lyris ListServ for about the past 10 years to handle all of our discussion list. Like most MLM’s, ListServ does have the ability to keep list archives…but naturally we opted to not use them for all of our lists. Big mistake.

Instead of list archives we have a user on our webserver called ‘archive’. Archive is subscribed to each and every list and gets copies of all the messages. When the messages come in, Archive processes them with a procmail script and separates them into mbox mailboxes for each of the lists.

Each of the mails are then piped to a program called mhonarc which converts them into html and provides an index, etc – which can be displayed on our current(old) website. But now we’ve got a new website coming up…

Enter GNU Mailman

Me:

Thank you for coming Mailman. I’m really glad to have you because you do a really good job, not to mention you’re free and uber-powerful… One thing though… We’ve got these uh, gulp “archives”. We, uh, need to keep them and everything but you know, they’re like, not in the greatest shape. See, there actually in mbox format…

Mailman:

Oh yeah, that’s not a problem at all. I’ve got a built in script to to do that. Just take all of the monthly mailbox files for each of your lists and drop them in my folder – I’ll knock them out in no time!

Me:

SWEET! But what did you say about monthly whatevers?

Mailman:

The mailbox files that you create every month for each list… You are using your procmail script to start a new mailbox file every month aren’t you? Putting 10 years worth of emails into a single monolithic file would be retarded…

Me:

Oh yeah yeah… Of course we did that. I thought you were talking about something else. Silly me. Anyway, so uh, yeah, I’ll get those file to you real soon.

Breaking up the monolith

So clearly I needed to edit the procmail script a little bit and reprocess all the mail – but WTH?

Last month I had a help desk ticket come my way about a list which was not appearing on the website, and hadn’t been for a number of months. After digging around, I realized that someone (almost certainly me) had made a mistake in the .procmailrc file which had kept it from processing mail for that list. This was embarrasing, but I discovered how to reprocess mail with formail.

I knew I could probably reprocess all of the mail (many thousands) but had absolutely no clue how to do it. I had only this one clue from my procmail recipe:

LOGFILE=$PMDIR/list_archive-`date +%Y-%m`.log

They had written it to rotate the logs, but not the mailbox names! Uncool! But at least I had my answer – `date +%Y-%m` can get the date into the names…But wait!

When I rewrote my procmail file like so:

#this is just one of many
:0 E
     * ^Sender:.*LIST-L
     {
       :0 c
       LIST-L.`date +%Y%m` #to match mailmans archive format...
     }

Totally not working! It created only one file, and the date was this month and this year.

I’ll fast-forward for the benefit of the reader at this point and just share an insight with you: date is a *nix command and has nothing to do with procmail and cannot get any data out of the emails themselves – like dates! Yes that’s right, you can put anything you want in between those little ticks, but because the date command only returns the system date, we’ve got to do two things:

1. Get the date the email was sent out of the email header (Magic)
2. Process the date field to have only the 4 digit year and 2 digit month (More Magic)

It gets easier from here…

Getting the date field

The first thing we need to do is nab the Date: header from the emails. This part is fairly straight forward. Procmail uses a variable $MATCH to hold the matched string for the rule that it’s matching on. We can use this to hold our Date header and then just pipe it to a script for processing.

Here’s the recipe magic!

# NOTE: I later found that this rule only seems to work
# when used as an ELSE rule (:0 E). I'm not sure why, but
# it was only matching the 'Date:' portion, and not the entire
# line. If you can help me understand why, please leave a comment.
# ANOTHER NOTE: The ticks in `echo $MATCH.... are ticks(the
# un-shifted tilde) and not single quotes.

:0 E
     * ^Sender:.*LIST-L
     {
       :0 c
       * ^\/Date:.*
       LIST-L.`echo $MATCH | php /path/to/dateconvert.php`
     }

Transforming the date into Mailman’s monthly format

Here’s the php code more magic to get your date cut down and switched around to a format Mailman will love. It uses php’s built in functions fairy dust to put the Date header into an array, and drop the empty elements. It was also necessary to create an array that maps the three letter month names used in the email header to their numerical equivalents.

< ?php
 $date = trim(fgets(STDIN));
 
 $datearray = array_values(array_filter(explode(" ", $date)));
 
 $month = $datearray[3];
 $year = $datearray[4];
 
 $montharray = array(
 "Jan" => "01",
 "Feb" => "02",
 "Mar" => "03",
 "Apr" => "04",
 "May" => "05",
 "Jun" => "06",
 "Jul" => "07",
 "Aug" => "08",
 "Sep" => "09",
 "Oct" => "10",
 "Nov" => "11",
 "Dec" => "12"
 );
 
 echo $year . $montharray[$datearray[3]];
?>

At this point you’re probably thinking one of two things:

1. “ZOMG you’re such a hack. You could have done that with so much less code! You have no style!”
2. “ZOMG you can totally do that with a sed/awk one liner!”

Sorry for wasting everyone’s time yet again.

After you’ve got the recipe in place and the php file all ready to go, just give it one of these:

Final product

Anyway, you should, after a few hours or days end up with a grip of mailbox files like this:

               LIST-L.200306  LIST-L.200502  LIST-L.200701
 	        LIST-L.200307  LIST-L.200503  LIST-L.200702
LIST-L.200112  LIST-L.200308  LIST-L.200504  LIST-L.200703
LIST-L.200201  LIST-L.200309  LIST-L.200505  LIST-L.200704
LIST-L.200202  LIST-L.200310  LIST-L.200506  LIST-L.200705
LIST-L.200203  LIST-L.200311  LIST-L.200507  LIST-L.200707
LIST-L.200204  LIST-L.200312  LIST-L.200509  LIST-L.200708
LIST-L.200205  LIST-L.200401  LIST-L.200511  LIST-L.200709
LIST-L.200206  LIST-L.200402  LIST-L.200512  LIST-L.200710
LIST-L.200207  LIST-L.200403  LIST-L.200601  LIST-L.200711
LIST-L.200208  LIST-L.200404  LIST-L.200602  LIST-L.200712
LIST-L.200209  LIST-L.200405  LIST-L.200603  LIST-L.200801
LIST-L.200210  LIST-L.200406  LIST-L.200604  LIST-L.200802
LIST-L.200211  LIST-L.200407  LIST-L.200605  LIST-L.200803
LIST-L.200212  LIST-L.200408  LIST-L.200606  LIST-L.200804
LIST-L.200301  LIST-L.200409  LIST-L.200607  LIST-L.200805
LIST-L.200302  LIST-L.200410  LIST-L.200608  LIST-L.200806
LIST-L.200303  LIST-L.200411  LIST-L.200610  LIST-L.200807
LIST-L.200304  LIST-L.200412  LIST-L.200611
LIST-L.200305  LIST-L.200501  LIST-L.200612

So there ya go. Now make with the comments.

This document is continually changing! One of the ways it gets changed is by people communicating with me using comments. In the future, I will host a wiki for this purpose – but for the time being please help me out by posting your suggested changes/improvements as comments!

In my previous post I described the troubles I had with standalone document management softwares. Many of the issues I had were related to a lack of flexibility and lack of integration with my CMS of choice: Drupal.

At first glance(and even after looking quite hard), Drupal seems to be weak when it comes to document management. But just like any Drupal solution, a careful examination of the available modules might turn up the ingredients for the perfect recipe!

In this article, I’m going to describe the steps required to get off the ground with a Drupal based document management solution that will provide:

• Organization of documents
• Revision control
• WebDav access
• Rich metadata
• Indexing for search
• In-browser display of documents
• Document conversion services
• All the goodness you get from building it inside Drupal
  • Free authentication
  • Free administration interface
  • Integration with other Drupal modules (Views anyone?)
  • Awesome community of developers

Getting started

I’d recommend testing this out on a fresh install of Drupal6.6 – should you encounter difficulty, the number of modules on an established site could make troubleshooting a bit more difficult. After you’ve got it down, you can move on to your active development site.

Thanks to Arto Bendiken, Miglius Alaburda, Justin Miller, Ben Lavender, Frank Febbraro, and of course Moshe Weitzman.

This article is based on Setting up your system for file conversions with File Framework. Ben gives a very helpful and accurate rundown of what it takes to get going under CentOS. Since I was trying it out under Ubuntu, I thought I’d spend the time documenting my troubles – and include instructions to add some extra bells and whistles.

System stuff

First things first, lets go ahead and get all the packages we need:

sudo apt-get install php5 php5-dev php-pear make php-getid3 libmagic-dev clamav swftools unrtf poppler-utils catdoc ghostscript tzdata tzdata-java alsa-tools alsa-utils libx11-6 libxext6 libxi6 libxtst6 asoundconf-gtk libfreetype6 libpng12-0 libjpeg62 giflib-tools libsm6 openjdk-6-jdk openoffice.org openoffice.org-headless code2html pstotext
sudo pecl install Fileinfo
sudo pear install http://download.pear.php.net/package/HTTP_WebDAV_Server-1.0.0RC4.tgz
sudo pear install http://download.pear.php.net/package/HTTP_WebDAV_Client-1.0.0.tgz

If you have trouble with the install of the pear modules, probably the version has changed – you should visit the HTTP packages page.

JODConverter

We also need to get the JOD Converter. It’s a few .jar files that we’ll stick in a directory in /opt. JODConverter is the piece that actually manages the conversion process through openoffice.

Run OpenOffice as a service

Long story short, use a version later than 2.3 to avoid problems running it ‘headless’. This is essential for the file conversion process.

edit: I realized that the OpenOffice service really needs to be running as www-data, so using an init script like this one is really necessary.

#!/bin/bash
#
# description: Open Office Service
#

export WEBUSER=www-data
export PATH=$PATH
export LANG=en_US.UTF-8

start() {
echo -n "Starting OpenOffice service: "
sudo -u $WEBUSER /opt/openoffice.org3/program/soffice -headless -accept="socket,host=127.0.0.1,port=8100;urp" -nofirststartwizard & 
echo "OpenOffice Started"
}

stop() {
echo -n "Stopping soffice: "
pkill soffice
echo "OpenOffice Stopped"
}

case "$1" in
start)
start
;;
stop)
stop
;;
status)
status soffice
;;
restart|reload|condrestart)
stop
start
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|status}"
exit 1
esac

exit 0

If you want OpenOffice3 like I’m using, you might want to remove the 2.4 packages with apt-get remove and go to openoffice.org and download the .deb packages. I installed by extracting the archive, cd’ing into the folder and using

sudo dpkg -i *.deb

and doing the same in the desktop integration folder. I can’t really recommend using OOo3 because the Ubuntu folks don’t have it in the repos…and the GUI is very crash happy.

Drupal stuff

Clean URLs

Pop over to the Drupal.org page describing how to set up clean urls if you don’t have that going already. Clean urls aren’t necessary, but due to a bug currently in bitcaching – it is.

Install Drush

If you aren’t using the Drush module, I highly recommend it. Although not related to or necessary for this project, since I discovered it one day ago, it’s become one of my favorite modules. It provides a familiar way to install and update your packages – and has a number of modules that extend it’s functionality.

• Install the Drush module by downloading the tarball to your modules directory (sites/all/modules) and extract it.
• Go into your modules page in Drupal and enable the Drush and associated modules. You won’t be able to turn on the simpletest runner module, that’s fine. Also – I wasn’t able to use the CVS support, so I have that disabled as well.

One last thing – you need to add a softlink to drush.php somewhere in your path. For me, I just echoed the path variable and picked the place that looked the best… Make sure you change any paths to whatever works.

% echo $PATH
/home/hopkinsju/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

% ln -s /var/www/drupal/sites/all/modules/drush/drush.php /home/hopkinsju/bin/drush

Now you should be able to type ‘drush’ and the computer will know what you’re talking about.

Add required modules with Drush

Now we just do this to get all the modules we need:

Drush will go out and grab the latest version of each module and extract it in your ’sites/all/modules’ directory.

note: As of this writing bitcache-alpha3 had a bug in it. Using alpha3 will result in the error “Fatal error: Unsupported operand types in serverpath/includes/common.inc on line 1546″. To resolve this, you can use either the alpha2 or dev versions of the bitcache module

A few other bits

The File Framework can get metadata for and play flash and mp3 files. You need only add a couple things to the vendor folder of fileframeworks:

edit: Using the commands below should get you going without a problem, but I wanted to clarify: You MUST use the ’slim’ version of the xspf player. Also, the path to getid3 should be /vendor/getid3 – there should also be a directory /vendor/getid3/getid3 containing the different modules.

update: new versions of getID3 and flowplayer as of Mar 18, 2009 – also you need to make folders for them…I’ll update the lines in a bit.

cd /path/to/drupal/sites/all/modules/fileframework/vendor
wget http://voxel.dl.sourceforge.net/sourceforge/getid3/getid3-1.7.9.zip
unzip getid3-1.7.9.zip

wget http://flowplayer.org/releases/flowplayer/flowplayer-3.0.7.zip
unzip flowplayer-3.0.7.zip

wget http://voxel.dl.sourceforge.net/sourceforge/musicplayer/xspf_player_slim-correct-0.2.3.zip
unzip xspf_player_slim-correct-0.2.3.zip

Enable the modules

Visit your modules page and enable the modules you need. When I first attempted this, I did run into an error where I had enabled one module or another without first enabling the modules it required(I think it was the RDF API module that needed to be enabled before the File formats). You’ll want to actually look at what you’re installing rather that just checking all the boxes of course. But basically – check all the boxes ;)

Drupal admin area things

• Visit admin/settings/dav/dav_fs and save the page to create the dav directory
• Enable DAV Server in admin/settings/dav
• If you want html highlighting for text files admin/settings/file/format/text
• Enable antivirus scanning (I chose to run it as a program) admin/settings/file/antivirus
• Enable file formats admin/settings/file/format
• Go tell the Fileserver that you want it to use the ‘Files’ vocab. Doing this will enable automatic creation of file nodes when items are added to that folder via WebDAV.

Please post your comments if you can improve on what I’ve done!

Happy document managing!

I’ve been developing a website for my job for quite some time now. We are getting very close to the launch date – and I’ve had one issue stuck in my craw for almost the duration: Document Management.

Document Management

What is document management? It seems pretty straight-forward to me, but many people have looked at me cross-eyed when I’ve used the term, and some have come right out and said “What do you mean, document management?”. Well, there’s this wiki, but basically I mean a system that gives me the ability to post my files on the web and have a decent amount of access control(user roles, permissions, etc). Naturally there are other features that are not definitive, but essential: Revisioning, search, and metadata is just a start.

There are many document management systems out there, both proprietary and open-source. Of what’s available, all are lacking. What strengths they have are countered with other hard to justify requirements: cost, complexity, administrative overhead.

I’ll take just a minute to talk about two solutions I’ve tried already: Micro$oft SharePoint 2007, and KnowledgeTree Community Edition.

SharePoint

I’ve been using SharePoint for around 6 months now. It’s certainly not bad at the job of document management – and in fact has the ability to go way beyond – given a full time asp.net developer and a fat budget.

I actually had to take a class to learn how to admin the system. Sure I could have probably picked it up on my own, given enough time…but it just speaks to the complexity issue. So, after 2 days of class – the take away was really this:

SharePoint can do anything. You might have to program it to do that, but it can do it. Community of developers? Oh uh, yeah there’s probably one out there… So what does it do out of the box really well? How about locking you into other M$ products – does that count? The honest truth is that out of the box SharePoint has numerous features that are shiny on the outside, but hollow on the inside.

One of the biggest stumbling blocks for our organization was the inability to use any type of authentication besides Active Directory – quite possibly a limitation imposed by the University to keep the system as secure as possible – but a real deal-breaker when you work with 62 different institutions.

KnowledgeTree Community Edition

Since this section of the article is really getting away from me…I’d better take a shortcut and include a part of some email correspondence with a KT sales rep:

My evaluation of KT is going well – although we have hit a few stumbling blocks. The mechanisms that KT uses to do the indexing seem to be overly done, requiring OpenOffice to be running as a background service, which is quite a bit just to allow for the indexing of word documents.

The documentation seems sparse – which is to be expected…but in addition the style of the commands seems to be an old and very specific style (i.e. chown -R nobody.nogroup as opposed to the more universal nobody:nogroup). We’ve had some difficulty getting the scheduler to run due to permission errors as well. From my point of view – not having expert knowledge of the history and reasoning behind the decisions about how KT is built – it seems to have a somewhat polished look on the outside, while being cobbled together on the inside…in need of a total rewrite.

Lastly, the integration with Drupal that I so badly needed didn’t work out very well. After applying the changes, the KT system became inaccessible, and a restore of the files and db from backups was necessary. This isn’t the fault of the KT software I’m sure, but nonetheless affects my ability to use it for my needs.

I’m afraid my KT project is on hold at this point due to time constraints.

I expect I will take it on again when time allows.

Ultimately, the system is fairly good and has a flexible authentication system, but the Drupal integration even if it did work was pathetic at best: can you say “iframe”?

Coming soon: How To: Setting Up Drupal File Framework On Ubuntu 8.10

WordPress 2.7-beta3-9863, as its name suggests…is a landmark update for WordPress.

Dashboard at a glance

Besides the new layout, there are other changes obvious in the first 5 minutes:

• A new “Quick Press” section – I-’m writing- started writing this post using it.
• Ajaxy accordion widgets replace the top-nav sections in the new sidebar menu.
• Sidebar can be minimized a la photoshop toolbar – child items pop-out on hover.
• Totally new(rockin’) icon set – apparently they had a contest – grats “BD”
• Did I mention the Ajax?? I don’t mean to be a web2.0 fanboy here – but honestly – UI design can make or break the experience…and unfortunately it’s more often breaking it.

No sooner than I say I’m not a web2.0 fanboy as I mentally say “Ooooeee!” upon noticing that the boxes can be repositioned via drag-and-drop. That’s, like, web2.0 meme numero uno.

Editing content

Viewing the list of my posts I noticed two things: An alternate view of posts that shows excerpts(just in case you don’t use the most descriptive titles), and the ability to “quick edit” the titles, categories, and a few other things about the post. Unfortunately you can’t edit the body of the post without clicking ‘edit’.

Revisions

This might have already been there – but revisions are great. At the bottom of the “Edit Post” page you get this:

Clicking one of the links takes you to a page showing the text of your post along with an array of radio buttons for all previous revisions and their restore links. Talk about handy.

Overall

I’m very very happy with the upgrade. I know WP updates don’t come with much fanfare – but this one was touted as having “major improvements”. I certainly think they came through.

There are some minor issues – browser related positioning errors and what-not – but I’d expect an official launch in the very near future.

Want what I have?

If you’d like to upgrade, you can just visit Tools>Update and get the “Nightly build”…Or, if you’re like me, and prefer to handle your upgrades with Subversion(it’s easier to go downgrade, faster to do multiple blogs, and simpler to merge your customizations with upstream code) – then you’d just ssh into your blog directory and…

If you are already tracking trunk(then you probably already know this):

svn update

OR

If you are tracking the latest stable version, but want to try the new stuff:

svn sw http://svn.automattic.com/wordpress/trunk/ .

If you hate it and want to switch back:

svn sw http://svn.automattic.com/wordpress/tags/2.6.3/ .

OR

If you don’t have a WP blog yet and want to get started with the latest stuff or don’t have anything in your media gallery worth keeping(you could always back that up) – and haven’t made any customizations to WP themes(again, you could always back those up). Basically if you don’t yet have your WP blog under revisioning, and need to get started with svn:

svn co http://svn.automattic.com/wordpress/trunk/ .

Enjoy!

Usually the word serendipity(as I hear it most often) is used to suggest that there was meaning in coincidence. Something unlikely that, simply because it occurred to the person in question, is somehow filled with meaning beyond chance.

Upon consulting the Wikipedia – I was relieved to find that the definition of serendipity didn’t exactly fit.

Serendipity is the effect by which one accidentally discovers something fortunate, especially while looking for something else entirely.

Last night, after we devoured a Chinese food feast – we each opened our fortune cookies. Everyone who has ever opened a fortune cookie knows it is very much like reading a horoscope, having your palm read, or consulting a crystal ball. Sometimes, most of the time, you can interpret your fortune to be “OMG that’s totally me!”.

Since we were, in fact, expecting fortunes – it really can’t be called serendipitous… but they did make us all feel sweet, so I thought I’d share.

Freya’s fortune

Life to you is a dashing and bold adventure.

Now that’s a well written fortune. Freya does live life as an adventure – “adventure” is what we call just about anything that involves leaving the house…

Laura’s fortune

You have a strong desire for a home and your family comes first.

Laura is all about making a home for our family. She sacrifices endlessly for us.

My fortune

Your present plans are going to succeed.

Perhaps the most like your usual fortune – but still being that I’m uber stressed about a project that I don’t want to fail…needed reassurance. I guess I’ll take what I can get at this point.

I guess it goes to show – things don’t have to have meaning to be nice.

Typo is blog software written in Ruby, and available as a gem for Rails. It’s what is currently running this website. It’ has it’s pluses but over time it’s grown aggravating to me. I’ve decided to move on to another platform, but I’m not sure which at this point.

If by the time you are done reading this post you feel like Typo would be a good fit for you, you can download Typo and try it yourself. I’m by no means the expert in these matters, and what might not work for me might be right up your alley.

Why I picked Typo in the first place

When I was originally looking for a blog software to use I had these basic requirements:

• Not what everyone else was using(Wordpress, Blogger)
• Not written in PHP

I didn’t want to use what everyone else was using because in most things – I like to be different. The fact is that I am different from most people, and I tend to make that known.

I wanted to use a product that was not written in PHP because as an aspiring programmer, I’d rather be using a more cutting edge language like Ruby or Python. Not only is the code of Ruby/Python easier to write and the languages more interesting, but the jobs you get when you know them pay very much more than with PHP, whose programmers are a dime a dozen. I’m not saying that it’s not worth knowing – indeed PHP may be the most useful language to know due to it’s ubiquity. Also, I don’t want to offend any of the PHP programmers who may be reading – I’m envious of your skills(read: skillz).

Ok, so why Typo? Well looking back I didn’t look into things all that well. I ended up turning all the negatives into positives which isn’t a very smart thing to do. At some point, mostly for very shallow reasons I decided that Ruby was a little cooler than Python, so having eliminated all of the Python blog software(there aren’t many) I had a very very short list to choose from. When I tried the demo of Typo I was fairly impressed by the admin interface. Also even though there were very few themes available, I really liked the one I have applied here(scribbish).

What I don’t like about Typo

Lack of community

One thing about open-source software is that it really needs a large and active community. This is extremely evident when you start using projects like Drupal, Ubuntu, Eclipse, and so on. Typo is not like that at all. It’s basically just one person developing with a few other making commits. This really leads to nowhere fast.

If you read the first blog post I wrote on this website you’ll see the difficulty I encountered just getting up and running. This was due almost entirely because of Typo development that wasn’t keeping pace with the development of it’s fundaments – Ruby/Rails. Even after working around the issues I was having, an entire section of my admin area is broken. Looking in the svn repo for Typo I’ve found that there is an experimental branch for Ruby 1.8.7, but it’s pretty clear that it’s not going to be finished before Ruby increments again.

Lack of features

Obviously since this is an open-source project, this is intimately tied to the lack of community – but there are things that should have been available at the outset. For example: users(myself included) cannot log into the public side of the website. This makes it very annoying for commenters – having to retype their info each time, and not being able to edit their comments – but also for me If I want to edit anything I have to do it from the admin side.

Another feature missing that is somewhat related to front-side logging in is a feature that I first saw in WP – private posts. The ability to make a post that only permitted users can see is essential for the entire concept of weblog. The idea that everything I publish should be public is ludicrous.

Plug-ins. Again, because plug-ins are almost by definition community contributed I wouldn’t expect to see too many of them – but the number available is really pathetic. Less than 20 are available, and almost all are totally useless to 99.9% of people on the internet.

This brings me to my next point:

Why I’m dumb

I managed to turn every one of the things I’ve identified as negative about Typo into a positive.

Q: Don’t you want to use a product with a huge community? A: No, I want to be different!

Q: Don’t you want to have thousands of plug-ins available? A: I’ll just write whatever I need!

Q: Don’t you want blogging to be easy? A: …

So yeah, now I realize that if you want to actually, say, write blogs then you have to have the time to do that. I want to learn Ruby – of course – but I want to write quick blogs about my accomplishments too. Starting a blog as a learning project means you have neither a blog or a project – just a bunch of stagnation. In my case I opted to write blog after blog and never have the features I wanted, feeling pissed off all the while. not good.

So, what to use now?

Honestly I’m leaning towards WordPress. It’s the obvious choice for obvious reasons, but until I have a couple hours to look again at all of the(free, open-source) options – I can’t make a final decision.

I’d still like to look at Ruby/Python blog softwares, and this time not because I want a pet project. I’ll look at performance issues – I know that Ruby, but especially Python will outperform PHP, but after caching is taken into account the difference may be negligible.

I don’t think WordPress is inherently insecure – as some assert it is due to it’s open-source nature(unlike windoze, which is SO secure!), but it might be vulnerable due to it’s high usage and therefore high rate of attack(just like windoze!). But that’s all anecdotal – I don’t know security in general much less the minute details of the WP code.

Mephisto is another Ruby blog, but it looks like it suffers from the same problems as does Typo. I’d love to do either one of these if I could get paid to work on them.

Anyway…I’ll just keep looking around for now. Post a comment if you want to make a suggestion! (Sorry it won’t remember your user info and I won’t be notified of new comments)

Since attending HighEdWeb 2008 and going to Jason Woodward’s Workshop on XML/XSLT, I’ve been really interested in the possibilities of using XSL to tranform a well-formed XML file into almost anything – especially another type of file, such as XHTML.

This topic is totally new to me, and I’ve never had the chance to use it in a real life scenario. I have played around a little bit with a hypothetical exercise though, and wanted to post my solution for others to use, and hopefully, offer some criticism on. Please do comment if you can think of a better or more elegant solution.

The business need

Develop a single web page that shows Henry’s current inventory. Develop a single page using XHTML standards that shows the inventory using XSL. CSS will be in the head of the XSL page and will be used to style the page. The page will show the widgets grouped by the type of metal in alphabetical order, and then the widgets in order of price with the lowest price item shown first (the list of widgets is only shown once in the page). Lastly, the out of stock widgets should be formatted differently than the ones that are in stock. Tables will be used to layout the web page.

The XML

I developed this XML a few weeks ago, and even wrote another post patting myself on the back for having authored a schema defining my widgets, but here is a snippet:

< ?xml-stylesheet type="text/xsl" href="inventory.xsl"?>
<inventory xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  xsi:noNamespaceSchemaLocation="inventory.xsd">
  <widget>
    <material>Aluminium</material>
    <length units="ft">1</length>
    <width units="in">2</width>
    <thickness units="in">.25</thickness>
    <price units="USD">15.00</price>
    <stockqty>25</stockqty>
  </widget>
...
</inventory>

The good stuff – XSLT

I think learning XSLT was equal parts learning to understand XSL and then learning the intricacies of Xpath – especially as it relates to <xsl:apply-templates /> and matching/selecting the elements. Anyway here’s what I came up with:

<xsl :stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  </xsl><xsl :template match="/">
    <html>
      <head>
        <title>Henry's Widget Shoppe</title>
        
      </head>
      <body>
        <h2>Here's my available widgets!</h2>
        <table id="Inventory">
          <tr class="tableHead">
            <td>Material Type</td>
            <td>Length</td>
            <td>Width</td>
            <td>Thickness</td>
            <td>Price</td>
            <td>Stock Qty</td>
          </tr>
          <xsl :apply-templates />
        </table>
      </body>
    </html>
    </xsl>

    <xsl :template match="inventory">
      </xsl><xsl :for-each select="widget">
        <xsl :sort select="material" order="ascending"/>
        <xsl :sort select="price" data-type="number" order="ascending"/>
        </xsl><xsl :choose>
          </xsl><xsl :when test="stockQty>0">
            <tr class="widget">
              <xsl :apply-templates select="." />
            </tr>
          </xsl>
          <xsl :otherwise>
            <tr class="widget outOfStock">
              <xsl :apply-templates select="." />
            </tr>
          </xsl>
          
                    
    

    <xsl :template match="widget">
            <td class="material">
              <xsl :value-of select="material"/>
            </td>
            <td class="length">
              <xsl :value-of select="length"/>
              <xsl :value-of select="length/@units"/>
            </td>
            <td class="width">
              <xsl :value-of select="width"/>
              <xsl :value-of select="width/@units"/>
            </td>
            <td class="thickness">
              <xsl :value-of select="thickness"/>
              <xsl :value-of select="thickness/@units"/>
            </td>
            <td class="price">
              <xsl :if test="price[@units] != 'USD'">$</xsl>
              <xsl :value-of select="price"/>
            </td>
            <td class="stockQty">
              <xsl :value-of select="stockQty"/>
            </td>
    </xsl>

Conclusion

I did make several improvements and tried alternate strategies over the weekend, so this isn’t exactly what I first puked out. All in all, I’m very pleased with it. This was probably one of the most valuable self learning exercises I’ve yet had.

Download the source here and/or View the transformed XML here