You know what rocks? This fetchmail init script for Red Hat!

Posted by Justin – June 23, 2010

This is from the contrib directory of the fetchmail source and is much better than what Google gives you.


#!/bin/sh
#
# fetchmail	This shell script takes care of starting and stopping
#		fetchmail.
#
# chkconfig: 2345 81 45
# description: The Fetchmail daemons allows to retrieve mail using various
#	       mail protocols and route them to the local MTA just as if
#	       the mail was sent directly to the local MTA. This is
#	       specially useful on intermittent dial-up connections.
# processname: fetchmail
# config: /etc/fetchmailrc
# author[s]:
#	Andrea Sterbini	<a .sterbini@itelcad.it>
#	ObiTuarY <obituary @freshmeat.net>

. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
if [ ${NETWORKING} = "no" ]
then
	exit 0
fi 
    
# See how we were called.
case "$1" in
  start)
	if [ -s /etc/fetchmailrc ]; then
		echo -n "Loading fetchmail: "
		daemon /usr/bin/fetchmail -f /etc/fetchmailrc
		echo
		touch /var/lock/subsys/fetchmail
	else
		exit 1
	fi
	;;
  stop)	
	echo -n "Shutting down fetchmail: "
	/usr/bin/fetchmail -q >/dev/null 2>&1 && echo fetchmail
#	killproc fetchmail
	rm -f /var/lock/subsys/fetchmail
	;;
  status)
        status fetchmail
        ;;
  restart|reload)
        $0 stop
        $0 start
        ;; 
	*)
	echo "Usage: fetchmail {start|stop|status|restart|reload}"
	exit 1
esac

exit 0

# === End of File ===
</obituary></a>

My Favorite Way To Delete Blank Lines WIth VIM

Posted by Justin – April 12, 2010 


:g/^$/d

Vector Outline of the State of Missouri

Posted by Justin – February 3, 2010

I think this is the 3rd year in a row that I’ve set out looking for an outline (vector format) of the state of Missouri. Each year I end up downloading one image or another from some Missouri state government website and creating a pen path around it in Photoshop or Illustrator.

You’d think that I would have saved the resulting outline one of these times, and well – I guess this is that time. I’m sure I have it somewhere on a network drive, but you know how that goes…

So I’m going to post it here: an Illustrator file with vectors for the Missouri outline and a Google Map underlay for locating cities and such, a .psd with path (check the paths palette), an SVG, an EPS, and a JPEG.

Missouri Outline Vector Files (.zip)


CC0

To the extent possible under law, Justin Hopkins has waived all copyright and related or neighboring rights to Outline of the State of Missouri. This work is published from United States.

Installing VirtualBox 3.0.12 on Fedora 12

Posted by Justin – November 29, 2009

If you want to get VirtualBox running on Fedora 12, it’s really not that hard. Please refer to the VirtualBox website for more up to date instructions.

FIrst thing you need to do is install dkms, and kernel-devel.


localhost$ su -
root@localhost$ yum install dkms
If you don’t do the above step, and are running a vanilla Fedora 12 the later step of installing VirtualBox will not be able to finish and complain about
“No precompiled module for this kernel found — trying to build one. Messages emitted during module compilation will be logged to /var/log/vbox-install.log. Compilation of the kernel module FAILED! VirtualBox will not start until this
problem is fixed. Please consult /var/log/vbox-install.log to find out why the
kernel module does not compile. Most probably the kernel sources are not found.
Install them and execute”

Now we can move on to adding the VirtualBox repo:


wget -q http://download.virtualbox.org/virtualbox/debian/sun_vbox.asc -O- | rpm --import -

Now you are good to go…


root@localhost$ yum install VirtualBox

# There is a compile step in this installer, and make take a few minutes.#

If you happened to have tried installing VirtualBox and ran into the above “Compilation of the kernel module FAILED!” message, just install dkms then ‘/etc/init.d/vboxdrv setup’ (Compiling will take some time probably).

KVM issues
Apparently the KVM kernel modules(which you would have if you were using the Fedora Virtual Manager) aren’t compatible with VirtualBox. There is a bug filed but as a workaround (or possibly just the easiest solution) is:

root@localhost$ modprobe -r kvm_intel

MySQL Shutdown Problem During Upgrade to Ubuntu 9.10 (Karmic Koala)

Posted by Justin – November 2, 2009

Immediately following the download of new packages for the upgrade to Karmic Koala, I noticed it had hung up while attempting to turn off services which were going to be upgraded.

mysql stopping...

It happened that MySQL was the hold up, and because killing the upgrade process seemed like a risky operation – I needed to find a way to move it along.

I wasn’t able to turn up any other posts from the community about this issue – so I decided to take the usual approach: Start killing process with fingers crossed and blog the outcome!

Just so happens I nailed it on the first try:

$ ps aux | grep mysql
<snip />
root      3624  0.0  0.0   5452  1744 pts/7    S+   09:05   0:00 /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf shutdown

$ sudo kill -9 3624 #your pid would be different...

The upgrader picked up and moved right along at that point. Hurray!

How To: Export and Import of .csv data out of and into MySQL

Posted by Justin – August 4, 2009

I keep having to do this, and have just got tired of Googling every time for it. Here is the method I use for pulling a table out of MySQL then putting it back later. It may not work for versions of MySQL other than 5, but hey, it works for me. Why do this? Because it’s a great way to schlep off work to others who don’t know MySQL. Rather than having folks submit changes to me, I like to just dump it out to a .csv, have them open it in Excel, make their changes, re-save as .csv, then I can import it at my leisure. I sometimes like to throw flour on my face (like in the old rice crispies commercial) and come out of my cube saying ‘Whew, that took forever! Next time tell them it’s going to take a couple weeks…’

Exporting the MySQL data into a comma delimited (.csv) file

SELECT * INTO OUTFILE '/var/tmp/data.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' FROM tblWhatever;

This will dump your table into a file with each row from the database being on it’s own line, columns separated by commas, and if there are any spaces or funny business going on with the data in a field, the data will be enclosed in double quotes. You can make changes to this if you like by, for instance, substituting ‘,’ with ‘\t’ would result in a tab delmited file.

some things: The reason I put the file into /var/tmp is because you need to put the file in a place that MySQL can write to. Put it wherever you want though. Also, your system may not use /var/tmp but /tmp or who knows what else. If you are using shared hosting, and are working with sensitive data – don’t do anything dumb like export credit card numbers to /tmp and then leave them there ;)

Importing the comma delimited (.csv) file back into MySQL

First thing, you’ll need to clear out your old table (otherwise the keys would collide):

DELETE FROM tblWhatever;

Then you can proceed with the import:

LOAD DATA INFILE '/home/youruser/data.csv' INTO TABLE tblWhatever FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n';

some things: Naturally the file will need to be readable by MySQL, but it can be anywhere. Older versions of MySQL used ‘LOAD DATA LOCAL INFILE’ but MySQL5 uses just ‘LOAD DATA INFILE’ and only accepts local files (a pretty good thing). So yeah you’ll have to scp your file to and fro.

How to get a .csv of subscribers on all your Mailman lists

Posted by Justin – May 16, 2009

The boss asked me to create a list of everyone subscribed to every discussion list we have. Ended up being like 3000 lines…luckly there is a python script that polls the mailman admin web interface. The below script makes it quite a bit easier to perform on all the lists.

First thing, you need a file with all of your list names – this works:

ls /usr/local/mailman/lists | cat > lists.txt ## This is where mailman is located on FreeBSD

Next, download mailman-subscribers.py to your home directory.

Finally, you need to create a shell script (i.e. getallsubscribers.sh) and paste the following (substitute the [[bracketed text]] with the info for your system):

while read line
do
 echo $line | cat >> listsAndMembers.csv
  ~/mailman-subscribers.py -c [[your.mailman.host]] $line [[listAdminPassword]] | cat >> listsAndMembers.csv
  echo | cat >> listsAndMembers.csv
  echo | cat >> listsAndMembers.csv
done < lists.txt
Now make the file executable and run it:
chmod +x getallsubscribers.sh
./getallsubscribers.sh

That's it. You should have a single file with all of your lists and their subscribers.

edit: It came up later that it would be nice to get a list of who from each domain was subscribed. This regex will turn up the listname and name/email for each subscriber in the .csv

(?:.*(?:gmail\.com)"|^[^"].*$)

Friends, don’t give away your passwords!

Posted by Justin – March 4, 2009

So, despite having logged into my blog just now without doing this – I’m going to share with you the easy way that probably 90% of you can protect your passwords while using the public wifi:

Step 1:

ssh -D 9999 -C somebox.youcan.sshto

Step 2

Go get something like FoxyProxy for Firefox and MM3 for Thunderbird (especially if you are using any version of Thunderbird 3 – nothing else is compatible).

For Firefox, set up a SOCKS5 proxy (Prefs > Advanced > Network > Settings) on localhost port 9999. Start using the proxy. Stop using the proxy when you kill your ssh connection and make sure you open the connection if you are using the proxy.

For thunderbird it’s basically the same if you are using FoxyProxy – but if you are using MM3 like me, you need to create (click edit, oh btw you will probably have to add the button to the toolbar first) a proxy with this config:

[Arbitrary_name
  socks=127.0.0.1:9999
]

That’s it. Now I can’t steal your passwordsss! If you want to encrypt DNS requests you can set network.proxy.socks_remote_dns to true in about:config

Margaret Waddell and Hummingbirdhouse Music Studio

Posted by Justin – February 18, 2009

I’ve been trying to write this for months now, but it always comes out cheesy sounding and not very well said. I’ve given up. Here is my best attempt at thanking a person.

Hummingbirdhouse Music StudioSo… this may seem a little bit odd if you have been to my site before. I’m not accustomed to writing about things other than geeky-computer-how-to type stuff, but I wanted to give a shout to a person who means a lot to myself and my family.

Margaret Waddell has been a wonderful friend of our family for a little over a year now. We first met in a natural foods store where she was, if memory serves(probably doesn’t), mixing herbs for a foot soaking cocktail. Of course, my very outgoing daughter Freya caught her attention and it quickly came to light that Margaret was a music teacher in need of marketing and web services. I could tell right away that we were a good match and offered to barter my web development services for her music classes. Best decision ever.

Freya absolutely loves Margaret. Freya likes just about everyone, but with Margaret it’s different. You can see it with Margaret too. A kinder person I’ve not met, but you can see and hear it when Margaret interacts with Freya…it’s love. Not only has Freya gained a lifelong friend, but her musical skills have gone through the roof. She loves listening to and making music all day long, it’s a real joy to see and hear.

You can find the website I built for Margaret at http://hummingbirdmusic.net if you have a child, and want them to have the benefits of an early childhood music program, do give her site a visit. Margaret teaches classes for little ones with singing, hand clapping, bounces, shakers, dancing, and so on…but also does piano/keyboard classes as well as Irish Tin Whistle. Check out the class listing here: http://hummingbirdmusic.net/classes.

Thanks Margaret, we all love you! (You too RavenWolf!)

How To: Convert mbox to mailman archives using procmail

Posted by Justin – December 10, 2008

Update: NATURALLY, after doing all of this I learned that I was given the wrong information. Turns out Mailman is more than happy to take a huge mbox file as input for the arch script. I did learn that running clean_arch on the mbox first is a good idea…

mailman Are you like me? Do you get upset when you have to deal with an almost decade old problem that you had nothing to do with? Well then I’ve got a story for you…

So we’ve got these archives…

Our organization was using Lyris ListServ for about the past 10 years to handle all of our discussion list. Like most MLM‘s, ListServ does have the ability to keep list archives…but naturally we opted to not use them for all of our lists. Big mistake.

Instead of list archives we have a user on our webserver called ‘archive’. Archive is subscribed to each and every list and gets copies of all the messages. When the messages come in, Archive processes them with a procmail script and separates them into mbox mailboxes for each of the lists.

Each of the mails are then piped to a program called mhonarc which converts them into html and provides an index, etc – which can be displayed on our current(old) website. But now we’ve got a new website coming up…

Enter GNU Mailman

Me:

Thank you for coming Mailman. I’m really glad to have you because you do a really good job, not to mention you’re free and uber-powerful… One thing though… We’ve got these uh, gulp “archives”. We, uh, need to keep them and everything but you know, they’re like, not in the greatest shape. See, there actually in mbox format…

Mailman:

Oh yeah, that’s not a problem at all. I’ve got a built in script to to do that. Just take all of the monthly mailbox files for each of your lists and drop them in my folder – I’ll knock them out in no time!

Me:

SWEET! But what did you say about monthly whatevers?

Mailman:

The mailbox files that you create every month for each list… You are using your procmail script to start a new mailbox file every month aren’t you? Putting 10 years worth of emails into a single monolithic file would be retarded…

Me:

Oh yeah yeah… Of course we did that. I thought you were talking about something else. Silly me. Anyway, so uh, yeah, I’ll get those file to you real soon.

Breaking up the monolith

So clearly I needed to edit the procmail script a little bit and reprocess all the mail – but WTH?

Last month I had a help desk ticket come my way about a list which was not appearing on the website, and hadn’t been for a number of months. After digging around, I realized that someone (almost certainly me) had made a mistake in the .procmailrc file which had kept it from processing mail for that list. This was embarrasing, but I discovered how to reprocess mail with formail.

I knew I could probably reprocess all of the mail (many thousands) but had absolutely no clue how to do it. I had only this one clue from my procmail recipe:

LOGFILE=$PMDIR/list_archive-`date +%Y-%m`.log

They had written it to rotate the logs, but not the mailbox names! Uncool! But at least I had my answer – `date +%Y-%m` can get the date into the names…But wait!

When I rewrote my procmail file like so:

#this is just one of many
:0 E
     * ^Sender:.*LIST-L
     {
       :0 c
       LIST-L.`date +%Y%m` #to match mailmans archive format...
     }

Totally not working! It created only one file, and the date was this month and this year.

I’ll fast-forward for the benefit of the reader at this point and just share an insight with you: date is a *nix command and has nothing to do with procmail and cannot get any data out of the emails themselves – like dates! Yes that’s right, you can put anything you want in between those little ticks, but because the date command only returns the system date, we’ve got to do two things:

  1. Get the date the email was sent out of the email header (Magic)
  2. Process the date field to have only the 4 digit year and 2 digit month (More Magic)

It gets easier from here…

Getting the date field

The first thing we need to do is nab the Date: header from the emails. This part is fairly straight forward. Procmail uses a variable $MATCH to hold the matched string for the rule that it’s matching on. We can use this to hold our Date header and then just pipe it to a script for processing.

Here’s the recipe magic!

# NOTE: I later found that this rule only seems to work
# when used as an ELSE rule (:0 E). I'm not sure why, but
# it was only matching the 'Date:' portion, and not the entire
# line. If you can help me understand why, please leave a comment.
# ANOTHER NOTE: The ticks in `echo $MATCH.... are ticks(the
# un-shifted tilde) and not single quotes.

:0 E
     * ^Sender:.*LIST-L
     {
       :0 c
       * ^\/Date:.*
       LIST-L.`echo $MATCH | php /path/to/dateconvert.php`
     }

Transforming the date into Mailman’s monthly format

Here’s the php code more magic to get your date cut down and switched around to a format Mailman will love. It uses php’s built in functions fairy dust to put the Date header into an array, and drop the empty elements. It was also necessary to create an array that maps the three letter month names used in the email header to their numerical equivalents. 

< ?php
 $date = trim(fgets(STDIN));
 
 $datearray = array_values(array_filter(explode(" ", $date)));
 
 $month = $datearray[3];
 $year = $datearray[4];
 
 $montharray = array(
 "Jan" => "01",
 "Feb" => "02",
 "Mar" => "03",
 "Apr" => "04",
 "May" => "05",
 "Jun" => "06",
 "Jul" => "07",
 "Aug" => "08",
 "Sep" => "09",
 "Oct" => "10",
 "Nov" => "11",
 "Dec" => "12"
 );
 
 echo $year . $montharray[$datearray[3]];
?>

At this point you’re probably thinking one of two things:

  1. “ZOMG you’re such a hack. You could have done that with so much less code! You have no style!”
  2. “ZOMG you can totally do that with a sed/awk one liner!”

Sorry for wasting everyone’s time yet again.

After you’ve got the recipe in place and the php file all ready to go, just give it one of these:

formail -s procmail -m /path/to/yourprocmailfile < /path/to/LIST-L

Final product

Anyway, you should, after a few hours or days end up with a grip of mailbox files like this:

               LIST-L.200306  LIST-L.200502  LIST-L.200701
 	        LIST-L.200307  LIST-L.200503  LIST-L.200702
LIST-L.200112  LIST-L.200308  LIST-L.200504  LIST-L.200703
LIST-L.200201  LIST-L.200309  LIST-L.200505  LIST-L.200704
LIST-L.200202  LIST-L.200310  LIST-L.200506  LIST-L.200705
LIST-L.200203  LIST-L.200311  LIST-L.200507  LIST-L.200707
LIST-L.200204  LIST-L.200312  LIST-L.200509  LIST-L.200708
LIST-L.200205  LIST-L.200401  LIST-L.200511  LIST-L.200709
LIST-L.200206  LIST-L.200402  LIST-L.200512  LIST-L.200710
LIST-L.200207  LIST-L.200403  LIST-L.200601  LIST-L.200711
LIST-L.200208  LIST-L.200404  LIST-L.200602  LIST-L.200712
LIST-L.200209  LIST-L.200405  LIST-L.200603  LIST-L.200801
LIST-L.200210  LIST-L.200406  LIST-L.200604  LIST-L.200802
LIST-L.200211  LIST-L.200407  LIST-L.200605  LIST-L.200803
LIST-L.200212  LIST-L.200408  LIST-L.200606  LIST-L.200804
LIST-L.200301  LIST-L.200409  LIST-L.200607  LIST-L.200805
LIST-L.200302  LIST-L.200410  LIST-L.200608  LIST-L.200806
LIST-L.200303  LIST-L.200411  LIST-L.200610  LIST-L.200807
LIST-L.200304  LIST-L.200412  LIST-L.200611
LIST-L.200305  LIST-L.200501  LIST-L.200612

So there ya go. Now make with the comments.