Author:

JustinI'm married. I like the internet, so much so I write code for it. I'm a Linux FanBoy.

Set Mailman’s Acknowledge Posts (ack) Option From The Command Line

Posted by Justin – September 13, 2010

Now that I’m managing our companies Mailman install my blog seems to be overrun…

So, as much of the internet has already noticed, Google/Gmail “helps” users by automatically archiving users own posts to discussion lists when they are received. This is stupid for a number of reasons, which I won’t go in to, but suffice to say that there is no way to convince Gmail of said stupidity.

While puzzling the situation it occurred to me that, at least for my group of people, the receipt of their own postings didn’t have anything to do with the message itself. The real reason we wanted to get our own posts was that we wanted an acknowledgement of our post letting us know everything went ok and that we could expect others to receive the message. Mailman will happily send you a “Your message entitled … was successfully received…” if you set AcknowledgePosts(ack) in your membership options. Because this email looks nothing like the original message you sent, Gmail won’t try to help you out! Magic!

Naturally the next question, as the title implies, is how to do this from the command line. If you’re like me, using Google Apps for your company, this is a problem for all of your users and one that you’d like to fix for everyone in one go. Mark Sapiro has written many helpful mailman scripts that can accomplish this type of task (http://www.msapiro.net/scripts/), but unfortunately this specific option didn’t get any love.

The following script was shamelessly copied/pasted/find/replaceded from the set_mod.py script found at the above url. Usage is described in the comments.




For one reason or another the –all flag of withlist doesn’t work here. Rather than figure it out, just use xargs…



[[code]]czo4OTpcIg0KbHMgL3Zhci9saWIvbWFpbG1hbi9saXN0cy8gfCB4YXJncyAtSSB7fSAuL3dpdGhsaXN0IC1yIHNldF9hY2sge30gLXN7WyYqJl19IC1kIFwnZXhhbXBsZS5jb21cJw0KXCI7e1smKiZdfQ==[[/code]]

			

				
		


	
		

				
Fixing mailman archive urls


Posted by Justin – August 13, 2010
				

			
	
			

				
I recently (tonight) migrated a number of mailman discussion lists from one server to another. To make things as complicated as possible, I changed OSes and domain names.



Moving the lists themselves wasn’t terribly bad, I just followed this post on debian-adminstration.org: http://www.debian-administration.org/article/Migrating_mailman_lists



That went fine up until I got down to 4.2, where I realized “Man, I’ve got way too many lists to do this by hand.” No biggie, I just modified the command to use xargs:




<pre>
# cd /usr/lib/mailman/bin
# ls /var/lib/mailman/lists | xargs -I listname ./withlist -l -r fix_url listname -u new-domain.example.org
</pre>




This worked quite well to fix the links on all of Mailmans internal html pages… but it didn’t do a thing for the archives. Many of the archived messages had links to attachments at the old domain name. So yeah:




<pre>
# cd /var/lib/mailman/archives/private
# find -type f -name *.html | xargs -I file perl -pi -w -e 's/olddomain\.example\.org/newdomain\.example\.org/g;' file
</pre>



Depending on the size and number of lists you have, this will take somewhere between a long time and a longer time.
			

				
		


	
		

				
You know what rocks? This fetchmail init script for Red Hat!


Posted by Justin – June 23, 2010
				

			
	
			

				
This is from the contrib directory of the fetchmail source and is much better than what Google gives you.



			

				
		


	
		

				
My Favorite Way To Delete Blank Lines WIth VIM


Posted by Justin – April 12, 2010
				

			
	
			

				
[[code]]czoxMTpcIg0KOmcvXiQvZA0KXCI7e1smKiZdfQ==[[/code]]

			

				
		


	
		

				
Vector Outline of the State of Missouri


Posted by Justin – February 3, 2010
				

			
	
			

				
I think this is the 3rd year in a row that I’ve set out looking for an outline (vector format) of the state of Missouri. Each year I end up downloading one image or another from some Missouri state government website and creating a pen path around it in Photoshop or Illustrator.



You’d think that I would have saved the resulting outline one of these times, and well – I guess this is that time. I’m sure I have it somewhere on a network drive, but you know how that goes…



So I’m going to post it here: an Illustrator file with vectors for the Missouri outline and a Google Map underlay for locating cities and such, a .psd with path (check the paths palette), an SVG, an EPS, and a JPEG.



Missouri Outline Vector Files (.zip)




  

    CC0

  

  To the extent possible under law, Justin Hopkins has waived all copyright and related or neighboring rights to Outline of the State of Missouri. This work is published from United States.
			

				
		


	
		

				
Installing VirtualBox 3.0.12 on Fedora 12


Posted by Justin – November 29, 2009
				

			
	
			

				
If you want to get VirtualBox running on Fedora 12, it’s really not that hard. Please refer to the VirtualBox website for more up to date instructions.



FIrst thing you need to do is install dkms, and kernel-devel.




If you don’t do the above step, and are running a vanilla Fedora 12 the later step of installing VirtualBox will not be able to finish and complain about 


“No precompiled module for this kernel found — trying to build one. Messages
emitted during module compilation will be logged to /var/log/vbox-install.log.

Compilation of the kernel module FAILED! VirtualBox will not start until this

problem is fixed. Please consult /var/log/vbox-install.log to find out why the

kernel module does not compile. Most probably the kernel sources are not found.

Install them and execute”



Now we can move on to adding the VirtualBox repo:



[[code]]czo5NDpcIg0Kd2dldCAtcSBodHRwOi8vZG93bmxvYWQudmlydHVhbGJveC5vcmcvdmlydHVhbGJveC9kZWJpYW4vc3VuX3Zib3guYXN7WyYqJl19YyAtTy0gfCBycG0gLS1pbXBvcnQgLQ0KXCI7e1smKiZdfQ==[[/code]]



Now you are good to go…



[[code]]czoxMjA6XCINCnJvb3RAbG9jYWxob3N0JCB5dW0gaW5zdGFsbCBWaXJ0dWFsQm94DQoNCiMgVGhlcmUgaXMgYSBjb21waWxlIHN0ZXB7WyYqJl19IGluIHRoaXMgaW5zdGFsbGVyLCBhbmQgbWFrZSB0YWtlIGEgZmV3IG1pbnV0ZXMuIw0KXCI7e1smKiZdfQ==[[/code]]



If you happened to have tried installing VirtualBox and ran into the above “Compilation of the kernel module FAILED!” message, just install dkms then ‘/etc/init.d/vboxdrv setup’ (Compiling will take some time probably).



KVM issues

Apparently the KVM kernel modules(which you would have if you were using the Fedora Virtual Manager) aren’t compatible with VirtualBox. There is a bug filed but as a workaround (or possibly just the easiest solution) is:



[[code]]czozNzpcInJvb3RAbG9jYWxob3N0JCBtb2Rwcm9iZSAtciBrdm1faW50ZWxcIjt7WyYqJl19[[/code]]
			

				
		


	
		

				
MySQL Shutdown Problem During Upgrade to Ubuntu 9.10 (Karmic Koala)


Posted by Justin – November 2, 2009
				

			
	
			

				
Immediately following the download of new packages for the upgrade to Karmic Koala, I noticed it had hung up while attempting to turn off services which were going to be upgraded.




<pre>
mysql stopping...
</pre>




It happened that MySQL was the hold up, and because killing the upgrade process seemed like a risky operation – I needed to find a way to move it along.



I wasn’t able to turn up any other posts from the community about this issue – so I decided to take the usual approach: Start killing process with fingers crossed and blog the outcome!



Just so happens I nailed it on the first try:



<pre>
$ ps aux | grep mysql
<snip />
root      3624  0.0  0.0   5452  1744 pts/7    S+   09:05   0:00 /usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf shutdown

$ sudo kill -9 3624 #your pid would be different...
</pre>




The upgrader picked up and moved right along at that point. Hurray!
			

				
		


	
		

				
How To: Export and Import of .csv data out of and into MySQL


Posted by Justin – August 4, 2009
				

			
	
			

				
I keep having to do this, and have just got tired of Googling every time for it. Here is the method I use for pulling a table out of MySQL then putting it back later. It may not work for versions of MySQL other than 5, but hey, it works for me. Why do this? Because it’s a great way to schlep off work to others who don’t know MySQL. Rather than having folks submit changes to me, I like to just dump it out to a .csv, have them open it in Excel, make their changes, re-save as .csv, then I can import it at my leisure. I sometimes like to throw flour on my face (like in the old rice crispies commercial) and come out of my cube saying ‘Whew, that took forever! Next time tell them it’s going to take a couple weeks…’



Exporting the MySQL data into a comma delimited (.csv) file



SELECT * INTO OUTFILE '/var/tmp/data.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' FROM tblWhatever;



This will dump your table into a file with each row from the database being on it’s own line, columns separated by commas, and if there are any spaces or funny business going on with the data in a field, the data will be enclosed in double quotes. You can make changes to this if you like by, for instance, substituting ‘,’ with ‘\t’ would result in a tab delmited file.



some things: The reason I put the file into /var/tmp is because you need to put the file in a place that MySQL can write to. Put it wherever you want though. Also, your system may not use /var/tmp but /tmp or who knows what else. If you are using shared hosting, and are working with sensitive data – don’t do anything dumb like export credit card numbers to /tmp and then leave them there ;)



Importing the comma delimited (.csv) file back into MySQL



First thing, you’ll need to clear out your old table (otherwise the keys would collide):



DELETE FROM tblWhatever;



Then you can proceed with the import:



LOAD DATA INFILE '/home/youruser/data.csv' INTO TABLE tblWhatever FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n';



some things: Naturally the file will need to be readable by MySQL, but it can be anywhere. Older versions of MySQL used ‘LOAD DATA LOCAL INFILE’ but MySQL5 uses just ‘LOAD DATA INFILE’ and only accepts local files (a pretty good thing). So yeah you’ll have to scp your file to and fro.
			

				
		


	
		

				
How to get a .csv of subscribers on all your Mailman lists


Posted by Justin – May 16, 2009
				

			
	
			

				
The boss asked me to create a list of everyone subscribed to every discussion list we have. Ended up being like 3000 lines…luckly there is a python script that polls the mailman admin web interface. The below script makes it quite a bit easier to perform on all the lists.



First thing, you need a file with all of your list names – this works:


ls /usr/local/mailman/lists | cat > lists.txt ## This is where mailman is located on FreeBSD




Next, download mailman-subscribers.py to your home directory.



Finally, you need to create a shell script (i.e. getallsubscribers.sh) and paste the following (substitute the [[bracketed text]] with the info for your system):


while read line
do
 echo $line | cat >> listsAndMembers.csv
  ~/mailman-subscribers.py -c [[your.mailman.host]] $line [[listAdminPassword]] | cat >> listsAndMembers.csv
  echo | cat >> listsAndMembers.csv
  echo | cat >> listsAndMembers.csv
done < lists.txt



Now make the file executable and run it:


chmod +x getallsubscribers.sh
./getallsubscribers.sh




That's it. You should have a single file with all of your lists and their subscribers.



edit: It came up later that it would be nice to get a list of who from each domain was subscribed. This regex will turn up the listname and name/email for each subscriber in the .csv


(?:.*(?:gmail\.com)"|^[^"].*$)

			

				
		


	
		

				
Friends, don’t give away your passwords!


Posted by Justin – March 4, 2009
				

			
	
			

				
So, despite having logged into my blog just now without doing this – I’m going to share with you the easy way that probably 90% of you can protect your passwords while using the public wifi:



Step 1:




<pre>
ssh -D 9999 -C somebox.youcan.sshto
</pre>




Step 2



Go get something like FoxyProxy for Firefox and MM3 for Thunderbird (especially if you are using any version of Thunderbird 3 – nothing else is compatible). 



For Firefox, set up a SOCKS5 proxy (Prefs > Advanced > Network > Settings) on localhost port 9999. Start using the proxy. Stop using the proxy when you kill your ssh connection and make sure you open the connection if you are using the proxy.



For thunderbird it’s basically the same if you are using FoxyProxy – but if you are using MM3 like me, you need to create (click edit, oh btw you will probably have to add the button to the toolbar first) a proxy with this config:




<pre>
[Arbitrary_name
  socks=127.0.0.1:9999
]
</pre>




That’s it. Now I can’t steal your passwordsss! If you want to encrypt DNS requests you can set network.proxy.socks_remote_dns to true in about:config