On Packaging Python

I’d love if there was a language-neutral library installation and management tool that really worked. Linux system package managers are absolutely not that tool; frankly it is absurd to even consider them as an alternative. So for now we do our best in our respective language communities. If we’re going to move forward, we’ll have to acknowledge what’s come before, and the reasoning for it.
A Few Corrections To “On Packaging” - Ian Bicking

I disagree with him so much I don't know where to start. Operating system packages are the best way to install anything on nice systems. Python is not 'special' in this regard, there's no reason not to use operating system packages.

I spent a couple hours a few months ago making it easier, and I should return to that project and get more done on it, at the moment it's far to pypi-centric.

Oh, and if someone could fix Python Issue 1533164 for Fedora 10, that'd be great too. :)


Mairix and Mutt for Mail

After a long time of using webmail and things, I've finally gotten around to hosting my mail locally on my computer.

Keeping with the 'search not sort' philosophy of mail, I have set up a system where when I get email (using 'getmail' to download it via pop from my email hosting) I deliver the email to two places. One is an archive (named for the day it was received on) and the other is an Inbox.

When I read email in the inbox, I delete it. The archive is always there.

I have a mairixrc configured to index all my archive. Unfortunately the globbing for mairix doesn't seem to work for subdirectories, so I have all my archives in yyyymmdd named folders instead of yyyy/mm/dd folders. This way I can say maildir=Maildir/Archive/*.

Now that I have my email sorted like this, and mairix is indexing it (on a cronjob), I can search it. I wrote a little script to search my email. The idea is that I can select some text and click a button on my toolbar, or I can hit a button my toolbar and enter some text into a search prompt, then mutt will spawn pointed at a folder containing my search results.

Here is the one that prompts for input:

selection=$(Xdialog --inputbox "Search For?" 18 45 2>&1)
# exit if Xdialog had a nonzero exit.
[ $? -ne 0 ] && exit "Okay, nevermind"

echo Searching for $selection
mkdir -p $searchdir
mairix -o $searchdir "$selection"
mutt -f $searchdir
rm -rf $searchdir

The one that uses the xclipboard just does selection=$(xclip -o) instead of the Xdialog dance.


Xrandr for dual head on IBM X200s

I tried to use xrandr on my new laptop (IBM X200s) and I had an error.
$ xrandr --output VGA --left-of LVDS
xrandr: screen cannot be larger than 1280x1280 (desired size 2960x1024)
It turns out that Fedora 10 has an X that doesn't need an xorg.conf, and therefore doesn't have one, so it's hard to put in the fix for this, which is putting in a Virtual line in the Screen.

This is what I had to put in xorg.conf to allow me to do dual head. Bear in mind that if you do this, then DRI will not work, and thus there will be no 3D acceleration.
Section "Device"
Identifier "Videocard0"
Driver "intel"

Section "Screen"
Identifier "Screen0"
Device "Videocard0"
SubSection "Display"
Virtual 2960 1050
And that was all that was needed. I can now use my external monitor via xrandr.


$15.24 for 2 zones

$15.24 for 2 zones
Originally uploaded by Stephen Thorne
Go-card is fail. A workmate of mine just got charged 6.5x the correct fare for his trip into work yesterday.

Total. Utter. Fail.


Booting and Stuff

So it turns out that my epic journey into "why doesn't this board boot" this week had a really anti-climatic ending.

Just so everyone knows, the Intel DG945GCLF board, which is a mini-itx board with 1.6ghz atom 230 chip on it, will not boot from a hdd unless one of the partitions on that drive is marked bootable.

This might seem obvious, but it doesn't make much sense :( the BIOS is supposed to just load the first 512 bytes of the drive and run it*.

So, things I learned this week:
  • grub-install is a shell script.
  • What grub stage1, e2fs_stage1_5 and stage2 are.
  • The layout of the partition table.
  • Various tricks with 'dd' and 'xxd' to read partition tables.
  • Editing hex values in partition tables and the grub stage1 on the disk is kinda fun the first time.
  • Knowing what bytes are what on a raw disk is knowledge that can't be erased with beer.
  • Beer is good for day 3 of dealing with non-booting systems.
So after something like 15 kernel compiles, various operating systems and rescue disks, it turns out that I needed to make a 2 character change to the sfdisk script that created our partition layout.

* yes, i know this is a lie.


SSH Fail

I'm very very angry just at the moment, I'm angry about a total mismatch of expectations in the name of 'usability' that has invalidated what I thought was a totally reasonable security mechanism that I more or less took for granted.

I have an SSH key. ~/.ssh/id_dsa. I have a passphrase on that key, so that if someone compromises my machine, all they have is a key, and they have to brute force my (rather long) passphrase. I have ssh-agent to remember my passphrase in memory so that I don't ever let that passphrase hit disk, but have the passphrase cached.

Running ssh-add -D should wipe that passphrase from memory so I have to type it again. I was toying with doing this nightly so that it would wipe the passphrase every night so when I log in the next morning my passphrase needs to be re-entered.

This is where the nightmare begins. On the weekend my machine crashed so I had to start a fresh this morning. I sit down, log in, fire up a terminal, and ssh into another host.

Bam, I'm straight in. No passphrase, no prompt, nothing. Just straight in. This shouldn't be possible. Either the passphrase has been removed from my key so that it can be used without a passphrase, or something is saving it to disk without my knowledge.

Fedora 9 in its default configuration will save your passphrase to disk if you're logged in under gnome. I don't know how to turn it off. I feel angry, violated, annoyed and really really frustrated. It was a simple thing and it's been fucked. I can't turn it off, I can't stop gnome from remembering my passphrase, I feel like pulling the drive, hitting it with a hammer, and going back to OSX instead.

At least OSX can get simple things like ssh-agent right.



Cafe Babe

Q: What's the difference between Old Java and New Open Source Java?

A: The Old Java worked in my browser dammit.


Firefox SSL Nags

Dear Lazyweb,

How do I get back the firefox 2 behaviour of allowing me to access SSL websites with 1 or 0 button clicks, instead of the multiple screen nagpages?



Trace that route

$ traceroute6 mirror.ipv6.internode.on.net
traceroute6 to mirror.ipv6.internode.on.net (2001:44b8:8020:7a80::20) from 2001:44b8:61::e3, 30 hops max, 12 byte packets
1 2001:44b8:61::e2 87.049 ms 89.126 ms 125.079 ms
2 vl67.cor1.adl6.internode.on.net 98.27 ms 77.689 ms 77.303 ms
3 mirror.internode.on.net 109.942 ms 177.193 ms 148.519 ms

Huzzah. Internode is now running a tunnel broker. :)


Gnnnr, Telstra.

This is a transcript of a conversation I had with Telstra today, this is verbatim from the 'chat' interface I used.

Jessica: Welcome to online assistance. You are chatting with Jessica NB:This Chat is being recorded and may be monitored for coaching purposes. If you do not wish to proceed, please close this window.
Jessica: The rates discussed in this Livechat are correct as at today's date, but may vary from time to time. Any variation will be done in accordance with the General Terms of Our Customer Terms which can be found by visiting http://www.telstra.com.au/customerterms/index.htm

Jessica: Hi, how may I help you with your enquiry today?
you: G'day.
you: I am moving house, and would like to know how much moving my phone service will cost, and if the phone line has been cancelled by the old tenants.
you: The address we are moving to is [REDACTED]
you: And our current phone number is [REDACTED]
Jessica: Ok Sure I can check this for you, one moment please
Jessica: Ok the phone line at [REDACTED] is still active the the old tenants name, So the connection fee will be only $59
you: Okay, have they notified you that they are going to disconnect?
you: And how much is the line rental that I am paying at the moment.
Jessica: I can not see if they have organised disconnection sorry.
Jessica: I will need to ask a few questions to access your account,
you: Sure.
Jessica: Can I please have your full name and Dob for the security of you account thanks,.
you: Stephen Thorne, [REDACTED]
Jessica: Thank you your currently on Homeline Budget 19.95 per month
you: oh, okay, and I can keep that?
you: (i.e. it won't go up when I move premesis?)
Jessica: yes you can,
Jessica: Oh sorry hang on,
you: are there any problems with me using that and ADSL2+?
Jessica: we have recently changed the business rules when using Homeline Budget in conjunction with ADSL from another provider. As you are disconnecting and re connecting at the new address these business rules would apply:
Jessica: HomeLine Budget customers must not acquire a broadband service from another service provider which is provided using Line Sharing Technology. If your Internet Service Provider uses Line Sharing Technology, you will be contacted by Telstra and you will need to choose a different HomeLine Plan other than HomeLine Budget. You will need to contact your current Internet Service Provider to confirm if they utilize line sharing equipment.
you: I see.
Jessica: I would connect to budget then if we find your ISP is using Line sharring we will automatically move you up to the next plan up which is Homeline Complete for 26.95 per month
you: I see.
Jessica: there is no charge to move plan
you: Thankyou for your time.
you: Can I have a copy of this transscript emailed to me?
Jessica: unfortunately we can not email this chat, however you should be able to copy and paste this into word, or notepad and save it
you: Thankyou
Jessica: Your welcome,
Jessica: Is there anything else I can assist you with today?
you: No, that will be all.
Jessica: Thank You for using Telstra.com. We look forward to assisting you in the future. Bye
Jessica Has Disconnected

I'm not impressed. $7 extra a month in order to connect ADSL2+ at the new place. I pay fees to my ISP that they pay to telstra, then telstra takes extra dollars from me as well.


High Availability Terminal Emulator

So I've got some software that glyph wrote. It's called the "High Availability Terminal Emulator". He put a link up to it a long while ago, and it's probably still around.

What it is, is a replacement for xterm/aterm/gnome-terminal/konsole etc. It is literally some python glue to make a window containing a libvte terminal, and it's written in python.

I have it in my '+junk' thingy on launchpad. It is accessable on code.launchpad.net. If you have bzr, you can do bzr branch lp:~jerub/+junk/hate.

There are many good reasons for writing this software. Being able to add features to terminal emulators is hard, writing python code is easy. I think it would be awesome to have a terminal emulator that did some advanced things like "survive X crashing", and have a key combo that says, "bring me a terminal that has nothing running in it and hasn't been used in a long while".

hate.py is supplied without warranty, etc etc. I also have no idea what license it's under.

If I have more contributors than just me and Glyph, I might even turn it into a 'real' project :)


Advanced Grub Notes

I have been doing a lot of hacking with grub and linux bootloading this week, and I wanted to make some notes about some of the more esoteric things in grub, that I feel are rather poorly documented.

First of all, I had a hell of a time getting the 'savedefault' behaviour in grub working. It should work like this when you run grub on the linux commandline:
grub> savedefault --default=2 --once
This means, "next time you boot, use the 2nd boot option, but only once". It does this by recording some information inside /boot/grub/stage_2 or something.

I simply could not get this behaviour to work, and found that this was a 'better' way of doing things introduced by a patch. "grub-0.97-once.patch" is the name. I backed out this patch and rebuilt, which reverted to an older behaviour for saving default state.

The use-case for us is an unattended bios update. Set the system to boot the bios installer just once, so it will reboot, install the bios, reboot, boot linux again.

Now I use this grub config (in menu.lst) to make grub boot an alternate bootloader, just once:
default saved

title Linux
kernel /boot/vmlinuz ...

title Alternate
kernel /boot/memdisk
savedefault 0
State is now no longer saved in some binary format, it is saved in /boot/grub/default. Which is a text file, and on the first line is a number which is the default option to boot. Zero indexed. The format of this file is not documented. There is a standard tool for setting the contents of this file which the patch above removed, called grub-set-default. If you have this executable, you probably don't have the patch applied to your version.

Usage of grub-set-default is trivial:
$ grub-set-default 0

Now the next thing I had trouble with was labels. I have a seriously large amount of machines that I maintain and create. One of the scripts that I have has the duty of figuring out what devices (/dev/sda1, etc) the partitions are on, and writing the fstab and grub configurations correctly.

So by using labels, I should be able to avoid having these scripts be dynamic.

First, labelling unlabelled drives, both ext3 and swap:
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 1.5G 552M 856M 40% /
# cat /proc/swaps
Filename Type Size Used Priority
/dev/sda5 partition 136544 0 -1
# tune2fs -L / /dev/sda1
# swapoff /dev/sda5
# mkswap -L SWAP /dev/sda5
# swapon LABEL=SWAP
Then, in your grub configuration, you need to make your kernel line look like this:
kernel /boot/vmlinuz root=LABEL=/
initrd /boot/initrd.img
The initrd.img is absolutely essental. Without it you will not be able to boot, and you will get the message "VFS: Cannot open root device "LABEL=/" or unknown-block(0,0)". We didn't have an initrd for our kernel, and we were compelled to make one in order to get labels to work.

The last thing is the pesky grub way of saying /dev/sda1. In grubspeak you say (hd0,0). This is supposed to be a magic way of saying "the first harddrive". So you would expect that when you have a recent motherboard where you have a PATA controller that performs better when you add the magic kernel line "hda=none" to make the drives appear as /dev/sda instead of /dev/hda (and incidentally have 8 times the throughput) that grub would automatically figure that out.

But that's wrong. There is a file in /boot/grub/device.map that is specifically placed there to map from hd0 to /dev/hda. And that will screw things around.

The last thing I had to do when setting up grub in my now nearly-totally-static configuration, with a static fstab and static grub configuration, is setup grub like this:
rm /boot/grub/device.map
grub-install --no-floppy
rm /boot/grub/device.map
Running grub-install will automatically detect the devices and put them in those files. By removing the file you can cause grub to automatically detect everything at boot time.

If anyone has easier ways of doing any of the above, or reasons why any of that is totally insane, I'm interested.