At Europython I learned...

Today I learned something magical that has transformed all my twisted
program examples:

from __future__ import print_function

Whee! This means that you can quickly and easily do
d.addCallback(print) without any pain. You can even do
d.addCallback(print, 'somedata', 1, 'etc').

>>> d = defer.succeed(None)
>>> d.addCallback(print, 'somedata', 1, 'etc')
None somedata 1 etc



Explaining For/Else In Python

In response to a thread on G+ I've decided to write a this article on 'how for/else works'.

I should start by saying that this isn't intuitive or simple. It's quite complicated, but it has a very clear use case. Let me start by stating as simply as I can the use-case.

for element in seq:
    if somecondition(element):

I think that example is reasonably clear, but it's important to explain what's happening here.

If a 'for' loop does not break, it's else is run, otherwise the else is not run. So:

  • The else suite is run if:
    • the sequence is empty
    • break is never executed
  • The else suite is bypassed if:
    • break is executed
    • a return statement in the loop is executed
    • an exception raises in the for loop

I hope that's clear. Any comments or addendums or errors I've made, please let me know in the comments.

Edit: I don't think using an 'else:' on a for loop is a bad idea. But I think it should only be used in exactly the idiom presented in the example, anything with more complicated flow control or more than 8 lines should be re-factored to use a function call with 'return' in the appropriate places.


Gmail Keyboard Shortcuts


I use Gmail a lot now. I also have come to know and love a few keyboard shortcuts. I can't imagine dealing with a large volume of email without them.

You have to turn them on under the mail settings first.

u will refresh the current mail search or your inbox if you're looking at your inbox.
u will return you to your mail search, label or inbox if you're looking at an email.
j/k to navigate between emails in your inbox view.
x selects an email, and dumps you back at the message list if you were in an email.
s applies a star to selected emails, or stars the email you're reading.
+/- will flag/unflag an email as important (if you're using priority inbox)
e archives selected emails, or archive an email you're reading.
replies to an email.
m will mute selected email threads so it won't reappear in your inbox when more posts are made.
gl lets you type the name of a label you want to view.
? brings up the help.

Most important button to me is u, as it refreshes my inbox when my android phone has chimed to let me know that I've got an important email, and the gmail UI hasn't shown it yet. The androids. They always know.

(Photo from Geek Sugar)


Appengine Development

I've been quite enjoying writing some code for Google Appengine this week. Just a small project for querying a variety of book retailer websites for information.

I'm really impressed with it so far. Having the ability to simply import and use things like memcache and use it is excellent. It works locally with some kind of cache, and then when I deploy my app it uses a memcache on the server side seamlessly.

I'm also liking that the urlfetch infrastructure built into appengine supports async downloading of webpages, so i can fire off 5+ API queries simultaneously and then collect the results, instead of doing them in sequence.

Here's a random bit of code I threw together that lets me cache the results of doing API queries in memcache so that a user doing reloads won't cause more API requests to occur than are strictly necessary:

from google.appengine.api import urlfetch
from google.appengine.api import memcache

CACHE_TIMEOUT = 3600 # 1 hour

class APIError(Exception):

def cache_result(rpc, url):
    result = rpc.get_result()
    if result.status_code == 200:
        data = result.content
        memcache.add(url, data, CACHE_TIMEOUT)

def apiquery(url):
    data = memcache.get(url)
    if data is not None:
        decoded_data = json.loads(data)
        return lambda:decoded_data

    logging.debug("Cache miss for %r", url)

    rpc = urlfetch.create_rpc()
    rpc.callback = lambda: cache_result(rpc, url)
    urlfetch.make_fetch_call(rpc, url, method=urlfetch.GET)

    def answer():
        result = rpc.get_result()
        if result.status_code == 200:
            data = result.content
            return json.loads(data)
        raise APIError, "Broken!"
    return answer

With the above code I can now do something like:

queries = [apiquery(url) for url in generate_api_queries()]
for query in queries:
    result = query()
    ... # do stuff with result

I don't really like the variable names I've used, but it's an interesting enough bunch of code linking together a few bits and pieces so I thought I'd throw it on my blog.


Cat Pictures


Now that's one statically charged cat. I wonder if someone was
taunting him with catnip or tuna at the top of the slide? At least
he's getting some exercise.


Favicon Finding

I was updating some of my blog settings, and noticed in blogspot that I can customise the favicon that it will use. I thought, "I know, I'll search for a favicon."

Wow, that was a mistake. The entire area of icons, and web icons, is SEO'd to hell (Asking me to spend £0.06 on an icon screams 'credit  card theft' to me, not 'bargain'), and once I found a source of  favicons I was immediately hit with that other problem - too many choices.

I guess I'll just leave the basic blogspot icon on http://shiny.thorne.id.au then.


Twisted Python Plugin Example

At EuroPython 2011 there was some very successful and popular training given on Twisted, presented
by Orestis Markou. The room was so packed the first time he gave the training on the Monday of the
conference, he was asked to do it a second time on the Wednesday.

Orestis asked me to put together a few things about best practice for deploying Twisted code. I have deployed many twisted servers so I was more than happy to oblige.

I put on bitbucket my twisted-plugin-example that I quickly composed for an example.

It shows the following:

  • The nice way to write a twisted/plugin/foo.py file.
  • How to use twistd to start your server instead of using a script containing reactor.run()
  • serverFromString + StreamServerEndpointService (i.e. string endpoint descriptions) to specify where to listen.
  • twisted.python.usage.Options
  • How to write a simple custom Service to do any additional setup required.

I think it's an acceptable starting point if someone has a Protocol/Factory and wants to start the transition from "I've experimented with this and it works" to "This is a body of code I would feel okay packaging with distutils and deploying in a VirtualEnv on a hundred servers"

If you have any comments, patches are accepted of course. :)


Doesn't Anyone Think This Is a Problem?

I can trivially get root on a machine if the user is member of the 'mock' group.

$ /usr/bin/mock -r epel-5-i386 --copyin /bin/sh /
$ /usr/bin/mock -r epel-5-i386 --shell 'chmod 4755 /sh'
$ /var/lib/mock/epel-5-i386/root/sh

In order to use mock properly, you have to be in the mock group. It just doesn't work properly even with sudo unless you're in that group. Once you're in that group, you can elevate privs in the chroot it creates and run arbitary scripts.

I did a quick google and I can't even find anything with keywords like "mock is a gigantic security hole because mock has lots of suid root stuff in it that lets you trivially root a machine if you've ever gotten it to work properly in the past."

It doesn't make any sense to me. Talking to two admins today led me to conclude that they both knew about this problem, and thought it wasn't an issue.


Broken Software is Broken

I have very low expectations of open source software, and I don't feel put out when I have to fix it. But when I use close source software I expect it to 'just work' and get frustrated when it doesn't.

sthorne@pearl~/w/netbox> dropbox stop
Dropbox isn't running!
sthorne@pearl~/w/netbox> dropbox start
Dropbox isn't running!
Dropbox is already running!
sthorne@pearl~/w/netbox> killall dropbox
sthorne@pearl~/w/netbox> dropbox start
Starting Dropbox...Done!

I really shouldn't complain too loudly. As I'm in the process of backing up all my email I've ever received to it.


Simple Twisted RPM Spec File

I needed to build twisted for RHEL5 (Really, centos5, but who cares, they’re the same thing with different lawyers).
This is a really simple spec file that lets me build the dratted thing with a minimum of fuss. Twisted tarball is the one from http://twistedmatrix.com/trac/wiki/Downloads.
%define name python-twisted
%define version 10.1.0
%define release 1
%{!?python_sitelib: %define python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")}
Summary: Event-based framework for internet applications
Name: %{name}
Version: %{version}
Release: %{release}
Source0: Twisted-%{version}.tar.bz2
License: MIT
Group: Development/Libraries
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot
Prefix: %{_prefix}
BuildArch: i386
Vendor: Your Name
Url: http://twistedmatrix.com/
BuildRequires: python-devel
Requires: python
Requires: python-zope-interface
Obsoletes: python-twisted-core
See summary.
%setup -q -c
cd Twisted-10.1.0
%{__python} setup.py build
cd Twisted-10.1.0
%{__python} setup.py install --root=$RPM_BUILD_ROOT