December 23, 2020 @ 08:30

📉 Monitoring is for running and understanding other people's code (aka "your infrastructure")

📈 Observability is for running and understanding your code -- the code you write, change and ship every day; the code that solves your core business problems.

That's 2021 for me right there. How about you?

(Quote from Charity Majors)

R.I.P. instagram

December 17, 2020 @ 23:22

Right so regarding my previous post - I looked at a few options and I've decided that I'm just not going to play this game, and so I pulled down a bunch of the Instagram posts manually and hacked this up a bunch. At least it won't go away this way, but doing this manually isn't ideal.

I guess I should consider what to do with that. Maybe there's some other reasonable thing I could cross-post to or even just do an instagram-style post via my phone. Maybe I should build an app for that.

Isn't the web such a better place now that we all need Facebook accounts? Grr.

related, related, related.


December 11, 2020 @ 12:15

Well I guess Instagram finally shut down the API I was using to pull my posts for the sidebar on right of this website.

So my choices are now:

  1. Jump through ridiculous hoops getting a Facebook developer account setup and using the Basic Display API - which means getting an actual Facebook account, something I've been actively avoiding for years.
  2. Just remove the sidebar.


April 26, 2020 @ 13:03

As I discussed in my previous post, everyone should be running SSL now... but the hacks I talked about were a bit annoying. So I'm trying out something new: Caddy.

Over the years I've changed my web sites from entirely dynamically-generated template-based stuff in Perl and Python, into this one, which is completely static, generated by Hugo. So I don't really run anything overly-complicated in my Nginx config anymore.

I spent some time over this weekend converting this and my other sites to use Caddy, which handles 100% of the SSL certificate generation, and otherwise works just like a web server.

I also converted both of the web servers fronting my Home Assistant servers, one of which runs on a Raspberry Pi.

Now I'm going to have a beer, instead of manually renewing three certificates. Cheers.

let's encrypt

February 6, 2020 @ 20:40

These days, everyone should be using SSL to secure, well, everything. It used to be that SSL certificates were really expensive, but with free providers like Let's Encrypt, there's not much excuse anymore.

Well... sorta.

In theory this is really easy to do, and easy to automate. In practice, well, a lot of the tools just plain suck, or they're designed for the most basic use-case and the most commonly used DNS providers. Or, they expect you use the certificate for a public website.

In my case, I have a number of private websites in addition to this one, and Postfix and Dovecot for my email. So I have to generate a few certificates, and then copy them to several machines and restart a bunch of daemons.

Also, for various reasons, I'm still using djbdns for my DNS, and so I've got to do things a little manually.

Here's my renew script, simply just call it with a list of domain names:


function join {
    for arg in $*; do
        echo -n "-d $arg "
domains=$(join $*)

certbot certonly --manual --preferred-challenges=dns \
                 --manual-auth-hook ~/bin/certbot-auth.sh \
                 --manual-public-ip-logging-ok --agree-tos \

That join function is a bit of a hack, but hey, it works.

Here's the auth-hook script - it generates a record suitable for import into djbdns and copies that to my server into the right place.


echo "'${rec}:${CERTBOT_VALIDATION}:300" > /tmp/${rec}
scp -i ~/.ssh/id_rsa /tmp/${rec} dns@myserver:/var/dns/extdns/root/dynamic/

sleep 30

That sleep there gives me 30 seconds to go manually run the "regenrate" process there, but this is better than nothing.

vim and powerline

January 24, 2020 @ 11:51

Do you use Vim and Powerline? If so, you may have got this error message at some point, when using a Virtualenv:

Traceback (most recent call last):
  File "<string>", line 4, in <module>
ModuleNotFoundError: No module named 'powerline'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<string>", line 9, in <module>
ModuleNotFoundError: No module named 'powerline'
An error occurred while importing powerline module.
This could be caused by invalid sys.path setting,
or by an incompatible Python version (powerline requires
Python 2.6, 2.7 or 3.2 and later to work). Please consult
the troubleshooting section in the documentation for
possible solutions.
Unable to import powerline, is it installed?

I know how to solve this! Put this in your .vimrc:

" workaround issue with powerline + virtualenv
" https://github.com/powerline/powerline/issues/1908
python3 << EOF
import sys
path = "/usr/lib/python{}.{}/site-packages/".format(
    sys.version_info.major, sys.version_info.minor)

Kubernetes Meetup at JW Player

March 1, 2019 @ 10:41

At JW Player last night, we hosted the New York Kubernetes Meetup. It went very well, we had about 80 people, and hung out with pizza and beer.

And I presented! I talked about the deployment system we’ve been working on which allows developers to deploy applications onto our Kubernetes clusters.

Update: So there used to be a video here, but since I’ve left JWPlayer, that link is no loger active. Oh well.

command-line search language

August 10, 2018 @ 09:22

At work, I've built a command-line client program for a number of microservices, which mostly do CRUD operations. I am looking to add search capabilities, but this is not as easy a task as I'd thought.


Let's call this client program boxer, because that's what it's actually named.

So using boxer, one can query a service to return an item in a database. Let's call this a thingy (not what it's really called). Getting a thingy is simple:

boxer get thingy [thing name]

And if we want a list of thingies:

boxer list thingies

With me so far? This is easy stuff. But what if we want to filter that list of thingies? Maybe it looks like this:

boxer list thingies where name = 'monkey'
boxer list thingies where name = 'goat' and version = 1

That's a little SQL-like. Or maybe it looks like this:

boxer list thingies --name 'monkey'
boxer list thingies --name 'goat' --version 1

But then the client program would have to know the structure of the data at the time it builds the argument parser - this is either really difficult, or involves querying the API for information. Probably not worth doing.

Or if there isn't a whole lot of data, the list call could return everything as JSON, and we could filter it using jq:

boxer list thingies | jq '.|(select(.name=="monkey"))'

This of course gets messy, and when you have a LOT of data, it's expensive to fetch ALL of it each time you only want a subset. Better to do the filtering at the back-end.

Common searches

Some things I figure people will commonly search for:

  • string 'like' or pattern in a string
  • item with an integer field max, as in "the thingy with the highest version"
  • item with the most recent date

Prior Art

I've been searching for any sort of "standard" for doing this, but so far I've come up empty. Lots of programs do this kind of thing, but it seems that everyone just re-invents the wheel each time, and there's no consistency.

So I'd be interested in ideas here :)

bunch of links

January 31, 2018 @ 00:15

Couple of links to stuff I’ve been working with recently.


A really nice Python date library.


A fork of another Python library I tried to contribute to, but that the author has basically abandoned.

IEX Developer Platform

Rather nice web API for obtaining real-time stock quotes.


Full-featured and highly-performant distributed reverse proxy engine. Not sure how I missed this one before… nor why I’m still using Amazon’s ELBs now.


A decent, lightweight Python asyncio IRC client library. AsyncIO all the things!

rebuilding things

January 30, 2018 @ 23:05

Recently I decided it’s about time to retire the ancient server that has been running my website (and mail and dns and a bunch of other things) - It’s a Northwood Pentium 4 Xeon from 2002, the kind with Hyperthreading. Running Linux 3.7.10. 32-bit.

Yes, that’s old.

Rather than upgrade, I’m probably moving all of this somewhere else, likely a cheap VPS somewhere… and since there’s a whole bunch of really old Perl running things here… I’ve been doing some rewriting.

Static webpage generation

The original website I put on this current server used a backend based on some unspeakable perl, but the most recent was using Pylons… and by “recent” I mean circa 2010. So yeah, that’s got to be redone. I pondered another web application, but decided that rather than doing on-the-fly content generation from templates, why not just pre-generate it?

So that’s what I’m doing. I’ve started building a bunch of Python scripts using, Jinja templates, and Markdown for blog content. Naturally, all run by Makefiles.

Let’s Encrypt

Since I’m doing it all new, I’m finally going to go full SSL, via Let’s Encrypt. Yeah, about time.

So yeah. Stay tuned.

in the queueueueueueue

June 12, 2012 @ 16:53

Recently, I’ve been playing with message queue-based interprocess communication in Python. I’ve got an application idea which uses a client / worker-pool distributed processing concept, and wanted to test out a few of the options available.

Tested libraries/services include:

Yes, I know that ZooKeeper isn’t really an MQ, but it’s pretty easy to implement one with it.


0MQ is exactly what they say: “sockets on steroids”. There’s no 0MQ “daemon”, no extra processes, just sockets and a library to talk via them. This is both good and bad.

Without a daemon, it means that there’s no extra point of failure to rely upon, you get full control over your queues and how they’re used. But also, you don’t get clustering unless you do it yourself, you don’t get any administration tools unless you do them yourself, no access control unless you do it yourself, &c.

Working with 0MQ

It took me a while to figure out how many queues I needed for my concept, and how to route them. I designed separate client and worker processes, with a single broker to manage the worker pool. This setup was a bit complex, with client having an output to the broker’s input queue, and an input queue for responses from workers. The broker has a single input queue, with various outputs to each of the workers in the pool. The workers each had an input queue.

Actual code for 0MQ is reasonably simple. I ended up making a few objects to wrap the low-level stuff, but it’s not like 0MQ is overly verbose. There’s a lot of different queue options, all of which do things slightly differently, I ended up using PULL for the worker and client input, PUSH for client -> broker, and PULL into the broker and PUSH out of the broker.


Decent. A bit limited. Not very robust unless you want to do your own heartbeat stuff, it’s a little hard to tell that you’ve actually connected and sent a message and that message was definitely received. I probably won’t use it for this.


Celery is a task/job queue built on message passing, which uses one of a handful of MQ systems. I used RabbitMQ via amqp, because I had it installed already.

I do like that it uses a backend MQ system, and that there’s a method and library for integrating it with Pylons (even though I’ve moved on to Pyramid now).

Working with Celery

Celery is a very high-level library, which uses decorators to indicate task procedures. You can do a lot with very minimal code.


I ended up stopping work on Celery, I just didn’t like how the system works. For one, I’m not really a fan of decorators which do a lot behind the scenes, because it’s not really obvious what is going on at a glance.

Also, there’s no clear way to have a callback when a task is complete, you can only check on the status of jobs from the caller. You can chain tasks, but the subtasks will run in the worker process, not the caller, which just isn’t going to be useful for me.

Apache ZooKeeper

As I said, ZooKeeper isn’t really a MQ system, it’s more of a shared storage system for maintaining configuration, naming, synchronization, and the like, but it’s easy to make a message queue. You just make a node for the queue, and therein make a sub-node for each message, letting ZooKeeper name them sequentially. Your worker can process them in order.

ZooKeeper is designed from the start to be robust and distributed, which allows the application developer to not really worry about these things.

Working with ZooKeeper

The python library included in the ZooKeeper source is a bit low-level, so I wrote a basic wrapper library to provide object- oriented access to ZNodes, then built a “ZQueue” object on top of my ZNode class. I found that treating the ZooKeeper tree as a tree of objects works very well when building higher level applications.

My method of making a ZNode for the queue and then ZNodes for each message means ZooKeeper acts as the broker, and there’s no need to write a middle layer.


I like ZooKeeper.

It’s not really a MQ, so I probably won’t use it as such, but I’m glad I tried it out, I can think of lots of other uses for it.

Pika and RabbitMQ

Pika is a pure-Python implementation of AMQP 0.9.1, the standard MQ protocol used by various MQ brokers, including my choice here, RabbitMQ.

RabbitMQ is a robust message broker with a management interface and bindings for many languages.

Pika is reasonably flexible, not too low-level but not too high. RabbitMQ is quite full-featured, including a full management interface which lets you inspect queues, and a full access-control system which is entirely optional.

Working with Pika 0.9.5

The Pika library is a little lower level than say, Celery, but it’s still reasonably easy to work with. Doing blocking work is rather easy, but doing anything asynchronous or CPS is a little more complex, you have to define mutliple callbacks. I just created objects for both my client and worker.

With RabbitMQ acting as the broker, there’s no need to build a broker layer like we did with 0MQ.


Pika has lots of nice features. You can create named, permanent queues, temporary unnamed queues, full message exchanges, tag messages with types and filter on those types, require or discard ACKs, &c. I see lots of possibilites there.

RabbitMQ servers can be clustered to form a single logical broker, and you can mirror queues across machines in a cluster to provide high availability.

In fact, Pika/Rabbit seems to have far more features than I’ll actually need for the project I have in mind - which is good, really.


So far, as should be obvious, I’m going to use Pika and RabbitMQ. I’ve only been playing with the combo for a day, so expect more to come here.

pong is not a physics problem

March 28, 2012 @ 10:13

So my friend Don has been learning iPhone/iPad programming lately, and he decided to implement a pong game to learn how to do game and graphics programming. He’d done this before, years ago in DOS, which makes it a good task to learn the new API.

He kept showing me code… and of course that got me thinking.

So I put together a quick pong demo for Android using AndEngine and the box2d physics engine.

You may now be saying, “But wait! Pong doesn’t need a physics engine!” Yes, yes, you’re right. The world doesn’t need yet another pong game either. But I did want to learn to use box2d anyway, so I’d call this a success.

And here is a test program

March 20, 2012 @ 10:42

I just whipped up a little multitouch bug tester program for Android. As noted in my previous post, lots of Android phones running 2.3.7 and earlier have buggy multitouch. In this example, I show the pointers (up to 5) with pointer numbers. You should be able to see at least two, and if the bug is not present, they should work properly. If you have the bug, you’ll notice 1 and 2 swapping back and forth as you touch 1, touch 2, lift 1, then tap 1.

Enjoy. multitouchbugtest.apk

quit tryin' to be clever

February 22, 2012 @ 16:16

I was trying to be clever, using the filename portion of the image URLs fed by Plus for the thumbnail images. Yeah, should know better. Now I’m just generating a filename from a UUID, ignoring what the Plus service provides.

So the images are showing up in the right column again. Huzzah!

The stars at night

January 11, 2012 @ 14:49

I’ve been working on an Android program which uses 2D graphics over the past few days, and since it’s always nice to have sample code to start from, I dug up my old Android starfield simulation.

I wrote this thing back in 2008, not that long after Android first came out, as an introduction to doing 2D graphics. I’ve written starfields before, first in C (and modeX) back in 1996, then in Flash’s ActionScript around 2002.

I just compiled the Android version, and discovered that nothing has changed in the API since 2008 - a very good thing. I did speed up the animation a little and add more stars, since modern phones are a lot faster, and there were odd gaps in the animation.

You can find all of these versions, with source code, right over here. Naturally, all source is copyright (c) me, and released under the Creative Commons by-nc-sa license.

Finally, a plus API.... well... almost

October 20, 2011 @ 13:06

So now there’s a Google Plus API for Python. You will notice I’ve added +1 buttons.

However, the sidebar on the right is gone for the moment, because there’s a rather major bug in either the Plus API, or the image resizer Google use for the thumbnail images. So right now I can’t actually pull thumbnails from Plus, I’d have to pull them down myself.

I’m pondering that, sure, but it seems the wrong way to do it.

Anyway, here’s the bug I’ve submitted to the Plus bug tracker. I suspect that’ll languish there for a year or so in the ‘new’ state, and by that time, I’ll have stopped caring.

You know, like most of the Android bugs I’ve cared about.


October 6, 2011 @ 17:57

I finally quit being lazy and updated my first Android app, calCOOLator. It was semi-broken because it was using standard java.lang.Double numbers, now I’m using java.math.BigDecimal and dealing with precision much better. Maintaining old applications, especially when it’s been years since you even looked at the code, is a little annoying.

I’m sure I’ll still get comments about how it doesn’t handle percentages correctly. “50 + 10% = 55” - wtf kind of math is that?

everyone's an amateur

August 18, 2011 @ 14:04

Yep, I’m doing it again. I’m trying to fix poorly-written libraries. This time it’s the python bindings for net-snmp, the most popular SNMP library for unix.

Libraries should never, just print error messages when they can instead throw an exception. Throwing an exception means the application developer using your library has an opportunity to handle the error gracefully, rather than having the library just crash, or in this case, not being able to determine an error condition even exists.

Bad practice. Here’s a link to my bug report.
Until they fix it, just edit netsnmp/client.py and replace the print stderr... line with one that throws a useful exception.

Stock quotes, currency, and airports

May 25, 2011 @ 23:59

For a recent project, I had need of simple Python libraries for currency conversion, stock quotes, and airport code lookup. In the past, I was using perl for this, so I used Finance::Currency::Convert::XE and Finance::Quote.

But since I’m using Python here, those aren’t available to me. And unfortunately, I couldn’t find anything on the web either.

And for the airport codes, I got a list and stuck it in a database. But then I have to maintain that… and that’s annoying.

So…. here you are. airportcodes.py, googlequote.py and xecurrency.py.

Drag and Drop in PyQt

January 14, 2011 @ 10:54

So you want to be able to drag items out of a QListView? The docs suck on this. Here’s how simple it actually is:

class MyListView(QtGui.QListView):
    def __init__(self, parent=None):
        super(MyListView, self).__init__(parent)  
        # enable dragging
        # I'm only interested in drag, not drop, so I disable drop
        # use this to allow selecting multiple entries

    def mouseMoveEvent(self, event):
    def dragObject(self):
      if not self.selectedIndexes():

      drag = QtGui.QDrag(self)  
      data = []
      for index in self.selectedIndexes():
          if not index.isValid(): continue

          # this assumes your model has a nodeFromIndex() method -
          # it's easy to set one up, you'll probably have a custom
          # model class anyways
          node = self.model().nodeFromIndex(index)

      # in this case I'm just making a newline-seperated list
      # of the data, you could do pretty much anything here
      md = Qt.QMimeData()
      md.setData('text/plain', "\n".join(data))  
      # this is important.  Without this, it won't do anything.
      # you can use different actions like Qt.MoveAction, which
      # would remove the item from your model, but then your model
      # has to be more complicated.  I'm only interested in copy here.
      dropAction = drag.exec_(Qt.Qt.CopyAction)  

Yup, that’s it. There’s lots of conflicting docs, some tell you to do complicated things with mousePressEvent(), others are older than modern Qt, others just plain wrong.

Google Buzz

November 4, 2010 @ 15:48

You may have noticed the word ‘buzz’ at the top of the right column on my site here. Yes, that does link to my Google Buzz account. Most things I post there, will also show up on this page.

This is actually quite easy to do.

The first thing you need is Google’z Buzz client library. I’m using Python, so I use this one here. The docs explain pretty well how to use it, but the basic code looks like this:

buzz_client = buzz.Client()
buzz_client.build_oauth_access_token(YOUR_API_KEY, YOUR_API_SECRET)

That will get you an active buzz client. Here’s how I fetch a list of my own posts:

    for post in buzz_client.posts(type_id='@self', user_id='@me', max_results=10).data:
        # do something with your posts here

except buzz.RetrieveError, e:
    # handle errors

The things I’m most interested in are the photos I post to my Picasa dropbox (usually taken from my phone while out in pubs and such). Photos are found in p.attachments, you’ll have to check for one with the attribute type equal to ‘photo’ (videos are unsuprisingly, ‘video’). Then you have attachment.preview and attachment.link - these are what show up on my page here.

Also interesting is attachment.type ‘article’.