Couple of links to stuff Iāve been working with recently.
A really nice Python date library.
A fork of another Python library I tried to contribute to, but that the
author has basically abandoned.
Rather nice web API for obtaining real-time stock quotes.
Full-featured and highly-performant distributed reverse proxy engine. Not sure
how I missed this one before⦠nor why Iām still using Amazonās ELBs now.
A decent, lightweight Python asyncio IRC client library. AsyncIO all the
things!
Recently I decided itās about time to retire the ancient server that has been
running my website (and mail and dns and a bunch of other things) -
Itās a Northwood
Pentium 4 Xeon
from 2002, the kind with Hyperthreading. Running Linux 3.7.10. 32-bit.
Yes, thatās old.
Rather than upgrade, Iām probably moving all of this somewhere else, likely a
cheap VPS somewhere⦠and since thereās a whole bunch of really old Perl
running things here⦠Iāve been doing some rewriting.
Static webpage generation
The original website I put on this current server used a backend based on some
unspeakable perl, but the most recent was using
Pylons⦠and by ārecentā I mean circa 2010.
So yeah, thatās got to be redone. I pondered another web application, but
decided that rather than doing on-the-fly content generation from templates,
why not just pre-generate it?
So thatās what Iām doing. Iāve started building a bunch of Python scripts
using, Jinja templates, and
Markdown for blog
content. Naturally, all run by Makefiles.
Letās Encrypt
Since Iām doing it all new, Iām finally going to go full SSL, via Letās
Encrypt. Yeah, about time.
So yeah. Stay tuned.
I made this pull request but the
author of the library thinks that not bothering to check HTTP status codes is
acceptable.
So my code goes from:
request_token, request_token_secret = self.oauth.get_request_token(method="POST")
auth_token = self.oauth.get_access_token(request_token, request_token_secret,method="POST")
self.session = self.oauth.get_session(auth_token)
To:
from rauth.service import process_token_request
from rauth.utils import parse_utf8_qsl
rsp = self.oauth.get_raw_request_token(method="POST")
rsp.raise_for_status()
request_token, request_token_secret = process_token_request(
rsp,
parse_utf8_qsl,
"oauth_token",
"oauth_token_secret")
rsp = self.oauth.get_raw_access_token(request_token, request_token_secret, method="POST")
rsp.raise_for_status()
auth_token = process_token_request(rsp, parse_utf8_qsl, "oauth_token", "oauth_token_secret")
self.session = self.oauth.get_session(auth_token)
Itās not horrible, but really, why would you ever think itās OK to not handle
errors?
Iāve been writing an API for a little project Iāve been working on for a while,
and in searching for a not-horrible way to do OAuth1 authentication, I actually
found a Python library that doesnāt suck.
Of course, itās not perfect. I noticed today that it doesnāt actually handle
HTTP error responses - it doesnāt even check the return code at all, just
assumes that any response itās given will be parseable. Which of course is not
at all true in many cases - including in mine.
So of course Iāve
forked it and am working on a fix.
Found another bug and made a pull request -
this time in the ārauthā library, which does OAuth in a reasonable sane way.
Except for this issue - I still have no idea why theyāre trying to parse the
OAuth response with a utility used for parsing HTTP requests, but hey, I
guess if it works for them, fine. For me though, I need to replace their use of
parse_utf8_qsl(s)
with json.loads(s.decode())
because my response is proper
JSON - shouldnāt OAuth responses be JSON anyway?
Whatever, itās late.
EDIT:
Okay so it turns out I was doing silly things like not reading the OAuth spec
and the response should be a query-string type thing like
oauth_token=foo&oauth_token_secret=bar
instead, which is what the library
parses just fine by default. Reading specs is a good plan, one I encourage
everyone to do.
My pull request is still valid though, if you really must break the spec, they
have the parser argument already, and it should work in a more sensible way.
Another bugfix for s3cmd.
Bugfix for s3cmd - some issues
with command-line arguments not working when I needed them to.
Today I created a program to pull data from the RSS feeds our service vendors
use for alerts, and either log, email, or instant message (we use Hipchat) to
various support groups.
AND, I open sourced it on github.
Enjoy!
February 27, 2014 @ 22:16
More python 3.3 porting, this time an interesting
Redis ORM.
EDIT: Changed my mind, someone else has started a 3.3 port which looks like
a way better method: mayfield/rom.
I see the Google plus article format returned by their Python API has changed
again. You will note the sidebar over on the right there only shows images and
no articles now. Iām getting really tired of fixing this every month.
Probably Iāll just not bother soon, and remove that whole sidebar altogether.
So Iāve been using this code in a few programs at work:
p = subprocess.Popen(...)
for line in p.stdout.readline():
...
print(line)
It turns out thereās a bunch of output buffering going on here.
You could put a sys.stdout.flush() after that print, but it wonāt help.
The iterator buffers. Do this:
p = subprocess.Popen(...)
while True:
line = p.stdout.readline()
if not line:
break
---
print(line)
Et violĆ ! No buffering.
Someone finally got me into the idea of using Python 3.x, so I started looking
at my current projects, and picked one to test with. This resulted in me having
to port a pretty major library, python-rrdtool. The trick about this is
that RRDTool is a C library, so the Python module is an extension module,
written in C.
And of course, strings and unicode are different from Python 2.x to 3.xā¦.
Anyway, hereās my pull request:
on github
December 21, 2012 @ 12:45
So Iām doing some REST stuff with SQLAlchemy classes, and I needed a simple way
to dump the objects to JSON. This base class will convert an object to a dict,
and has some options for funny fields and such.
You may (will) want to use a custom JSON encoder (see simplejson docs) to
convert some of the data types, like datetime.datetime in particular.
Itās certainly not ideal, in particular the isinstance() call, but it works
at the moment for what Iām doing.
class JsonObject(object):
def __to_json__(self, exclude_fields = []):
d = {}
for prop in dir(self):
if prop.startswith('_') or prop in exclude_fields:
continue
val = getattr(self, prop)
if callable(val):
continue
if isinstance(val, (sqlalchemy.schema.MetaData, JsonObject)):
continue
if hasattr(self, '__formatters__') and prop in self.__formatters__:
d[prop] = self.__formatters__[prop](data)
else:
d[prop] = val
return d
September 26, 2012 @ 12:19
I got a new laptop. More on that later, perhaps.
This laptop doesnāt have hardware controls for the screen brightness, itās done
via software. In linux, this is done via poking at stuff in /sys.
So to make my life easier, I wrote a little Python script to do it in a sane
way. I do a similar thing for the keyboard backlight, but that only has 3
levels of brightness, so the math is much easier.
backlight.py
Recently, Iāve been playing with message queue-based interprocess communication
in Python. Iāve got an application idea which uses a client / worker-pool
distributed processing concept, and wanted to test out a few of the options
available.
Tested libraries/services include:
Yes, I know that ZooKeeper isnāt really an MQ, but itās pretty easy to implement
one with it.
0MQ
0MQ is exactly what they say: āsockets on steroidsā. Thereās no 0MQ ādaemonā,
no extra processes, just sockets and a library to talk via them. This is both
good and bad.
Without a daemon, it means that thereās no extra point of failure to rely upon,
you get full control over your queues and how theyāre used. But also, you donāt
get clustering unless you do it yourself, you donāt get any administration tools
unless you do them yourself, no access control unless you do it yourself, &c.
Working with 0MQ
It took me a while to figure out how many queues I needed for my concept, and
how to route them. I designed separate client and worker processes, with a
single broker to manage the worker pool. This setup was a bit complex, with
client having an output to the brokerās input queue, and an input queue for
responses from workers. The broker has a single input queue, with various
outputs to each of the workers in the pool. The workers each had an input
queue.
Actual code for 0MQ is reasonably simple. I ended up making a few objects to
wrap the low-level stuff, but itās not like 0MQ is overly verbose. Thereās a
lot of different queue options, all of which do things slightly differently, I
ended up using PULL for the worker and client input, PUSH for client -> broker,
and PULL into the broker and PUSH out of the broker.
Results
Decent. A bit limited. Not very robust unless you want to do your own
heartbeat stuff, itās a little hard to tell that youāve actually connected and
sent a message and that message was definitely received. I probably wonāt use
it for this.
Celery
Celery is a task/job queue built on message passing, which uses one of a handful
of MQ systems. I used RabbitMQ via amqp, because I had it installed already.
I do like that it uses a backend MQ system, and that thereās a method and
library for integrating it with Pylons (even though
Iāve moved on to Pyramid now).
Working with Celery
Celery is a very high-level library, which uses decorators to indicate task
procedures. You can do a lot with very minimal code.
Results
I ended up stopping work on Celery, I just didnāt like how the system works.
For one, Iām not really a fan of decorators which do a lot behind the scenes,
because itās not really obvious what is going on at a glance.
Also, thereās no clear way to have a callback when a task is complete, you can
only check on the status of jobs from the caller. You can chain tasks, but
the subtasks will run in the worker process, not the caller, which just isnāt
going to be useful for me.
Apache ZooKeeper
As I said, ZooKeeper isnāt really a MQ system, itās more of a shared storage
system for maintaining configuration, naming, synchronization, and the like, but
itās easy to make a message queue. You just make a node for the queue, and
therein make a sub-node for each message, letting ZooKeeper name them
sequentially. Your worker can process them in order.
ZooKeeper is designed from the start to be robust and distributed, which allows
the application developer to not really worry about these things.
Working with ZooKeeper
The python library included in the ZooKeeper source is a bit low-level, so I
wrote a basic wrapper library to provide object- oriented access to ZNodes, then
built a āZQueueā object on top of my ZNode class. I found that treating the
ZooKeeper tree as a tree of objects works very well when building higher level
applications.
My method of making a ZNode for the queue and then ZNodes for each message means
ZooKeeper acts as the broker, and thereās no need to write a middle layer.
Results
I like ZooKeeper.
Itās not really a MQ, so I probably wonāt use it as such, but Iām glad I tried
it out, I can think of lots of other uses for it.
Pika and RabbitMQ
Pika is a pure-Python implementation of AMQP 0.9.1, the standard MQ protocol
used by various MQ brokers, including my choice here, RabbitMQ.
RabbitMQ is a robust message broker with a management interface and bindings for
many languages.
Pika is reasonably flexible, not too low-level but not too high. RabbitMQ is
quite full-featured, including a full management interface which lets you
inspect queues, and a full access-control system which is entirely optional.
Working with Pika 0.9.5
The Pika library is a little lower level than say, Celery, but itās still
reasonably easy to work with. Doing blocking work is rather easy, but doing
anything asynchronous or CPS is a little more complex, you have to define
mutliple callbacks. I just created objects for both my client and worker.
With RabbitMQ acting as the broker, thereās no need to build a broker layer like
we did with 0MQ.
Results
Pika has lots of nice features. You can create named, permanent queues,
temporary unnamed queues, full message exchanges, tag messages with types and
filter on those types, require or discard ACKs, &c. I see lots of possibilites
there.
RabbitMQ servers can be clustered to form a single logical broker, and you can
mirror queues across machines in a cluster to provide high availability.
In fact, Pika/Rabbit seems to have far more features than Iāll actually need for
the project I have in mind - which is good, really.
Conclusion
So far, as should be obvious, Iām going to use Pika and RabbitMQ. Iāve only
been playing with the combo for a day, so expect more to come here.
Time for an update to feedformatter.
It seems that the latest ElementTree (1.3.0) does things much differently than
the 1.2.6 Iād been using before.
This change
updates my āCDATA hackā and seems to work on both versions now.
See also my response at StackOverflow.
February 22, 2012 @ 16:16
I was trying to be clever, using the filename portion of the image URLs fed by
Plus for the thumbnail images. Yeah, should know better. Now Iām just
generating a filename from a UUID, ignoring what the Plus service provides.
So the images are showing up in the right column again. Huzzah!
Iāve just made a commit to webob,
a Python library which āprovides wrappers around the WSGI request environment,
and an object to create WSGI responsesā (according to
the webob website).
I admit that this is based solely on wanting to return
āIām a teapotā from
a Pyramid application, but it does make sense
to support arbitrary HTTP response codes as well.
Iāve been messing with building REST APIs in
Pyramid, and one thing that lacking was the
ability to send HTTP error messages to the client as JSON objects. Pyramid only
had support for HTML or plain text, neither of which is easy to parse when
youāre writing a JavaScript client.
Soā¦. this commit
to the Pyramid project ought to take care of things nicely.
Hopefully theyāll accept my pull request. If not, I can always just maintain my
own fork.
November 23, 2011 @ 15:04
As you can see, the right column of this page is actually full of stuff again.
I decided to just deal with the Google Plus API as
is, and instead of using their image URLs, Iām just locally caching all of the
images (and thumbnails) and hosting them here, rather than linking to broken
links.
It works decently, I suppose. Though I noticed that Plus is weird about what it
considers to be the ātitleā of a given post - most of these donāt actually have
titles, itās just guessing. So theyāre a little weird sometimes. Better than
nothing though.
November 17, 2011 @ 16:42
I just discovered this. So cool!
import readline
import rlcompleter
readline.parse_and_bind("tab: complete")
Put that in a startup file you use with $PYTHONSTARTUP.
So now thereās a Google Plus API for Python. You will notice Iāve added +1
buttons.
However, the sidebar on the right is gone for the moment, because thereās a
rather major bug in either the Plus API, or the image resizer Google use for the
thumbnail images. So right now I canāt actually pull thumbnails from Plus, Iād
have to pull them down myself.
Iām pondering that, sure, but it seems the wrong way to do it.
Anyway,
hereās the bug Iāve submitted
to the Plus bug tracker. I suspect thatāll languish there for a year or so in
the ānewā state, and by that time, Iāll have stopped caring.
You know, like most of the Android bugs Iāve
cared
about.
Yep, Iām doing it again. Iām trying to fix poorly-written libraries. This time
itās the python bindings for net-snmp, the most
popular SNMP library for unix.
Libraries should never, just print error messages when they can instead
throw an exception. Throwing an exception means the application developer using
your library has an opportunity to handle the error gracefully, rather than
having the library just crash, or in this case, not being able to determine an
error condition even exists.
Bad practice. Hereās a
link to my bug report.
Until they fix it, just edit netsnmp/client.py
and replace the print stderr...
line with one that throws a useful exception.
Google need to hurry up and release a G+ API, so I can replace the sidebar there
-> with my G+ feed rather than my Buzz feed.
Sorta feels like Buzz is irrelevant now, doesnāt it?
For a recent project, I had need of simple Python libraries for currency
conversion, stock quotes, and airport code lookup. In the past, I was using
perl for this, so I used
Finance::Currency::Convert::XE and
Finance::Quote.
But since Iām using Python here, those arenāt available to me. And
unfortunately, I couldnāt find anything on the web either.
And for the airport codes, I got a list and stuck it in a database. But then I
have to maintain that⦠and thatās annoying.
Soā¦. here you are.
airportcodes.py,
googlequote.py and
xecurrency.py.
Iām now officially a contributor to
feedformatter, a Python library to
generate RSS and Atom feeds. Iāll probably use it for this site soon.
Iāve been doing a lot of work on two Python package tracking libraries lately:
Iāve got one app right now that does package tracking, and a future website Iām
working on will need to do this, so I started looking for some useful code
already done. Naturally there were a lot of semi-working or partly started
things out there.
Python-fedex does work pretty well, after fixing a bug or two.
Packagetrack is a wrapper for FedEx, UPS, and USPS package tracking, that
uses the individual APIs and provides a generic result object with common
fields, so you can basically just not care what sort of tracking number you
have. It also returns objects that inherit from dict so I can use it to
make JSON objects for web stuff later.
One thing that sucks is that the docs on how to use the shippers APIs are pretty
overcomplicated. But then, isnāt that the case for most API docs? Good reading
if you want to fall asleep, but frustrating when you want to actually get Real
Work done.
Iām amused actually, since this is the first time in a LONG while that Iāve
contributed to a public project. Nearly all of my work has been either for-hire
(and thus proprietary), or so specific to my personal needs that there isnāt any
point in cleaning it up for release. Iām enjoying this, even if in this case
Iāve had to learn how to use git.
So you want to be able to drag items out of a QListView? The docs suck on this.
Hereās how simple it actually is:
class MyListView(QtGui.QListView):
def __init__(self, parent=None):
super(MyListView, self).__init__(parent)
self.setDragEnabled(True)
self.setAcceptDrops(False)
self.setDragDropMode(QtGui.QAbstractionItemView.DragOnly)
self.setSelectionMode(QtGui.QAbstractionItemView.ExtendedSelection)
def mouseMoveEvent(self, event):
self.dragObject()
def dragObject(self):
if not self.selectedIndexes():
return
drag = QtGui.QDrag(self)
data = []
for index in self.selectedIndexes():
if not index.isValid(): continue
node = self.model().nodeFromIndex(index)
data.append(str(node))
md = Qt.QMimeData()
md.setData('text/plain', "\n".join(data))
drag.setMimeData(md)
dropAction = drag.exec_(Qt.Qt.CopyAction)
Yup, thatās it. Thereās lots of conflicting docs, some tell you to do
complicated things with mousePressEvent()
, others are older than modern Qt,
others just plain wrong.
You may have noticed the word ābuzzā at the top of the right column on my site
here. Yes, that does link to my Google Buzz account. Most things I post there,
will also show up on this page.
This is actually quite easy to do.
The first thing you need is Googleāz Buzz client library. Iām using Python, so
I use this one here.
The docs explain pretty well how to use it, but the basic code looks like this:
buzz_client = buzz.Client()
buzz_client.oauth_scopes=[buzz.FULL_ACCESS_SCOPE]
buzz_client.use_anonymous_oauth_consumer(oauth_display_name=g.buzz_name)
buzz_client.build_oauth_access_token(YOUR_API_KEY, YOUR_API_SECRET)
That will get you an active buzz client. Hereās how I fetch a list of my own
posts:
try:
for post in buzz_client.posts(type_id='@self', user_id='@me', max_results=10).data:
except buzz.RetrieveError, e:
The things Iām most interested in are the photos I post to my Picasa dropbox
(usually taken from my phone while out in pubs and such). Photos are found in
p.attachments
, youāll have to check for one with the attribute type
equal to
āphotoā (videos are unsuprisingly, āvideoā). Then you have attachment.preview
and attachment.link
- these are what show up on my page here.
Also interesting is attachment.type
āarticleā.
So Iām testing out a new blogging enigne Iāve written in Python.
Itās based on Pylons and MongoDB, with a few nice extras, Genshi, WTForms,
jQuery, Markdown, the usual stuff. I did this mostly as a playground for using
these new technologies in various work and not-work projects.