Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We currently reside in Sydney after almost a decade in Berlin, have three kids, and have far too little time to really justify having a blog.
Quite some time ago at an AWS Meetup in 2014 I gave a talk (sadly, I can't find the slides) about how we used S3 and built tooling to interact with it based on our understanding of how the internals of the platform worked, and best practices as advised by AWS themselves. One large aspect of the tooling we built was a custom base 62 encoding using the math/big package from the Go standard library.
The general idea is that we had a filenames generated randomly based on a character set of [a-zA-Z0-9] and 12 characters in length. Since S3 is not an indexed data store, you can only enumerate objects serially (unless you know what you are looking for). By treating the keyspace as a a range of numbers, you can divide up the set of numbers by a number of parallel workers, then convert each range of numbers back into filenames within the character set above. The process by which we did this used the big number package as mentioned above, and some…
I've written a couple of times in the past about my use of IPv6 at home. Sadly the state of native IPv6 for consumers is not so much better, so I'm left without native connectivity to my router (to be fair, I have assumed nothing has changed but it is possible Deutsche Telekom is now offering it without making any fanfare about its arrival).
So I still have the humble Hurricane Electric tunnel…
I've been doing a lot more programming in Go recently, mostly because it has awesome concurrency primitives but also because it is generally a pretty amazing language. Unlike other languages which have threads, fibres or event-driven frameworks to achieve good concurrency, Go manages to avoid all of these but still remain readable. You can also reason about its behaviour very effectively due to…
As I wrote previously, I found that the mismatch between the goals of large cloud services like Amazon Web Services and the languages they support slightly conflict with the notion of making highly concurrent and parallelised workflows.
Of course the obvious followup to that post (even embarrassingly obvious since I've been copiously mentioning Go so much recently) is to point out that Google's App Engine is doing this right by supporting Go as a first-class language, even getting an SDK…
Almost two years ago now, I wrote a post about how I was fed up with resorting to shell scripting as my knee-jerk reaction to computer problems. At the time, I had been attacking any problem that required more than a couple of commands at the prompt by writing a shell (usually BASH) script and hit major limitations that I really should have been solving with a legitimate programming language. I…
As I've previously mentioned, I've been doing a lot of work recently with various aspects of AWS on a daily basis (or close to it). My primary language these days is still Ruby, but I've been labouring through the excellent Seven Languages in Seven Weeks book in the hope I can broaden my horizons somewhat. I'm fairly comfortable with Python, somewhat familiar with Javascript now after playing with…