Writings on various topics (mostly technical) from Oliver Hookins and Angela Collins. We currently reside in Sydney after almost a decade in Berlin, have three kids, and have far too little time to really justify having a blog.
I'm continuing my winning streak of publishing a blog post once every 5 months, and what better reason than to rant about something? I feel slightly conflicted about this one but at the same time I'm annoyed enough to write this... anyway, enjoy.
Email is an annoying domain of the internet. It features heavily on my list of things I don't want to be responsible for anymore, in addition to DNS, BGP…
Well, I almost made it two years between posts. Life, changing jobs, mental health, and the general state of the world will do that. I actually have a few blog posts I've got saved up for writing and it's a bit of a race between me penning them here or $work getting a technical blog site running (since invariably these kinds of posts always start from a $work problem).
Anyhoo, let's start with the…
One of the services my team runs in AWS makes good use of Memcached (via the ElastiCache product). I say "good" use as we manage to achieve a hit rate of something like 98% most of the time, although now I realise that it comes at a significant cost - when this cache is removed, it takes a significant toll on the application. Unlike other applications that traditionally cache the results of MySQL…
One of the notably lacking metrics from CloudWatch has been the current and previous AutoScaling group sizes - in other words, how many nodes are in the cluster. I've worked around this by using the regular EC2 APIs, querying the current cluster size and the desired size and logging this to Graphite. However, it only gives you the current values - not anything in the past, which regular CloudWatch…
I don't have to do much data analysis fortunately, but when I do there are two options: either the data is local to our own datacenter and I can use our own Hadoop cluster, or it is external and I can use Elastic MapReduce. Generally you don't run an Elastic MapReduce cluster all the time, so when you create your cluster you still need to get that data into the system somehow. Usually the easiest…
I've been doing a lot more programming in Go recently, mostly because it has awesome concurrency primitives but also because it is generally a pretty amazing language. Unlike other languages which have threads, fibres or event-driven frameworks to achieve good concurrency, Go manages to avoid all of these but still remain readable. You can also reason about its behaviour very effectively due to…
As I wrote previously, I found that the mismatch between the goals of large cloud services like Amazon Web Services and the languages they support slightly conflict with the notion of making highly concurrent and parallelised workflows.
Of course the obvious followup to that post (even embarrassingly obvious since I've been copiously mentioning Go so much recently) is to point out that Google's App Engine is doing this right by supporting Go as a first-class language, even getting an SDK…
Unlike many, I'm actually a good boy and do backups of my personal data (for which I can mostly thank my obsessive-compulsive side). However, up until now I've been remiss in my duties to also take these backups off-site in case of fire, theft, acts of god or gods etc. Without a tape system or rotation of hard drives (not to mention an actual "off-site" site to store them), this ends up being a…
As I've previously mentioned, I've been doing a lot of work recently with various aspects of AWS on a daily basis (or close to it). My primary language these days is still Ruby, but I've been labouring through the excellent Seven Languages in Seven Weeks book in the hope I can broaden my horizons somewhat. I'm fairly comfortable with Python, somewhat familiar with Javascript now after playing with…
I've been using S3 a lot in the last couple of months, and with the Amazon SDK for Ruby it really is dead simple to work with (as well as all of the other AWS services the SDK supports currently). So simple in fact, that you could quite easily delete all of your objects with very little work indeed. I did some benchmarks and found that (with batch operations) it took around 3 minutes to delete…