adrift on a cosmic ocean

I recently wrote about how I'd historically been using Pig for some daily and some ad-hoc data analysis, and how I'd found Hive to be a much friendly tool for my purposes. As I mentioned then, I'm not a data analyst by any stretch of the imagination, but have occasional need to use these kinds of tools to get my job done. The title of this post (while originally a placeholder for something more…

This post title is inspired by this Google Plus post, although I've been meaning to write this post for a few days anyway (it's just a catchier title to describe the same thing). I'm not a data scientist, analyst or even a hard-code Hadoop user by any stretch of the imagination, but on occasion I need to do some log analysis when there is simply too much data to force through Awk (as much as I…

The background to this story is that I spent the bulk of one week recently working on getting a prototype service deployed with AWS CloudFormation, and the experience was still reasonably painful. My team has other services deployed with CloudFormation, which is working perfectly fine (now) but I had hoped there would be some improvements available since the last time we went through the process…

I don't have to do much data analysis fortunately, but when I do there are two options: either the data is local to our own datacenter and I can use our own Hadoop cluster, or it is external and I can use Elastic MapReduce. Generally you don't run an Elastic MapReduce cluster all the time, so when you create your cluster you still need to get that data into the system somehow. Usually the easiest…

Perhaps a little late in the year to be conducting a personal retrospective on the years past, but I feel at this point I'm starting to wonder about the challenges ahead. The last two to three years I've distinctly changed my career direction from systems engineering, to "DevOps" (whatever that means anymore), to developer. Sure, I'm technically a Computer Scientist by tertiary education standards…

Feature flags, flippers, toggles, selective rollouts and various other terms have been used to describe systems that allow you to deploy code without necessarily forcing it on all of your users. I won't describe it here as it has already been covered in depth by a variety of companies and people, notably (and not necessarily in date order) Flickr, Github, Forrst, Martin Fowler and many others. The…

I notice my pace has yet again slowed between the last chapter of the book - Erlang - and this one. Another five months has passed since I finished the chapter on Erlang! In actual fact, I haven't been slaving away on the next language that whole time - decompression of sorts has to follow each chapter, and dealing with a manic three-year-old, finding some time for a bit of exercise and trying to…

I seem to recall first being introduced to MOSH after I saw a link to it in my RSS feed, and almost simultaneously in my work's developer mailing list (where people tend to post about new and interesting technologies). People were literally losing their shit over it (well, ok, not literally) and declaring they would never use anything but MOSH ever again. I may be dramatising somewhat. It's…

I'm on vacation in Thailand at the moment and noticed a correlation here that I hadn't before. So many houses, shanties, shops, restaurants and various other abodes have awnings or coverings made of synthetic material and advertising various products. The awnings (I'm not sure what else to call them, as they provide various different services depending on the needs of the person) provide shade…

A couple of years ago now at least, at my previous employer, I was just starting to feel reasonably comfortable with Ruby. I can attribute a large part of that to having had a great boss who gave me the time to sink a bunch of time into coding up our IaaS and Config Management systems while still being effectively a Systems Administrator, and another large part to reading Metaprogramming Ruby. A…