Google Cloud Monitor Metrics: a Review

We’ve been running a bit of a baking-off of observability tools for the past few years. Over the last couple months we integrated Google Cloud Monitoring (formerly Stackdriver Metrics) to track custom application metrics. The impetus for this was two-fold:

1. Reduced cognitive load: it’s baked into Google Cloud so it’s theoretically one less tool to have (although GCP Console is so vast I think of it as many tools)
2. Reduced operational costs

The upshot is that it is inordinately complicated to get working (both the engineering and the visualizations) and I don’t think the investment in effort to implement it is worth any savings in on-going costs. Even though Stackdriver was founded in 2012, the tooling feels really basic (likely because of the long integration time into Google Cloud) compared to competitors. The off-the-shelf reports are very high-level, and the interface is buggy and slow.

Environmental Impact Face-off: Cows vs. Soda

I’ve been digging into the incredibly broad world of climate impact and the organizations working to address it. I read that Climeworks is working to sell captured carbon dioxide to the carbonated beverage industry. That got me thinking: carbonated beverages have CO2 in them, so when you crack the can (and when you burp) all the CO2 is going back into the atmosphere. Then, a friend shared an article about a British company working on collecting methane from cow’s burps. (It turns out 95% of methane from cows comes from their burps; not what I thought!) So how do cow burps and people burps stack up in their impact on climate change?

Shelter in Sourdough

With everyone locked-in at home for COVID-19 mitigation, it appears many are baking. I can’t find flour at the grocery story and I’ve read that the yeast supply chain is flattened. Fortunately, I still have yeast. Now, folks are getting into sourdough. I spent the first two years after my son’s birth making sourdough bread every week. Below I share some of my discoveries. I blogged the experience in excruciating details over on Tumbler if you want to dig in.

This isn’t “Normal” Work-from-home

Over the past three weeks I’ve supported a number of people who have transitioned to working from home during the Covid-19 lockdown. The one thing that keeps coming up is that this is different than standard remote work. I’ve worked remotely for 12 years of my career and this is definitely not “normal”. I think that’s a critical thing for folks working remotely for the first time to understand. When Covid-19 lockdowns are over, working from home will be easier; this is a lot harder.

Harnessing Resiliency for Security

On Thursday evening I got an alert that one of our server’s outbound network traffic exceeded normal level. By about 100-fold.

Why do we have these alerts? For security. I want to know if inbound or outbound traffic is too high. For inbound traffic, this could be the sign of a DoS attack. For outbound traffic this could indicate that one of our servers has been compromised and is now being used to send spam, distribute malware, or exfiltrate data.

I won’t get into the investigation here (the server wasn’t compromised). Instead I want to talk about how we’ve reduced the risk of a compromised server by harnessing the resiliency we designed into our operations. We can just blow everything away and start from clean images. This is a really big hammer; but, because of in-built resiliency, it’s non-disruptive.

DIY Risers for Wooden Train Sets

A few years ago, as my kid was getting into wooden trains, I made these risers so he could build bridges and elevated tracks. While there are lots of risers available for purchase, I really like these because they were fairly simple to make, and have a lot of stability. The key difference in the design is the angled sides which adds the stability. I was inspired by Incan buildings that have withstood earthquakes for more than half a millennium.

Interlocking blocks and sloping walls make Inca buildings extremely resistant, but not immune, to earthquake damage. 500 years of earthquakes have done remarkably little damage to Inca structures left in their complete state.

Inca Architecture, Ancient History Encyclopedia.

Loneliness and the Remote Office

I miss working with people, face-to-face, on the same challenge. For several years (and many times in my career) I have worked remotely. Don’t get me wrong there are great benefits too; but it is lonely.

Photo by Kewal on Unsplash

Beginner’s Luck: When Kittens Walk Through Walls

I really enjoy working with junior developers. They are super valuable in my team. They bring rapid innovation to stagnant areas of the code base or the process. They add a breath of fresh air to stuffy corners in senior developer-dom. They walk through walls.

Photo by AndriyKo Podilnyk on Unsplash

Managing Cron Jobs in Kubernetes with Ruby

We run a Rails application as Docker containers in Kubernetes. Our application and services have a fair number of scheduled tasks and before we moved to containers these ran using cron in the VM that the server ran on. When we moved to Docker we first migrated by deploying a container that just ran our cron jobs. We’ve now migrated those to native Kubernetes Cron jobs which vastly improved the resiliency of our system.

The New Google Cloud Build

We recently moved several of our projects to the new Google Cloud Build for building container images and pushing them to the repository. It’s a pretty simple system (not a full CI) but it does the job well, and I liked having the “build” part separate from the “run tests” part of the toolchain. That said, I feel like this is among the many tools that leave me writing bash scripts in YAML.