Why I'm Not an SRE

SRE (Site Reliability Engineer) as coined by Google in 2003, came into prominence in 2016 with the release of “The SRE Book”. It defines a particular job in an engineering org. This job is mostly focused on of course, reliability, but also performance, observability/monitoring, code deployment, capacity planning, and many other adjacent activities. Some people say SRE is Google’s “version of DevOps” (aka a “DevOps implementation”). This is a little tricky to talk about, because DevOps, started as a rallying cry for shortening deployment times and adding more automation in the SDLC, has turned into a role itself.

Read Full Post…

The Incidental Complexity of Kubernetes

Kubernetes is the quintessential source of incidental complexity for our age, and our industry has been duped into believing that it is “good”. (50,000 people can’t be wrong!) In the classic No Silver Bullet, Brooks (the author), carves out two different types of complexity: Accidental complexity: the introduction of more code or moving parts “by accident”, and can be corrected. Essential complexity: inherit to the problem at hand, independent of the surrounding environment (environment, programming language, hardware constraints, etc).

Read Full Post…

WLED on a Ustellar RGB Flood 6 Pack

Previously, I bought this 6-pack of Outdoor RGBW Flood lights and disassembled them. These have the SEO words of: Ustellar RGB Flood Light Outdoor, Smart WiFi RGBW Color Changing Landscape Lights+Warm White 2700K, 40W App Control Christmas Spotlight Works with Alexa, IP66 Waterproof Uplights, Plug & Play (6 Pack) They have a part number of UT88865-RGBW-US-1. Fundamentally they are a string of 6 24v WS2814A RGBW flood lights with the first flood as the controller with a Tuya CBU module for wifi control.

Read Full Post…

AWS re:Invent 2023 Talk: Iterating Faster on Stateful Services in the Cloud

This is a joint talk with one of my Netflix Coworkers JS Jeannotte. In this talk we survey the different options for running stateful workloads on EC2. We also present a hybrid method of using containers to run workloads using Kubelet in standalone mode, combined with the EC2 replace-root-volume API. This method enables us to iterate quickly through: Loading new container images and restarting kubelet for workload software upgrades Rebooting the server into a new OS and kernel quickly for upgrading everything else All without losing the on-disk state.

Read Full Post…

re:Invent 2023 Badge Teardown

NOTE: This blog post is just a physical teardown and documentation of the device’s advertised bluetooth capabilities. No reverse engineering was done (I’m not that good). I recently attended AWS re:Invent 2023, and the badge had a curious circular object attached to it. In extremely small print it had some sort of serial, 00234867 silk screened on the case. Inside was a small circular PCB and a CR2016 battery: (click for a larger image)

Read Full Post…

Hosting an Apple Tasting Party

I recently hosted a “Great Apple Tasting Experiment”, where some friends and I blind tasted 27 different apple cultivars. Luckily I live near Berkely Bowl, and incredible grocery story with even more than this number of unique apples. Special thanks to the cashier who checked us out. Here is the list were were able to buy in this November of 2023: Juici Braeburn Rave Ambrosia Beni Shogun Fugi Red Delicious Smitten Granny Smith Ambrosia Pink Lady Jazz Opal Autumn Glory Gala Lady Golden Delicious Sweetango Lady Alice Apple Jonagold Gold Rush Hunnyz Envy Rome Beauty Organic Sweetie Honey Crisp Sugarbee Crimson Gold We all tried and rated each of these apples, without knowing their name.

Read Full Post…

Halloween Crafts 2023

Here is a sampling of craft projects for Halloween 2023 that I made with wife help. Click on a picture to see a full size image. Sorry for the terrible image quality. Transmutation Circle Materials: Charcoal chalk LED Candles Magnet Wire Wemos D1 controller running WLED Straight edge, vinyl stencils, wire/compass. It is kinda surreal to see this in real life. This thing is big enough to create a human from raw elements!

Read Full Post…

Using Per-Instance-Type Kernels in a Single AWS EC2 AMI

Sometimes it is uself to have an EC2 Machine Image (AMI) change its behavior depending on its runtime environment. This adds complexity to the AMI, but reduces the complexity of having to maintain N AMIs with N behaviors. One of the hardest things about an AMI to change at runtime is the Linux kernel. By the time you are acutally running, the Linux kernel is there, and it is too late to upgrade or change.

Read Full Post…

Using golang's pprof over a Unix Socket

Python Manhole When working with a complex python daemon, I really appreciated the python-manhole library. This tool allows you to listen on a Unix domain socket, and serve a REPL, get stack traces, and more. From the README, it is as simple as this: Install it: pip install manhole Add it to your python code somewhere in the start sequence: import manhole ... def main(): manhole.install() # this will start the daemon thread .

Read Full Post…

There Is No Such Thing As Stock Kubernetes

Sometimes I hear the phrase “Stock Kubernetes” and wonder exactly what one means when they say that. Often it is in contrast to a “Managed” kubernetes (k8s) offering, for example (just the big three): Amazon EKS Google GKE Azure AKS But these are all on cloud providers. When someone says “Stock Kubernetes”, they may mean they want to run k8s on a cloud provider, but not use the cloud provider’s service.

Read Full Post…