[{"content":"My K3s cluster runs on six Intel NUCs. They\u0026rsquo;re great for the 50+ containers that make up my homelab — Plex, Home Assistant, Frigate, Paperless, the usual suspects. But they\u0026rsquo;re terrible at machine learning. Four Skylake cores and 32 GB of RAM per node doesn\u0026rsquo;t get you far when Immich wants to classify 80,000 photos or Frigate wants a vision model to describe who\u0026rsquo;s at your door.\nThe Mac Studio sitting under my desk — M1 Ultra, 64 GB unified memory — is the opposite problem. Absurd single-machine ML performance, but I don\u0026rsquo;t want to migrate my entire cluster to it. I want the K8s cluster to stay the control plane for routing, TLS, service discovery, and GitOps, while the Mac handles the heavy compute.\nThe solution turned out to be one of Kubernetes\u0026rsquo; oldest and least glamorous features: manual Endpoints.\nThe pattern In Kubernetes, a Service normally discovers its backends through label selectors — it finds pods with matching labels and routes traffic to them. But if you create a Service without a selector and pair it with a hand-written Endpoints resource, you can point that Service at any IP address. Including one outside the cluster.\napiVersion: v1 kind: Service metadata: name: ollama spec: type: ClusterIP ports: - port: 11434 targetPort: 11434 name: http # No selector — backends come from the Endpoints resource --- apiVersion: v1 kind: Endpoints metadata: name: ollama # Must match the Service name subsets: - addresses: - ip: 192.168.1.200 # Mac Studio\u0026#39;s LAN IP ports: - port: 11434 name: http That\u0026rsquo;s it. Every pod in the cluster can now reach ollama.ollama.svc.cluster.local:11434 and the traffic routes to the Mac Studio. Add an Ingress in front and it\u0026rsquo;s also available at ollama.example.com with TLS termination, external-dns, and Homepage dashboard integration — all the same infrastructure every other service gets.\nThe Mac Studio doesn\u0026rsquo;t know or care that Kubernetes exists. It\u0026rsquo;s just listening on a port.\nWhat runs on the Mac Six services currently use this pattern:\nService Port What it does llama-swap 11434 LLM inference (OpenAI-compatible API) Immich ML 3003 Photo classification and face detection Pet Tagger 2287 Individual dog identification (YOLO + CLIP) ComfyUI 8188 Image generation (Stable Diffusion) Fooocus 7865 Image generation (simplified UI) Stable Diffusion 7860 Image generation (A1111 WebUI) The first three run as Docker Compose on the Mac. The image generation tools run natively. llama-swap runs as a macOS LaunchAgent via a plist:\n\u0026lt;plist version=\u0026#34;1.0\u0026#34;\u0026gt; \u0026lt;dict\u0026gt; \u0026lt;key\u0026gt;Label\u0026lt;/key\u0026gt; \u0026lt;string\u0026gt;com.llama-swap\u0026lt;/string\u0026gt; \u0026lt;key\u0026gt;ProgramArguments\u0026lt;/key\u0026gt; \u0026lt;array\u0026gt; \u0026lt;string\u0026gt;/usr/local/bin/llama-swap\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;--config\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;/opt/models/llama-swap.yaml\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;--listen\u0026lt;/string\u0026gt; \u0026lt;string\u0026gt;0.0.0.0:11434\u0026lt;/string\u0026gt; \u0026lt;/array\u0026gt; \u0026lt;key\u0026gt;RunAtLoad\u0026lt;/key\u0026gt; \u0026lt;true/\u0026gt; \u0026lt;key\u0026gt;KeepAlive\u0026lt;/key\u0026gt; \u0026lt;true/\u0026gt; \u0026lt;/dict\u0026gt; \u0026lt;/plist\u0026gt; Each K8s-side app follows the same three-resource template: a Service (no selector), an Endpoints pointing to the Mac\u0026rsquo;s IP, and an Ingress for external access. Adding a new proxied service takes about two minutes.\nKeeping the Mac in the GitOps loop The obvious downside of running workloads outside the cluster is that ArgoCD can\u0026rsquo;t manage them. The Docker Compose files and configs aren\u0026rsquo;t Kubernetes resources. But they\u0026rsquo;re still code, and they still need version tracking and dependency updates.\nI solved this by committing the Mac\u0026rsquo;s Docker Compose files into the same repo under hosts/mac-studio/:\nhosts/mac-studio/ ├── README.md ├── immich-ml/ │ └── docker-compose.yml └── pet-tagger/ ├── docker-compose.yml └── .env.example Renovate\u0026rsquo;s docker-compose manager is scoped to hosts/**/docker-compose.yml, so image updates get the same automated PRs as everything else in the cluster. When Immich releases a new version, Renovate opens a single PR that bumps both the in-cluster immich-server and the Mac\u0026rsquo;s immich-machine-learning — keeping them in lockstep, which matters because their internal API isn\u0026rsquo;t versioned.\nDeploying a Mac-side update is still manual — SSH in, docker compose pull \u0026amp;\u0026amp; docker compose up -d — but at least the change is tracked in Git and the version is always visible.\nWhy not just run the Mac as a K8s node? I tried this. K3s runs on macOS with some effort, and you can register a Mac as a worker node. The problems:\nDevice access is painful. The M1\u0026rsquo;s GPU is the whole point, but exposing Apple Silicon\u0026rsquo;s unified memory to a container runtime is an unsolved problem. Docker Desktop on Mac doesn\u0026rsquo;t pass through the GPU. Native containers barely exist. launchd vs. kubelet. macOS already has a solid process supervisor. Fighting it to let kubelet own everything creates more problems than it solves. The Mac reboots for updates. macOS updates are not optional in the way Linux unattended-upgrades are. Having a K8s node disappear for 10 minutes during a macOS update is disruptive. The Endpoints approach avoids all of this. The Mac is a black box that happens to speak HTTP on known ports. Kubernetes doesn\u0026rsquo;t need to schedule pods on it, manage its lifecycle, or understand its hardware.\nThe one service that skips the proxy Immich ML is the exception. Rather than routing through a Service and Endpoints, the Immich server pod points directly at the Mac via an environment variable:\nenv: - name: IMMICH_MACHINE_LEARNING_URL value: http://192.168.1.200:3003 This was a pragmatic choice. Immich ML processes thousands of photos during bulk imports, and the extra hop through kube-proxy added enough latency to be noticeable. For a service that only talks to one consumer, the direct connection is simpler and faster.\nWhat I\u0026rsquo;d change Static IPs are fragile. The Mac\u0026rsquo;s static IP is hardcoded in six Endpoints resources. If it changes, I need to update all of them. A DNS-based approach (ExternalName Service pointing at a hostname) would be cleaner, but ExternalName Services don\u0026rsquo;t support port remapping, which several of these services need.\nNo health checking. With real pod backends, Kubernetes removes unhealthy endpoints automatically. With manual Endpoints, if the Mac is down, traffic routes to a dead IP and times out. I could add a sidecar that periodically checks the Mac and updates the Endpoints resource, but it hasn\u0026rsquo;t been painful enough to build yet.\nDocker Compose deploys are manual. A webhook that triggers docker compose pull \u0026amp;\u0026amp; up -d on the Mac when Renovate merges a PR would close the GitOps loop completely. Watchtower could do this, but I\u0026rsquo;d rather build something that only acts on merged PRs, not on every image push.\nDespite these rough edges, the pattern has been running for about six months without issues. The Mac handles all the ML inference for the cluster — photo classification, LLM serving, image generation, pet identification — while the NUCs handle everything else. The Endpoints proxy is invisible to every consumer; they just call a .svc.cluster.local address and get an answer.\n","permalink":"https://www.rustybower.com/posts/mac-studio-kubernetes-compute-satellite/","summary":"\u003cp\u003eMy K3s cluster runs on six Intel NUCs. They\u0026rsquo;re great for the 50+ containers that make up my homelab — Plex, Home Assistant, Frigate, Paperless, the usual suspects. But they\u0026rsquo;re terrible at machine learning. Four Skylake cores and 32 GB of RAM per node doesn\u0026rsquo;t get you far when Immich wants to classify 80,000 photos or Frigate wants a vision model to describe who\u0026rsquo;s at your door.\u003c/p\u003e\n\u003cp\u003eThe Mac Studio sitting under my desk — M1 Ultra, 64 GB unified memory — is the opposite problem. Absurd single-machine ML performance, but I don\u0026rsquo;t want to migrate my entire cluster to it. I want the K8s cluster to stay the control plane for routing, TLS, service discovery, and GitOps, while the Mac handles the heavy compute.\u003c/p\u003e","title":"Mac Studio as a Kubernetes Compute Satellite"},{"content":"My Bird Buddy smart feeder takes a photo every time a bird visits. Over a few months, that adds up to thousands of images — all sitting in Bird Buddy\u0026rsquo;s cloud app with no good way to search, organize, or back them up. I wanted them in Immich, my self-hosted photo library, organized by species and properly timestamped.\nThe Bird Buddy app doesn\u0026rsquo;t have an export feature. But the pybirdbuddy Python library can authenticate to their API and walk the feed. So I built a two-stage pipeline: a CronJob that downloads new photos daily, and a second CronJob that syncs them into Immich. Both run in Kubernetes, and the whole thing is about 300 lines of Python.\nThe timestamp problem Bird Buddy images arrive as plain JPEGs with no EXIF metadata. If you drop them into Immich as-is, every image gets dated to when the CronJob ran, not when the bird visited. A cardinal at sunrise and a chickadee at sunset both show up as \u0026ldquo;3:00 AM\u0026rdquo; because that\u0026rsquo;s when download.py ran.\nThe Bird Buddy API does return a createdAt timestamp for each postcard. The fix is to write that timestamp into the JPEG\u0026rsquo;s EXIF data before Immich ever sees the file:\ndef apply_timestamp(filepath, dt, is_video): \u0026#34;\u0026#34;\u0026#34;Embed capture time so Immich dates the asset correctly.\u0026#34;\u0026#34;\u0026#34; local = dt.astimezone(LOCAL_TZ) if not is_video: stamp = local.strftime(\u0026#34;%Y:%m:%d %H:%M:%S\u0026#34;).encode() raw_off = local.strftime(\u0026#34;%z\u0026#34;) offset = (raw_off[:3] + \u0026#34;:\u0026#34; + raw_off[3:]).encode() if raw_off else None exif = piexif.load(str(filepath)) exif[\u0026#34;Exif\u0026#34;][piexif.ExifIFD.DateTimeOriginal] = stamp exif[\u0026#34;Exif\u0026#34;][piexif.ExifIFD.DateTimeDigitized] = stamp if offset: exif[\u0026#34;Exif\u0026#34;][piexif.ExifIFD.OffsetTimeOriginal] = offset piexif.insert(piexif.dump(exif), str(filepath)) # Also set file mtime — Immich\u0026#39;s fallback when EXIF is absent (videos) ts = dt.timestamp() os.utime(filepath, (ts, ts)) Immich reads DateTimeOriginal first, which is the standard EXIF field for \u0026ldquo;when was this photo actually taken.\u0026rdquo; For videos (MP4s from the feeder\u0026rsquo;s motion clips), EXIF doesn\u0026rsquo;t apply, so we fall back to setting the file\u0026rsquo;s modification time — Immich uses that as a last resort.\nThe timezone offset matters too. Without it, Immich has to guess whether \u0026ldquo;2026-07-04 08:32:15\u0026rdquo; is UTC, local time, or something else. Writing the explicit offset (-05:00 for Central) removes the ambiguity.\nStage 1: Download The downloader runs daily at 3 AM as a Kubernetes CronJob. It authenticates to Bird Buddy, walks the feed for the last 24 hours, and downloads any new media into species-organized directories on an NFS share:\n/photos/birdbuddy/ ├── American_Robin/ │ ├── abc123.jpg │ └── def456.mp4 ├── Black-capped_Chickadee/ │ ├── ghi789.jpg │ └── ... ├── Northern_Cardinal/ │ └── ... └── .downloaded.json ← state file for deduplication Deduplication is handled by a simple JSON state file that tracks every media ID we\u0026rsquo;ve already downloaded. The state file lives on the same NFS share as the photos, so it survives pod restarts.\nThe CronJob itself uses a stock python:3.12-slim image with pip-installed dependencies — no custom Docker image to maintain:\napiVersion: batch/v1 kind: CronJob metadata: name: birdbuddy-downloader spec: schedule: \u0026#34;0 3 * * *\u0026#34; timeZone: \u0026#34;America/Chicago\u0026#34; concurrencyPolicy: Forbid jobTemplate: spec: template: spec: containers: - name: downloader image: python:3.12-slim command: - /bin/sh - -c - | pip install --quiet pybirdbuddy aiohttp piexif tzdata \u0026amp;\u0026amp; \\ python /scripts/download.py envFrom: - secretRef: name: birdbuddy-credentials env: - name: OUTPUT_DIR value: /data/birdbuddy volumeMounts: - name: scripts mountPath: /scripts - name: data mountPath: /data/birdbuddy volumes: - name: scripts configMap: name: birdbuddy-downloader-scripts - name: data nfs: server: 192.168.1.10 path: /mnt/storage/media/photos/birdbuddy The Python script is delivered as a Kustomize-generated ConfigMap — the same pattern I use for AppDaemon automations. No Dockerfile, no container registry, no build pipeline. Change the Python, push to Git, ArgoCD syncs the ConfigMap.\nThe downloader supports two modes: an incremental daily sync (last 24 hours of the feed) and a full backfill that walks all collections. The initial backfill pulled about 2,000 images. Daily runs typically download 5-20 new photos.\nStage 2: Sync to Immich Thirty minutes after the downloader finishes, a second CronJob reconciles the NFS directory with Immich:\nTrigger a library scan. Immich has an \u0026ldquo;external library\u0026rdquo; feature that watches a filesystem path. The sync script pokes the API to start a scan and waits 60 seconds for it to index new files.\nEnsure a \u0026ldquo;Bird Buddy\u0026rdquo; album exists. All bird photos get collected into a single album for easy browsing.\nAdd missing assets to the album. Pages through every asset Immich imported from the BirdBuddy path and adds any that aren\u0026rsquo;t already in the album.\nArchive everything. Sets each asset\u0026rsquo;s visibility to archive, which removes it from the main Photos timeline while keeping it visible in the album and Archive views.\nThe archiving step is the key UX decision. Without it, my family\u0026rsquo;s photo timeline would be 80% bird pictures. Archiving keeps them organized and searchable without drowning out the actual family photos.\ndef archive_assets(asset_ids): for batch in chunked(asset_ids): api(\u0026#34;PUT\u0026#34;, \u0026#34;/api/assets\u0026#34;, {\u0026#34;ids\u0026#34;: batch, \u0026#34;visibility\u0026#34;: \u0026#34;archive\u0026#34;}) The sync script uses only Python\u0026rsquo;s standard library — no requests, no third-party HTTP client. urllib.request is ugly but it means the CronJob doesn\u0026rsquo;t need a pip install step at all.\nSecrets management Bird Buddy credentials and the Immich API key are stored as Sealed Secrets:\napiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: birdbuddy-credentials namespace: birdbuddy-downloader spec: encryptedData: BIRDBUDDY_EMAIL: AgBTcv5LFLpb610j... BIRDBUDDY_PASSWORD: AgBipbrCt5fI3I8V... The download script reads them as environment variables via envFrom. Nothing sensitive touches Git in plaintext.\nWhat I learned EXIF timestamps are non-negotiable for photo imports. Without them, bulk imports are useless — every photo shows up on the same date. This applies to any pipeline that downloads photos from an API and imports them into a photo manager. The few lines of piexif code saved hours of manual re-dating.\nTwo CronJobs are better than one. Separating download from sync means I can re-run either independently. If Immich has a hiccup, I re-run the sync without re-downloading everything. If Bird Buddy\u0026rsquo;s API is down, the next day\u0026rsquo;s sync picks up whatever was already downloaded.\nNFS as the integration layer. The downloader writes to NFS. Immich\u0026rsquo;s external library reads from NFS. The sync script talks to Immich\u0026rsquo;s API. No service-to-service coupling, no message queues, no shared databases. The filesystem is the contract.\nThe pipeline has been running unattended for several months now. Every morning, any new bird visitors appear in the Bird Buddy album, correctly dated and organized by species. The total infrastructure cost is two CronJobs and an NFS share that was already there.\n","permalink":"https://www.rustybower.com/posts/birdbuddy-immich-photo-pipeline/","summary":"\u003cp\u003eMy \u003ca href=\"https://mybirdbuddy.com/\"\u003eBird Buddy\u003c/a\u003e smart feeder takes a photo every time a bird visits. Over a few months, that adds up to thousands of images — all sitting in Bird Buddy\u0026rsquo;s cloud app with no good way to search, organize, or back them up. I wanted them in \u003ca href=\"https://immich.app/\"\u003eImmich\u003c/a\u003e, my self-hosted photo library, organized by species and properly timestamped.\u003c/p\u003e\n\u003cp\u003eThe Bird Buddy app doesn\u0026rsquo;t have an export feature. But the \u003ccode\u003epybirdbuddy\u003c/code\u003e Python library can authenticate to their API and walk the feed. So I built a two-stage pipeline: a CronJob that downloads new photos daily, and a second CronJob that syncs them into Immich. Both run in Kubernetes, and the whole thing is about 300 lines of Python.\u003c/p\u003e","title":"Automating a Smart Bird Feeder's Photo Archive with Kubernetes and Immich"},{"content":"Home automations are code that controls physical things. When they break, the consequences aren\u0026rsquo;t a 500 error — they\u0026rsquo;re a garage door that won\u0026rsquo;t close in a snowstorm, or window shades that open at 3 AM, or a tablet that charges to 100% and stays plugged in for months until the battery swells.\nI run about 20 AppDaemon automations in my homelab, all deployed as Kubernetes ConfigMaps and tested with appdaemontestframework. The testing isn\u0026rsquo;t academic — it\u0026rsquo;s caught real bugs that would have been annoying or expensive to discover in production.\nWhy AppDaemon over HA automations? Home Assistant\u0026rsquo;s built-in automations are great for simple trigger-action rules. \u0026ldquo;When the sun sets, turn on the porch lights.\u0026rdquo; But anything with branching logic, state tracking, or multiple interacting conditions becomes hard to read and impossible to test.\nAppDaemon lets you write automations as Python classes. Each one gets an initialize() method where you set up listeners and timers, and callback methods that fire when things happen. It\u0026rsquo;s just Python — you can use conditionals, loops, data structures, and (critically) unit tests.\nThe tradeoff is deployment complexity: you need to run AppDaemon as a separate service alongside Home Assistant, manage the Python files, and wire up the configuration. Kubernetes and Kustomize make this manageable.\nThe deployment model Every AppDaemon automation lives as a .py file in the Git repo. Kustomize bundles them into a ConfigMap and mounts them into the AppDaemon container:\n# environments/bowerhaus/apps/appdaemon/kustomization.yaml configMapGenerator: - name: appdaemon-config files: - appdaemon/apps.yaml - appdaemon/bedroom_shades.py - appdaemon/garage_door_auto_close.py - appdaemon/tablet_charger.py - appdaemon/doorbell_person_frigate_snapshot.py # ... 15+ more The deployment mounts this ConfigMap at /conf/apps, which is where AppDaemon looks for automation modules:\nvolumes: - name: config-volume configMap: name: appdaemon-config containers: - name: appdaemon volumeMounts: - name: config-volume mountPath: /conf/apps The workflow is: edit Python → push to Git → ArgoCD syncs the ConfigMap → AppDaemon picks up the changes. No SSH, no file copying, no manual restarts.\nExample: temperature-aware garage doors This one has saved me real money on heating bills. In cold weather, if a garage door is left open, the automation starts a countdown based on the current temperature:\nTemperature Timeout Below 30°F 7 minutes Below 40°F 15 minutes Below 50°F 30 minutes 50°F+ No auto-close Before closing, it flashes the garage lights to warn anyone who might be in the way:\nclass GarageDoorAutoClose(hass.Hass): def get_timeout_for_temperature(self): temp = float(self.get_state(self.temp_sensor, attribute=\u0026#34;temperature\u0026#34;)) if temp \u0026lt; 30: return 7 * 60 elif temp \u0026lt; 40: return 15 * 60 elif temp \u0026lt; 50: return 30 * 60 return None def auto_close_door(self, kwargs): door = kwargs[\u0026#34;door\u0026#34;] # Re-check temperature — it might have warmed up during the wait if self.get_timeout_for_temperature() is None: self.log(f\u0026#34;Temperature now above 50°F — skipping auto-close\u0026#34;) return # Flash lights as warning, then close self.flash_lights_then_close(door, ...) The re-check on close is important. If the door opens at 29°F and warms to 52°F during the 7-minute wait, the auto-close cancels. Temperature thresholds are checked twice — once when the timer starts and once when it fires.\nExample: forecast-aware window shades The bedroom shade automation is the most complex one. It manages motorized shades based on time of day, weather forecast, and thermostat state:\nMorning: Open at 8:30 AM, unless the forecast high is above 85°F (solar heat gain would fight the AC all day). Midday: If the thermostat starts cooling, close the shades to reduce heat gain. When cooling stops, re-open — but only if it\u0026rsquo;s before 5 PM. Evening: Close at sunset or 7 PM, whichever comes first. The same BedroomShades class is reused across rooms with different parameters:\n# apps.yaml bedroom_shades: module: bedroom_shades class: BedroomShades covers: - cover.upstairs_master_bedroom_valance - cover.upstairs_master_bedroom_south thermostat: climate.master_bedroom_thermostat forecast_high_threshold: 85 morning_open_enabled: true # Guest room: close-only (no morning open, no cooling re-open) guest_shades: module: bedroom_shades class: BedroomShades covers: - cover.upstairs_guest_bedroom_valance thermostat: climate.guest_bedroom_thermostat morning_open_enabled: false forecast_high_threshold: 78 That morning_open_enabled: false flag exists because of a bug. Some rooms\u0026rsquo; shades were opening unexpectedly when the AC cycling stopped — the \u0026ldquo;cooling stopped → re-open\u0026rdquo; path didn\u0026rsquo;t check whether the shades were supposed to be auto-opened in the first place. The test that caught this:\n@automation_fixture( (BedroomShades, { \u0026#34;covers\u0026#34;: [\u0026#34;cover.guest_south\u0026#34;], \u0026#34;thermostat\u0026#34;: \u0026#34;climate.guest\u0026#34;, \u0026#34;morning_open_enabled\u0026#34;: False, }) ) def guest_shades_app(): pass def test_no_reopen_when_morning_open_disabled(given_that, guest_shades_app): opens = [] guest_shades_app.open_shades = lambda reason: opens.append(reason) guest_shades_app.sun_down = lambda: False guest_shades_app.is_before_reopen_cutoff = lambda: True guest_shades_app.closed_for_cooling = True guest_shades_app.thermostat_changed( \u0026#34;climate.guest\u0026#34;, \u0026#34;hvac_action\u0026#34;, \u0026#34;cooling\u0026#34;, \u0026#34;idle\u0026#34;, {} ) assert opens == [] # Shades should NOT have opened The @automation_fixture decorator from appdaemontestframework instantiates the real app class with mock Home Assistant services injected. You can call any method directly and assert on the side effects. The test above monkey-patches open_shades to record calls instead of actually calling the cover service — pure behavioral testing.\nExample: tablet battery management Three Fire tablets run Fully Kiosk Browser as Home Assistant dashboards around the house. Each one sits on a Shelly smart plug. The automation keeps batteries between 25% and 80%:\nclass TabletCharger(hass.Hass): def battery_check(self, entity, attribute, old, new, kwargs): battery = int(float(new)) plug_on = self.get_state(cfg[\u0026#34;plug\u0026#34;]) == \u0026#34;on\u0026#34; if battery \u0026lt;= 25 and not plug_on: self.turn_on(plug) elif battery \u0026gt;= 80 and plug_on: self.turn_off(plug) def periodic_check(self, kwargs): # Every 5 minutes, catch edge cases the state listener might miss for name, cfg in self.tablets.items(): battery = int(float(self.get_state(cfg[\u0026#34;battery\u0026#34;]))) if battery \u0026gt;= 100 and self.get_state(cfg[\u0026#34;plug\u0026#34;]) == \u0026#34;on\u0026#34;: self.turn_off(cfg[\u0026#34;plug\u0026#34;]) # Hard cutoff at 100% The state listener handles the normal case. The periodic check is a safety net — if a state change event is missed (HA restart, network hiccup), the 5-minute sweep catches it. The 100% hard cutoff is separate from the 80% target because a tablet that jumps from 79% to 100% between state reports would otherwise stay charging indefinitely.\nTesting with time travel Some automations are timer-based — they do something after N hours. The test framework has a time_travel helper that fast-forwards timers without actually waiting:\ndef test_projector_sends_notification_after_6_hours( given_that, theatre_projector_app, time_travel, assert_that ): given_that.state_of(\u0026#34;media_player.theater\u0026#34;).is_set_to(\u0026#34;on\u0026#34;) theatre_projector_app.projector_turned_on( \u0026#34;media_player.theater\u0026#34;, \u0026#34;state\u0026#34;, \u0026#34;off\u0026#34;, \u0026#34;on\u0026#34;, {} ) time_travel.fast_forward(21600) # 6 hours in seconds theatre_projector_app.check_projector_6_hours( {\u0026#34;entity\u0026#34;: \u0026#34;media_player.theater\u0026#34;} ) assert_that().notification().was_sent_with_message( \u0026#34;Projector has been on for 6 hours\u0026#34; ) This tests that the projector automation sends a push notification after 6 hours and force-powers-off the projector after 12. Without time travel, you\u0026rsquo;d have to wait 12 actual hours to verify the behavior.\nThe testing payoff The tests have caught three classes of bugs:\nFlag interactions. The morning_open_enabled bug above — a flag meant to disable one behavior accidentally gated a different behavior. Timer state leaks. A cancelled timer\u0026rsquo;s callback still firing because the handle wasn\u0026rsquo;t properly cleared. Edge case arithmetic. Temperature thresholds that didn\u0026rsquo;t handle None (sensor unavailable) or string values from HA. None of these would have been caught by \u0026ldquo;deploy it and watch what happens\u0026rdquo; — they\u0026rsquo;re the kind of bugs that only manifest under specific conditions (cold snap + garage door + recently restarted HA) and are nearly impossible to reproduce manually.\nRunning the tests is just pytest:\ncd environments/bowerhaus/apps/appdaemon pip install appdaemontestframework pytest tests/ No Kubernetes cluster needed, no Home Assistant instance, no real hardware. The tests run in CI alongside kustomize build validation.\nIs it over-engineered? Maybe. Plenty of people run Home Assistant automations from the UI and never look back. But I manage 20+ automations across six thermostats, three garage doors, five window shade motors, three tablets, and a projector. At that scale, \u0026ldquo;edit YAML in the HA UI and hope for the best\u0026rdquo; stops working. Automations interact with each other in ways that are hard to predict without tests, and bugs in physical-world automations have real consequences.\nThe ConfigMap deployment model means every change goes through Git, gets reviewed in a diff, and can be rolled back. The tests mean I can refactor an automation without manually verifying every scenario. It\u0026rsquo;s the same discipline we apply to production services — it just happens to control window shades instead of API endpoints.\n","permalink":"https://www.rustybower.com/posts/unit-tested-appdaemon-automations-kubernetes/","summary":"\u003cp\u003eHome automations are code that controls physical things. When they break, the consequences aren\u0026rsquo;t a 500 error — they\u0026rsquo;re a garage door that won\u0026rsquo;t close in a snowstorm, or window shades that open at 3 AM, or a tablet that charges to 100% and stays plugged in for months until the battery swells.\u003c/p\u003e\n\u003cp\u003eI run about 20 \u003ca href=\"https://appdaemon.readthedocs.io/\"\u003eAppDaemon\u003c/a\u003e automations in my homelab, all deployed as Kubernetes ConfigMaps and tested with \u003ca href=\"https://github.com/FlorianKemworthy/appdaemontestframework\"\u003eappdaemontestframework\u003c/a\u003e. The testing isn\u0026rsquo;t academic — it\u0026rsquo;s caught real bugs that would have been annoying or expensive to discover in production.\u003c/p\u003e","title":"Unit-Testing Home Automations That Run in Kubernetes"},{"content":"My grandfather had about 10,000 35mm slides. I rented a SlideSnap X1 and spent a weekend feeding them through — 33 boxes worth, organized into folders by box. The scanner itself was great, but when you\u0026rsquo;re pushing through thousands of slides in a weekend, some inevitably go in upside down or backwards. No metadata, no EXIF orientation flags — just thousands of JPEGs, some right-side up, some not, sitting on a NAS.\nManually reviewing 10,000 images isn\u0026rsquo;t realistic. But a vision LLM running on local hardware can look at each one and answer a simple question: is this upside down?\nThe problem Scanned slides can be wrong in four orientations:\nOrientation What happened Correct Slide was loaded properly Upside down (180°) Slide was inserted flipped vertically Mirrored Slide was scanned from the emulsion side Mirrored + upside down Both problems at once Upside-down images are by far the most common issue. Mirror flips are harder to detect without readable text in the image, so I focused on rotation first.\nWhy not digiKam or OpenCV? I tried digiKam first. It has an auto-rotate feature that uses DNN-based orientation detection, and it kind of works — but it felt painfully laggy when processing thousands of images, and the accuracy wasn\u0026rsquo;t great. It would confidently leave obviously upside-down photos untouched. For a few hundred photos it\u0026rsquo;s probably fine, but at this scale I needed something I could script, run unattended, and verify afterwards.\nPure classical computer vision doesn\u0026rsquo;t help much either. You could try detecting faces and checking if they\u0026rsquo;re inverted, but many of these photos are landscapes, buildings, or candid shots without clear faces. Edge detection and gradient analysis can suggest a dominant \u0026ldquo;up\u0026rdquo; direction, but they\u0026rsquo;re unreliable on photos with ambiguous composition — a snow-covered mountain reflected in a lake, for example.\nThe task requires the kind of semantic understanding that a human uses: sky goes up, people stand on their feet, buildings point upward, text reads left-to-right. A vision language model does exactly this.\nThe setup I already had Ollama running on a Mac Studio (M1 Ultra, 64 GB RAM) for Frigate\u0026rsquo;s camera event descriptions. The Mac Studio sits on the same network as the NAS, so the pipeline is:\nRead image from NAS (SMB mount) Send to Ollama\u0026rsquo;s vision model via HTTP API Parse the response Record the result For the model, I used llama3.2-vision (11B parameters). It fits comfortably in 64 GB of RAM and processes each image in about 15 seconds. The smaller minicpm-v was faster but significantly less accurate.\nWhat didn\u0026rsquo;t work: the grid approach My first attempt was clever but wrong. I created a 2x2 grid showing all four possible orientations of each image (original, mirrored, rotated 180°, mirrored+rotated), labeled A through D, and asked the model to pick which one looked correct.\n┌─────────┬─────────┐ │ A: orig │ B: mirr │ ├─────────┼─────────┤ │ C: 180° │ D: both │ └─────────┴─────────┘ The model said \u0026ldquo;A\u0026rdquo; for every single image. The grid made each sub-image too small to reason about, and the model defaulted to the first option. Five for five wrong.\nWhat worked: one question at a time Splitting the problem into a simple binary question on the full-resolution image worked much better. The prompt:\nLook at this scanned photograph. Is it UPSIDE DOWN? Check these things: - Are people\u0026#39;s heads at the BOTTOM of the image? That means upside down. - Is the sky or ceiling at the BOTTOM? That means upside down. - Are buildings, trees, or poles pointing DOWNWARD? That means upside down. - Is the ground or floor at the TOP of the image? That means upside down. Respond in EXACTLY this format (two lines, nothing else): UPSIDE_DOWN: YES or NO REASON: Brief explanation On a test set of 5 images (3 upside-down, 2 correct), this got all 5 right. The key insight: give the model the full image at a reasonable resolution, ask one simple question, and tell it exactly what format to respond in.\nBlur detection for free While I was processing each image, I also wanted to flag blurry scans that might need to be re-done. This part doesn\u0026rsquo;t need an LLM at all — the Laplacian variance method works well:\ndef calculate_blur_score(image_path): img = cv2.imread(str(image_path), cv2.IMREAD_GRAYSCALE) # Normalize for different scan resolutions h, w = img.shape if max(h, w) \u0026gt; 1024: scale = 1024 / max(h, w) img = cv2.resize(img, (int(w * scale), int(h * scale))) return float(cv2.Laplacian(img, cv2.CV_64F).var()) The Laplacian operator approximates the second derivative of the image — sharp edges produce high values, blurry regions produce low values. The variance of the Laplacian across the whole image gives a single sharpness score. Higher is sharper.\nA threshold of 100 worked well for these scans. Anything below that was visibly soft.\nSampling before committing Running 10,000 images through a vision model at 15 seconds each would take about 42 hours. Before committing to that, I sampled 3 random images from each of the 33 folders to estimate which ones had problems:\nFolder Total Sampled Upside↓ % ------------------------- ------ -------- -------- ------ Batch 01 81 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 02 70 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 03 79 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 04 81 3 2 67% \u0026lt;\u0026lt;\u0026lt; Batch 05 78 3 1 33% Batch 06 541 3 1 33% Batch 07 441 3 1 33% Batch 08 420 3 0 0% Batch 09 1305 3 0 0% Batch 10 1085 3 0 0% Batch 11 1373 3 0 0% ... 99 sample images, 13 minutes, and I had a priority map. Four folders had 67% upside-down rates — those ~311 images should be processed first. The large folders (4,000+ images) looked clean. The sampling pass likely saved 30+ hours of unnecessary processing.\nThe script The full script is about 400 lines of Python. The core loop is straightforward:\nfor img_path in image_files: # Blur score (instant, no LLM needed) blur_score = calculate_blur_score(img_path) # Orientation check (sends image to Ollama) img_b64 = image_to_b64(img_path, max_size=1024) text = ask_vision(ollama_url, model, UPSIDE_DOWN_PROMPT, img_b64) is_upside_down = parse_response(text) results.append(ImageResult( path=str(img_path), blur_score=blur_score, orientation=\u0026#34;C\u0026#34; if is_upside_down else \u0026#34;A\u0026#34;, correction=\u0026#34;rotate_180\u0026#34; if is_upside_down else \u0026#34;none\u0026#34;, )) It generates an HTML report with side-by-side thumbnails (as-scanned vs. corrected) so you can visually verify before applying fixes. The fixes themselves are non-destructive — each corrected image gets a .original backup:\n# Review first python slide_analyzer.py \u0026#34;/Volumes/slides/batch-01\u0026#34; # Apply corrections from the report python slide_analyzer.py --fix-from slide_report.json Performance Metric Value Model llama3.2-vision (11B) Hardware Mac Studio M1 Ultra, 64 GB Time per image ~15 seconds Accuracy (test set) 5/5 Sample pass (99 images) 13 minutes Estimated full run (10,000 images) ~42 hours Estimated full run (priority folders only) ~1.3 hours What I\u0026rsquo;d do differently Mirror detection is hard. I initially tried to detect horizontal flips too, but the model was unreliable — it flagged correctly-oriented images as mirrored. Without readable text in the photo, there often aren\u0026rsquo;t enough visual cues. Mirror detection probably needs a more capable model or a different approach (OCR on the image, then check if the text is backwards).\nA larger model might help. The 11B llama3.2-vision worked well for upside-down detection, which is a relatively easy spatial reasoning task. Mirror detection is subtler and might benefit from a 34B+ parameter model. With 64 GB of RAM, llava:34b would fit, but I haven\u0026rsquo;t tested it yet.\nBatch the Ollama calls. The current script processes images sequentially because Ollama handles one inference at a time on a single GPU. If you had multiple GPUs or were using a cloud API, you could parallelize this significantly.\nConclusion The combination of \u0026ldquo;cheap classical CV for the easy stuff\u0026rdquo; (blur detection) and \u0026ldquo;local vision LLM for the semantic stuff\u0026rdquo; (orientation detection) turned a multi-day manual review into something that runs overnight. The sampling strategy — check a few images per folder before processing everything — is the kind of optimization that seems obvious in retrospect but saves enormous amounts of time.\nThe script, the Ollama model, and the NAS are all local. No images leave the network, no API costs, no rate limits. That matters when the photos are your grandfather\u0026rsquo;s personal history.\n","permalink":"https://www.rustybower.com/posts/fixing-upside-down-scanned-slides-vision-llm/","summary":"\u003cp\u003eMy grandfather had about 10,000 35mm slides. I rented a \u003ca href=\"https://www.slidesnap.com/\"\u003eSlideSnap X1\u003c/a\u003e and spent a weekend feeding them through — 33 boxes worth, organized into folders by box. The scanner itself was great, but when you\u0026rsquo;re pushing through thousands of slides in a weekend, some inevitably go in upside down or backwards. No metadata, no EXIF orientation flags — just thousands of JPEGs, some right-side up, some not, sitting on a NAS.\u003c/p\u003e","title":"Fixing 10,000 Upside-Down Scanned Slides with a Local Vision LLM"},{"content":"CMS recently published provider-level Medicaid spending data from T-MSIS — every fee-for-service, managed care, and CHIP claim from 2018 through 2024, aggregated by billing provider, procedure code, and month. 227 million rows. $1.09 trillion in payments. I wanted to see what falls out when you run some basic fraud heuristics against it.\nThe dataset is available at opendata.hhs.gov/datasets/medicaid-provider-spending/.\nThe dataset The download is a single 2.94 GB parquet file with seven columns:\nColumn Type Description BILLING_PROVIDER_NPI_NUM string NPI of the billing provider SERVICING_PROVIDER_NPI_NUM string NPI of the servicing provider HCPCS_CODE string Procedure code CLAIM_FROM_MONTH string Month (YYYY-MM) TOTAL_UNIQUE_BENEFICIARIES int64 Unique patients TOTAL_CLAIMS int64 Claim count TOTAL_PAID double Dollars paid by Medicaid Here\u0026rsquo;s what the first few rows look like — the data is sorted by TOTAL_PAID descending, so the biggest line items are at the top:\n┌──────────────────────────┬────────────────────────────┬────────────┬──────────────────┬────────────────────────────┬──────────────┬──────────────┐ │ BILLING_PROVIDER_NPI_NUM │ SERVICING_PROVIDER_NPI_NUM │ HCPCS_CODE │ CLAIM_FROM_MONTH │ TOTAL_UNIQUE_BENEFICIARIES │ TOTAL_CLAIMS │ TOTAL_PAID │ ├──────────────────────────┼────────────────────────────┼────────────┼──────────────────┼────────────────────────────┼──────────────┼──────────────┤ │ 1376609297 │ 1376609297 │ T1019 │ 2024-07 │ 39765 │ 1205701 │ 118887675.31 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-08 │ 39677 │ 1152534 │ 115561066.11 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-05 │ 39678 │ 1157235 │ 112823255.3 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-06 │ 39834 │ 1164582 │ 111449173.13 │ │ 1376609297 │ 1376609297 │ T1019 │ 2024-09 │ 39527 │ 1099808 │ 111199832.57 │ └──────────────────────────┴────────────────────────────┴────────────┴──────────────────┴────────────────────────────┴──────────────┴──────────────┘ Right away you notice NPI 1376609297 billing $118M in a single month for T1019 (personal care services) with 1.2 million claims. That\u0026rsquo;s going to come up again.\nA quick summary of the full dataset:\ncon.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT COUNT(*) as total_rows, COUNT(DISTINCT BILLING_PROVIDER_NPI_NUM) as unique_billing_npis, COUNT(DISTINCT SERVICING_PROVIDER_NPI_NUM) as unique_servicing_npis, COUNT(DISTINCT HCPCS_CODE) as unique_procedures, MIN(CLAIM_FROM_MONTH) as earliest_month, MAX(CLAIM_FROM_MONTH) as latest_month, SUM(TOTAL_PAID) as total_paid FROM \u0026#39;{PATH}\u0026#39; \u0026#34;\u0026#34;\u0026#34;) 227M rows, 617,503 billing providers, 1.6M servicing providers, 10,881 procedure codes, spanning 84 months (Jan 2018 – Dec 2024), totaling $1.09 trillion.\nWhy parquet, not CSV HHS offers this dataset as a ZIP download. Inside is parquet, not CSV. This matters a lot for a file this size.\nCompression. The parquet file is 2.94 GB. The same data in CSV would be roughly 15-20 GB. Parquet uses column-oriented compression (dictionary encoding for the repeated NPI strings, run-length encoding for sorted columns, and Snappy/ZSTD compression on top). The three NPI/code string columns have high cardinality but lots of repetition within row groups — exactly the pattern parquet compresses best.\nColumn pruning. If your query only touches BILLING_PROVIDER_NPI_NUM and TOTAL_PAID, parquet readers skip the other five columns entirely. With CSV, you always read every byte of every row. For a 7-column file, that\u0026rsquo;s up to 5/7ths of I/O you never have to do.\nPredicate pushdown. DuckDB can push WHERE clauses down into the parquet scan, skipping entire row groups whose min/max statistics prove no rows match. When I filter WHERE TOTAL_CLAIMS \u0026gt; 1000, DuckDB doesn\u0026rsquo;t even read row groups where the max claims value is under 1,000.\nType preservation. CSV makes you guess types and parse strings. Parquet knows TOTAL_PAID is a double and TOTAL_CLAIMS is an int64 — no parsing overhead, no type ambiguity.\nFor a dataset with 227M rows, these differences compound. The parquet file isn\u0026rsquo;t just smaller — it\u0026rsquo;s fundamentally faster to query.\nDon\u0026rsquo;t load 227M rows into pandas My first attempt was naive:\nimport pandas as pd df = pd.read_parquet(\u0026#39;/data/medicaid-provider-spending.parquet\u0026#39;) The Jupyter kernel OOM-killed immediately on a 16 GB pod.\nThe math makes it obvious in hindsight. Parquet decompresses to a much larger in-memory representation, and pandas stores each string as a Python object with ~56 bytes of overhead regardless of string length. Three string columns across 227M rows:\n227M rows × 3 string columns × ~56 bytes/object ≈ 38 GB + 2 int64 columns × 8 bytes × 227M ≈ 3.6 GB + 1 float64 column × 8 bytes × 227M ≈ 1.8 GB ≈ 43 GB total Even a 64 GB pod would be tight once you start doing groupbys that create intermediate DataFrames.\nThe fix: DuckDB. It queries parquet files directly on disk using memory-mapped I/O, streaming through row groups without materializing the full dataset. Peak memory usage stays in the low single-digit GBs. Every query in this analysis ran in under 3 minutes on that same 16 GB pod.\nimport duckdb PATH = \u0026#39;/data/medicaid-provider-spending.parquet\u0026#39; con = duckdb.connect() con.sql(f\u0026#34;SELECT COUNT(*) FROM \u0026#39;{PATH}\u0026#39;\u0026#34;) # → (227083361,) No read_parquet(), no DataFrames, no OOM. DuckDB treats the parquet file as a table and pushes the full query plan — filters, aggregations, window functions — down to the scan.\nThe trick is to do all the heavy lifting in DuckDB SQL, then call .df() only on the final result sets (which are thousands of rows, not millions):\n# Heavy aggregation stays in DuckDB — result is ~68K rows, perfectly fine for pandas outliers = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT ... FROM \u0026#39;{PATH}\u0026#39; GROUP BY ... HAVING z_score \u0026gt; 3 \u0026#34;\u0026#34;\u0026#34;).df() Five fraud signals Healthcare fraud detection comes down to finding providers whose billing patterns deviate sharply from their peers. I built five independent signals and combined them into a composite risk score. Here\u0026rsquo;s every query.\n1. Cost-per-beneficiary outliers For each provider and procedure code, I sum the total dollars paid and divide by unique beneficiaries across all months. Then z-score within each procedure code (restricting to procedures with 20+ providers for stable statistics). Anything above z=3 is flagged.\noutliers_cost = con.sql(f\u0026#34;\u0026#34;\u0026#34; WITH prov_proc AS ( SELECT BILLING_PROVIDER_NPI_NUM, HCPCS_CODE, SUM(TOTAL_PAID) as total_paid, SUM(TOTAL_UNIQUE_BENEFICIARIES) as total_benes, SUM(TOTAL_CLAIMS) as total_claims, COUNT(DISTINCT CLAIM_FROM_MONTH) as months_active, SUM(TOTAL_PAID) / GREATEST(SUM(TOTAL_UNIQUE_BENEFICIARIES), 1) as paid_per_bene FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM, HCPCS_CODE ), proc_stats AS ( SELECT HCPCS_CODE, COUNT(*) as num_providers, AVG(paid_per_bene) as mean_ppb, STDDEV_POP(paid_per_bene) as std_ppb FROM prov_proc GROUP BY HCPCS_CODE HAVING COUNT(*) \u0026gt;= 20 ), scored AS ( SELECT p.*, (p.paid_per_bene - s.mean_ppb) / NULLIF(s.std_ppb, 0) as z_paid_per_bene FROM prov_proc p JOIN proc_stats s USING (HCPCS_CODE) ) SELECT * FROM scored WHERE z_paid_per_bene \u0026gt; 3 ORDER BY z_paid_per_bene DESC \u0026#34;\u0026#34;\u0026#34;).df() This scans the full 2.94 GB parquet file, builds the two-level aggregation entirely in DuckDB, and returns only the outliers. It ran in 192 seconds.\nResult: 68,037 provider/procedure combinations flagged across 27,348 unique billing NPIs.\nThe top hit: NPI 1467653303 billing $10,416 per beneficiary for CPT 99213 — a standard 15-minute office visit that typically pays around $57. A z-score of 183. They billed $541K for 52 beneficiaries in a single month. Either those were the most expensive office visits in the history of medicine, or something\u0026rsquo;s wrong.\nOther standouts:\nNPI 1437412525: $22,064/beneficiary for 92507 (speech therapy), z=94 NPI 1073608998: $65,778/beneficiary for J3490 (unclassified drugs), z=68 NPI 1427138726: $8,817/beneficiary for 99232 (subsequent hospital care), z=62 — and they did it across 16 months, billing $6M total 2. Billing mill detection In Medicaid, the billing provider is the entity that submits the claim, and the servicing provider is the one who actually performed the service. Most providers bill for their own services — BILLING_PROVIDER_NPI_NUM = SERVICING_PROVIDER_NPI_NUM. A billing mill funnels claims through many servicing providers under one billing entity, taking a cut.\nbilling_mill = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT BILLING_PROVIDER_NPI_NUM, COUNT(DISTINCT SERVICING_PROVIDER_NPI_NUM) as num_servicing_npis, COUNT(DISTINCT HCPCS_CODE) as num_procedures, SUM(TOTAL_PAID) as total_paid, SUM(TOTAL_CLAIMS) as total_claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as total_benes, COUNT(DISTINCT CLAIM_FROM_MONTH) as months_active FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM ORDER BY num_servicing_npis DESC \u0026#34;\u0026#34;\u0026#34;).df() The distribution tells the story:\ncount 617,503 mean 4.6 std 33.0 25% 1.0 50% 1.0 ← median is 1 75% 1.0 ← 75th percentile is still 1 max 5,746 The median and 75th percentile are both 1. Most providers bill for themselves. Then there\u0026rsquo;s NPI 1679525919 with 5,746 servicing NPIs, billing $863 million across 84 months with 1,579 distinct procedure codes. That\u0026rsquo;s either a massive health system or something worth investigating.\nI flag the top 1% by servicing NPI count as potential billing mills.\n3. Volume impossibilities Each row in this dataset is already aggregated to one billing-provider/servicing-provider/procedure/month combination. So when TOTAL_CLAIMS exceeds 1,000 for a single row, that means one provider billed over 1,000 claims for one procedure code in one month — averaging 50+ per working day with zero days off.\nhigh_volume = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT BILLING_PROVIDER_NPI_NUM, SERVICING_PROVIDER_NPI_NUM, HCPCS_CODE, CLAIM_FROM_MONTH, TOTAL_CLAIMS, TOTAL_UNIQUE_BENEFICIARIES, TOTAL_PAID FROM \u0026#39;{PATH}\u0026#39; WHERE TOTAL_CLAIMS \u0026gt; 1000 ORDER BY TOTAL_CLAIMS DESC \u0026#34;\u0026#34;\u0026#34;).df() Result: 1,783,926 rows exceeded 1,000 claims/month across 30,143 billing NPIs, totaling $391 billion.\nThe single most extreme: NPI 1225163876 billed 1,607,071 claims in February 2022 for code 1286Z — but only to 1,906 beneficiaries, and was paid just $230K. That\u0026rsquo;s 842 claims per beneficiary in a single month. The low payment amount suggests these may be bulk capitation or encounter records rather than fee-for-service fraud, but the ratio is still bizarre.\nThe real volume monster is NPI 1376609297 (the same provider from the first row of the dataset) — they consistently bill 1.1-1.2 million claims per month for T1019, each month paying out $100M+. They occupy 19 of the top 20 highest-volume rows.\n4. Spending spike detection Fraudulent providers often show a \u0026ldquo;ramp and run\u0026rdquo; pattern — billing spikes dramatically before the entity disappears or gets caught. I used LAG() to compute month-over-month spending growth per provider and flagged anything with \u0026gt;5x growth where the spike month exceeded $50K:\nspikes = con.sql(f\u0026#34;\u0026#34;\u0026#34; WITH monthly AS ( SELECT BILLING_PROVIDER_NPI_NUM, CLAIM_FROM_MONTH, SUM(TOTAL_PAID) as monthly_paid, SUM(TOTAL_CLAIMS) as monthly_claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as monthly_benes FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM, CLAIM_FROM_MONTH ), with_prev AS ( SELECT *, LAG(monthly_paid) OVER ( PARTITION BY BILLING_PROVIDER_NPI_NUM ORDER BY CLAIM_FROM_MONTH ) as prev_paid FROM monthly ) SELECT BILLING_PROVIDER_NPI_NUM, CLAIM_FROM_MONTH, prev_paid, monthly_paid, monthly_paid / GREATEST(prev_paid, 1) as growth_ratio, monthly_claims, monthly_benes FROM with_prev WHERE monthly_paid / GREATEST(prev_paid, 1) \u0026gt; 5 AND monthly_paid \u0026gt; 50000 AND prev_paid IS NOT NULL ORDER BY growth_ratio DESC \u0026#34;\u0026#34;\u0026#34;).df() This query aggregates 227M rows to provider-month level, computes lag-based growth ratios via a window function, and filters — all pushed down into DuckDB\u0026rsquo;s execution engine. It ran in 91 seconds.\nResult: 18,916 spike events across 13,527 providers.\nThe top spikes all share a pattern: prev_paid of $0 (or near-zero) followed by millions in the next month. NPI 1770700221 went from $0 to $4.76M in July 2023. NPI 1336117670 went from $0 to $4.74M in February 2018. These are providers that appear out of nowhere with massive billing.\nNPI 1144347824 is interesting — they appear four times in the top 20 with spikes in Dec 2020, Feb 2021, Apr 2021, and Jun 2021. Repeated $0-to-$550K+ cycles suggest an on-again-off-again billing pattern.\n5. Procedure concentration Most providers bill across multiple procedure codes, even within a narrow specialty. A provider deriving \u0026gt;95% of all Medicaid revenue from a single HCPCS code at high dollar volumes can indicate upcoding (always picking the most expensive code) or phantom billing (billing for services never rendered using a single code).\nconcentrated = con.sql(f\u0026#34;\u0026#34;\u0026#34; WITH prov_proc AS ( SELECT BILLING_PROVIDER_NPI_NUM, HCPCS_CODE, SUM(TOTAL_PAID) as total_paid, SUM(TOTAL_CLAIMS) as total_claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as total_benes FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM, HCPCS_CODE ), prov_total AS ( SELECT BILLING_PROVIDER_NPI_NUM, SUM(total_paid) as provider_total FROM prov_proc GROUP BY BILLING_PROVIDER_NPI_NUM ), ranked AS ( SELECT p.*, t.provider_total, p.total_paid / t.provider_total as pct_of_total, ROW_NUMBER() OVER ( PARTITION BY p.BILLING_PROVIDER_NPI_NUM ORDER BY p.total_paid DESC ) as rn FROM prov_proc p JOIN prov_total t USING (BILLING_PROVIDER_NPI_NUM) WHERE t.provider_total \u0026gt; 500000 ) SELECT * FROM ranked WHERE rn = 1 AND pct_of_total \u0026gt; 0.95 ORDER BY provider_total DESC \u0026#34;\u0026#34;\u0026#34;).df() Result: 25,397 providers with \u0026gt;$500K total and \u0026gt;95% from one code, representing $193 billion.\nThe dominant code across the top 20 is T1019 (personal care services). NPI 1376609297 tops the list at $5.5 billion, 98% from T1019. The next five are also T1019 providers at $1-3B each. These are likely large home and community-based services (HCBS) agencies — the concentration on one code may be legitimate program design, but the scale demands scrutiny.\nNPI 1932341898 stands out: $997M, 99.99% from H0044 (supported employment), with only two procedure codes total. That\u0026rsquo;s extreme even for this list.\nComposite scoring Each provider gets a binary flag for each of the five signals. The composite fraud score is just the sum.\nimport pandas as pd all_providers = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT BILLING_PROVIDER_NPI_NUM, SUM(TOTAL_PAID) as total_spending FROM \u0026#39;{PATH}\u0026#39; GROUP BY BILLING_PROVIDER_NPI_NUM \u0026#34;\u0026#34;\u0026#34;).df() risk = all_providers.copy() flag1_npis = set(outliers_cost[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_cost_outlier\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag1_npis).astype(int) threshold_mill = billing_mill[\u0026#39;num_servicing_npis\u0026#39;].quantile(0.99) flag2_npis = set(billing_mill[ billing_mill[\u0026#39;num_servicing_npis\u0026#39;] \u0026gt;= threshold_mill ][\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_billing_mill\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag2_npis).astype(int) flag3_npis = set(high_volume[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_high_volume\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag3_npis).astype(int) flag4_npis = set(spikes[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_spike\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag4_npis).astype(int) flag5_npis = set(concentrated[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;]) risk[\u0026#39;flag_concentrated\u0026#39;] = risk[\u0026#39;BILLING_PROVIDER_NPI_NUM\u0026#39;] \\ .isin(flag5_npis).astype(int) flag_cols = [ \u0026#39;flag_cost_outlier\u0026#39;, \u0026#39;flag_billing_mill\u0026#39;, \u0026#39;flag_high_volume\u0026#39;, \u0026#39;flag_spike\u0026#39;, \u0026#39;flag_concentrated\u0026#39; ] risk[\u0026#39;fraud_score\u0026#39;] = risk[flag_cols].sum(axis=1) This is the one step where I use pandas — the DuckDB queries already reduced 227M rows down to manageable result sets (tens of thousands of NPIs), so the set-membership lookups and joins are trivial.\nDistribution:\nScore Providers 0 539,161 1 57,424 2 17,690 3 3,034 4 194 2+ 20,918 The 20,918 providers with 2+ flags account for $535 billion — nearly half of all Medicaid spending in the dataset.\nThe top 5 For the highest-scoring providers, I pulled their full procedure breakdowns and monthly spending ranges:\nfor npi in top5: proc_breakdown = con.sql(f\u0026#34;\u0026#34;\u0026#34; SELECT HCPCS_CODE, SUM(TOTAL_PAID) as paid, SUM(TOTAL_CLAIMS) as claims, SUM(TOTAL_UNIQUE_BENEFICIARIES) as benes FROM \u0026#39;{PATH}\u0026#39; WHERE BILLING_PROVIDER_NPI_NUM = \u0026#39;{npi}\u0026#39; GROUP BY HCPCS_CODE ORDER BY paid DESC LIMIT 5 \u0026#34;\u0026#34;\u0026#34;).df() NPI 1700090834 — Score 4/5 ($1.13B) Flags: cost outlier, billing mill, high volume, spike\nHCPCS_CODE paid claims benes pct H0019 553,957,700 2,884,496 155,197 53.2% H0004 188,211,147 1,572,012 505,021 18.1% H0005 108,477,327 1,623,733 203,179 10.4% H0020 103,053,473 7,020,805 276,149 9.9% H0006 88,021,893 1,166,081 255,743 8.4% Monthly spending range: $37,576 — $25,877,770 All H-codes (behavioral health). H0019 alone is $554M for day treatment services across 155K beneficiaries. The monthly range swinging from $38K to $25.9M means there were massive ramp-up periods. This has the profile of a large behavioral health managed care entity — but one that also trips cost outlier and billing mill flags.\nNPI 1932341898 — Score 4/5 ($997M) Flags: cost outlier, high volume, spike, concentrated\nHCPCS_CODE paid claims benes pct H0044 996,688,991 304,663 283,937 100.0% T2028 112,131 1,468 36 0.0% Monthly spending range: $512,298 — $21,818,712 $997M from a single code. H0044 is supported employment — helping people with disabilities find and maintain jobs. 283,937 beneficiaries at an average of $3,511 each. Only two procedure codes ever billed. The 43x spread between minimum and maximum monthly spending ($512K to $21.8M) is hard to explain with normal program growth.\nNPI 1114931391 — Score 4/5 ($382M) Flags: cost outlier, high volume, spike, concentrated\nHCPCS_CODE paid claims benes pct S3620 370,523,694 1,324,711 974,114 97.1% 83655 8,882,287 685,614 667,070 2.3% 85018 1,601,721 623,389 607,346 0.4% 80061 691,330 45,838 44,922 0.2% 82947 49,396 11,154 10,951 0.0% Monthly spending range: $47 — $15,497,422 97% from S3620 (newborn screening panel). The monthly range from $47 to $15.5M is the most extreme spread in the top 5. This likely represents a state-contracted newborn screening lab — the code and beneficiary counts support that. But a $47 month to $15.5M month means either the contract changed drastically or there are data quality issues. The secondary codes (83655, 85018) are basic lab tests, consistent with a lab operation.\nNPI 1629283197 — Score 4/5 ($306M) Flags: billing mill, high volume, spike, concentrated\nHCPCS_CODE paid claims benes pct T1015 298,897,046 718,309 655,601 99.1% 99393 722,772 52,037 39,635 0.2% 99392 688,366 47,287 38,251 0.2% 99394 608,918 31,656 24,001 0.2% 92014 591,683 27,987 16,169 0.2% Monthly spending range: $594,623 — $5,028,749 99% from T1015 (clinic-based behavioral health). The secondary codes (99392-99394) are preventive care E\u0026amp;M visits, and 92014 is an eye exam — suggesting this billing entity has a clinic component too. But the billing mill flag means claims are flowing through many servicing NPIs under this one billing number.\nNPI 1619341716 — Score 4/5 ($303M) Flags: cost outlier, billing mill, high volume, spike\nHCPCS_CODE paid claims benes pct 99211 45,853,871 488,964 412,175 34.6% 90832 37,594,923 253,130 125,580 28.4% 99213 29,286,349 255,935 204,037 22.1% 99214 9,967,573 88,118 73,463 7.5% 99212 9,740,312 66,604 55,192 7.4% Monthly spending range: $66,177 — $7,160,676 The most diversified billing pattern of the top 5. A mix of E\u0026amp;M visits (99211-99214) and psychotherapy (90832). The cost outlier flag means they\u0026rsquo;re charging significantly more per beneficiary than peers for these common codes. The billing mill flag means many servicing NPIs are operating under this one billing entity. The 108x spread between min and max monthly billing ($66K to $7.2M) indicates major scaling events.\nCaveats These are flags, not findings. Several patterns have legitimate explanations:\nLarge health systems and MCOs naturally have many servicing NPIs — that\u0026rsquo;s not fraud, it\u0026rsquo;s organizational structure. The billing mill signal needs to be interpreted in context. State-contracted labs (like the newborn screening provider at NPI 1114931391) can have legitimate volume and spending spikes tied to contract awards and state program changes. T1019/T1015/H-codes are commonly used in home and community-based services (HCBS) and behavioral health, where high volumes per billing entity may reflect program design rather than fraud. States often contract with large managed care entities for these services. This data is aggregated — we can\u0026rsquo;t see individual claim details, diagnosis codes, patient demographics, or service locations. Many fraud patterns only become visible at the claim level. The real value is in combining signals. A provider that\u0026rsquo;s both a cost outlier and has billing mill patterns and shows spending spikes is far more suspicious than one that only trips a single wire. The 194 providers at score 4/5 are the ones I\u0026rsquo;d start investigating.\nTechnical notes The full analysis runs as a Jupyter notebook backed by DuckDB on a 16 GB Kubernetes pod. Some implementation details worth noting:\nDuckDB\u0026rsquo;s parquet performance is remarkable. The z-score query (Signal 1) does a full scan → two-level aggregation → window function → filter, all on 227M rows of parquet. It completes in 192 seconds. The spending spike query uses LAG() window functions over the aggregated monthly data — 91 seconds. These are the kinds of queries that would require Spark or a data warehouse for most teams.\nThe pandas handoff is intentional. DuckDB returns result sets via .df() only after reducing 227M rows to thousands. The composite scoring step — five set-membership lookups and a sum — is trivial in pandas. There\u0026rsquo;s no reason to push that into SQL.\nGREATEST(x, 1) prevents division by zero. Several providers have zero beneficiaries or zero prior-month spending. Rather than filtering them out (and potentially missing interesting cases), I clamp the denominator to 1. This means zero-bene providers get a per-bene cost equal to their total paid — which is correct behavior for flagging purposes.\n","permalink":"https://www.rustybower.com/posts/medicaid-fraud-detection-duckdb-227m-rows/","summary":"\u003cp\u003eCMS recently published provider-level Medicaid spending data from T-MSIS — every fee-for-service, managed care, and CHIP claim from 2018 through 2024, aggregated by billing provider, procedure code, and month. 227 million rows. $1.09 trillion in payments. I wanted to see what falls out when you run some basic fraud heuristics against it.\u003c/p\u003e\n\u003cp\u003eThe dataset is available at \u003ca href=\"https://opendata.hhs.gov/datasets/medicaid-provider-spending/\"\u003eopendata.hhs.gov/datasets/medicaid-provider-spending/\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 id=\"the-dataset\"\u003eThe dataset\u003c/h2\u003e\n\u003cp\u003eThe download is a single 2.94 GB parquet file with seven columns:\u003c/p\u003e","title":"Finding Fraud in $1 Trillion of Medicaid Data with DuckDB"},{"content":"My Home Assistant dashboard has evolved significantly over the past year. What started as the default auto-generated cards has become a carefully organized interface that my whole family actually uses. Here\u0026rsquo;s how I built it, with a focus on the whole home audio system that ties everything together.\nDashboard Philosophy Before diving into implementation, a few principles guided the design:\nFunction over flash - Every card earns its screen space One-tap actions - Common tasks shouldn\u0026rsquo;t require drilling into menus Status at a glance - The overview tells you what\u0026rsquo;s happening without interaction Mobile-first - Tablets mounted around the house are the primary interface The Stack Lovelace YAML mode - Full control over layout and structure Mushroom cards - Clean, consistent UI components Custom Layout Card - Responsive grid layouts that work on any screen Stack-in-card - Grouping related controls together Overview Page The overview page is designed to answer \u0026ldquo;what\u0026rsquo;s happening right now?\u0026rdquo; at a glance:\n- type: custom:mushroom-chips-card alignment: center chips: - type: weather entity: weather.katw - type: template entity: person.rusty_bower icon: mdi:account icon_color: \u0026#34;{{ \u0026#39;green\u0026#39; if is_state(\u0026#39;person.rusty_bower\u0026#39;, \u0026#39;home\u0026#39;) else \u0026#39;grey\u0026#39; }}\u0026#34; content: Rusty - type: template icon: mdi:lightbulb-group icon_color: \u0026#34;{{ \u0026#39;amber\u0026#39; if states.light | selectattr(\u0026#39;state\u0026#39;, \u0026#39;eq\u0026#39;, \u0026#39;on\u0026#39;) | list | count \u0026gt; 0 else \u0026#39;disabled\u0026#39; }}\u0026#34; content: \u0026#34;{{ states.light | selectattr(\u0026#39;state\u0026#39;, \u0026#39;eq\u0026#39;, \u0026#39;on\u0026#39;) | list | count }} lights\u0026#34; The chip bar at the top shows:\nCurrent weather Who\u0026rsquo;s home (person tracking) How many lights are on (tap to navigate to lighting) Garage door status (red = open, green = closed) HVAC status (heating/cooling active) Upcoming trash/recycling days Quick actions provide one-tap scenes: Goodnight, Spa Mode, All Lights Off, Close All Garage Doors.\nWhole Home Audio Architecture The audio system uses a Xantech DAX66 multi-zone amplifier - a commercial-grade matrix amplifier that can route any of 6 sources to any of 18 zones independently. Each zone has its own volume control and can play different content simultaneously.\nThe Zones The house is wired with in-ceiling speakers across 15 zones spanning three floors - main living areas, bedrooms/offices upstairs, and utility/recreation spaces in the basement. Plus a dedicated Theater with an Anthem AV receiver for serious listening.\nIntegration Approach The Xantech communicates via RS-232, but I needed network access from Kubernetes. The solution is a socat bridge running as a sidecar:\ncontainers: - name: socat image: alpine/socat:latest args: - \u0026#34;-d\u0026#34; - \u0026#34;-d\u0026#34; - \u0026#34;TCP-LISTEN:7001,reuseaddr,fork\u0026#34; - \u0026#34;TCP:10.0.10.209:4999,keepalive,forever,interval=10\u0026#34; This bridges TCP connections from Home Assistant to the serial-to-ethernet adapter connected to the Xantech. The keepalive,forever ensures the connection recovers from network hiccups.\nHome Assistant sees each zone as a media_player entity with source selection and volume control.\nAudio Dashboard Design Designing the audio UI was tricky. With 15 zones, a traditional media player card for each would be overwhelming. The solution is a two-column layout: streaming controls on the left, zone controls on the right.\nStreaming Column The maxi-media-player card provides a full-featured player with album art as a blurred background:\n- type: custom:maxi-media-player entities: - media_player.spotify - media_player.zone_1 - media_player.zone_2 # ... all zones sections: - player artworkAsBackgroundBlur: true This gives you playback controls, track info, and the ability to cast to any zone - all in one card.\nZone Controls Column For the individual zones, I use mushroom-media-player-card with a horizontal layout. Tap to toggle on/off, volume buttons for quick adjustments:\n- type: custom:stack-in-card cards: - type: custom:mushroom-title-card title: Whole Home Audio subtitle: Upstairs - type: custom:mushroom-media-player-card entity: media_player.zone_1 name: Zone 1 layout: horizontal volume_controls: - volume_buttons - volume_set media_controls: [] collapsible_controls: false tap_action: action: toggle - type: custom:mushroom-media-player-card entity: media_player.zone_2 name: Zone 2 layout: horizontal volume_controls: - volume_buttons - volume_set media_controls: [] collapsible_controls: false tap_action: action: toggle # ... more zones - type: custom:mushroom-title-card title: \u0026#34;\u0026#34; subtitle: Main Floor # ... main floor zones The key settings:\nlayout: horizontal - Compact single-line layout volume_controls: [volume_buttons, volume_set] - Both +/- buttons and a slider media_controls: [] - Hide play/pause since we control that from the streaming card tap_action: action: toggle - Tap the card to turn the zone on/off This puts all zones in a scrollable card organized by floor, with the streaming player always visible on the left.\nScene Integration The real power comes from combining audio with other automations:\nSpa Mode One tap: dims the outdoor lights, starts a relaxation playlist in the relevant zone, adjusts the HVAC.\n- type: custom:mushroom-template-card primary: Spa Mode icon: mdi:hot-tub icon_color: cyan tap_action: action: call-service service: scene.turn_on target: entity_id: scene.spa_mode Goodnight Turns off all audio zones, sets overnight temperature setpoints, dims lights to 5% in hallways, locks doors.\nParty Mode Syncs all main floor and outdoor zones to the same source at matched volumes.\nTheater Integration The theater runs separately on an Anthem AV receiver, integrated via IP control. It gets its own section with proper media controls:\n- type: custom:mushroom-media-player-card entity: media_player.anthem_av name: Anthem AV use_media_info: true show_volume_level: true volume_controls: - volume_set - volume_mute The theater lighting also has dedicated scene buttons: Off, Movie (5% seating lights), Intermission (50%), and Full brightness for cleaning.\nTablet Deployment Fire tablets run Fully Kiosk Browser showing the dashboard at key locations - one on the main floor for everyday use, one upstairs, and a portable tablet that docks at a charging station when not in use.\nThe Admin page shows tablet connectivity status so I know if one has gone offline.\nLessons Learned YAML mode is worth it - The visual editor can\u0026rsquo;t do responsive layouts or template chips Group by function, not location - \u0026ldquo;Lighting\u0026rdquo; view is more useful than \u0026ldquo;Living Room\u0026rdquo; view Limit visible options - Show the 80% use case, hide the edge cases behind taps Test with family - My initial designs were too information-dense What\u0026rsquo;s Next Music Assistant integration for better source management Voice control via local Whisper for \u0026ldquo;play jazz in the kitchen\u0026rdquo; Presence-based audio - automatically route music to follow occupancy The dashboard continues to evolve, but the core principle stays the same: make the smart home actually easier to use than the dumb version.\n","permalink":"https://www.rustybower.com/posts/home-assistant-dashboard-whole-home-audio/","summary":"\u003cp\u003eMy Home Assistant dashboard has evolved significantly over the past year. What started as the default auto-generated cards has become a carefully organized interface that my whole family actually uses. Here\u0026rsquo;s how I built it, with a focus on the whole home audio system that ties everything together.\u003c/p\u003e\n\u003ch2 id=\"dashboard-philosophy\"\u003eDashboard Philosophy\u003c/h2\u003e\n\u003cp\u003eBefore diving into implementation, a few principles guided the design:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eFunction over flash\u003c/strong\u003e - Every card earns its screen space\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eOne-tap actions\u003c/strong\u003e - Common tasks shouldn\u0026rsquo;t require drilling into menus\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStatus at a glance\u003c/strong\u003e - The overview tells you what\u0026rsquo;s happening without interaction\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMobile-first\u003c/strong\u003e - Tablets mounted around the house are the primary interface\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"the-stack\"\u003eThe Stack\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eLovelace YAML mode\u003c/strong\u003e - Full control over layout and structure\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMushroom cards\u003c/strong\u003e - Clean, consistent UI components\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCustom Layout Card\u003c/strong\u003e - Responsive grid layouts that work on any screen\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStack-in-card\u003c/strong\u003e - Grouping related controls together\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"overview-page\"\u003eOverview Page\u003c/h2\u003e\n\u003cp\u003eThe overview page is designed to answer \u0026ldquo;what\u0026rsquo;s happening right now?\u0026rdquo; at a glance:\u003c/p\u003e","title":"Building a Home Assistant Dashboard with Whole Home Audio Control"},{"content":"Managing DNS records for a homelab with dozens of services is tedious. Every time you deploy something new, you have to remember to add a DNS record. I solved this by using external-dns to automatically create Pi-hole DNS entries from Kubernetes ingress resources.\nThe Goal When I create an ingress like this:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: frigate annotations: external-dns.alpha.kubernetes.io/hostname: frigate.bowerha.us spec: rules: - host: frigate.bowerha.us # ... I want Pi-hole to automatically create a DNS record pointing frigate.bowerha.us to my ingress controller\u0026rsquo;s IP. No manual steps, no forgetting to update DNS.\nComponents Pi-hole - DNS server and ad blocker external-dns - Kubernetes controller that syncs DNS records Kubernetes ingress-nginx - Ingress controller with a known IP Setting Up external-dns Deployment apiVersion: apps/v1 kind: Deployment metadata: name: external-dns namespace: external-dns spec: replicas: 1 selector: matchLabels: app: external-dns template: metadata: labels: app: external-dns spec: containers: - name: external-dns image: registry.k8s.io/external-dns/external-dns:v0.14.0 args: - --source=ingress - --provider=pihole - --pihole-server=http://pihole-web.pihole.svc.cluster.local - --pihole-password=$(PIHOLE_PASSWORD) - --domain-filter=bowerha.us - --registry=noop - --policy=upsert-only - --interval=1m env: - name: PIHOLE_PASSWORD valueFrom: secretKeyRef: name: pihole-password key: password Key arguments:\n--source=ingress - Watch ingress resources for DNS records --provider=pihole - Use Pi-hole as the DNS backend --domain-filter=bowerha.us - Only manage records for this domain --policy=upsert-only - Create/update but never delete records --interval=1m - Check for changes every minute Pi-hole Password Secret apiVersion: v1 kind: Secret metadata: name: pihole-password namespace: external-dns type: Opaque stringData: password: \u0026#34;your-pihole-admin-password\u0026#34; RBAC external-dns needs permission to read ingresses:\napiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: external-dns rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;services\u0026#34;, \u0026#34;endpoints\u0026#34;, \u0026#34;pods\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;watch\u0026#34;, \u0026#34;list\u0026#34;] - apiGroups: [\u0026#34;extensions\u0026#34;, \u0026#34;networking.k8s.io\u0026#34;] resources: [\u0026#34;ingresses\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;watch\u0026#34;, \u0026#34;list\u0026#34;] - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;nodes\u0026#34;] verbs: [\u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: external-dns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: external-dns subjects: - kind: ServiceAccount name: external-dns namespace: external-dns Annotating Ingresses The key annotation is external-dns.alpha.kubernetes.io/hostname:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: home-assistant namespace: homeassistant annotations: cert-manager.io/cluster-issuer: letsencrypt-prod external-dns.alpha.kubernetes.io/hostname: ha.bowerha.us spec: ingressClassName: nginx rules: - host: ha.bowerha.us http: paths: - path: / pathType: Prefix backend: service: name: home-assistant port: number: 8123 tls: - hosts: - ha.bowerha.us secretName: home-assistant-tls Within a minute of applying this ingress, Pi-hole will have a local DNS record for ha.bowerha.us pointing to the ingress controller\u0026rsquo;s LoadBalancer IP.\nHow It Works external-dns watches for ingress resources with the hostname annotation It queries the ingress controller\u0026rsquo;s service to find its external IP It calls the Pi-hole API to create/update a Local DNS Record Pi-hole serves that record to all clients on the network The records appear in Pi-hole under Local DNS \u0026gt; DNS Records.\nTroubleshooting external-dns CrashLoopBackOff Usually means it can\u0026rsquo;t reach Pi-hole. Check:\nIs the Pi-hole service URL correct? Is Pi-hole actually running and responding on port 80? Is the password correct? I had an issue where Pi-hole\u0026rsquo;s FTL process got stuck after a volume issue. Restarting Pi-hole fixed external-dns.\nRecords Not Appearing Check external-dns logs:\nkubectl logs -n external-dns deployment/external-dns Look for:\nConnection errors to Pi-hole Domain filter mismatches Ingress resources missing the annotation Wrong IP Address external-dns gets the IP from the ingress controller\u0026rsquo;s service. If you\u0026rsquo;re using MetalLB or a cloud LoadBalancer, make sure the service has an external IP assigned:\nkubectl get svc -n ingress-nginx Making It Standard Practice I added this to my team\u0026rsquo;s deployment guidelines in CLAUDE.md:\n## DNS Management When creating ingress resources for `bowerha.us` domains, always include the external-dns annotation: annotations: external-dns.alpha.kubernetes.io/hostname: myapp.bowerha.us This automatically creates a Pi-hole DNS record pointing to the ingress controller. No manual DNS configuration needed. Now DNS is just part of the deployment - no separate step to remember.\nResult What used to be a manual process:\nDeploy app Create ingress Remember to add DNS Log into Pi-hole Add Local DNS record Test that it works Is now:\nDeploy app with annotated ingress Done The DNS record appears automatically within a minute. When I tear down services, I use --policy=upsert-only so records persist (useful for debugging), but you could use --policy=sync for automatic cleanup.\nThis small automation removes friction from deploying new services and eliminates a whole category of \u0026ldquo;why can\u0026rsquo;t I reach this?\u0026rdquo; debugging sessions.\n","permalink":"https://www.rustybower.com/posts/external-dns-pihole-kubernetes/","summary":"\u003cp\u003eManaging DNS records for a homelab with dozens of services is tedious. Every time you deploy something new, you have to remember to add a DNS record. I solved this by using external-dns to automatically create Pi-hole DNS entries from Kubernetes ingress resources.\u003c/p\u003e\n\u003ch2 id=\"the-goal\"\u003eThe Goal\u003c/h2\u003e\n\u003cp\u003eWhen I create an ingress like this:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003enetworking.k8s.io/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eIngress\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003efrigate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eannotations\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eexternal-dns.alpha.kubernetes.io/hostname\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003efrigate.bowerha.us\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erules\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ehost\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003efrigate.bowerha.us\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c\"\u003e# ...\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eI want Pi-hole to automatically create a DNS record pointing \u003ccode\u003efrigate.bowerha.us\u003c/code\u003e to my ingress controller\u0026rsquo;s IP. No manual steps, no forgetting to update DNS.\u003c/p\u003e","title":"Auto-Populating Pi-hole DNS from Kubernetes Ingresses"},{"content":"One of the challenges of running a homelab with dozens of services is keeping track of what\u0026rsquo;s running and where. I recently deployed Homepage - a modern, fully static dashboard that automatically discovers services from Kubernetes ingresses.\nWhy Homepage? I evaluated several dashboard options including Homarr and Heimdall. Homepage stood out for a few reasons:\nKubernetes-native service discovery - no manual configuration needed Real-time pod status - shows if services are actually running Clean, modern UI - dark theme, customizable layout Lightweight - just a static site with no database The Setup Homepage runs as a simple deployment in Kubernetes with a ServiceAccount that has read access to the cluster. The magic happens through ingress annotations.\nBase Deployment apiVersion: apps/v1 kind: Deployment metadata: name: homepage spec: replicas: 1 template: spec: serviceAccountName: homepage initContainers: - name: copy-config image: busybox:1.36 command: [\u0026#39;sh\u0026#39;, \u0026#39;-c\u0026#39;, \u0026#39;cp /config-source/* /config/\u0026#39;] volumeMounts: - name: config-source mountPath: /config-source - name: config mountPath: /config containers: - name: homepage image: ghcr.io/gethomepage/homepage:v0.10.9 volumeMounts: - name: config mountPath: /app/config volumes: - name: config-source configMap: name: homepage-config - name: config emptyDir: {} The init container pattern is important here - Homepage needs a writable config directory to create log files, but ConfigMaps are read-only. We copy the config to an emptyDir volume at startup.\nService Discovery via Annotations The real power comes from ingress annotations. For each service you want on the dashboard, add these annotations:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: frigate annotations: gethomepage.dev/enabled: \u0026#34;true\u0026#34; gethomepage.dev/name: \u0026#34;Frigate\u0026#34; gethomepage.dev/group: \u0026#34;Home Automation\u0026#34; gethomepage.dev/icon: \u0026#34;frigate.png\u0026#34; gethomepage.dev/pod-selector: \u0026#34;app=frigate\u0026#34; The pod-selector annotation is crucial for status monitoring. Homepage uses this to find the actual pods and show whether they\u0026rsquo;re running. Without it, you\u0026rsquo;ll see \u0026ldquo;not found\u0026rdquo; errors.\nConfigMap for Layout The ConfigMap controls the dashboard layout and widgets:\napiVersion: v1 kind: ConfigMap metadata: name: homepage-config data: settings.yaml: | title: Homelab theme: dark color: slate headerStyle: clean layout: Home Automation: style: row columns: 4 Media: style: row columns: 4 Infrastructure: style: row columns: 4 widgets.yaml: | - kubernetes: cluster: show: true cpu: true memory: true showLabel: true label: \u0026#34;cluster\u0026#34; nodes: show: true cpu: true memory: true kubernetes.yaml: | mode: cluster services.yaml: | [] docker.yaml: | Note that services.yaml and docker.yaml must be valid YAML (even if empty). Setting services.yaml: \u0026quot;[]\u0026quot; prevents JavaScript errors from empty files.\nGotchas I Encountered 1. Empty Config Files Cause JS Errors If services.yaml or docker.yaml are empty or contain only comments, Homepage throws a classList null reference error. Always set them to valid YAML.\n2. Pod Selector Mismatches Homepage defaults to looking for pods with app.kubernetes.io/name=\u0026lt;ingress-name\u0026gt;. Most of my deployments use simpler labels like app=frigate. The gethomepage.dev/pod-selector annotation fixes this.\n3. Icon Names Icons come from the Dashboard Icons project. Some names aren\u0026rsquo;t obvious:\nArgoCD: argo-cd.png (not argocd.png) Z-Wave JS: mdi-z-wave (Material Design Icons format) Result Now every time I add a new service with the proper annotations, it automatically appears on my dashboard with real-time status. No more manually updating bookmark pages or forgetting what port something runs on.\nThe dashboard shows CPU/memory usage for the cluster and nodes, groups services logically, and immediately tells me if something is down. It\u0026rsquo;s become my starting point for accessing everything in the homelab.\n","permalink":"https://www.rustybower.com/posts/homepage-kubernetes-service-discovery/","summary":"\u003cp\u003eOne of the challenges of running a homelab with dozens of services is keeping track of what\u0026rsquo;s running and where. I recently deployed \u003ca href=\"https://gethomepage.dev/\"\u003eHomepage\u003c/a\u003e - a modern, fully static dashboard that automatically discovers services from Kubernetes ingresses.\u003c/p\u003e\n\u003ch2 id=\"why-homepage\"\u003eWhy Homepage?\u003c/h2\u003e\n\u003cp\u003eI evaluated several dashboard options including Homarr and Heimdall. Homepage stood out for a few reasons:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eKubernetes-native service discovery\u003c/strong\u003e - no manual configuration needed\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReal-time pod status\u003c/strong\u003e - shows if services are actually running\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eClean, modern UI\u003c/strong\u003e - dark theme, customizable layout\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLightweight\u003c/strong\u003e - just a static site with no database\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"the-setup\"\u003eThe Setup\u003c/h2\u003e\n\u003cp\u003eHomepage runs as a simple deployment in Kubernetes with a ServiceAccount that has read access to the cluster. The magic happens through ingress annotations.\u003c/p\u003e","title":"Building a Self-Updating Dashboard with Homepage and Kubernetes"},{"content":"For years I used Keel to automatically update container images in my Kubernetes clusters. It worked, but as I moved to GitOps with ArgoCD, Keel\u0026rsquo;s push-based approach became a liability. I migrated to Renovate for PR-based image updates, and it\u0026rsquo;s been a significant improvement.\nThe Problem with Keel Keel watches for new container images and updates deployments directly in the cluster. You can configure it via annotations:\nmetadata: annotations: keel.sh/policy: major keel.sh/trigger: poll When a new image appears, Keel modifies the deployment in-place.\nWhy This Broke Down GitOps drift - Keel updates the cluster, but not git. ArgoCD sees drift and wants to revert to what\u0026rsquo;s in git. You end up fighting your own tools.\nNo review process - Updates happen automatically. A bad image goes live immediately. Rolling back means finding the old tag and manually updating.\nNotification-only visibility - Keel can send Slack notifications, but there\u0026rsquo;s no audit trail, no PR history, no way to see what changed when.\nLimited version control - Keel\u0026rsquo;s policies (major/minor/patch) are annotation-based and hard to customize per-image.\nRenovate: PR-Based Updates Renovate takes a different approach. It scans your repository, finds container images, checks for updates, and opens pull requests. The update only happens when you merge the PR.\nThis fits GitOps perfectly - git remains the source of truth.\nMy Renovate Setup Self-Hosted CronJob I run Renovate as a Kubernetes CronJob rather than using the hosted service:\napiVersion: batch/v1 kind: CronJob metadata: name: renovate namespace: renovate spec: schedule: \u0026#34;0 */4 * * *\u0026#34; # Every 4 hours concurrencyPolicy: Forbid jobTemplate: spec: template: spec: restartPolicy: Never containers: - name: renovate image: ghcr.io/renovatebot/renovate:42.19.5 env: - name: RENOVATE_CONFIG_FILE value: /config/renovate.json - name: RENOVATE_TOKEN valueFrom: secretKeyRef: name: renovate-env key: RENOVATE_TOKEN volumeMounts: - name: renovate-config mountPath: /config volumes: - name: renovate-config configMap: name: renovate-config Why self-hosted:\nFull control over scan frequency No rate limits from Renovate\u0026rsquo;s hosted service Works with private registries Runs inside my cluster, close to my git server Configuration The repository-level renovate.json controls what gets updated:\n{ \u0026#34;$schema\u0026#34;: \u0026#34;https://docs.renovatebot.com/renovate-schema.json\u0026#34;, \u0026#34;extends\u0026#34;: [\u0026#34;config:recommended\u0026#34;], \u0026#34;enabledManagers\u0026#34;: [\u0026#34;helm-values\u0026#34;, \u0026#34;kustomize\u0026#34;, \u0026#34;kubernetes\u0026#34;], \u0026#34;kubernetes\u0026#34;: { \u0026#34;managerFilePatterns\u0026#34;: [ \u0026#34;/(^|/)base/.+/(?:[^/]*deployment)\\\\.ya?ml$/\u0026#34; ] }, \u0026#34;prConcurrentLimit\u0026#34;: 20, \u0026#34;prHourlyLimit\u0026#34;: 0, \u0026#34;packageRules\u0026#34;: [ // ... rules ] } The managerFilePatterns is crucial - it tells Renovate to only look at deployment files in my base/ directory. This prevents it from trying to update the same image in multiple overlay locations.\nPackage Rules: The Power Feature This is where Renovate really shines. You can define complex rules for how different images should be handled.\nPinning Major Versions Some apps I want to stay on a specific major version until I\u0026rsquo;m ready to upgrade:\n{ \u0026#34;description\u0026#34;: \u0026#34;Keep linuxserver/lidarr on 2.x\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;lscr.io/linuxserver/lidarr\u0026#34;], \u0026#34;allowedVersions\u0026#34;: \u0026#34;\u0026lt;3.0.0\u0026#34; } Automerging Safe Updates Minor and patch updates for stable images can merge automatically:\n{ \u0026#34;description\u0026#34;: \u0026#34;Automerge minor/patch for Linuxserver images\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;lscr.io/linuxserver/*\u0026#34;], \u0026#34;matchUpdateTypes\u0026#34;: [\u0026#34;minor\u0026#34;, \u0026#34;patch\u0026#34;], \u0026#34;automerge\u0026#34;: true, \u0026#34;automergeType\u0026#34;: \u0026#34;pr\u0026#34; } The PR is still created (for visibility), but it merges without manual intervention.\nHandling Weird Versioning LinuxServer images sometimes use date-based tags like 2021.12.15. These confuse semver parsing:\n{ \u0026#34;description\u0026#34;: \u0026#34;Ignore Linuxserver date-style 2021 tags\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [ \u0026#34;lscr.io/linuxserver/tautulli\u0026#34;, \u0026#34;lscr.io/linuxserver/overseerr\u0026#34; ], \u0026#34;allowedVersions\u0026#34;: \u0026#34;!/^2021\\\\./\u0026#34; } Avoiding .0 Releases Home Assistant .0 releases are often buggy. I skip them:\n{ \u0026#34;description\u0026#34;: \u0026#34;Home Assistant: avoid .0 releases\u0026#34;, \u0026#34;matchDatasources\u0026#34;: [\u0026#34;docker\u0026#34;], \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;ghcr.io/home-assistant/home-assistant\u0026#34;], \u0026#34;allowedVersions\u0026#34;: \u0026#34;!/\\\\.0$/\u0026#34; } I also disable major/minor updates entirely for Home Assistant - I want to control those manually:\n{ \u0026#34;description\u0026#34;: \u0026#34;Home Assistant: disable major/minor updates\u0026#34;, \u0026#34;matchPackageNames\u0026#34;: [\u0026#34;ghcr.io/home-assistant/home-assistant\u0026#34;], \u0026#34;matchUpdateTypes\u0026#34;: [\u0026#34;major\u0026#34;, \u0026#34;minor\u0026#34;], \u0026#34;enabled\u0026#34;: false } The Workflow Now Renovate scans every 4 hours PRs are created for available updates I review (or automerge handles it) Merge to master ArgoCD syncs the new image to the cluster Everything flows through git. I can see exactly when an image was updated, who approved it, and what the previous version was.\nDependency Dashboard Renovate creates an issue called \u0026ldquo;Dependency Dashboard\u0026rdquo; that shows:\nPending updates waiting for PRs Open PRs awaiting merge Updates blocked by version constraints Errors from failed lookups This single issue gives you visibility into your entire update backlog.\nMigration Tips 1. Start with Automerge Disabled Get comfortable with the PR flow before enabling automerge:\n{ \u0026#34;extends\u0026#34;: [\u0026#34;config:recommended\u0026#34;, \u0026#34;:automergeDisabled\u0026#34;] } 2. Use Semantic Commits Enable semantic commits for cleaner git history:\n{ \u0026#34;semanticCommits\u0026#34;: \u0026#34;enabled\u0026#34; } PRs get titles like fix(deps): update lscr.io/linuxserver/sonarr to v4.0.2.\n3. Limit Concurrent PRs Initially Don\u0026rsquo;t flood yourself with PRs on the first run:\n{ \u0026#34;prConcurrentLimit\u0026#34;: 5 } Increase once you\u0026rsquo;ve caught up on the backlog.\n4. Remove Keel Gradually I removed Keel annotations from one app at a time, verified Renovate was creating PRs, then moved to the next. Don\u0026rsquo;t rip out Keel all at once.\nKeel vs Renovate: Summary Aspect Keel Renovate Update method Direct cluster modification Pull requests GitOps compatible No (causes drift) Yes Review process None (notification only) Full PR review Version constraints Basic annotation policies Powerful regex rules Rollback Manual Git revert Audit trail Logs only Full git history Automerge N/A (always auto) Optional per-package Result My image updates now have:\nVisibility - PRs show exactly what\u0026rsquo;s changing Control - Rules prevent unwanted major updates History - Git log shows every update Safety - Automerge only for trusted packages Consistency - GitOps stays clean, no drift The slight delay (PR review vs instant deploy) is worth the reliability. When something breaks, I know exactly which merge caused it and can revert cleanly.\nIf you\u0026rsquo;re running GitOps with ArgoCD or Flux, Renovate is the right choice for container image updates. Keel solved a problem for the pre-GitOps era, but PR-based updates are the modern approach.\n","permalink":"https://www.rustybower.com/posts/keel-to-renovate-kubernetes-image-updates/","summary":"\u003cp\u003eFor years I used Keel to automatically update container images in my Kubernetes clusters. It worked, but as I moved to GitOps with ArgoCD, Keel\u0026rsquo;s push-based approach became a liability. I migrated to Renovate for PR-based image updates, and it\u0026rsquo;s been a significant improvement.\u003c/p\u003e\n\u003ch2 id=\"the-problem-with-keel\"\u003eThe Problem with Keel\u003c/h2\u003e\n\u003cp\u003eKeel watches for new container images and updates deployments directly in the cluster. You can configure it via annotations:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eannotations\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ekeel.sh/policy\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emajor\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ekeel.sh/trigger\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epoll\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWhen a new image appears, Keel modifies the deployment in-place.\u003c/p\u003e","title":"From Keel to Renovate: Better Container Image Updates for GitOps"},{"content":"I manage two Kubernetes environments - a home cluster (bowerhaus) and a cloud cluster (rustycloud) - using GitOps with Kustomize and ArgoCD. After running this setup for a while, I\u0026rsquo;ve learned what works, what doesn\u0026rsquo;t, and some non-obvious gotchas.\nThe Architecture kustomize/ ├── base/ # Shared, environment-agnostic configs │ ├── media/ │ │ ├── lidarr/ │ │ ├── radarr/ │ │ └── sonarr/ │ ├── home-automation/ │ │ ├── home-assistant/ │ │ └── frigate/ │ └── data-analytics/ │ ├── prometheus/ │ └── grafana/ ├── environments/ │ ├── bowerhaus/ │ │ ├── applicationsets/ # ArgoCD ApplicationSet │ │ └── apps/ # Per-app overlays │ │ ├── frigate/ │ │ ├── home-assistant/ │ │ └── prometheus/ │ └── rustycloud/ │ ├── applicationsets/ │ └── apps/ │ ├── plex/ │ ├── sonarr/ │ └── grafana/ The key principle: base contains environment-agnostic resources, environments contain overlays that customize for each cluster.\nArgoCD ApplicationSets Instead of creating individual ArgoCD Applications for each service, I use ApplicationSets with a git directory generator:\napiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: bowerhaus-appset namespace: argocd spec: generators: - git: repoURL: https://github.com/RustyBower/kustomize.git revision: master directories: - path: environments/bowerhaus/apps/* template: metadata: name: \u0026#39;{{path.basename}}\u0026#39; spec: project: default source: repoURL: https://github.com/RustyBower/kustomize.git targetRevision: master path: \u0026#39;environments/bowerhaus/apps/{{path.basename}}\u0026#39; kustomize: {} destination: server: https://kubernetes.default.svc namespace: \u0026#39;{{path.basename}}\u0026#39; syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true Every directory under environments/bowerhaus/apps/ automatically becomes an ArgoCD Application. Add a new folder, push to git, and ArgoCD deploys it.\nThe kustomize: {} block forces ArgoCD to use Kustomize even if there\u0026rsquo;s no explicit kustomization.yaml (though you should always have one).\nBase/Overlay Pattern Base: Generic, Reusable The base contains the core deployment without environment-specific details:\n# base/media/lidarr/lidarr-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: lidarr spec: replicas: 1 template: spec: containers: - name: lidarr image: lscr.io/linuxserver/lidarr:2.8.2 ports: - containerPort: 8686 volumeMounts: - name: config mountPath: /config volumes: - name: config persistentVolumeClaim: claimName: lidarr-config No NFS paths, no environment-specific storage, no ingress hostnames.\nOverlay: Environment-Specific The overlay adds what\u0026rsquo;s unique to each environment:\n# environments/rustycloud/apps/lidarr/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../../../../base/media/lidarr/ - ingress.yaml namespace: lidarr patches: - path: deployment-patch.yaml # environments/rustycloud/apps/lidarr/deployment-patch.yaml apiVersion: apps/v1 kind: Deployment metadata: name: lidarr spec: template: spec: volumes: - name: media nfs: server: 10.0.0.105 path: \u0026#34;/cephfs/media/\u0026#34; - name: download nfs: server: 10.0.0.105 path: \u0026#34;/cephfs/data/download/complete/music\u0026#34; containers: - name: lidarr volumeMounts: - mountPath: /download/complete/music name: download - mountPath: /data/music name: media subPath: Music The patch adds NFS volumes specific to rustycloud\u0026rsquo;s storage infrastructure.\nKustomize Challenges I\u0026rsquo;ve Hit 1. ClusterRoleBinding Namespace Issues ClusterRoleBindings reference ServiceAccounts with a namespace. The base might have:\nsubjects: - kind: ServiceAccount name: prometheus namespace: monitoring But your overlay uses namespace: prometheus. Kustomize\u0026rsquo;s namespace transformer doesn\u0026rsquo;t update references inside ClusterRoleBindings.\nSolution: Use an inline patch in your overlay:\npatches: - target: kind: ClusterRoleBinding name: prometheus patch: |- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus subjects: - kind: ServiceAccount name: prometheus namespace: prometheus 2. Strategic Merge vs JSON Patches Kustomize\u0026rsquo;s default strategic merge patches work well for adding fields but struggle with:\nRemoving fields Modifying array items by index Complex nested structures When strategic merge fails, use JSON patches:\npatches: - target: kind: Deployment name: myapp patch: |- - op: remove path: /spec/template/spec/containers/0/resources/limits 3. ConfigMap/Secret Name Hashing Kustomize appends hashes to ConfigMap and Secret names by default. This breaks references if you\u0026rsquo;re not careful.\ngeneratorOptions: disableNameSuffixHash: true I disable hashing for configs that need stable names (like Renovate\u0026rsquo;s config).\n4. Image Tags in Different Locations Renovate and Kustomize both want to manage image tags. I keep images in the base deployments and let Renovate update them there. The kubernetes manager in Renovate handles this:\n{ \u0026#34;kubernetes\u0026#34;: { \u0026#34;managerFilePatterns\u0026#34;: [ \u0026#34;/(^|/)base/.+/(?:[^/]*deployment)\\\\.ya?ml$/\u0026#34; ] } } This tells Renovate to only look at deployment files in the base directory.\n5. ArgoCD Sync Waves When deploying interconnected services, order matters. Use sync-wave annotations:\nmetadata: annotations: argocd.argoproj.io/sync-wave: \u0026#34;-1\u0026#34; # Deploy before wave 0 Negative waves deploy first. I use this for:\n-2: Namespaces and CRDs -1: Secrets and ConfigMaps 0: Main deployments (default) 1: Ingresses and monitoring Directory Structure Tips One App Per Directory Each app gets its own directory with a kustomization.yaml. This maps cleanly to ArgoCD Applications and makes it obvious what\u0026rsquo;s deployed where.\nConsistent Naming I use the app name as:\nDirectory name: apps/frigate/ Namespace: namespace: frigate ArgoCD Application name: {{path.basename}} → frigate This consistency makes debugging easier.\nBase Categories Group bases by function:\nbase/media/ - Plex, *arr stack base/home-automation/ - Home Assistant, Frigate base/data-analytics/ - Prometheus, Grafana, InfluxDB base/dev-tools/ - Gitea, Drone, Renovate Self-Healing and Pruning I enable both in the ApplicationSet:\nsyncPolicy: automated: prune: true # Delete resources not in git selfHeal: true # Revert manual changes This ensures git is the source of truth. Any kubectl edits get reverted. Deleted files remove resources.\nWarning: prune: true means deleting a directory from git deletes the entire deployment. Good for cleanup, dangerous if you accidentally remove something.\nThe Workflow Make changes in a branch Test with kustomize build environments/bowerhaus/apps/myapp Push and merge to master ArgoCD syncs within 3 minutes (or trigger manually) Watch the sync in ArgoCD UI No kubectl apply, no remembering which cluster you\u0026rsquo;re on, no drift between environments.\nTakeaways ApplicationSets eliminate per-app boilerplate Base/overlay pattern enforces separation of concerns Strategic merge patches work 80% of the time; JSON patches handle the rest Self-heal + prune keeps clusters matching git Test locally with kustomize build before pushing The initial setup takes effort, but the ongoing maintenance is minimal. Adding a new service is just creating a directory and pushing to git.\n","permalink":"https://www.rustybower.com/posts/kustomize-argocd-homelab-gitops/","summary":"\u003cp\u003eI manage two Kubernetes environments - a home cluster (bowerhaus) and a cloud cluster (rustycloud) - using GitOps with Kustomize and ArgoCD. After running this setup for a while, I\u0026rsquo;ve learned what works, what doesn\u0026rsquo;t, and some non-obvious gotchas.\u003c/p\u003e\n\u003ch2 id=\"the-architecture\"\u003eThe Architecture\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-fallback\" data-lang=\"fallback\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekustomize/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── base/                          # Shared, environment-agnostic configs\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   ├── media/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │   ├── lidarr/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │   ├── radarr/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │   └── sonarr/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   ├── home-automation/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │   ├── home-assistant/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │   └── frigate/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   └── data-analytics/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│       ├── prometheus/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│       └── grafana/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── environments/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   ├── bowerhaus/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │   ├── applicationsets/       # ArgoCD ApplicationSet\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │   └── apps/                  # Per-app overlays\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │       ├── frigate/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │       ├── home-assistant/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   │       └── prometheus/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   └── rustycloud/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│       ├── applicationsets/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│       └── apps/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│           ├── plex/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│           ├── sonarr/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│           └── grafana/\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe key principle: \u003cstrong\u003ebase\u003c/strong\u003e contains environment-agnostic resources, \u003cstrong\u003eenvironments\u003c/strong\u003e contain overlays that customize for each cluster.\u003c/p\u003e","title":"GitOps for Homelabs: Kustomize + ArgoCD Patterns and Pitfalls"},{"content":"I have six Honeywell T6 Pro Z-Wave thermostats throughout my house controlling a multi-zone HVAC system. Out of the box, these thermostats have their own scheduling and \u0026ldquo;smart\u0026rdquo; features that conflict with Home Assistant automations. Here\u0026rsquo;s how I configured them to let HA take full control.\nThe Setup 6x Honeywell T6 Pro Z-Wave thermostats (TH6320ZW2003) Z-Wave JS UI running in Kubernetes Home Assistant with Z-Wave JS integration Gas furnace + AC (not heat pump) with multi-zone dampers The zones:\nGreat Room \u0026amp; Music Room - shared outdoor unit, multi-zone Master Bedroom, Office, Kids Room - shared outdoor unit, multi-zone Basement - standalone (heat only, no AC) The Problem The T6 Pro has built-in features that fight against Home Assistant control:\nBuilt-in scheduling - The thermostat tries to follow its own 5-2 or 5-1-1 schedule Adaptive Intelligent Recovery - Pre-heats/cools based on learned patterns Temperature resolution - Reports in 1°F increments by default, limiting precision When HA sends a setpoint change, these features can override it or cause unexpected behavior.\nKey Parameters to Change After digging through the Z-Wave JS node configuration, here are the critical parameters:\nParameter 1: Schedule Type Set to 0 (No schedule)\nParameter: 1 Value: 0 (None) Options: 0=None, 1=Occupancy, 2=Every Day, 3=5-2, 4=5-1-1 This disables the thermostat\u0026rsquo;s built-in scheduling entirely. HA will handle all scheduling via automations.\nParameter 27: Adaptive Intelligent Recovery Set to 0 (Disabled)\nParameter: 27 Value: 0 (Disabled) Options: 0=Disabled, 1=Enabled AIR learns how long it takes to reach setpoint and starts heating/cooling early. Sounds nice, but it means the thermostat ignores your setpoint timing. Disable it and let HA handle pre-conditioning if needed.\nParameter 44: Temperature Display Resolution Set to 0 (1°F)\nParameter: 44 Value: 0 (1°F) Options: 0=1°F, 1=0.5°F Actually, keep this at 0 for Fahrenheit. If you use Celsius, setting to 1 gives 0.5° resolution which can be useful for tighter control.\nParameter 6: Cool Stages (Basement Only) Set to 0 for heat-only zones\nParameter: 6 Value: 0 (No cooling) Options: 0=None, 1=1 Stage, 2=2 Stages My basement has no AC, so I set cool stages to 0. This prevents the thermostat from ever calling for cooling.\nHow to Change Parameters Via Z-Wave JS UI Navigate to your thermostat node Go to Configuration Parameters Find the parameter number Set the new value and click Save The changes take effect immediately - you\u0026rsquo;ll see the thermostat update its display.\nVia Home Assistant Service Call You can also use the zwave_js.set_config_parameter service:\nservice: zwave_js.set_config_parameter target: device_id: abc123... data: parameter: 1 value: 0 However, I found the Z-Wave JS UI more reliable for bulk changes across multiple devices.\nMulti-Zone Considerations With multi-zone HVAC, the thermostats operate dampers rather than directly controlling the furnace/AC. A few things to keep in mind:\nZone coordination - If one zone calls for heat while another calls for cooling, you have a conflict. HA automations should prevent this. Minimum airflow - Most systems need at least one zone open. Configure a \u0026ldquo;dump zone\u0026rdquo; or ensure automations don\u0026rsquo;t close all dampers. Equipment protection - The furnace/AC has its own safeties, but avoid rapid cycling by adding delays between mode changes in HA. My HA Automation Strategy With the thermostats now in \u0026ldquo;dumb\u0026rdquo; mode, HA handles:\nTime-based scheduling - Different setpoints for sleep, away, home Occupancy detection - Adjust zones based on presence sensors Weather integration - Pre-condition before temperature swings Energy optimization - Coordinate zones to minimize equipment runtime The thermostats become simple actuators - they receive a setpoint from HA and maintain it. No more fighting between two \u0026ldquo;smart\u0026rdquo; systems trying to be helpful.\nResult After making these changes across all six thermostats:\nSetpoint changes from HA take effect immediately No more mysterious temperature swings from AIR Schedules are 100% controlled by HA automations The system behaves predictably The T6 Pro is a solid Z-Wave thermostat once you disable its autonomous features. It reports temperature accurately, responds quickly to commands, and the hardware is reliable. Just don\u0026rsquo;t let it think for itself.\n","permalink":"https://www.rustybower.com/posts/zwave-thermostats-home-assistant/","summary":"\u003cp\u003eI have six Honeywell T6 Pro Z-Wave thermostats throughout my house controlling a multi-zone HVAC system. Out of the box, these thermostats have their own scheduling and \u0026ldquo;smart\u0026rdquo; features that conflict with Home Assistant automations. Here\u0026rsquo;s how I configured them to let HA take full control.\u003c/p\u003e\n\u003ch2 id=\"the-setup\"\u003eThe Setup\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003e6x Honeywell T6 Pro Z-Wave thermostats\u003c/strong\u003e (TH6320ZW2003)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eZ-Wave JS UI\u003c/strong\u003e running in Kubernetes\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHome Assistant\u003c/strong\u003e with Z-Wave JS integration\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGas furnace + AC\u003c/strong\u003e (not heat pump) with multi-zone dampers\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe zones:\u003c/p\u003e","title":"Optimizing Z-Wave Thermostats for Home Assistant Control"},{"content":"I run a fully self-hosted CI/CD pipeline using Drone for builds, Gitea for git hosting, and Harbor for container registry. No GitHub Actions, no Docker Hub, no external dependencies. Here\u0026rsquo;s how it all fits together.\nThe Stack Gitea - Lightweight git server with OAuth2 support Drone - Container-native CI/CD platform Harbor - Enterprise container registry with vulnerability scanning BuildKit - Modern Docker builder for efficient image builds Why Self-Hosted? Privacy - Code never leaves my network No rate limits - Build as often as needed Offline capability - Works during internet outages (for local images) Learning - Understanding the full DevOps stack Cost - No per-minute billing for CI runners Architecture ┌─────────────────────────────────────────────────────────────┐ │ Kubernetes │ │ │ │ ┌──────────┐ ┌─────────────────────────────┐ │ │ │ Gitea │────▶│ Drone │ │ │ │ (git) │ │ ┌───────┐ ┌──────────┐ │ │ │ └──────────┘ │ │Server │ │ Runner │ │ │ │ │ └───────┘ └────┬─────┘ │ │ │ └────────────────────│────────┘ │ │ │ │ │ ┌──────────┐ ┌──────▼─────┐ │ │ │ Harbor │◀──────────────────│ BuildKit │ │ │ │(registry)│ │ (builds) │ │ │ └──────────┘ └────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘ Push code to Gitea Gitea webhook triggers Drone Drone spawns a build job via the Kubernetes runner BuildKit builds the container image Image is pushed to Harbor ArgoCD deploys the new image (separate workflow) Gitea Setup Gitea is straightforward - a single deployment with PostgreSQL backend. The key configuration is creating an OAuth2 application for Drone:\nGo to Gitea Settings → Applications Create new OAuth2 application Set redirect URI to https://drone.rustybower.com/login Note the Client ID and Client Secret Drone Components Drone Server The main Drone server handles the web UI, webhook processing, and build coordination:\napiVersion: apps/v1 kind: Deployment metadata: name: drone spec: template: spec: containers: - name: drone image: drone/drone:2.26.0 envFrom: - configMapRef: name: drone-env volumeMounts: - mountPath: /data name: drone-data Configuration via ConfigMap:\napiVersion: v1 kind: ConfigMap metadata: name: drone-env data: DRONE_GITEA_SERVER: \u0026#34;https://gitea.rustybower.com\u0026#34; DRONE_GITEA_CLIENT_ID: \u0026#34;your-oauth-client-id\u0026#34; DRONE_GITEA_CLIENT_SECRET: \u0026#34;your-oauth-client-secret\u0026#34; DRONE_RPC_SECRET: \u0026#34;shared-secret-for-runners\u0026#34; DRONE_SERVER_HOST: \u0026#34;drone.rustybower.com\u0026#34; DRONE_SERVER_PROTO: \u0026#34;https\u0026#34; DRONE_USER_CREATE: \u0026#34;username:rusty,admin:true\u0026#34; Drone Kubernetes Runner The runner executes pipeline steps as Kubernetes pods:\napiVersion: apps/v1 kind: Deployment metadata: name: runner spec: template: spec: containers: - name: runner image: drone/drone-runner-kube:latest envFrom: - configMapRef: name: drone-env serviceAccountName: drone-runner The runner needs RBAC permissions to create pods:\napiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: drone-runner rules: - apiGroups: [\u0026#34;\u0026#34;] resources: [\u0026#34;pods\u0026#34;, \u0026#34;pods/log\u0026#34;, \u0026#34;secrets\u0026#34;] verbs: [\u0026#34;get\u0026#34;, \u0026#34;list\u0026#34;, \u0026#34;watch\u0026#34;, \u0026#34;create\u0026#34;, \u0026#34;delete\u0026#34;] BuildKit for Efficient Builds Instead of Docker-in-Docker (security nightmare), I use BuildKit as a separate service:\napiVersion: apps/v1 kind: Deployment metadata: name: buildkitd spec: template: spec: containers: - name: buildkitd image: moby/buildkit:v0.26.3 args: - --addr - tcp://0.0.0.0:1234 - --config - /etc/buildkit/buildkit.toml securityContext: privileged: true # Required for building volumeMounts: - name: config mountPath: /etc/buildkit - name: harbor-docker-config mountPath: /root/.docker BuildKit needs registry credentials to push images:\napiVersion: v1 kind: Secret metadata: name: harbor-docker-config type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: \u0026lt;base64-encoded-docker-config\u0026gt; Harbor Configuration Harbor provides:\nContainer image storage Vulnerability scanning with Trivy Robot accounts for CI/CD access Image replication between registries Create a robot account for Drone:\nHarbor → Administration → Robot Accounts Create with push/pull permissions Use credentials in Drone secrets Pipeline Example Here\u0026rsquo;s a real pipeline for my Hugo blog:\n# .drone.yml kind: pipeline type: kubernetes name: default clone: depth: 1 submodules: true steps: - name: init submodules image: alpine/git commands: - git submodule update --init --recursive - name: docker build and push image: plugins/docker settings: registry: harbor.rustybower.com repo: harbor.rustybower.com/hugo/hugo-site tags: - latest dockerfile: Dockerfile build_args: - HUGO_BASEURL=https://www.rustybower.com/ username: from_secret: harbor_username password: from_secret: harbor_password The Dockerfile uses multi-stage builds:\n# Build Stage FROM hugomods/hugo:exts AS builder ARG HUGO_BASEURL= ENV HUGO_BASEURL=${HUGO_BASEURL} COPY . /src RUN hugo --minify --enableGitInfo # Final Stage FROM hugomods/hugo:nginx COPY --from=builder /src/public /site Secrets Management Drone secrets are stored per-repository:\nGo to repository settings in Drone UI Add secrets for registry credentials Reference in pipeline with from_secret Never commit credentials to .drone.yml - always use secrets.\nTroubleshooting Tips Build Not Starting Check the runner logs:\nkubectl logs -n drone deployment/runner Common issues:\nRPC secret mismatch between server and runner Runner can\u0026rsquo;t reach Drone server RBAC permissions missing Registry Push Fails Verify BuildKit has credentials:\nkubectl exec -n drone deployment/buildkitd -- cat /root/.docker/config.json Check Harbor robot account permissions.\nPipeline Stuck The Kubernetes runner creates pods for each step. Check for stuck pods:\nkubectl get pods -n drone Look for ImagePullBackOff or resource constraints.\nIntegration with GitOps After the image is pushed to Harbor, I use Renovate to create PRs updating the image tag in my Kustomize repo. ArgoCD then deploys the new image.\nThe full flow:\nPush code to Gitea Drone builds and pushes to Harbor Renovate detects new image, creates PR Merge PR to update manifest ArgoCD syncs new image to cluster This keeps the GitOps principle intact - images are built automatically, but deployment still flows through git.\nPerformance Optimizations Layer Caching BuildKit caches layers automatically. For faster builds:\nOrder Dockerfile commands from least to most changing Use multi-stage builds to minimize final image size Mount build caches for package managers Resource Limits Set appropriate limits in Drone steps:\nsteps: - name: build image: node:20 resources: limits: memory: 2Gi cpu: 2 Parallel Steps Independent steps can run in parallel:\nsteps: - name: test image: node:20 commands: - npm test - name: lint image: node:20 commands: - npm run lint Security Considerations Network policies - Restrict what build pods can access Robot accounts - Use minimal permissions for registry access No privileged unless needed - Only BuildKit needs privileged mode Scan images - Harbor\u0026rsquo;s Trivy integration catches vulnerabilities Audit logs - Gitea, Drone, and Harbor all log activity Result My self-hosted CI/CD:\nBuilds in ~2 minutes for most projects Zero external dependencies Full control over the pipeline Complete audit trail across all components The initial setup is more complex than GitHub Actions, but the understanding you gain of CI/CD internals is valuable. Plus, there\u0026rsquo;s something satisfying about git push triggering a build on your own infrastructure.\n","permalink":"https://www.rustybower.com/posts/drone-gitea-harbor-self-hosted-cicd/","summary":"\u003cp\u003eI run a fully self-hosted CI/CD pipeline using Drone for builds, Gitea for git hosting, and Harbor for container registry. No GitHub Actions, no Docker Hub, no external dependencies. Here\u0026rsquo;s how it all fits together.\u003c/p\u003e\n\u003ch2 id=\"the-stack\"\u003eThe Stack\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eGitea\u003c/strong\u003e - Lightweight git server with OAuth2 support\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDrone\u003c/strong\u003e - Container-native CI/CD platform\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHarbor\u003c/strong\u003e - Enterprise container registry with vulnerability scanning\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBuildKit\u003c/strong\u003e - Modern Docker builder for efficient image builds\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"why-self-hosted\"\u003eWhy Self-Hosted?\u003c/h2\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003ePrivacy\u003c/strong\u003e - Code never leaves my network\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNo rate limits\u003c/strong\u003e - Build as often as needed\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eOffline capability\u003c/strong\u003e - Works during internet outages (for local images)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLearning\u003c/strong\u003e - Understanding the full DevOps stack\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCost\u003c/strong\u003e - No per-minute billing for CI runners\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"architecture\"\u003eArchitecture\u003c/h2\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-fallback\" data-lang=\"fallback\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e┌─────────────────────────────────────────────────────────────┐\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│                         Kubernetes                           │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│                                                              │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  ┌──────────┐     ┌─────────────────────────────┐           │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  │  Gitea   │────▶│          Drone              │           │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  │  (git)   │     │  ┌───────┐    ┌──────────┐  │           │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  └──────────┘     │  │Server │    │  Runner  │  │           │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│                   │  └───────┘    └────┬─────┘  │           │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│                   └────────────────────│────────┘           │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│                                        │                     │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  ┌──────────┐                   ┌──────▼─────┐              │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  │  Harbor  │◀──────────────────│  BuildKit  │              │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  │(registry)│                   │  (builds)  │              │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│  └──────────┘                   └────────────┘              │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│                                                              │\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e└─────────────────────────────────────────────────────────────┘\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003col\u003e\n\u003cli\u003ePush code to Gitea\u003c/li\u003e\n\u003cli\u003eGitea webhook triggers Drone\u003c/li\u003e\n\u003cli\u003eDrone spawns a build job via the Kubernetes runner\u003c/li\u003e\n\u003cli\u003eBuildKit builds the container image\u003c/li\u003e\n\u003cli\u003eImage is pushed to Harbor\u003c/li\u003e\n\u003cli\u003eArgoCD deploys the new image (separate workflow)\u003c/li\u003e\n\u003c/ol\u003e\n\u003ch2 id=\"gitea-setup\"\u003eGitea Setup\u003c/h2\u003e\n\u003cp\u003eGitea is straightforward - a single deployment with PostgreSQL backend. The key configuration is creating an OAuth2 application for Drone:\u003c/p\u003e","title":"Self-Hosted CI/CD with Drone, Gitea, and Harbor"}]