ADR: Tailnet-per-environment (Prod / Test / Dev) with declarative node sharing

Status

Rejected. The homelab will not split Prod/Test/Dev into separate tailnets. Environment separation stays as it is today: a single tailnet with tag-based grants.

The investigation is retained in full because the rejection is the useful part — it documents why an appealing structural-isolation story doesn’t survive contact with an already-working Services deployment.

Superseded in direction by ADR-0007: Workload isolation in a single tailnet — access pools, which keeps the workload/homelab separation inside the tailnet as ACL-defined access pools, with a separate tailnet retained only as a deferred off-ramp. Its core finding — that a tailnet boundary pays off where paths should be closed anyway (non-homelab workload), not between homelab environments — still stands.

Context

Today the homelab runs a single tailnet with tag-based environment separation. Services are already the primary exposure mechanism for containers across Prod/Test/Dev — advertised via Docktail labels and ScaleTail sidecars, with per-service MagicDNS names, per tailscale.md. That part of the design is settled and this ADR does not revisit it.

What is not settled is the environment boundary. Per tailscale-grants.md, isolation between Prod/Test/Dev is enforced entirely by grant rules:

{"src": ["tag:prod"], "dst": ["tag:prod"], "ip": ["*:*"]}
{"src": ["tag:test"], "dst": ["tag:test"], "ip": ["*:*"]}
{"src": ["tag:dev"], "dst": ["tag:dev"], "ip": ["*:*"]}

That works, but the isolation is one policy-file edit deep. A bad merge, a mis-tagged node, or a fat-fingered tag:* destination collapses Prod/Test/Dev into one flat network. The blast radius of a policy mistake is the entire homelab.

Two Tailscale capabilities change the calculus:

1. Additional tailnets via the API. POST /api/v2/organizations/-/tailnets provisions a fully isolated tailnet under the same account, no console clicks. The response carries a tailnet-scoped OAuth client — creation is org-scoped, but administration and deletion require that per-tailnet credential (worked example).
2. Declarative node sharing (docs, alpha, waitlist-gated). Cross-tailnet access becomes a policy-file construct — externalTailnets plus grants referencing group://<alias>/<name> or tag://<alias>/<name>. It is explicitly designed for machine-to-machine access between trusted tailnets, which is exactly the homelab’s shape.

The question this ADR frames: is a tailnet boundary the right isolation primitive for the homelab’s environments, and is the cross-tailnet seam cheap enough to live with?

What a tailnet split does to the existing Services layer

Because Services are already the exposure model, the split’s real cost lands here rather than on greenfield design:

• Service names become tailnet-qualified. Every service currently resolves under one tailnet’s MagicDNS suffix. Three tailnets means three suffixes, and the naming convention documented in tailscale.md needs an environment-aware revision. OPEN: does plex.<prod-tailnet>.ts.net replace the current flat convention, or does an alias layer hide the split?
• httpsEnabled is per-tailnet and off by default. Any new tailnet needs the setting turned on before TLS-terminating services work at all — a provisioning step that doesn’t exist today because the current tailnet was configured once, long ago.
• Docktail/ScaleTail config becomes tailnet-scoped. Sidecars and label-driven advertisement authenticate to a specific tailnet. Auth keys, tags, and tagOwners all become per-tailnet concerns.
• Cross-environment service consumption is the seam. Anything in one environment consuming a service in another (observability, backups, promotion tooling) stops being an in-tailnet lookup and becomes an externalTailnets grant.

OPEN and worth answering early: whether a Service can be published across a tailnet boundary. If yes, the boundary contract is service-level and clean. If not, cross-environment access is node-and-port level, which is coarser than the exposure model already in use — a genuine regression against current practice, and arguably the strongest argument for staying on one tailnet.

Correction to an earlier framing

An earlier draft of this ADR analyzed the seam using link-based sharing (sharing docs) and concluded the model was largely unworkable — tags stripped, shares user-scoped so tagged machines can’t accept them, quarantine by default. Those constraints are real but belong to link-based sharing, not declarative node sharing. Declarative sharing operates at the policy level and supports tag references and machine-to-machine access directly. The two mechanisms coexist; picking the wrong one produces the wrong conclusion, which is why they’re separated explicitly below.

Drivers

The two sharing mechanisms

Getting this distinction right is most of the decision.

Consequence: declarative sharing is the mechanism this design should be built on. Link-based sharing remains the right tool only for the partner/guest case — handing one machine to one outside human.

Shape of the config

Double opt-in, per the declarative node sharing docs. Receiving tailnet declares what may be referenced:

{
  "externalTailnets": {
    "prod": {
      "externalID": "<prod-tailnet-id>",
      "allowIncomingConnections": false,
      "allowExternalReferencesTo": ["tag:komodo-controller"]
    }
  }
}

Sharing tailnet accepts the relationship and writes the grant:

{
  "externalTailnets": {
    "dev": { "externalID": "<dev-tailnet-id>", "allowIncomingConnections": true }
  },
  "grants": [
    { "src": ["tag://dev/komodo-controller"], "dst": ["tag:host"], "ip": ["8120"] }
  ]
}

A grant referencing something absent from the other side’s allowExternalReferencesTo is silently ignored — no error, just no connectivity. That failure mode needs a verification step in any runbook.

Options

Retained as the investigation record. Outcome noted on each.

Option A — Status quo: one tailnet, tag-based separation — CHOSEN

• Pros: zero migration; flat MagicDNS; one policy file; single console; no alpha dependency; Services and access model untouched.
• Cons: isolation is advisory, not structural; one bad grant flattens all environments; no per-environment audit boundary.

Option B — Three tailnets (Prod / Test / Dev), declarative sharing between them — rejected

• Pros: isolation is structural — a Dev policy mistake cannot reach Prod; per-tailnet audit logs, auth keys, and tagOwners; Dev becomes genuinely disposable; whole lifecycle is API-drivable (create, httpsEnabled, policy push, scoped auth keys); tags and m2m survive the boundary; double opt-in means neither side can unilaterally widen access.
• Cons: depends on an alpha, waitlist-gated feature; three policy files and three scoped OAuth credentials to manage; FQDNs become tailnet-qualified; silent-ignore failure mode on mismatched references; per-tailnet settings (notably HTTPS certs, off by default) must be provisioned explicitly; rewrites a working Services exposure surface.

Option C — Two tailnets: Prod isolated, Test+Dev together — rejected

• Pros: captures the blast-radius win that matters at two-thirds the overhead; Test↔Dev promotion stays in-tailnet.
• Cons: Test loses a hard boundary from Dev, so “does Test mirror Prod?” gets fuzzier; still carries the full alpha dependency for one seam — paying the same risk for less of the benefit.

Option D — One tailnet, hardened — CARRIED FORWARD (with A)

Policy-file CI with ACL tests in GitHub Actions, mandatory review on the policy file, ban tag:* destinations, and tighten per-service grants on the Services already in place.

• Pros: near-zero migration; no alpha dependency; converts soft isolation into tested isolation.
• Cons: still one flat trust domain; a mistake that passes tests still lands everywhere; audit boundary unchanged.

Decision

Rejected — keep the current single-tailnet structure with tag-based environment separation.

The structural-isolation argument is real, and declarative node sharing does make tailnet-per-environment technically achievable in a way link-based sharing never did. It is still the wrong trade for this homelab. Four reasons, in order of weight:

1. It breaks access that currently works, to solve a problem that hasn’t bitten

Services are already the exposure mechanism for most containers, and access works today — admin tiers, partner access, per-service MagicDNS names. A tailnet split rewrites that working surface: tailnet-qualified DNS suffixes, tailnet-scoped Docktail/ScaleTail sidecar auth, per-tailnet httpsEnabled, and per-tailnet auth keys and tagOwners. The failure it prevents — a fat-fingered grant flattening environments — has not actually occurred. Trading working access for hypothetical blast-radius reduction is the wrong direction.

2. Cross-boundary sharing is coarser than what’s already in place

The unresolved question from the investigation is decisive: if a Service cannot be published across a tailnet boundary, then cross-environment access degrades to node-and-port grants. That is coarser than the per-service exposure model already running. And if the shareable app has to physically live in the tailnet it’s shared from, containers move off the Prod Docker host — which drags in Komodo stack placement and the promotion path from ADR-0001. Either answer is worse than the status quo for a boundary nobody asked for.

3. The complexity is not proportionate to a single-operator homelab

Three tailnets means three policy files, three consoles, three sets of tailnet-scoped OAuth credentials, three auth-key rotations, and per-tailnet provisioning steps that don’t exist today. Add the silent-ignore failure mode — a grant referencing something absent from the other side’s allowExternalReferencesTo produces no error, just no connectivity — and routine debugging gets materially harder. Isolation that is expensive to operate gets worked around, and a worked-around boundary is worse than an honest tag.

4. The alpha dependency lands on the wrong path

Declarative node sharing is alpha and waitlist-gated. Under this ADR it would sit on the path between the Komodo controller and Prod periphery agents — meaning homelab manageability depends on an alpha feature. That inverts the risk this ADR set out to reduce.

What is kept

• Environment separation stays tag-based (tag:prod / tag:test / tag:dev) in one tailnet. No grants rewrite, no tag taxonomy change, no Services churn.
• Partner access stays link-based sharing, per the adversarial model in tailscale.md. Outside humans are not modeled as external tailnets.
• Policy-file CI with ACL tests is still worth doing (the former Option D) and is the only recommendation carried forward from this ADR. It converts advisory isolation into tested isolation at near-zero cost, and it is independently useful.

Future option: collapse three environments to two

Noted, not decided: the Prod/Test/Dev split could later collapse to Test and Prod only. Under ADR-0007, Dev’s remaining purpose is being reconsidered — if non-homelab work moves to a workload environment, a third homelab environment may not be earning its keep. That is a separate decision and deliberately out of scope here.

Where the tailnet boundary does belong

The investigation’s real finding is that the boundary axis was wrong. Splitting environments imposes a boundary across paths that need to stay open (admin, Komodo, observability, backups). Splitting workload from homelab puts the boundary where the paths should be closed anyway. See ADR-0007.

Cross-environment flows that made the split expensive

Each of these works today only because everything is one tailnet. Each would have needed an explicit externalTailnets/grant design — collectively, this list is the concrete form of reason #1 above:

• Komodo controller (NUC, tag:admin) → Prod/Test/Dev periphery agents — the canonical m2m case; technically workable via tag:// references, but it puts homelab manageability behind an alpha feature
• Promotion path DevDocker → Test → Prod (per ADR-0001, promotion is a file move; the network path is currently implicit and would have to become explicit)
• Observability scrape/push paths across environments — direction matters against allowIncomingConnections
• Backup flows to tag:storage NAS (UnRaid = Prod, Synology = Test)
• Admin access from the operator’s devices to all three — one identity in three tailnets, or a device per tailnet, neither cheap
• Exit node — one per tailnet, or one shared
• Partner/guest access — stays on link-based sharing regardless
• ProxMox host, which hosts the Dev VM but is itself infrastructure

Trade-offs that would have been accepted (and were judged too costly)

• Alpha-feature dependency on the critical path for cross-environment automation.
• Longer FQDNs and a documentation sweep to match.
• Per-tailnet provisioning steps that don’t exist today: httpsEnabled PATCH, tagOwners, auth keys, policy push.
• Tailnet-scoped OAuth credentials must be captured at creation — the org token cannot administer or delete a tailnet it created.
• Cross-environment automation gets deliberately harder. Intended for environments; unhelpful for admin paths.

Consequences / follow-on work

Because the split is rejected, most of the original follow-on work is moot. What remains:

• Add policy-file CI with ACL tests. The one carried-forward recommendation. Ban tag:* destinations, require review on the policy file, run tests in GitHub Actions.
• No changes to tailscale-grants.md or tailscale.md from this ADR — the tag taxonomy and Services naming convention stand as documented.
• Record the rejection so the option isn’t re-litigated from scratch; the mechanism comparison above is the reusable part.
• Continue in ADR-0007 for the workload boundary, where the provisioning artifact and waitlist question actually apply.

Open questions — resolved or relocated

The rejection closes most of these. Recorded so the option isn’t re-litigated from scratch:

1. Plan/cost implications of additional tailnets — moot here; relocated to ADR-0007.
2. Waitlist timing for declarative node sharing — no longer gating this decision; still worth joining for ADR-0007.
3. Quarantine semantics vs. allowIncomingConnections for declaratively shared nodes — unresolved; relocated to ADR-0007, where it actually matters.
4. Operator identity/devices across three tailnets — moot; admin access stays in one tailnet, which is the point of the rejection.
5. Komodo topology across a boundary — moot; controller and periphery agents stay in one tailnet.
6. Can a Service be published across a tailnet boundary? — unresolved and now load-bearing for ADR-0007. Answering it decides whether shareable apps can stay on their current host or must physically move.
7. Does Test lose meaning as a Prod rehearsal under Option C? — moot; Option C rejected. Related but distinct question of collapsing to Test+Prod is noted as a future option above.
8. Migration order — moot; no migration.
9. Synced groups (SCIM/Google) can’t be referenced across tailnets — relocated to ADR-0007; only relevant where a boundary exists.

References

Primary sources

• Declarative node sharing — the mechanism this ADR is built on. Alpha, waitlist-gated. externalTailnets, double opt-in, tag:///group:// external reference syntax, allowExternalReferencesTo, allowIncomingConnections. Also the source for the SCIM/Google synced-group limitation.
• summer-with-tailscale — worked, runnable examples for the API-driven path. Modules used here:
  • 01-tailnet-sandboxes — OAuth token exchange, POST /organizations/-/tailnets, and the tailnet-scoped OAuth credential that the org token cannot substitute for on delete
  • 02-tailnet-membership — tsnet embedded membership; relevant if services join tailnets directly rather than via host daemons
  • 03-declarative-sharing — externalTailnets config walkthrough and scoping external access by group/tag instead of blanket dst: ["*"]
  • 04-api-is-the-way — policy file as data via GET/POST /tailnet/-/acl, the httpsEnabled PATCH, scoped/tagged/short-lived auth keys, plus end-to-end.sh

Supporting

• Share your machines with other users — link-based sharing. Source of the tag-stripping, quarantine, and user-scoped constraints; retained here for the partner/guest case and to keep the two mechanisms distinguishable.
• Tailscale Services GA — cited for the GA-era additions relevant to a split (declarative on-node JSON config, per-service audit logs, ACL tests), not as an introduction to Services, which the homelab already runs
• Grants syntax
• Tailnet policy file
• Tailscale API reference

ADR: Workload isolation in a single tailnet — access pools

Status

Proposed. Direction settled: workload/dev traffic stays in the current tailnet, separated by ACLs organized as access pools. The separate-tailnet option is deferred to a named trigger, not discarded.

Extends ADR-0006, which rejected splitting Prod/Test/Dev into separate tailnets. That investigation’s useful finding was that the boundary axis was wrong, not the boundary idea. This ADR takes the next step of that finding: separation of workload from homelab is real, but the mechanism is pools within one boundary, with structural separation preserved as an off-ramp.

Context

Two distinct things are currently conflated inside one tailnet:

1. The homelab — Prod and Test infrastructure, private applications, the Komodo GitOps pipeline, storage, admin interfaces. Everything here is either operator-only or shared with the partner as a trusted user.
2. Dev — nominally a homelab environment, but in practice where non-homelab work happens. It has no promotion relationship worth preserving with Test/Prod for that class of work.

The earlier draft of this ADR weighed two mechanisms: a separate tailnet for workload (structural isolation, independent policy, clean audit boundary) against staying in the shared tailnet with tag separation (zero migration, isolation only advisory). The pivotal consideration — that a boundary’s alpha dependency sits on a path that is allowed to degrade, unlike ADR-0006’s admin/Komodo paths — is retained.

The judgment has since shifted on the trigger. The conditions that would justify a boundary — outside resources managing elements, sustained workload or external-party growth — are not real today. Building the structural split now pays complexity for a hypothetical, and it puts reachability that does not yet need to exist on an alpha, waitlist-gated feature (declarative node sharing). The boundary is kept as the off-ramp, priced and designed, but not built.

Decision

Single tailnet, workload separated by access pools. The structural split is deferred, not discarded.

1. One tailnet for now

Workload/Dev elements stay in the current tailnet, protected by ACLs rather than a boundary. Nothing new to provision, no per-tailnet settings (httpsEnabled, tagOwners, auth keys), and the Services/Docktail/ScaleTail exposure layer works exactly as today. External sharing, when it happens, stays link-based — one machine handed to one invited user — which covers ad-hoc guest access without committing to a separate tailnet.

2. Scope framing shifts from “homelab first” to “environment with pools”

The tailnet is no longer scoped as the homelab with everything else grafted on. It is the environment housing purpose-defined pools — provided services as one pool, development/workloads as another. The pool set is open: additional pools are named when a need clears the bar; none are pre-declared here (see Open Questions). This reframing drives everything that follows: nodes are organized by purpose AND access, and the ACL question becomes “which pools can which users and services reach.”

3. ACLs are pool-based, not environment-tag-based

The access model follows the pools: grants are written in terms of who may reach which pool, in which direction. Two concept principles, without enumerating the pools themselves:

• One pool per node. A node belongs to exactly one pool. Cross-pool reach is an explicit, named grant — never a second membership. This keeps the ACL table non-ambiguous: a node’s pool is its identity, and exceptions are visible rules rather than overlapping tags.
• Directional by default. Workload → provided services is the natural direction and the common grant (dev needs to reach the things the environment provides). Services → workload is a named exception, not part of the default posture. Reverse-direction access is the thing that erodes the separation, so it must always be a visible rule.

The pool taxonomy itself — names, membership, concrete grants — is deliberately unspecified in this ADR. It is a follow-on design task (see Consequences).

4. Separate tailnet = deferred off-ramp

If the environment grows and outside resources need to manage elements, the workload moves to a new set of devices in a separate tailnet — the homelab tailnet untouched. The earlier draft’s separate-tailnet analysis is retained as the off-ramp design: structural isolation, independent policy file, clean audit boundary, external-group sharing without exposing private applications. Its alpha dependency (declarative node sharing) is acceptable once the boundary is real, because it then sits on a workload path that is allowed to degrade — the same risk posture, just correctly timed.

The pool taxonomy chosen now is the seam that cut will follow: pools map cleanly to future tailnet membership, and a split becomes a placement/policy exercise rather than a redesign. Policy-file CI should be in place first — a split that is mostly policy movement is only cheap if the policy is already tested.

Why this succeeds where ADR-0006 failed

Future option: separate tailnet (deferred, priced)

Noted, not built. When the trigger fires — outside resources managing elements, or sustained external-party growth — execute the off-ramp: new device set in a separate tailnet, provisioned via the API chain (OAuth exchange → create tailnet → httpsEnabled → tagOwners/policy → scoped auth keys), homelab tailnet untouched, cross-boundary access via declarative node sharing. Joining the waitlist for declarative sharing is only needed when the trigger becomes real; doing it now starts a clock for a feature nothing active depends on.

Open questions

1. Pool taxonomy — deliberately unspecified here. The concrete pools, names, membership, and grants are the follow-on design task. Implicit constraint from this ADR: no “miscellaneous” pool; a catch-all pool is how pools rot into a flat network.
2. Does the partner need any workload access? Default assumption: no — workload pools are operator-only. Link-based sharing remains available per-need; the answer affects whether the future split needs external-group modeling at all.
3. Exit node — does workload need its own, or use none?
4. Split trigger threshold — what concretely constitutes “outside resources managing elements”? Naming the threshold now makes the off-ramp decision unemotional later.
5. Quarantine semantics / synced groups — relocated to the off-ramp; only apply once a boundary exists.

Consequences / follow-on work

• Design the pool taxonomy and draft the HuJSON policy — tracked TODO, not done here. A policy skeleton with the pool structure and directional grants, reviewed against the principles above.
• Policy-file CI with ACL tests — now more load-bearing than under either prior framing: pools keep isolation advisory, and CI is what converts it into tested isolation. Ban tag:* destinations, require review on the policy file, map pool grants to tests, run in GitHub Actions.
• Update tailscale-grants.md once the pool taxonomy is concrete; convert the environment-tag framing to pool framing.
• Update tailscale.md to document workload pools, link-based sharing for guests, and the deferred-split framing.
• Do not join the declarative-sharing waitlist yet — retracted from the earlier draft; only needed once the off-ramp trigger is real.
• Rename Dev → Workload in homelab docs, and be explicit that it is outside the promotion chain, so it isn’t mistaken for a pipeline stage.
• Revisit ADR-0001: if Dev leaves the promotion chain, the per-environment ResourceSync branch layout has one fewer environment to steer.

References

• ADR-0006: Tailnet-per-environment — the rejected environment split; contains the link-based vs. declarative sharing mechanism comparison and the API provisioning details
• Declarative node sharing — alpha, waitlist-gated; the off-ramp’s cross-boundary mechanism, not active today
• Share your machines with other users — link-based sharing; the active mechanism for external access under this ADR
• summer-with-tailscale — worked examples for the API-driven tailnet provisioning chain the off-ramp will use
• tailscale.md — current Services, Docktail/ScaleTail, and access-tier design
• tailscale-grants.md — current tag taxonomy and grant rules

ADR: Orchestration Model — Docker-native vs Kubernetes/ArgoCD for the Homelab

Context

The homelab currently standardizes on Docker (rootful, on Fedora Server baremetal hosts) managed through Komodo for GitOps. The prior narrative decision (“Komodo over Kubernetes (K3s + ArgoCD)” in planning/decisions.md) chose Docker-native Komodo over a K3s + ArgoCD stack on the grounds of complexity, risk tolerance, and concept alignment with existing familiarity with compose files.

This ADR reopens that axis. Two things have shifted the picture:

1. Declarative lifecycle is now a hard requirement, not a nice-to-have. The goal is to easily and declaratively spin up / tear down / manage resources across environments. That is exactly what native GitOps (ArgoCD reconciling manifests) does out of the box — and what Komodo does with more manual steering (ResourceSync branch gymnastics, per-environment TOML edits on merge, see ADR 0001).
2. ArgoCD is the more common / more documented route than Komodo. If the operational surface is “declare desired state, let a controller reconcile,” ArgoCD is the industry-default implementation. Komodo gets us most of the way but is a smaller ecosystem with rougher edges (Podman compat, stack-status quirks).

So the earlier “Kubernetes is overkill for a homelab” conclusion is being stress-tested against a simpler question: is the thing we’re optimizing for simplicity of the runtime or simplicity of declarative management? Those pull in different directions.

The tension

Options considered

Option A: All Docker-native (Komodo)

Standardize everything on Docker + Komodo ResourceSync.

• Pros: lowest runtime complexity; builds on existing Compose investment; Dev OCP cluster still provides separate K8s skills.
• Cons: declarative management is real but requires the merge-hygiene rituals from ADR 0001; smaller ecosystem; no first-class “one command spins up the whole graph.”

Option B: All Kubernetes (K3s/K8s + ArgoCD)

Replace Komodo with a K3s or K8s control plane and ArgoCD as the GitOps controller for every workload.

• Pros: single, ubiquitous declarative model; ArgoCD is the common, well-documented path; spin up/down is a Git commit; one pane of glass for everything.
• Cons: real operational complexity (control plane, CRDs, more moving parts); the original “overkill for homelab” risk is still valid; more failure modes to own 24/7.

Option C: Split model (leading direction, not yet decided)

Use Docker for “prod apps” and Kubernetes/ArgoCD for “workload apps.”

• Keeps the low-friction Docker path for stable, user-facing services (Plex, etc.) where runtime simplicity matters most.
• Puts the heavier / more dynamic / more experiment-driven workloads on K8s/ArgoCD where declarative reconciliation pays off and aligns with work skills.
• The only clear advantage of “all Kubernetes” (Option B) over this split is the single pane of glass — ArgoCD as the one view of everything. That is a real UX win, but it is currently the only decisive argument for unifying onto K8s, and it may not outweigh the standing complexity cost.

Decision

No conclusion yet, this ADR is intentionally left open.

The leading direction is Option C (split model), driven by the desire for easy declarative management without forcing the entire homelab onto a K8s control plane it may not need. The single strongest argument for abandoning the split and going all-in on Kubernetes (Option B) is the unified ArgoCD dashboard; that has not yet been judged worth the operational cost.

What would move this from Draft to a decision:

• A concrete definition of which apps count as “prod apps” (Docker) vs “workload apps” (K8s); the boundary is currently intuitive, not enumerated.
• A validated K3s + ArgoCD bring-up path on the homelab hardware (the original decision flagged K3s-in-prod as risky; that risk needs a fresh look).
• A judgment on whether “single pane of glass” is worth the control-plane burden, or whether two dashboards (Komodo + ArgoCD) is acceptable.

Consequences (potential)

If Option C is adopted:

• Two management planes to operate (Komodo for Docker, ArgoCD for K8s) — more tooling surface, but each stays within its comfort zone.
• Prod apps keep the low-ceremony Docker/Compose lifecycle.
• Workload apps get first-class declarative reconciliation and stronger skills transfer to the work OCP stack.
• Loses the “one ArgoCD view of everything” that Option B would provide.

If Option B is adopted instead:

• Single declarative model and dashboard for the whole homelab.
• Pays the standing K3s operational cost the original decision sought to avoid.
• Supersedes the “Komodo over Kubernetes” narrative in decisions.md — that section would need revisiting.

Open Questions

• Where exactly is the “prod app” vs “workload app” line drawn?
• Does “prod app” vs “workload app” need to be an treated as alternative approach to my originally declared “Prod/Test/Dev” environments from ADR 0001?
  • Idea: If this is the case, it would call into question the order of evolution in regards to hardware, with “Prod” still running on a dedicated device, having “Test” running within the ProxMox cluster, and having “Workload/Dev” on a larger version of the OCP cluster.
• Is the K3s-in-prod risk from the original decision still valid, or has the operational picture changed?
• If my personal work is focused largely on kubernetes, will it be better to have everything involved with kubernetes, or will having “prod apps” in docker (option C) encourage me to not mess with prod apps and focus my time on just the kubernetes side of the field?
• Is there a “single pane of glass” option that can manage both docker and kubernetes, or at least have visibility on both?
• Does losing a single pane of glass (under the split) actually hurt day-to-day, or is it tolerable?
• Does this reopen the runtime question (ADR 0003 / 0002) for the K8s-hosted workloads, or do they stay rootful Docker-in-VM / containerd as K8s dictates?

References

• Narrative decision: Komodo over Kubernetes (K3s + ArgoCD) — the earlier conclusion this ADR re-examines.
• ADR 0001: Per-environment Git branches via Komodo ResourceSync — the merge-hygiene cost of Docker-native GitOps.
• ADR 0002: Baremetal Docker Host OS Selection
• ADR 0003: Container Runtime Tradeoffs

ADR: Skip local Docker registry, focus on git pipeline

Context

The homelab needs a strategy for managing container images between the local environment and production. Initial considerations included running a local container registry (Artifactory/Lightwell analog) or relying on Docker Hub with version pinning. The homelab already uses Komodo GitOps, GitHub Actions for rendering, and Ansible validation playbooks.

Goals:

1. Avoid infrastructure overhead — A local registry adds a running service, storage management, and maintenance burden for a feature (offline access) that isn’t actually required.
2. Leverage existing pipeline investment — GitHub Actions, Ansible validation, and Komodo ResourceSync are already in place and can be extended to cover image governance.
3. Security evaluation through PR process — Image scanning and review can happen in Git before anything gets deployed, rather than after images sit in a registry.
4. Version pinning is already the pattern — Komodo configs and Git branches already provide effective version control; a registry would be redundant.

Decision

Skip running a local Docker registry as a stop-gap. Instead:

1. Pin images explicitly in Komodo TOML configs (e.g., nginx:1.25-alpine instead of latest) — this is the primary guardrail.
2. Extend the existing Git pipeline to include:
   • Image scan on PR (Trivy or Anchore)
   • Push approved images to Docker Hub / existing registry with pinned tags
   • Komodo picks up the new commit SHA / tag on sync
3. Use Docker Hub rate-limit-friendly practices — pull once, cache locally via Docker’s built-in store, and pin versions.
4. Revisit local registry later only if offline access or rate limits become actual problems.

Considered alternatives

Consequences

Positive

• No extra running service or storage to manage
• Security gates move left — images are scanned and reviewed in PR before any deployment
• Existing Git/GitHub Actions investment leveraged; no new infra needed
• Version pinning already the pattern via Komodo configs; registry would be redundant
• Simpler onboarding for new services — just add to pipeline, no registry config needed

Negative / risks

• Image scanning depends on CI runner availability and speed
• If Docker Hub rate limits become problematic, no local cache (but this hasn’t been an issue in practice)
• Security evaluation relies on PR review quality — same as any code change

Mitigations

• Add Trivy scan to existing GitHub Action as a checklist step — fail PR if critical vulnerabilities found
• Cache scanned results in GitHub Actions artifact so subsequent scans are fast
• If rate limits bite, add a registry later as a targeted fix (not infrastructure upfront)
• Document the “pinning is the guardrail” principle so teams don’t revert to latest

Status

Accepted — proceed with extending the Git pipeline for image governance instead of running a local registry.

References

• Decisions: decisions.md
• Komodo GitOps: Komodo ResourceSync docs
• Existing ADR: 0001-komodo-resourcesync-branch-per-environment.md
• Trivy scanning: trivy image-scan --exit-code 1 --severity HIGH,CRITICAL

ADR: Baremetal Docker Host OS Selection

Context

The homelab needs a host OS for baremetal Docker nodes (LenovoMini 1 for Prod, LenovoMini 2 for Test, and the DevDocker VM on ProxMox). Requirements:

1. RPM/DNF-based — user preference, already familiar with the ecosystem.
2. Komodo GitOps compatibility — Periphery agent must be able to manage container workloads.
3. Low-maintenance — set once and forget; not something to constantly troubleshoot.
4. Easily automatable — host provisioning should be reproducible via Ansible, including Periphery agent setup.

Options Considered

Host OS

Container Runtime

Research Findings

Podman + Komodo Compatibility

After investigation, Podman support in Komodo relies on workarounds:

• Requires DOCKER_HOST env pointing at Podman’s Docker API socket (/run/user/$UID/podman/podman.sock)
• Stacks may incorrectly show as “DOWN” after successful deploy due to compose network label differences (GitHub issue #504)
• No first-class testing or documentation from Komodo maintainers
• Conclusion: Support is too rough for a low-maintenance setup

Rootful vs Rootless Decision

Rootless Docker was initially preferred for security, but three factors drove the switch to rootful:

• No practical benefit for a homelab — all workloads are user-deployed and trusted. The socket is already protected by host-level access controls (SSH, firewall, physical security).
• Avoids community-driven maintenance burden — rootless Docker with Komodo Periphery relies on community patterns not official docs. Rootful is the documented, tested path.
• Simplifies Ansible automation — rootful Docker is a single dnf install + daemon enable. No dockerd-rootless-setuptool.sh, no loginctl enable-linger, no environment variable wiring.

Host OS Alignment

• Fedora Server ships recent kernel + DNF, packages are fresh enough for both the Docker CE repo and Ansible management
• Rocky/Alma would work but are conservative; no advantage for this use case
• Fedora CoreOS is immutable — no package install at runtime, conflicts with Ansible-based provisioning

Decision (Provisional)

1. Fedora Server for the host OS — DNF-native, recent kernel, familiar tooling, easy Ansible provisioning.
2. Docker (not Podman) for the container runtime — Komodo’s first-class support removes any reason to fight Podman compatibility.
3. Rootful Docker — simplicity wins for a homelab with trusted workloads. No port restrictions, no privileged container limitations, no extra setup steps.
4. Containerized Periphery — deployed as a Docker container managed by Ansible, with /var/run/docker.sock mounted for API access. Updates via image pull, not manual binary replacement.

Implementation: Containerized Periphery via Ansible

Periphery runs as a Docker container, managed by the same Ansible playbook that provisions the host. The Komodo maintainer’s filesystem semantics concern (discussion #220) applies to rootless Docker where user namespaces cause path remapping — with rootful Docker, container and host paths are the same, so the concern doesn’t apply.

Ansible Role Structure

The host provisioning playbook will include a role komodo-periphery that handles Periphery as a container:

roles/
  komodo-periphery/
    tasks/
      main.yml          # pull, create dirs, run container
    templates/
      periphery.config.toml.j2   # config template
    vars/
      main.yml          # defaults (core URL, port, etc.)

Tasks (main.yml)

- name: Ensure Periphery data directory exists
  ansible.builtin.file:
    path: /opt/periphery
    state: directory
    mode: '0755'

- name: Deploy Periphery container
  community.docker.docker_container:
    name: periphery
    image: "ghcr.io/moghtech/periphery:{{ periphery_version }}"
    restart_policy: unless-stopped
    network_mode: host
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/opt/periphery:/config"
      - "/:/mnt/host:ro"
    env:
      PERIPHERY_CONFIG_PATH: /config/periphery.config.toml

Key choices:

• network_mode: host — Periphery needs to reach containers by their host-level ports for monitoring. Host networking avoids port mapping conflicts and simplifies the network topology.
• /var/run/docker.sock mount — gives Periphery the standard Docker API access. With rootful Docker this is the native socket, no indirection needed.
• /:/mnt/host:ro — bind mounts the host root so Periphery can read filesystem stats for disk monitoring.
• restart_policy: unless-stopped — survives reboots and daemon restarts.

Config Template (periphery.config.toml.j2)

[periphery]
port = {{ periphery_port | default(8120) }}
root_directory = "/mnt/host"

[[connections]]
address = "wss://{{ core_address }}/connection"
connect_as = "{{ inventory_hostname }}"
onboarding_key = "{{ periphery_onboarding_key }}"

The config template is rendered by Ansible with host-specific variables (core address, hostname, onboarding key), sourced from Ansible vault or group vars.

Ansible Playbook Structure

- hosts: docker_hosts
  become: yes
  roles:
    - role: docker-install        # dnf install docker, enable + start
    - role: komodo-periphery      # pull image, deploy container

Gotchas

1. SELinux may block container socket access. On Fedora Server with SELinux enforcing, the Periphery container needs the :z flag on volume mounts or a dedicated SELinux policy. If Periphery fails to connect, check ausearch -m avc and add :z to the socket mount: "/var/run/docker.sock:/var/run/docker.sock:z".

2. Host networking limits to one Periphery per host. With network_mode: host, port 8120 is consumed directly on the host. Only one Periphery instance can run per machine. This is the expected topology — one agent per host.

3. Periphery container image tag strategy. Use a specific version tag (e.g., :0.2.5) not :latest for reproducible deployments. Update by bumping the version in Ansible vars and re-running the playbook.

4. Onboarding key rotates. The onboarding_key in [[connections]] is a one-time use key from the Komodo Core UI. After first connection, Periphery persists its session — the key is no longer needed. If re-deploying from scratch, generate a new key.

5. Config changes require container restart. If periphery.config.toml changes, Ansible must trigger a container restart. The docker_container module handles this automatically when config changes are detected via template diff.

6. Ansible vault for secrets. The onboarding key is sensitive and should be stored in Ansible vault, not in plaintext group vars.

Consequences

Positive:

• Familiar RPM/DNF management
• Komodo Periphery has native, first-class support for rootful Docker
• All workloads work without restriction — privileged containers, ports <1024, cgroup limits
• Containerized Periphery auto-updates via docker pull; Ansible handles version pinning
• Ansible-driven provisioning means new hosts are reproducible
• No special setup beyond standard Docker CE install

Negative / risks:

• Docker socket is root-equivalent — compromised container has host access. Mitigated by homelab trust model (all workloads are user-deployed).
• Containerized Periphery adds a Docker-in-Docker layer that the Komodo maintainer flags for potential filesystem confusion, but with rootful Docker (no user namespace) this is not a practical concern.
• Fedora ~13mo support window means OS upgrades every ~6-12mo (acceptable for homelab).
• Periphery config changes need container restart — automated via Ansible.

Open Questions

• Which Periphery image tag to pin initially? Start with :latest during bring-up, pin to a specific version once stable.
• How to handle Periphery version upgrades? Ansible var bump + playbook re-run. Need a process for checking upstream releases.

References

• ADR 0003: Container Runtime Tradeoffs — deeper analysis of Docker rootful, rootless, and Podman comparison
• FoxxMD — Migrating to Komodo (rootless Periphery via socket-proxy, systemd agent recommendation)
• Docker — Rootless Docker setup
• Komodo — Periphery installation docs
• Komodo issue #504 — Podman compose network status mismatch
• Komodo discussion #220 — Maintainer recommends systemd agent over containerized

ADR: Container Runtime Tradeoffs — Docker (Rootful), Docker (Rootless), and Podman

Context

Selecting a container runtime for baremetal Docker hosts involves balancing security, operational convenience, and tooling compatibility. This ADR breaks down three approaches and the fundamental tension between central-socket management and daemonless isolation.

The host OS decision is handled in ADR 0002: Baremetal Docker Host OS Selection.

Runtime Options

Docker (Rootful)

The classic model: dockerd runs as root, exposes /var/run/docker.sock.

Docker (Rootless)

dockerd runs under a user namespace, exposes a user-scoped socket.

Podman (Daemonless)

No central daemon. Each container is launched directly by the user’s Podman client.

Analysis

Security: The Central Socket Risk

A Docker socket is equivalent to root access on the host. Any process that can write to the socket can:

• Start containers with host filesystem mounts (read/write any file)
• Create privileged containers that break out of isolation
• Modify iptables, network namespaces, cgroups
• Access secrets mounted into any container

Rootful Docker places this power on a single, well-known path (/var/run/docker.sock). The docker group is effectively root. The blast radius of a compromised process with socket access is the entire host.

Rootless Docker mitigates this through user namespaces. The daemon and all containers run within the user’s namespace — root inside a container is the user outside. A compromised socket can’t escape the user namespace, so the blast radius is that user’s resources only. However, it’s still a central daemon — if the rootless dockerd process is exploited, all that user’s containers are compromised.

Podman eliminates the central socket entirely. No process management attachment, no daemon to target. Each container is a child of its launching process. The blast radius is per container — no socket to poison. To use management tooling, you must explicitly start a podman system service socket, which reintroduces a central endpoint — but it’s opt-in, not the default.

Management: Daemonless vs. Central Socket

This is the inverse of the security tradeoff:

Without a central socket, management tooling faces a discovery problem:

• How does Komodo’s Periphery find the container runtime? It can’t — it needs an explicit path to podman system service socket, which must be running as a systemd user service.
• How does docker ps / podman ps enumerate running containers across a session? Podman uses a per-user boltdb database, but there’s no persistent daemon registering container state.
• How do compose stacks get tracked? Podman’s compose support (podman-compose) uses a different project model than Docker Compose — Komodo’s stack status detection breaks (issue #504).

With a central socket, management is trivial: connect to the well-known path, issue standard API calls, get predictable results. The cost is the security risk described above.

Operational Considerations

Recommended Direction

This ADR does not make a standalone decision — it informs ADR 0002. The current direction:

Docker (rootful) for maximum simplicity and tooling compatibility. The socket-security risk is accepted as an acceptable tradeoff for a homelab with trusted workloads.

Podman is the more secure architecture by design, and rootless Docker is a middle ground. Rootful Docker optimizes for operational simplicity — no setup ceremony, no port restrictions, no privileged container limitations. For a homelab where all workloads are user-deployed and trusted, the central-socket risk is manageable.

If the homelab expands to include untrusted or multi-tenant workloads, this should be revisited — rootless Docker or Podman would contain blast radius in a way rootful Docker cannot.

Consequences

If Docker (rootful) is adopted (current direction):

• All management tooling works natively with zero configuration — standard Docker API on the well-known socket
• Setup is minimal — dnf install docker, enable daemon, run containers
• No workload restrictions — privileged containers, ports <1024, cgroup limits all work
• Socket is root-equivalent — any process with socket access owns the host. Acceptable for homelab trusted workloads.

If Docker (rootless) or Podman is reconsidered later:

• Stronger isolation — compromise is contained to a user namespace or individual container
• Some workloads (privileged containers, Swarm mode) are restricted
• Management complexity increases — socket discovery, env vars, emulation layers required for Komodo
• Ports <1024 need workarounds

References

• Docker: Rootless Docker
• Docker: Docker socket security
• Podman: Rootless Podman
• Podman: Systemd integration
• Komodo issue #504 — Podman compose network status mismatch
• ADR 0002: Baremetal Docker Host OS Selection

ADR: Per-environment Git branches via Komodo ResourceSync

Context

The homelab uses Komodo with ResourceSync to declare servers, stacks, deployments, and related resources from Git (an “app-of-apps” style flow: Git remains the audit trail; Komodo reconciles).

Goals:

1. main shows the whole picture — dev/, test/, and prod/ (or equivalent komodo/{dev,test,prod}/) layouts exist together on main so reviews and long-lived docs match reality after merges.
2. Per-environment steering — each environment’s sync can follow main or follow a named branch (for example to soak a stack change in Dev before it lands on main).
3. Homelab-realistic ergonomics — accept some manual merge hygiene instead of building full release automation up front.

An earlier sketch of this approach lived in wip/komodo-artifact-templates/scratch/branch_management_ideas.md.

Decision

1. Single infrastructure repo; main is the default line of truth
   After work merges, main holds the canonical layout for all managed environments (folder-separated configs, per existing repo structure docs).

2. One ResourceSync configuration per Komodo-managed environment (Dev, Test, Prod — or whichever environments are GitOps-managed)
   Each [[resource_sync]] (or equivalent Komodo sync configuration) sets, at minimum:

   • Repository (and git provider / account as required).
   • Branch (or other ref field supported by Komodo’s ResourceSync schema — see Resource sync config) independently per environment.
   • Resource path scope limited to that environment’s subtree (for example only komodo/dev/** for Dev), so a branch used for experiments does not accidentally apply Prod definitions.

3. Branch-testing workflow

   • Create a feature branch for the change under test.
   • Point the target environment’s ResourceSync at that branch (for example Dev follows feature/xyz while Test and Prod continue to follow main).
   • After validation, merge the feature branch to main.
   • Manually reconcile the ResourceSync metadata on main so environments that should track main again have their branch/ref set back to main (or another agreed default).

   Known quirk (accepted): the ResourceSync TOML that sets branch = "feature/xyz" lives in Git. On the feature branch that file typically must reference the same branch so the sync pulls a self-consistent tree. After merge to main, that same file’s branch field must be updated to main (or removed to default) for environments that should no longer track the feature branch. This is intentional small, explicit edits on main, not hidden automation, until/unless you add tooling.

4. Releases / tags as a pin for Prod (optional, not required now)
   Using tags or GitHub Releases so Prod tracks refs/tags/v… while main moves forward is a valid pattern when you want Prod immutability without branch gymnastics. With separate per-environment directories and per-sync branch control, the urgency is lower than in a single-shared-path model. We defer mandatory release-based promotion; revisit if main churn becomes painful for Prod.

Consequences

Positive

• Clear mental model: folders = promotion path; branch on ResourceSync = “which revision of that tree is live for this environment.”
• Feature work can soak on Dev/Test without forking the whole repo layout conceptually on main.
• Decisions stay in Git and stay reviewable.

Negative / risks

• Merge hygiene: forgetting to repoint a ResourceSync after merge can leave an environment tracking a stale or abandoned branch.
• Self-consistency: sync config and synced paths must agree on branch, or behavior is confusing during transitions.

Mitigations (recommended, non-blocking)

• Short checklist when merging infra PRs: “ResourceSync branches updated for Dev/Test/Prod as intended?”
• Keep ResourceSync definitions in small, dedicated files (for example under komodo/sync/ or similar) so diffs are obvious.
• Optionally add CI later (grep/assert expected branch for Prod) if mistakes recur.

Alternatives considered

References

• Komodo: Sync resources
• Homelab repo layout: Planned repository structure (includes promotion flow between environment folders)
• Prior scratch note: wip/komodo-artifact-templates/scratch/branch_management_ideas.md