BigFix Agent DaemonSet Deployment - DaemonSet design model for executing BigFix security agents on underlying OpenShift worker nodes.
Consolidation Roadmaps
Infrastructure Consolidation Roadmap - Multi-phase migration plan for transitioning core homelab services and workloads from independent virtual machines to unified Kubernetes/OpenShift containers.
Subsections of Architecture & System Designs
GitOps Cluster Consolidation & Version Control Roadmap
This roadmap outlines the strategic phased approach for consolidating disparate, tag-drifted, or unmanaged GitOps clusters (environments) back into a fully version-controlled lifecycle managed off of a single main branch with tagging.
Objective
To eliminate repository drift, simplify environment configuration, and establish a clear promotion path by:
Baselining all clusters onto a single trunk (main) with no active tags or branches.
Cleaning up all unused/stale git branches and tags.
Separating testing environments onto a dedicated long-lived Testing branch.
Pinning all remaining environments (production, shared services) to a new, immutable release tag (v2.0).
High-Level Flowchart
flowchart TD
%% Current State
subgraph Current [Current State: Drift & Disparate Mapping]
C_Prod[Prod Clusters / Tags/Branches]
C_Test[Test Clusters / Diverse Tags]
C_Unmanaged[Unmanaged Clusters / No Sync]
end
%% Phase 1
subgraph Phase1 [Phase 1: Trunk Consolidation]
Main_Branch[(Git Repo: main branch)]
P1_Sync[Point ALL Cluster GitOps/ResourceSync to main branch]
P1_Clean[Prune all stale tags & inactive branches]
end
%% Phase 2
subgraph Phase2 [Phase 2: Testing Separation]
Test_Branch[(Git Repo: Testing branch)]
P2_Sync[Point Test Clusters to 'Testing' branch]
end
%% Phase 3
subgraph Phase3 [Phase 3: Version Stabilization]
V2_Tag[(Git Tag: v2.0)]
P3_Sync[Point Production & Other Clusters to 'v2.0' tag]
end
%% Transitions
Current -->|Align and Point to main| P1_Sync
P1_Sync -->|Prune old Git refs| P1_Clean
P1_Clean -->|Create 'Testing' from main| Test_Branch
Test_Branch -->|Configure Sync| P2_Sync
P1_Clean -->|Tag main as v2.0| V2_Tag
V2_Tag -->|Configure Sync| P3_Sync
%% Styling
style Current fill:#fdd,stroke:#f66,stroke-width:2px;
style Phase1 fill:#ffd,stroke:#dd6,stroke-width:2px;
style Phase2 fill:#dfd,stroke:#6b6,stroke-width:2px;
style Phase3 fill:#ddf,stroke:#66b,stroke-width:2px;
Goal: Consolidate all GitOps controllers/ResourceSync configurations to point strictly to the main branch with zero tag/revision overrides.
Map and Align Configurations:
Update all cluster GitOps configs (e.g., Komodo [[resource_sync]] target revisions or ArgoCD target revisions) to use the main branch.
Ensure all cluster-specific directories (e.g., komodo/prod/, komodo/test/, komodo/dev/) are unified on main.
Reconcile and Validate:
Allow GitOps controllers to reconcile. Confirm that all clusters are running stable configurations served purely from main.
Repository Pruning:
Delete all unused, stale, and tracking branches.
Purge old tags from the remote git origin to prevent metadata clutter and reference confusion.
Phase 2: Testing Isolation
Goal: Branch out the testing clusters to a dedicated, long-lived branch to isolate validation work from main production configurations.
Branch Creation:
Branch Testing directly from the cleaned and verified main branch.
Target Re-pointing:
Modify the GitOps sync configurations of your Test Clusters to track the Testing branch instead of main.
Promotion Workflow Baseline:
Features/workloads are committed to main (or individual feature branches), validated, merged to main, and then merged from main to Testing for soak tests.
Phase 3: Release Pinning (v2.0 Tag)
Goal: Establish release stability by pinning production and all non-test clusters to an immutable point-in-time release.
Tag Generation:
Tag the stable main branch with the new version tag: v2.0.
Target Re-pointing:
Update the GitOps sync configurations for all remaining clusters (e.g., Production, Shared Core Services) to target refs/tags/v2.0.
Establish Lifecycle Operations:
All subsequent production updates are promoted from Testing back to main, and then promoted to production by updating/moving the target tag (e.g., v2.1, v2.2) or updating the tag pointer.
Key Benefits
Clean Git Slate: Drastically reduces active git refs, making branch history easy to read and manage.
Predictable Promotion: Features move from main $\rightarrow$ Testing branch $\rightarrow$ v2.0 tag.
Auditability: Clear division of what is being tested versus what is running in production based purely on git references.
Long-Term Management
To ensure continuous operational stability and transparency, clusters are categorized by environment type and mapped to specific Git target references, utilizing a flat repository structure for environment settings.
Cluster Target References
Different environments track different Git references depending on their operational requirements and strictness levels:
Test Clusters:
Target Reference:Testing branch.
Usage: Serves as the immediate validation target for developers and GitOps sync, ensuring new features and configurations are continuously integrated and tested.
Pre-Production and Normal Production Clusters:
Target Reference:Main branch.
Usage: Serves as the primary stable branch. Pre-production acts as the final validation gate, while normal production tracks the main trunk directly for rapid, reliable delivery of verified features.
Strictly Managed Clusters:
Target Reference: Tagged releases (e.g., v2.x).
Usage: Pin to immutable release tags to guarantee absolute control over rollout schedules and facilitate rigorous change management.
Flat Repository Structure
Instead of deep hierarchies or branching drift for configuration variants, the repository is organized with a flat directory structure at its root or specific config directory (e.g., komodo/ or gitops/).
For each environment, a dedicated, self-contained subdirectory contains all specific configurations and override settings:
gitops-repo/
βββ test/ # Settings specific to Test clusters (tracks 'Testing' branch)
βββ preprod/ # Settings specific to Pre-production (tracks 'Main' branch)
βββ prod/ # Settings specific to Normal Production (tracks 'Main' branch)
βββ prod-strict/ # Settings specific to Strictly Managed Production (tracks immutable tags)
βββ shared/ # Shared base resources or templates referenced by environments
This flat design guarantees that:
High Visibility: All environment-specific settings are completely visible in their respective subdirectories.
No Hidden Overrides: Eliminates nested inheritance or complex drift, making configurations easy to audit, compare, and modify.
Declarative Consistency: All target references can read from the same commit or release tag while cleanly resolving environment-specific configurations via their designated subdirectory.
BigFix Host Agent Client Deployment on OpenShift via GitOps
Overview
This guide details the architecture, design decisions, and configuration required to deploy the HCL BigFix Host Agent Client onto a Red Hat OpenShift Container Platform (OCP) cluster using Argo CD.
This implementation rigorously adheres to and matches the standardized Helm component structures, sync-waves, and folder layouts defined in the helm-component-pattern artifact collection.
The Host Agent Client runs directly on each physical/virtual cluster node. It monitors host operating system settings, hardware properties, and installed package inventories, and applies host-level configuration patches.
Architecture
Because the BigFix Host Agent inspects and manages the underlying Red Hat Enterprise Linux CoreOS (RHCOS) or RHEL host OS, it must execute with root/privileged credentials.
hostPID: true: Direct visibility into the host’s process tree.
hostNetwork: true: Direct communication with the network via host IP interfaces.
Host Volume Mounts:
Host / mounted to /host so the agent can read/write root configuration files.
Host /var/opt/BESClient mounted to persist client database, certificates, and logging state across container rebuilds.
OpenShift Security Posture & SCCs
By default, OpenShift strictly forbids containers from running as root, mounting host paths, or accessing host namespaces. To deploy a privileged host agent, you must grant the DaemonSet’s ServiceAccount the privileged Security Context Constraint (SCC).
This is achieved securely in GitOps by creating a RoleBinding within the target namespace that binds the ServiceAccount to the cluster-provided system:openshift:scc:privilegedClusterRole:
The deployment is self-contained under artifacts/openshift/bigfix-agent-deployment/.
Namespace & ServiceAccount:
The namespace hcl-bigfix is isolated with monitoring active.
Masthead Secret Injection:
Every BigFix client requires an actionsite.afxm masthead file to verify keys and trace upstream Relays. This is injected as a Kubernetes Secret of type Opaque.
DaemonSet Setup:
The DaemonSet includes explicit tolerations to ensure the client is deployed onto Master/Control-Plane nodes as well as worker nodes, ensuring 100% host coverage:
Missing Master Tolerations:
Without master/control-plane node tolerations, control-plane nodes will lack auditing coverage, creating a blind spot in compliance and security tracking.
State Persistence:
If the /var/opt/BESClient host volume mount is omitted, the client will generate a new unique ComputerID on every pod restart. This will bloat the BigFix Console with thousands of duplicate offline entries. Always ensure this directory is mapped to a host path (using DirectoryOrCreate).
BigFix Inventory (BFI) Discovery & NFD Deployment via GitOps
Overview
This guide details the architecture, design decisions, and configuration required to deploy the HCL BigFix Inventory (BFI) discovery and sub-capacity licensing integration on Red Hat OpenShift Container Platform (OCP).
This implementation rigorously adheres to and matches the standardized Helm component structures, sync-waves, and folder layouts defined in the helm-component-pattern artifact collection.
The BFI discovery integration is a headless, API-level integration. It does not install a resident server. Instead, it deploys a ServiceAccount and a long-lived API token that allows the external BigFix Inventory server to query OpenShift APIs. It also deploys the Red Hat Node Feature Discovery (NFD) Operator to map and label host CPU sockets and cores, which BFI queries to calculate Processor Value Unit (PVU) licensing.
Architecture & Integration Flow
The integration bridges the external BFI server and the OpenShift Control Plane:
API Discovery: The external BFI server connects via the big-fix-serviceaccount token to read nodes, namespaces, and virtualmachineinstances (to track OpenShift Virtualization VM cores).
CPU Topology Labeling: The NFD Operator discovers core and socket details on each bare-metal or virtual node, and matches them against an ILMT-specific NodeFeatureRule to apply standard capacity labels (e.g., ibm.ilmt.cores, ibm.ilmt.sockets). BFI queries these labels to generate compliant sub-capacity reports.
GitOps Sync-Wave Strategy
Applying operators and their dependent Custom Resources (CRs) simultaneously inside GitOps can cause synchronization deadlocks because the Custom Resource Definitions (CRDs) do not exist when the CR templates are parsed.
To ensure a seamless, zero-race-condition bootstrap, we separate the operator from its CR instances and structure the workloads across Argo CD Sync-Waves:
Wave
Component
Action
2
components/openshift-nfd/operator
Installs the NFD namespace, OLM OperatorGroup, and Subscription to register the CRDs.
3
components/openshift-nfd/instance
Installs the NodeFeatureDiscovery controller instance custom resource.
4
components/hcl-bigfix-discovery
Deploys the discovery namespace, API Token Secret, RBAC rules, and the NFD NodeFeatureRule CR.
Configuration Details
RBAC Permissions for BFI Server
The external BFI Server relies on the ServiceAccount token to authenticate and call APIs. The ClusterRoles must be isolated to read-only resource lists:
The NFD NodeFeatureRule injects values into labels using Go template notation ({{ }}). Because Helm also parses Go templates, you must escape these braces in Helm source templates:
This forces Helm to output the braces literally, allowing the NFD operator to interpret them at runtime.
Common Pitfalls
ServiceAccount Tokens in OpenShift 4.11+:
Kubernetes 1.24+ and OpenShift 4.11+ no longer generate static long-lived token Secrets automatically when a ServiceAccount is created. You must manually define a Secret of type kubernetes.io/service-account-token with the ServiceAccount annotation:
CRD Lifecycle Race Conditions:
Do not combine NFD Operators and NodeFeatureRule into a single Helm component. Always run them in separate waves or as distinct applications to ensure the operator is running and ready to handle the NodeFeatureRule Custom Resource.
This guide details how to set up an on-demand, dynamic Tailscale integration in OpenBao. When a client (such as a Docker container) requests a secret from docker/tailscale/auth-token/<name> (for example, docker/tailscale/auth-token/nginx), OpenBao will automatically call the Tailscale API to generate a short-lived, preauthorized, and ephemeral authentication key labeled for that specific container, returning it to the client immediately.
Architectural Choices & Routing
Can we support other paths later (e.g., docker/tailscale/idp)?
Yes, absolutely. You have two clean options to achieve this:
Option A: Extend the Plugin (Recommended)
Our plugin is modular. If you need a new path later, such as docker/tailscale/idp that interfaces with another service or maps configuration, you can simply add a new path handler inside the plugin code (e.g., creating path_idp.go and adding it to Paths in backend.go). This keeps all Tailscale-related integrations housed within a single secrets engine binary.
Option B: OpenBao Nested Mount Routing
OpenBao (and HashiCorp Vault) routes API requests based on the most specific matching path prefix.
If you mount our plugin at docker/tailscale, it owns everything under docker/tailscale/*.
However, you can mount a completely separate secrets engine (like a standard Key-Value store or a database engine) at a nested subpath, such as docker/tailscale/idp:
bao secrets enable -path=docker/tailscale/idp kv-v2
In this scenario, a request to docker/tailscale/idp/my-secret is routed to the KV engine, while a request to docker/tailscale/auth-token/nginx is routed to our Tailscale plugin.
Prerequisites
OpenBao running in dev mode on the host (127.0.0.1:8200 with root token dev-only-token).
Docker installed on the host (used to compile the plugin without requiring a Go installation on your machine).
A Tailscale API Key (e.g., example-api-key-here) and your Tailnet name.
Step-by-Step Configuration
Step 1: Compile the Plugin using Docker
Since Go is not required on your host, you can compile the plugin binaries for macOS and Linux using a containerized Go environment.
Navigate to the plugin source folder:
cd wip/openbao-tailscale-integration
Build the Docker builder image:
docker build -t openbao-tailscale-builder .
Run the builder to extract the binaries into a bin/ directory on your host:
docker run --rm -v "$(pwd)/bin:/out" openbao-tailscale-builder
This creates four binaries in your wip/openbao-tailscale-integration/bin/ folder:
To load external plugins, OpenBao must be configured with a plugin directory.
If running OpenBao in Dev Mode on your macOS host, start the server using the -dev-plugin-dir flag and point it to the absolute path of your built binaries:
# Create a dedicated plugin directorymkdir -p /Users/sam/Workspace/openbao-plugins
# Copy the appropriate binary to your plugins directory# For Apple Silicon macOS:cp bin/openbao-plugin-secrets-tailscale-darwin-arm64 /Users/sam/Workspace/openbao-plugins/openbao-plugin-secrets-tailscale
chmod +x /Users/sam/Workspace/openbao-plugins/openbao-plugin-secrets-tailscale
# Start OpenBao dev server with the plugin folder loadedbao server -dev -dev-root-token-id="dev-only-token" -dev-plugin-dir="/Users/sam/Workspace/openbao-plugins"
Step 3: Register the Plugin in the Catalog
Before enabling the secrets engine, OpenBao requires you to register the binary in its internal catalog with a SHA256 signature to guarantee code integrity.
This script can run as the entrypoint inside your docker container (in this case, for an nginx service) to automatically register itself on Tailscale using the token fetched from OpenBao:
#!/bin/sh
set -e
# Define the service nameSERVICE_NAME="nginx"echo "Fetching Tailscale auth token from OpenBao for ${SERVICE_NAME}..."# Fetch the dynamic auth token using curlBAO_RESPONSE=$(curl -s \
--header "X-Bao-Token: dev-only-token"\
http://host.docker.internal:8200/v1/docker/tailscale/auth-token/${SERVICE_NAME})# Extract the token value from the JSON responseTAILSCALE_AUTH_KEY=$(echo "$BAO_RESPONSE" | grep -o '"auth_token":"[^"]*' | grep -o '[^"]*$')if[ -z "$TAILSCALE_AUTH_KEY"]; then echo "Error: Failed to retrieve Tailscale auth key from OpenBao." exit 1fiecho "Successfully retrieved auth token. Starting Tailscale..."# Authenticate and bring up Tailscale node in the containertailscaled --state=mem: &
sleep 2tailscale up --authkey="${TAILSCALE_AUTH_KEY}" --accept-dns=false
echo "Tailscale is connected!"exec "$@"
Example: Docker Compose configuration
version: '3.8'services:
app-node:
image: alpine:latestcontainer_name: tailscale_nginxcap_add:
- NET_ADMINdevices:
- /dev/net/tun:/dev/net/tunvolumes:
- ./entrypoint.sh:/entrypoint.shentrypoint: ["/bin/sh", "/entrypoint.sh"]
command: ["ping", "google.com"]
extra_hosts:
- "host.docker.internal:host-gateway"# Resolves the host OpenBao address on Linux
Troubleshooting & Common Pitfalls
Permission Denied (Execution): If OpenBao logs fork/exec /Users/sam/Workspace/openbao-plugins/openbao-plugin-secrets-tailscale: permission denied, verify that the plugin file is executable by running chmod +x on it.
Architecture Mismatch: Ensure you copied the correct architecture binary. If OpenBao runs on Apple Silicon macOS, you must use the darwin-arm64 binary. If it runs inside a Linux Docker container, you must use the linux-amd64 or linux-arm64 binary depending on your host.
Invalid Tag Prefix: When requesting a token with tags, Tailscale API strictly requires tags to begin with tag: (e.g. tag:docker, not docker).