Checking OpenShift Certificates

This guide covers methods for checking the expiration dates of critical OpenShift certificates, including the API server and Ingress controller. While OpenShift typically auto-rotates most internal certificates, manually verifying them can be useful for troubleshooting and custom monitoring.

Key Certificate Locations

Critical certificates are stored as Kubernetes Secrets across different namespaces:

Quick Checks

Use these one-liners to manually inspect specific certificates.

API Server Certificate

To check the expiration date directly:

oc get secret -n openshift-kube-apiserver kube-apiserver-cert-key-pair \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -enddate -noout

To view full certificate details including the validity period:

oc get secret -n openshift-kube-apiserver kube-apiserver-to-kubelet-signer \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout | grep -A2 "Validity"

Default Ingress Controller Certificate

To check the expiration date:

oc get secret -n openshift-ingress router-certs-default \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -enddate -noout

Automated Expiration Checks

When dealing with many certificates, automation is necessary. Here are several approaches for scripting certificate checks.

Option 1: Multi-Namespace Checking

This script iterates over key namespaces and extracts the end date for all TLS secrets:

#!/bin/bash

NAMESPACES=("openshift-kube-apiserver" "openshift-ingress" "openshift-config-managed")

for ns in "${NAMESPACES[@]}"; do
    echo "======== $ns ========"
    oc get secret -n "$ns" -o json 2>/dev/null | \
      jq -r '.items[] | select(.type == "kubernetes.io/tls") | .metadata.name' | \
      while read secret; do
        echo -n "$secret: "
        oc get secret -n "$ns" "$secret" -o jsonpath='{.data.tls\.crt}' | \
          base64 -d | openssl x509 -enddate -noout | cut -d= -f2
    done
    echo ""
done

Option 2: Comprehensive Warning Script (Recommended)

This script calculates the remaining days for each certificate in a namespace and provides warnings if they are close to expiring. It handles epoch time conversion for robust comparison.

#!/bin/bash

NAMESPACE="openshift-kube-apiserver"
DAYS_WARNING=30

oc get secret -n "$NAMESPACE" -o json | jq -r '.items[] | select(.type == "kubernetes.io/tls") | .metadata.name' | while read secret; do
    echo "=== Certificate: $secret ==="
    
    # Extract and decode certificate
    CERT=$(oc get secret -n "$NAMESPACE" "$secret" -o jsonpath='{.data.tls\.crt}' | base64 -d)
    
    # Get expiration date in seconds since epoch
    EXPDATE=$(echo "$CERT" | openssl x509 -enddate -noout | cut -d= -f2)
    EXPDATE_EPOCH=$(date -d "$EXPDATE" +%s 2>/dev/null || date -j -f "%b %d %T %Y %Z" "$EXPDATE" +%s)
    NOW=$(date +%s)
    DAYS_LEFT=$(( ($EXPDATE_EPOCH - $NOW) / 86400 ))
    
    # Display results
    echo "Expires: $EXPDATE"
    echo "Days remaining: $DAYS_LEFT"
    
    # Warning if expiring soon
    if [ $DAYS_LEFT -lt $DAYS_WARNING ]; then
        echo "⚠️  WARNING: Certificate expires in $DAYS_LEFT days!"
    elif [ $DAYS_LEFT -lt 0 ]; then
        echo "❌ CRITICAL: Certificate has expired!"
    else
        echo "✓ OK"
    fi
    echo ""
done

Note: Save as check-certs.sh, run chmod +x check-certs.sh, and configure it as a cron job for regular monitoring.

Option 3: Quick One-Liners

If you prefer one-liners over a script, you can use jq to loop through secrets:

oc get secret -n openshift-kube-apiserver -o json | \
  jq -r '.items[] | select(.type == "kubernetes.io/tls") | .metadata.name' | \
  while read secret; do
    echo "=== $secret ==="
    oc get secret -n openshift-kube-apiserver "$secret" \
      -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -enddate -noout
  done

Built-In Monitoring

While scripts are helpful for point-in-time checks, OpenShift provides built-in mechanisms for monitoring:

1. Prometheus Alerts: OpenShift’s default monitoring stack includes Prometheus alerts for certificate expiration.
2. Web Console: Check Observe → Alerting in the OpenShift web console for active certificate expiration warnings.
3. CSR Approval: For pending Certificate Signing Requests (often relevant during node addition or cert rotation), check status with oc get csr -w and approve manually if necessary using oc adm certificate approve <csr-name>.

Chaining GitHub Actions Workflows Without a PAT or GitHub App

How to trigger one workflow from another when you can’t (or don’t want to) use Personal Access Tokens or GitHub Apps.

The Problem

You have two workflows:

1. Workflow A — Detects branches and auto-creates PRs against main
2. Workflow B — Runs tests on PRs opened against main

Manual PRs trigger Workflow B. But when Workflow A creates a PR, Workflow B never runs.

Why This Happens

GitHub Actions has a deliberate security feature:

Actions performed using the default GITHUB_TOKEN do not trigger other workflows.

This prevents accidental infinite loops (Workflow A triggers B triggers A triggers B…). When Workflow A creates a PR using GITHUB_TOKEN, GitHub suppresses the pull_request event that Workflow B listens for.

Mitigation Options

Why Not PAT or GitHub App?

In enterprise environments, engineers often:

• Don’t have permission to create GitHub Apps at the org level
• Can’t or shouldn’t create PATs tied to personal accounts for shared workflows
• Face security policies that restrict token creation
• Need solutions that work within the default GITHUB_TOKEN permissions

The workflow_run approach works entirely within GitHub’s default permission model.

Solution: Using workflow_run Trigger

Instead of Workflow B listening for pull_request events (which won’t fire), have it listen for Workflow A’s completion.

Architecture

┌─────────────────┐         ┌─────────────────┐
│   Workflow A    │         │   Workflow B    │
│ (Auto-Create PR)│         │  (Run Tests)    │
└────────┬────────┘         └────────▲────────┘
         │                           │
         │ completes                 │ triggers on
         │                           │ workflow_run
         ▼                           │
    ┌────────────┐                   │
    │   PR       │───────────────────┘
    │  Created   │   (B finds PR from A's run)
    └────────────┘

Implementation

Workflow A: Auto-Create PR

# .github/workflows/auto-pr.yml
name: Auto Create PR

on:
  push:
    branches:
      - 'feature/**'
      - 'fix/**'

jobs:
  create-pr:
    runs-on: ubuntu-latest
    outputs:
      pr-number: ${{ steps.create-pr.outputs.pull-request-number }}
      pr-url: ${{ steps.create-pr.outputs.pull-request-url }}
    steps:
      - uses: actions/checkout@v4

      - name: Create Pull Request
        id: create-pr
        uses: peter-evans/create-pull-request@v5
        with:
          title: "Auto PR: ${{ github.ref_name }}"
          body: |
            Automated PR for branch `${{ github.ref_name }}`
            
            Created by workflow run: ${{ github.run_id }}
          base: main
          branch: ${{ github.ref_name }}

      - name: Output PR info
        run: |
          echo "Created PR #${{ steps.create-pr.outputs.pull-request-number }}"
          echo "URL: ${{ steps.create-pr.outputs.pull-request-url }}"

Workflow B: Run Tests on Workflow A Completion

# .github/workflows/pr-tests.yml
name: PR Tests

on:
  # Still trigger on manual PRs
  pull_request:
    branches: [main]

  # Also trigger when Auto Create PR workflow completes
  workflow_run:
    workflows: ["Auto Create PR"]
    types: [completed]

jobs:
  test:
    runs-on: ubuntu-latest
    # Only run if workflow_run was successful (skip if Workflow A failed)
    if: |
      github.event_name == 'pull_request' ||
      (github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success')
    
    steps:
      - name: Get PR info (for workflow_run trigger)
        id: pr-info
        if: github.event_name == 'workflow_run'
        uses: actions/github-script@v7
        with:
          script: |
            // Find PRs associated with the triggering workflow's head branch
            const headBranch = context.payload.workflow_run.head_branch;
            const headSha = context.payload.workflow_run.head_sha;
            
            const { data: prs } = await github.rest.pulls.list({
              owner: context.repo.owner,
              repo: context.repo.repo,
              head: `${context.repo.owner}:${headBranch}`,
              state: 'open'
            });
            
            if (prs.length === 0) {
              core.setFailed(`No open PR found for branch ${headBranch}`);
              return;
            }
            
            const pr = prs[0];
            core.setOutput('pr-number', pr.number);
            core.setOutput('pr-head-sha', pr.head.sha);
            core.setOutput('pr-head-ref', pr.head.ref);
            
            console.log(`Found PR #${pr.number} for branch ${headBranch}`);

      - name: Checkout PR code
        uses: actions/checkout@v4
        with:
          # Use PR ref for workflow_run, default for pull_request
          ref: ${{ github.event_name == 'workflow_run' && steps.pr-info.outputs.pr-head-ref || github.head_ref }}

      - name: Run tests
        run: |
          echo "Running tests..."
          # Your test commands here
          npm test  # or pytest, go test, etc.

      - name: Report status to PR
        if: github.event_name == 'workflow_run'
        uses: actions/github-script@v7
        with:
          script: |
            const prNumber = '${{ steps.pr-info.outputs.pr-number }}';
            const sha = '${{ steps.pr-info.outputs.pr-head-sha }}';
            
            // Create a check run or comment on the PR
            await github.rest.repos.createCommitStatus({
              owner: context.repo.owner,
              repo: context.repo.repo,
              sha: sha,
              state: 'success',  // or 'failure' based on test results
              context: 'PR Tests (via workflow_run)',
              description: 'Tests passed'
            });

Key Points

1. Dual triggers: Workflow B responds to both pull_request (manual PRs) and workflow_run (auto PRs)

2. Conditional logic: The if condition ensures we only run on successful Workflow A completions

3. PR discovery: Since workflow_run doesn’t have PR context, we query the API to find the PR by branch name

4. Status reporting: We manually create commit statuses so the PR shows test results

Caveats

Check Runs and PR Status

When triggered via workflow_run, the workflow runs in the context of the default branch, not the PR. This means:

• PR checks list won’t automatically show the workflow
• You need to manually create commit statuses (shown above)
• Or use the Checks API to create check runs

Race Conditions

If Workflow A completes before the PR is fully created (rare but possible), the PR lookup might fail. Add a short delay or retry logic if needed:

- name: Wait for PR to be indexed
  run: sleep 5

Branch Name Matching

The PR lookup uses the head branch. Ensure Workflow A’s branch naming is predictable so Workflow B can find the right PR.

Verification

After implementing:

1. Push a branch matching your pattern (e.g., feature/test-chain)
2. Confirm Workflow A runs and creates the PR
3. Confirm Workflow B triggers on Workflow A completion
4. Confirm tests run and status appears on the PR

Summary

The workflow_run trigger lets you chain workflows without PATs or GitHub Apps. It’s slightly more complex than the token-based approaches, but works entirely within GitHub’s default permission model — making it ideal for enterprise environments where token creation is restricted.

Fixing Podman Start issues with Krunkit on MacOS

Podman on macOS can run its Linux VM through libkrun via the krunkit binary (installed from Homebrew). When required shared libraries are missing, krunkit crashes immediately and Podman reports an unhelpful abort-trap error instead of the underlying dyld failure.

This guide documents the symptoms, troubleshooting steps, fix, and prevention for that failure mode.

Symptoms

Starting a libkrun-backed Podman machine fails:

podman machine start

Example output:

Starting machine "podman-machine-default"
Error: krunkit was terminated by signal: abort trap

Related indicators:

• podman machine list shows the machine in stopped state with Never under Last Up
• podman version fails with a socket/SSH connection error because the VM never started
• The machine was created with VM type libkrun (default when krunkit is installed)

podman machine list
# NAME                     VM TYPE     ...
# podman-machine-default*  libkrun     ...  Never

Root Cause

krunkit depends on a chain of Homebrew libraries (libkrun-efi, virglrenderer, and others). If any link in that chain is missing, macOS dynamic linker (dyld) aborts the process before Podman can surface a clear error.

In this case, two libraries were not installed:

Podman only reported the generic abort trap because krunkit exited before the VM could boot.

Troubleshooting

Work through these steps to isolate the failure.

1. Confirm machine state and provider

podman machine list
podman machine inspect podman-machine-default

Look for VM TYPE: libkrun and State: stopped.

2. Run krunkit directly

This usually reveals the real dyld error:

which krunkit
krunkit --version

If krunkit crashes with Library not loaded, note the missing .dylib path.

3. Check Homebrew dependency tree

brew deps krunkit
brew deps --installed krunkit libkrun-efi

Compare installed deps against what krunkit --version reports as missing.

4. Inspect linked libraries (optional)

otool -L "$(which krunkit)"

Follow the chain into /opt/homebrew/Cellar/ if multiple libraries are missing.

Resolution

Install the missing libraries reported by krunkit --version, then start the machine.

For the missing libepoxy and molten-vk libraries:

brew install libepoxy molten-vk

Verify krunkit runs:

krunkit --version
# krunkit 1.1.1

Start the Podman machine:

podman machine start

If dependencies are still broken or versions are mismatched, reinstall the krunkit stack so Homebrew relinks everything:

brew reinstall krunkit libkrun-efi
podman machine start

Verification

Confirm client–server connectivity:

podman version

Run a test container:

podman run --rm hello-world

Expected: image pulls successfully and prints the Podman hello-world banner.

Prevention

• Install krunkit from the maintained tap so Homebrew tracks dependencies:

  brew tap slp/krun
  brew install krunkit

• Avoid orphaning dependencies — if you run brew cleanup or manually remove packages, check that krunkit --version still works before starting Podman.

• After major macOS or Homebrew upgrades, verify the stack:

  krunkit --version && podman machine start

• Rootless vs rootful — the default machine runs rootless. If you need ports below 1024 or hit Docker-client compatibility issues:

  podman machine set --rootful

Environment (reference)

This issue was diagnosed on:

References

• containers/krunkit — libkrun VM launcher for macOS; Homebrew install instructions
• krunkit usage docs — VM configuration and device options

HCL BigFix Agent Architecture Guide

Overview

HCL BigFix is a multi-platform endpoint management platform that enables IT operations and security teams to discover, manage, patch, and secure global IT estates. Whether operating on-premises, in the cloud, or in hybrid environments, BigFix uses a unique single-agent, hierarchical architecture to manage thousands of devices efficiently with minimal bandwidth overhead.

This guide is designed for technical consultants planning a new BigFix deployment with a client. It outlines the core architectural components, infrastructure decisions, prerequisite checklists, deployment flows, and discovery questions necessary to align with a client’s requirements.

BigFix Core Architecture & Components

BigFix does not rely on active scanning or heavy database polling. Instead, it operates on a pull-based, “Relevance” evaluation model.

                                +-------------------+
                                |  Windows Console  |
                                +---------+---------+
                                          |
                                          | (HTTPS - TCP 52311)
                                          v
+-------------------+           +-------------------+           +-------------------+
|    Web Reports    |<----------|    Root Server    |---------->|  HCL License /    |
+-------------------+  (TCP)    +---------+---------+  (HTTPS)  |  Content Servers  |
                                          |                     +-------------------+
                                          | (TCP/UDP 52311)
                                          v
                                +-------------------+
                                |    Top-Level      |
                                |      Relay        |
                                +---------+---------+
                                          |
                   +----------------------+----------------------+
                   | (TCP/UDP 52311)                             | (TCP/UDP 52311)
                   v                                             v
         +-------------------+                         +-------------------+
         |   Branch Relay    |                         |   Branch Relay    |
         |    (Location A)   |                         |    (Location B)   |
         +---------+---------+                         +---------+---------+
                   |                                             |
         +---------+---------+                         +---------+---------+
         |                   |                         |                   |
         v                   v                         v                   v
   +-----------+       +-----------+             +-----------+       +-----------+
   |  Client   |       |  Client   |             |  Client   |       |  Client   |
   +-----------+       +-----------+             +-----------+       +-----------+

1. The Root Server

The central brain of the deployment. It:

• Collects and aggregates reports from agents.
• Downloads patch feeds and fixlets from the HCL Content Servers.
• Manages and distributes action payloads.
• Runs the core database services.

2. BigFix Relays

Relays are intermediate staging points. Instead of 10,000 agents downloading a 100MB patch over a WAN link from the Root Server, the local Relay downloads it once, and the local agents fetch it from that Relay over the local LAN.

• Relays are standard endpoints running a lightweight relay service.
• Hierarchical layering: Relays can report to other parent relays (Top-Level Relays), shielding the Root Server from direct load.

3. BigFix Client (Agent)

A lightweight agent (~10-20MB RAM, <2% CPU) running as a system service on every managed machine.

• It continuously evaluates the endpoint state using Relevance Language (a declarative query language).
• If an agent determines a patch or configuration is “relevant”, it flags itself in the Console.
• It executes Action Scripts when instructed by the operator.

4. BigFix Console

The primary administrative interface used by operators to view inventory, author actions, and deploy patches.

• Constraint: The Console is strictly a Windows-only desktop application.
• It connects directly to the Root Server via HTTP/HTTPS.

5. Web Reports

A web-based interface for secondary reporting, viewing historical trends, and sharing read-only compliance dashboards with executive stakeholders.

Infrastructure Sizing & Database Choices

Selecting the right OS and Database backend is the most critical decision during the initial phase.

Supported Operating Systems for Root Server

• Windows Server: 2016, 2019, 2022 (Standard and Datacenter editions).
• Red Hat Enterprise Linux (RHEL): 8.x, 9.x (64-bit).

Database Backends & Constraints

• Microsoft SQL Server (for Windows Server deployments):
  • SQL Server Enterprise & Standard: Supported for versions 2014 through 2022.
  • SQL Server Express: Supported only for proof-of-concept (POC) environments or environments with fewer than 500 endpoints due to the 10GB database size limit.
• IBM DB2 (for Linux deployments):
  • DB2 v11.5 is the standard for Linux deployments.
  • An evaluation/bundled version of DB2 is typically included in the Linux server installer.

Hardware Sizing Guidelines

Note: Sizing depends on endpoint counts and active modules (e.g., Lifecycle, Compliance, Inventory).

Prerequisites

1. Network & Firewall Port Requirements

The BigFix platform uses TCP/UDP port 52311 as its default communication channel.

Note on UDP 52311: If UDP is blocked across subnets, agents will still check in via TCP on a polling interval (defaulting to every 4 hours), but real-time “gather now” command execution will be delayed.

2. DNS & Name Resolution

• The Root Server must be reachable via a fully qualified domain name (FQDN) or a dedicated IP.
• It is highly recommended to use a CNAME alias (e.g., bigfix.clientdomain.com) during installation rather than a specific server host name. This allows easy migration of the hardware/VM in the future without redeploying agents.

Configuration & Installation Sequence

The general deployment path follows these distinct phases:

Phase 1: License Authorization & Masthead Creation

1. Run the BigFix Installation Generator on a Windows staging machine.
2. Submit your HCL license authorization file (.BesLicenseRequest) to the HCL License server.
3. HCL returns a Signed License file.
4. Generate the Masthead file (actionsite.afxm). This file acts as the anchor of trust. It contains the server FQDN, configuration parameters, and the public key used by agents to verify that actions sent to them are authentically signed.

Phase 2: Installing the Root Server

• On Windows: Run setup.exe and choose “Install Server”. Configure your connection to MS SQL Server and select your generated masthead.
• On Linux: Extract the tarball, ensure database (DB2/external) is ready, and run:

  ./install.sh

  Select Option [1] All Components or [2] Server and Client Only when prompted.

Phase 3: Deploying the Console

1. Run the console installer on a Windows administrator workstation.
2. Connect to the Root Server using the operator account created during server installation.

Phase 4: Deploying Relays

1. Install the standard BigFix Agent on dedicated relay VMs or branch office hosts.
2. In the BigFix Console, right-click the designated computers and select “Install BigFix Relay”.
3. Configure the relay settings (e.g., maximum cache size, parent relays).

Phase 5: Agent Rollout

• Use the Client Deploy Tool (CDT) built into the Console to push agents over SSH or WMI.
• For large scale deployments, deploy the agent MSI/RPM package via GPO, SCCM, or Ansible, bundled with the generated actionsite.afxm masthead.

Consultant’s Discovery Checklist

Use these specific discovery questions during your initial meeting with the client to gather architectural requirements:

1. How many total endpoints (workstations, servers, VMs, containers) are in scope?
   • Why: Determines whether SQL Server Express is acceptable (< 500) and sets hardware specifications.
2. What is the current distribution of Operating Systems across these endpoints?
   • Why: Identifies if they need specific OS agents and helps plan the deployment method.
3. What is the network topology? Do you have remote branch offices or bandwidth-constrained links?
   • Why: Drives the relay placement strategy and WAN bandwidth throttle planning.
4. Do you have a preferred Operating System for the central BigFix server (Windows Server vs RHEL)?
   • Why: Dictates database licensing requirements (MS SQL Server vs DB2) and administrative skillsets.
5. How are endpoints currently patched or managed? Do we need to plan co-existence or a migration from tools like WSUS, SCCM, or Ansible?
   • Why: Highlights potential software conflicts or open firewall port struggles.
6. Will endpoints reporting from outside the corporate firewall (remote/WFH workers) need management?
   • Why: Determines if we need to deploy an Internet-Facing Relay (DMZ Relay) with public DNS records.

Common Pitfalls & Anti-Patterns

• Hardcoding Server Names: Installing with a specific server name (e.g., srv-bf-prod-01) instead of a DNS CNAME alias (bigfix.company.com). If the server hardware dies, migrating the system requires manually updating the masthead on every single agent.
• Neglecting Relay Cache Limits: Leaving relay cache limits at default values. In environments deploying heavy OS patches or major software rollouts, relays will run out of cache space, causing them to fetch payloads repeatedly across WAN links.
• Ignoring UDP 52311: Assuming TCP is sufficient. Without UDP 52311 inbound to clients, operators cannot trigger “Instant” actions, severely impacting incident response speeds.
• Insufficient Space in Temporary Directories: Installing without verifying disk space in /var/tmp (Linux) or C:\Windows\Temp (Windows). BigFix client downloads are staged here, and major patch actions will fail if these directories are constrained.

References

• HCL BigFix Enterprise Support Portal - Central knowledge base and product updates.
• BigFix Developer Reference - Authoring guides, Relevance query guides, and REST API references.

Installing Podman on MacOS

Podman Desktop is the graphical front end for Podman on macOS. Containers do not run natively on macOS — Podman launches a lightweight Linux VM (a Podman machine) and runs containers inside it. The desktop app handles onboarding, machine management, and optional Docker compatibility so existing Docker workflows keep working.

This guide covers installation, first-run setup, Docker compatibility, rootless vs rootful mode, and common gotchas — including the krunkit abort trap failure on Homebrew installs.

Prerequisites

Optional but useful:

• Homebrew — alternative install path (see gotchas below)
• Docker CLI — not required; Podman can serve the Docker API. Install via onboarding or brew install docker if you want the docker command

Choose an Install Path (Important)

Podman Desktop and the Podman CLI can be installed two ways. Pick one and stay on it.

Do not mix install methods

If Podman is already installed via Homebrew, either:

• Uninstall Homebrew Podman before using the .dmg installer, or
• Use Homebrew only — do not also install the .dmg

Mixing paths causes duplicate binaries, conflicting machine configs, and hard-to-debug startup failures.

# Check what you already have
which podman
podman --version
brew list --cask podman-desktop 2>/dev/null
ls /Applications/Podman\ Desktop.app 2>/dev/null

Installation

Option A: DMG installer (recommended)

1. Download the Universal .dmg from Podman Desktop macOS downloads.
2. Open the .dmg and drag Podman Desktop to Applications.
3. Launch from Applications or Spotlight (Cmd+Space → “Podman Desktop”).
4. Complete onboarding (see First-run setup below).

The installer bundles krunkit and avoids the Homebrew dependency-chain issues described in Podman Machine krunkit Abort Trap.

Option B: Homebrew

brew install --cask podman-desktop

Homebrew installs Podman Desktop and the Podman engine if not already present. Open the app from /Applications.

If you use libkrun (krunkit is on your PATH), install krunkit from the maintained tap so dependencies are tracked:

brew tap slp/krun
brew install krunkit

After any Homebrew install, verify krunkit before creating a machine:

krunkit --version

If this crashes, see the krunkit abort trap guide before proceeding.

First-run setup

Podman Desktop onboarding walks through engine install, machine creation, and optional CLIs. You can also finish setup later from Settings → Resources or the Dashboard notification.

Onboarding checklist

1. Install Podman engine — enter your macOS password when prompted.
2. Create a Podman machine — default name is podman-machine-default.
3. Install optional CLIs — kubectl and compose (Podman Compose).
4. Open Dashboard — confirm the machine shows Running under Settings → Resources.

CLI equivalent

If you prefer the terminal (or need to script setup):

podman machine init
podman machine start
podman info
podman run --rm hello-world

Customize resources before or after init:

podman machine set --cpus 6 --memory 4096 --disk-size 100
podman machine start

Machine providers

On Apple Silicon with krunkit installed, new machines default to libkrun (GPU-capable, uses Apple’s Hypervisor framework). Without krunkit, Podman falls back to applehv.

podman machine list
# VM TYPE column: libkrun or applehv

Rootless vs rootful

Every Podman machine runs containers in one of two modes. Rootless is the default.

When to stay rootless

• General development and learning
• Running apps on high ports (3000, 8080, etc.)
• You want the safer default

When to switch to rootful

• Binding ports below 1024
• Tools that assume root-owned containers or the root Podman socket
• Permission errors editing registries or storage inside the VM

How to switch

Podman Desktop: Settings → Resources → select your machine → toggle rootful mode, then restart the machine.

CLI:

podman machine stop
podman machine set --rootful
podman machine start

To revert:

podman machine stop
podman machine set --rootful=false
podman machine start

Podman maintains separate connection endpoints for each mode:

podman system connection list
# podman-machine-default       → rootless socket
# podman-machine-default-root  → rootful socket

Switch the active connection if needed:

podman system connection default podman-machine-default-root

Docker compatibility

Podman exposes a Docker-compatible API so tools and scripts written for Docker can talk to Podman instead. On macOS, Podman Desktop enables Third-Party Docker Tool Compatibility by default.

What gets mapped

When the machine is running, macOS exposes a system socket symlink:

ls -la /var/run/docker.sock
# → ~/.local/share/containers/podman/machine/podman.sock

Podman Desktop reports socket status under Settings → Docker Compatibility.

Using the Docker CLI with Podman

1. Ensure the Podman machine is running.
2. Enable Docker compatibility in Podman Desktop (on by default on macOS).
3. Install the Docker CLI if you do not have it (onboarding or brew install docker).
4. Verify:

docker info --format '{{.ServerVersion}}'
# Should report a Podman version, not Docker Desktop

DOCKER_HOST (explicit configuration)

Some tools (Gradle, Testcontainers, CI scripts) need an explicit socket path instead of relying on /var/run/docker.sock:

export DOCKER_HOST="unix://${HOME}/.local/share/containers/podman/machine/podman.sock"

Or create a Docker context:

docker context create podman \
  --docker "host=unix://${HOME}/.local/share/containers/podman/machine/podman.sock"
docker context use podman

Compose

• Podman Compose — installed during onboarding; invoke as podman compose (preferred on Podman).
• Docker Compose — works against the Docker-compatible socket when compatibility is enabled.

podman compose version
podman compose up -d

Compose project naming and networking differ slightly from Docker Compose in edge cases; prefer podman compose for Podman-native behavior.

Coexistence with Docker Desktop

Do not run Docker Desktop and Podman Desktop socket forwarding at the same time — both compete for /var/run/docker.sock. Quit Docker Desktop (or disable its socket) before enabling Podman Docker compatibility.

Gotchas

1. krunkit abort trap (Homebrew + libkrun)

Symptom: podman machine start fails with krunkit was terminated by signal: abort trap.

Cause: Missing Homebrew libraries in the krunkit dependency chain (libepoxy, molten-vk, etc.).

Fix: See Podman Machine krunkit Abort Trap.

Quick check:

krunkit --version   # should print a version, not crash

2. Mixed Homebrew and DMG installs

Installing Podman via Homebrew and Podman Desktop via .dmg (or vice versa) leaves duplicate podman/krunkit binaries and conflicting machine state. Uninstall one path completely before adopting the other.

3. krunkit not found

Podman Desktop onboarding or podman machine init may report that krunkit is missing when using a partial Homebrew install.

brew tap slp/krun
brew install krunkit

The official .dmg bundles krunkit and avoids this.

4. Rootless privileged ports

Binding host port 80 in rootless mode fails silently or with permission errors. Switch to rootful or use a high port (8080) with a reverse proxy.

5. ARM64 image platform

On Apple Silicon, prefer multi-arch images or specify the platform:

podman run --platform linux/arm64 ...

Running x86-only images requires emulation and is slower.

6. Docker CLI still talks to Docker Desktop

If docker info shows Docker Desktop after switching to Podman:

• Quit Docker Desktop
• Confirm Podman machine is running
• Check Settings → Docker Compatibility in Podman Desktop
• Run docker context ls and switch away from desktop-linux

7. Machine resource defaults

The default 2 GiB RAM is tight for multiple containers or image builds. Increase before heavy use:

podman machine set --memory 8192 --cpus 4
podman machine start

Changes apply on next start.

8. Registries and VM config edits

Files like /etc/containers/registries.conf live inside the VM, not on macOS. Edit from the host:

podman machine ssh
# or, if permission denied in rootless mode:
podman machine set --rootful && podman machine start

Verification

Run this checklist after setup:

# Machine running
podman machine list

# Client ↔ VM connectivity
podman version

# Basic container
podman run --rm hello-world

# Docker API socket (when compatibility enabled)
ls -la /var/run/docker.sock

# Optional: Docker CLI against Podman
docker info --format '{{.ServerVersion}}'

In Podman Desktop: Settings → Resources shows the machine as Running; Settings → Docker Compatibility shows the socket as reachable.

Quick reference

References

• Podman Desktop macOS install — official install and onboarding
• Managing Docker compatibility — socket mapping and CLI setup
• Podman installation (macOS) — CLI and machine overview
• containers/krunkit — libkrun VM backend for Apple Silicon

NVIDIA Container Toolkit & CDI Setup on Proxmox GPU Passthrough VMs

Overview

NVIDIA Container Toolkit with CDI (Container Device Interface) enables Docker containers to access physical GPUs. On Proxmox VMs with GPU PCI passthrough, CDI specs must be explicitly generated and Docker must be configured to use them.

Prerequisites

• Proxmox VM with NVIDIA GPU PCI passthrough configured (IOMMU, vfio-pci)
• NVIDIA driver installed and nvidia-smi working on the host
• Docker installed

Step-by-Step

1. Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

2. Generate CDI Specs

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Verify the spec was created:

ls -la /etc/cdi/nvidia.yaml

3. (If Needed) Create AMD CDI Stub

On Proxmox GPU passthrough VMs, Docker’s CDI resolution may also require an amd.yaml spec. If NemoClaw or other GPU-passthrough tools fail with “AMD CDI spec not found”, create a minimal stub:

sudo tee /etc/cdi/amd.yaml > /dev/null <<'EOF'
---
cdiVersion: "0.5.0"
kind: "amd.com/gpu"
devices:
  - name: "stub"
    containerEdits:
      env:
        - ROCR_VISIBLE_DEVICES = ""
EOF

4. Configure Docker for CDI

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

5. Verify GPU Access

docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

You should see the GPU listed with driver version and CUDA capabilities.

Common Pitfalls

References

• NVIDIA Container Toolkit Installation Guide
• NVIDIA CDI Support
• OpenShell GPU Passthrough

NVIDIA vGPU Manager: build, integrated registry, and ClusterPolicy (OpenShift Virtualization)

This note validates a common on-cluster workflow against NVIDIA’s published steps, then records that workflow when Quay (or another external registry) is not used and images land in the integrated OpenShift image registry instead.

Primary NVIDIA reference: NVIDIA GPU Operator with OpenShift Virtualization — Building the vGPU Manager image and Creating a ClusterPolicy.

Related workspace guide: Using the integrated OpenShift container image registry.

End-to-end workflow (integrated registry)

1. Download the vGPU Manager .run

• NVIDIA Licensing Portal → Software Downloads → Driver downloads → Linux KVM complete vGPU package for the Product Version you intend to support.
• Unzip and locate NVIDIA-Linux-x86_64-*-vgpu-kvm.run (or the AIE variant renamed per NVIDIA).

2. Clone NVIDIA’s build context and copy the installer

git clone https://gitlab.com/nvidia/container-images/driver
cd driver

Change into the vgpu-manager subdirectory for your OS line (NVIDIA’s OpenShift example historically used vgpu-manager/rhel8; confirm the right path for your OCP / RHCOS generation):

cd vgpu-manager/rhel8    # example only — pick the path NVIDIA documents for your release
cp /path/to/extracted/NVIDIA-Linux-x86_64-*-vgpu-kvm.run ./

3. Point the build at the integrated registry and RHCOS

Set VERSION to the vGPU software / driver version from the portal (NVIDIA’s examples look like 510.73.06; use your shipped version string).

Set OS_TAG to rhcos4.<minor> matching your OpenShift workers (NVIDIA documents this pattern).

Set PRIVATE_REGISTRY to the registry host and namespace prefix you will push to. For the integrated registry, that is commonly the external route hostname (for pushes from your workstation) plus project path, for example:

default-route-openshift-image-registry.apps.<cluster-base>/<project>

…where <project> is often nvidia-gpu-operator or a dedicated images project. See using-integrated-openshift-registry.md for login and push mechanics.

Example (illustrative only — adjust host, project, and versions):

export VERSION=510.73.06
export OS_TAG=rhcos4.15
export PRIVATE_REGISTRY=default-route-openshift-image-registry.apps.cluster.example.com/nvidia-gpu-operator

Build and push using podman or docker as in NVIDIA’s doc:

podman build --build-arg DRIVER_VERSION="${VERSION}" \
  -t "${PRIVATE_REGISTRY}/vgpu-manager:${VERSION}-${OS_TAG}" .
podman push "${PRIVATE_REGISTRY}/vgpu-manager:${VERSION}-${OS_TAG}"

Use a meaningful tag (here ${VERSION}-${OS_TAG}) so ClusterPolicy and support tickets stay unambiguous.

4. Reference the image from in-cluster pulls

Operands pull using the in-cluster registry DNS (typically image-registry.openshift-image-registry.svc:5000/<project>/vgpu-manager:<tag>). Your ClusterPolicy vgpuManager.repository / image / version (or equivalent fields) must match how your installed GPU Operator version assembles the pull spec. Always confirm with:

oc explain clusterpolicy.spec.vgpuManager --recursive

Conceptually you are wiring repository + image + version so they resolve to the same image you pushed (often image: vgpu-manager, version: tag string like 510.73.06-rhcos4.15 if that is what you pushed).

5. Update ClusterPolicy (operator already installed)

Generate or export the example ClusterPolicy, then merge at least:

• spec.sandboxWorkloads.enabled: true
• spec.sandboxDevicePlugin.enabled: true
• spec.vgpuManager.enabled: true with repository, image, version, and imagePullSecrets if your registry requires them
• spec.vgpuDeviceManager.enabled: true

NVIDIA’s JSON-oriented example is in Creating a ClusterPolicy for the GPU Operator using the CLI. If a ClusterPolicy already exists, prefer oc get clusterpolicy -o yaml, edit, and oc apply.

6. Verify

oc get clusterpolicy -o yaml | less
oc get pods -n nvidia-gpu-operator
oc get nodes -l nvidia.com/gpu.workload.config=vm-vgpu

When the vGPU Manager and sandbox operands are healthy and nodes are labeled, continue with NVIDIA’s HyperConverged mediated device steps and VM attachment.

Common pitfalls

• Wrong vgpu-manager subdirectory or OS_TAG for your RHCOS line → build succeeds but driver mismatches the node OS.
• ClusterPolicy fields do not match the pushed tag → ImagePullBackOff on daemonsets or operator-managed pods.
• Image in project A, operands in nvidia-gpu-operator without system:image-puller (or a pull secret) → pull forbidden across namespaces.
• Enabling vgpuManager before the image exists → transient failures; enable or point after push.

References

• NVIDIA GPU Operator with OpenShift Virtualization
• NVIDIA driver container repository
• Red Hat: Accessing the registry

Using the integrated OpenShift container image registry

OpenShift ships with a cluster-internal container registry (the Image Registry Operator, namespace openshift-image-registry). You can push images there instead of an external registry (for example Quay.io), as long as workloads that need the image can pull from the namespace where the image lives.

This is the same pattern NVIDIA documents for hosting built images (for example vGPU Manager) on the cluster’s own registry.

What you get

• In-cluster pull hostname (from any pod on the cluster):

  image-registry.openshift-image-registry.svc:5000

• Full image reference:

  image-registry.openshift-image-registry.svc:5000/<project>/<image>:<tag>

  Example: image-registry.openshift-image-registry.svc:5000/nvidia-gpu-operator/vgpu-manager:9.0.0.0

• Private by default: images are not public on the internet; access is via Kubernetes RBAC and image pull credentials where needed.

Prerequisites

1. Registry is running — ManagementState should be Managed:

   oc get configs.imageregistry/cluster -o jsonpath='{.spec.managementState}{"\n"}'

2. Storage — the registry needs a PVC or other supported backend (install-time or day-2 configuration). If the registry is not ready, pushes fail until storage is fixed.

3. Pushing from your laptop — you need a route (or port-forward) and authentication. Check for the default route:

   oc get route -n openshift-image-registry

   If there is no route, an admin can enable the default route or you can push from a Job or Build inside the cluster that uses the internal service DNS and a service account token.

Basic workflow: push an image to the integrated registry

1. Create or pick a project (namespace) to hold the image, for example nvidia-gpu-operator or a dedicated gpu-images project.

2. Log in to the registry from your workstation (using the external route host Red Hat documents for your version), with an OpenShift user token:

   REGISTRY=$(oc get route default-route -n openshift-image-registry -o jsonpath='{.spec.host}')
   podman login -u "$(oc whoami)" -p "$(oc whoami -t)" "$REGISTRY"

   Use docker instead of podman if that is what you use.

3. Tag and push using the external host for the push client (OpenShift documents the exact hostname pattern; it is often default-route-openshift-image-registry.apps.<cluster-domain>).

   podman tag mylocal/vgpu-manager:9.0.0.0 "$REGISTRY/nvidia-gpu-operator/vgpu-manager:9.0.0.0"
   podman push "$REGISTRY/nvidia-gpu-operator/vgpu-manager:9.0.0.0"

4. Reference the image from manifests using the in-cluster form (*.svc:5000/...) so nodes and pods resolve it without depending on the apps route from inside the cluster.

Use explicit tags (for example 9.0.0.0 or a driver version string), not only latest, so rollbacks and support questions stay clear.

Pulling from another namespace (for example GPU Operator operands)

If the image lives in project gpu-images but pods run in nvidia-gpu-operator, those pods need permission to pull. Typical approaches:

• Grant the nvidia-gpu-operator default service account (or the operand SA) access to pull from the other namespace (for example oc policy add-role-to-user system:image-puller system:serviceaccount:nvidia-gpu-operator:default -n gpu-images), or
• Copy/mirror the image into nvidia-gpu-operator and reference it there, or
• Use an image pull secret in nvidia-gpu-operator that can authenticate to the registry (often unnecessary for same-cluster internal pulls when RBAC is set correctly).

Exact policy depends on your org’s security model; the important part is: internal registry does not remove the need for RBAC across namespaces.

Wiring operators (example: NVIDIA ClusterPolicy / vgpuManager)

Point repository, image, and version (and imagePullSecrets if you use them) at the integrated registry path the same way you would for Quay. The operator concatenates fields according to its CRD; confirm with:

oc explain clusterpolicy.spec.vgpuManager --recursive

Conceptually you want the operands to pull something equivalent to:

image-registry.openshift-image-registry.svc:5000/<project>/<image>:<tag>

Verification

oc get configs.imageregistry/cluster -o yaml | less
oc get route -n openshift-image-registry
oc get imagestream -n nvidia-gpu-operator   # if you use ImageStreams; direct push without IS is also common

Further reading

• Red Hat: Accessing the registry (push/pull, routes, and authentication for your OCP version).

See also: NVIDIA vGPU Manager: build, integrated registry, and ClusterPolicy — end-to-end flow when hosting the vGPU Manager image on the integrated registry.

VSCode Setup for Enterprise Development

This guide covers setting up Visual Studio Code (VSCode) as your primary development environment on enterprise devices for working with OpenShift, Ansible, Python, YAML, and Bash.

Prerequisites

Before configuring VSCode, ensure the following core tools are installed on your system. While VSCode is assumed to be installed, these underlying dependencies are required for functionality.

1. Git (git-scm)

Download and install Git from the official git-scm.com site.

• Windows: Use the Git for Windows installer. Ensure “Git from the command line and also from 3rd-party software” is selected during installation.
• macOS: Install via Homebrew (brew install git) or by installing Xcode Command Line Tools (xcode-select --install).

2. Python 3

Install Python 3.x from python.org or your system’s package manager.

• Ensure pip is included in the installation.
• Add Python to your system PATH.

3. Ansible Lint

Once Python is installed, install ansible-lint to enable playbook validation in VSCode:

pip install ansible-lint

Authentication: Personal Access Tokens (PAT)

In an enterprise environment, you will typically use a Personal Access Token (PAT) instead of your account password for Git operations over HTTPS.

Generating a PAT (GitHub Example)

1. Log in to your GitHub account.
2. Go to Settings > Developer settings > Personal access tokens > Fine-grained tokens (or Tokens classic).
3. Click Generate new token.
4. Give it a descriptive name (e.g., “VSCode Enterprise Desktop”).
5. Select the required scopes (typically repo for full control of private repositories).
6. Copy the token immediately. You will not be able to see it again.

Using the PAT

When you clone a repository or push changes for the first time, Git will prompt for your credentials.

• Username: Your GitHub username.
• Password: Paste your PAT here.

Note: On Windows and macOS, the system credential manager will securely store this token for future use.

Recommended Extensions

To get the best experience, install the following extensions from the VSCode Marketplace.

Core Languages & Tools

Version Control & Collaboration

Utilities

• Error Lens: Highlights errors and warnings inline in your code.
• Path Intellisense: Autocompletes filenames.
• Dotenv: Support for .env file syntax highlighting.

Installing Extensions

To install any of the recommended extensions:

1. Click the Extensions icon in the Activity Bar on the side of VSCode (or press Ctrl + Shift + X).
2. Type the name of the extension in the search bar.
3. Click Install.

Recommended Settings

Add these to your user settings.json (Ctrl + Shift + P > “Open User Settings (JSON)”) for a smoother experience:

{
  "editor.formatOnSave": true,
  "editor.renderWhitespace": "all",
  "files.insertFinalNewline": true,
  "files.trimTrailingWhitespace": true,
  "git.autofetch": true,
  "git.confirmSync": false,
  "workbench.editor.restoreViewState": true,
  "yaml.format.enable": true,
  "ansible.validation.lint.enabled": true
}

Setting Up Your Workspace

1. Pulling Repositories

Use the integrated terminal (Ctrl + \``) or the Command Palette (Ctrl + Shift + P`) to clone your repositories.

git clone https://github.com/your-org/your-repo.git

2. Creating a Workspace

If you work across multiple repositories (e.g., one for Ansible playbooks, one for Python tools), use Multi-root Workspaces:

1. Open your first repository folder.
2. Go to File > Add Folder to Workspace…
3. Select your second repository folder.
4. Save the workspace: File > Save Workspace As…

3. Version Control Workflow

Basic Git Usage

• Stage Changes: Click the + icon next to files in the Source Control tab.
• Commit: Enter a message in the text box and click Commit.
• Push/Pull: Use the sync icon in the status bar or the ... menu in Source Control.

Creating Pull Requests (PRs)

With the GitHub Pull Requests and Issues extension:

1. Push your branch to the remote repository.
2. Click the GitHub icon in the Activity Bar.
3. Click the + (Create Pull Request) icon.
4. Follow the prompts to select base/head branches and enter a title/description.

Plugin Configuration Tips

Ansible & YAML

To ensure the YAML extension recognizes OpenShift/Kubernetes schemas:

1. Open Settings (Ctrl + ,).
2. Search for yaml.schemas.
3. Add entries for your Kubernetes/OpenShift manifest patterns.

Python

Ensure you select the correct Python interpreter:

1. Press Ctrl + Shift + P.
2. Type Python: Select Interpreter.
3. Choose the version or virtual environment (.venv) relevant to your project.

Related Ideas & Next Steps

• Dev Containers: Use the Remote - Containers extension to define your development environment in a Dockerfile or docker-compose.yaml within the repo.
• Snippets: Create custom snippets for common Ansible tasks or OpenShift manifest headers.
• Tasks: Define VSCode Tasks (tasks.json) to run your playbooks or tests with a single shortcut.