Skip to main content

Collect a troubleshooting bundle

Collect a troubleshooting bundle from an Infrahub deployment and share it with OpsMill support. Collection is read-only and safe to run at any time, including while the instance is degraded. See Infrahub Collect for what a bundle contains and how secrets are masked. The same command produces the same bundle on Docker Compose and Kubernetes.

Prerequisites​

Before collecting a bundle:

  • The infrahub-collect binary is installed (installation guide)
  • Docker and Docker Compose are available to your user
  • The Infrahub Compose project is running (fully or partially)

The binary is self-contained and collection works offline by default: it uses your existing Docker or kubectl access and performs no network access beyond your deployment itself. The one exception is the opt-in --benchmark, which downloads a benchmark image.

Step 1: Verify environment detection​

Confirm the tool can find your Infrahub deployment:

infrahub-collect environment detect

The tool auto-detects Docker Compose projects and Kubernetes namespaces containing Infrahub. To list every deployment it detects:

infrahub-collect environment list

Step 2: Create the bundle​

For a single Infrahub project, run:

infrahub-collect create

If several Infrahub projects run on the same host, target one explicitly:

infrahub-collect create --project=infrahub-production

By default the bundle is written to ./infrahub_bundles. Use --output-dir to choose another location:

infrahub-collect create --output-dir=/tmp/support

Step 3: Monitor collection progress​

The command streams progress as each collector runs:

INFO[0000] Detecting deployment environment...
INFO[0000] Kubernetes environment detected (namespace: infrahub-prod)
INFO[0000] Creating collection ID: 20260702_141530
INFO[0001] Collecting logs: infrahub-server (2 replicas)
INFO[0004] Collecting logs: task-worker (3 replicas, 1 restarted - including previous logs)
INFO[0009] Collecting database logs...
INFO[0012] Collecting message-queue status...
INFO[0013] Collecting cache status...
INFO[0014] Collecting task-manager state...
INFO[0016] Collecting server info...
INFO[0017] Collecting container metrics...
INFO[0018] Creating compressed archive...
INFO[0021] Bundle created: infrahub_bundles/support_bundle_20260702_141530.tar.gz

If a service is down or unreachable, collection continues with the remaining services and records the failure:

WARN[0013] Collector failed: cache status (container not running) - continuing

A partial bundle is still a useful bundle — a degraded instance is precisely when you need one.

Step 4: Verify the bundle​

Check the archive and review what was collected:

# Verify archive integrity
tar -tzf infrahub_bundles/support_bundle_20260702_141530.tar.gz > /dev/null && echo "Archive is valid"

# View the collection manifest
tar -xzOf infrahub_bundles/support_bundle_20260702_141530.tar.gz bundle/bundle_information.json | jq '.'

The manifest records what was attempted and what succeeded:

{
"manifest_version": 2026110500,
"collect_id": "20260702_141530",
"created_at": "2026-07-02T14:15:30Z",
"tool_version": "1.2.0",
"infrahub_version": "1.5.2",
"environment": "kubernetes",
"edition": "enterprise",
"helm": {
"release_name": "infrahub",
"chart": "infrahub-enterprise",
"chart_version": "1.2.3"
},
"log_lines": 100000,
"collectors": [
{"name": "logs/infrahub-server", "status": "success"},
{"name": "logs/task-worker", "status": "success"},
{"name": "cache-status", "status": "failed", "reason": "container not running"},
{"name": "benchmark", "status": "skipped", "reason": "not requested"}
]
}

The edition field reports whether the deployment runs Infrahub Community or Enterprise. On Docker Compose it is classified from the infrahub-server container image, and on Kubernetes from the Helm chart name. It is omitted when the edition cannot be determined.

The bundle layout is identical on Docker Compose and Kubernetes. Inside the archive, all files live under a top-level bundle/ directory:

  • bundle_information.json - Collection manifest
  • logs/<service>/ - Container logs, one file per replica (plus *.previous.log for restarted pods)
  • database/ - Neo4j server logs
  • message-queue/ - RabbitMQ queues, exchanges, connections, and status
  • cache/ - Redis info, clients, configuration, and slow log
  • task-worker/ - Task worker state, one directory per replica
  • task-manager/ - Work pools, work queues, recent flow runs, the pending and running flow and task runs, events, and automations
  • server/ - Version, installed packages, API information, configuration, schema, worker schema-sync status (infrahub_status.json), environment variables
  • metrics/ - Container resource metrics
  • traces/ - Traces of slow requests, one OTLP JSON file per trace, slowest first (only with --include-traces)

Each file holds the output of the command that produced it and nothing else, so a .json file always parses. When a command fails, its file is absent and a sibling <name>.err.txt records what ran, how it failed, and what the command reported — for example task-manager/flow-runs.err.txt in place of flow-runs.json. The manifest lists the same failure as a reason on the collector.

Review before sharing

Masking is based on key names: environment variables and configuration keys containing password, secret, token, or key are replaced with ******** before they are written to the bundle. Secrets stored under other key names are not detected. Service logs are collected as-is, and database query logs (--include-queries) can contain customer data. Review the bundle contents before sharing.

Step 5: Share the bundle with support​

Send the archive to OpsMill support through your usual support channel. The bundle is a plain local file, so environments with restricted egress can transfer it through whatever mechanism their security policy allows.

Advanced usage​

Include a backup for issue reproduction​

Support often asks for a backup alongside logs so they can reproduce your issue locally. Create both in one run:

infrahub-collect create --include-backup

This runs the standard infrahub-backup logic and records the backup in the bundle manifest.

warning

Unlike log collection, the backup step follows the normal backup behavior for your deployment, which can include stopping application containers while the snapshot is taken. Plan for a short interruption, or skip --include-backup and create the backup separately during a maintenance window.

Include database query logs​

Query logs can help diagnose slow or failing database operations, but they may contain customer data, so they are excluded by default:

infrahub-collect create --include-queries

Adjust the log volume limit​

By default, collection captures the most recent 100,000 log lines per container. Raise or lower the limit as needed:

# Collect more history for a long-running issue
infrahub-collect create --log-lines=500000

# Or via environment variable
export INFRAHUB_LOG_LINES=500000
infrahub-collect create

The effective limit is recorded in the manifest, so support knows whether logs were truncated.

Include the traces of slow requests​

When Infrahub is slow, the traces of the slow requests show which component and which database query took the time. If your deployment sends traces to Tempo, for example with the Infrahub observability stack, add them to the bundle:

infrahub-collect create --include-traces

By default, the collector reads the traces of the last 2 hours that lasted longer than 1 second. Change the window and the minimum duration as needed:

# A slowness that is still going on: the last 30 minutes
infrahub-collect create --include-traces --traces-since=30m

# A slowness that ended: an absolute window, in RFC 3339
infrahub-collect create --include-traces \
--traces-start=2026-11-04T14:00:00Z --traces-end=2026-11-04T15:00:00Z

# Only the traces longer than 5 seconds
infrahub-collect create --include-traces --traces-min-duration=5s

The collector writes one file per trace to traces/, numbered from 0001 for the slowest trace. It stops when the files reach the size limit, 100 MB by default (--traces-max-size, in megabytes of uncompressed JSON), or after 5 minutes of fetching. Because the slowest traces are written first, the traces left out are always faster than the traces kept. The traces entry of bundle_information.json records the window, the minimum duration, and how many traces were found, written and left out.

Personal data is removed. The collector keeps only these fields of each trace:

  • On every span: the trace, span and parent span IDs, the span name and kind, the start and end times, and the status code.
  • On the process that sent the span: service.name, service.version and worker.id.
  • On database query spans (execute_db_query_with_metadata): query_name and rows.
  • On GraphQL spans (execute_graphql): type, operation, name and branch.
  • On HTTP server spans: http.method, http.route and http.status_code.

Everything else is removed before the file is written, including the text of database queries, URLs, client IP addresses, user agents, exception messages and span events.

Where the collector finds Tempo. The collector sends its requests from the machine where it runs, so it works while the Infrahub containers or pods are down:

  • On Docker Compose, it reads the published port 3200 of the infrahub-tempo service of the project. With a remote Docker context, this port is on the remote host; use --traces-tempo-url instead.
  • On Kubernetes, it finds the service labelled app.kubernetes.io/name=tempo in the Infrahub namespace and forwards a local port to its port 3100 for the duration of the collection. The infrahub-observability Helm chart creates this service.

The collector works with Tempo 2.5 and later.

When no Tempo is found, when Tempo does not answer the search within 60 seconds, or when your Kubernetes role lacks a permission, the traces entry is skipped and the reason names what is missing.

Your own trace backend. To read the traces from another Tempo-compatible backend, give its URL. If the backend needs a token, set it in your shell; the collector sends it as a bearer token and never writes it to the bundle or the output:

export INFRAHUB_TRACES_TEMPO_TOKEN=<token>
infrahub-collect create --include-traces --traces-tempo-url=https://tempo.example.com

When the variable is not set and the command runs in a terminal, the collector asks for the token without showing what you type. An empty answer means no token.

The trace sampling ratio. Infrahub records a trace for only a share of the requests. To let support know that share, give an Infrahub API token of a user with the super_admin permission. The collector uses it only to read the sampling ratio (Infrahub 1.12 or later) and records the ratio in the bundle:

export INFRAHUB_TRACES_API_TOKEN=<token>
infrahub-collect create --include-traces

Without the variable, the collector asks for the token in a terminal; leave the answer empty to skip. If the slowness can be reproduced, a super admin can raise the ratio for a limited time with PUT /api/troubleshooting/trace-sample-ratio before reproducing it, and remove the temporary ratio with DELETE on the same endpoint after the bundle is created.

note

Tempo can take about a minute before a new trace appears in its search. If you create the bundle right after the slowness, wait a minute first.

Run a performance benchmark​

For performance investigations, support may ask you to include a benchmark run:

infrahub-collect create --benchmark

The benchmark requires downloading the OpsMill benchmark container image and measures whether the host meets Infrahub's resource requirements — disk IOPS, memory, and single-core CPU — rather than generating load against your instance. If the image cannot be pulled — for example in an air-gapped environment — the benchmark is skipped with a warning and the rest of the collection completes normally.

Troubleshoot collection​

No deployment detected​

If infrahub-collect create reports that no environment was found, list what the tool can see:

infrahub-collect environment list

On Docker Compose, confirm the project is running with docker compose ls; the project name must contain infrahub, or you must name it with --project. On Kubernetes, the namespace is discovered through the app.kubernetes.io/name=infrahub label; for deployments with custom labels, name the namespace with --k8s-namespace.

Permission denied on Kubernetes​

Log collection needs the pods/log permission and the status collectors need pods/exec. If your role lacks one of them, the affected collectors are recorded as failed in the manifest. Ask your cluster administrator for read access to both — collection needs no write or scale permissions.

With --include-traces, the collector finds the deployment's Tempo and forwards a port to it, which needs list and get on services, get and list on pods, and create on pods/portforward. kubectl port-forward reads the service and the pod behind it before it opens the port. Without these permissions, the traces entry is skipped and the reason names what is missing. With --traces-tempo-url, the collector reads a URL that your machine reaches and needs none of them.

Docker or kubectl not found​

The tool depends on the Docker CLI or kubectl being installed and configured on the machine where it runs. When neither is usable, the command fails immediately and the error names the missing CLI.

Individual collectors failed​

Collector failures on a degraded deployment are expected — a stopped cache container cannot answer status queries. The command still exits successfully, and the manifest records each failure with a reason. Send the partial bundle as-is; support reads the manifest first.

Validation​

Confirm your collection works:

  • infrahub-collect environment detect finds your deployment
  • Bundle archive is created and passes the integrity check
  • Manifest lists an explicit outcome for every collector
  • Logs are present for every service replica
  • Masked environment output contains no plaintext secrets