Operating day to day

Every run ends as success, failed or skipped, and every one of them is on disk. This page is about reading them.

Checking how things are going

pipelines status# latest runs of every pipelinepipelines status nightly-report -n 10pipelines show nightly-report <run-id># the full run.jsonpipelines show nightly-report <run-id> collect# a single step, with its denials if anypipelines web# local read-only dashboard at http://127.0.0.1:7717 (overview, /costs, pipelines, runs and logs)

status lists the latest runs (5 by default) with their id, status, cost and skip reason. The run record includes a notify entry whenever the channel was invoked: the channel, its exit code, the timestamp and the tail of its output. That way you can confirm the notification really went out without opening your inbox.

For a visual view of the same data, and of what your agent steps cost this month, see The dashboard.

Reading a skipped run

A skipped caused by a throttle is normal and expected: a docs-review pipeline with a 46-hour throttle on a nightly cron skips most nights by design. What you should look at is skipReason in run.json, or the reason column in status.

A pipeline can go on skipping silently for weeks because of a broken guard, and the data repo will look exactly as if it were working. It has happened to us. That's why every pipeline with triggers and notify must declare notify.stale_after (the linter enforces it, rule R4): it turns that silence into a notification.

Reading a failed run

pipelines show <name> <run-id> tells you which step failed and, in its error, the exit code plus the last lines of stderr. Your notification channel receives the same information in PIPELINES_NOTIFY_REASON, so the failure notification can carry it. If the failure was an external hiccup, resume from the failed step with the original params:

pipelines resume <name> <run-id>

A failed isn't always a fault of the engine or of the pipeline. A repo-audit pipeline, for example, might have its final verdict step exit with 1 on purpose when it finds problems: the red run is the alert.

Checking state under launchd

launchctl print gui/$(id -u) | grep cat.start.pipelines# loaded jobs and their last exit codelaunchctl print-disabled gui/$(id -u)# which ones are disabled

To check how a pipeline behaves under its schedule, fire the real plist instead of imitating its environment with env -i: launchd provides USER, LOGNAME, SHELL and TMPDIR, and a hand-made imitation doesn't. A loaded job can be triggered right now with:

launchctl kickstart -k gui/$(id -u)/cat.start.pipelines.<name>

This runs the real pipeline, guards included: if the throttle hasn't expired, you'll see a skipped, which is information too. From the CLI, pipelines run <name> --force skips the when guards, but not the requires checks.

Checking state under systemd

systemctl --user list-timers 'cat.start.pipelines.*'# next firingssystemctl --user status cat.start.pipelines.<name>.timerjournalctl --user -u cat.start.pipelines.<name>.service# service start-uptail -f ~/.local/state/pipelines/logs/pipelines-<name>.log# the job's outputsystemctl --user start cat.start.pipelines.<name>.service# fire it now

If systemctl --user says it can't connect to the bus, open a real session (SSH as that user, not su or sudo -u) or set XDG_RUNTIME_DIR=/run/user/$(id -u).

Where to look when something doesn't add up

The job's log holds what happened before the engine got as far as writing to .runs/: on macOS, ~/Library/Logs/pipelines-<name>.log by default; on Linux, ~/.local/state/pipelines/logs/pipelines-<name>.log. Locks, installed params, secrets and workspaces are listed in What lives on disk.

If a run died holding the lock, run exits with code 3 until the lock in .runs/.locks/<pipeline>.lock is cleared.