Events and audit logs

How Flux's event log doubles as the audit substrate — what's recorded, what isn't, and how to query it for forensics.

Flux does not ship a separate audit log. The execution_events table — the same one that drives replay, resume, and the timeline view in the console — is the audit substrate for everything that happens inside a workflow. Authentication and authorization decisions, by contrast, are not persisted at all: they go to stdlib logging and end up in the server’s stdout. Knowing which signal lives where decides whether a forensic question is answerable at all.

The execution event log

Every state transition a workflow or task makes is appended as one row in execution_events (flux/models.py:610ExecutionEventModel, with a (execution_id, source_id, event_id, type, name, value, time, subject) shape). The row never moves. value is a PickleType(pickler=dill) column, so the inputs and outputs that bracket each event are reconstituted verbatim on read. Cascade-on-delete is wired up to executions.execution_id, which means dropping an execution wipes its events — there is no append-only retention guarantee at the schema level.

The event vocabulary is defined in flux/domain/events.py::ExecutionEventType and currently has 27 types across five lifecycle groups:

GroupCountExamples
Workflow lifecycle12WORKFLOW_SCHEDULED, WORKFLOW_CLAIMED, WORKFLOW_STARTED, WORKFLOW_COMPLETED, WORKFLOW_FAILED, WORKFLOW_PAUSED, WORKFLOW_RESUMING, WORKFLOW_RESUMED, WORKFLOW_RESUME_SCHEDULED, WORKFLOW_RESUME_CLAIMED, WORKFLOW_CANCELLING, WORKFLOW_CANCELLED
Task lifecycle6TASK_STARTED, TASK_COMPLETED, TASK_FAILED, TASK_PAUSED, TASK_RESUMED, TASK_PROGRESS
Retry chain3TASK_RETRY_STARTED, TASK_RETRY_COMPLETED, TASK_RETRY_FAILED
Fallback chain3TASK_FALLBACK_STARTED, TASK_FALLBACK_COMPLETED, TASK_FALLBACK_FAILED
Rollback chain3TASK_ROLLBACK_STARTED, TASK_ROLLBACK_COMPLETED, TASK_ROLLBACK_FAILED

There are no SECURITY_*, LOGIN_*, or KEY_* event types. Auth decisions stay out of this table.

Authentication events live in stdout

flux/security/auth_service.py, flux/security/providers/api_key.py, and flux/security/providers/oidc.py all route their notable moments through the module logger returned by flux.utils.get_logger. Auth-disabled requests become a logger.warning("Auth disabled — request treated as admin (anonymous)"). Expired API keys log at WARNING; a disabled principal logs at WARNING with the subject. OIDC token validation failures log at WARNING with the underlying exception. Provider exceptions log at ERROR. None of those writes touch the database.

There is no audit_log table, no security_events table, and AuthService.create_api_key, grant_role, revoke_api_key, and delete_principal do not emit anything beyond their commit. If you need a record of who created a key or who reassigned a role, you must capture it from the server’s stdout — typically by shipping it to a SIEM via journald, Loki, CloudWatch, or whatever your log pipeline is.

How to read what’s there

Three avenues, in increasing order of effort:

  1. Per-execution timeline. flux execution show <id> --detailed (flux/cli.py:728) hits GET /executions/{id}?detailed=true and prints every event for that run as JSON. This is the right tool for “what happened inside this specific failure.”
  2. Server stdout. Tail the server process (or whatever you’ve redirected its stdout into) for auth, OIDC, and provider warnings. Anything starting with Auth disabled, OIDC token, API key, or Principal '...' is security-relevant.
  3. Direct SQL. For aggregate questions — “how many TASK_FAILED events did namespace etl produce last week?”, “list every WORKFLOW_CANCELLED triggered by principal X” — there is no first-class CLI. Open the database and query execution_events joined on executions.

The executions row itself records who scheduled the run via scheduling_subject and scheduling_principal_issuer (flux/models.py:534-535), not a principal_id column. Cross-reference those against the principals table when answering “who triggered this?”

For schedule history specifically, flux schedule history <id> (flux/cli.py:1479) dumps the full run sequence for a single schedule, including dispatches and the executions they produced.

What is not captured

Retention and three failure modes

Retention is indefinite. execution_events grows for as long as executions remain — there is no built-in TTL or rotation. Three concrete operational hazards follow:

  1. Unbounded growth. A high-cardinality workflow (hundreds of tasks, thousands of runs/day) will balloon the table. You need a periodic job that deletes old executions (cascade removes their events), or a partitioned table on Postgres, or both.
  2. Per-secret access blindness. No table records who fetched which secret when. If your threat model needs that, add the audit-proxy described above before you need it, not after.
  3. Security events only in stdout. A misconfigured deployment that swallows the server’s stdout silently loses every auth event Flux produces. Confirm the log pipeline is collecting the server process before you trust it as your audit channel.