Secrets management

A Flux secret is a named string that workflows request by name. The server stores it AES-GCM-encrypted in the secrets table (flux/models.py::SecretModel wraps the value column in EncryptedType) and releases it only to a worker running an execution whose workflow declared the name in secret_requests. Workflows declare via @task.with_options(secret_requests=[...]) or @workflow.with_options(secret_requests=[...]); the runtime injects a secrets kwarg into the task call (flux/task.py:284-286).

CLI lifecycle

The Click group is flux secrets (plural, flux/cli.py:1529). Four subcommands, all of which talk to the server’s /admin/secrets routes:

flux secrets set DB_PASSWORD 's3cret-value'   # create or overwrite
flux secrets list                              # names only, never values
flux secrets get DB_PASSWORD                   # prompts for confirmation, then prints plaintext
flux secrets remove DB_PASSWORD                # permanent delete

Each subcommand accepts --server-url and --format json|text. get prompts for confirmation before printing; pass --yes to skip in scripts. The CLI does not write to the database directly — it always goes through HTTP, so the server must be reachable.

REST equivalents

The admin routes (flux/server.py:2185-2284), all gated by admin:secrets:manage (writes) or admin:secrets:read (reads):

Scoping

Secrets are global in 0.56.0. SecretModel has a single primary-key column, name, with no namespace, workflow_id, or principal_id discriminator. If two teams want logical separation, use a prefix convention (team-a/db_password, team-b/db_password) — Flux treats them as ordinary names.

At-rest encryption

EncryptedType (flux/models.py:177) derives an AES-GCM key from flux.security.encryption.encryption_key via PBKDF2 and refuses to load if the setting is unset: ValueError: Encryption key is not set in the configuration. Set the key via FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY or flux.toml. See Encryption at rest for key generation and rotation.

Runtime delivery to workers

When a worker claims an execution it installs a RemoteSecretManager into the context (flux/worker.py:501-506). On first read, that manager POSTs /workers/{name}/secrets/batch with {"execution_id": ..., "names": [...]}. The server (flux/server.py:2141-2183) checks four things before answering:

1. The bearer token belongs to the worker named in the URL.
2. The execution exists and ctx.current_worker matches.
3. The workflow was found in the catalog.
4. Every requested name appears in the workflow’s declared secret_requests. Undeclared names return 403 Secrets not declared by workflow: [...].

Only then does the server decrypt and return plaintext. The worker holds the values in process memory for the duration of the task and never persists them to the event log.

Rotation

flux secrets set NAME NEWVALUE overwrites the row in place (save() in flux/secret_managers.py:52-66 does a session.get + assignment). There is no version history table in 0.56.0 — the previous ciphertext is replaced and unrecoverable. In-flight executions that have already read the secret keep the old value in process memory until the task completes; the next read returns the new value.

Env-var fallback

None. DatabaseSecretManager.get (flux/secret_managers.py:79-90) issues a single SELECT against the secrets table; any requested name not present in the row set is collected into missing and raises ValueError: The following secrets were not found: [...]. There is no os.environ.get fallback at any layer (secret_managers.py, remote_managers.py, or task.py). If a secret is not registered, the workflow fails.

What can go wrong

Secret requested but not registered. await SecretManager.current().get([...]) raises, the task fails, and the failure surfaces in the event log as a TASK_FAILED with The following secrets were not found: ['api_key']. Fix: flux secrets set api_key <value>. Verify with flux secrets list.

Plaintext logged accidentally. Anything a task writes to stdout, structured logs, or a return value is captured in the execution event log. The runtime cannot tell a string came from secrets["…"] rather than from a literal. Keep secret-bearing variables out of return values, exception messages, and log lines.

Encryption key missing or wrong. EncryptedType.process_result_value raises during the decrypt step, so flux secrets get and any task reading a secret returns 500. The server log shows Encryption key is not set in the configuration. or a GCM tag mismatch. Fix: set FLUX_SECURITY__ENCRYPTION__ENCRYPTION_KEY to the value the rows were written under. Rotating the key without re-encrypting existing rows leaves the store unreadable; see Encryption at rest.