Vault secret rotation per tenant

Direct answer

Changing a value in Vault is only the first step of rotation. A production platform must synchronize the new value, identify every dependent workload, restart only applications that cannot reload it in place, verify readiness, and preserve a rollback path. OpenTenant automates that tenant-scoped sequence across Vault, Vault Secrets Operator or External Secrets Operator, and targeted Kubernetes rollouts.

The goal is not “restart on every Secret event.” It is a controlled credential transition with contained blast radius.

Three separate lifecycle problems

Secret systems are often discussed as one feature, but rotation crosses three controllers:

1. Secret authority: Vault creates, stores, leases, or updates the credential.
2. Delivery: VSO, External Secrets Operator, CSI, or an agent makes the value available to a workload.
3. Consumption: The application reloads the value in place or a rollout replaces its pods.

A green status at one layer does not prove the next layer completed. OpenTenant records each stage against one tenant release.

Vault Secrets Operator

HashiCorp's Vault Secrets Operator documentation says VSO watches custom resources, synchronizes supported Vault sources into destination Kubernetes Secrets, and keeps source changes replicated over the destination's lifetime. HashiCorp also documents automatic drift remediation and secret rotation support for Deployments, ReplicaSets, and StatefulSets.

This works well for Helm charts that already consume Kubernetes Secrets. The application does not need a direct Vault client, but the destination Secret becomes part of the security boundary. Kubernetes RBAC, etcd encryption, operator permissions, and namespace isolation remain important.

External Secrets Operator

External Secrets Operator also reconciles external values into Kubernetes Secrets. Its refresh behavior is explicit:

• Periodic is the default and updates at refreshInterval when the interval is greater than zero.
• CreatedOnce creates the Secret and does not follow future source changes.
• OnChange updates when the ExternalSecret's metadata or specification changes.

Those modes are operational choices. A database password expected to rotate should not silently use CreatedOnce. A bootstrap password persisted by the application may become dangerous if regenerated periodically. OpenTenant stores the refresh policy with the tenant secret binding instead of applying one global default to every credential.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-database
  namespace: tenant-acme
spec:
  refreshPolicy: Periodic
  refreshInterval: 15m
  secretStoreRef:
    name: tenant-acme-vault
    kind: SecretStore
  target:
    name: app-database
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: tenants/acme/database
        property: password

The example illustrates desired state; names, paths, and intervals must be generated from trusted tenant identity and tested against the installed operator version.

Synchronization is not application reload

Kubernetes updates a Secret object, but many applications read environment variables only at process start. Mounted Secret volumes may update eventually, yet the application may still cache the old value. The platform must know each workload's consumption behavior.

There are three common strategies:

Native hot reload

Prefer an application-native reload when it is documented, observable, and does not interrupt requests. The workload should expose proof that it accepted the new credential.

Operator-triggered rollout

Use VSO's supported rollout behavior when the selected resource and secret CRD support it. Keep the mapping explicit so one tenant secret cannot trigger unrelated workloads.

Reloader-triggered rollout

Stakater Reloader watches ConfigMaps and Secrets and triggers workload rollouts. Its search-and-match annotations allow a workload to reload only when it references a resource explicitly marked for reload—useful in a multi-tenant cluster where global auto-reload would be too broad.

OpenTenant generates targeted annotations from the dependency graph. A shared platform Secret should never restart every tenant merely because a controller can discover all references.

Rotation needs an overlap plan

Replacing a credential immediately can break live connections before new pods are ready. Where the upstream system permits it, safer rotation uses an overlap window:

1. Create the new credential or lease.
2. Synchronize it to the tenant scope.
3. Roll or reload dependent workloads.
4. Verify authentication and readiness with the new value.
5. Revoke the previous credential after the grace period.

Some systems only permit a single active password. Those rotations need maintenance behavior, connection draining, and rollback rules specific to the application. OpenTenant cannot manufacture dual-credential support when the dependency lacks it; the deployment profile must say so.

How OpenTenant contains blast radius

Vault version or lease changes
  → operator reconciles tenant destination
  → dependency index resolves affected workloads
  → tenant rollout policy selects reload or restart
  → readiness and authentication checks run
  → previous credential is revoked when safe
  → event and release provenance are recorded

The dependency index is the key. Each secret binding records the tenant, Vault path or role, destination, consuming workload, and rollout strategy. Rotation of tenants/acme/database affects Acme's API and workers—not every release using the same base chart.

Failure handling

A secure rotation controller fails closed and exposes the stage that failed.

• Vault update failed: do not change Kubernetes state.
• Synchronization stalled: keep the previous workload running and alert on version mismatch.
• Rollout failed: stop revocation while the old credential remains valid, when the authority supports overlap.
• Readiness passed but authentication failed: treat the release as failed and restore the previous reference or credential according to the dependency's rollback capability.
• Repeated events: coalesce updates so a burst of Secret changes does not create endless pod churn.

Verification checklist

• Vault policies restrict each tenant to its own secret paths or dynamic roles.
• Operator credentials follow least privilege and cannot read the entire tenant registry unnecessarily.
• Refresh policies match credential semantics.
• The destination Secret is encrypted and RBAC-restricted.
• Every consuming workload declares reload behavior.
• Negative tests prove tenant A's secret cannot restart or update tenant B.
• Rotation tests cover old connections, new connections, readiness, and rollback.
• Audit records connect the Vault version to the Kubernetes Secret version and rollout revision.
• Offboarding revokes credentials before deleting dependent resources.

Sources

• Vault Secrets Operator — HashiCorp
• ExternalSecret refresh behavior — External Secrets Operator
• Reloader — Stakater

OpenTenant makes rotation a tenant operation rather than a cluster-wide event: synchronize the intended value, touch only the declared dependents, and prove the release recovered before retiring the old credential.