Logging
kube-oidc-proxy writes one structured stream. Every first-party record is a
single JSON object on stdout — startup, readiness, per-request access
decisions, cache lookups, shutdown — encoded by log/slog and carrying the
same envelope. There is no second text log to collect and no separate access-log
file: --logging-format picks the encoding for the whole stream and -v picks
how much of it you see.
- Record shape
- Compatibility
- Verbosity
- Event registry
- Field reference
- Startup, shutdown and exit status
- Correlation
- Worked queries
- ECS mapping
- Redaction
- See also
Record shape
Section titled “Record shape”Every record the proxy emits itself carries the same six keys before any event-specific field:
| Field | Type | Values / format | Notes |
|---|---|---|---|
time | string | RFC 3339 with sub-second precision | slog default |
level | string | ERROR, WARN, INFO, DEBUG | slog default |
msg | string | static text per event, never interpolated | query on event_type, not msg |
schema_version | int | 1 | bumped only on a breaking field change |
component | string | startup, server, oidc, readiness, request, tokenreview, sar, audit, upstream, shutdown, metrics, k8s | which subsystem spoke; k8s marks bridged library output |
event_type | string | <domain>.<object>.<action>, one of the 43 registered values | absent on component=k8s records |
An access decision looks like this (line-wrapped here, one line in the stream):
{"time":"2026-09-04T10:53:24.615018Z","level":"INFO","msg":"access decision","schema_version":1, "component":"request","event_type":"request.access.decided", "request_id":"7f1a9c1e-6a2b-4a1f-9f0e-3d9d1c2b5a04","event":"AuSuccess","src_ip":"10.42.1.3", "path":"/api/v1/namespaces/default/pods","forwarded_for_untrusted":"10.42.0.5","http_method":"GET", "auth_method":"oidc","issuer_name":"corp","k8s_verb":"list","k8s_api_group":"","k8s_resource":"pods", "k8s_namespace":"default","decision":"allow","inbound_user":"alice@example.com", "inbound_groups":["platform-admins","system:authenticated"], "inbound_extra":{"Remote-Client-IP":["10.42.1.3"]},"inbound_extra_omitted":1}Records bridged from the Kubernetes libraries (client-go, apimachinery, the
apiserver filters) carry component=k8s and no event_type. That absence
is the filter: event_type present means the proxy said it, component=k8s
means a library did.
Compatibility
Section titled “Compatibility”Five rules govern the event set. They are what makes a saved query keep working across upgrades.
- Append-only. New
event_typevalues may be added in any release. An existing value is never repurposed and never renamed in place; a retired value keeps its constant so the string is never reused. - Adding a value does not bump
schema_version. A query that matches one value is unaffected by the existence of a new one, so a new event is not a breaking change.schema_versionmoves only when field parsing breaks. - The registry is the source. One entry per event, with its component, default level, mandatory fields and one-line summary, lives in the code and generates the table below; CI fails on a diff. How to change it is in development: log contract.
- Query on
event_type, notmsg.msgis static human text and may be reworded.event_typeis the machine contract. component=k8srecords have noevent_type. They come from Kubernetes libraries through the klog bridge, keep their own wording, and are excluded from every rule above.
Versioning
Section titled “Versioning”What changes between releases, and what each kind of change means for a saved query:
- Adding an
event_typevalue is allowed in any release and does not bumpschema_version. A query for one value is unaffected by a new one. - Renaming a value after it has shipped is breaking. A record carries one
event_type, so there is no overlap window and emitting both names double-counts. The recovery is: add the new value, stop the old one in the same release, keep the old constant marked retired so the string is never reused, publish anevent_type IN (old, new)migration query, and announce it in the changelog. At least one minor release of notice, one major for anything underrequest.*orauthn.*. - Retiring a value because its code path was removed is not breaking. Queries return nothing instead of failing. Keep the constant marked retired.
schema_versionbumps only when field parsing breaks: a field removed or retyped, or a closed value set narrowed. Not for a new event, a new optional field, or a widened value set.
Verbosity
Section titled “Verbosity”-v is the single verbosity knob. There is no independent --log-level.
| Level | Content | Visible at |
|---|---|---|
| ERROR | The process cannot serve correctly: audit backend start or flush failure, the API server unreachable for a SubjectAccessReview or TokenReview, an internal invariant breach, an unreadable CA. | always |
| WARN | Degraded state and anomalies: an issuer pending with a reason (on state change only), a forwarded header dropped, repeated reserved-identity attempts, header-cap attempts. | always |
| INFO | Lifecycle (effective config and version, issuer configured and initialized, ready, serving, shutdown milestones, hook results) and the access record for every request, unsampled. | -v=0 |
| DEBUG | Per-request internals: the auth path taken, cache hit or miss, the live SubjectAccessReview, impersonation header names, client cancellation. | -v>=1 |
WARN and ERROR are never hidden, whatever -v says. Denials are INFO on the
access record and are never rate-limited: a bad token must not be able to buy a
warning line, and denial rates belong in metrics. Only
request.anomaly.detected, request.headers.dropped and
log.warning.suppressed are token-bucketed; when the bucket drops records,
log.warning.suppressed reports warning_reason, suppressed_count and
interval_seconds.
--logging-format selects json (the default) or text for the whole stream.
Both flags are documented in the
configuration reference.
Event registry
Section titled “Event registry”43 registered values, generated from the registry. “Required” lists the fields
that must be present beyond time, level, msg, schema_version,
component and event_type; conditional fields are described in the summary.
event_type | components | level | required | summary |
|---|---|---|---|---|
audit.backend.failed | audit | ERROR | error_message | The audit backend did not start. |
audit.backend.started | audit | INFO | backend_kind | The audit backend is running. |
audit.flush.completed | audit | INFO | duration_ms | The pre-shutdown audit flush succeeded. |
audit.flush.failed | audit | ERROR | error_message | The pre-shutdown audit flush failed. |
authn.oidc.failed | oidc | DEBUG | request_id, reason | The OIDC authenticator rejected the token. The denial itself is INFO on request.access.decided. |
authn.oidc.succeeded | oidc | DEBUG | request_id, issuer_name | The OIDC authenticator accepted the token. Names the issuer that accepted it. |
authn.token.missing | tokenreview | DEBUG | request_id | No bearer token was presented on the TokenReview path. |
authn.tokenreview.completed | tokenreview | DEBUG | request_id, authenticated | A live TokenReview answered, with duration_ms. A cache hit is reported by cache.tokenreview.lookup instead. |
authn.tokenreview.failed | tokenreview | ERROR | request_id, reason, error_message | The API server was unreachable or returned a status error. Carries reason=authentication_dependency_error. |
authz.impersonation.resolved | sar | DEBUG | request_id, target_kind, target_name | The whole impersonation sequence was allowed. A denial is request.access.decided with decision=deny and reason=impersonation_denied. |
authz.sar.completed | sar | DEBUG | request_id, decision, duration_ms, request_coalesced, target_kind | A live or shared SubjectAccessReview returned. Fires once per impersonation header value, so one request can emit several. |
authz.sar.failed | sar | ERROR or DEBUG | request_id, reason, error_message | The SubjectAccessReview call failed. Carries reason=authorization_dependency_error, or reason=client_canceled at DEBUG when the client hung up before the review answered. |
cache.sar.lookup | sar | DEBUG | request_id, cache_result | One SubjectAccessReview cache consultation. Carries decision on a hit, never the cache key. |
cache.tokenreview.lookup | tokenreview | DEBUG | request_id, cache_result | One TokenReview cache consultation. Carries authenticated on a hit, never the cache key. |
log.warning.suppressed | any | WARN | warning_reason, suppressed_count, interval_seconds | Token-bucket summary of dropped warning records. |
metrics.scrape.failed | metrics | ERROR | error_message | Gathering or encoding the exposition failed; the scrape was answered with 500. |
metrics.server.failed | metrics | ERROR | error_message | The metrics HTTP server returned an error. |
metrics.server.started | metrics | INFO | address | The metrics listener is serving /metrics. |
oidc.issuer.configured | oidc | INFO | issuer_name, issuer_count | Once per configured issuer at startup. |
oidc.issuer.initialized | oidc | INFO | issuer_name, issuer_state, ready_issuers, total_issuers | An issuer’s JWKS loaded. Carries issuer_state=initialized. |
oidc.issuer.pending | oidc | WARN | issuer_name, issuer_state, pending_reason, ready_issuers, total_issuers | The pending set or a pending reason changed. Carries issuer_state=pending; not emitted on every scrape. |
proxy.config.invalid | startup | ERROR | reason, error_message | The configuration cannot be used, including a CA bundle that cannot be read. |
proxy.config.loaded | startup | INFO | version, config_hash, issuer_count, readiness_mode | The effective non-secret configuration is fixed. |
proxy.hook.completed | shutdown | INFO | hook, duration_ms | One record per pre-shutdown hook that finished. |
proxy.hook.failed | shutdown | ERROR | hook, error_message | A pre-shutdown hook returned an error. |
proxy.server.started | server | INFO | address | The secure listener is serving. |
proxy.server.stopped | server | INFO | duration_ms | The secure listener stopped. |
proxy.shutdown.completed | shutdown | INFO | duration_ms | Listeners stopped, readiness stopped and pre-shutdown hooks finished. |
proxy.shutdown.started | shutdown | INFO | signal | A termination signal was received. The forced exit is the same event with forced=true. |
proxy.startup.failed | startup | ERROR | error_message | The process could not reach serving after the logger was built; it exits non-zero without a second, unstructured error line. |
readiness.proxy.ready | readiness | INFO | ready_issuers, total_issuers, readiness_mode | Readiness latched to ready. |
readiness.server.failed | readiness | ERROR | error_message | The readiness HTTP server returned an error. |
request.access.decided | request | INFO | request_id, event, src_ip, path, http_method, auth_method, decision | Authentication, authorization and proxy admission decided for a request. Carries event=AuSuccess|AuFail. |
request.anomaly.detected | request | WARN | request_id, src_ip, reason | A rejection that indicates an exploit attempt or gross misconfiguration. Token-bucketed; the access record still carries the outcome. |
request.handler.failed | request | ERROR | request_id, reason, error_message | The proxy hit an internal error while handling a request; the access record carries reason=internal_error. |
request.headers.dropped | request | WARN | request_id, src_ip, dropped_headers | X-Forwarded-For or X-Real-Ip removed for an untrusted peer, a fully trusted chain or a malformed hop. Carries forwarded_for_untrusted when the client sent an X-Forwarded-For; a client that sent only X-Real-Ip has no chain to report. Token-bucketed. |
request.headers.rewritten | request | DEBUG | request_id, src_ip, forwarded_for_untrusted | X-Forwarded-For collapsed to the resolved client IP on the trusted-proxy path. Diagnostic, not a warning: it is the trusted-proxy contract working as configured and fires on every request behind a trusted ingress. |
request.impersonation.applied | request | DEBUG | request_id, outbound_user, impersonated_header_names | Outbound impersonation headers built. Header names only, never values. |
request.impersonation.skipped | request | DEBUG | request_id, skip_reason | Impersonation was disabled by flag, or the request took the TokenReview passthrough path. |
request.response.completed | request | INFO | request_id, http_status, duration_ms, termination | The handler returned. The terminal record for every request, mirroring the audit stage ResponseComplete. |
request.response.started | request | INFO | request_id, http_status, time_to_headers_ms | First final status on a long-running request; a forwarded 1xx does not start the response. Mirrors the audit stage ResponseStarted. |
upstream.request.canceled | upstream | DEBUG | request_id, reason | The client went away before the upstream response completed. Carries reason=client_canceled. |
upstream.request.failed | upstream | ERROR or DEBUG | request_id, reason, termination, error_message | The reverse proxy transport failed. Carries reason=upstream_error and a classified termination; drops to DEBUG when the client canceled the request. |
Field reference
Section titled “Field reference”Status: frozen means the name and value format exist today and must not change — SIEM rules key on them. new means added by the structured-logging work.
src_ip and path are the only emitted names for the client address and the
request path. No alias of either is emitted and no deprecation window is open.
If an alias is ever added it goes in this table with a status of alias.
On request-scoped records
Section titled “On request-scoped records”| Field | Type | Values / format | Status | Records | Notes |
|---|---|---|---|---|---|
request_id | string | UUID v4 minted by the proxy, max 64 chars | new | all request records | the same value as the Audit-ID header sent upstream and echoed to the client |
client_request_id | string | inbound Audit-ID or X-Request-ID, printable ASCII, max 64 chars | new | request.access.decided | present only when a valid inbound value existed; never used as request_id unless the peer is a trusted proxy |
event | string | AuSuccess, AuFail | frozen | request.access.decided | the SIEM contract |
src_ip | string | IPv4 or IPv6, trusted-proxy resolved | frozen | all request records | the raw peer address with its port is never logged |
forwarded_for_untrusted | string | raw X-Forwarded-For chain, sanitized | frozen | request.access.decided, request.headers.* | forensic, not identity |
path | string | URL path, query string stripped, sanitized | frozen | request.access.decided | |
http_method | string | GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS, CONNECT | new | request.access.decided | |
k8s_verb | string | get, list, watch, create, update, patch, delete, deletecollection, proxy, connect | new | request.access.decided | from the request-info resolver; absent on non-resource paths |
k8s_api_group | string | API group, empty string for the core group | new | request.access.decided | |
k8s_resource | string | resource name, for example pods | new | request.access.decided | |
k8s_subresource | string | for example exec, log, portforward | new | request.access.decided | absent when none |
k8s_namespace | string | namespace | new | request.access.decided | absent for cluster-scoped requests |
k8s_name | string | object name, max 256 chars | new | request.access.decided | absent on list and watch |
auth_method | string | oidc, tokenreview, none | new | request.access.decided | none is the no-token 401 path |
issuer_name | string | configured issuer name, max 256 chars | new | request.access.decided, oidc.issuer.* | never the full issuer URL |
reason | string | closed set, see reason vocabularies | new | request.access.decided on AuFail, and on error records | |
decision | string | allow, deny | new | request.access.decided, cache.sar.lookup, authz.sar.completed | authorization outcome |
inbound_user | string | username, sanitized, max 256 chars | frozen | request.access.decided | |
inbound_uid | string | UID, sanitized | frozen | request.access.decided | absent when empty |
inbound_groups | array of string | first 32 groups, sanitized | frozen | request.access.decided | the cap is new |
inbound_groups_omitted | int | groups beyond the cap | new | request.access.decided | absent when zero |
inbound_extra | object | allowlisted impersonation extras only | frozen | request.access.decided | originaluser.jetstack.io-extra is no longer on the allowlist |
inbound_extra_omitted | int | count of extras not logged | frozen | request.access.decided | |
outbound_user, outbound_uid, outbound_groups, outbound_groups_omitted, outbound_extra, outbound_extra_omitted | as the inbound_* fields | as the inbound_* fields | frozen (the caps are new) | request.access.decided when impersonating | compare with inbound_* to see who acted as whom |
target_kind | string | user, group, uid, extra, serviceaccount, unknown | new | request.access.decided on impersonation_denied, authz.sar.completed, authz.impersonation.resolved | replaces parsing the error text |
target_name | string | impersonation target, sanitized, max 256 chars | new | same as target_kind | |
http_status | int | HTTP status written to the client | new | request.response.started, request.response.completed | a 200 implied by the first Write, or by a handler that returned without writing, is recorded as 200; 0 only when no response went out, because the connection was hijacked or the handler aborted before writing |
time_to_headers_ms | int | milliseconds from request start to the first final status | new | request.response.started | long-running requests only; a forwarded 1xx does not start the response |
duration_ms | int | milliseconds from request start to handler return | new | request.response.completed, authz.sar.completed, authn.tokenreview.completed | |
response_bytes | int | bytes written to the client | new | request.response.completed | absent when the connection was hijacked |
termination | string | normal, client_cancel, upstream_reset, upstream_timeout, proxy_error, hijacked, panic | new | request.response.completed, upstream.request.failed | |
error_message | string | sanitized, max 512 chars | new | DEBUG records, ERROR records and internal_error | never at INFO or WARN for a client-caused failure |
dropped_headers | array of string | x-forwarded-for, x-real-ip | new | request.headers.dropped | |
skip_reason | string | disabled, passthrough | new | request.impersonation.skipped | |
impersonated_header_names | array of string | Impersonate-User, Impersonate-Group, Impersonate-Uid, Impersonate-Extra-<key> | new | request.impersonation.applied | header names only, never values |
On cache and review records
Section titled “On cache and review records”| Field | Type | Values / format | Records | Notes |
|---|---|---|---|---|
cache_result | string | hit, miss, bypass | cache.sar.lookup, cache.tokenreview.lookup | bypass covers a disabled cache and an oversized SAR spec; miss includes expired |
request_coalesced | bool | true, false | authz.sar.completed | singleflight shared one live check between requests |
decision | string | allow, deny | cache.sar.lookup, authz.sar.completed | on a SAR cache hit |
authenticated | bool | true, false | authn.tokenreview.completed, cache.tokenreview.lookup | on a TokenReview cache hit |
duration_ms | int | live call duration | authz.sar.completed, authn.tokenreview.completed | absent on a cache hit |
Cache keys, token hashes and the serialized SubjectAccessReview spec are never
logged. The cache identity is in the event type, not in a field.
On lifecycle records
Section titled “On lifecycle records”| Field | Type | Values / format | Records |
|---|---|---|---|
version | string | build version | proxy.config.loaded |
config_hash | string | hash of the effective non-secret configuration | proxy.config.loaded |
issuer_count | int | configured issuers | proxy.config.loaded, oidc.issuer.configured |
issuer_state | string | pending, initialized | oidc.issuer.initialized, oidc.issuer.pending |
pending_reason | string | not_initialized, transient, error | oidc.issuer.pending |
ready_issuers, total_issuers | int | counts | oidc.issuer.initialized, oidc.issuer.pending, readiness.proxy.ready |
readiness_mode | string | any, all | proxy.config.loaded, readiness.proxy.ready |
hook | string | pre-shutdown hook name | proxy.hook.completed, proxy.hook.failed |
warning_reason | string | the suppressed record’s reason | log.warning.suppressed |
suppressed_count | int | records dropped in the interval | log.warning.suppressed |
interval_seconds | int | summary interval | log.warning.suppressed |
address | string | bound listen address | proxy.server.started |
signal | string | SIGTERM, SIGINT | proxy.shutdown.started |
forced | bool | true on the final signal | proxy.shutdown.started, absent otherwise |
backend_kind | string | audit backend type | audit.backend.started |
Reason vocabularies
Section titled “Reason vocabularies”reason is a closed set, but there are two of them and they never mix.
Request failures — on request.access.decided with event=AuFail, on
request.response.completed when the request ended in an error, and on the
authn.*, authz.*, upstream.* and request.handler.failed records:
unauthorized, reserved_identity, impersonation_denied,
too_many_impersonation_values, no_username_claim,
authentication_dependency_error, authorization_dependency_error,
upstream_error, internal_error, client_canceled.
Startup configuration failures — on proxy.config.invalid only, a
startup-scoped vocabulary that is deliberately disjoint from the request set so
a query for a configuration problem can never match request traffic:
ca_file_unreadable, invalid_configuration.
ca_file_unreadable is emitted when the CA bundle named by
--oidc-ca-file cannot be read; invalid_configuration is reserved for an
effective configuration the process refuses.
Values explicitly excluded
Section titled “Values explicitly excluded”Bearer tokens, Authorization and Cookie values, any header not on the
allowlist, request and response bodies, cache keys, arbitrary claims and
extras, configured extra-header values, the User-Agent header, full issuer
URLs, and raw peer addresses with ports.
Startup, shutdown and exit status
Section titled “Startup, shutdown and exit status”The log stream is the only place the process reports its own failures once the
root logger exists. A startup error after that point (an unreadable kubeconfig,
a certificate that does not load, a port already bound) emits
proxy.startup.failed with error_message and the process exits 1 without
printing a second, unstructured line. Only a failure before the logger can be
built, such as an unknown flag or an invalid --logging-format, is printed to
stderr as plain text, because there is nothing else to report through yet.
On shutdown the audit backend is flushed before the process exits, and the
outcome is logged. With the backends bundled from k8s.io/apiserver (log,
and the buffered webhook) a lost flush is visible as a bridged
component=k8s ERROR record, audit.flush.completed still follows with
duration_ms, and the process exits 0: those backends do not report an error
from Shutdown(), because the upstream audit.Backend interface has no return
value there, and the buffered backend logs a delivery failure through the
Kubernetes error handler instead. So with a bundled backend, alert on the
component=k8s ERROR, not on the exit status.
A backend that implements the optional ShutdownErr() error extension gets
stricter handling: a flush it reports as failed emits audit.flush.failed,
propagates out of the pre-shutdown hooks as proxy.hook.failed, and makes the
process exit non-zero, on the grounds that a dropped audit event for a
request this process already served is not a warning to be tidied away. None
of the bundled backends implement the extension today.
Correlation
Section titled “Correlation”One request produces several records — an access decision, cache lookups, a
SubjectAccessReview, a terminal request.response.completed. request_id
joins them, and it reaches beyond this process:
- The outermost filter mints a UUID v4 per request and sets it as the
Audit-IDrequest header. - Both proxy audit chains read that header, so the proxy’s own audit events carry the same value instead of minting two.
- The reverse proxy forwards the header upstream, so the kube-apiserver audit
event’s
auditIDis the same string. One id spans the proxy log, the proxy audit log and the API server audit log. - The response carries
Audit-IDback to the client, so a user can quote the id of the request they are complaining about.
In short: request_id = the Audit-ID header = the kube-apiserver
auditID.
An inbound Audit-ID or X-Request-ID is not adopted as request_id
unless the immediate peer is inside a configured --trusted-proxies network. A
syntactically valid inbound value from an untrusted peer is kept as
client_request_id on the access record — visible for correlation with an
ingress, never authoritative. See
trusted proxies and client IP.
Two decisions are recorded for a forwarded request, and they answer different questions:
| Field | Who decides | Question answered |
|---|---|---|
decision on the proxy’s access record | the proxy | Was the request admitted: token verified and any Impersonate-* headers authorized? allow means it was forwarded. |
authorization.k8s.io/decision on the API server’s audit event | the API server | Was the impersonated identity allowed to perform the action? allow or forbid. |
decision=allow followed by an upstream 403 is a normal, healthy outcome:
the proxy admitted the request and RBAC refused the action. The upstream
result is http_status on request.response.completed and the annotation on
the API server’s event. decision=deny means the proxy refused the request
itself, so no API server audit event exists for it, and reason says why.
event=AuSuccess|AuFail and decision=allow|deny are the frozen contract SIEM
rules already key on. event names the outcome, event_type names the record
shape; AuSuccess is only ever emitted with decision=allow and AuFail only
with decision=deny, and a test enforces the pairing.
Worked queries
Section titled “Worked queries”Three questions, in three query languages. $POD is a proxy pod, $RID a
request id.
Everything for one request
Section titled “Everything for one request”kubectl -n kube-oidc-proxy logs "$POD" | grep "\"request_id\":\"$RID\""{app="kube-oidc-proxy"} | json | request_id = "<request-id>"index=kubernetes sourcetype=kube-oidc-proxy request_id="<request-id>"| sort 0 _time| table _time level event_type decision reason http_status duration_msThe same id in the API server audit log finds the upstream half:
auditID == "<request-id>".
Denials by reason in the last hour
Section titled “Denials by reason in the last hour”kubectl -n kube-oidc-proxy logs "$POD" --since=1h \ | jq -r 'select(.event_type == "request.access.decided" and .decision == "deny") | .reason' \ | sort | uniq -c | sort -rnsum by (reason) ( count_over_time( {app="kube-oidc-proxy"} | json | event_type = "request.access.decided" | decision = "deny" [1h] ))index=kubernetes sourcetype=kube-oidc-proxy earliest=-1h event_type="request.access.decided" decision="deny"| stats count by reason| sort -countIssuer state
Section titled “Issuer state”An issuer’s lifecycle is two record types: oidc.issuer.pending, emitted when
the pending set or a pending reason changes, and oidc.issuer.initialized,
emitted once its JWKS has loaded. Its current state is therefore the newest of
the two per issuer_name, within one process; a query on pending records
alone shows pending history and would miss the initialization that ended it.
kubectl -n kube-oidc-proxy logs "$POD" \ | jq -rs 'map(select(.event_type == "oidc.issuer.pending" or .event_type == "oidc.issuer.initialized")) | group_by(.issuer_name) | map(last) | .[] | "\(.issuer_name)\t\(.issuer_state)\t\(.pending_reason // "-")\t\(.ready_issuers)/\(.total_issuers)"'In Loki a log query cannot keep only the newest line per issuer, so the
line-level form below is the lifecycle history, newest first, not the current
state; read the last line for each issuer_name and pod.
{app="kube-oidc-proxy"} | json | event_type =~ "oidc.issuer.(pending|initialized)" | line_format "{{.issuer_name}} {{.issuer_state}} {{.pending_reason}} {{.ready_issuers}}/{{.total_issuers}}"For the current state as a table, use the fact that an issuer never returns
to pending within one process: every pod and issuer pair that has logged
oidc.issuer.initialized is ready, and a configured issuer missing from the
result is still pending.
sum by (pod, issuer_name) ( count_over_time({app="kube-oidc-proxy"} | json | event_type = "oidc.issuer.initialized" [$__range]))index=kubernetes sourcetype=kube-oidc-proxy event_type IN ("oidc.issuer.pending", "oidc.issuer.initialized")| stats latest(issuer_state) AS state latest(pending_reason) AS pending_reason latest(_time) AS last_seen by pod, issuer_nameGroup by pod as well as issuer when several replicas run: each process initializes its issuers independently. See multi-issuer readiness.
Every replica at once
Section titled “Every replica at once”kubectl logs deploy/<name> reads one pod. Select by label to merge all
replicas, and lift the per-pod line cap kubectl applies to a selector:
kubectl -n kube-oidc-proxy logs -l app.kubernetes.io/name=kube-oidc-proxy \ --since=15m --tail=-1 --prefixLoki and Splunk already see every pod; the equivalent there is a label or sourcetype match, which the queries above use.
One line per request
Section titled “One line per request”An access decision and a terminal request.response.completed are two records
of one request; a kubectl --as request at -v=1 adds one cache.sar.lookup
per impersonation header value. Grouping on request_id puts the decision, the
upstream status and the cache outcome on the same line.
kubectl -n kube-oidc-proxy logs -l app.kubernetes.io/name=kube-oidc-proxy --since=15m --tail=-1 \ | jq -rs 'map(select(.request_id != null)) | group_by(.request_id) | map((map(select(.event_type == "request.access.decided"))[0] // {}) as $a | (map(select(.event_type == "request.response.completed"))[0] // {}) as $r | (map(select(.event_type == "cache.sar.lookup") | .cache_result) | join(",")) as $c | select($a.time != null) | [$a.time, $a.event, ($a.reason // "-"), ($a.inbound_user // "-"), ($a.k8s_verb // "-"), ($a.k8s_api_group // ""), ($a.k8s_resource // "-"), ($a.k8s_namespace // "-"), (if $c == "" then "-" else $c end), (($r.http_status // "-") | tostring), $a.request_id] | @tsv) | .[]'index=kubernetes sourcetype=kube-oidc-proxy earliest=-15m request_id=*| stats earliest(_time) AS _time values(event) AS event values(reason) AS reason values(inbound_user) AS user values(k8s_verb) AS verb values(k8s_api_group) AS api_group values(k8s_resource) AS resource values(k8s_namespace) AS namespace values(cache_result) AS cache values(http_status) AS status by request_id| where isnotnull(event)| sort _timek8s_verb, k8s_api_group, k8s_resource and k8s_namespace are the
request-info dimensions, in the vocabulary RBAC rules use: list rather than
GET, and the group separate from the resource with the core group empty.
In LogQL there is no join across lines; filter on request_id for one request
(above), or alert on request.response.completed with http_status and pull
the decision by ID when a line needs explaining. A shell function that renders
this as a table, with a worked example of what each row means, is in
operations: watching requests.
ECS mapping
Section titled “ECS mapping”The proxy’s field names are the contract; they are not renamed to Elastic Common Schema. This table is for whoever writes the ingest pipeline that copies them.
| Proxy field | ECS field | Notes |
|---|---|---|
time | @timestamp | |
level | log.level | |
msg | message | |
component | log.logger | |
event_type | event.action | |
request_id | trace.id | also the kube-apiserver auditID; http.request.id if you prefer a transaction-scoped field |
src_ip | source.ip | the resolved client IP |
path | url.path | |
http_method | http.request.method | |
http_status | http.response.status_code | |
response_bytes | http.response.body.bytes | |
duration_ms | event.duration | event.duration is nanoseconds — multiply by 1,000,000 |
time_to_headers_ms | — | no ECS equivalent; keep as a custom field |
event | event.outcome | AuSuccess → success, AuFail → failure; ECS also allows unknown. Map from event, not from decision |
reason | event.reason | |
inbound_user | user.name | the authenticated identity |
inbound_groups | user.roles | |
outbound_user | user.effective.name | ECS’s user.effective.* reusable set is exactly the impersonation case |
outbound_groups | user.effective.roles | |
error_message | error.message | |
k8s_verb, k8s_resource, k8s_namespace, … | — | no ECS equivalent; keep as custom fields under your own namespace |
Redaction
Section titled “Redaction”- Never logged: bearer tokens,
AuthorizationandCookievalues, any header not on the allowlist, request and response bodies, cache keys, arbitrary claims and extras, configured extra-header values, theUser-Agentheader, full issuer URLs. originaluser.jetstack.io-extrais not on the loggable extras allowlist. That extras existed is reported as a bounded count (inbound_extra_omitted,outbound_extra_omitted), never as content.- Every user-influenced string passes through the
sanitizehelper, which strips control characters, so nothing a client sends can inject a second record or fake a field. - Bounds:
request_id64 characters,error_message512, identity strings 256, group lists capped at 32 withinbound_groups_omitted/outbound_groups_omittedreporting the drop. - Prefer a classified
reasonover raw error text at INFO and WARN. A truncatederror_messageappears only on internal failures and at DEBUG. issuer_nameis the configured name. Full issuer URLs are never logged.
See also
Section titled “See also”- Operations: reading the request log — the operator narrative and the fields to grep first.
- Shipping logs and audit events — a node-level agent recipe that moves this stream and the audit events to AWS Kinesis.
- Configuration: logging —
--logging-formatand-v, and the chart values that render them. - Caching —
cache.sar.lookup,cache.tokenreview.lookupandauthz.sar.completed. - Multi-issuer —
oidc.issuer.*andissuer_name. - Metrics — the same vocabularies as counters and histograms; denial rates live there.
- CONTRIBUTING — how to add an event.