Continuous authorization—making access decisions dynamically using live signals—has moved from theory to operational requirement for zero‑trust networks in 2026. The key enabler is telemetry: well-structured, timely, and trustworthy signals about identity, device posture, network flows, and application behavior. This guide walks practitioners through designing, deploying, and operating a scalable telemetry pipeline that turns raw signals into enforceable authorization decisions using OpenTelemetry, Apache Kafka, Open Policy Agent (OPA), and a SIEM/observability backend.

Why a dedicated telemetry pipeline matters for zero trust

Zero trust requires frequent re-evaluation of privileges. Static, infrequent checks (e.g., once-per-login MFA) are insufficient for modern threats such as session hijacking, lateral movement, or compromised endpoints. A telemetry pipeline provides:

  • Real-time signals for continuous authorization (device health, user behavior, session context).
  • Centralized enrichment and normalization so diverse sources map to consistent attributes.
  • Scalable transport and buffering to decouple producers (agents, apps) from consumers (policy engines, SIEM).
  • Auditable storage and analytics for forensics, compliance, and model training.

High-level architecture

The reference architecture in this guide uses four logical layers:

  1. Signal sources: identity providers (IdP), endpoints (MDM/EDR), service meshes/API gateways, authentication logs, network sensors.
  2. Collection and instrumentation: OpenTelemetry agents/SDKs for traces/metrics/logs with dedicated collectors at edge and core.
  3. Transport and buffering: Apache Kafka (or cloud-managed streaming) as a durable, scalable bus.
  4. Processing, decisioning and storage: stream processors for enrichment, OPA (sidecar or service) for decisioning, and SIEM/long-term store for analysis.

Decision enforcement points (DEPs) such as gateways, service mesh sidecars, or NAC devices query the Policy Decision Point (PDP) after event enrichment. Synchronous decisions (allow/deny) must meet tight latency budgets; asynchronous risk scoring can feed future decisions or session termination.

Step 1 — Inventory signals and set event schema

Start by cataloguing available signals and desired signals. Typical categories:

  • Identity signals: IdP events, SCIM changes, session tokens, role/entitlement changes.
  • Device posture: MDM status, FIDO2 attestation, disk encryption state, patch level.
  • Network telemetry: flow records, connection metadata, TLS server name, source/destination.
  • Application traces/metrics: request latency, error rates, unusual endpoints.
  • Security alerts: EDR detections, IOC matches, sandbox results.

Define an event schema that captures required attributes for policy decisions: timestamp, subject (user or service), identity attributes (role, group), device id & posture, source IP, destination, resource id, action, risk_score, and trace_id. Use consistent attribute names and types across source adapters.

Step 2 — Instrument and collect with OpenTelemetry

OpenTelemetry (OTel) is the practical choice in 2026 for traces, metrics and structured logs. Deploy collectors near producers (sidecar/host-level) configured to:

  • Accept OTLP over gRPC/HTTP from SDKs and agents.
  • Apply lightweight processing (attribute renaming, sampling policy, redaction) to keep sensitive PII out of transit streams.
  • Export to Kafka as the primary transport and to local buffering if Kafka is temporarily unavailable.

Practical tips:

  • Standardize attribute names using a mapping table in the collector. E.g., map "user.idp.email" → "subject.email".
  • Use deterministic sampling for low-volume critical events (auth attempts) and rate-based sampling for high-volume metrics.
  • Sign or HMAC events at the collector when provenance is required for downstream trust.

Step 3 — Transport and durability with Kafka

Kafka provides durability, partitioning and retention controls that fit continuous authorization needs:

  • Use separate topics for high-priority real-time signals (auth, device posture) and lower-priority analytics streams.
  • Set retention short for real-time topics (e.g., hours to days) and longer for audit topics.
  • Use partition keys that align with consumption patterns: subject_id for per-user ordering, resource_id for resource-centric policies.

Deploy Kafka with multi-AZ replication, enforce TLS and mutual TLS for client connections, and limit topic ACLs so only authorized processors can read or write.

Step 4 — Enrich and normalize in streaming processors

Before policy evaluation, events often need enrichment:

  • Map identity attributes via lookups to the IdP or HR systems (SCIM queries).
  • Enrich device data with MDM status and recent EDR findings.
  • Aggregate recent signals into session-level context (e.g., last 5 minutes of failed auths).

Use stream processors (Kafka Streams, ksqlDB, Flink) to implement these enrichments close to the transport layer. Enrichment reduces latency for policy evaluation and provides consistent attributes to the PDP.

Step 5 — Policy evaluation: OPA and decision caching

Open Policy Agent (OPA) running as a scalable PDP evaluates enriched events against Rego policies. Deployment patterns:

  • Sidecar PDPs for ultra-low-latency decisions within service mesh or DEPs.
  • Centralized PDP farm for complex policies and correlation (with a caching layer at the DEP).

Design considerations:

  • Keep Rego policies modular: identity, device posture, network constraints, risk thresholds.
  • Use a fast attribute lookup service or local caches to avoid synchronous calls during PDP evaluation.
  • Employ decision caching at the DEP with short TTLs (e.g., 30–120s) for similar requests to satisfy latency SLAs while still enabling frequent re-evaluation.

Example of a minimal Rego rule for device posture:

package ztna.policy

default allow = false

allow {
  input.action == "connect"
  input.device.posture == "compliant"
  input.identity.role == "engineer"
  input.resource.risk_level  5
}

In production, combine this with risk-score thresholds and interactive responses (step-up MFA, limited access) rather than binary allow/deny.

Step 6 — Enforcement points and latency tradeoffs

DEPs vary: API gateways, service mesh proxies (e.g., Envoy), cloud-native ingress, or on-prem NAC. Patterns:

  • For sub-100ms decisions, use sidecar PDPs or client-side policy enforcement with local caches.
  • For complex correlation, use asynchronous signals: allow initially, score continuously, and revoke when risk thresholds cross.
  • Design enforcement to support graceful degradation: default-deny for high-risk flows, default-allow with monitoring for low-risk flows where availability is critical.

Step 7 — Storage, analytics and feedback loops

Ship enriched events and decisions to a SIEM and an analytics store (e.g., Elasticsearch, Splunk, or cloud data warehouse) for:

  • Historical analysis and forensics.
  • Model training for ML-based risk scoring.
  • Audit trails to meet compliance needs (retain identity + decision + reason).

Feedback loops are crucial: alerts and analyst judgments should feed back into enrichment services and risk scoring to improve future decisions.

Security, privacy and governance

Design with provenance and least privilege in mind:

  • Sign or HMAC critical events at source to prevent tampering.
  • Encrypt data in transit and at rest. Apply field-level encryption or tokenization for sensitive PII before wider distribution.
  • Implement RBAC/ABAC for all pipeline components; separate duties between streaming, decisioning, and storage teams.
  • Document retention and access policies to satisfy GDPR, CCPA, or sector-specific rules.

Operational considerations and SLAs

Key operational KPIs:

  • Decision latency (p95): target under 100ms for synchronous use cases.
  • Event delivery lag (p99): target under 2s for real-time topics.
  • Pipeline availability: plan for multi-region redundancy with automatic failover.
  • Data integrity: monitor event loss via sequence numbers and signed heartbeats.

Create runbooks for incidents: Kafka broker loss, OPA overload, enrichment service timeouts. Practice chaos testing for PDP and transport failures to validate fail-open vs fail-closed behavior.

Practical rollout plan (90-day roadmap)

  1. Weeks 1–2: Inventory signals, define schema, prioritize 3–5 critical use cases (e.g., remote admin access, DB access, vendor sessions).
  2. Weeks 3–6: Stand up OTel collectors and Kafka test cluster; instrument a pilot application and one DEP.
  3. Weeks 7–10: Implement enrichment processors and a simple OPA policy; validate latency and decision correctness.
  4. Weeks 11–12: Integrate SIEM, enable auditing, and run a phased rollout to high-priority services.
  5. Post-90 days: Expand sources, harden security controls, and tune policies with observed telemetry.

Example: fintech use-case (concrete decisions)

A mid-sized fintech in 2026 used this approach to protect database access for customer data. Signals included IdP session events, MDM posture, recent EDR alerts, and connection origin. They routed these via OTel to Kafka, enriched with HR group membership and recent failed login counts, and evaluated policies in OPA sidecars. Result: automated session termination within 30s of an EDR high-confidence detection and a 70% reduction in manual investigations due to richer audit trails.

Checklist before you go live

  • Defined event schema and mapping tables.
  • Instrumented critical sources with OTel collectors.
  • Secured Kafka with mTLS and ACLs; configured topic retention.
  • Implemented enrichment processors and tested IdP/SCIM lookups.
  • Deployed OPA PDP with policies and caching; validated latency.
  • Connected SIEM and set retention/audit controls.
  • Prepared runbooks and incident playbooks.

Further reading and tools

  • OpenTelemetry project documentation — for collection and exporter patterns.
  • Apache Kafka documentation — topics, partitions, retention and security.
  • Open Policy Agent (OPA) & Rego — policy authoring best practices.
  • NIST SP 800-207 (Zero Trust Architecture) — conceptual guidance aligned with continuous authorization.

Conclusion: Building a telemetry-first pipeline is a practical, high-impact way to operationalize continuous authorization in a zero‑trust architecture. By standardizing signals, using durable transport, enriching data close to the source, and evaluating policies with performant PDPs, organizations can shift from periodic checks to dynamic, auditable, and scalable access decisions. Start small, instrument deliberately, and iterate policies with data-driven feedback.