Implementing Zero-Trust Authentication and Authorization in Distributed Data Lake House Architectures

Abstract

Distributed data lake house platforms connect business intelligence tools, distributed query engines, metadata catalogs, orchestration frameworks, and cloud object storage into a single analytical surface that spans hybrid and multi-cloud infrastructure. This disaggregation removes the network perimeter that earlier access control models assumed, and it leaves authentication and authorization scattered across components that each enforce a different native model: identity providers issue tokens, query engines apply engine-level roles, metadata catalogs hold table-level ownership, and storage systems apply bucket-level policies. No existing zero-trust reference model maps these control points onto a single, continuously verified architecture for a disaggregated lake house stack. This paper proposes a reference architecture that anchors fine-grained authorization at the metadata catalog, where the platform first has full semantic visibility into which tables, columns, and rows a request may reach, while identity federation and workload identity manage authentication upstream and downstream of that decision point. The architecture is evaluated against zero-trust tenets and compared with engine-native and storage-native enforcement alternatives on the basis of granularity, cross-engine consistency, and auditability. The paper also formalizes the latency cost of continuous verification and a composite behavioral risk score for detecting mid-query trust decay, and it closes with a discussion of the approach's limitations and the conditions under which catalog-anchored enforcement is the stronger design choice.