Unified Data Lakehouse Architecture for Real-Time and Batch Data Integration in Multi-Cloud Environments
Keywords:
: Batch Processing, Data Governance, Data Lakehouse, Data Quality, Multi-Cloud Architecture, Real-Time StreamingAbstract
Enterprise data ecosystems increasingly span transactional platforms, event streaming infrastructure, cloud object storage, and machine learning environments — often without a unifying architectural layer to govern how data flows between them. Fragmentation across these systems produces inconsistencies in data quality, lineage, and access control that erode analytical trust and complicate regulatory compliance. This article presents a unified data lakehouse architecture designed to integrate real-time and batch data processing within a governed, scalable, and cloud-agnostic analytical foundation. The framework combines open table format storage, metadata-driven governance, zone-based data organization, and a unified access abstraction to support diverse workload types across multi-cloud environments. Streaming pipeline patterns are focused on low-latency manipulation of events․ Batch patterns are used more commonly for reconciliation‚ historical analysis and regulatory reporting․ Data quality checks can be enforced at ingestion‚ transformation‚ and consumption‚ which shows the architecture to be valuable in regulated‚ high-throughput application spaces‚ such as healthcare and financial services․ The proposed framework may reduce architectural fragmentation, improve data trustworthiness, and establish a consistent governance model that scales with enterprise data complexity.
Downloads
References
M. Armbrust, A. Ghodsi, R. Xin, and M. Zaharia, "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics," in Proc. 11th Conf. on Innovative Data Systems Research (CIDR), 2021. Available: http://cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
M. Armbrust et al., "Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores," Proc. VLDB Endow., vol. 13, no. 12, pp. 3411–3424, 2020. Available: https://dl.acm.org/doi/10.14778/3415478.3415560
S. Nadal et al., "Operationalizing and Automating Data Governance," J. Big Data, vol. 9, 2022. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9736715/
Dražen Oreščanin and Tomislav Hlupić, "Data Lakehouse — A Novel Step in Analytics Architecture," in 2021 44th International Convention on Information, Communication and Electronic Technology (MIPRO), 2021. Available: https://ieeexplore.ieee.org/document/9597091/
S. A. Errami, H. Hajji, K. A. El Kadi, and H. Badir, "Spatial Big Data Architecture: From Data Warehouses and Data Lakes to the LakeHouse," J. Parallel Distrib. Comput., vol. 176, pp. 70–79, 2023. Available: https://doi.org/10.1016/j.jpdc.2023.02.007
T. Hlupić; D. Oreščanin; D. Ružak; M. Baranović, "An Overview of Current Data Lake Architecture Models," in 2022 45th Jubilee International Convention on Information, Communication and Electronic Technology (MIPRO), 2022. Available: https://ieeexplore.ieee.org/document/9803717/
Ahmed A. Harby, Farhana Zulkernine a, "Data Lakehouse: A Survey and Experimental Study," Inf. Syst., vol. 127, 2025. Available: https://doi.org/10.1016/j.is.2024.102460
D. Durner, V. Leis, and T. Neumann, "Exploiting Cloud Object Storage for High-Performance Analytics," Proc. VLDB Endow., vol. 16, no. 11, pp. 2769–2782, 2023. Available: https://dl.acm.org/doi/10.14778/3611479.3611486
Y. Zhao et al., "A Zone-Based Data Lake Architecture for IoT, Small and Big Data," in IDEAS '21: Proceedings of the 25th International Database Engineering & Applications Symposium, 2021. Available: https://dl.acm.org/doi/10.1145/3472163.3472185
A. Boufassil et al., "Data Catalog: Approaches, Trends, and Future Directions," in 2023 17th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), 2024. Available: https://ieeexplore.ieee.org/document/10472799/
R. Gupta et al., "Secured and Privacy-Preserving Multi-Authority Access Control System for Cloud-Based Healthcare Data Sharing," Sensors, vol. 23, no. 5, 2023. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10007450/
T. K. H. Le et al., "Enhancing Healthcare Interoperability with FHIR: A Systematic Approach to Online Data Management," in ICISE '24: Proceedings of the 2024 9th International Conference on Information Systems Engineering, 2024. Available: https://dl.acm.org/doi/10.1145/3711954.3711958
D. Chrimes, "Big Data Analytics of Predicting Annual US Medicare Billing Claims with Health Services," in 2022 IEEE International Conference on Big Data (Big Data), 2023. Available: https://ieeexplore.ieee.org/document/10020524/
A. Kumar et al., "Integrating Big Data Analytics with Financial Risk Management: Challenges and Opportunities," in 2025 Seventh International Conference on Computational Intelligence and Communication Technologies (CCICT), 2025. Available: https://ieeexplore.ieee.org/abstract/document/11087991/
K. Peddireddy, "Streamlining Enterprise Data Processing, Reporting and Realtime Alerting using Apache Kafka," in 2023 11th International Symposium on Digital Forensics and Security (ISDFS), 2023. Available: https://ieeexplore.ieee.org/document/10131800
M. Zaharia et al., "Apache Spark: A Unified Engine for Big Data Processing," Communications of the ACM, vol. 59, no. 11, pp. 56–65, 2016. Available: https://dl.acm.org/doi/10.1145/2934664
L. Zineb and F. Rachid, "ETL Technologies for Big Data: A Comparative Study," in 2023 IEEE International Conference on Advances in Data-Driven Analytics And Intelligent Systems (ADACIS), 2024. Available: https://ieeexplore.ieee.org/document/10424341
C. Ma, X. Hu et al., "A Data Analysis Privacy Regulation Compliance Scheme for Lakehouse," in ADMIT '23: Proceedings of the 2023 2nd International Conference on Algorithms, Data Mining, and Information Technology, 2023. Available: https://dl.acm.org/doi/abs/10.1145/3625403.3625405
Vijay Kumar Butte and Sujata Butte, "Enterprise Data Strategy: A Decentralized Data Mesh Approach," 2022 International Conference on Data Analytics for Business and Industry (ICDABI), 2023. Available: https://ieeexplore.ieee.org/document/10041672/
E. Begoli, I. Goethert, and K. Knight, "A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks," in 2021 IEEE International Conference on Big Data (Big Data), 2021. Available: https://ieeexplore.ieee.org/document/9671534/
V. K. Pamula, "The Medallion Architecture in Practice: A Framework for Building Scalable and Governed Data Lakehouses on Microsoft Fabric," in 2026 IEEE 5th International Conference on AI in Cybersecurity (ICAIC), 2025. Available: https://ieeexplore.ieee.org/document/11395677/
D. Chowdhury and P. Kulkarni, "Application of Data Analytics in Risk Management of Fintech Companies," in 2023 International Conference on Innovative Data Communication Technologies and Application (ICIDCA), 2023. Available: https://ieeexplore.ieee.org/abstract/document/10099795/
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
