Ensuring Security in Modern Data Pipelines: Practical Strategies for Data Engineers
Keywords:
Strategic processes, security, compliance, contemporary data pipelines, construction concerns, security practices, obligatory structures, vulnerabilities, ingestion, transformation, storage levels, interprets encryption, access-control, secret-management technologies, enhancing data protection, continuous monitoring, data lineage tracking, regulatory compliance, GDPR, CCPA, effective security measures, pipeline security, operational effectiveness, compliance with regulations.Abstract
In this research study, the authors explore strategic processes that can be used to maintain security and compliance in contemporary data pipelines. It explores construction concerns, most important security practices, and obligatory structures that can alleviate the vulnerabilities across ingestion, transformation, and storage levels. The research interprets encryption, access-control and secret-management technologies as the ways of enhancing protection of data. It also highlights the fact that continuous monitoring and data lineage tracking along with regulatory compliance must be tracked according to GDPR and CCPA. The results highlight the need to establish effective security measures and end with effective recommendations that widen pipeline security, operational effectiveness, and compliance with regulations.
DOI: https://doi.org/10.17762/ijisae.v12i22s.8090
Downloads
References
Prasad, V.K., Bhattacharya, P., Maru, D., Tanwar, S., Verma, A., Singh, A., Tiwari, A.K., Sharma, R., Alkhayyat, A., Țurcanu, F.E. and Raboaca, M.S., 2022. Federated learning for the internet-of-medical-things: A survey. Mathematics, 11(1), p.151.
Khan, A.Q., Nikolov, N., Matskin, M., Prodan, R., Roman, D., Sahin, B., Bussler, C. and Soylu, A., 2023. Smart data placement using storage-as-a-service model for big data pipelines. Sensors, 23(2), p.564.
Gudavalli, S., Mokkapati, C., Chinta, U., Singh, N.I.H.A.R.I.K.A., Goel, O. and Ayyagari, A.R.A.V.I.N.D., 2021. Sustainable Data Engineering Practices for Cloud Migration. Iconic Research and Engineering Journals (IREJ), 5(5), pp.269-287.
Zeydan, E. and Mangues-Bafalluy, J., 2022. Recent advances in data engineering for networking. Ieee Access, 10, pp.34449-34496.
Aslan, Ö., Aktuğ, S.S., Ozkan-Okay, M., Yilmaz, A.A. and Akin, E., 2023. A comprehensive review of cyber security vulnerabilities, threats, attacks, and solutions. Electronics, 12(6), p.1333.
Raptis, T.P., Cicconetti, C., Falelakis, M., Kalogiannis, G., Kanellos, T. and Lobo, T.P., 2023. Engineering resource-efficient data management for smart cities with Apache Kafka. Future Internet, 15(2), p.43.
Mubeen, M., Arslan, M. and Anandhi, G., 2022. Strategies to avoid illegal data access. Journal of Communication Engineering & Systems, 12(3), pp.29-40.
Devaki, K. and Leena Jenifer, L., 2022. A study on challenges in data security during data transformation. In Computer networks, big data and IoT: proceedings of ICCBI 2021 (pp. 49-66). Singapore: Springer Nature Singapore.
Pandian, R.S.R. and Columbus, C., 2022. An analytical approach for optimal secured data storage on cloud server for online education platform. Geoscientific Instrumentation, Methods and Data Systems Discussions, 2022, pp.1-36.
Bharati, S. and Podder, P., 2022. Machine and deep learning for iot security and privacy: applications, challenges, and future directions. Security and communication networks, 2022(1), p.8951961.
Chadwick, D.W., Fan, W., Costantino, G., De Lemos, R., Di Cerbo, F., Herwono, I., Manea, M., Mori, P., Sajjad, A. and Wang, X.S., 2020. A cloud-edge based data security architecture for sharing and analysing cyber threat information. Future generation computer systems, 102, pp.710-722.
Gudepu, B.K. and Jaladi, D.S., 2022. Data Discovery and Security: Protecting Sensitive Information. International Journal of Acta Informatica, 1(1), pp.176-187.
Owobu, W.O., Abieba, O.A., Gbenle, P., Onoja, J.P., Daraojimba, A.I., Adepoju, A.H. and Ubamadu, B.C., 2021. Review of enterprise communication security architectures for improving confidentiality, integrity, and availability in digital workflows. IRE Journals, 5(5), pp.370-372.
Brickley, J.C. and Thakur, K., 2021. Policy of least privilege and segregation of duties, their deployment, application, & effectiveness. Int J Cyber Secur Digit Forens, 10(4), pp.112-119.
Kousalya, A. and Baik, N.K., 2023. Enhance cloud security and effectiveness using improved RSA-based RBAC with XACML technique. International Journal of Intelligent Networks, 4, pp.62-67.
Fotache, M., Munteanu, A., Strîmbei, C. and Hrubaru, I., 2023. Framework for the assessment of data masking performance penalties in SQL database servers. Case Study: Oracle. IEEE Access, 11, pp.18520-18541.
Nadipalli, R., 2022. Data Integrity in MySQL-Driven Cloud Systems Using DevSecOps Pipelines. Journal of Scientific and Engineering Research, 9(1), pp.225-232.
Omotunde, H. and Ahmed, M., 2023. A comprehensive review of security measures in database systems: Assessing authentication, access control, and beyond. Mesopotamian Journal of CyberSecurity, 2023, pp.115-133.
James, U.U., 2022. Machine learning-driven anomaly detection for supply chain integrity in 5G industrial automation systems. International Journal of Scientific Research in Science, Engineering and Technology, 9(2), pp.2017-2023.
Awaysheh, F.M., Aladwan, M.N., Alazab, M., Alawadi, S., Cabaleiro, J.C. and Pena, T.F., 2021. Security by design for big data frameworks over cloud computing. IEEE Transactions on Engineering Management, 69(6), pp.3676-3693.
Alharbi, A., 2023. Applying Access Control Enabled Blockchain (ACE-BC) Framework to Manage Data Security in the CIS System. Sensors, 23(6), p.3020.
Sadaf, M., Iqbal, Z., Javed, A.R., Saba, I., Krichen, M., Majeed, S. and Raza, A., 2023. Connected and automated vehicles: Infrastructure, applications, security, critical challenges, and future aspects. Technologies, 11(5), p.117.
Ardagna, C.A., Bellandi, V., Damiani, E., Bezzi, M. and Hebert, C., 2021. Big Data Analytics-as-a-Service: Bridging the gap between security experts and data scientists. Computers & Electrical Engineering, 93, p.107215.
Majeed, A. and Lee, S., 2021. Applications of machine learning and high-performance computing in the era of COVID-19. Applied System Innovation, 4(3), p.40.
Almufarreh, A. and Arshad, M., 2023. Promising emerging technologies for teaching and learning: Recent developments and future challenges. Sustainability, 15(8), p.6917.
Al-Amri, R., Murugesan, R.K., Man, M., Abdulateef, A.F., Al-Sharafi, M.A. and Alkahtani, A.A., 2021. A review of machine learning and deep learning techniques for anomaly detection in IoT data. Applied Sciences, 11(12), p.5320.
Chadwick, D.W., Fan, W., Costantino, G., De Lemos, R., Di Cerbo, F., Herwono, I., Manea, M., Mori, P., Sajjad, A. and Wang, X.S., 2020. A cloud-edge based data security architecture for sharing and analysing cyber threat information. Future generation computer systems, 102, pp.710-722.
Alghofaili, Y., Albattah, A., Alrajeh, N., Rassam, M.A. and Al-Rimy, B.A.S., 2021. Secure cloud infrastructure: A survey on issues, current solutions, and open challenges. Applied Sciences, 11(19), p.9005.
Nath, S. and Arrawatia, R., 2022. Trade-offs or synergies? Hybridity and sustainable performance of dairy cooperatives in India. World Development, 154, p.105862.
Al-Jumaili, A.H.A., Muniyandi, R.C., Hasan, M.K., Paw, J.K.S. and Singh, M.J., 2023. Big data analytics using cloud computing based frameworks for power management systems: Status, constraints, and future recommendations. Sensors, 23(6), p.2952.
Yevseiev, S., Milov, O., Pribyliev, Y., Zviertseva, N., Lezik, A., Komisarenko, O., Nalyvaiko, A., Pogorelov, V., Katsalap, V. and Husarova, I., 2023. Development of the concept for determining the level of critical business processes security. Eastern-European Journal of Enterprise Technologies, 1(9), p.121.
Pollini, A., Callari, T.C., Tedeschi, A., Ruscio, D., Save, L., Chiarugi, F. and Guerri, D., 2022. Leveraging human factors in cybersecurity: an integrated methodological approach. Cognition, Technology & Work, 24(2), pp.371-390.
Ogeawuchi, J.C., Akpe, O.E., Abayomi, A.A., Agboola, O.A., Ogbuefi, E.J.I.E.L.O. and Owoade, S.A.M.U.E.L., 2022. Systematic review of advanced data governance strategies for securing cloud-based data warehouses and pipelines. Iconic Research and Engineering Journals, 6(1), pp.784-794.
Hewage, U.H.W.A., Sinha, R. and Naeem, M.A., 2023. Privacy-preserving data (stream) mining techniques and their impact on data mining accuracy: a systematic literature review. Artificial Intelligence Review, 56(9), pp.10427-10464.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
