Securing Multicast Networks in ISP Environments: A Forgotten Attack Surface

Abstract

Internet Service Providers (ISPs) increasingly leverage IP multicast for efficient content delivery of streaming media, IoT data, and financial information, yet the security implications remain largely unaddressed in mainstream literature. Multicast protocols inherently introduce unique attack vectors including rogue rendezvous point attacks, source impersonation, and control plane flooding that threaten network infrastructure integrity. This paper identifies critical security gaps in current multicast deployments through a comprehensive analysis of protocol vulnerabilities and operational practices. We propose M-SENTINEL, an innovative security framework incorporating protocol-level segmentation with dynamic fallback logic to isolate trust domains and maintain service continuity during attacks. Our implementation features hierarchical key distribution and behavioral-based anomaly detection with mathematical formulations for threat quantification. Simulation results demonstrate 94.8% attack detection rates with under 2.1% false positives while maintaining sub-50ms fallback activation during security incidents. This work establishes a foundational security methodology for multicast infrastructures that have long represented a neglected attack surface in ISP environments, with specific contributions in trust segmentation, attack resilience, and operational continuity mechanisms.