Workshops Archive

Learning to Schedule: A Supervised Learning Framework for Network-Aware Scheduling of Data-Intensive Workloads

Workshop: The 12th Annual International Workshop on Innovating the Network for Data-Intensive Science (INDIS)

Authors: Sankalpa Timilsina and Susmit Shannigrahi (Tennessee Technological University)

Abstract: Distributed cloud environments hosting data-intensive applications often experience slowdowns from network congestion, asymmetric bandwidth, and inter-node data shuffling. These factors are typically not captured by host-level metrics such as CPU or memory. Scheduling without considering them can cause poor placement, longer transfers, and degraded job performance. We present a network-aware scheduler that uses supervised learning to predict job completion time. Our system collects real-time telemetry from all nodes, applies a trained model to estimate job duration per node, and ranks them to select the best placement. We evaluate the scheduler on a geo-distributed Kubernetes cluster deployed on the FABRIC testbed using network-intensive Spark workloads. Compared to the default Kubernetes scheduler, which relies on current resource availability alone, our supervised scheduler achieved 34–54% higher accuracy in selecting optimal nodes. The novelty of our work lies in demonstrating supervised learning for real-time, network-aware scheduling on a multi-site cluster.

Back to The 12th Annual International Workshop on Innovating the Network for Data-Intensive Science (INDIS) Archive Listing Back to Full Workshop Archive Listing