The problem
Spot instances cost up to 70–90 % less than normal ones but can be reclaimed at short notice. Workflow schedulers either avoid them or lose work when an instance disappears. Choosing which tasks to place on spot machines, and when to checkpoint, decides the saving.
Who needs it
Data engineering, bioinformatics and media-processing teams that run large batch workflows in AWS, Azure or Alibaba Cloud.
Idea of the approach
- Use public spot-price histories and scientific workflow shapes (Montage, CyberShake from the Pegasus gallery).
- Model interruption risk from the price history.
- Extend a budget/deadline workflow scheduler (BUDA, BUCAS) with spot placement and checkpointing decisions.
- Evaluate in a simulator (WorkflowSim/CloudSim or a Python simulator).
What you will build and deliver
- Scheduler + simulator code and datasets.
- Cost and makespan results versus on-demand only.
- Paper.
Skills needed
Python or Java, algorithms, probability basics.
Research contribution & where to publish
Contribution: risk-aware spot placement inside a list-scheduling algorithm. Venues: ICA3PP, IEEE CLUSTER, Journal of Parallel and Distributed Computing, IEEE TCC.
Key references — start reading here
- S. Yi, D. Kondo, A. Andrzejak. Reducing Costs of Spot Instances via Checkpointing in the Amazon Elastic Compute Cloud. IEEE CLOUD, 2010.
- M. A. Rodriguez, R. Buyya. Deadline Based Resource Provisioning and Scheduling Algorithm for Scientific Workflows on Clouds. IEEE Transactions on Cloud Computing, 2014.
- H. Topcuoglu, S. Hariri, M.-Y. Wu. Performance-Effective and Low-Complexity Task Scheduling for Heterogeneous Computing. IEEE Transactions on Parallel and Distributed Systems, 2002.
- H. Djigal, L. Liu, J. Luo, J. Xu. BUDA: Budget and Deadline Aware Scheduling Algorithm for Task Graphs in Heterogeneous Systems. IEEE/ACM IWQoS, 2022.
Interested?
Send the topic title and the courses you have taken.
Contact Dr. Hamza DjigalOther topics in Edge and cloud scheduling
Budget- and deadline-aware autoscaling of Kubernetes microservices with reinforcement learning
Carbon-aware scheduling of data-pipeline workflows across cloud regions
Predicting serverless cold starts and pre-warming functions with machine learning
A digital twin of a multi-access edge computing network for testing task-offloading policies (open simulator and benchmark)