← All capstone topics
Industry + research Edge and cloud scheduling

Carbon-aware scheduling of data-pipeline workflows across cloud regions

Run each step of a data pipeline at the time and in the cloud region where electricity is cleanest, without missing the deadline.

Level: Undergraduate Format: Individual project Duration: 1–2 semesters Supervisor: Dr. Hamza Djigal

The problem

The carbon intensity of electricity changes by hour and by region. Batch jobs such as nightly data pipelines or model training can often wait a few hours or run elsewhere. Most schedulers ignore this; companies now report their cloud emissions.

Who needs it

Companies with sustainability reporting obligations; cloud providers (Google, Microsoft) already shift flexible jobs.

Idea of the approach

  • Use public carbon-intensity data (e.g. Electricity Maps / national grid datasets) for several regions.
  • Model a pipeline as a task graph (DAG) with deadlines.
  • Extend a list-scheduling heuristic (HEFT/IPPTS) with a carbon cost and time windows; compare with carbon-agnostic scheduling.
  • Simulate with real traces; optionally run a small demo with Apache Airflow.

What you will build and deliver

  • The carbon-aware scheduler and a simulator.
  • Results: carbon saved vs delay added.
  • Paper.

Skills needed

Python, algorithms; data handling with pandas.

Research contribution & where to publish

Contribution: deadline-constrained, carbon-aware DAG scheduling across regions. Venues: ACM e-Energy, IEEE CLOUD, Sustainable Computing: Informatics and Systems, IEEE TSUSC.

Key references — start reading here

  1. A. Radovanović, R. Koningstein, I. Schneider, B. Chen, A. Duarte, B. Roy, et al.. Carbon-Aware Computing for Datacenters. IEEE Transactions on Power Systems, 2023.
  2. P. Wiesner, I. Behnke, D. Scheinert, K. Gontarska, L. Thamsen. Let's Wait Awhile: How Temporal Workload Shifting Can Reduce Carbon Emissions in the Cloud. ACM/IFIP Middleware, 2021.
  3. A. Souza, N. Bashir, J. Murillo, W. Hanafy, Q. Liang, D. Irwin, P. Shenoy. Ecovisor: A Virtual Energy System for Carbon-Efficient Applications. ACM ASPLOS, 2023.
  4. H. Djigal, J. Feng, J. Lu, J. Ge. IPPTS: An Efficient Algorithm for Scientific Workflow Scheduling in Heterogeneous Computing Systems. IEEE Transactions on Parallel and Distributed Systems, 2020.

Interested?

Send the topic title and the courses you have taken.

Contact Dr. Hamza Djigal

Other topics in Edge and cloud scheduling