Skip to content

medea.md

Latest commit

 

History

History
28 lines (18 loc) · 1.46 KB

medea.md

File metadata and controls

28 lines (18 loc) · 1.46 KB
Raw

Medea: Scheduling of long running applications in shared production clusters

Metadata

Presented in EuroSys 2018.

Authors: Panagiotis Garefalakis, Konstantinos Karanasos, Peter Pietzuch, Arun Suresh, Sriram Rao (Imperial College London & Microsoft)

Understanding the paper

Methodology

  • Medea adopts a two-scheduler design:
    • For the placement of long-running applications (LRAs), it uses a dedicated scheduler.
    • For the placement of task-based applications, it directly uses a traditional scheduler.
  • It formulates the placement of LRAs with constraints as an integer linear program (ILP), and solves it as an online optimization problem.
  • It considers multiple LRA container requests at once to achieve higher-quality placements and global objectives (e.g., minimizing the violation of placement constraints, resource fragmentation, any load imbalance, or the number of machines used).
  • It also investigates heuristics that trade placement quality for lower scheduling latency:
    1. Tag popularity: prioritizes the placement of containers that have more constraints.
    2. Node candidates: prioritizes the placement of containers that have smaller nodes allowed to be placed.

Implementation

Built as an extension to Apache Hadoop YARN.

Evaluation

Evaluated on a 400-node pre-production cluster.