• What is orchestration?

• What we have built?

• Plans

• Architecture

• Why not docker?

• Why not kubernetes/mesos/...?

• Why rust?

• Auto-Deployment

• Scheduling / HW Sharing

• Monitoring

• Self-Healing

• Minimum components

• Availability first

• Introspection

• Minimum configuration

• Upgrades

• Only One Orchestration

• Security

• Network Async and Lossy

• Leader Election

• Restart Process

• Scheduling in Failing Mode

• Minimum components

• Availability first

• Introspection

• Minimum configuration

• vagga

• lithos

• cantal

• verwalter

Containers for dev.env.

$ git clone git@host:proj
$ cd proj
$ vagga
Available commands:
  run   Start app with postgres and redis
  doc   Build docs with sphinx
  test  Run unit tests
  js    Compile javascripts

$ vagga run
Okay, server started at http://localhost:8080

$ vagga doc
Installing sphinx
...
Docs are built in doc/_build/html/index.html

$ git pull
$ vagga test
Requirements txt changed
Rebuilding container "test"
................
Ok 20 successful

• Self-documentation

• IDE folders with deps

• Versioning

• Continuous Integration

Production-grade containers

• Start process

• Set up environment

• Mount file systems

• Restart on failure

• Downloading

• Build

• Network API

Configuration files

introspection

security

• Sandbox config

• Read-only/nosuid/noexec

• Fixed command-line (*)

(*) You control only version and number of instances

• Upgrade w/o restart

• PID 1

Monitoring

Metrics

• System metrics

• CGroups

• Application metrics

• 2 sec scan interval

• 1 hour in-memory

• Web UI

• -> Carbon(graphite) -> Graphana

• Peer discovery

• Aggregated stats

3k-10k metrics in 10-40ms

1h in 15-35MiB

Debug build

Cluster-wide Scheduling

Metrics -> Config

• Inputs

• Leader-Election

• Monolithic Scheduler

• Render

• Web UI

• Metrics

• Peers

• UI Buttons

• Prev. Schedules

Raft-like protocol

No fsync

Low Traffic

Consistency

• Side-Effect Free

• Reproducible

• Transformer

Monolithic scheduler

JSON

Handlebars -> nginx.conf

nginx -t

nginx -s reload

10 Djangos → 5 Running, 77 Err/sec → 1 min → Blacklist

Met → Sch → Met → Sch

• Ship It

• Dokku-like thing

• HTTP Routing

• P2P File Sync

• Multi-DC