Docker
Runs each execution as a container on the local Docker daemon, or as a Swarm service if the daemon is a Swarm manager.
- Driven by
- The Docker Engine API
- Container images
- Pulled, with fallbacks tried in order
Crankshaft is a headless Rust library for running tasks. Your engine builds the tasks, and Crankshaft submits, monitors, and cancels them on Docker, GA4GH TES, or any scheduler that can be run from a shell. It was developed at St. Jude for the bioinformatics execution engine, Sprocket, and is designed to manage tens to hundreds of thousands of concurrent tasks, but it can be used for any kind of work.
cargo add crankshaftRegister as many as you need, each under its own name, and pick one when you spawn.
Runs each execution as a container on the local Docker daemon, or as a Swarm service if the daemon is a Swarm manager.
Sends each task to a GA4GH Task Execution Service, such as Funnel, TESK, or a cloud provider's. The server runs the containers.
Runs tasks through submit, monitor, and kill commands you write. That's enough for LSF, Slurm, or PBS, locally or over SSH.
Crankshaft runs inside your Tokio runtime, and you decide when tasks are canceled. It handles scheduling, calls to the backends, and tracking each task.
max_tasks limit.CancellationToken you control.Needs tokio, tokio-util, nonempty, and anyhow. Getting started has the full setup.
use crankshaft::Engine;
use crankshaft::config::backend::{Config, Kind, docker};
use crankshaft::engine::Task;
use crankshaft::engine::task::Execution;
use nonempty::NonEmpty;
use tokio_util::sync::CancellationToken;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// 1. Register a backend under a name.
let backend = Config::builder()
.name("docker")
.kind(Kind::Docker(docker::Config::default()))
.max_tasks(50)
.build();
let engine = Engine::default().with(backend).await?;
// 2. Describe the work.
let task = Task::builder()
.name("hello")
.executions(NonEmpty::new(
Execution::builder()
.images(["alpine"])?
.program("echo")
.args([String::from("hello, world!")])
.build(),
))
.build();
// 3. Spawn it on the backend by name.
let handle = engine
.spawn("docker", task, CancellationToken::new())
.await?;
// 4. Wait for one result per execution.
let results = handle.wait().await?;
println!("{}", results.first().status);
engine.shutdown().await;
Ok(())
}Subscribe to the engine and each task announces itself as it moves: created, image pulled, started, and finished. The same stream feeds the gRPC monitor and the terminal console.
| Kind | Time | Event | Task | Backend | Detail |
|---|---|---|---|---|---|
| 09:41:34 | ImagePullFinished | polish-gizmo | docker | burnish:latest | |
| 09:41:27 | TaskCreated | count-cogs | lsf | ||
| 09:41:25 | ImagePullStarted | polish-gizmo | docker | burnish:latest | |
| 09:41:18 | ImagePullFailed | polish-gizmo | docker | burnish:3.1 | |
| 09:41:16 | TaskCreated | tune-kazoo | tes | tes_id task-7f3c | |
| 09:41:09 | ImagePullStarted | polish-gizmo | docker | burnish:3.1 | |
| 09:41:02 | TaskCreated | polish-gizmo | docker |
Want to run workflows, not build an engine? Sprocket is a WDL workflow engine built on Crankshaft. If you need something that works today, start there.
Go to Sprocket →