Skip to content

Task execution for workflow engines

Crankshaft is a headless Rust library for running tasks. Your engine builds the tasks, and Crankshaft submits, monitors, and cancels them on Docker, GA4GH TES, or any scheduler that can be run from a shell. It was developed at St. Jude for the bioinformatics execution engine, Sprocket, and is designed to manage tens to hundreds of thousands of concurrent tasks, but it can be used for any kind of work.

Get started
cargo add crankshaft

Part of the Sprocket project by St. Jude Rust Labs · API reference on docs.rs

1DockerLocal daemon or Swarm2TESGA4GH Task Execution Service3GenericLSF, Slurm, and othersShared core: Engine and Task.Each throw: a named backend.
Every backend accepts the same Task.

Backends

Register as many as you need, each under its own name, and pick one when you spawn.

Docker

Runs each execution as a container on the local Docker daemon, or as a Swarm service if the daemon is a Swarm manager.

Driven by
The Docker Engine API
Container images
Pulled, with fallbacks tried in order
Docker backend →

TES

Sends each task to a GA4GH Task Execution Service, such as Funnel, TESK, or a cloud provider's. The server runs the containers.

Driven by
HTTP, with basic or bearer auth
Container images
Passed to the TES server
TES backend →

Generic

Runs tasks through submit, monitor, and kill commands you write. That's enough for LSF, Slurm, or PBS, locally or over SSH.

Driven by
Your shell commands
Container images
Ignored
Generic backend →

Four steps from nothing to an exit code

Crankshaft runs inside your Tokio runtime, and you decide when tasks are canceled. It handles scheduling, calls to the backends, and tracking each task.

  1. Register a backendGive it a name, a kind, and a max_tasks limit.
  2. Describe a TaskOne or more executions, each with an image, a program, and arguments.
  3. Spawn it by nameWith a CancellationToken you control.
  4. Wait for resultsOne exit status per execution, or a typed error.

Needs tokio, tokio-util, nonempty, and anyhow. Getting started has the full setup.

rust
use crankshaft::Engine;
use crankshaft::config::backend::{Config, Kind, docker};
use crankshaft::engine::Task;
use crankshaft::engine::task::Execution;
use nonempty::NonEmpty;
use tokio_util::sync::CancellationToken;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Register a backend under a name.
    let backend = Config::builder()
        .name("docker")
        .kind(Kind::Docker(docker::Config::default()))
        .max_tasks(50)
        .build();
    let engine = Engine::default().with(backend).await?;

    // 2. Describe the work.
    let task = Task::builder()
        .name("hello")
        .executions(NonEmpty::new(
            Execution::builder()
                .images(["alpine"])?
                .program("echo")
                .args([String::from("hello, world!")])
                .build(),
        ))
        .build();

    // 3. Spawn it on the backend by name.
    let handle = engine
        .spawn("docker", task, CancellationToken::new())
        .await?;

    // 4. Wait for one result per execution.
    let results = handle.wait().await?;
    println!("{}", results.first().status);

    engine.shutdown().await;
    Ok(())
}

Every task reports in

Subscribe to the engine and each task announces itself as it moves: created, image pulled, started, and finished. The same stream feeds the gRPC monitor and the terminal console.

engine.subscribe()Example output
Example Crankshaft event stream, newest first
KindTimeEventTaskBackendDetail
09:41:34ImagePullFinishedpolish-gizmodockerburnish:latest
09:41:27TaskCreatedcount-cogslsf
09:41:25ImagePullStartedpolish-gizmodockerburnish:latest
09:41:18ImagePullFailedpolish-gizmodockerburnish:3.1
09:41:16TaskCreatedtune-kazootestes_id task-7f3c
09:41:09ImagePullStartedpolish-gizmodockerburnish:3.1
09:41:02TaskCreatedpolish-gizmodocker

Want to run workflows, not build an engine? Sprocket is a WDL workflow engine built on Crankshaft. If you need something that works today, start there.

Go to Sprocket →

Licensed MIT or Apache-2.0. Part of St. Jude Rust Labs.