Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Multi-version vLLM support: the compatibility shim

How the simulator builds against more than one vLLM line from a single main branch. Read versioning.md first for the strategy (build matrix, one image per line, compat.toml as source of truth) and conformance.md for capture/replay. This doc covers the code side: how we absorb the protocol crate's API drift.

Contents

The shape of the problem

The wire protocol comes from one git dependency, vllm-engine-core-client, which lives in the vLLM repo (rust/src/engine-core-client/). Its API drifts across releases. Cargo can hold only one rev of a git dep per build, so each line is a separate build (the matrix). The job of the shim is to let the same source compile against each line's crate, isolating the divergences in one place.

Per-line builds: pin, don't patch

Cargo rejects a [patch] that redirects a git dependency to a different rev of the same source ("patches must point to different sources"). So the per-line rev is swapped in [workspace.dependencies], not via --config patch. build.rs cannot do it either: dependency resolution happens before any build script runs.

cargo xtask pin-vllm <line> reads compat.toml and rewrites Cargo.toml: it sets the vllm-engine-core-client rev to the line's protocol_rev, and inserts, rewrites, or removes the fork [patch] to match the line's patch_repo/patch_rev (a fork is a different source, so it is allowed to [patch]). The committed Cargo.toml carries no [patch] block (the default line builds upstream), so a forked line's block is inserted, not rewritten; the script strips any existing block first, so it's idempotent. After the rewrite the rev no longer matches Cargo.lock, so per-line builds omit --locked.

Gotcha: the script changes the manifest but not the environment, and build.rs reads the line from VLLM_TARGET_VERSION (falling back to the compat.toml default). So a per-line build must set both: run the script and export VLLM_TARGET_VERSION=<tag>. The CI matrix does both; a local older-line build must too:

cargo xtask pin-vllm 0.27
VLLM_TARGET_VERSION=v0.27.1 cargo build --workspace   # no --locked

Capability cfgs

Where the crate's API genuinely diverges in a way owning a type can't hide (a field whose type differs per line), the engine gates on a discrete capability, not a version number. build.rs maps the target line to cfgs and declares them with cargo::rustc-check-cfg:

  • vllm_engine_id_u16 — EngineId::from_engine_index narrowed its parameter from u32 to u16 in 0.28 (the wire encoding was always two-byte little-endian). Gates sim_protocol::vllm::engine_id_from_index. On 0.28+.

Retired cfgs: vllm_outputs_enum (the 0.25 protocol restructure: types moved into protocol::{request, output, sampling} and EngineCoreOutputs became the classified enum over a private flat struct) went when 0.24 left the window, and vllm_cache_creation_tokens (PrefillStats.num_cache_creation_tokens, added in 0.26) went when 0.25 did. Every supported line shares both, so the outputs shim is unconditional and the field is assigned directly.

The capability list itself lives in sim_compat::capabilities, not in a build script: cfgs from build.rs only reach the crate that owns it, so both the root crate and sim-protocol (which owns the outputs shim) run their own build script and call the same capabilities::emit. A crate that grows cfg-gated code needs a build.rs of its own.

rust-analyzer does not run build scripts against the pinned line, so it shows false-positive errors on whichever branch is inactive; cargo build is the truth.

What the shim owns

The principle: own a tolerant decode wherever possible (no cfg), and reach for a capability cfg only when a field's type differs per line. Owned types deserialize the same wire on every line (serde ignores unknown fields).

ConcernDivergenceShim
Handshake harness typesmock_engine module absent before 0.23 (we never used its behavior, only structs)sim-protocol::mock_engine owns MockEngineSockets/MockEngineDataSockets/MockCoordinatorSockets + DEFAULT_MOCK_MAX_MODEL_LEN + default_dtype()
Request-type frameEngineCoreRequestType::from_frame is head-onlysim-protocol::wire::request_type_from_frame (1-byte decode)
Lora requestwire form is a positional array whose trailing fields differ per lineLoraSpec{lora_int_id,lora_name} (own, reads positions 0/1) for the add_lora call + registry
Protocol type pathsnone since 0.24 rolled off; every line uses protocol::{request,output,sampling}sim-protocol::vllm re-exports the whole surface; every crate imports from there, never from vllm_engine_core_client::protocol directly, so the next path move lands in one file
Output envelopethe classified enum serializes through a private flat struct, so it cannot be built inlinesim-protocol::vllm::Envelope alias + request_batch()/utility() constructors and request_outputs()/scheduler_stats()/utility_output() accessors
Engine identityEngineId::from_engine_index takes u32 (0.27) vs u16 (0.28+)sim-protocol::vllm::engine_id_from_index(u32), vllm_engine_id_u16-gated; validates the two-byte wire range on every line
Ready response (decode)EngineCoreReadyResponse.vllm_version absent before 0.23tap decodes its own tolerant CapturedReadyInfo{vllm_version:Option<String>}
Ready response (emit)0.27 frontends require the parallel-config sizes, scheduler caps, instance_id, and kv_events_config; 0.29 frontends require supports_lora and max_loras (and reject supports_lora != (max_loras > 0))sim-owned map-encoded SimReadyResponse superset: every field any line requires is always emitted, older frontends skip unknown keys, no cfg
Output decodedecode_engine_core_outputs takes any AsRef<[u8]> (<=0.28) vs &[Bytes] (0.29+)the tap passes the Bytes frames zeromq already hands it, which satisfy both signatures, no cfg
Lora requestLoraRequest::new returned Self (<=0.28), Result (0.29), and is gone on maintests build the struct literal; the fields are stable across the window
Utility requestEngineCoreUtilityRequest derives Deserialize only on 0.23+ (crate was client-only)engine_core::UtilityRequestSpec (Deserialize_tuple, matches the wire tuple)

The wire types still come from the crate (the matrix's whole point: catch drift at compile time). The shim only covers the spots where our decoding/server role needs something the client-oriented crate lacks on an older line.

The first three rows predate the current window: the lines that forced them (0.22, 0.23) have rolled off. They stay because owning them is still the right call for a server-side role — a tolerant decode we control cannot be broken by an upstream field addition — not because any supported line requires them.

Testing across lines

The matrix runs per line (see ci.yml):

  • cargo build --workspace — the "does the wire still compile" gate.
  • cargo test --workspace --lib — unit tests (compile + pass on every line).
  • cargo test --test conformance — the conformance runner (skips until goldens).

The full-stack e2e integration tests (tests/engine_core_e2e.rs, tests/tap_e2e.rs) drive the real EngineCoreClient, whose API is incomplete on older lines, so they target the default line via the build-and-test job, not each matrix leg.

The shim's own contract (that an Envelope built by the constructors survives a msgpack round trip) is unit-tested in sim-protocol::vllm, so it runs on every matrix leg — that is what proves the inactive cfg branch is real and not just compiling.

Current window

The window is N, N-1, N-2: three stable lines, plus the two trackers riding ahead of them. No line in the window needs a [patch] fork; every one builds against upstream, so the repo has no external-fork dependency.

  • nightly (nightly): tracks vLLM main, protocol_rev is the latest post-merge commit (bumped regularly). build.rs treats the non-vX.Y tag as the newest line, so all capability cfgs are on. It exists to catch wire drift before a release lands: the live-HEAD nightly canary pins to upstream main, builds, runs unit tests, runs the HEAD-client protocol e2e suite, and runs the conformance runner.
  • rc (v0.28.0rc2): the newest release candidate ahead of the stable window, bumped (tag + rev) by the release watcher. Never default, never fidelity-validated.
  • 0.29 (v0.29.0, default): builds against upstream 98dff2a8. Grew the handshake again: supports_lora and max_loras are required (absorbed by the SimReadyResponse superset), decode_engine_core_outputs takes &[Bytes], and LoraRequest::new became fallible.
  • 0.28 (v0.28.0): builds against upstream 2cf0a691. Narrowed EngineId::from_engine_index to u16 (vllm_engine_id_u16); everything else it added is a trailing #[serde(default)] field.
  • 0.27 (v0.27.1): builds against upstream 6e448d0e. The oldest line and the only one without vllm_engine_id_u16, so it is what keeps that branch honest.

0.27 hard-gates CI: it has its three goldens (two prefix-cached multiturn seeds plus one nocache multiturn, Qwen3-8B on H200), captured against that line's released engine image and replaying byte-identically. 0.28 and 0.29 entered the window without goldens and run non-gating until the Golden Capture workflow registers theirs. See conformance.md.

Fork patches

No line in the window needs one today, but the mechanism stays: a line can set patch_repo/patch_rev in compat.toml to [patch]-override the crate with a fork carrying a fix that isn't upstream yet. A fork is a different source, so Cargo allows it where a same-source rev patch is rejected. cargo xtask pin-vllm inserts, rewrites, or removes the block per leg; the committed Cargo.toml has none, since the default line builds upstream.

The last use was the 0.22/0.21 serde-defaults fork (vllm-project/vllm#45848 backported for the tap's capture-time decode). Both lines left the window in the 0.26 roll, so the external wseaton/vllm dependency is gone with them.

Follow-ups

None currently.