# MinIO MemKV RELEASE.2026-08-14T18-23-15Z Released: 2026-08-18 A vLLM spec that offloads KV blocks straight to a MemKV cluster, InfiniBand support, storage-tier reclaim under capacity pressure (on by default), a per-server transfer rate cap, and a plaintext credential in client logs. --- ## Downloads ### Server Binary | Platform | Architecture | Download | | -------- | ------------ | -------- | | Linux | amd64 | [memkv](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv) | | Linux | arm64 | [memkv](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv) | ### NIXL Plugin (for Dynamo / KVBM integrations) | Platform | Architecture | Download | | -------- | ------------ | -------- | | Linux | amd64 | [libplugin_MEMKV.so](https://dl.min.io/aistor/memkv/release/linux-amd64/libplugin_MEMKV.so) | | Linux | arm64 | [libplugin_MEMKV.so](https://dl.min.io/aistor/memkv/release/linux-arm64/libplugin_MEMKV.so) | ### LD_PRELOAD Shim (for MLPerf-Storage kvcache workloads) | Platform | Architecture | Download | | -------- | ------------ | -------- | | Linux | amd64 | [libmemkv_preload.so](https://dl.min.io/aistor/memkv/release/linux-amd64/libmemkv_preload.so) | | Linux | arm64 | [libmemkv_preload.so](https://dl.min.io/aistor/memkv/release/linux-arm64/libmemkv_preload.so) | ### Packages `.deb`, `.rpm`, and `.apk` packages bundle the server + both `.so` sidecars + the LMCache, sglang, and vLLM Python wheels into a single per-arch install. | Format | Architecture | Download | | ------ | ------------ | -------- | | DEB | amd64 | [memkv\_20260814182315.0.0_amd64.deb](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv_20260814182315.0.0_amd64.deb) | | DEB | arm64 | [memkv\_20260814182315.0.0_arm64.deb](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv_20260814182315.0.0_arm64.deb) | | RPM | amd64 | [memkv-20260814182315.0.0-1.x86_64.rpm](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv-20260814182315.0.0-1.x86_64.rpm) | | RPM | arm64 | [memkv-20260814182315.0.0-1.aarch64.rpm](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv-20260814182315.0.0-1.aarch64.rpm) | | APK | amd64 | [memkv\_20260814182315.0.0_x86_64.apk](https://dl.min.io/aistor/memkv/release/linux-amd64/memkv_20260814182315.0.0_x86_64.apk) | | APK | arm64 | [memkv\_20260814182315.0.0_aarch64.apk](https://dl.min.io/aistor/memkv/release/linux-arm64/memkv_20260814182315.0.0_aarch64.apk) | The NIXL plugin is auto-symlinked to `/opt/nvidia/nvda_nixl/lib/plugins/` when that directory exists (postinstall hook). ### Python Plugins Install from PyPI, for amd64 and arm64: ```bash pip install -U memkv-lmcache memkv-sglang # Python 3.8+ pip install -U memkv-vllm # Python 3.10+, needs vLLM 0.25.0+ ``` PyPI serves **1.0.7** for this release. The wheels `dlopen` the RDMA libraries at runtime — the transport needs host `rdma-core`, otherwise it falls back to TCP. Without network access, the deb/rpm installs the same code at `/usr/share/memkv/wheels/`. Those wheels are stamped 1.0.5, the version the tagged packages were built with, so pip would treat them as already installed over 1.0.5 from the previous release and do nothing — use `--force-reinstall`: ```bash cd /usr/share/memkv/wheels pip install --force-reinstall memkv_lmcache-*.whl memkv_sglang-*.whl pip install --force-reinstall memkv_vllm-*.whl ``` ### Container Image ```bash docker pull quay.io/minio/memkv:RELEASE.2026-08-14T18-23-15Z docker pull quay.io/minio/memkv:latest ``` The container ships the server and the NIXL plugin (under `/usr/local/lib/plugins/`). For the LD_PRELOAD shim and the Python wheels, use the deb/rpm. ### Verification Each binary is signed with both minisign (preferred) and GPG; sha256sums are published alongside. ```bash # minisign minisign -Vm memkv -P RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav # sha256 sha256sum -c memkv.sha256sum ``` --- ## Changes since RELEASE.2026-08-04T22-19-45Z ### New Features - **vLLM direct KV offload — `MemKVOffloadingSpec`.** A native vLLM offloading spec whose medium is a MemKV cluster: the worker is handed the live GPU KV tensors, packs each block into one value, and MemKV servers move it with one-sided RDMA. No vLLM CPU pool in front, no LMCache, no vLLM patch, no server change — config only: ``` --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both", "kv_connector_extra_config":{"spec_name":"MemKVOffloadingSpec", "spec_module_path":"memkv_vllm.spec"}}' ``` One decision to make: `"memkv_scratch_medium"` is `host` (default) or `gpu`. GPUDirect reaches line rate only where the NIC and that rank's GPU share a PCIe switch; on 8× H200 against two nodes over 2×400 Gb, restore ran 86.8 GB/s (89% of line rate) with `host` and 23.1 GB/s with `gpu` when most ranks lacked such a NIC. Set `gpu` on rail-optimised nodes, `host` everywhere else. Both media write the same values under the same keys. `memkv bench --vllm` profiles the path in inference terms and recommends a pool size. - **InfiniBand fabrics are now addressed correctly.** MemKV built every RDMA address vector for RoCE and never inspected the port's link layer, so on a native IB fabric every transfer failed with `DC READ timeout after 1s` — a timeout, not an error, which read like a fabric fault. Address vectors now branch on the link layer: InfiniBand peers are addressed by LID, RoCE keeps the existing global-route path byte for byte. An IB port with no LID is a startup error naming the subnet manager instead of a silent timeout later, and a LID the subnet manager reassigns is now noticed via the device's async event queue. No wire-format change; RoCE deployments see identical bytes and take an identical code path. See Known Limitations. - **Storage-tier reclaim on capacity pressure, on by default.** The NVMe tier only ever freed a slot on an explicit client `DELETE`, which works for LMCache but not for SGLang HiCache — its storage interface has no per-key delete hook, so a HiCache deployment filled the drives and then failed every write. Reclaim now runs on watermarks, LRU over the index first and FIFO by block id for blocks a restart left unindexed: ```yaml storage: eviction: enabled: true high_watermark: 70 # start reclaiming at 70% slot fill low_watermark: 60 # drain back to 60% ``` This is a default-on change in retention: blocks that used to persist indefinitely are now reclaimed once the tier crosses the high watermark. Set `enabled: false` to keep the old behavior. Reads are gated against concurrent reclaim, so a block freed mid-read is refused rather than served from a slot that has since been reused. - **Per-server transfer rate cap.** `network.throttle` puts an aggregate byte-rate ceiling on bulk transfers so a rail shared with training or storage traffic can keep headroom: ```yaml network: throttle: enabled: true bits_per_sec: 200 Gbps # about half of one 400GbE rail burst_window_ms: 100 # unused rate banked as burst credit; 0 paces strictly ``` Because the server initiates every bulk transfer under MemKV's direct-I/O pattern, one server-side limiter covers RDMA as well as TCP — including the paths the NIXL, LMCache, and SGLang integrations drive. Off by default. `memkv_throttle_bytes_total`, `memkv_throttle_delay_seconds_total`, and `memkv_throttle_waits_total` separate a throttled server from a slow drive. Per server, not cluster-wide: N servers admit N× the cap. Control messages and the admin endpoint are never throttled. - **Metrics history over the admin API.** The server now samples itself once a second into three ring buffers — 1m at 1s, 1h at 60s, 24h at 900s — covering throughput, latency, memory, storage, RDMA, eviction, and cache hit rate, and serves them at `GET /v1/metrics/history`. History persists across restarts to `--metrics-history-path` (default `memkv-metrics-history.bin`; `none` keeps it in memory for the life of the process). ### Improvements - **Batch transfer failures now name their cause.** The batched client API returned a bare `Vec`, so a cold cache, a dead rail, an MR registration failure, a truncated reply, and a server-side storage error were all the same `false` — a vLLM worker could report "3/8 block writes failed" and nothing that distinguishes a hardware problem from a cache that was never warm. It now returns `Ok`, `Miss`, `Transport(cause)`, or `Server(cause)`, propagating the per-entry cause the server already sends. The vLLM worker drops its 50 ms retry with it: a transport error will not have cleared by the time the retry reaches the same rail, and a miss means the block was evicted, so both ended in recompute anyway — now they end there having logged which one it was. ### Bug Fixes - **The write-path saturation guard was dead on every NVMe deployment.** The JBOF manager implemented neither capacity probe the guard reads, so `storage_saturated()` short-circuited to false everywhere. Drives filled to 100% and writes then failed with an untyped error instead of the typed out-of-space signal the client's TCP-streaming fallback keys off, so the fallback never engaged. Both probes are now implemented from the freelist counters; memory-mode deployments pick up the same probe. - **vLLM plugin works with vLLM 0.26 and later.** vLLM commit `a9531edfa` ("Define clean backend configuration boundary"), between v0.25.1 and v0.26.0, renamed several offloading-spec fields the plugin reads directly and reworked spec construction. The plugin now detects and adapts to either side of that rename rather than pinning to one. Fixed in wheel **1.0.6** — a `pip install -U` is required, the 1.0.5 wheels on PyPI predate it. ### Security Updates - **The client no longer logs the auth key or the license.** `Engine::new` logged its whole config, and the config derived `Debug`, so every client startup wrote the HMAC shared key and the license JWT in plaintext to the first line of output — into each node's log file, into the logs of anything embedding the client, and into any captured console or CI output. Both now render as `Some()`; whether a key is configured stays visible, the value does not. The server was never affected. **Operator action:** this stops new leakage, it does not scrub logs already written. Rotate any key that was captured. ### Known Limitations - **The monitoring console UI is not bundled in this release.** The server side ships and `GET /v1/metrics/history` is live, but the release build did not produce the console bundle, so the admin server's `/` route serves a placeholder. Scrape the endpoint directly, or build the console into the binary from source. Fixed in the build for the next release. - **A chunked overwrite can still lose earlier chunks under capacity pressure.** Whole-value writes hold a per-entry gate that reclaim respects, but a chunked overwrite spans several calls, each gated individually. A gap of more than ~500 ms between chunks while the tier is above the high watermark can let reclaim free the chain in between. Closing it needs a per-key write lease across calls. - **InfiniBand addressing is same-subnet only.** Traffic crossing subnets needs a global route to the local router, which is refused rather than mis-addressed. When the subnet manager reassigns a LID, established sessions fail and reconnect; recovery is the reconnection, not transparent. Both the data path and the LID-reassignment path have been validated on InfiniBand hardware on Nebius; CI itself remains SoftRoCE. - **`MemKVOffloadingSpec` scope.** Single KV cache group; an offloaded block is one GPU block; keys are rank-qualified, so a restore requires the same TP topology that stored them; RDMA only, with no TCP fallback on this path. The transport is measured (94% of 2×400 Gb on the save direction at 4 rails); an end-to-end serving campaign has not been run against the corrected path, so there are no request-level figures to quote yet. --- ## Documentation - Hosted docs: - Embedded docs (in the binary): `memkv doc` serves the same site locally. ## Support - Security disclosures: security@min.io - Licensed customers and contributors: contact MinIO support.