# MinIO AIStor RELEASE.2026-05-04T23-02-27Z Released: 2026-05-05 This release adds a safeguard that blocks hot pool expansion while a rebalance is active, fixes an S3 Select output-buffer sharing bug that could crash the server under specific Select workloads, and lands a broad set of correctness fixes across replication, batch jobs, audit logging, configuration reload, Delta Sharing, and Kafka notifications. Several long-standing edge cases — including phantom delete markers on Spark legacy-optimize replication, lost batch key-rotation tags, and a goroutine leak in Kafka client retries — are also resolved. --- ## Downloads ### Binary Downloads | Platform | Architecture | Download | | -------- | ------------ | ------------------------------------------------------------------------------ | | Linux | amd64 | [minio](https://dl.min.io/aistor/minio/release/linux-amd64/minio) | | Linux | arm64 | [minio](https://dl.min.io/aistor/minio/release/linux-arm64/minio) | | macOS | arm64 | [minio](https://dl.min.io/aistor/minio/release/darwin-arm64/minio) | | macOS | amd64 | [minio](https://dl.min.io/aistor/minio/release/darwin-amd64/minio) | | Windows | amd64 | [minio.exe](https://dl.min.io/aistor/minio/release/windows-amd64/minio.exe) | ### FIPS Binaries | Platform | Architecture | Download | | -------- | ------------ | ------------------------------------------------------------------------------ | | Linux | amd64 | [minio.fips](https://dl.min.io/aistor/minio/release/linux-amd64/minio.fips) | | Linux | arm64 | [minio.fips](https://dl.min.io/aistor/minio/release/linux-arm64/minio.fips) | ### Package Downloads | Format | Architecture | Download | | ------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------- | | DEB | amd64 | [minio_20260504230227.0.0_amd64.deb](https://dl.min.io/aistor/minio/release/linux-amd64/minio_20260504230227.0.0_amd64.deb) | | DEB | arm64 | [minio_20260504230227.0.0_arm64.deb](https://dl.min.io/aistor/minio/release/linux-arm64/minio_20260504230227.0.0_arm64.deb) | | RPM | amd64 | [minio-20260504230227.0.0-1.x86_64.rpm](https://dl.min.io/aistor/minio/release/linux-amd64/minio-20260504230227.0.0-1.x86_64.rpm) | | RPM | arm64 | [minio-20260504230227.0.0-1.aarch64.rpm](https://dl.min.io/aistor/minio/release/linux-arm64/minio-20260504230227.0.0-1.aarch64.rpm) | ### Container Images ```bash # Standard docker pull quay.io/minio/aistor/minio:RELEASE.2026-05-04T23-02-27Z podman pull quay.io/minio/aistor/minio:RELEASE.2026-05-04T23-02-27Z # FIPS docker pull quay.io/minio/aistor/minio:RELEASE.2026-05-04T23-02-27Z.fips podman pull quay.io/minio/aistor/minio:RELEASE.2026-05-04T23-02-27Z.fips ``` ### Homebrew (macOS) ```bash brew install minio/aistor/minio ``` --- ## New Features - **Block hot pool expansion during active rebalance (#4455)** — Adding a server pool via `SIGHUP` reload while a rebalance is running could leave objects duplicated across source and destination pools, produce missing bucket volumes that made objects on *all* pools invisible until a deep heal, and write inconsistent `xl.meta` across disks in the same erasure set. Hot expansion is now rejected with an actionable operator log while a rebalance is in progress, and `globalEndpoints` plus the grid/lock-grid mesh are rolled back to their pre-expansion state. Cold-restart expansion remains the supported path for adding pools mid-rebalance and is unaffected. --- ## Performance Improvements - **Avoid per-entry NSLock in Delta Sharing config scans (#4417)** — Three internal Delta Sharing scan paths (`cleanupExpiredDeltaSharingTokens`, `listAllShares`, `listTokensForShare`) previously called `readConfigFromStorage` per entry, fanning a directory scan of N entries into N distributed NSLock RPCs. Switched to `Walk` with `Unsorted: true` + `readConfigWithOpts(NoLock: true)` so a scan is now a single namespace walk, eliminating contention with concurrent writers (e.g. token-last-used updates). Follows the same fix as PR #4132 for the MCP/server-config scan paths. - **Lock-free `logOnceIf` (#4500)** — Reworked the once-only error-log helper to use an atomic-keyed `xsync.Map` instead of a mutex-protected map, removing a hot-path lock from logging across the server. Also includes the log subsystem in the dedupe key and drops unused fields. --- ## Bug Fixes ### Replication - **Preserve internal reset headers on delete-marker resync (#4577)** — Repeated `mc replicate resync` passes over a bucket containing a delete marker corrupted the reset key in the delete-marker's `MetaSys`, dropping the `x-minio-internal-replication-reset-` prefix on every subsequent pass. `xlMetaV2DeleteMarker.ToFileInfo` then forwarded these unfiltered entries into `fi.Metadata`, causing XML syntax errors in `ListObjectVersions?metadata=true` responses. The reset-metadata handling now matches the object-version path and consistently preserves the prefixed key across resyncs. - **Clear stale KMS headers before replicating older objects (#4646)** — Plain-text objects uploaded between 2022 and 2024 sometimes carry `X-Amz-Server-Side-Encryption` and `X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id` headers (typically introduced by historical copy operations), causing replication to fail. Replication now strips these extraneous headers before dispatch; KMS headers are still re-added correctly when the object is genuinely encrypted. - **Skip replication stats update when local metadata persist fails (#4791)** — `PutObjectMetadata` errors after a successful target replication were silently ignored, causing per-bucket `ReplicatedSize` to inflate every scanner cycle when local persistence chronically failed (observed after manual erasure-set object moves to recover from a separate upstream issue). Stats are no longer credited when local metadata persist fails — under-counting is preferred over runaway overstatement. - **Permanent-delete null version on versionless replica DELETE (#4210, #4279)** — With `_MINIO_SPARK_LEGACY_OPTIMIZE=on`, a Hadoop/Spark `DROP PARTITION PURGE` on the source cluster purges null versions and sends versionless DELETEs to the replica. The replica handler misidentified these as create-delete-marker requests, producing phantom delete markers that propagated back to the source — `HeadObject` then returned `404 x-amz-delete-marker: true` and broke Hive/Spark reads on both clusters. When `globalSparkLegacyOptimize` (or `IsPurgeOnDeleteSpark`) is active and a replica DELETE arrives without a version, the version is now normalized to `nullVersionID` so the version-specific delete path runs. PR #4279 extends the same normalization to the `DeleteObjects` (bulk) path. Clusters not using Spark legacy-optimize retain prior delete-marker semantics. - **Suppress `VersionNotFound` log noise in replication tracker (#4623)** — `replicateDelete` and `requeueTrackedDeletes` now check for `VersionNotFound`/`ObjectNotFound` before logging metadata-update failures on tracker objects under `.minio.sys/buckets/.trackdelete/`. These conditions are expected when the tracker is cleaned up concurrently and were producing noisy false-positive errors in customer logs. - **Fix data race on `lrgworkers` slice in `queueReplicaTask` (#4084)** — `queueReplicaTask` indexed `p.lrgworkers` inside a `select` without holding `p.mu`, racing with `ResizeLrgWorkers` and risking a send-on-closed-channel panic when the worker pool resized. Now holds `p.mu.RLock()` across the whole `select` and guarantees `MaxWorkers`/ `MaxLWorkers` are ≥ 1 in `getPoolOpts`, mirroring the existing `getWorkerCh` pattern. ### S3 API - **Fix output-buffer sharing crash in S3 Select (#4665)** — Two long-standing defects in `internal/s3select` (in place since 2020) could crash the server and trigger systemd/container restarts under specific Select workloads: `marshal()` indexed `buf.Bytes()[buf.Len()-1]` without checking for an empty buffer, and `messageWriter.start` ran the response-streaming loop without panic recovery. A customer observed 15+ restarts over two days; the fix guards the trailing-newline trim, adds a deferred recover in the streaming goroutine, and ensures `Evaluate.Finish`/`FinishWithError` no longer deadlock on the recovered path. - **Add missing S3 namespace to `PostResponse` XML (#4454)** — `POST Object` with `success_action_status=201` returned an XML body whose root element was missing the `http://s3.amazonaws.com/doc/2006-03-01/` namespace attribute, the only such omission across all S3 XML response structs. Strict S3 clients that validate namespaces now interoperate correctly. - **Cap SigV4 streaming-trailer buffers (#4502)** — The signed (`STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER`) and unsigned (`STREAMING-UNSIGNED-PAYLOAD-TRAILER`) chunked-upload trailer readers previously accumulated trailer bytes into an unbounded buffer with only the request-body limit as a ceiling, allowing malformed inputs to drive large transient heap allocations before being rejected. Trailer content is now capped at **64 KiB** via `io.LimitedReader`, oversized input fails fast with HTTP 400 `BadRequest`, and the signed-path signature check no longer leaves trailer state partially mutated when validation fails. ### Lifecycle / Restore - **Correct version ID in "versioning not enabled" error message (#4452)** — `POST Object Restore` with `?versionId=` against a non-versioned bucket returned `InvalidArgument` with an empty version ID interpolated into the error message. The handler now reports the parsed version ID so the operator can identify the offending request. ### Batch Jobs - **Preserve original modification time on key rotation of non-current versions (#4474)** — Key-rotation batch jobs targeting a non-current version stamped the current time into `MTime`, causing the rotated version to appear as the "latest" in time-sorted listings even though the version ID was unchanged. Rotation now sets `MTime` in the destination `ObjectOptions` so the version stays in its original list position. - **Preserve object tags during key rotate jobs (#4382)** — `keyrotate` batch jobs silently dropped all `x-amz-tagging` user tags from every processed object — both already-encrypted objects undergoing rotation and plaintext objects being encrypted for the first time — because `cleanMetadata` strips `x-amz-tagging` into `UserTags` and the write paths only forwarded `UserDefined`. Tags are now re-injected immediately after `Clone()` so `CopyObject` (key-rotate) and `PutObject` (plaintext-encrypt + multipart) all carry them through. ### Audit & Logging - **Do not log internode Bearer JWT on grid WS upgrades (#4504)** — Grid WebSocket routes (`/minio/grid/v1`, `/minio/grid/lock/v1`) were wrapped in `adminMiddleware`, whose deferred `logger.AuditLog` captured `r.Header` verbatim on every peer connect — putting the internode root-secret-signed Bearer JWT into audit records that may flow to external log sinks. Grid is internode RPC (authenticated by `serverRequestValidate` and traced via `grid.ManagerOptions.TraceTo`), so admin middleware was redundant; the routes are now mounted directly without it. - **Stop overwriting `InputBytes` with `Content-Length` in audit records (#4451)** — `AuditLog` set `auditEntry.API.InputBytes` from the trace request recorder (which counts actual bytes received) and then unconditionally reassigned it to `r.ContentLength`. The recorder value is now used when present; `Content-Length` is only used as a fallback. Audit `InputBytes` is reliable again for chunked transfers and aborted uploads. ### Configuration - **Make `config.yaml` rewrite atomic on reload (#3937)** — `writeMigratedConfig` used `os.WriteFile` (non-atomic), so a peer's `getConfigFileInfo` RPC reading the file mid-rewrite could see partial bytes; the consistency gate then computed a garbage checksum that never matched any valid normalized hash, rejecting otherwise-identical configs across the cluster. Switched to a `CreateTemp` + `fsync` + `Rename` atomic write, and `getConfigFileInfo` now returns the parse error instead of silently hashing raw bytes when the on-disk config is unparsable. - **Prevent double-write in `SetConfigKVHandler` for dynamic configs (#4307)** — When `applyDynamicConfigForSubSys` failed, `applyDynamic` wrote an error response, after which `SetConfigKVHandler` still called `writeSuccessResponseHeadersOnly`. The gzip wrapper buffered both writes and the second `WriteHeader(200)` overwrote the buffered error status, so `madmin.SetConfigKV` returned `restart=true, err=nil` for dynamically reloadable subsystems (scanner, compression, heal). The three callers (`SetConfigKVHandler`, `DelConfigKVHandler`, `DeleteIDPConfig`) now return early when the dynamic apply already wrote a response. - **Propagate QoS config deletion to peer nodes (#4315)** — Peer `LoadBucketMetadataHandler` skipped `updateGlobalQOS` when `meta.qosConfig == nil`, so deleting a bucket QoS rule never tore down the in-memory `Throttle` on peer nodes — they kept enforcing the deleted rate limits and returning "traffic exceeds configured rate" errors until restart. The nil guard is removed; `updateGlobalQOS` correctly issues a no-op `DeleteThrottle` for buckets that never had QoS configured. ### Storage / Erasure - **Return 503 instead of 404 when commit metadata is missing (#4206)** — When `commitRenameDataDir` succeeded but the subsequent `CommitXL` failed with `errFileNotFound` (the staged `xl.meta` disappeared between rename and commit — a write-quorum failure, not a missing object), the error mapped to `404 NoSuchKey`. Clients could then conclude the object was absent and skip retries. The path now returns `InsufficientWriteQuorum` (HTTP 503) so clients retry. - **Fix empty `RenameData` Sign computation (#3710)** — `slices.Grow(dst, 16)` extends capacity but not length, so `copy(dst[len(dst):], …)` (which uses destination length, not capacity) wrote nothing — `Sign` was always empty. `reduceCommonVersions` consequently skipped its fast-path version-set comparison during heal (full metadata comparison still ran, but the optimization was disabled for objects with ≤ 10 versions). Replaced with an `append`-based build of the per-version-id signature buffer. ### Notifications - **Close previous Kafka client before retry in `initKafka` (#4348)** — `once.Init` retries `initKafka` on every failed ping, and each retry called `kgo.NewClient` while overwriting `target.client` without closing the previous one — leaking three background goroutines per retry (`reapConnectionsLoop`, `updateMetadataLoop`, `pushMetrics`). Production profiles showed 750k+ goroutines while a broker was transiently unreachable. The previous client is now closed before each replacement. ### Delta Sharing - **Preserve empty shares, clamp token expiry, serialize mutations (#4258)** — Three coordinated correctness fixes: - Schemas and shares are now preserved on table drop (matching Databricks Unity Catalog semantics where grants outlive contents); `deleteSharesReferencingTable` no longer auto-deletes the share when its last table is removed. - Token `ExpiresIn`/`ExpiresAt` are clamped at 100 years. Extreme values like `100000000d` overflowed `int64 time.Duration` on older `xtime` builds and landed in the past, causing the expired-token GC to wipe shares within hours. - Mutations on share, share index, per-share token index, and global token index are now serialized via an advisory write-lock on a mutation key separate from the data path, closing lost-update races between concurrent table drops, admin updates, and token create/delete; reads dedup through singleflight on hot auth paths. ### Metrics - **Use `TotalCores` for CPU load percentage calculation (#4563)** — `CPUCount` was incorrectly seeded with the logical core count and then incremented per CPU socket, producing an off-by-N value (e.g. 13 on a single-socket 12-thread host). The load-percentage calculation now uses `TotalCores` — the total schedulable thread count Linux measures load average against — restoring accurate CPU load reporting in metrics. --- ## Security & Compliance ### Software Bill of Materials (SBOM) This release includes comprehensive SBOM documentation in multiple formats: - [SPDX JSON](sbom-RELEASE.2026-05-04T23-02-27Z.spdx.json) — Standard SBOM format - [CycloneDX JSON](sbom-RELEASE.2026-05-04T23-02-27Z.cyclonedx.json) — Security scanner compatible - [Go Modules](go-modules-RELEASE.2026-05-04T23-02-27Z.txt) — Human-readable dependency list SBOM files document all direct and transitive dependencies for security auditing and compliance requirements. --- ## Upgrade Instructions For detailed upgrade instructions, please read: https://docs.min.io/enterprise/aistor-object-store/upgrade-aistor-server/ Platform-specific upgrade guides: - **Linux/Bare Metal**: https://docs.min.io/enterprise/aistor-object-store/upgrade-aistor-server/upgrade-aistor-linux/ - **Kubernetes with Helm**: https://docs.min.io/enterprise/aistor-object-store/upgrade-aistor-server/upgrade-aistor-kubernetes-helm/ ### Support For enterprise support: - SUBNET Support: https://subnet.min.io - Documentation: https://docs.min.io