# MinIO AIStor EDGE.2025-11-27T00-47-50Z

Released: November 27, 2025

This release introduces comprehensive Apache Iceberg catalog support with native Trino and Spark integration, transforming MinIO AIStor into a full-featured lakehouse platform. Major operational enhancements include persistent log recording for compliance, lease-based distributed locking to eliminate stale locks, and 146 updated Prometheus metrics with a new Customer Operations Dashboard. Performance improvements include Grid RPC compression reducing internal communication overhead and real-time replication metrics collection.

## New Features

### Apache Iceberg Catalog API
- **Complete Iceberg REST Catalog implementation** - Full support for table management, namespaces, and view operations compatible with Apache Iceberg
- **Iceberg Views API** - Native support for materialized and logical views with Trino and Spark integration (#2276)
- **TableAuthSys caching** - Cached authorization for table operations improving query performance (#2286)
- **Register-table endpoint** - Support for registering existing Iceberg tables
- **HEAD endpoints** - HEAD support for namespaces and tables for existence checks
- **Inventory support for AIStor Tables** - Full inventory integration for lakehouse workloads

### Monitoring & Observability
- **Persistent log recorder** - Disk-backed storage for API, Error, and Audit logs with configurable flush intervals enabling long-term compliance retention (#2173)
  - New environment variables: `MINIO_API_LOG_ENABLE`, `MINIO_API_LOG_DISK_LIMIT`, `MINIO_API_LOG_FLUSH_COUNT`, `MINIO_API_LOG_FLUSH_INTERVAL`
  - Similar variables for error logs (`ERROR_LOG_*`) and audit logs (`AUDIT_LOG_*`)
- **Customer Operations Dashboard** - Production-ready Grafana dashboard with 146 updated Prometheus metrics (#2120)
- **Admin trace support for Tables** - Real-time request tracing for Iceberg catalog operations
- **Enhanced inventory status API** - Comprehensive execution metrics including nextScheduledTime, startTime, endTime, scannedCount, matchedCount, executionTime, numStarts, numErrors, numLockLosses, and manifestPath (#1871, #1808)
- **Auto-generated cluster names** - Memorable two-part names like 'GriffinWave' or 'UnicornFrost' for easier cluster identification (#1897)

### High Availability & Operations
- **Lease-based distributed locking** - Eliminates stale locks with automatic expiration using absolute timestamps instead of refresh-based validity (#2073)
- **Cordon functionality** - Graceful node draining for maintenance without removing from cluster (#1433)
- **Update cancellation and downgrade** - Ability to cancel in-progress updates and perform controlled version downgrades (#2119)
- **Binary tree update distribution** - Optimized server binary updates across large clusters using tree topology instead of fan-out, with recursive fallback for offline nodes (#1967)

### Identity & Access Management
- **Multiple LDAP identity providers** - Configure multiple LDAP servers with new LDAP IAM type supporting diverse directory structures (#1931)
- **Service account parent status** - Shows parent user status for service accounts improving account management (#2252)
- **Persistent console UI configuration** - Console settings survive server restarts (#1923)
- **Deployment ID preset** - Support for `_MINIO_DEPLOYMENT_ID` environment variable for controlled cluster identity in containerized environments (#1652)

### Storage & Data Protection
- **eBPF LSM storage protection** - Kernel-level protection preventing unauthorized deletion of MinIO AIStor data directories, even by root (#2009, #1945)
- **Goroutine panic recovery** - Critical path protection with automatic recovery and structured error handling (#2051)
- **Write verification response headers** - Client-visible confirmation via `X-Amz-Write-Verified` header (#1955)
- **CRC32C bitrot for ARM64** - Hardware-accelerated CRC32C with SSE4.2 support for improved data integrity verification (#2196)

## Improvements & Enhancements

### Performance
- **Grid RPC compression** - Automatic compression for internal server communication payloads over 4KB reducing network bandwidth (#1754, #2210)
- **Real-time replication metrics** - Windowed time-series collection with per-bucket and site-level tracking (#2128, #2183)
- **Metrics cardinality reduction** - Consolidating labels and removing redundant metrics reducing memory overhead (#2277)
- **ServerInfo API optimization** - Reduced cache TTL to 10s with zero-overhead Grid connection state queries (#2048)
- **IO metrics optimization** - Eliminated redundant DiskInfo() calls per disk per second (#2070)

### Replication & Site Management
- **IAM parity validation** - Detect IAM configuration inconsistencies across replicated sites (#1947)
- **Site replication rollback** - Automatic revert of partially failed site join operations (#1872)
- **Batch replication multipart fix** - Correct part number handling for non-sequential multipart objects (#2329)
- **Purge-on-delete improvements** - Fixed deadlock and directory object version buildup (#2186, #2135)

### Compatibility & Standards
- **Conditional delete headers** - Amazon S3's If-Match and If-None-Match for DeleteObject operations (#2114)
- **Bucket replication through load balancers** - Allows replication setup through load balancers on different ports (#2202)
- **Historic KES ciphertext support** - Backward compatibility for msgp ciphertext format from older KES versions (#2083)
- **FTPS certificate reload** - Dynamic certificate reloading without server restart (#2011)

### Monitoring & Diagnostics
- **Missing V3 metrics** - Added 41 metrics from V2 to V3 including healing, ILM, KMS, lambda, locks, logger webhook, software info, and tiering (#2054)
- **Network, CPU, and memory metrics** - Comprehensive system resource metrics for V3 (#2102)
- **Lock monitoring metrics** - Detailed distributed lock metrics including wait times, hold durations, and contention statistics (#2130)
- **Inventory V2 metrics** - Complete inventory feature metrics in V2 Prometheus endpoint (#2055)
- **Set query enhancements** - Additional metadata and statistics fields in set query API results (#2019)
- **Drive query healing info** - Basic healing information in Drive Query API (#1924)

### Developer Experience
- **Enhanced debug logging** - Trace support for tracked deletion, write operations, and internal calls (#2132, #2197)
- **Improved error messages** - Clearer deployment ID guidance and better validation messages (#1985)
- **Reduced log noise** - Suppressed expected context.Canceled and NoSuchInventoryConfiguration errors (#2106, #2094)
- **Lock cleanup optimization** - Reduced expired lock cleanup interval from 2 hours to 5 minutes (#2110)

## Bug Fixes

### Critical Fixes
- **Key rotation on versioned buckets** - Fixed key rotation failures with multiple object versions (#2320)
- **Multipart part number handling** - Corrected GetObject PartNumber for non-sequential parts (#2323)
- **Batch replicate multipart partnumber** - Fixed part number tracking in batch replication (#2329)
- **KMS connectivity tracking** - Accurate health monitoring from MinIO AIStor perspective (#2289)
- **Table creation race conditions** - Allow table creation if previously staged, eliminating precondition failures (#2313)

### Data Integrity
- **Offline drive healing** - Restored xattr tracking for delete and write operations fixing drive healing (#1988)
- **Scanner bucket scan info** - Fixed API incorrectly reporting scan status for individual buckets (#2193)
- **Object naming mode preservation** - Ensures replacement drives inherit cluster's object naming mode (#2188)
- **Health check panic** - Fixed panic in heterogeneous cluster configurations with different pool sizes (#1666)

### Replication & Lifecycle
- **Site replication state races** - Eliminated race conditions in peer edit operations by refactoring saveToDisk (#2319)
- **LDAP import of site-replicator-0** - Allows IAM import to overwrite site-replicator-0 service account for recovery (#2184)
- **Replication purge hang** - Resolved deadlock in purge-on-delete feature (#2186)
- **ILM expiry replication** - Fixed ExpiredObjectDeleteMarker flag not being replicated to peer sites (#2013)

### Metrics & Monitoring
- **QoS metrics regression** - Restored QoS metrics functionality with activity-based filtering (#2155)
- **Grid RPC daily stats** - Fixed statistics calculation (#2330)
- **Merged metrics** - Updated madmin-go dependency to fix merge metrics issues (#2198)
- **V3 metrics issues** - Renamed /cluster/iam to /iam, clarified queue descriptions, removed redundant metrics (#2185)
- **ClusterAPIStats empty** - Fixed inverted condition causing empty LastDayAPI metrics (#1944)
- **GetWindow stale data** - Ensured windowed metrics update to current time before returning (#2181)
- **Offline drive metrics** - Fixed offline drive counting in metrics (#1968)
- **Single-node metrics** - Fixed NodesQuery and DrivesQuery for single-node deployments (#2080, #2014)

### Operations & Stability
- **Health check during updates** - Fixed sporadic failures in rolling update integration tests (#2201)
- **Cordon quorum check** - Corrected to use write quorum instead of read quorum (#2077)
- **Stale lock detection** - Fixed reporting after transition to per-drive locking (#2109)
- **Node name for single-drive** - Returns local node name for single-drive endpoint configurations (#2017, #1990, #1965)
- **V4 API performance** - Fixed slowness when servers are offline (#2023)
- **Double response headers** - Prevented duplicate header writes for PreconditionFailed errors (#1973)
- **Inventory race conditions** - Resolved 'Precondition failed' errors from concurrent job operations (#2127)
- **Inventory context cancelled** - Fixed ETag consistency in metadata updater (#1863)

### Tables & Iceberg
- **Warehouse bucket deletion** - Protected warehouse buckets from S3 API deletion
- **Namespace validation errors** - Improved error handling and validation
- **IAM policy resource handling** - Correct ARN handling for table resources
- **SigV4 character escaping** - Proper URL encoding in authentication

## Security Updates

- **eBPF LSM enforcement** - Kernel-level storage protection preventing direct filesystem access bypassing MinIO AIStor (#2009, #1945)
- **KMS connection monitoring** - Enhanced visibility into encryption key management connectivity and latency (#2289)
- **POST policy tag enforcement** - Fixed security issue where tagging in POST policy uploads was not enforcing policy-specified tags (#2005)

## Performance Improvements

- **Grid RPC compression** - Automatic compression for internal network traffic when payloads exceed 4KB (#1754)
- **Binary tree update distribution** - Reduced bandwidth usage and improved resilience during updates in large clusters (#1967)
- **Real-time replication metrics** - Eliminated expensive periodic bucket scans with windowed time-series collection (#2128)
- **ServerInfo API** - Sub-millisecond response times with 10s cache TTL and background refresh (#2048)
- **IO metrics optimization** - Eliminated redundant DiskInfo() calls per disk per second (#2070)
- **CRC32C for ARM64** - Hardware-accelerated data integrity verification on ARM64 systems (#2196)

---

## Upgrade Instructions

For detailed upgrade instructions, visit: https://docs.min.io/enterprise/aistor-object-store/upgrade-aistor-server/upgrade-aistor-linux/

### Quick Upgrade Steps:

1. Download the latest EDGE binary for your architecture from https://dl.minio.io/aistor/minio/edge/
2. Stop the MinIO AIStor service
3. Replace the existing binary with the new version
4. Restart the MinIO AIStor service

**Important**: This is an EDGE release containing the latest development features and improvements. For production deployments, use official RELEASE versions.

For complete release details and downloads, visit: https://dl.minio.io/aistor/minio/edge/