IBM introduces Hyperstore for accelerated remote data retrieval through NVMe-oF in Storage Scale – Blocks and Files

IBM is developing a new feature called Hyperstore within its Storage Scale system to improve the speed of accessing data from remote drives using NVMe over Fabrics (NVMe-oF). This is an advancement of IBM’s GPFS file system, which allows servers to operate in parallel, speeding up file reads and writes.

Frank Kraemer, an IBM IT architect, has expressed interest in implementing NVMe-oF for speed, as suggested by Tom Lyon in the “NFS must die” article. However, IBM plans to maintain the classic form of file system interface and erasure coding for ease of use and safe operations.

The concept of Hyperstore was mentioned in a presentation by Chris Maestas, chief architect of storage file and object systems at IBM, during the HPC user forum. He highlighted the need for data access in a hybrid and multi-cloud environment, with both CPU and GPU-based computing requiring access to remote data as if it were local.

Hyperstore aims to provide an NVMe-oF performance cluster within Storage Scale, allowing administrators to emulate local storage on GPU compute nodes, enabling remote data to run closer to the compute. This was demonstrated at the SC22 event using the IBM ESS 3500 with SSD, offering high IOPS and GBps for computing clients.

Hyperstore is a tiered system within Storage Scale, providing a reliable buffer pool using native declustered RAID (GNR), a performance pool, and access to local drives on client compute nodes. Clients access the trusted storage pool using network shared disks (NSDs), which are logical groupings of storage disks on a network.

Files are split across NSD servers, which store the stripes as blocks, allowing accessing clients to perform real-time parallel I/O. Drives in the performance group, a subset of the drives in the reliable group, are accessed using NVMe-oF for faster performance, with a shared cache between all compute nodes.

Details about Hyperstore are expected to be revealed in the coming months as IBM continues to develop this feature within its Storage Scale system.

Article Source
https://blocksandfiles.com/2024/06/21/ibm-storage-scale-hyperstore/