Global Shared Memory on a Linux cluster

Grigorios Magklis
Athanasios Papathanasiou

The ultimate goal of this project is to extend the shared memory semantics of modern operating systems to enable applications on a cluster to share pages accross machines.

1. The idea

Most modern Unix-like operating system (including Linux) support shared memory through two different mechanisms: System V shared memory, and mmap. The limitation of these two mechanism is that the processes sharing a page must reside on the same machine. In this project we propose a mechanism to allow processes on different nodes of a cluster to share a page. One of the main differences between the proposed mechanism and System V or mmap shared memory is found in the consistency model. The latter use Processor Consistency (PC), while our system will provide Lazy Release Consistency (LRC).

2. Design and Implementation

The proposed system should be comprised of two parts:

  • A set of C/C++ library routines: the routives will provide applications with a set of functions to allocate, free and manipulate shared memory segments.

  • A kernel module that will support Distributed Shared Memory across a cluster of work stations (using LRC). The module will make use of the algorithms developed in the Cashmere project.

Our main intention while implementing the system is to minimize the modifications to the core of the Linux kernel. A second goal is to make this system available only to applications who need distributed shared memory. This way not every process in the system will have to pay the overhead of this extra functionality.

The main additions to the system's kernel will be two data structures:

  • A per process virtual-to-global address translation table. This will translate the process specific virtual page address to a cluster-wide global page address, used by the rest of the system to identify pages. This extra level of abstraction enables processes to map the share segments in any virtual address.

  • A per node page directory. The page directory will be indexed using the global address and will provide with the necessary information to implement the consistency model.

Applications will co-operate with the kernel module through the user-level library and the page fault mechanism. The library routines will be used to ``register'' themselves with this system and manipulate shared memory segments. The page fault mechanism will be used to track down page accesses, and provide applications with the LR consistency model.

Registered applications will use the new page fault path, while unregistered applications will behave in the standard way. Thus unregistered applications will not have to pay the potential overhead of this new path.

Short-term goals

We expect this system to span the time frame of this course, so we have to set some short-term goals. The implementation of the two data structures, the page fault mechanism and the user-level library routines (ie. all the entry point to the protocol) is imperative so we plan on working on these first. Next in order is a simple allocator-deallocator process that will try to allocate and free global shared segments (a simple testbed app). After this we're going to work on the consistency mechanisms for the shared segments and implement the full protocol. We expect to finish the first part of this project in this class, namely implementation of the entry points and the testbed application.