CSC 2/458: Parallel and Distributed Systems 23 February 2026 Message Passing May be - Language-level mechanism, usually seen as an alternative to shared memory for use within a single process (failure domain). Go, Ada, Erlang, ... - Library-level package for communication among processes on multiple machines, usually within a cluster or data center. MPI, RPC, higher-level packages like Map-Reduce and Spark Requires some sort of failure detection and maybe recovery - OS-supported mechanism for Internet communication, typically with extensive library support. Java or Unix TCP and UDP suport Requires failure recovery Regardless of degree of distribution and failure model, need to address - naming -- how do we say who we're sending to/receiving from? - send semantics -- after sending, when do we continue? - receive semantics -- is receive explicit? Naming of processes, operations, channel abstractions must also address binding -- allows program to delay connection between program-level name and underlying path ordering -- usually messages on the same path arrive in order (UDP a notable exception) arity of ends of paths can >1 thread send to / receive from the same path? Send Semantics no-wait send -- sender continues immediately, or as soon as underlying system takes responsibility for the buffer maximal concurrency buffering and flow control a problem have to exert backpressure, which can't in the general case be hidden by the implementation error reporting a BIG problem Java, Go, MPI synchronization send -- sender waits for receiver to receive fixes error-reporting problem requires high-level acks (inefficient on many substrates) Occam, Go (CSP); MPI remote invocation send -- sender waits for reply matches many common algorithms no more expensive than synchronization send on most substrates (cheaper than pair of synchronization sends) Occam, Ada mutual emulations, cost acks for error detection S send may require 2 underlying messages RI send may require 4 can sometimes _piggyback_ to reduce this count RI often only 3 marshaling nice if language lets you send language-level objects nice if it does type checking error reporting exceptions work fine for S send and RI send, not for NW send buffering and flow control any NW send system has a limit can write a program that breaks when the limit is exceeded collective communication -- broadcast, multicast, etc. MPI has rich set Receive Semantics explicit and implicit receipt explicit -- some running thread says "receive" implicit -- message causes a new thread to be created selectivity based on source, local state, availability of message, and sometimes message content (tag in PVM/MPI) guarded commands Ada, Erlang, Go, Occam, ... timeout polling ---------------------------------------- MPI messages sent to processes; receipt is explicit by default (extra features for one-sided communication) automatic data representation conversion _communicator_ mechanism to allow communication (esp. collective) among subsets of the processes of a program, and to support various interconnection topologies sender has to specify receiver; receiver can receive from any sender if desired receiver can select based on availability, sender, and *tag* -- special field can receive from wild-card sender "blocking" semantics wait until buffer is reusable because underlying system (no-wait) or receiver (synchronization) has it "non-blocking" semantics allow sender to continue immediately, but buffer isn't safely reusable until after executing a ``post-send'' operation, which blocks Implementations can make either no-wait or synchronization send the default; programmer can insist on one or the other if desired, but there may be a cost. No-wait send fails if there isn't enough buffer space. ---------------------------------------- Java message passing via the java.net library relatively pleasant encapsulation of UDP (datagram) and TCP (reliable) sockets C or C++ programs use more flexible but cumbersome libraries supported by the OS kernel. UDP -- best-effort (unordered, unreliable) messaging DatagramSocket mySocket = new DatagramSocket(portId); DatagramPacket myMsg = new DatagramPacket(buf, len, addr, port); ... // initialize message mySocket.send(myMsg); mySocket.receive(myMsg); ... // parse content of myMsg TCP -- reliable in-order messaging ServerSocket myServerSocket = new ServerSocket(portId); Socket clientConnection = myServerSocket.accept(); Socket serverConnection = new Socket(hostName, portId); BufferedReader in = new BufferedReader( new InputStreamReader(clientConnection.getInputStream())); PrintStream out = new PrintStream(clientConnection.getOutputStream()); // This is in the server; the client would make streams out // of serverConnection. ... String s = in.readLine(); out.println("Hi, Mom\n"); Selective receipt Can be done w/ threads or using the java.nio library. Latter provides a /channel/ abstraction that is sort of a generalization of an IO stream. ServerSocketChannel ssc = ServerSocketChannel.open(); ssc.configureBlocking(false); ServerSocket ss = ssc.socket(); InetSocketAddress address = new InetSocketAddress(ports[i]); ss.bind(address); Selector sel = Selector.open(); SelectionKey k = ssc.register(sel, SelectionKey.OP_ACCEPT); sel.select(); Set selectedKeys = sel.selectedKeys(); for (k : selectedKeys) { if ((k.readyOps() & SelectionKey.OP_ACCEPT) == SelectionKey.OP_ACCEPT) { ServerSocketChannel ssc = (ServerSocketChannel)k.channel(); SocketChannel sc = ssc.accept(); sc.configureBlocking(false); sc.register(sel, SelectionKey.OP_READ); } else if (k.readyOps() & SelectionKey.OP_READ) == SelectionKey.OP_READ { SocketChannel sc = (SocketChannel) k.channel(); // handle read request } } ---------------------------------------- Rendezvous = remote invocation send plus explicit receipt RPC = remote invocation send plus implicit receipt typically built on top of UDP or TCP, and provides an interface reminiscent of local procedure calls. IDL for data declarations (parameter types) RPC 'stub compiler' generates caller and callee stubs Caller stub "marshalls" parameters and sends and receives messages On server side, each server thread calls into the RPC library and, in a loop, repeatedly receives a message and calls the appropriate callee stub, which in turn "unmarshalls" parameters, calls the appropriate local procedure, and sends back the results. transparency is challenging parameter modes -- prob. don't want to use by-reference Ada uses value-result performance failure semantics Typically provide "at most once" semantics Lots of implementations of RPC out there. Google RPC (GRPC) a popular open-source choice.