Notes for CSC 2/458 Wednesday 4-9-03 ff ========================================================== NB: my previous impressions of these systems were misleading. The SunFire system is broadcast within domains up to 24 processors, and directory-based among up to 4 domains. The Regatta uses snooping across the whole machine for operations that miss in the L2. --------------------- SunFire system 5-state MOESI protocol within coherence domain. Additional 3-state MSI tags (Mtags) that indicate global state and, on home node, identities of sharers. Stored in main memory with cache line. Also replicated in caches of processors with copies. SSM (scalable shared memory) "agent" in each domain responsible for bumping operations up to the global switch when necessary, and acting on behalf of remote domains within the local domain. Processors physically organized 4 per board. Separate address and data networks. Multiple (up to 18) address buses, partitioning physical address space within each domain. Board address repeater relays to other 3 processors and to domain-level repeater, which relays to other boards (including SSM agent) Board-level data switches are 3-port crossbars. Three per board. Two connect processors in pairs, and connect to board-level switch, which in turn connects to backplane switch. --------------------- Regatta system 2 processors (cores) per chip 4 chips per MCM (multi-chip module -- chips surface-mounted to substract without plastic or ceramic packaging) up to 4 MCM's in the machine, all on one board caches L1I per core, internal direct-mapped 128-byte line 64KB L1D per core, internal 2-way set-associative 128-byte line 32KB write-through L2 per chip, internal 8-way set-associative 128-byte line 1.5MB write-back, inclusive L3 per chip, external, but tags and controller internal 8-way set-associative 512-byte line, with coherence data per 128-byte sub-line 32MB can be interleaved among chips of the MCM to be 2- or 4-way shared 64MB or 128MB write-back, not nec. inclusive 7-state L2 coherence protocol I invalid SL shared local (i.e. local "owner") -- responsible for sourcing to requestors in this MCM S shared -- will not source to requestors in this MCM M exclusive, modified by me will source to anybody Me exclusive, but not modified entered via LL or prefetch from memory can downgrade without writeback Mu exclusive and modified but not by me entered via LL or prefetch from dirty remote cache will source to anybody (I think) T modified, but not exclusive (dropped out of M) requires negotiation to figure out who should source it 5-state L3 coherence protocol I invalid S shared -- can source only to own L2(s) T modified; may be shared; sourced from local memory Trem same at T, but sourced from remote memory O prefetched -- not modified; sourced from local memory; status in other caches unknown Within an MCM there are four coherence buses, each written by one chip and snooped by all four. All transactions between an L2 and its L3 go out on the corresponding bus. Across MCMs the buses are connected by rings that function logically as busses. Each MCM writes to one ring and snoops all four.