Lecture notes for CSC 252, Thurs. Feb. 13 ff, 2014 A3 is on the web. Trivia due Tues. Feb. 18; main due Fri. Feb. 28. Midterm exam Tues. Mar. 4. Special event Thurs. Mar. 6: Bob Colwell ACADEMIC HONESTY Chapter 4 reading: skim whole chapter read the sections on the course schedule page particularly carefully Topics: quick review of low-level basics, inc. clocking issues in instruction set design and encoding sequential and pipelined processor implementations hazards and their mitigation, interrupts superscalar and out-of-order processors ============================== Building blocks can learn much more by taking ECE courses Bottom level: solid state physics then: transistors then: gates & clocking hardware units microarchitecture ------------------------------ Digital Circuits (review?) Combinational Start with AND, OR, NOT, NAND, NOR, etc. gates Build n-bit Boolean function of m Boolean inputs Acyclic --> no memory. We can build arbitrary functions with two levels of logic, but potentially exponential area (size). Various techniques (e.g. Karnaugh maps, multiple levels) to reduce complexity of circuit. ALUs a nice example. Also control logic (e.g. instruction decode). Mux/demux circuits used a lot for control; choose among several possible inputs: select --+---NOT---, | | | AND----, A ------------' | | OR----- out '---------, | AND----' B ------------' Semi-obvious hierarchical multi-bit extension. Use k select lines to choose among 2^k possible inputs. (Could also build a flat circuit, but that requires unreasonable fan-in.) Sequential Outputs depend on current input AND saved state. Requires cycles (loops) in wiring. Flip-flop the basic circuit: remembers one bit. ,---------------------------, | | AND-------------------, | load -----+----NOT-----' old OR-----+---- out | ,----' '------------, | AND--------------' in ------------------' new load = 1 --> old = 0 --> out = new = in load = 0 --> new = 0 --> out = old Obvious multi-bit extension. Build registers, other sorts of memory. Practical issues area, depth fan-in, fan-out power (related to area, degree of switching (changing bits)) ------------------------------ Clocking Signals in a digital circuit propagate at rates determined by solid-state physics and circuit complexity. Signals running through two different circuits will probably come out the end at different times. If they form the two inputs to something else, we need to do something to sync them up. The standard solution is CLOCKING. We divide the chip into lots of little circuits. On (each line of) the output of each circuit we put a flipflop (latch) that holds a signal until instructed to pass it on. We then superimpose a global network of wires that delivers "pass it on" signals to the latches. Suppose all 'load' lines are currently low. Latches are remembering values; have potentially different values at their inputs. Momemtarily drive all 'load' lines high to grab the input values, then low again before new signals have had a chance to propagate to the inputs. Pass-it-on network, and the oscillator that drives it, are the CLOCK. On a 1GHz processor, the clock delivers a signal to every latch every nanosecond. This time interval (one nanosecond in this example) is the CYCLE TIME (= 1/frequency). Its length is chosen to be the maximum, over all little circuits on the chip, of the time it takes signals to propagate from the inputs to the outputs of the circuit. [NB: at the circuit level, clocked latches are called "registers". This is a slightly different meaning from the ISA level. ISA "registers" are implemented with a circuit called a "register file": READ PORT(S) WRITE PORT(S) (k) ,--------------, (k) reg specifier ------>| |<------ reg specifier (t) | | (t) reg value <------| |<------ reg value | | (k) | | reg specifier ------>| | (t) | | reg value <------| | | | ... | |<------ clock '--------------' ] A typical instruction goes through several (roughly 5 to 20 on most machines) intermediate steps. So the latency to execute a single instruction is several cycles. PIPELINING allows new instructions to begin executing before old ones have finished, so we have multiple instructions "on the fly" at once. CPI (cycles per instruction) takes pipelining into account. You get it by dividing some large span of cycles by the number of instructions completed during that span. To run the clock faster you have to either shrink the propagation time, typically by making everything smaller, or reduce the number of levels of logic through which signals must propagate per clock. ------------------------------ Integrated circuit manufacturing ingots, wafers, and dies defect: microscopic imperfection in wafer die cost = wafer cost / (dies per wafer X die yield) big chips cost much more because you get fewer that work. ** In practice, die cost goes up roughly with the cube of the area. As chips come off the assembly line they're tested at various clock speeds. Throw out the ones that don't work at all; sort the rest into bins. Get fewer in the higher-clock bins; sell them for more $. ------------------------------ Instruction Set Architecture (ISA) provides the interface between hardware and low-level software Goal: Find a language that makes it easy to build both the hardware and the compiler while maximizing performance and minimizing cost Years ago, the ISA was one of the most important aspects of computer design. It had a great impact on the software (program code size, ease of compilation, backward compatibility) and the hardware (could it be implemented, amenability to h/w optimizations). Today the importance of the ISA has been greatly reduced. [The engineer arguably most responsible for that breakthrough is our guest speaker on March 6.] Code size hardly matters; compiler optimizations are better automated; technology exists to translate from one ISA to another, statically or dynamically. ------------------------- Processor evolution As technology advances, there are occasionally times when some threshold is crossed that suddenly makes it possible to design machines in a very different way. An early example of such a "paradigm shift" was the development of microprogramming in the mid 1960s. During the 1950s and the early 1960s, the instruction set of a typical computer was implemented by soldering together large numbers of discrete components that performed the required operations. To build a faster computer, one generally designed extra, more powerful instructions, which required extra hardware. This had the unfortunate effect of requiring assembly language programmers (or compiler writers, though there weren't many of them back then) to learn a new language every time a new and better computer came along. As computers got cheaper and demand increased, a low-end market developed. IBM realized they could exploit this market -- and the spread between it and the high end -- by introducing a implementation technique called MICROPROGRAMMING. This allowed them to provide the *same* instruction set across a whole line of computers, from cheap slow machines to expensive fast machines. The basic idea of microprogramming was to build a "microengine" in hardware that executed a interpreter program in "firmware". The interpreter implemented the IBM 360 instruction set. More expensive machines had fancier microengines, with more of the 360 functionality in hardware. The top-of-the-line machines (e.g. the groundbreaking 360 model 91, which pioneered many key techniques, including out-of-order execution) had everything in hardware. Microprogramming also made it easy to extend instruction sets, which people did. They ran studies to identify instructions that often occurred in sequence (e.g. the sequence that jumps to a subroutine and updates bookkeeping information in the stack), and then provided new instructions that performed the function of the sequence. By clever programming in the firmware, it was generally possible to make the new instruction faster than the old sequence, and programs got faster. The microcomputer revolution of the late 1970s (a second "paradigm shift" 8086 = 1978) occurred when it became possible to fit a microengine onto a single chip -- a microprocessor -- and personal computers were born. PCs didn't induce a major change in (low end) ISA design, but they caused a dramatic drop in prices. Over this whole period of time, ISAs were getting larger and more complex. There was one primary motivation for this. Designers wanted the assembly-level language to closely resemble the high-level language. They believed that this would reduce the code size, improve performance because of the efficient transformation from high-level code to actual hardware, and make compilation easy. The x86 ISA is an example of complex instruction set computing (CISC). The VAX was the most complex to be a major commercial success. Others of note include the IBM 360/370 and the Motorola 68K. Later, various researchers pointed out the pitfalls of the CISC approach. (1) Compilation is not really simplified as the compiler has to find the exact code sequences that fit a certain instruction. It turned out that compilers generally produced very simple instructions. It is also harder to apply compiler optimizations to a complex ISA. (2) CISC programs were not all *that* much smaller: while there are fewer instructions, the average encoding for each instruction is longer. (3) The dedicated hardware for the complex instructions didn't improve execution speed all that much, either: the control unit became more complex and that slowed down the simpler instructions (which were in a majority). Supercomputer designers knew this. Companies like CDC (Control Data) built *pipelined* machines in the 1960s with simple, *regular* (uniform) instructions. Pipelining made the machines fast, but also complex and therefore expensive. A third "paradigm shift" occurred in the early 1980s (commercialized in the mid 1980s) when people realized that VLSI density had advanced to the point where you could put a pipelined processor on a single chip, if the ISA were simple enough. [NB: People didn't exactly adopt supercomputer ISAs. Those tended to have large register sets, vector instructions, and an emphasis on floating point. Microprocessors didn't have room for these; nor was FP important for the target market.] This was the "RISC revolution". A common misconception is that small instruction sets are the distinguishing characteristic of RISC machines. A somewhat better characterization is that RISC machines are machines in which the ISA has been designed to facilitate (1) stating at least one new instruction every cycle (2) making the cycles very short, and (3) maximizing the number of hardware units that can operate in parallel during a given cycle. Maximal parallelism benefits from giving the instructions a very regular, predictable format, so that the interactions between instructions are clear and the processor can begin working on the next instruction before the previous one has finished. RISC architectures have operations that are usually register-to-register, with only simple load and store operations accessing memory. This simplifies the control unit. It results in a large register file and in simple addressing modes. They also have simple fixed-length instruction formats. It is, therefore, possible to read register operands while still decoding the opcode. There is no definitive definition of what RISC means. In fact, there are very few pure RISC or CISC processors as most RISC processors have characteristics that are typical of CISCs and vice versa. Here are some generalizations of the RISC vs CISC comparison: RISC CISC new instruction(s) every cycle multi-cycle instructions load/store architecture any instruction may reference memory highly pipelined not pipelined or less pipelined hardwired microprogrammed (old x86 stuff) fixed format instructions variable format instructions few instructions, modes many instructions, modes complexity in the compiler complexity in the microprogram multiple register sets single register set The Pentium Pro (P6) [1995] Given Intel's large installed base with the x86, it could not move away from the instruction set. However, the PentiumPro was a drastically different implementation that resembled a RISC processor. It was basically a RISC processor that ran a CISC instruction set by translating the CISC instructions into sets of smaller instructions. Still the basis of the Intel Core (Pentium M) architecture. Excellent book on this: The Pentium Chronicles (2005), by Robert Colwell (chief architect). Cf. earlier book on CISC project: The Soul of a New Machine (1981), by Tracy Kidder (reporter). Data General Eclipse MV/8000 (Eagle). The decoder converts each instruction into "micro-ops". These are buffered in a micro-op queue. They then undergo register renaming and out-of-order execution like in other RISC processors. Thus, parts of various different CISC instructions can execute simultaneously as they may not have any dependences -- the original CISC instructions may have had dependences that would have prevented the second one from starting before the completion of the first. This also differentiates the micro-op conversion from a traditional microprogrammed CISC processor. Because the micro-ops are simple operations, the P6 was also heavily pipelined - it had a 14-stage pipeline. ------------------------------ Basic Issues In Instruction Set Design The typical RISC machine is 3-address, load-store, with separate integer and FP registers. The number of registers has grown over time 6502 had about 4, all different x86 has 8 (not counting PC, segment registers, CCs, SSE, FP), all different VAX had 16 Cray 1 had 611 64-bit registers, of 5 different kinds Most RISC machines have 32 int and 32 fp IA64 has 128 of each It's common to have more physical registers than architectural, and to do register _renaming_. More on this later. In SPARC and IA64 (Itanium), the mapping from architectural registers to physical registers changes on subroutine calls (with some overlap for passing arguments). ------------------------------ Instruction Format Can be fixed length, variable length, or a hybrid Code density argues for variable-length instructions, but simplicity (and thus cycle time) argues for fixed. CISC machines tend to be variable (VAX was probably the most extreme example); RISC machines are all fixed (all 32 bits, in fact). One interesting exception: 32b ARM has a "Thumb" mode in which most instructions are only 16 bits, a few are 32 bits, and a fair number are unavailable. It can switch back and forth between Thumb mode and ordinary 32b mode in a single cycle -- usually as part of the subroutine calling sequence. [IA64 is in another category, called LIW (large instruction word), which Intel prefers to call EPIC (explicitly parallel instruction-set computer). IA64 has 128-bit instructions, consisting of three 40-bit sub-instructions and 8 bits of information about dependences among those instructions.] ------------------------------ RISC instructions (NB: 64-bit MIPS, Power, SPARC, and ARM are all pretty similar, in terms of number of fields and number of bits devoted to each) All MIPS instructions are 32 bits. Load and store are the *only* memory access instructions. Displacement is the only addressing mode: register contents plus immediate = address. First 6 bits (op) indicate operation That's 2^6 = 64 possible instructions. We need a few more than that, so some ops cover >1 instruction, with extra bits borrowed elsewhere to distinguish them Remaining bits used for 3 register names and 11 extra bits or 2 register names and a 16-bit "immediate operand" used for arithmetic when one operand is a constant and for load/store (displacement address and 16-bit offset) and for branches (with displacement in words, not bytes) (branches use 2 regs for comparison, e.g. be r1, r2, label ) or 26 bits of jump target, which is left-shifted two bits, and replaces the 28 low-order bits of the PC. (unconditional only) ------------------------------ RISC wrinkles Loading large absolute/immediate values 32-bit instruction size means you can't say "load" and specify a 32-bit value in a single instruction. Therefore have to use a two-instruction sequence. Example: load address 0x12345678 into a register on the MIPS: lui $5, 0x1234 # "load upper immediate": sets upper 16 bits ori $5, $5, 0x5678 Other multi-instruction sequences often needed in comparison to CISC machines: subroutine call/return, push/pop, arithmetic on memory, ... Load and Branch Delay Slots: more on these later Register windows: take 254 Hardware register renaming: more on this later ---------------------------------------- Processor design The "Y86" (from the text) similar to the x86, but much simpler load-store 13 total instructions, 3 (OP, jXX, cmov) w/ multiple variants all instructions 1 (nop, halt, ret), 2 (OP, push, pop, rrmov, cmov), or 6 (load, store, irmov, jump, call) bytes long register specifiers always in the same place in the instruction immediate operands always in the same place, and 4 bytes long < see Figure 4.2 > Implementation strategies: - simple single-cycle (very SLOW cycle) control - multicycle control, still single instruction at a time, built using microprogramming (not in the book). Advantages: - can use same hardware unit more than once in a single instruction - can finish simple instructions in fewer cycles than complex instructions. Suppose single-cycle design has a 10MHz clock. That's designed to accommodate the most time-consuming instruction (probably a load). The multi-cycle design might have a 100MHz clock, with the longest instruction taking 12 cycles. That lengthy instruction will be slower on this machine, but simpler instructions (which are probably more common) might finish in just 2 or 3 cycles. Overall we win, both in speed and in area. Y86 is designed in such a way that there isn't much of a win for this: instructions take approx. equal amounts of time to execute. - pipelined control, still in-order Advantage: multiple instructions in flight, for lower CPI - superscalar and out-of-order execution ======================================== single-cycle control Can't use any hardware unit twice in one cycle. If any datapath element is needed for two purposes by the same instruction (e.g. ALU and PC incrementer) we'll need two copies. If a datapath element is needed for different purposes by *different* instructions, we can reuse it if we hook up the inputs and outputs with MUXes. Multiple reads or writes to memory or register file require multiporting. ---------------------------------------- SEQ implementation (from book) Basic idea: divide instruction execution into stages for conceptual purposes -- helps us organize design, and will be useful later for pipelining latch PC condition codes register values memory contents let everything update during one long slow clock. We could probably run a system like this with current technology at around 100MHz if it used a very small (on-chip) memory. For bigger memory we'd need a much slower clock (~1MHz) or a mechanism to force a wait on loads and maybe stores, ideally combined with a cache to avoid the wait on hits How do register file and memory work? Partially combinational, partially sequential. read address | | write address | | | | write value | | | v v v ,-------,-------------------, | | | | | |<--- clock | | | | | | | | | | '-------------------| | | '---------------------------' | v read value So long as the clock is low we have, effectively, a combinational function from read address(es) to read values (as many pairs as there are read ports). When the clock goes high the function changes. Now the whole processor effectively works like this (written flowing up the board/page, as in the book): ,-------------------------------, | | ,-----------------------, | | | ,-----------, | '-------, |---->| | | ^ | lots of | | registers | | CCs are | | combinational |<----| | | either ,---, | logic | '-----------' | inputs or |CCs| | | | outputs, '---' | | ,-----------, | depending ^ | |---->| | | on inst. | | | | memory | | (never ,-------' |<----| | | both) | | '-----------' | '-----------------------' | ^ | | ,------, | '--------------| PC |---------' '------' << go over figure 4.23 in detail >> - light blue boxes are data path, combinational or mostly so - grey boxes are control path, combinational - white ovals are wire names -- no logic - sequential state is CC, PC, instruction memory, data memory, register file *** latching these is all we need to make the processor work (The latching concept is a little subtle: basically, we read old values out when the clock is low and write new values in when the clock is high. So, for example, each write port of the register file has a demux in front of it, and each read port has a mux after it. The latch is on the output of the demux, which includes the 'load' signals of the register file. Those load signals are distinct from the clock, which triggers the latch.) - wide lines are word-wide data - narrow lines are sub-word data - dotted lines are control signals not all wires (esp. control wires) are shown - ALU fun. box requires icode and ifun as inputs - rA and rB have to be fed to reg. file - srcA tells register file which register to read on line valA needs icode and rA as inputs - srcB tells register file which register to read on line valB needs icode and rB as inputs - dstM tells register file which register (if any) to update from memory needs icode and rA as inputs - dstE tells register file which register (if any) to update from ALU needs icode and rB as inputs - ALU A box decides whether to use valC or valA needs icode as input - ALU B box decides whether to use B or 0 needs icode as input - set CC depending on ALU output && right kind of inst - etc. popl is the only instruction that writes two registers in one instruction. ALU used for arithmetic stack pointer dec./inc. in push/pop, call/ret effective addr. calc. in load, store +0 in RR, IR moves PC increment is another ALU needs icode as input real memory isn't anywhere near as fast as the rest of the machine; would limit this design to around 1MHz ------------------------------ SEQ+ implementation (from book) Problem with SEQ: have to finish previous instruction before we know PC for next instruction. Not a problem now, but will inhibit pipelining. Solution: rotate PC updating around from end of circuit to beginning. Latch data from which it is computed, rather than value itself. Now in the absence of control flow we have all the new data very soon, which will allow us to start the next instruction. Dealing with branches will be a challenge. More on that next time. << go over figure 4.40 in detail >> Note in particular the feedback loop from valP back to pValP. During pipelined execution this will allow us to start a new instruction every cycle. ------------------------------ [ multiple-cycle control (not in book) [ [ This is how CISC machines were historically implemented. [ [ If we go multicycle we have the option of re-using datapath components [ in different cycles of the same instruction. [ We can also fetch instructions and data from the same memory system [ without multiporting. [ [ We save chip area, and can do simple instructions faster. [ [ Control of course is now more complex. [ [ GOAL: try to do an equal amount of work in each cycle, so we can make [ the cycle time as short as possible. [ [ Control for a multi-cycle processor can be hardwired (FSM) or microcoded. [ Historically, microcode tended to require less chip area. ============================== Pipelining The key performance-enhancing idea in modern microprocessors Dates from CDC and IBM supercomputers of the 1960s. Factory assembly line is the traditional analogy. basic idea stages (*example*) F instruction fetch D instruction decode and register fetch E execution or effective address calculation M memory access W write back (of results to registers) F D E M W instrs F D E M W | F D E M W | F D E M W v ... time --> requires all the hardware of the single-slow-cycle implementation plus latches to store intermediate values (inst, ALU output, PC) for use by later stage (as in the multi-cycle implementation) these are PIPELINE REGISTERS latches and comparators for forwarding (see below) MUCH higher memory bandwidth -- aggressive caches, separate I and D caches yet more complex control performance improvement speedup = (average instruction time w/out pipeline) / (ave inst time w/ pipeline) REMEMBER: denominator gets to divide by number of stages, but has to eat pipeline overhead and is limited by the slowest stage. EXAMPLE: suppose steps above take 50, 50, 60, 50, 50 ns, respectively. Non-pipelined machine takes 260 ns/inst. If pipelined machine has 5ns setup per stage, it takes 65 ns/inst -- 4-fold speedup, not 5. Note that relative overhead of pipeline registers increases as size of stages shrinks --> diminishing returns from very deep pipelining. ------------------------------ PIPE- implementation (from book) << go over figure 4.41 in detail >> 5 pipeline registers, one to latch inputs to each stage predict that all branches are taken (right ~60% of the time) note that inputs to latches include not only data but also control signals for example, we have 4 different icode values (for 4 different instructions) latched in pipeline registers similarly, dstE and dstM are passed through the execute and memory stages so we can delay their use until the writeback stage We latch a *lot* of state. Sometimes we can overload things. For example, valP (new PC) is needed in execute stage, where we discover that a conditional jump was not taken (i.e. was mispredicted). valP is also needed in the memory stage, where we need to store it for call instructions. Neither of these needs valA, so we can let valA and valP share space in the pipeline register -- like a union, essentially. The Select_A box in the fig chooses which to save. Note, though, that we can't merge valE and valM, because a pop instruction produces both of them. Convention in book: upper-case prefix means data value or control signal latched in input register for this stage; lower-case prefix means data value or control signal computed in this stage Pipeline works using *predicted* PC value. More later on what happens if the prediction is wrong. *** PIPE- runs some programs just fine but NOT programs with short-range dependences among instructions ------------------------------ hazards cause stalls (aka bubbles) in a correct implementation (not in PIPE-) stall everything behind, too, with in-order execution hazard types structural data control ------------------------------ structural hazards register file instruction and data fetches (if you miss in L1 caches and collide in L2) FP units, etc. (for multi-cycle instructions) can't start FP divides on consecutive cycles standard solutions: capitalize on fact that register file is among the fastest pipeline stages: write to it (W stage) in one half of the cycle; read from it (D stage) in other half of the cycle live with L2 collisions rely on compiler to schedule other stuff between FP instructions ------------------------------ data hazards sources condition codes not a problem in PIPE- because they're generated and used in same stage (E) loads book assumes machine can do memory loads and stores in one cycle. Unless we do something special, will have to stall two cycles if E stage of current instruction needs load result of previous instruction; one cycle if we need the load result from two instructions ago. We'll reduce this to 1 and 0 below. in real machines, L1 cache works in 2-3 cycles; misses take much longer contents of general registers book focuses on these Note that all 3 sources become more severe in a deeper pipeline. data -- WR (true), RW (anti), WW (output) dependences latter 2 can be addressed with register renaming (more on this later) For now, focus on true dependences. consider add $10, %edx F D E M W add $3, %eax F D E M W nop F D E M\ W nop F D E \ M W nop F D \E M W add %edx, %eax F D E M W halt F D E M W No problem. But if we remove a nop, we don't have %eax in time. If we remove 2 nops, we don't have %edx in time. Solutions (1) programmer's responsibility to include the nops. Cray 1 worked this way. If they're missing, behavior is undefined (and certainly wrong) (2) stall in HW (interlocking) injects bubbles (dynamic nops) add $10, %edx F D E M W add $3, %eax F D E M W add %edx, %eax F ( bubble ) ` F ( bubble ) ` F ( bubble ) ` F D E M W whatever < draw this incrementally, left-to-right > (3) forward (shortcut) instead of moving value from W to D, move it from E to E add $10, %edx F D E M W add $3, %eax F D E M W \ add %edx, %eax F D E M W no bubble But some significant complexity: forwarding unit, which provides value to execute unit input pipeline register. Key idea is to snag values from "previous" instructions that will eventually reach our input registers, but aren't there yet. Must be prepared to take input from any of several sources. To tell which, every value moving down the pipe has to be tagged with the register to which it is supposed to be written. Forwarding logic can then tell whether to use value in register file or soon-to-be-written value from a pipeline register, the ALU, or the memory read port. if (reg == e_dstE) use e_valE // ALU result from 1 instruction ago elsif (reg == M_dstE) use M_valE // ALU result from 2 instructions ago elsif (reg == M_dstM) use m_valM // load result from 2 instructions ago elsif (reg == W_dstE) use W_valE // ALU result from 3 instructions ago elsif (reg == W_dstM) use W_valM // load result from 3 instructions ago else use reg // > 3 instructions ago << figure 4.52 >> full PIPE diagram << figure 4.56 >> detail of forwarding logic ------------------------------ Now what about loads? Even with forwarding we can't get the value when we need it. Interlock HW introduces one-cycle bubble, assuming a hit in a fast L1 cache: mov foo, %eax F D E M W addl $3, %eax F ( bubble ) ( bubble ) ` F D E M W pushl %eax F D E M W To avoid performance hit on in-order processors, modern compilers "schedule" an unrelated instruction into the "load delay slot". [ Early RISC machines provided the [ interlock for the cache miss, but NOT for the expected one-cycle [ delay (saved a little chip area and complexity that way). This is [ an architecturally visible *delayed load*. Compiler had to fill [ delay slot with nop if nothing else was available. Note, however, [ that if you add the interlock in later machines (which they did) you [ can stop inserting NOPs in new programs, and still have backward [ compatibility. H&P report about half the loads force stalls in naive code (with a single-cycle load delay); about a quarter with a good compiler. More aggressive solution is to do out-of-order execution: lets instructions behind you continue even if you're stalled. Originally used in supercomputers; standard in modern microprocessors with a small number of cores. More on this later. ------------------------------ control hazards -- branches Good branch prediction is very important. ~60% accuracy on average if you predict all branches are taken. ~65% accuracy on average if you predict *backward* branches are taken. This does a good job with loops, but not if-then-else. Good dynamic history predictors do better than 90%. HW shadow stack for return addresses similarly does very well. No bubble in PIPE for unconditional jump or call. PIPE assumes jumps are always taken. For mispredicted conditional jump, know address of next instruction when execute stage completes. Forward to fetch unit for 2-cycle delay: jl F D E M W xxx F D - - - yyy F - - - - mov F D E M W Note that for two cycles we're working on the wrong instructions. Turn them into bubbles as soon as we realize it. For return there isn't an easy static prediction -- no analog to "always taken, with address in instruction". The authors don't want to add the complexity of the HW shadow stack to PIPE, so they simply always stall after a return. Note that even with forwarding down to the fetch stage, we suffer a 3-cycle bubble: ret F D E M W xxx F - - - - yyy F - - - - zzz F - - - - mov F D E M W [ Early RISC machines had *delayed branches* -- reduced impact of [ mispredictions by always executing instruction after the branch. [ Unfortunately, we're stuck with those for backward compatibility. [ Nullifying branches (in recent versions of legacy RISC ISAs) [ Undo the effect of the instruction in the delay slot if the static [ prediction turns out to be wrong. Relieves the compiler/assembler [ of the need to ever put NOPs in delay slots. ------------------------------ This isn't the end by any means! This is still a single-issue in-order design, with a simple 5-stage pipe. Just imagine what a multi-issue, out-of-order processor with register renaming and aggressive branch prediction looks like. That was mainstream a decade ago. High-end cores may also have multithreading trace caches value speculation lock elision run-ahead threads thread-level speculation Power-efficient cores won't have most of that. Tomorrow's chips may have very large numbers of cores (100s or 1000s) heterogeneous architectures or even ISAs (witness Cell, PowerEN, cell phones) "clustered" organizations, with multiple "sub-cores" multiple clock domains, with dynamic voltage and frequency scaling device interfaces full memory systems We can expect variants of the RISC/CISC debate to resurface. At what point does it pay to simplify the design in order to make it smaller/cheaper/cooler -- e.g. to fit more processors on chip? ------------------------------ Additional challenges for pipeline implementation interrupts other terms trap -- deliberate program-caused exception -- non-deliberate program-caused "interrupt" sometimes includes these; sometimes restricted to external quick overview of how they work vector table masking, sometimes at multiple levels RFE instruction Ideally should be PRECISE, meaning (a) when you wake up in the interrupt handler all previous instructions have completed and no future instructions have made any changes to the machine state; (b) if the interrupt was an exception, user-mode PC points at the faulting instruction. Precision typically not important for external interrupts (as long as it's possible to resume afterward), or for fatal program-caused ones, but crucial for recoverable program-caused ones. Make it easier to resume after I/O or page fault. Required for standards comformance with IEEE FP arithmetic. Precision a challenge due to branch delay slots (explain) post-interrupt PCs may be non-sequential OOO execution multi-cycle (e.g. FP) instructions cache misses TLB misses (w/ HW reload) mispredicted branches don't want to announce an exception on an untaken path Options (for our simple PIPE machine, w/out most of the above) (1) ignore the problem -- naah (2) don't issue an instruction until the previous one is guaranteed to complete (MIPS R2/3K) -- simple but potentially performance-limiting (3) stop dispatching; drain pipe; deliver on commit. (This is what PIPE does.) (4) generate right away, but save whole pipe contents to memory or special registers. Interrupt handler can choose to toss (if exception is fatal), resume from (if handling can safely be imprecise -- e.g. I/O completion), or emulate through to precision in SW (some SPARCs have done this) ------------------------------ accessing memory PIPE design above assumes memory responds in a single cycle, which forces a one-cycle delay if we need the result of the M stage in the next E stage. L1 may be that fast on some machines, but often needs 2 or even 3 cycles on modern machines. Must have interlocks. Also, address translation can lead to TLB and page faults. HW TLB reload stalls the pipeline. SW TLB load and page faults cause interrupts, which the OS has to handle before restarting the faulting thread. ------------------------------ register renaming deal with W-W and R-W dependences (we've been focusing on W-R so far; they're the only ones that matter in an in-order, single-issue pipeline that updates registers only in the W stage) complicated differentiate between register *names* and physical registers. hardware tracks dependences among instructions. When a register name is used for separate values in different instructions, the hardware may allocate separate physical registers and execute the instructions simultaneously. Simple example: r1 = A B = r1 r1 = C # this use of r1 is independent of the first D = r1 If we change this to p12 = A B = p12 p35 = C D = p35 Then 2nd pair of instructions can happen in parallel with 1st pair, or even before -- e.g., if load of A is a cache miss. [ Bigger example: [ a := b + (c * d); [ e := f + (g * h); [ [ Suppose the compiler has generated: [ [ 1 r1 = c [ 2 r2 = d [ 3 r1 = r1 * r2 [ 4 r2 = b [ 5 r1 = r1 + r2 [ 6 a = r1 [ 7 r1 = g [ 8 r2 = h [ 9 r1 = r1 * r2 [ 10 r2 = f [ 11 r1 = r1 + r2 [ 12 e = r1 [ [ The hardware may convert this to [ [ 1 R1 = c [ 2 R2 = d [ 3 R1 = R1 * R2 [ 4 R3 = b [ 5 R1 = R1 * R3 [ 6 a = R1 [ 7 R4 = g [ 8 R5 = h [ 9 R4 = R4 * R5 [ 10 R6 = f [ 11 R4 = R4 + R6 [ 12 e = R4 [ [ Now instructions 3 and 4 can execute in parallel (they couldn't before), [ as can instructions 9 and 10. Moreover instructions 1-6 can execute [ in parallel with instructions 7-12. ============================== Modern high-end microprocessors ------------------------------ Superscalar machines Start more than one instruction per cycle, assuming they're independent. May still execute all instructions in order. ------------------------------ Out-of-order execution Decouple instruction fetch/decode from instruction execution from instruction completion. Original concepts developed for the IBM 360 model 91 in the late 1960s. Moved into microprocessors only in the late 1990s. Several possible implementation strategies. Here's a *very* brief outline of one. It's roughly analogous to fig. 5.11 in the book, but structured a bit differently. I-cache fetch fetch queue dispatch -- includes rename and tracking of dependences; instruction goes to appropriate issue queue and to ROB (called "retirement unit" in book) issue "queues" issue execution units completion reorder buffer (ROB) commit Fetch unit fetches into fetch queue, which serves to even out rates of I-cache and dispatch unit (dispatch unit tends to be burstier, though I-cache can be in the event of misses). Dispatch unit does register renaming and tracks dependences among instructions. Each instruction is tagged with an indication of the inputs (physical registers) it needs before it can proceed, and is placed in an issue queue. Simultaneously it is placed in the ROB. Typically there are separate issue queues for integer, FP, and load/store instructions. Instructions issue (begin execution) when all their operands are ready and an appropriate hardware unit is available. This means they can leave the "queue" out-of-order. So the issue queue is really more like a set. Ideally you'd like to issue the most critical ready instruction(s) (the ones that future ops depend on most). This is hard to figure out. Oldest is a so-so approximation. Various heuristics are used in practice. Completed instructions are tagged in the ROB and their results made available to future instructions, both via writeback to the register file and via direct forwarding to the issue queue entries of dependent instructions. The ROB is structured as a queue, with instructions in program order. When a completed instruction reaches the head of the queue it *commits* and all its resources are reclaimed. In-order commit exists for the purpose of supporting precise interrupts. Note that branch prediction is crucial to keep the pipeline going, and branch mis-predicts have a *serious* penalty. It's even worse in modern Pentium processors, which have an extra unit on the front end to translate IA32 instructions into RISC-like internal instructions. A branch has to change the *IA32* instruction stream. ------------------------------ fine-grain multithreading tolerate cache misses by switching threads on each cycle, or perhaps on each cache miss. ------------------------------ simultaneous multithreading (SMT) Multiple threads with separate front ends (fetch, decode), register sets, and reorder/writeback stages, but shared functional and memory units. Can issue instructions from different threads in same cycle. ------------------------------ chip-level multiprocessors (CMP, aka multicore) Multiple full processors on one chip. May share lower levels of on-chip cache. May have fast intra-chip communications. Each processor is called a _core_. Most manufacturers now sell multicore ships -- 2 to 8 cores are common; more in some cases. [Cavium Networks makes a 32-core (MIPS ISA) chip. Tilera makes one with 100 cores (!) ISA similar to MIPS. These are moving targets.] "Hyperthreaded" Pentiums are 2-way SMT. Recent Niagara chips from Oracle are 8-way multithreaded. Operating system treats all of these (FGM, SMT, CMP) like a multiprocessor. Aggressively superscalar OOO designs are very power hungry. CMPs made from simpler processors may be more energy efficient. They will require more aggressively multithreaded code, however. Concurrent programming is going mainstream. The next big change is likely to be heterogeneity. Intel, AMD, and Nvidia already make integrated CPU/GPU chips. Qualcomm and other cell phone manufacturers do slightly less integrated SoCs. << Describe how GPUs work >>