Chip Design: From Logic Gates to AI Accelerators

The Primitive: Multiply-Accumulate (MAC)

At the base of all AI computation is the multiply-accumulate operation. Reiner Pope explains that AI chips prioritize MAC because it is the fundamental step in matrix multiplication. By using a 4-bit multiplication followed by an 8-bit accumulation, designers account for the fact that errors accumulate during summation, requiring higher precision in the accumulator than in the multiplier. The circuit implementation uses AND gates for the multiplication and 'full adders' (3-to-2 compressors) to sum the partial products efficiently, following the Dadda multiplier architecture.

The Cost of Data Movement

A critical insight is that the actual logic (the MAC unit) is a tiny fraction of the total chip area. The vast majority of a traditional processor's area is consumed by the 'muxes' (multiplexers) and register files required to move data into and out of the logic units. In a standard CPU or CUDA core, the cost of selecting and routing data from a register file to the ALU scales linearly with the number of registers, often dwarfing the cost of the computation itself. This 'data movement tax' is the primary bottleneck in chip design.

Systolic Arrays and Scaling

To overcome the inefficiency of moving data for every single operation, modern AI accelerators like TPUs and Tensor Cores utilize systolic arrays. Instead of performing one MAC at a time, these architectures bake larger loops of the matrix multiplication algorithm directly into the hardware. By increasing the granularity of the fixed-function logic, designers can perform more computations per trip to the register file, effectively amortizing the cost of data movement. This shift explains why smaller bit-precision (like FP4) is so favorable: it reduces the quadratic area cost of the multiplier while allowing for higher throughput.

Architectural Trade-offs

The lecture contrasts CPUs, GPUs, and FPGAs. CPUs are optimized for general-purpose logic and branch-heavy code, requiring large caches and complex control logic. GPUs (and specifically Tensor Cores) are essentially massive arrays of tiny, specialized TPUs designed for high-throughput, predictable data movement. FPGAs offer reconfigurability at the cost of efficiency, as they must implement logic gates using programmable interconnects, which are significantly slower and more power-hungry than the hard-wired logic of an ASIC.