harishsg993010 /tiny-NPU

opensource NPU for LLM inference (this run gpt2)

★ 299⑂ 70👁 3⑃ 1● 2▤ 0
SystemVerilogCreated 2026-02-09
+0
Daily
299
stars
70
forks
3
watchers
1
prs
2
issues
0
releases

README

tiny-npu

A minimal neural processing unit in SystemVerilog, optimized for learning how NPUs work from the ground up.

Built with fully documented SystemVerilog RTL, two execution modes (LLM Mode for transformer inference and Graph Mode for ONNX models), a complete 128-bit microcode ISA, working GPT-2, LLaMA, Mistral and Qwen2 inference running on real HuggingFace weights, KV-cache optimization, an ONNX compiler supporting Gemm, Conv, Reduce, Exp/Log/Sqrt, Gather, Slice, Concat, BatchNorm, AvgPool, MaxPool, Pad, Resize, Cast and more, FP16 arithmetic (MAC, math LUTs, softmax, layernorm), async dispatch with DMA+GEMM overlap and performance counters, 50-model fuzz testing, and full Verilator simulation with cycle-accurate verification.

 Table of Contents

- Overview
- Architecture
  - NPU
  - Memory
  - Engines
- Execution Modes
  - LLM Mode
  - Graph Mode (ONNX)
- LLM Mode ISA
- LLM Mode Execution
  - Microcode Controller
  - Transformer Block
  - KV-Cache Decode
- Graph Mode ISA
- Graph Mode Execution
  - Graph Pipeline
  - Tensor Descriptor Table
  - ONNX Compiler
- Inference Demo
- Simulation
- Advanced Functionality
- Next Steps

 Overview

If you want to learn how a CPU or GPU works at a hardware level, there are excellent resources available - from textbooks to open-source implementations like tiny-gpu.

NPUs are a different story.

The AI hardware market is one of the most competitive in the industry. The internal architectures of production NPUs from NVIDIA (Tensor Cores), Google (TPU), Apple (Neural Engine), and others remain proprietary. While there are plenty of resources on using these accelerators through frameworks like PyTorch and TensorFlow, there's almost nothing to help you understand how they actually work at the RTL level.

The best option is to go through research papers and try to reverse-engineer what's happening, or dig through complex open-source implementations that prioritize completeness over readability.

This is why I built tiny-npu.

 What is tiny-npu?

 [!IMPORTANT]

 tiny-npu is a minimal, fully synthesizable neural processing unit in SystemVerilog, optimized for learning about how NPUs work from the ground up.

 It supports two execution modes: LLM Mode for running real transformer models (GPT-2, LLaMA, Mistral, Qwen2) with a 128-bit microcode ISA, and Graph Mode for running ONNX models with a dedicated graph ISA, tensor descriptor table, and 13 hardware engines (GEMM with INT8+FP16 dual MAC, Softmax, Reduce, Math LUT, Gather, Slice, Concat, AvgPool, MaxPool, Pad, Resize Nearest, Cast, plus inline element-wise). Graph Mode includes async dispatch with scoreboard-based DMA+GEMM overlap, hardware performance counters, and FP16 support throughout the pipeline. Both modes share the same compute core and on-chip SRAM.

With this motivation in mind, we can strip away the complexity of production-grade accelerators (multi-chip interconnects, HBM controllers, sparsity engines) and focus on the core elements that make ne

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON