LPU Language Processing Unit
The LPU (Language Processing Unit) is a specialized inference chip launched by Groq, designed from the ground up for the computational patterns of transformer models, representing a thorough revolution in AI inference hardware architecture. Unlike the GPU's SIMT architecture, the LPU employs a deterministic execution model, eliminating thread scheduling overhead and cache miss latency, with inference latency as low as milliseconds and token generation speeds reaching thousands of tokens per second in LLM inference scenarios. Groq LPU is now available as an API service through GroqCloud, supporting mainstream open-source large models such as Llama 3, Mixtral 8x7B, and Gemma, which developers can call via standard REST APIs. The LPU's token generation speed far exceeds similarly priced GPU solutions, making it suitable for latency-sensitive applications such as real-time dialogue, code completion, voice assistants, and agent systems that demand extremely fast response times. Choosing an LPU requires accepting that its software ecosystem is still growing, but the differentiation advantage in inference speed is very significant in the current large model inference market.
This category includes the following AI accelerator chips/compute cards: