DBRX Instruct

DBRX Instruct is a 132B parameter mixture-of-experts (MoE) large language model developed by Databricks. Specializing in few-turn interactions, it is an instruction fine-tuned version of DBRX Base. The transformer-based, decoder-only model is trained on 12 trillion tokens of text and code data. Utilizing a fine-grained MoE architecture, it activates 36B parameters per input, enhancing model quality. It supports a context length of up to 32K tokens and incorporates advanced techniques like rotary position encodings, gated linear units, and grouped query attention.