Nvidia and Mistral AI Pack NeMo 12B AI Model into Smaller Form Factor

The Mistral-NeMo-Minitron 8B is a miniaturized version of the existing NeMo 12B AI model. Based on the “pruning and distillation” process, the miniaturized version can run locally on laptops and PCs.

Nvidia and Mistral AI launch a small AI model, Mistral-Nemo-Minetron 8Bwhich can be run natively on laptops and computers with less computing power. The model is a miniaturized version of the NeMo 12B and performs well in benchmarks with similar small models. The success of this smaller type is a combination of two techniques: “pruning and distillation”.

“Trimming and distillation”

The new Mistraal-NeMo-Minitron 8B mini AI model is a miniaturized version of the existing NeMo 12B. The model has been completely “pruned” from 12 billion to 8 billion parameters. To ensure this mini model remains efficient, the developers used a “pruning and distillation” technique.

“Pruning shrinks the neural network by removing model weights that contribute less to accuracy. During the distillation process, the team retrained this trimmed model on a smaller dataset to significantly improve accuracy, which was reduced by the pruning process,” said Brian Catanzaro, vice president of deep learning research at Nvidia. Blog post from Nvidia.

Read also

Authors Sue Nvidia Over Mishandling of NeMo Training

The optimized language model was trained on “a portion of the original dataset,” saving significant costs in terms of initial computation. The amount of computing power saved makes the model suitable for running natively on laptops and PCs, and it is said to run nine language-based AI benchmarks of the same size.

“Packaged as an NVIDIA NIM microservice, the model is optimized for low latency, which means faster responses to users, and high throughput, which equates to higher computational efficiency in production,” the blog post said. Furthermore, Nvidia is offering its own custom modeling service, AI Foundry, so that Minitron 8B can also run on smartphones.