Neural Magic focuses on sparsifying and quantizing LLMs for efficient deployment

Neural Magic, led by Professor Near Shavit, specializes in sparsifying and quantizing Language Models (LLMs) for optimal deployment. Their technique allows for the reduction of bits per weight and activation without sacrificing accuracy, resulting in smaller and more efficient models for execution on CPUs. By employing sparsity and quantization during the fine-tuning process, Neural Magic provides users with tools to optimize their models. Their aim is to enable running LLMs locally on devices, eliminating the need for cloud providers. They have achieved up to 4x reduction in the number of model bits and a 6-8x speedup compared to full-precision models on CPUs.