Multiverse Computing researchers developed a "Quantization-Aware Healing" method that shrunk OpenAI's GPT-OSS model from 120 billion parameters to 60 billion. The smaller model outperformed its larger counterpart on 7 of 9 tests by learning from the original full-precision model instead of a halfway copy.
This breakthrough in AI model compression could significantly reduce the cost and hardware requirements for running advanced AI, making powerful models more accessible to a wider range of developers and potentially enabling on-device AI applications.
Researchers at Multiverse Computing have developed a novel method called Quantization-Aware Healing, which allows for the shrinking of large AI models while simultaneously improving their performance. Published on the Hugging Face blog on August 25, the technique was applied to OpenAI's open GPT-OSS model.
The team reduced the model's parameters from 120 billion to 60 billion and compressed its memory to 4-bit. Counterintuitively, this smaller version surpassed the original, full-quality model on 7 out of 9 benchmark tests. The researchers explained that the success lies in teaching the shrunken model directly from the original, highly capable model, rather than from a compromised, halfway-compressed version.
This approach contrasts with traditional model shrinking methods, which often result in a loss of accuracy due to excessive compression. The researchers noted that their method treats the quantization step not as a cost-saving measure to be minimized, but as an opportunity for further "teacher supervision." This yields a model that is not only cheaper to operate and lighter in memory but also at least as accurate as its full-precision predecessor.
The Hypernova-60B model, the result of this process, requires approximately a quarter of the memory and half the parameters of the original. This significant reduction in resource requirements could enable powerful AI models to run on less powerful hardware, such as desktops or even mobile phones, making advanced AI more accessible to smaller labs and individual developers. The team has released the healed Hypernova-60B model as open weights on Hugging Face.