PrismML brings tiny LLMs to Qualcomm-powered smart glasses

Key facts

• PrismML has created a version of its tiny language models for smart glasses running on Qualcomm's Snapdragon chips.
• The 1-bit Bonsai LLM is a 2-billion-parameter model tuned for vision and language.
• The model allows wearers to ask what they are looking at in real time.
• PrismML's goal is open-weight AI that runs on devices and makes better use of existing computing power.
• The model fits 4x larger models onto the same memory footprint as 4-bit precision models.
• The model generates tokens at more than 2x the speed of 4-bit precision models.

PrismML, an AI Lab founded by Caltech researchers and advised by UC Berkeley’s Ion Stoica, has developed a specialized version of its compact language models for smart glasses that utilize Qualcomm's Snapdragon platforms. The company showcased its 1-bit Bonsai LLM at Qualcomm's Snapdragon Summit, demonstrating its ability to run locally on AI smart glasses built on the Snapdragon AR1 Gen 1 Platform. This model, a 2-billion-parameter vision-language model, is designed to enable real-time understanding of what the wearer is observing and respond accordingly. PrismML's broader ambition is to advance open-weight AI that operates directly on devices, maximizing the use of existing computing power and offering an alternative to cloud-dependent proprietary AI solutions. This approach prioritizes on-device inference, which enhances privacy and reduces latency, particularly crucial for wearables with limited power and thermal capacity. PrismML collaborated with Qualcomm Technologies to optimize the model's weights and architecture for the Qualcomm Hexagon NPU, ensuring efficient performance on the constrained hardware of smart glasses. While this represents a significant step towards PrismML's vision, no smart glasses incorporating these models have been announced for consumer release yet.