Key facts
- OpenAI's Decisions API reduces latency from 1-3 seconds to 100-180 milliseconds.
- Speak cut its action time from 1.4 seconds to 0.13 seconds using the API.
- Newer Jev models from TypeSafe top benchmarks like JevMan, while OpenAI's model ranks lower on some indices.
OpenAI has launched its Decisions API, making a model designed for fast, low-cost AI decision-making widely available to users. This move enters a competitive market for specialized AI models, exemplified by TypeSafe's Jev, which has achieved a $7.6 billion valuation. OpenAI's offering is based on its ChatGPT Luna model, reconfigured to significantly improve inference speed and reduce costs. Nikunj Handa, product lead for OpenAI’s APIs, stated that the Decisions API has shrunk processing times from one to three seconds down to 100-180 milliseconds. The API was announced on September 29 and is now available to all customers, with added voice and visual capabilities and geographically targeted inference resources to ensure low latency. Early adopters like Box, Datadog, Ramp, and Speak have used the API to replace traditional LLMs for higher throughput, with Speak's CTO Andrew Hsu noting a reduction in action time from 1.4 seconds to 0.13 seconds.