OpenAI has launched a series of significant updates to its API offerings, introducing vision fine-tuning capabilities, a new Realtime API, and model distillation tools. These advancements aim to provide developers with more sophisticated AI tools for building innovative applications across various industries.
The company has introduced vision fine-tuning on its GPT-4o model, allowing developers to customise the model's image understanding capabilities. This new feature is designed to enable enhanced visual search functionality, improved object detection for autonomous vehicles, and more accurate medical image analysis. According to OpenAI, developers can improve model performance with as few as 100 images, with larger datasets driving even higher performance.
OpenAI's newly introduced Realtime API enables developers to build low-latency, multimodal experiences in their applications. This API supports natural speech-to-speech conversations using six preset voices, similar to ChatGPT's Advanced Voice Mode. The company has also announced plans to add audio input and output capabilities to its Chat Completions API in the coming weeks.
In addition to these features, OpenAI has unveiled a Model Distillation suite, which includes Stored Completions, Evals (in beta), and Fine-tuning integration. This suite allows developers to distill the capabilities of larger models like GPT-4o into more cost-efficient models such as GPT-4o mini, potentially reducing operational costs while maintaining performance.
To encourage adoption, OpenAI is offering 1M free training tokens per day for GPT-4o vision fine-tuning through October 31, 2024.
The company has also introduced Prompt Caching, which offers automatic discounts on inputs that the model has recently seen. This feature provides a 50% discount and faster prompt processing times for cached inputs, potentially reducing costs and latency for developers using the API.
OpenAI stress their commitment to safety and privacy, stating that they continuously run automated safety evaluations on fine-tuned models and monitor usage to ensure applications adhere to their usage policies. The company also assures that fine-tuned models remain entirely under the developer's control, with full ownership of business data.