Rate Limits Eliminated with an Auto-Scaling LLM Deployment on Azure
A self-managed Mistral deployment with autoscaling for a German AI data-query platform
About the customer
Our client is a German AI platform that helps businesses query, extract and analyze large datasets, including unstructured sources, using machine learning and natural language processing (NLP). Users ask questions in plain language, and OpenAI’s large language models (LLMs) answer them. As the datasets grew, the client kept hitting OpenAI’s rate limits, the caps on how many requests can be sent in a given time, even after raising its quotas. We deployed an LLM on Microsoft Azure that scales automatically with demand.
What needed solving
Built on OpenAI
The client used OpenAI’s LLMs to let users query vast datasets in natural language
Frequent rate limits
As its datasets grew, the team hit rate limits frequently, despite increasing their quotas
Limited scale
The bottleneck hurt throughput and user experience, limiting the platform’s ability to scale
How we built it
LLM deployment
Deployed the Mistral model with vLLM, a tool for running LLMs efficiently, on Microsoft Azure for high availability. vLLM’s PagedAttention makes better use of the GPUs.
Autoscaling
SkyPilot adjusts the number of instances to the workload, and releases resources when demand is low.
Load balancing
SkyPilot spreads requests across multiple LLM instances.
Version control and deployment
Docker and GitHub handle version control and deployment automation.
The client can handle high-volume queries from its business customers
Technologies
Domains
Have a Query? Let's Have a Conversation
Do you have a project in mind? We’d love to know what it is. Email us or send a message from our contact page.
Have a similar problem?
Tell us what you are trying to solve and our team will get back to you.