Euro open source specialist proposes LLM gateway to ease token bills

European open source database service provider Aiven says it has saved 80 percent of its AI token costs internally by moving to open-weight models and is building a gateway to help devs swap models in and out as required. The Finland-based company built its business on hosting and providing open-source databases as a service, and in the last couple of years has moved into machine learning. Speaking to The Register , Ville Lehto, Aiven’s VP of strategy, said that when dealing with open source AI infrastructure, it's not trivial to make something that can run across the different clouds. However, in giving its developers access to LLMs, the company wanted to manage its own token costs and control its "own destiny when it comes to AI usage." After a wave of debate around so-called token-maxxing, developers have generally become more interested in reducing their costs by using a mix of models, including cheaper Chinese alternatives to the Western frontier models, and open-weight models which come at a lower cost. Open-weight models have publicly released learned parameters, principally their weights and biases, and can be run locally or hosted at lower cost than the frontier models from OpenAI and Anthropic. Examples include the Llama Family from Meta, Qwen Family from Alibaba Cloud, China’s DeepSeek, Mistral AI’s Medium 3.5 and Devstral and, Google's Gemma. "When it comes to this token use, as we witnessed early on this year, there are a fraction of users that are absolutely heavy users, and when it comes to their spend, a lot of it could be saved by actually using open-weight models," Lehto said. He said Aiven faced a choice to either have strict token budgets or try to find a more efficient way of using tokens. Opting for the latter, it implemented an internal gateway to help developers work with open-weight models more easily, Lehto said. "We wanted to make it as frictionless as possible. We can offer our employees different open-weight models, be it Western models or the ones from Asia, and take on the frontier models. We've been focusing on the heavy users, and with them we see roughly 80 percent decline in spend." However, since the performance of open-weight and frontier models varies, it was also important to use other metrics, such as the cost per task, he said. Aiven built its gateway around the open source framework called LiteLLM , which it hosted. It added new features such as budgeting, access management, and added other open-source components. …