Why LLM agnostic solutions are the future of dev tools

We have been spoiled by choices. With so many AI models, tools, and LLMs available, we no longer depend on one for all tasks. This brings us to an important question: what does LLM agnostic mean? 

For instance, I use OpenAI’s o1 model for code generation, especially when the task at hand is more critical and requires more thinking, and I use Claude 3.5 sonnet for generating texts. The main reason I do so is that after multiple tests and trials, I have seen that certain models do some tasks better. 

The same applies when building an application. You want to give your users the flexibility to use different LLMs for different tasks, we already see this happening in tools like Pieces and GitHub Copilot. In this article, we will learn more about what LLM agnosticism is and how it is helpful.

What is LLM agnostic?

By definition, being LLM agnostic means that a system or application is not dependent on a particular LLM and can work with different LLMs. 

More and more applications are built with an LLM agnostic architecture, as it provides more flexibility to users who can use different LLMs for specific tasks without being dependent on a single LLM provider. Being LLM agnostic also helps in building more resilient software.

 It also allows developers to use task-agnostic prompts to design queries that work consistently across different providers without requiring task-specific tuning. 

Recently, on January 23rd, there was a huge outage of ChatGPT along with disruptions to the GPT-4, 4o, and mini models. Building a system that is model agnostic, supports 

multiple models, help in avoiding these issues and building a more resilient tool.

You need a tool that allows applications to dynamically leverage the best AI for a given task, ensuring flexibility and improved performance.

Why build an AI-agnostic architecture for your product?

One thing we all agree on in the AI landscape is how fast it is evolving. This includes not only technological advancements but also acquisitions, company partnerships, executives quitting jobs, and questions about regulation.

These have led to people choosing and moving from one LLM to another. If your software or application has the ability to choose from models, it can also help in onboarding more users.

AI also brings with it a lot of rules and regulations. New laws and regulations are being developed in countries to address the potential impact of AI.

For example, Meta decided not to release its new open-source LLM in the European Union due to what it describes as "regulatory uncertainty". This is where an LLM agnostic architecture shines. 

If your tool has the ability to choose from multiple models and even run models locally, then users in the EU can continue to use your tool, which wouldn’t have been the case if it depended only on one LLM.

Building an LLM agnostic architecture

The LLM Agnostic architecture is built in a way that allows the integration of multiple LLMs from various providers. 

At its core, it decouples the application logic from the underlying LLM implementations, which allows switching between different LLMs.

Here’s all the layers that it contains:

• Interface Layer – It acts as the entry point for users or clients. This layer usually has APIs to abstract the interaction with LLMs.

• Application Logic Layer – This layer acts as the brain, where the main logic is added.

• Model Layer – It provides access to various LLMs.

• Data Layer – This manages context storage and allows retrieval for more accurate responses. This is also useful for Retrieval Augmented Generation (RAG).

• Observability Layer – This layer tracks the system performance. 

• Security Layer – This layer makes sure that the LLMs maintain safety and compliance guidance.

Benefits of an LLM agnostic architecture

There are multiple benefits to building an architecture that is LLM agnostic. Here are the key ones:

1. No vendor lock-in: In an LLM agnostic system, users are not tied to a particular LLM. They have the flexibility to switch between LLMs and use it for specific tasks.

2. Better performance and scalability: You can balance workloads across multiple LLMs to ensure consistent performance under high demand.

3. Faster adoption of newer models: You can modify your software to adapt to newer models without having to re-architect the system.

4. Increased reliability: Since the architecture uses multiple LLMs, it builds a failover mechanism, thus reducing the risk of downtime from a single provider.

5. Data privacy: Tools like Pieces that offer support to multiple LLMs, offers models that can be used locally.

6. Better integration: LLM agnostic architecture makes it easy to integrate vector databases, RAG pipelines and even APIs.

Challenges in an LLM agnostic architecture

While building an LLM agnostic architecture comes with many benefits, it also presents a few challenges:

• Complexity in building: Creating a system with a unified API layer to handle multiple I/O formats is complex, along with managing workflows that route requests to different LLMs and error handling.

• Performance variation: Different LLMs provide different results for a particular prompt and might also have different response times, which can lead to a decreased user experience.

• Continued maintenance: As the LLMs keep evolving, there will be frequent updates to APIs, which can require constant updates to the architecture.

• Cost management: LLM providers charge based on token usage, model type, and latency requirements, making it hard to predict costs.

• Lack of customization: While LLM agnostic architecture provides flexibility for various integrations, it is hard to fine-tune or perform model-related optimizations.

Best practices for implementing LLM agnostic system

Now that we understand the meaning of an LLM agnostic system, how it is built, its perks, and its cons, here are some best practices you can follow for implementing an agnostic AI architecture:

1. Create a unified layer that abstracts the differences between AI/LLM providers.

2. Build a logic that routes tasks to specific models based on performance, cost and latency.

3. Use frameworks like LangChain or Prefect to manage multi-model workflows and allow chaining tasks across different models.

4. Introduce logic to handle failover and fallback mechanisms.

5. Use local or open-source models when handling regulated data to comply with laws like GDPR or HIPAA.

6. Implement observability tools to track model performance, latency, token usage, and costs.

Conclusion

AI is growing tremendously fast. Just last week, we saw our notifications filled with the new DeepSeek model, and we saw how developers are already considering moving away from popular LLMs and choosing it. 

This is how fast the AI industry is moving. If the applications you build continue to be dependent on a single LLM, you will be forced to move to an LLM agnostic architecture to keep up and also continue your product’s growth. 

With this article, I hope you got an idea of what that would look like and things to keep in mind.

Here are some other resources that you might find helpful:

• Research paper on Generating Topic-Agnostic Conversations With LLMs

• MAML-en-LLM: Model agnostic meta-training of LLMs for improved in-context learning

• AI Wars: How Salesforce’s Agnostic LLM Approach Works