Multimodal AI: First hand experience integrating it into team's workflow

I have seen AI slowly become a part of all our tasks, whether writing a social media post, quick research, fixing grammar, writing code, or even decision-making.

We want AI to help us be more productive and optimize processes. While traditional AI can lack in a few areas, like accuracy, multimodal AI fixes this since it can handle multiple data inputs, resulting in a more accurate output.

The multimodal AI market has been growing quite a bit. We have seen companies like Mistral release their Pixtral 12B Multimodal AI model in September 2024.

If you have used GPT-4o, DALL-E, Gemini, or Claude, they are also examples of multimodal AI.

In this article, we will introduce you to multimodal AI, how it works, and how you, as a leader or a decision-maker, can use it to make more informed decisions.

How does unimodal differ from multimodal AI?

Before understanding what multimodal AI means, we must know how it differs from traditional AI. 

So, how is multimodal AI different from other examples of AI?

Unimodal AI, or traditional AI systems, are designed to work only on a single type of data. For example, text-only natural language processing and image-only computer vision. Since they can process only a single type of data, they provide great results in their specific domain but fail to deliver accurate results in domains that need an understanding of more than one data type.

On the other hand, multimodal AI provides better responses and is more contextually aware, as it can process multiple data types (image, text, audio, video) simultaneously. 

Multimodal AI was made possible because of transformer architecture. Two such examples are OpenAI's CLIP and DALL-E. 

Image source: NIPS'17

CLIP can classify images using natural language, while DALL-E can generate images using natural language. 

This shows how multimodal AI can work across different data types and can be used for complex tasks like Visual Question Answering and Natural Language for Visual Reasoning.

For a lot of us, multimodal AI is a new concept, even though we might use it almost every day (if you have used ChatGPT, it is a multimodal AI model). We are mostly familiar with generative AI and its differences with llms. 

So, here’s a quick overview on what is the difference between generative AI and multimodal AI: 

Generative AI generates results such as text, image, code, etc based on learned patterns, while multimodal AI processes multiple types of data to generate results. 

What does it mean for your team? 

Integrating Multimodal AI into your workflow can help maintain focus, improve decision-making, and automate repetitive tasks. 

According to a report by Gartner, 40% of Generative AI applications will be multimodal by 2027 and will have a transformational impact on businesses, as it enables use cases that weren’t possible before. 

So it is safe to say that multimodal AI will be the future of businesses. 

For individuals or teams, we are seeing growth in workflows becoming more AI-assisted. 

Here’s a recent update from Vercel, where they introduced an AI workflow that will help in checking writing style and guide for their docs. This is a great example of how teams will move ahead with integrating multimodal AI into their daily workflows.

Let’s begin…!

What is Multimodal AI?

Multimodal AI is a system that integrates and processes multiple data types also known as modalities to generate results. Unlike traditional AI models that can handle only a single type of data, multimodal AI models can take in different types of data such as text, image, audio, video to output more human-like accurate results that are also contextually aware.

Google defines multimodal AI as models that process and integrate multiple data types, such as text, images, audio, and video. Their Gemini models, like Gemini 2.0, enhance multimodal capabilities with real-time interactions and improved reasoning.

So, when is an AI model considered multimodal? It is considered multimodal when the model can process multiple modalities. 

“Modality” in AI refers to the type of data or information a system can understand. This includes:

• Text: Written or spoken language.

• Video: Combining visual and auditory data.

• Images: Visual data such as photographs and graphics.

• Sensor Data: Applications like glucose monitors that can be used in healthcare.

• Audio: This includes audio of any format: spoken words, music, and environmental sounds.

For example, think of an AI assistant for data analysts that can:

• Analyze data, research papers (text),

• Review reports, dashboards (images and videos)

And combine all of these data to give a result. This is more accurate and reliable than solutions that just take in data from one source. 

A multimodal AI system contains three components:

• Input module: The input module is made up of several unimodal neural networks. Each network handles a different type of data, collectively making up the input module.

• Fusion module: The input module collects the data and passes it on to the fusion module to process the information coming from each data type.

• Output module: This component outputs the results.

How does Multimodal AI work?

Historically, AI development has focused on unimodal models that are capable of processing only one type of data. 

However, this had many restrictions and generated inaccurate results that were not contextually aware. This led to the growth of multimodal AI. 

Understanding how multimodal AI is used in generative AI can help you get the most out of the commonly used AI tools that use the power of both. 

An article by IBM on Multimodal AI, mentions the three characteristics of multimodal AI: heterogeneity, connection, and interactions where Heterogeneity refers to the diverse qualities, structures, and representations of modalities. 

Connections refer to the complementary information shared between different modalities and interactions refer to how different modalities interact when they are brought together.

Multimodal model architecture usually includes an encoder, a fusion mechanism, and a decoder. 

Encoders – Encoders transform raw multimodal data into machine-readable feature vectors or embeddings that models use as input to understand the content.

Multimodal models often have three types of encoders for each data type - image, text, and audio.

• Image Encoders: Image encoders convert image pixels into feature vectors to help the model understand critical image properties.

• Text Encoders: Text encoders transform text descriptions into embeddings that models can use for further processing.

• Audio Encoders: Audio encoders convert raw audio files into usable feature vectors that capture critical audio patterns, including rhythm, tone, and context.

Fusion mechanism – The fusion mechanism helps in combining the embeddings that are transformed by encoders. This allows the model to be more contextually aware of the data coming in from all modalities.

Here are some key fusion strategies.

• Early Fusion: Combines all modalities before passing them to the model for processing.

• Intermediate Fusion: Projects each modality onto a latent space and fuses the latent representations for further processing.

• Late Fusion: Processes all modalities in their raw form and fuses the output for each.

• Hybrid Fusion: Combines early, intermediate, and late fusion strategies at different model processing phases.

Decoders – Decoders help in processing the feature vectors from different modalities to produce the required output. It can contain cross-modal attention networks to focus on different parts of input data and produce relevant outputs. 

Implementing multimodal AI in your day-to-day tasks

By now, you know how advanced multimodal AI is, especially in terms of being contextually aware and generating accurate results.

Here are some ways how you can use it for your day-to-day work:

For more thorough decision-making 

Since multimodal AI can take in data from multiple sources and use it to generate human-like responses, you can use it to combine text reports, dashboards, and real-time data streams to get data-driven insights. 

You can also use it to forecast trends using predictive analysis.

Here’s an example of how you can use long-term memory that already has the context of your code and saved snippets to write more accessible code using LLMs of your choice: 

For making communication smoother and more efficient

Multimodal AI models are great for creating summaries and translations (multiple languages, image to text, and vice versa). 

You can use it to summarize meetings, make announcements, and automate other types of communication workflows.

Here’s how I used an AI assistant to turn my meeting notes into actionable points:

By the way, you can test-drive it yourself since it’s free and has both local and cloud options. 

For improving productivity 

And by “productivity” I mean not you working 24/7 like a machine that doesn’t need its oil to be changed, but actually focusing on the tasks, having less stress going back and forth, and finally completing that to-do list of yours! 

AI is seen as an assistant and has been built to help us simplify repetitive processes. 

If you have to send daily reports of your work, you can build automations that generate summaries based on your work tickets/commits and send them to the appropriate channel.

Here’s an example of how I’m using AI to create a script that could help me send bulk emails and skip manually adding email addresses 👇

For better planning 

As a leader, you might often have to deal with "what-ifs" and questions that do not have any definitive answers. This is where multimodal AI can help. 

Since it can process information from multiple data sources, it can assist you in making strategic decisions.

When in doubt, use AI. Here’s how I have used ChatGPT to help brainstorm ideas for better promoting a hybrid conference:

For research and competitive analysis

Business and product decisions are often made with proper research and competitive analysis, and this is a lengthy process that takes days of research. 

You can use multimodal AI to generate an analysis as it can process multiple types of data such as financial reports, news articles, customer reviews, and more.

For example, I am using AI in the example below to help me build a new product that will help me compete with existing competitors.

Long story short, I could continue this list, but I hope I covered the main part as I’m using it in my day-to-day life. Saves me tons of time, and nerves. 

Speaking of engineers, my peers and team find it no less useful. 

How engineers can make use of multimodal AI

As an engineer, you can not only make use of multimodal AI but also build with it and take learnings to build better systems.

Here’s an example of me using multimodal AI to code better. 

I have been building an application with Next.js, again with Pieces,  and it was able to suggest code changes based on `Saved Materials` and `Captured Context`.

You can also make use of the APIs of the already built systems to help in processes such as:

• Design and prototyping – By building applications that can generate sketches or images from natural language.

• Customer support systems – Applications that can analyze text, audio, and images to provide smart customer support.

• Content creation – Developing tools that can help in creating video, image, and written content. For example, tools like Descript AI, and Whisper.

• Accessibility and assistive tech solutions – Building solutions that can help people with special needs, for example, being able to generate text from sign language.

Lastly, you can learn from the already-built systems to build solutions that are AI-enabled, more scalable, and flexible.

Here are some focus areas of learning:

• From OpenAI’s APIs, you can learn how using cloud AI platforms can offload multimodal AI processing and help in building scalable and cost-efficient models.

• From Google’s Gemini and OpenAI’s GPT-4o, you can learn how transformer architectures can help in building more contextually aware systems. 

This is a great example of how it is not always necessary to reinvent the wheel. If you have certain use cases, you should be using pre-trained multimodal models and customizing them instead of training from scratch.

From Tesla’s self-driving car, you can learn how to optimize real-time data fusion strategies for multimodal decision-making.

Key considerations when adopting multimodal AI

As we wrap up, we have seen how helpful multimodal AI is and how it can assist us. 

While it offers significant benefits, there are also important considerations to keep in mind:

1. Ethical and privacy concerns: Multimodal AI has access to a vast amount of data. If strict safeguarding practices are not in place, it can raise security concerns.

2. Computational demands: Since it processes multiple types of data, it might lead to increased costs and slower processing times.

3. Data quality and availability: Multimodal AI works with a lot of data, and at times it can be difficult to get high-quality data, which can lead to decreased accuracy in results.

Using multimodal AI for your teams' long-term game

Once you choose to use Multimodal AI in some capacity in your teams, here are some recommendations that I have:

• Multimodal AI depends a lot on data from various sources.

To make your applications better and more contextually aware, make sure you focus on data quality.

• Focus on privacy from day one. 

Keep in mind the ethical use of AI, and see how you can include privacy in the architectural design of the application you are building.

• Make your tools human-collaborative. 

Look for ways to build features that can complement human decision-making abilities with AI’s ability to act as a co-pilot. That’s what I did, through Pieces.

Get a free AI companion with long-term memory without risking your budget, security, and time, and check this out yourself. 

Lastly, here are some helpful resources if you want to learn more about Multimodal AI:

• Managing context for repository-aware code generation in no time

• Agent AI: Surveying the Horizons of Multimodal Interaction

• Exploring text segmentation in retrieval-augmented generation (RAG)