Custom AI beyond LLMs: Vision, Audio, Multimodal

Introduction

Custom AI solutions extend beyond language models, encompassing advanced vision, audio, and multimodal technology capabilities. These diverse AI models offer unique adaptability, showcasing innovation in fields that transcend traditional text analysis. From sophisticated image generation to intricate audio processing, the scope of customizable AI is vast and full of untapped potential.

‍

Type of AI Models

Text

Text AI models are designed to understand, interpret, and generate human language. They are used in chatbots, content generation, and language translation applications.

Commercial Example - ChatGPT, developed by OpenAI, is a conversational AI model for generating human-like text responses.

Open Source Example - Mistral offers open-weight models like Mixtral 8x7B, known for their efficiency and adaptability in various use cases, supporting multiple languages and coding abilities.

Vision

Vision AI models specialize in interpreting and generating visual content. They are crucial in applications like image generation, enhancement, and analysis, using deep learning to understand and recreate visual elements from given data inputs.

Commercial Example - DALL-E, developed by OpenAI, represents a significant advancement in vision AI. It's a text-to-image model capable of creating complex and creative images from textual descriptions. This model exemplifies the commercial application of vision AI in creative and design fields.

Open Source Example - Stable Diffusion is a notable open-source counterpart in vision AI. Stable Diffusion is particularly noteworthy for its ability to operate in latent space, which allows for the high-quality and flexible generation of images conditioned on various inputs like text or bounding boxes. It stands out for its efficiency and ability to run on consumer-grade GPUs. Trained on a diverse dataset, Stable Diffusion can generate detailed and varied images from textual prompts. One of its key features is the flexibility of end-users to fine-tune the model for specific use cases, such as generating personalized or stylistically unique images. The model has been developed to emphasize computational and memory efficiency, making it accessible to many users and applications​​​​​​.

Audio

Audio AI models, particularly in speech-to-text, are pivotal in transforming spoken language into written text. They are widely used for transcriptions, voice-controlled applications, and accessibility tools.

Commercial Example - AWS Transcribe: a service by Amazon Web Services, offers advanced speech recognition capabilities, allowing for the transcription of audio files into text. It's designed for high accuracy and can handle various accents and languages. For more information, visit AWS Transcribe.

Open Source Example - Whisper, Developed by OpenAI, Whisper is an open-source speech recognition system known for its robust performance across multiple languages and contexts. It stands out for its adaptability and accuracy in different environments and applications. Further details can be found at OpenAI Whisper.

Multimodal

Multimodal AI models integrate and analyze data from multiple different modes or types of input, such as text, images, and audio. These models can understand and generate complex content that spans different forms of media.

Commercial Example - GPT-4V, represents a cutting-edge development in multimodal AI, offering capabilities that combine text and visual inputs for diverse applications.

Open Source Example - LLaVA, is an open-source multimodal AI model that showcases advancements in integrating vision and language processing. More information about LLava can be found at LLava.

Video

Video AI is a developing field that applies the principles of image AI to videos. These models are designed to interpret and generate video content, often analyzing data frame-by-frame.

The "ViViT: A Video Vision Transformer" research paper is a key reference in this domain. This study explores the use of transformer-based models for video classification, a method that has shown promising results, especially in handling spatiotemporal data efficiently. 

The research emphasizes the effectiveness of these models even on comparatively small datasets, underscoring their potential in various video-related applications.​

‍

Two High-Level and Major Approaches

Two major approaches take center stage when understanding the detailed workings of large language models (LLMs) like ChatGPT: Next Token Prediction and the Diffusion Approach.

‍

Next Token Prediction in AI Models

Next Token prediction is a core mechanism in many AI models, particularly language models. It involves predicting the subsequent word or token in a sequence based on the context provided by the preceding tokens. This technique is fundamental in generating coherent and contextually relevant text in models like ChatGPT.

For a more detailed exploration of next token prediction and its role in the mechanics of Language Models (LLMs), refer to our previous blog post, "What are the mechanics inside LLM?" 

Token interpretation

1. Text: Part of the Word

Token interpretation in text-based AI involves analyzing segments of words rather than whole words. This approach allows for a deeper understanding of language nuances and syntax. It enables the AI to handle complex linguistic elements such as morphemes and compound words more effectively. This granular focus leads to more precise and contextually appropriate text generation, making these models highly adaptable to the intricacies of human language.

2. Image: Pixel

Image-based AI models interpret each pixel as a token, especially in multimodal and open-source platforms. These models predict the next pixel using data from surrounding pixels, much like text-based models predict the next word segment. This pixel-focused approach allows AI to generate detailed images precisely, playing a crucial role in image generation and enhancement tasks.

3. Multimodal

Multimodal AI models synthesize information from text and images, interpreting words and pixels together to generate responses. The attached figure shows that these models integrate visual data encoded by a vision encoder with textual instructions processed by a language model. 

The combined data is then used to produce a language response that accurately reflects the image's content, context, and associated text. This integrated approach allows multimodal AI to understand and respond to complex queries that require an analysis of visual elements alongside textual data.

‍

Diffusion Approach

The diffusion approach in AI, exemplified by models like Midjourney and DALLE, represents a generative technique that progressively refines images from random noise to detailed visuals. For an in-depth look into the latent diffusion models that underpin this process, further details are available on the following research page.

‍

Conclusion

The potential to personalize AI goes beyond language and can be applied to various domains. Vision, audio, and multimodal AI models demonstrate an extraordinary capacity for adaptation, driven by underlying mechanisms that are remarkably similar across different modalities.

Text AI models dissect language to its smallest units, offering nuanced interpretations. Vision AI transforms pixel data into complex images, and audio AI converts sound waves into meaningful text. Multimodal AI, perhaps the most sophisticated of all, weaves words and visual elements together to respond intelligently to multifaceted inputs.

The same principles that allow us to tailor language models to specific needs also apply to vision, audio, and multimodal models. Whether it's through next token prediction or the diffusion approach, the customization techniques are fundamentally interconnected. By understanding these relationships and the shared techniques behind them, we are better equipped to push the boundaries of AI even further, framing solutions that are as diverse and dynamic as the challenges they aim to address.

‍

References

1. Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual Instruction Tuning. Retrieved from arXiv:2304.08485v2.
2. OpenAI. (2023). Introducing Whisper. Retrieved from OpenAI. 
3. Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). LLaVA: Large Language and Vision Assistant. Visual Instruction Tuning. NeurIPS 2023 (Oral). Retrieved from https://llava-vl.github.io/.
4. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., et al. (2024). Mixtral of Experts. Retrieved from  arXiv:2401.04088v1.
5. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., & Schmid, C. (2021). ViViT: A Video Vision Transformer. Retrieved from arXiv:2103.15691.