#028 Training Multi-Modal AI, Inside the Jina CLIP Embedding Model

Today we are talking to Michael Günther, a senior machine learning scientist at Jina about his work on JINA Clip.

Some key points:

• Uni-modal embeddings convert a single type of input (text, images, audio) into vectors
• Multimodal embeddings learn a joint embedding space that can handle multiple types of input, enabling cross-modal search (e.g., searching images with text)
• Multimodal models can potentially learn richer representations of the world, including concepts that are difficult or impossible to put into words

Types of Text-Image Models

1. CLIP-like Models
   • Separate vision and text transformer models
   • Each tower maps inputs to a shared vector space
   • Optimized for efficient retrieval
2. Vision-Language Models
   • Process image patches as tokens
   • Use transformer architecture to combine image and text information
   • Better suited for complex document matching
3. Hybrid Models
   • Combine separate encoders with additional transformer components
   • Allow for more complex interactions between modalities
   • Example: Google's Magic Lens model

Training Insights from Jina CLIP

1. Key Learnings
   • Freezing the text encoder during training can significantly hinder performance
   • Short image captions limit the model's ability to learn rich text representations
   • Large batch sizes are crucial for training embedding models effectively
2. Training Process
   • Three-stage training approach: 
     • Stage 1: Training on image captions and text pairs
     • Stage 2: Adding longer image captions
     • Stage 3: Including triplet data with hard negatives

Practical Considerations

• Similarity Scales
  • Different modalities can produce different similarity value scales
  • Important to consider when combining multiple embedding types
  • Can affect threshold-based filtering
• Model Selection
  • Evaluate models based on relevant benchmarks
  • Consider the domain similarity between training data and intended use case
  • Assessment of computational requirements and efficiency needs

Future Directions

1. Areas for Development
   • More comprehensive benchmarks for multimodal tasks
   • Better support for semi-structured data
   • Improved handling of non-photographic images
2. Upcoming Developments at Jina AI
   • Multilingual support for Jina ColBERT
   • New version of text embedding models
   • Focus on complex multimodal search applications

Practical Applications

• E-commerce
  • Product search and recommendations
  • Combined text-image embeddings for better results
  • Synthetic data generation for fine-tuning
• Fine-tuning Strategies
  • Using click data and query logs
  • Generative pseudo-labeling for creating training data
  • Domain-specific adaptations

Key Takeaways for Engineers

1. Be aware of similarity value scales and their implications
2. Establish quantitative evaluation metrics before optimization
3. Consider model limitations (e.g., image resolution, text length)
4. Use performance optimizations like flash attention and activation checkpointing
5. Universal embedding models might not be optimal for specific use cases

Michael Guenther

• LinkedIn
• X (Twitter)
• Jina AI
• New Multilingual Embedding Modal

Nicolay Gerold:

• ⁠LinkedIn⁠
• ⁠X (Twitter)

00:00 Introduction to Uni-modal and Multimodal Embeddings 00:16 Exploring Multimodal Embeddings and Their Applications 01:06 Training Multimodal Embedding Models 02:21 Challenges and Solutions in Embedding Models 07:29 Advanced Techniques and Future Directions 29:19 Understanding Model Interference in Search Specialization 30:17 Fine-Tuning Jina CLIP for E-Commerce 32:18 Synthetic Data Generation and Pseudo-Labeling 33:36 Challenges and Learnings in Embedding Models 40:52 Future Directions and Takeaways