Ahmed Masry's picture

8 3 4

Ahmed Masry PRO

ahmed-masry

·

https://ahmedmasryku.github.io/

Ahmed_Masry97

AI & ML interests

Multimodal Chart Understanding, Multimodal Document AI, Multimodal Vision - Language Models,

Recent Activity

posted an update 1 day ago

Happy to announce AlignVLM 📏 – a novel approach to bridging vision and language latent spaces for multimodal understanding in Vision-Language Models (VLMs) 🌍📄🖼 🔗 Read the paper: https://huggingface.co/papers/2502.01341 🧐 What’s the challenge? Aligning visual features with language embeddings remains a major bottleneck in VLMs. Existing connectors such as Multi-layer perceptron (MLPs) often introduce noise that degrades performance. ❌ 🎯 Our Solution: ALIGN Connector We propose AlignVLM, a method that maps vision features into a weighted average of LLM text embeddings, ensuring they remain in a space that the LLM can effectively interpret. ✅ 🔬 How does it perform? We compared ALIGN against common connectors like MLPs, Perceiver Resampler, and Ovis trained under similar configurations. The results? ALIGN outperforms them all 🏆 on diverse document understanding tasks 📄. 📊 Meet the AlignVLM Model Family! We trained Llama 3.1 (1B, 3B, 8B) using our connector and benchmarked them against various models. The results: ✅ AlignVLM surpasses all Base VLMs trained under similar configurations. ✅ Our models also perform competitively against Instruct VLMs such as Qwen2-VL and InternVL-2.5 🚀. 🤔 What about robustness to noise? We injected Gaussian noise (μ=0, σ=3) into the vision encoder’s outputs before feeding them to the connector: ✅ ALIGN Connector: Minimal drop (↓1.67%) – proving its high robustness! ❌ MLP Connector: Severe degradation (↓25.54%) – struggling with noisy inputs. Code & model weights coming soon! Stay tuned! 🔥

authored a paper 2 days ago

AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding

upvoted a paper 2 days ago

AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding

View all activity

Organizations

ahmed-masry's activity

commented a paper 2 days ago

AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding

Paper • 2502.01341 • Published 3 days ago • 31 •

New activity in ahmed-masry/unichart-qa-data about 1 month ago

Dataset Source?

#3 opened about 1 month ago by

New activity in ahmed-masry/ColFlor 3 months ago

Update README.md

#1 opened 3 months ago by

New activity in ahmed-masry/ChartGemma 5 months ago

how the handle hallucination?

#2 opened 6 months ago by

New activity in ahmed-masry/unichart-table-data 6 months ago

License

#2 opened 6 months ago by

New activity in ahmed-masry/chartgemma 6 months ago

Missing license

#1 opened 6 months ago by

commented 2 papers 7 months ago

ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild

Paper • 2407.04172 • Published Jul 4, 2024 • 23 •

ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild

Paper • 2407.04172 • Published Jul 4, 2024 • 23 •

New activity in ahmed-masry/ChartInstruct-LLama2 7 months ago

This was working great until recently where it errors out

#1 opened 7 months ago by