Zyphra has announced the release of its new Zamba2-VL models, a family of vision-language models (VLMs) designed to significantly reduce the time-to-first-token by approximately an order of magnitude. The release includes models with 1.2B, 2.7B, and 7B parameters, leveraging a hybrid Mamba2–Transformer architecture for enhanced accuracy and efficiency.
What are Zamba2-VL Models?
Zamba2-VL models are open-source vision-language models that integrate a hybrid Mamba2–Transformer backbone. These models are capable of reading images and text simultaneously, making them effective in applications such as chart, document, and photo analysis. Zyphra has aimed to achieve competitive accuracy with lower latency through this innovative architecture.
The architecture of Zamba2-VL includes a vision encoder and a lightweight MLP adapter, which projects image features into the language model’s space. This allows the models to support single and multi-image understanding, crucial for applications requiring comprehensive visual context.
How Does the Hybrid Mamba2-Transformer Architecture Work?
The hybrid Mamba2-Transformer architecture utilized in Zamba2-VL models combines Mamba2 state-space layers with shared transformer blocks. This design facilitates linear time operation with a fixed-size state, while a small number of shared attention layers ensure in-context retrieval capabilities. This hybrid approach balances expressivity and efficiency, offering a unique advantage over traditional VLMs.
Zamba2-VL models use the Mistral v0.1 tokenizer, trained on a substantial dataset of 100 billion tokens, sourced from open web datasets. This comprehensive training ensures the models can handle various vision-text and pure-text data seamlessly.
What Benchmarks Did Zamba2-VL Models Achieve?
Zamba2-VL models were evaluated across 14 benchmarks, including chart, diagram, and document understanding, as well as general perception and reasoning. The models demonstrated strong performance, particularly in visual counting and document understanding, with a 90.9% score on the DocVQA test for the 2.7B model.
While Zamba2-VL models excel in certain areas, they lag behind larger models in knowledge-heavy reasoning tasks, such as those found in the MMMU and MathVista benchmarks. Nevertheless, their performance in document and visual counting applications positions them as a practical choice for specific use cases.
Where Can These Models Be Applied?
Zyphra’s Zamba2-VL models are ideal for applications requiring efficient document and form extraction, such as invoice parsing and receipt digitization. Their strengths in visual counting make them suitable for retail and inventory management tasks. Additionally, their low latency makes them well-suited for on-device assistants and edge deployments.
With the ability to process long visual inputs efficiently, these models offer significant advantages in handling complex visual data, such as multi-page PDFs, further broadening their potential applications.
Frequently Asked Questions
What makes Zamba2-VL models different from other vision-language models?
Zamba2-VL models use a hybrid Mamba2–Transformer architecture, providing lower latency and competitive accuracy in processing vision and text data simultaneously. This approach allows for efficient on-device and edge deployment.
What sizes are available for the Zamba2-VL models?
The Zamba2-VL family includes models with 1.2B, 2.7B, and 7B parameters, catering to different deployment needs ranging from edge devices to more robust applications.
How do Zamba2-VL models perform in document understanding tasks?
The 2.7B Zamba2-VL model scored 90.9% on the DocVQA test, showcasing strong performance in document understanding tasks, making it suitable for applications like invoice parsing.
What are the technical requirements for deploying Zamba2-VL models?
Zamba2-VL models require a CUDA GPU for optimal latency, as the optimized Mamba2 kernels are designed for GPU acceleration. Deployment involves using Zyphra’s transformers fork and additional dependencies.
Related coverage
Sources
- MarkTechPost – Zyphra Release Zamba2-VL: Hybrid Mamba2–Transformer Vision-Language Models
- Zyphra – Our Work: Zamba2-VL
