NeoMME: an efficient Multimodal-native and Multilingual Encoder
Hcompany introduces NeoMME, a new family of 260M and 800M multilingual multimodal encoders that process text and raw image patches in a single bidirectional Transformer, trained from scratch with a masked discrete-diffusion objective.
Intelligence analysis by Gemini 2.5 Flash
NeoMME represents a significant advancement in multimodal AI by integrating vision and language processing into a unified Transformer architecture, eliminating the need for separate pretrained vision towers or causal language models. This novel approach, trained from scratch, aims to enhance efficiency and reduce computational overhead, particularly for visual document retrieval tasks.
Imagine a super-smart robot brain that can understand both words and pictures at the same time, without needing two separate brains for each. NeoMME is like that brain, built to be really fast and efficient at finding information in documents that have both text and images, like finding a specific page in a picture book just by describing it, without having to read every single word first.
Analysis
NeoMME introduces a paradigm shift in multimodal encoding by consolidating text and image processing into a single, bidirectional Transformer. Unlike many existing visual language models that rely on separate pretrained vision encoders and causal language models, NeoMME is trained from scratch using a masked discrete-diffusion objective. This integrated architecture allows for a more streamlined computational path, which is particularly beneficial for tasks like retrieval and classification that do not require autoregressive text generation. The model's design emphasizes efficiency, offering significant improvements in throughput and storage reduction for visual document retrieval.
NeoMME Architecture
The NeoMME encoder backbone is a single Transformer designed to handle both text and image inputs natively. It comes in two sizes, 260M and 800M parameters, both sharing the same core architecture. Text inputs utilize factorized token embeddings, while images are divided into 32x32 non-overlapping patches and projected into the same Transformer encoder space. A key feature is dynamic image resolution, allowing the model to adapt token usage based on image information density, such as high-resolution document pages. The models boast a long bidirectional context length of 16,384 tokens, incorporating modern encoder improvements like grouped-query attention, 2D rotary position embeddings, and squared-ReLU MLPs. Furthermore, a BPE tokenizer with a 131k-token vocabulary was trained from scratch on diverse multilingual text, code, mathematics, and image transcripts.
Masked Discrete-Diffusion
NeoMME's pretraining strategy involves a discrete masked-diffusion text denoiser, learning from images by reconstructing masked text. For text-only examples, a corruption rate is uniformly sampled, and eligible text tokens are independently masked. For multimodal examples, corruption rates range from 0.3 to 1, ensuring image patches remain visible while NeoMME reconstructs the masked text. This high masking strategy forces the model to leverage visible image evidence, preventing reliance on language-only shortcuts and encouraging image-grounded descriptions. The pretraining dataset is a mix of multilingual text, code, mathematics, natural images, and document images, with each model processing approximately 524 billion packed input tokens, including 290 billion from text-only examples. The NorMuon optimizer was chosen to enhance data efficiency during this training process, given the relatively smaller text budget compared to some predecessors.
ColPali's Page-Image Approach
To evaluate the backbone's downstream performance, NeoMME was fine-tuned for visual document retrieval using the page-image methodology introduced by ColPali. This approach allows NeoMME-Retriever to rank document page screenshots directly, bypassing the often-complex and error-prone optical character recognition (OCR) steps typically required to extract text from PDFs. This method preserves crucial layout and contextual information present in the visual representation of a document page. The 260M model, at a matched 2048x2048 image input size on an NVIDIA L40S GPU, demonstrates impressive efficiency, encoding about 51 pages per second, which is approximately twice the throughput of ColModernVBERT. Additionally, hierarchical token pooling and asymmetric quantization significantly reduce late-interaction index storage from about 1.5 MB to a mere 6 kB per page, a 255x reduction, while maintaining over 95% of the baseline nDCG@10 performance.
Key points
- NeoMME is a family of 260M and 800M multilingual multimodal encoders.
- It uses a single bidirectional Transformer to process both text tokens and raw image patches, trained from scratch.
- The model employs a masked discrete-diffusion objective for pretraining, learning from images by reconstructing masked text.
- NeoMME-Retriever is fine-tuned for visual document retrieval, bypassing traditional OCR steps.
- It achieves significantly higher throughput and reduces late-interaction index storage by 255x while maintaining high performance.
NeoMME's integrated architecture and efficiency gains could lead to faster, more cost-effective multimodal AI applications, particularly in document processing and retrieval, making advanced AI more accessible and scalable for various industries. Its ability to bypass OCR steps could streamline workflows significantly.
While efficient, the model's relatively smaller text budget compared to some predecessors might limit its pure language understanding depth in certain complex scenarios. The novelty of its architecture also means its long-term robustness and adaptability across an even wider range of tasks beyond document retrieval still need extensive validation.
