Publications

Instead of generating coordinates as text, MolmoPoint points by selecting the visual tokens that contain the target concept via special grounding tokens, followed by finer-grained subpatch and location tokens. It achieves state-of-the-art image pointing and improves video pointing and tracking.

An open family of video-language models with strong video understanding and point-driven grounding. I led long-context post-training, scaling to 512+ frames and 65k+ token contexts with context parallelism; our 4B variant outperforms Qwen3-VL 8B on average across long-video benchmarks.

VideoNet: a benchmark of 1,000+ domain-specific actions across dozens of domains that tests video-language models with few-shot in-context examples. Fine-tuning Molmo2-4B on our ~500k video QA training set surpasses all open-weight 8B models on the benchmark.

TrajViT tokenizes videos into panoptic sub-object trajectories instead of space-time patches, cutting tokens by up to 10x while outperforming ViT3D on retrieval and as a VideoLLM encoder.

A large-scale image editing dataset of real user requests and human-made edits from Reddit. A model trained on it outperforms competitors by up to 165 Elo points in human judgment.

We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. The best-in-class 72B model within the Molmo family not only outperforms others in the class of open weight and data models but also compares favorably against proprietary systems like GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human evaluation.

A benchmark for evaluating foundation models on complex actions which requires large enough frame sampling rate to understand fast movements, tracking, and understanding sequence of fine movements.

We showed how one can merge any task-specific expert model from open-source model zoos into foundation models (FMs) such as CLIP. This enhances the visual features of FM for dense prediction and localization tasks without collecting any supervised data.

We introduced a new sample adaptive inference method called SHARCS. It routes samples to different sub-networks with varying widths within any transformer network based on the hardness of input sample.

We introduced a new parameter-efficient fine-tuning method based on prompt tuning. In our method, prompts for some source tasks are learnt and for each sample in a new target task an attentional mixture of source prompts is used as the target prompt.

We introduce MERLOT Reserve, which learns from 20 million YouTube videos through all their modalities (audio, vision, and text). Learning from audio helps broadly -- even on single-image tasks like VCR. Our model learns state-of-the-art representations, that also transfer well to video-based tasks in a zero-shot setting.

Paraphrase generation by retrieving similar paraphrase pairs from a pre-existing corpus and editing them using multi-head attention mechanism.