CoSyn: The open-source tool that’s making GPT-4V-level vision AI accessible to everyone

Researchers at the University of Pennsylvania and the Allen Institute for Artificial Intelligence have developed a groundbreaking tool that allows open-source AI systems to match or surpass the visual understanding capabilities of proprietary models like GPT-4V and Gemini 1.5 Flash, potentially reshaping the competitive landscape between open and closed AI development.

The tool, called CoSyn (Code-Guided Synthesis), addresses a critical bottleneck in AI development: the scarcity of high-quality training data for teaching machines to understand complex visual information like scientific charts, medical diagrams, and financial documents. Rather than scraping millions of images from the internet — a practice fraught with copyright and ethical concerns — CoSyn leverages the coding abilities of existing language models to generate synthetic training data.

“We have, we lack of such data to train the model. We lack of data, like documents, charts with rich annotations to train a vision language model to do question answering over ...