Meta, a prominent tech company, has recently unveiled CM3Leon, an AI model that claims to achieve state-of-the-art performance in text-to-image generation. Unlike previous models, CM3Leon is capable of generating captions for images, which lays the foundation for more advanced image-understanding systems in the future. According to Meta, CM3Leon’s capabilities enable image generation tools to produce more coherent and contextually appropriate imagery based on input prompts.
Most image generators currently available rely on a process called diffusion, where noise is gradually subtracted from an initial noisy image to approach the target prompt. While this technique produces impressive results, it is computationally intensive and not practical for real-time applications due to its high cost and slow operation. In contrast, CM3Leon is a transformer model that utilizes attention mechanisms to weigh the relevance of input data such as text or images. This architecture allows for faster model training and easier parallelization.
Meta claims that CM3Leon is even more efficient than most transformers, requiring five times less compute power and a smaller training dataset compared to previous transformer-based methods. To train CM3Leon, Meta utilized a dataset of millions of licensed images from Shutterstock. The most powerful version of CM3Leon developed by Meta consists of 7 billion parameters, twice as many as OpenAI’s DALL-E 2.
One key factor contributing to CM3Leon’s improved performance is supervised fine-tuning (SFT), a technique used in training text-generating models like OpenAI’s ChatGPT. Meta theorized that SFT could be beneficial when applied to the image domain as well. Indeed, instruction tuning enhanced CM3Leon’s performance not only in image generation but also in generating image captions and responding to questions about images.
Unlike many other image generators that struggle with complex objects or text prompts with multiple constraints, CM3Leon demonstrates better performance in these scenarios. Examples provided by Meta include generating images based on prompts such as a small cactus wearing a straw hat and neon sunglasses in the Sahara desert or a raccoon main character in an Anime preparing for an epic battle with a samurai sword. Compared to DALL-E 2, CM3Leon’s images were generally closer to the prompts and more detailed.
Another advantage of CM3Leon is its ability to understand instructions for editing existing images. For instance, when presented with the prompt “Generate a high-quality image of a room that has a sink and a mirror with a bottle at location (199, 130),” the model can generate a visually coherent image that includes all the specified objects. In contrast, DALL-E 2 often fails to capture these nuances and omits objects mentioned in the instructions.
Furthermore, CM3Leon outperforms specialized image captioning models in generating captions and answering questions about images, despite receiving less text data during training.
While Meta did not announce any specific plans regarding the release of CM3Leon, they emphasized the importance of transparency in addressing challenges related to bias within generative AI models. As the industry continues to evolve, Meta believes that transparency will be crucial for accelerating progress.
Meta’s CM3Leon represents a significant breakthrough in text-to-image generation. With its improved performance, efficiency, and ability to generate captions and edit images based on instructions, CM3Leon showcases promising advancements in generative AI technology.