Training Krea 2: Data Curation and Pipeline Strategies

Data Curation and Filtering

Krea 2 achieves stylistic diversity by avoiding synthetic AI-generated training data, which the team views as a source of aesthetic "stickiness" and model bias. The data pipeline relies on approximately 30 to 40 custom in-house filters to curate a high-quality dataset from billions of images. Key filtering methods include:

• Deduplication: The team performs initial deduplication using P-hash or MD5 hashes, followed by semantic near-duplicate removal using embedding-based methods like SSCD.
• Sparse Autoencoders (SAE): The team uses SAEs as an unsupervised tagging system to identify and filter out undesirable artifacts such as watermarks, signatures, and border issues.
• World Knowledge Coverage: To ensure the model understands diverse concepts, the team ranks Wikipedia articles by PageRank and uses these high-value concepts to verify representation within the training corpus.
• Captioning Pipeline: Images undergo OCR to extract visible text, followed by a second pass with a Vision Language Model (VLM) to generate detailed captions. Filters are specifically designed to remove images where the captioner fails to capture context, such as paintings framed on walls, to prevent the model from incorrectly learning to include frames in every generated painting.

Training Pipeline and Architecture

The training process mirrors modern Large Language Model (LLM) workflows, moving from low-resolution semantic learning to high-resolution structural refinement. The pipeline follows a structured progression: pre-training, mid-training, supervised fine-tuning (SFT), preference optimization, and reinforcement learning (RL).

• Prompt Expansion: The team employs a small, autoregressive language model to expand user prompts into longer, more detailed descriptions, which aligns better with the training data distribution.
• Preference Optimization: The model is polished using collected preference pairs to align outputs with desired aesthetic and structural standards.
• Future Directions: The team is exploring multi-expert on-policy distillation, where specialized experts (e.g., photography, text rendering) are merged into a single student model to aggregate capabilities.