Customizing Flux: From Generative Media to Robotics

The Customization Pipeline

Black Forest Labs approaches model customization through a six-step inference pipeline: user input, prompt upsampling, moderation, model weight execution, post-processing, and deployment. The primary goal is to bridge the gap between a general-purpose foundational model and specific enterprise requirements without needing to retrain the entire model from scratch.

Prompt Upsampling as a Steering Mechanism

Prompt upsampling acts as a critical intermediary layer where a VLM or LLM interprets user intent and maps it to the model's training distribution. This is essential because users often provide vague prompts (e.g., "a cabin in the forest") that fail to meet specific design constraints (e.g., subject placement, negative space for text, or brand-specific aesthetics). By customizing the system prompt, engineers can force the model to adhere to strict layout rules without altering the underlying model weights.

Moderation: Config vs. Weights

Moderation is treated as a primary product surface rather than a secondary filter. For most enterprise use cases, modular configuration (API-level safety parameters and blocklists) is preferred for its scalability and flexibility. However, for edge-deployed models where deterministic filters aren't feasible, the team uses "LoRA sliders." By training a LoRA on the delta between modest and sexualized captions, they can create a negative-alpha weight adjustment that effectively "desexualizes" model output at inference time, achieving significant reductions in unwanted content without degrading overall image quality.

Embodied AI: Flux Action

Moving beyond media, the team extended Flux into robotics via "Flux Action." They fine-tuned the model to predict both video frame continuations and a six-dimensional vector representing robot arm movement. By embedding the robot's embodiment (the specific axes and constraints of the Lero robot) directly into the model weights, the model can process visual input from the robot's cameras and output the precise motor commands required to complete pick-and-place tasks.

Notable Quotes

• "The model technically can generate all of these different things, but really the design requirements end up being the bottleneck."
• "Moderation is actually one of the most important product surfaces that we have... it decides what the user sees and what not."
• "The delta between modest and sexualized captions is essentially the concept of sexualization."
• "We actually teach flux that this specific robot has six axes and the movement of these axes is essentially represented by a vector."