This article details an open reproduction of Surya Narreddi’s project where a language model is trained via reinforcement learning (RL) to paint watercolours by generating JavaScript code using the p5.brush library. The entire pipeline runs on Hugging Face: the RL environment and scorer model run as Spaces, a pairwise judge uses Inference Providers, and all artifacts (pool dataset, training scripts, trained models) are published.

The RL environment wraps the p5.brush library (47 methods, restricted to 10 to enforce a watercolour style), headless Chromium for rendering, and a gate that rejects non-compiling or cheating sketches. The reward has four terms: gate (0.05), length (0.05), a pairwise judge (0.60), and HPSv3 (0.30). The pairwise judge (Qwen3-VL-30B-A3B-Instruct) compares candidate paintings against four references randomly drawn from a hand-rated pool of 178 images (split into ‘love’ and ‘okay’ tiers). HPSv3 is an open 7B preference model. Three runs were trained with different reward mixes: judge-led (0.60 judge, 0.30 HPSv3), hps-led (0.30 judge, 0.60 HPSv3), and hps-only (0.00 judge, 0.90 HPSv3).

The base model is Qwen3.5-35B-A3B, fine-tuned with LoRA via TRL‘s GRPOTrainer. Key configuration changes that enabled learning: raising the learning rate to 5e-5, using a constant learning rate scheduler with warmup, setting scale_rewards to ‘none’, and using all-linear LoRA to cover all linear layers (the MoE architecture has fused routed experts left frozen).

Results: All three runs showed improving mean group reward. The hps-only run improved from 0.58 to 0.71 over 60 steps; judge-led from 0.45 to 0.72 over 110 steps; hps-led from 0.57 to 0.82 over 110 steps. The more weight the pairwise judge carried, the noisier the climb and the lower the start. The pairwise judge term itself climbed, confirming the model learned to win more comparisons against the pool. The main improvement in hps-only came from reducing bad paintings; the pairwise judge additionally pushed the top-end quality and paint coverage. The model ignored the prompt instruction for 15–30 shapes, settling on 7–9 shapes, since that command was not rewarded. Diversity was limited because the pool contains only one subject (hibiscus); more diverse output would require a more diverse reference pool.

Infrastructure challenges dominated the effort: 1.5% of rollouts scored zero due to timeouts or unresponsive scorers; those are now excluded (return None). A bug in OpenEnv (cached closed websocket) cost two runs; the fix was submitted upstream. Each render takes 69–96 seconds in a CPU Space (no GPU), and a training step of eight rollouts takes 15–18 minutes. Total cost for a 60-step run: one H200 (~18 hours), an a100-large Space for HPSv3, a cpu-upgrade Space for the environment, and Inference Providers quota for the pairwise judge.

Limitations explicitly discussed: the pool contains no human-made paintings, only model-generated ones. The reward encodes the author’s personal taste as defined by the pool ratings. The method is not specific to flowers—others have used it for animals and animations. The project does not improve on the original idea; it reproduces it openly. The author provides a list of suggested next steps: multi-turn training (the model paints without seeing its own output), using smaller models (a 4B already passes the gate), SFT on the pool before RL, explicit pigment reward, and dynamic reference difficulty.

Every piece is published: the reference pool, environment Space, scorer Space, training scripts, rollout datasets, and trained models.