Generative models like diffusion typically operate in compressed latent spaces for efficiency, while there is growing interest in leveraging high-quality pretrained visual representations (e.g., DINO, SigLIP) that are understanding-oriented. The core tension is that representation encoders produce high-dimensional latents that capture multiple hypotheses for masked regions, whereas generative models prefer low-dimensional latents that faithfully preserve injected noise. Previous work tried to bridge this gap with complex objectives and architectures, but FAE shows a simpler path is possible.

FAE (Feature Auto-Encoder) adapts pretrained visual encoders into generation-friendly low-dimensional latents using as little as a single attention layer. The key design is coupling two separate deep decoders: one reconstructs the original feature space, and a second takes those reconstructed features as input for image generation. This keeps the latent space compact while retaining enough information for both reconstruction and understanding. FAE is generic—it works with different SSL encoders and can be plugged into diffusion models or normalizing flows.

For serious builders, the takeaway is that representation–generation mismatch can be resolved with a surprisingly lightweight adaptation. On ImageNet 256×256, FAE with diffusion and CFG achieves an FID of 1.29 (800 epochs) and 1.70 (80 epochs), and without CFG it reaches state-of-the-art FID of 1.48 (800 epochs). This shows both high quality and fast learning, suggesting that pretrained visual encoders can be efficiently repurposed without heavy modifications to the generative pipeline.