MonsterHouse is an interactive Halloween projection that turns a garage door into a grumpy old pirate ship. It listens to trick-or-treaters, answers in character, and softens when visitors say ‘please.’ The system is built on the NVIDIA Nemotron Voice Agent and runs on a single workstation with an NVIDIA RTX PRO 6000 Blackwell Max-Q and an NVIDIA RTX PRO 4000 Blackwell. The pipeline combines Nemotron Speech ASR for listening, NVIDIA Nemotron 3.5 Bolt E4B 4GB for dialogue, Magpie TTS Zeroshot for voice synthesis, and Audio2Face-3D for mouth animation in Unreal Engine 5; a Nemotron 3 Nano Omni VLM optionally describes costumes while YOLO and ByteTrack count and track visitors.

The character rig was built in Autodesk Maya and exported to Unreal Engine, where virtual fire and effects sit behind the door. To give the house a cantankerous personality, the author invented a backstory: the house was built from timbers of a sunken pirate ship. A LoRA adapter trained on over 5,000 multi-turn example conversations shapes Bolt E4B into this specific stubborn character. The Voice Agent lets any LLM be plugged in; the author tested more than a dozen models and chose Bolt E4B’s 4-bit quantized release, citing a smaller footprint than Qwen3 30B-A3B and Qwen 3.6 35B-A3B.

Latency is managed with a text bank and cached recordings. Median end-of-speech-to-audio is 1.70 seconds with Bolt Q4 and TTS, and 0.72 seconds for cached responses; speech-end detection currently waits for 300 ms of silence. Magpie TTS Zeroshot uses short reference samples rather than a trained voice, and the house has seven moods selected by emotion tags from the LLM. Audio2Face-3D receives clean speech, while a second audio stream adds real-time vocal effects for the deep demonic voice; a 100 ms adjustment compensates for Audio2Face-3D buffering. The language/voice pipeline and the Unreal Engine graphics run on separate GPUs to keep animation frame rate stable.

A control desk UI provides tabs for conversations, model selection, recordings, prompts, performance monitoring, and camera feeds, plus a soundboard and custom dialogue field. Live mode runs the trick-or-treat character, demo mode lets the house explain itself, and microphone mode bypasses recognition, dialogue generation, and synthesis so the author can speak through the house directly.

Visual awareness is handled by Nemotron 3 Nano Omni interpreting camera images to describe costumes, with YOLO and ByteTrack providing counts. A synthetic test with five costumed sheep correctly identified pirate, vampire, witch, pumpkin, and ghost outfits, but the author stresses it is not evidence from a real Halloween night; real driveway groups, costumes, and night lighting still need testing. MonsterHouse code is not packaged for release, but the underlying NVIDIA models are available. The recommended path is to clone NVIDIA Nemotron Voice Agent, run the Generic Assistant example, swap in Bolt E4B, add a LoRA trained via NVIDIA NeMo AutoModel, use Magpie TTS Zeroshot for a custom voice, add Audio2Face-3D for animation, and optionally use the Omni Assistant Subagents example with a people tracker. Cached response banks and saved audio can make even a slow monster feel quick, and a fully cached live version can run with just ASR and a Python conversation router.