Skill Distillation: Teaching Small Models Procedures via Markdown
The article describes a personal agent architecture that moves beyond chatbot-style interaction toward a small operating system. The core tension is that frontier models are too expensive and large to run locally for routine work, yet smaller models lack the procedural knowledge to handle complex tasks like managing an inbox or deal pipeline. The author’s system solves this by separating the generation of workflows from their execution. A frontier model writes and refines atomic SKILL.md files, each describing a single job, while a smaller local model like Qwen 35B or Gemma 26B simply follows those steps. This flips the usual AI pattern: the small model doesn’t need to understand the deep reasoning behind a task, only how to execute a well-defined procedure. The author calls this procedural knowledge retrieval, distinct from both RAG’s factual retrieval and classical weight-based distillation.
The concrete approach has three layers. First, a local markdown knowledge base called QMD stores roughly eighty workflow files in ~/memories as playbooks. Second, a frontier model authors and evaluates SKILL.md files until accuracy converges, also checking recall against QMD so the right keywords surface the right skill. Third, an agent loop runs Plan → Tool Call → Observe → Refine, calling out to seventeen Rust APIs and MCP integrations. The key operational insight is that the same frontier model writes both the skills and their evaluations, then rewrites each skill until accuracy converges. A nightly system scans historical logs to understand what new skills should be generated, mirroring a loop Pete Koomen described at Y Combinator.
The serious takeaway is that skill distillation offers a practical path to building capable local agents without requiring small models to pack the world’s knowledge into their weights. The teacher (frontier model) transfers procedural knowledge to the student (small local model) through inspectable, versionable, hot-swappable markdown files. This means the librarian of skills becomes the company’s institutional knowledge, while the student model can be swapped out every quarter for whichever model is cheapest. For builders, this suggests a cleaner architecture: separate the expensive reasoning from the reliable execution, and let markdown be the interface between them.
