Scout: Charting the Incremental Index

Every good navigation system starts with a complete survey of the coastline before it can track the tides. Today we dug into Scout's indexing pipeline to understand exactly where it stands — and more importantly, where the gaps are.

What exists today: scout init performs a full parallel crawl across the markdown corpus, extracting metadata and wikilinks from each file, then builds an in-memory index and serializes it to a binary cache via postcard. Subsequent runs load that cached index if it's present. Simple. Functional. A solid keel.

What doesn't exist: any form of incremental update. There's no filesystem watcher, no update() or deindex_file() methods, no diff-based rebuild. If the cache disappears, Scout does a full rebuild from scratch. The architecture spec had always envisioned incremental patching — VaultIndex::update, index_file, deindex_file — but those methods lived in the parent crate and were never ported when Scout became standalone. They were noted as an open decision. Today we closed that decision: it's time to build them.

The design vision is clear: one full crawl in production builds (where speed is acceptable), then a filesystem watcher that patches the index incrementally as files are created, modified, renamed, or deleted. You don't resurvey the entire harbor every time a boat docks — you watch the entrance and update your log.

The implementation work ahead is non-trivial. It requires a filesystem watcher dependency, new mutation methods on the Index struct, cache invalidation logic, and careful handling of edge cases like renames, directory moves, and rapid-fire events that can race each other. We spent time mapping this out against the project hierarchy — structuring the work as Milestone → Goal → Issue rather than a flat task list — so each piece is trackable and independently verifiable.

Nothing shipped today. What shipped was the clarity: we now know exactly what the index needs to become, what's missing, and how the work breaks down. Sometimes the most productive day at sea is the one where you finish reading the charts before you touch the helm.