> product // whispering-wombat // the making of
Whispering Wombat
how it was made.
Whispering Wombat started as one question: could a serverless crawler running on Cloudflare do the job of a desktop SEO tool like Screaming Frog. The answer arrived in 72 commits over about six weeks, and it was yes. A Durable Object that chains its own alarms to walk a whole site, an analysis pipeline that grew to eleven phases and then got pruned back, and a Bearer-token API that quietly turned the crawler into something an AI agent could drive on its own.
The short version
It began on 27 February 2026 as a proof of concept: a crawler, Google OAuth, a custom domain, and the unglamorous founding-day work of decoding HTML entities and fixing code review findings. Over the next weeks it grew an analysis pipeline phase by phase, from broken links and metadata up to readability, duplicate detection, content clustering and JS rendering. Two turns mattered more than the feature count. First a redesign from crawl-centric to domain-centric, so the tool tracked sites over time rather than treating each crawl as an island. Second a Bearer-token API for API consumers and an SEO manager agent, which is the seam that later let the whole engine be exposed through MCP. The proof of concept earned its keep: it became the engine behind the Whispering Wombat SEO server in the catalogue.
// the build log · mined from the commit history, nothing dramatised
Crawler, auth, and the boring foundations
Six commits on the founding day. The POC crawler landed, then Google OAuth and a custom domain, then the work nobody photographs: decoding HTML entities in titles and H1 text, a pass of security and database-index fixes from code review, a CLAUDE.md for project context, and the vitest config wrangled into shape. A working spine before the first day was out.
From a crawler to a tool you can read
The raw crawl became something legible: a sortable data grid, filtered views, an Overview tab and a page-detail sheet. SERP previews, social cards and schema validation followed, then crawl comparison with change detection, link analysis with anchor text and SEO scoring, sitemap validation and crawl-depth tracking. This is also where the chain of small fixes shows the honest grind: a links endpoint reworked to use a JOIN instead of a phantom is_broken column, anchor-text field names aligned to the actual API response.
Bearer tokens, then a redesign
On 2 March a quiet commit added Bearer token auth for API consumers and an SEO manager agent. It read like plumbing at the time, but it is the seam the MCP server would later run through. The day after came the bigger move: a redesign of the architecture from crawl-centric to domain-centric, so a site could be tracked over time rather than as a series of disconnected crawls, plus Google Search Console and Analytics 4 integration.
Eleven phases, and learning to be wrong less often
The analysis pipeline climbed phase by phase: search intelligence, portfolio views, notifications, then accessibility, image and breadcrumb checks, mobile user-agent config, content clustering via Workers AI embeddings, and JS rendering detection. Just as telling is the cleanup. One commit fixed fifteen false-positive issue types at once, with status-code filters and trailing-slash exclusions, and paginated pages were carved out of duplicate detection so the tool stopped crying wolf.
Replacing three phases with one good call
The spell checker was swapped from nspell to AI spelling via Gemini, splitting genuine errors from suggestions. Then a larger simplification: AI-first page analysis through a direct Gemini API call, replacing three separate phases with one. The commits that followed are dogfood fixes, issue normalisation, deduplication and label truncation, the sign of a tool being used in anger rather than just shipped.
Turning findings into something to send
A reports foundation, then client-facing branding and a client dashboard, a site-structure graph, a conversational SEO chat, and a modular report builder with a GA4 channel breakdown and a weekly performance template. The crawler had become a reporting tool an agency could put a name on, with PDF generation and scheduled email reports carrying the output out the door.
Wrapping up the proof of concept
The closing weeks are PRs and corrections: tabs that wrap on small screens, a crawl error rendered as a string instead of object Object, external-link domain drill-down, sitemap discovery decoupled from robots handling, and H1 detection bugs. The last commit adds an ASSETS binding so static assets serve through the Worker, a small but real Cloudflare lesson learned in production. Modest, focused, done.
git log: “Add Bearer token auth for API consumers and SEO manager agent”
A 2 March commit that looked like plumbing at the time. It is the seam that later let the whole crawler be driven by an AI through MCP.
// the roads not taken
Tried, measured, set aside: the judgement lives here as much as in what shipped.
Crawl-centric to domain-centric
The early model treated each crawl as a standalone thing. It was redesigned to be domain-centric so a site could be tracked over time, which is what an SEO tool actually needs to do.
Three analysis phases collapsed into one AI call
Several separate analysis phases were replaced by a single AI-first page analysis via the Gemini API. Fewer moving parts, and the AI handled what the hand-rolled phases had been approximating.
Fifteen false positives killed in one pass
A crawler that flags problems is only useful if the problems are real. One commit fixed fifteen false-positive issue types, and paginated pages were excluded from duplicate detection, so the tool earned trust by being wrong less often.
nspell swapped for AI spelling
The dictionary-based spell checker was replaced with AI spelling via Gemini, which could separate genuine errors from stylistic suggestions rather than flagging every unfamiliar word.
Want something built like this?
This is how we work: in the open, measured, honest about the dead ends.