Is this release ready to document?
A demonstration run against two public projects, caddyserver/caddy and twbs/bootstrap. Public data, no client involved, and every figure below can be re-checked against those repositories. What a paid engagement covers is set out at the end.
What this is. A demonstration run against two public projects,caddyserver/caddyandtwbs/bootstrap. Public data, no client involved, and every figure below can be re-checked against those repositories. What a paid engagement covers is set out at the end.
A release manager has three separate records of the same release: the milestone, the documentation site, and the release notes. First-party tools will happily draft from any one of them, and none of them will tell you that the three do not name the same work.
What follows is that same question asked of two public releases, using only the structural pass and no AI model at any point.
Assumptions going in
- That a well-run project would produce a short, boring report.
- That a rolling 90-day window is a reasonable default scope.
- That the documentation site and the GitHub release body amount to interchangeable evidence.
The first was partly true. The second and third were wrong.
What organizations already run
Organizations in this position already have Jira's release-notes builder, GitHub's generated changelog, and Confluence page templates. They usually have a writer in the sprint or an engineer drafting on the tag, and Rovo or a similar vendor agent if the stack is Atlassian.
Every one of those tools is good inside its own system. The question a release manager actually needs answered is a different one: is this release accounted for across all the systems we already pay for?
The gap
A 90-day window answers the question of what happened lately, which is a dashboard number. It is the wrong question entirely during the week you are trying to ship.
I ran the same engine against caddyserver/caddy three times in the same week, changing only the scope:
| Scope | Issues | Merges | issue→PR | Findings |
|---|---|---|---|---|
| 90-day window | 39 | 70 | 77% | 59 |
milestone v2.11.4 |
1 | 10 | 100% | 4 |
milestone v2.11.5 |
1 | 29 | 100% | 12 |
Fifty-nine findings is residue, and a disciplined team will never work through it. Four findings against a named release is a meeting somebody can actually hold and finish.
On twbs/bootstrap v3.3.7 the same question was asked of three sources at once (milestone, versioned site, GitHub release body). Counts matched GitHub's own search: 24 closed issues, 48 merged pull requests (12 closed-unmerged correctly excluded). The site corroborated 8 of 24 issues. The release notes corroborated 9 of 48 pull requests. Neither source alone saw the other's work. Issue #19626 was correctly flagged: its fix (#19628) belongs to v4.0.0-alpha.3.
What a paid engagement covers
Everything above was measured on public projects. Here is how the same work runs when a client is paying for it.
The reconciliation is scoped to the release itself rather than to a rolling window, which in practice means the milestone plus the release tag whenever the notes live on GitHub. More than one documentation source is read in the same run, so that a page and a release body are never mistaken for each other. The structural pass compares identifiers only, and it sends nothing outside your network.
On Caddy that is one run against caddyserver/caddy at milestone v2.11.4, reading the GitHub release body alongside the documentation site, and returning a Markdown report.
That is the fixed-price audit in full. You receive a report and a walkthrough, and you leave with a decision: ship the notes as they stand, or work through the rows that are still open.
What this will not do
A milestone is not the whole release. Across four Caddy releases, the milestone accounted for 72% of what actually shipped, simply because maintainers do not assign every merge to one. Notes drafted from that scope are a starting point rather than a finished artifact, and the remaining 28% is why a person still reads them.
The check that looks for undocumented changes almost never fires on a product site, because most documentation does not cite pull-request numbers. When that happens the report says the check was skipped. It does not say that every change was documented.
It will not publish the notes, open the missing tickets, or run itself on a schedule. Each of those remains a person's job, or the job of tooling I have not built.
Figures: August 2026 public runs of caddyserver/caddy and the recorded twbs/bootstrap v3.3.7 milestone. Companion: You already have a drafter.