An empty transcript proves nothing until you can prove the input deserved a transcript. We learned this by wrongly accusing two of our own tools: both returned empty on a YouTube URL that was live, popular and perfectly retrievable — and happened to be an animated short with a music track and no spoken words. The tools worked. The test could never have passed.
What does a failed transcript test actually tell you?
On its own, almost nothing. An empty result is one observable with at least five distinct causes: the URL is dead, the platform is blocking retrieval, the video has no speech, the tool’s parser broke, or your input violated a constraint the schema did not surface. Each cause demands a different fix, and four of the five are not the tool’s fault.
The run log usually separates them immediately. In our case the log showed the fetch succeeding — title, duration, 23 million views all retrieved — followed by a single line: no speech was detected in the media. That line converts “the tool is broken” into “the test is broken” in one read. Check the log before forming any other opinion; it is the cheapest evidence available and the most commonly skipped.
How do you prove an input deserves to pass?
Three properties, each independently verifiable before you spend a run:
Liveness. The URL must currently resolve to the content you think it does. For YouTube and TikTok, the oEmbed endpoint is the clean check — a server-side request that returns the video’s title if it exists and an error if it does not, no JavaScript rendering required. For platforms without oEmbed, a direct fetch of the page works, with one caution: some sites answer 200 for every path and route the miss client-side, so a 200 alone is not proof.
Speech. A transcription test needs words. Music videos, gameplay clips, animation shorts and ambience footage all produce honest empty transcripts. Pick something where a person verifiably talks — an interview, a lecture, a vlog — and confirm it by knowing the content, not by guessing from the title.
Schema fit. The value must mean what the field expects, not merely satisfy its type. We once passed a date field the string “week” — the schema said string, validation passed, the run returned nothing. Read the field’s description and example, not just its type.
What is the sibling test?
The strongest single diagnostic when you run more than one tool against the same platform: feed the identical input to a closely related tool and compare.
If a URL transcribes correctly through tool A and returns empty through tool B, the input is exonerated and suspicion localizes to B. If it fails through both — as ours did across two independently built TED tools on two different talks — suspicion moves to the platform itself. One comparison, run in minutes, replaces an afternoon of speculation.
The test needs honest limits. Downloader-passes-but-transcript-fails does not convict the transcript tool, because a clip can be retrievable and still contain no usable speech track. Sibling evidence narrows the search; it convicts only when the compared tools genuinely exercise the same path.
What does a fair verdict process look like?
The sequence that survived our own retesting, in order of cost:
- Read the run log. The failure often names itself.
- Verify the input is alive via oEmbed or a direct fetch, independent of the tool.
- Verify the input fits semantically — speech for transcribers, results-bearing queries for search tools.
- Run the sibling test where a related tool exists.
- Retest with a corrected input before writing anything down as broken.
We ran nine suspected-broken tools through this sequence. Five were exonerated — every one of them a victim of our own inputs, including a “broken” profile scraper that returned all 21 fields on retest. The four real defects came out sharper too, because each carried decisive evidence instead of a shrug.
Why this belongs in a production pipeline, not just debugging
Any pipeline that gates on live verification runs — publishing checks, monitoring, contract tests — inherits this problem at scale. A gate that tests a transcript tool with whatever URL was on hand will file false defect reports forever, and false defects are corrosive: they train people to ignore the gate.
The fix costs one design decision: every automated verification input carries a note on why it qualifies — verified live on a date, verified to contain speech, verified to return results. Inputs age, platforms change, and last month’s qualifying URL is this month’s silent false alarm. An input with a paper trail can be re-qualified in a minute; an input chosen by convenience has to be re-debugged from scratch.
Continue in the directory
Turn the guide into a real sample run.
Open the current AgentX contract, check pricing and fields, then validate a narrow output.