# Video and audio to text: transcript, subtitles and notes: build brief

Assembled by AgentGid's starter kit builder from recipe `transcribe` v0.3 (https://agentgid.com/build/kits/transcribe/). Data snapshot: 2026-10-12.
Constraints: commercial use; free parts only; crediting authors is fine.
Recipe status: test-built on 2026-10-12 (2 builds) with exactly these parts. The fixes from the last build are in this version, which has not been built again as a whole.
Licenses and prices change. Before you ship, open the linked AgentGid pages and confirm.

## What to build

A Python tool: give it a video or audio file and it writes a plain transcript, `.srt` and `.vtt` subtitles, a Markdown transcript with timestamps and a JSON file with every word's time.

## Definition of done

- [ ] `python -m transcribe run media/test-en.mp4` writes `out/test-en/` with `transcript.txt`, `transcript.srt`, `transcript.vtt`, `transcript.md` and `words.json`.
- [ ] The test clips are made on the machine from a known text (macOS `say`, Linux `espeak-ng`) and wrapped into an .mp4 with a still picture, so the video path is tested and nobody else's recording is needed.
- [ ] Word accuracy on the English test clip is at least 90% against the known text, measured by a word-level edit distance in the tests.
- [ ] A strict test parser reads `transcript.srt` and `transcript.vtt` back: SRT cues are numbered and use a comma before the milliseconds, the VTT file starts with `WEBVTT` and uses a dot, cues are in order, none is longer than 7 seconds or 84 characters, and times never go backwards.
- [ ] `transcript.md` has the file name, the detected language, the length and paragraphs that each start with a `[mm:ss]` mark.
- [ ] A folder can be passed instead of a file: every audio and video file in it (not in subfolders) is done once, and a second run skips files whose results exist (`--force` redoes them). Two files with the same name and different extensions (`talk.mp3`, `talk.wav`) get separate folders (`out/talk-mp3/`, `out/talk-wav/`).
- [ ] A long file shows progress as a percentage of its duration while it runs.
- [ ] Unit tests cover the subtitle cue splitter, the timestamp formats and the paragraph builder, and pass with `python -m pytest`.
- [ ] After the first run downloads the model, a second run works with the network off. If the model is already cached from earlier work, check the download with an empty cache (`HF_HOME` set to a new folder) or say that the first-run download was not seen.

Left for the person, because no check can do it:

- Read one transcript of a real recording of yours before trusting the tool with important material: names, numbers and technical terms are where speech models slip.

## Before you start

These files are already in this folder. Keep them, and never let a scaffolder overwrite them:

- `AGENTS.md`: this brief.
- `CLAUDE.md`: one line that imports AGENTS.md for Claude Code.
- `.mcp.json`: the MCP servers listed under "Tools you have".
- `CREDITS.md`: the license record. Add to it as you add files.
- `kit.json`: this kit as data (parts, versions, checks). `install.txt`: the install commands below. `first-prompt.txt`: the message the person starts you with.

You may do these without asking:

- Install the packages in the Stack section.
- Read the documentation this brief links to.
- Let faster-whisper download its model from the Hugging Face Hub on first run (about 484 MB for `small`, 1.6 GB for `large-v3-turbo`). No account or token is needed.
- Use `git`, the system speech command (`say` or `espeak-ng`) and, if they are installed, `ffmpeg` and `ffprobe` without asking.
- Import numpy in tests or test-clip code: it comes in with faster-whisper.

Stop and ask the person for:

- an account, a key or a payment;
- a runtime dependency, asset source or API that this brief does not list;
- a check that still fails after two real fixes (say what you tried).

## Stack

Use these parts and versions. Ask before swapping one.

| Part | Choice | Version | Kind |
|---|---|---|---|
| Runtime | Python 3.12 | - | runtime |
| Speech recognition | Faster Whisper | 1.2.1 | Python package |
| Also | pytest (unit tests) | 9.1.1 | Python package |
| Also | PyAV (reads and writes audio and video files) | below 19 | Python package |

Install exactly these versions (the same lines are in `install.txt`):

```sh
uv venv --seed --python 3.12 .venv && source .venv/bin/activate   # uv fetches Python 3.12 if it is missing; --seed puts pip into the venv, so the pip line below installs there and not into the system Python
# without uv: python3.12 -m venv .venv && source .venv/bin/activate
python --version   # must print 3.12.x
pip install faster-whisper==1.2.1 pytest==9.1.1 "av<19"
```

System packages (use the OS package manager): espeak-ng (Linux only: speaks the test clip (macOS uses its built-in `say`)).

Checks that apply to this kit:

- PyAV is pinned below 19: faster-whisper 1.2.1 passes an argument to `av.open` that PyAV 19 removed, and the fix is not released yet.
- faster-whisper decodes audio and video with PyAV, which bundles FFmpeg's libraries: transcription needs no system FFmpeg.

## Rules for this project

- Transcription is one function that takes a path and returns segments with words. The command line and the tests call that function; nothing else talks to the speech engine.
- Media files are read and written with PyAV (`import av`), not by calling the `ffmpeg` command: PyAV is already in the stack and works on machines without a system FFmpeg.
- Load the model once per process and reuse it. Loading takes seconds and hundreds of MB of memory.
- Never write an API key into a log or an error message.

## Tools you have

These MCP servers are in the config file listed under "Before you start".

- Context7: Up-to-date library docs for the agent
- AgentGid: Live data for choosing tools, assets and APIs while building

If a server's tools are missing from your session, either the person has not approved them yet (Claude Code holds the servers of a project `.mcp.json` as "Pending approval" until the person accepts them), or the config was added after the session started. Ask the person to approve the servers or restart the agent in this folder; until then, use the links in this brief.

## Plan

Work one step at a time. Do not start a step until the previous one meets its "Done when" lines.

### 1. Scaffold

- Create the virtual environment with Python 3.12 explicitly, as the install lines of the Stack section show, and check `python --version` inside it. If Python 3.12 is missing, `uv python install 3.12` gets it.
- Install the packages from the Stack section. On Linux also install `espeak-ng` for the test clips; macOS needs no system package (it has `say`, and PyAV bundles FFmpeg). Do not install `ffmpeg`: nothing in this plan needs it.
- `git init`, with a `.gitignore` that holds `.env`, `.venv/`, `out/`, `media/` and `uploads/`, and a README.md that says how to run each command.
- Lay out `transcribe/` with modules `engine.py` (speech recognition), `formats.py` (txt, srt, vtt, md, json), `batch.py` (files and folders) and a `__main__.py` command line. Put `pythonpath = ["."]` under `[tool.pytest.ini_options]` in pyproject.toml.
- Done when:
  - `python -m transcribe --help` lists the commands and `python -m pytest` runs.
  - The kit files listed under "Before you start" are still there.

### 2. Test clips

- Add `python -m transcribe make-test-clips`. It writes a known English text (60 to 90 words, with a name and a number in it; write numbers as digits, "42", because that is how Whisper writes them) to `media/test-en.txt`, and speaks it with the system voice (`say -o test-en.aiff` on macOS, `espeak-ng -w test-en.wav` on Linux).
- Wrap the audio into `media/test-en.mp4` with PyAV: decode the speech, then write a 1-frame-per-second gray `libx264` video stream (`yuv420p`, 640x360) and an `aac` audio stream into one container. PyAV's wheels bundle FFmpeg with both encoders, so no system FFmpeg is needed:

  ```python
  out = av.open('media/test-en.mp4', 'w')
  v = out.add_stream('libx264', rate=1); v.width, v.height, v.pix_fmt = 640, 360, 'yuv420p'
  a = out.add_stream('aac', rate=16000, layout='mono')   # the default layout is stereo
  # a.frame_size is 0 until the first encode: feed AAC frames of 1024 samples
  # encode gray av.VideoFrame.from_ndarray(...) frames (pts = second) and av.AudioFrame chunks, mux every packet, flush both encoders, close
  ```

- Also write a two-second silent clip `media/silence.wav` (16 kHz, `pcm_s16le`, zeros) for an edge-case test.
- Done when: `av.open('media/test-en.mp4')` shows one video and one audio stream, and its length matches the audio within a second.

### 3. Speech recognition

- Transcribe with faster-whisper on the CPU with int8, word timestamps and the voice-activity filter on. `transcribe()` returns a generator and does the work while you iterate over it; `info.duration` is known at once, so progress is `segment.end / info.duration`:

  ```python
  from faster_whisper import WhisperModel
  model = WhisperModel(MODEL, device='cpu', compute_type='int8')   # MODEL: see below
  segments, info = model.transcribe(path, language=LANG, word_timestamps=True, vad_filter=True)
  for s in segments:   # nothing is transcribed until this loop runs
      print(f'{s.end / info.duration:.0%}', end='\r')
      words = [(w.word, w.start, w.end, w.probability) for w in s.words]
  ```

- `MODEL` is `small`. For English-only recordings `small.en` is a little faster; the `.en` models cannot write any other language.
- `LANG` is `None`: Whisper detects the language from the first 30 seconds only. Add `--language` so a recording that opens with music or another language can be fixed, and write the detected language and its probability into every result. When `--language` is given, write that language with `"language_source": "--language"` and no probability: faster-whisper then reports a probability of 1, which is not a detection.
- In languages written with spaces, word entries that do not begin with a space continue the word before them ("30,000" arrives as " 30" and ",000"). Join them before counting or displaying words. Not for `zh`, `ja`, `th`, `lo`, `my` and `yue`: there faster-whisper already splits words by Unicode and no entry starts with a space, so this rule would glue a whole segment into one word.
- Return plain data: a list of segments, each with `start`, `end`, `text` and `words` (`text`, `start`, `end`, `p`), plus `language`, `language_probability` and `duration`. Formats and tests use this, never the engine's own objects.
- Read first: https://github.com/SYSTRAN/faster-whisper#readme (sections Word-level timestamps and VAD filter)
- Done when:
  - The English test clip comes back with at least 90% word accuracy (a test computes it with a word-level edit distance after lower-casing and stripping punctuation).
  - The silent clip gives zero segments and no invented text. (Whisper still reports a language for it; with no segments, write the language as unknown.)

### 4. Formats

- `transcript.txt`: the text in paragraphs. Start a new paragraph after a pause of 2 seconds or more, or after about 60 seconds of speech at the next sentence end.
- `transcript.srt` and `transcript.vtt`: cues of at most 2 lines of 42 characters and at most 7 seconds, split at word boundaries using the word times, never across a pause longer than 1 second. SRT writes `00:01:02,345` with a comma and numbers its cues; VTT starts with `WEBVTT` and writes `00:01:02.345` with a dot.
- `transcript.md`: a header with the file name, detected language and length, then the paragraphs, each starting with a `[mm:ss]` mark (`[h:mm:ss]` past an hour).
- `words.json`: the plain data from the speech step, as is.
- Read first: https://www.w3.org/TR/webvtt1/ (section 4, file structure)
- Done when:
  - Unit tests with fixed word timings check cue splitting, both timestamp formats (including times past one hour) and paragraph breaks.
  - A test writes every format for the English clip into a temporary folder and reads the subtitles back with a strict parser of its own (numbered SRT cues with commas; `WEBVTT` first line and dots), because FFmpeg's readers also accept an SRT with dots and a VTT without the header.

### 5. Files and folders

- `run <path>` takes a file or a folder. In a folder it picks audio and video files by extension (mp3, m4a, wav, ogg, flac, mp4, mov, mkv, webm) and writes `out/<file name>/` for each.
- Skip a file whose `out/` folder already holds all results, unless `--force` is given. A file that fails is reported and the rest go on; the command exits with an error code if any failed.
- Done when: Running `run media/` twice transcribes each clip once.

### 6. Verify

- Commit, make a clean copy of the repository in a temporary folder (`git clone . /tmp/check`) and follow only the README there: install, make the test clips, run the tests.
- Run the English clip again with the network off (the model is cached now). On Linux, `unshare -rn -- python -m transcribe run media/test-en.mp4 --force` runs a command without a network; on macOS, `sandbox-exec -p '(version 1)(allow default)(deny network*)' python -m transcribe run media/test-en.mp4 --force` does the same. If neither works, say that this was not checked.
- Transcribe one real recording of at least 10 minutes if the person gives one, and write its length, the time taken and the model into PROGRESS.md.
- Done when: Every line under Definition of done is true, and anything that could not be verified is named in PROGRESS.md.

## How to check your work

- After each step, run its command on `media/test-en.mp4` and open the output files, not only the exit code.
- Compare transcripts with the known text by a word-level edit distance, never by eye.
- Synthetic speech is easier than a real recording: treat the test-clip accuracy as a floor check, not as the tool's accuracy.
- Keep the model cached between runs; never delete the Hugging Face cache in a test.

## Known traps

- faster-whisper 1.2.1 accepts any PyAV from 11 up, but PyAV 19 (Sep 29, 2026) removed the `metadata_errors` argument it passes to `av.open`, so a fresh install fails at the first `transcribe()` with a TypeError. The fix is on faster-whisper's main branch but not released; the Stack pins PyAV below 19. (PyAV changelog and faster-whisper source, checked Oct 12, 2026)
- `WhisperModel.transcribe()` returns a lazy generator: nothing runs until you loop over the segments, so timing or error handling around the call itself measures nothing. (faster-whisper README)
- `WhisperModel.transcribe()` has the voice-activity filter off by default (`vad_filter=False`), while `BatchedInferencePipeline.transcribe()` has it on. Pass `vad_filter=True` yourself. (faster-whisper 1.2.1 source)
- Without a `language`, Whisper detects it from the first 30 seconds only. A recording that opens with music, a jingle or a greeting in another language can be transcribed in the wrong language from start to end. (faster-whisper 1.2.1 docstring)
- FFmpeg's subtitle readers are lenient: `ffmpeg -i x.srt` accepts dots instead of commas, and a .vtt file without the `WEBVTT` line opens too. A read-back with FFmpeg or PyAV does not prove the file is valid for players. (W3C WebVTT, and measured with PyAV 18.1.0 (bundled FFmpeg), Oct 12, 2026)
- Models whose names end in `.en` know English only. With Russian or any other language they write English-sounding nonsense instead of failing. (openai/whisper README)
- SRT and WebVTT look alike but differ: SRT numbers its cues and writes milliseconds after a comma; WebVTT needs the `WEBVTT` first line and a dot. Players reject the wrong separator silently or show nothing. (W3C WebVTT)

## Warnings you can ignore

- "You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN ...": the Whisper models are public and need no token. Do not ask the person for one.
- A progress bar from huggingface_hub on the first run while the model downloads.

## Limits

- Transcribe only recordings you have the right to use. Recording people may need their consent where you live; that is the person's call, not the tool's.
- This kit does not download media from YouTube or other sites: their terms decide what you may download.
- Never read or print the contents of `.env` files, and never commit keys.

## When you need something that is not listed

Do not pick a library, asset source or API from memory. Ask the AgentGid MCP server first:

- `find_tools(goal)` for libraries, APIs and agents that fit a task.
- `find_assets(category, query, free_commercial_only=true)` for media sources and their license terms. It knows sources such as Kenney or Pixabay, not single packs, models or fonts.
- `get_api(api)` for prices, free tiers and limits of entries in AgentGid's API catalog. Media sources are not in it: use `find_assets`.
- `check_status(names)` when a service seems down.

Then tell the person what you want to add and why, cite the AgentGid URL, and wait for a yes before you add a runtime dependency.

## When this brief is wrong

Keep a list of what in this brief was wrong, missing or out of date, with the line and what worked instead (in PROGRESS.md). When you finish, send it to AgentGid's editors with `report_kit_issue(recipe="transcribe", version="0.3", text=...)` on the AgentGid MCP server: only they read it. Leave out keys, personal data and the person's code.

## Sources

- https://www.python.org/downloads/
- https://agentgid.com/tools/faster-whisper/
- https://agentgid.com/mcp/servers/io-github-upstash-context7/
- https://agentgid.com/connect/
- https://raw.githubusercontent.com/PyAV-Org/PyAV/main/CHANGELOG.rst
- https://github.com/SYSTRAN/faster-whisper#readme
- https://raw.githubusercontent.com/SYSTRAN/faster-whisper/v1.2.1/faster_whisper/transcribe.py
- https://www.w3.org/TR/webvtt1/
- https://github.com/openai/whisper#available-models-and-languages