# Short-video pipeline: script to finished file: build brief

Assembled by AgentGid's starter kit builder from recipe `short-video` v0.3 (https://agentgid.com/build/kits/short-video/). Data snapshot: 2026-10-12.
Constraints: commercial use; free parts only; crediting authors is fine.
Recipe status: test-built on 2026-10-12 (2 builds) with exactly these parts. The fixes from the last build are in this version, which has not been built again as a whole.
Licenses and prices change. Before you ship, open the linked AgentGid pages and confirm.

## What to build

A Python command line tool: give it a script file and it writes `final.mp4` (1080x1920, narrated, captioned, cut to stock footage) plus a credits file.

## Definition of done

- [ ] With a footage key in `.env`, `python -m pipeline make --script scripts/example.json` writes `out/example/final.mp4`. Without a key it stops with a message that names the missing key, and `--footage saved` still writes the video from the saved example response.
- [ ] ffprobe reports 1080x1920, 30 fps, H.264 video and AAC audio, and a length between 20 and 60 seconds.
- [ ] The audio track covers the whole video and is not silent (mean level above -40 dB). Every caption fits inside 960 px, and frames grabbed at the midpoint of three captions, the widest one among them, each show their own caption with complete letters.
- [ ] Footage changes every 3 to 6 seconds, the picture changes at every planned cut frame, and no shot has a uniformly black band on opposite edges.
- [ ] `out/example/credits.json` lists every clip with its source, page URL, author and license, plus the voice package, the voice model and the font.
- [ ] `python -m pipeline make --dry-run` works with no network and no keys: placeholder clips, a generated tone track in place of speech, and word timings spread from the script. It writes to `out/example-dry/`, never over a real video.
- [ ] Unit tests cover caption grouping and shot timing and pass with `python -m pytest`.
- [ ] Keys live in `.env`, `.env.example` lists them, `.env` is in `.gitignore`, and no log line or error message contains a key.

Left for the person, because no check can do it:

- Listen to one finished video before anything is published: a check can measure loudness, not whether the voice reads well. Decide then whether the pauses between sentences are too long (`--trim-silence`) and whether the voice is loud enough.
- Create the Pixabay account and put the key into `.env`.

## Before you start

These files are already in this folder. Keep them, and never let a scaffolder overwrite them:

- `AGENTS.md`: this brief.
- `CLAUDE.md`: one line that imports AGENTS.md for Claude Code.
- `.mcp.json`: the MCP servers listed under "Tools you have".
- `CREDITS.md`: the license record. Add to it as you add files.
- `kit.json`: this kit as data (parts, versions, checks). `install.txt`: the install commands below. `first-prompt.txt`: the message the person starts you with.

You may do these without asking:

- Install the packages in the Stack section.
- Download the files linked under Assets, from the links given.
- Read the documentation this brief links to.
- Let the packages fetch what they need on first run: the voice model (about 330 MB) and the speech-recognition model (about 150 MB) from Hugging Face, and the spaCy model `en_core_web_sm` (about 12 MB), which the voice package installs with pip from github.com.
- Use `git`, `ffmpeg` and `ffprobe` without asking.
- Work without a footage key: build and test the footage stage against a saved response shaped like the documented one, download and crop the example clip from Pixabay's API documentation, and report the live search as unverified.

Stop and ask the person for:

- an account, a key or a payment;
- a runtime dependency, asset source or API that this brief does not list;
- a check that still fails after two real fixes (say what you tried).

## Stack

Use these parts and versions. Ask before swapping one.

| Part | Choice | Version | Kind |
|---|---|---|---|
| Runtime | Python 3.12 | - | runtime |
| Script writing | Your coding agent writes the scripts | - | - |
| Voiceover | Kokoro-82M | 0.9.4 | Python package |
| Caption timing | Faster Whisper | 1.2.1 | Python package |
| Video assembly | MoviePy | 2.2.1 | Python package |
| Image handling | Pillow | 10.4 to 11.1 | Python package |
| Also | Requests (HTTP calls to the footage API) | 2.34.2 | Python package |
| Also | python-dotenv (reads keys from .env) | 1.2.4 | Python package |
| Also | pytest (unit tests) | 9.1.1 | Python package |
| Also | SoundFile (writes the WAV files) | 0.14.0 | Python package |
| Also | PyAV (pinned for Faster Whisper) | below 19 | Python package |
| Also | NumPy (pinned for MoviePy) | below 2.5 | Python package |

Install exactly these versions (the same lines are in `install.txt`):

```sh
python3.12 -m venv .venv && source .venv/bin/activate   # with uv: uv venv --python 3.12 .venv && source .venv/bin/activate, then write "uv pip install" below
python --version   # must print 3.12.x
# no NVIDIA GPU: run the next line first (CPU-only PyTorch, about 2 GB less to download). With an NVIDIA GPU, skip it.
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install kokoro==0.9.4 faster-whisper==1.2.1 moviepy==2.2.1 "pillow>=10.4,<11.2" requests==2.34.2 python-dotenv==1.2.4 pytest==9.1.1 soundfile==0.14.0 "av<19" "numpy<2.5"
```

System packages (use the OS package manager): ffmpeg (for ffprobe, which the check command uses).

Checks that apply to this kit:

- Pillow is pinned to 10.4 to 11.1. moviepy 2.2.1 accepts pillow >=9.2.0 and <12.0, but with 11.2 or newer its captions are cut off at the bottom. The pin installs 11.1.0 today.
- numpy is pinned below 2.5: with 2.5, moviepy 2.2.1 prints a DeprecationWarning for every decoded frame.
- PyAV is pinned below 19: faster-whisper 1.2.1 passes an argument to `av.open` that PyAV 19 removed, and there is no newer faster-whisper yet.
- Use Python 3.12. kokoro 0.9.4 requires Python <3.13,>=3.10. pip refuses a newer interpreter; uv does not.
- Kokoro pulls in PyTorch and over a hundred packages: about 3 GB with the default GPU wheels, about 1.7 GB with CPU-only PyTorch. First run downloads a voice model of about 330 MB.

## Rules for this project

- Every stage is a command that can run alone and reads its input from files the previous stage wrote. That is what makes a failed stage cheap to repeat.
- Check a package's version with `python -c "from importlib.metadata import version; print(version('moviepy'))"`. `moviepy.__version__` reports 2.1.2 for 2.2.1, and in an environment made by uv, `pip show` asks the system Python.
- Never write a full request URL into a log or an error message: the Pixabay key travels in the query string, and the HTTP library puts the URL into its own error text. Remove the key before logging, and test that. (test build)

## Assets and licenses

Add a row to CREDITS.md for every file you bring into the project, at the moment you add it.

### Stock footage: Pixabay

- API limits: 100 requests per 60 seconds per key; cache responses for 24 hours; higher limits may be granted on request.
- Source terms: Commercial use OK; Credit when you use its API; Monetized videos with conditions.
- Note: Do not scrape its website. Use its API, one request at a time, and download only the files you use.
- The key is free and needs the person's Pixabay account. It goes into `.env` as `PIXABAY_API_KEY`.
- For a command line tool, "show Pixabay as the source" means: name Pixabay and the clip's page URL in the log line of every clip and in the credits.
- AgentGid's rules for Pixabay: Cache API responses for 24 hours; do not mass-download. Show Pixabay as the source when displaying search results.
- License page: https://pixabay.com/service/license-summary/ (AgentGid read it on Oct 10, 2026). Details: https://agentgid.com/assets/pixabay/

### Caption font: Google Fonts, Poppins

- Font file: https://raw.githubusercontent.com/google/fonts/main/ofl/poppins/Poppins-ExtraBold.ttf (151 KB)
- License text to keep next to it: https://raw.githubusercontent.com/google/fonts/main/ofl/poppins/OFL.txt
- Specimen: https://fonts.google.com/specimen/Poppins
- Designer: Indian Type Foundry, Jonny Pinhorn, Ninad Kale. License of this font: OFL-1.1.
- Source terms: Commercial use OK; No credit needed.
- AgentGid's rules for Google Fonts: Keep the license file with font files you bundle in an app.
- License page: https://github.com/google/fonts (AgentGid read it on Oct 10, 2026). Details: https://agentgid.com/assets/google-fonts/

## Licenses of the tools above

- Kokoro-82M: Apache-2.0. Keep the Apache-2.0 license notice when you redistribute the model. https://agentgid.com/assets/kokoro/
- Two things carry the name: the Python package `kokoro` 0.9.4 and the model file `kokoro-v1_0.pth` from hexgrad/Kokoro-82M on Hugging Face. Record both in the credits.
- For English the package loads `phonemizer-fork` and espeak-ng, both GPL-3.0-or-later, and `num2words` (LGPL). That does not touch the videos you make; it matters if you pass the tool itself on, as an app or an image.
- MoviePy: MIT.  https://agentgid.com/assets/moviepy/
- MoviePy 2 does not need a system FFmpeg to render: its dependency imageio-ffmpeg brings an FFmpeg binary, and FFmpeg's own license applies to that binary. It brings no ffprobe, which is why ffmpeg is listed under system packages.

## Tools you have

These MCP servers are in the config file listed under "Before you start".

- Context7: Up-to-date library docs for the agent
- AgentGid: Live data for choosing tools, assets and APIs while building

If a server's tools are missing from your session, either the person has not approved them yet (Claude Code holds the servers of a project `.mcp.json` as "Pending approval" until the person accepts them), or the config was added after the session started. Ask the person to approve the servers or restart the agent in this folder; until then, use the links in this brief.

## Plan

Work one step at a time. Do not start a step until the previous one meets its "Done when" lines.

### 1. Scaffold

- Create the virtual environment with Python 3.12 explicitly, as the install lines of the Stack section show, and check `python --version` inside it. If Python 3.12 is missing, `uv python install 3.12` gets it.
- On a machine without an NVIDIA GPU, install CPU-only PyTorch first, as the install lines show. Then install the rest.
- Install the packages from the Stack section and the system package listed under it.
- `git init`, with a `.gitignore` that holds `.env`, `.venv/`, `out/` and `cache/`, and a README.md that says how to run each command.
- Lay out `pipeline/` with one module per stage (script, voice, captions, footage, assemble, credits) and a `__main__.py` command line. Put `pythonpath = ["."]` under `[tool.pytest.ini_options]` in pyproject.toml, so that plain `pytest` finds the package too.
- Done when:
  - `python -m pipeline --help` lists the commands and `python -m pytest` runs.
  - The kit files listed under "Before you start" are still there.

### 2. Script format

- A script is a JSON file with `title`, `hook`, `beats` and `closing`. Each beat has `text` and `keywords`: a list of one to five search words, joined with spaces for the search, at most 100 characters in all. The narrated lines are the hook, the text of each beat, and the closing line. The hook is shown over the first beat's footage and the closing line over the last beat's, unless `hook_keywords` or `closing_keywords` are given. The title is not narrated. Validate on load.
- Write beats of 9 to 15 words (3 to 6 seconds of speech) and 55 to 150 words in total. Validation warns outside those ranges, at about 2.8 words per second.
- Write three example scripts by hand into `scripts/`; the first is `scripts/example.json`.
- Done when: An invalid script fails with a message that names the missing field.

### 3. Voice

- Generate the narration with Kokoro, one WAV file per narrated line at 24 kHz, and record each duration. The call returns a generator with one item per chunk of text, so join the chunks of a line:

  ```python
  from kokoro import KPipeline
  import numpy as np, soundfile as sf
  pipeline = KPipeline(lang_code='a', repo_id='hexgrad/Kokoro-82M')   # 'a' is American English
  chunks = [np.asarray(audio) for _, _, audio in pipeline(text, voice='af_heart')]
  sf.write(path, np.concatenate(chunks), 24000)
  ```

- Kokoro pads every chunk with silence, about 0.3 s before and 0.45 s after the speech. With the 0.25 s pause below, the gap between two lines is therefore about 1 s, and the word counts in this brief assume that. Measure where the sound of each line starts and ends (first and last sample above -50 dBFS) and write it down for the caption stage. Add a `--trim-silence` option that cuts the padding to 0.05 s; leave it off by default.
- Join the lines with a 0.25 s pause and save `narration.wav` with two identical channels: MoviePy reads a one-channel file as stereo and 3 dB quieter. Write the durations and the pause length to a file the next stage reads.
- Done when:
  - `narration.wav` holds the whole script, and the per-beat durations plus the pauses add up to its length.
  - The narration is between 20 and 60 seconds long; the stage refuses to go on otherwise.

### 4. Caption timing

- Transcribe `narration.wav` with faster-whisper and keep start and end of every word. An entry that does not begin with a space continues the word before it ("30,000" arrives as " 30" and ",000"):

  ```python
  from faster_whisper import WhisperModel
  model = WhisperModel('base.en', device='cpu', compute_type='int8')   # about 150 MB on first run
  segments, _ = model.transcribe('narration.wav', word_timestamps=True)   # a generator: nothing runs until you iterate
  words = []
  for s in segments:
      for w in s.words:
          if words and not w.word.startswith(' '):
              words[-1] = (words[-1][0] + w.word.strip(), words[-1][1], w.end)
          else:
              words.append((w.word.strip(), w.start, w.end))
  ```

- Give each word to the line whose time window (from the voice stage) contains the word's midpoint. Recognised times run early at the start of a line: no word may start before the sound of its line starts, and the last word lasts until the sound ends.
- Show the script's spelling: when the number of recognised words in a line equals the script's, keep the script words with the recognised times; when nothing was recognised, spread the script words evenly over the line; otherwise keep the recognised words and log a warning that names the line.
- Group words into captions of two to four words that never cross a line boundary. Measure every candidate with the caption font (`ImageFont.truetype(FONT_PATH, 84).getlength(text)`, plus the stroke on both sides): a caption must fit in 960 px. Put fewer words into it when it does not, and make the type smaller only when two words alone are too wide. A one-word line gives a one-word caption.
- Done when:
  - A unit test feeds fixed word timings in and gets the expected caption groups out.
  - For the example script, the recognised word count equals the script's in every line, and no caption is wider than 960 px.

### 5. Footage

- Give `make` a `--footage` option: `pixabay` (the default; without a key it stops with a message that names `PIXABAY_API_KEY`), `saved` (the example response from the API documentation, saved in the repository, with its clip downloaded for real) and `placeholder` (generated clips, as in a dry run).
- Search `https://pixabay.com/api/videos/` with `key`, `q` (the line's keywords), `per_page=20` and `safesearch=true`. Each hit has `duration`, `pageURL`, `user` and four renditions under `videos` (`large`, `medium`, `small`, `tiny`), each with `url`, `width`, `height` and `size`. `large` can be empty, and `duration` is rounded: a clip can be half a second shorter than it says.
- Prefer hits at least as long as the shot. Inside a hit, take the smallest rendition whose 9:16 centre crop is at least 1080x1920 (usually `large`, 3840x2160). A 1920x1080 rendition works but is enlarged 1.78 times: say so in the log.
- Plan shots in whole frames: a line shorter than 3 s shares a shot with the line after it (the last line with the one before), using the keywords of the line that is on screen longest, and a span longer than 6 s is split into equal parts.
- A clip is used once per video while unused ones are left. When they run out, take the least used clip again from a start point at least a second away from where its last use ended, and say so in the log. A clip shorter than its shot holds its last frame.
- Download clips to a cache folder, keep each clip's credit next to it there, and cache search responses for 24 hours.
- Write each clip's source, page URL, author and license into the credits record as it is downloaded.
- Fall back to a plain coloured background when a search returns nothing or a download fails, and say so in the log.
- Read first: https://pixabay.com/api/docs/ (section Search Videos)
- Done when:
  - Running the stage for the example script fills the cache and the credits record, and a second run makes no network calls.
  - A property test over random scripts (the standard `random` module is enough) finds every shot between 90 and 180 frames long.

### 6. Assembly

- Build the video with MoviePy 2. Let FFmpeg scale while decoding, crop to 1080x1920, cut with `subclipped`, give every caption a start, a length, a position and a margin, set the length of the composite yourself, and name the audio codec:

  ```python
  clip = VideoFileClip(path, target_resolution=(w, h))   # (width, height) that covers 1080x1920
  shot = clip.cropped(x_center=w / 2, y_center=h / 2, width=1080, height=1920).subclipped(0, seconds)
  cap = (TextClip(text=words, font=FONT_PATH, font_size=84, color='white', stroke_color='black', stroke_width=6, margin=(12, 42))
         .with_start(start_frame / 30).with_duration(frames / 30).with_position(('center', 1160)))
  video = (CompositeVideoClip([picture, *captions], size=(1080, 1920), use_bgclip=True)
           .with_duration(narration.duration).with_audio(narration))   # without with_duration the video ends with the last caption
  video.write_videofile(out, fps=30, codec='libx264', audio_codec='aac')
  ```

- `FONT_PATH` is the path of `Poppins-ExtraBold.ttf` in the repository, next to its license text. The caption sits a little below the middle of the frame, clear of the buttons that short-video players draw over the bottom.
- Do not join the shots with `concatenate_videoclips`: it adds float durations, and about one cut in six lands a frame late even when every shot is a whole number of frames long. Make `picture` one `VideoClip` whose frame function turns the time into a frame number (`round(t * 30)`) and finds the shot by comparing whole numbers.
- Read first: `https://zulko.github.io/moviepy/getting_started/updating_to_v2.html`
- Done when:
  - `final.mp4` exists and ffprobe reports 1080x1920, 30 fps, one H.264 video stream and one AAC audio stream, as long as the narration.
  - A frame grabbed in the middle of the widest caption shows complete letters: nothing is cut off at the sides, and descenders such as g, j, p and y are whole.
  - The picture differs between the frame before and the frame at every planned cut.

### 7. Checks and credits

- Add a `check` command: ffprobe for size, frame rate, length and codecs; FFmpeg's `volumedetect` for the mean audio level; three frames grabbed at caption midpoints (always including the widest caption), each compared with a reference drawing of the same caption made with Pillow; the picture before and at every planned cut; and a test for uniformly black bands on opposite edges of each shot.
- Write `credits.json` and a ready-to-paste description block with the credits each license asks for. A dry run says in both that it is a dry run.
- Done when: `python -m pipeline check out/example` prints a pass for every line.

### 8. Verify

- Commit, make a clean copy of the repository in a temporary folder (`git clone . /tmp/check`) and follow only the README there: install, run the dry run, run the tests.
- Run the dry run with the network off. On Linux, `unshare -rn -- python -m pipeline make --dry-run` runs a command without a network; elsewhere, switch the network off by hand or say that this was not checked.
- If a footage key is present, run the whole pipeline on a script the tests have not seen.
- Done when: Every line under Definition of done is true, and anything that could not be verified (for example the live footage search without a key) is named in PROGRESS.md.

## How to check your work

- After each stage, run its command on `scripts/example.json` and look at the output file, not only at the exit code.
- Use ffprobe to check a video instead of trusting that it was written.
- Look at frames: grab one in the middle of a caption after every change to the caption code.
- Keep network calls behind a cache so repeated runs are fast and do not spend API quota.

## Known traps

- MoviePy 2 removed `moviepy.editor`. Import from `moviepy`; `set_*` methods became `with_*`; `resize`, `crop` and `subclip` became `resized`, `cropped` and `subclipped`; effects are classes passed to `with_effects([...])`, which takes a list. (MoviePy docs, checked by running 2.2.1)
- With Pillow 11.2 or newer, MoviePy 2.2.1's TextClip cuts off the bottom of the text and of the stroke. The Stack pins Pillow below 11.2, and the vertical `margin` of half the font size protects you if the pin is ever lifted. The 12 px at the sides keep the stroke of the first and last letter whole. (measured with Pillow 11.1.0 and 11.3.0, Oct 12, 2026)
- Always pass TextClip the path of a font file you ship. A family name raises `ValueError: Invalid font`; leaving `font` out does not fail, it quietly uses Pillow's default font. (MoviePy docs and test build)
- `write_videofile` puts MP3 audio into an .mp4 unless you pass `audio_codec='aac'`. (checked by running 2.2.1, Oct 12, 2026)
- Google Fonts ships some families only as a variable file whose default weight is thin (Montserrat is one). TextClip cannot choose a weight, so captions come out as hairlines. Use the static file named under Assets. (google/fonts, checked Oct 12, 2026)
- kokoro 0.9.4 needs Python below 3.13. pip refuses it on 3.13, but uv installs it there without a word, so check the interpreter yourself. (PyPI, checked with pip and uv, Oct 12, 2026)
- When its espeak fallback is missing, Kokoro skips words it does not know and says nothing: its logger is switched off. The package brings its own espeak-ng through espeakng-loader, so the system package the README asks for was not needed in the test build. Check anyway: the recognised word count must equal the script's. (Kokoro source and second test build)
- The speech recogniser returns what it heard, not what you wrote: numbers, names and homophones can differ. Captions show the script's spelling with the recognised times. (test build)
- "At least 1080 px tall" is not enough for a vertical video: a 1920x1080 clip passes that test and still has to be enlarged 1.78 times. Test the size of the 9:16 crop. (Pixabay API docs and test build)
- Cache search results for 24 hours and download only the clips you use; Pixabay asks for both. (AgentGid: Pixabay)
- faster-whisper 1.2.1 breaks with PyAV 19, which came out on Sep 29, 2026: `TypeError: open() got an unexpected keyword argument 'metadata_errors'` at the first `transcribe`. The Stack pins `av` below 19. (PyAV changelog and second test build)
- A four-word caption at 84 px can be 1,400 px wide. TextClip makes the clip as wide as the text and the composite crops what does not fit, without an error. (second test build)
- `CompositeVideoClip(..., use_bgclip=True)` takes its length from the captions, not from the background: without `with_duration` the video stops when the last caption does. (checked by running 2.2.1, Oct 12, 2026)
- A TextClip has no start, no length and position (0, 0) until you set them: left alone, every caption sits in the top left corner for the whole video. (second test build)

## Warnings you can ignore

- Kokoro prints "WARNING: Defaulting repo_id to hexgrad/Kokoro-82M" when `repo_id` is not passed. The snippet in the plan passes it.
- "You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN ...": the two models are public and need no token. Do not ask the person for one.
- Three PyTorch warnings on the first Kokoro run (a dropout option, `weight_norm` is deprecated, `torch.jit.script` is deprecated). They come from the model code and do not affect the audio.
- MoviePy prints "unclosed file" resource warnings from its audio reader when warnings are switched on. They do not affect the output.

## Limits

- Every clip, track and voice has its own terms. The credits file is the record that shows what each video used.
- Do not clone a real person's voice without their explicit consent.
- Stock footage does not come with rights to the brands, landmarks or people in it. Avoid clips where those are the subject.
- Never read or print the contents of `.env` files, and never commit keys.

## When you need something that is not listed

Do not pick a library, asset source or API from memory. Ask the AgentGid MCP server first:

- `find_tools(goal)` for libraries, APIs and agents that fit a task.
- `find_assets(category, query, free_commercial_only=true)` for media sources and their license terms. It knows sources such as Kenney or Pixabay, not single packs, models or fonts.
- `get_api(api)` for prices, free tiers and limits of entries in AgentGid's API catalog. Media sources are not in it: use `find_assets`.
- `check_status(names)` when a service seems down.

Then tell the person what you want to add and why, cite the AgentGid URL, and wait for a yes before you add a runtime dependency.

## When this brief is wrong

Keep a list of what in this brief was wrong, missing or out of date, with the line and what worked instead (in PROGRESS.md). When you finish, send it to AgentGid's editors with `report_kit_issue(recipe="short-video", version="0.3", text=...)` on the AgentGid MCP server: only they read it. Leave out keys, personal data and the person's code.

## Sources

- https://www.python.org/downloads/
- https://agentgid.com/assets/kokoro/
- https://agentgid.com/tools/faster-whisper/
- https://agentgid.com/assets/moviepy/
- https://agentgid.com/tools/pillow/
- https://agentgid.com/assets/pixabay/
- https://agentgid.com/assets/google-fonts/
- https://fonts.google.com/specimen/Poppins
- https://agentgid.com/mcp/servers/io-github-upstash-context7/
- https://agentgid.com/connect/
- https://zulko.github.io/moviepy/getting_started/updating_to_v2.html
- https://pypi.org/pypi/moviepy/json
- https://raw.githubusercontent.com/google/fonts/main/ofl/montserrat/METADATA.pb
- https://pypi.org/pypi/kokoro/json
- https://raw.githubusercontent.com/hexgrad/kokoro/main/README.md
- https://pixabay.com/api/docs/
- https://raw.githubusercontent.com/PyAV-Org/PyAV/main/CHANGELOG.rst
- https://pypi.org/pypi/numpy/json