# MomentIQ Clip Evaluation Kit Use this kit before building `/v1/media/find-clips`, `/v1/media/score-clip`, or the heavy AI clip judge. The point is to prove that MomentIQ returns better clips than random timestamps and better clips than heuristics alone. ## Why This Matters MomentIQ's product promise is: > Scan long media cheaply first, then spend deeper review only on the best > candidate moments. That only works if we can measure "best." The evaluation set gives us the ground truth needed to tune heuristics, compare heavy AI providers, and avoid shipping a clip finder that merely sounds smart. ## Dataset Targets Start small, then expand. ### Minimum useful set - 5 podcast/interview videos - 2 comedy/laughter-heavy videos - 2 educational/webinar videos - 1 creator/livestream video ### Strong beta set - 20 podcast/interview videos - 10 comedy/laughter-heavy videos - 10 webinar/education videos - 10 creator/livestream videos Use videos that you are allowed to process. For public demos, use videos that you are allowed to show. ## Labeling Workflow Start with the source manifest before editing labels: ```bash npm run eval:sources -- docs/clip-source-manifest-template.csv \ --labels docs/clip-label-template.csv \ --plan-md results/eval/run-plan.md ``` If you are choosing real videos for the first time, use [clip-eval-owner-inputs.md](clip-eval-owner-inputs.md) first. It explains what source videos are acceptable, what the first dataset mix should look like, and what must exist before the heavy AI judge can be trusted. When you have one source ready, follow [first-clip-eval-packet-walkthrough.md](first-clip-eval-packet-walkthrough.md). To create working copies of the manifest and label templates, run: ```bash npm run eval:init ``` The generated working CSVs start header-only by default so example rows are not accidentally counted as real evaluation progress. The template files in `docs/` still show example rows for reference. The working CSVs and `results/eval/` outputs are gitignored because they may contain private source paths, signed URLs, and human labels. Keep real evaluation data local unless you intentionally sanitize it for a public demo. After initialization, the default status command auto-detects `docs/clip-source-manifest-working.csv`, `docs/clip-label-working.csv`, and a single packet folder under `results/eval/` when those files exist: ```bash npm run eval:status ``` For one specific source, generate a local copy-paste runbook: ```bash npm run eval:source:runbook -- podcast_001 ``` This writes `results/eval//runbook.md` with the exact preflight, dry-run, upload, job creation, fetch, packet, review, and label commands for that source. The runbook is gitignored with the rest of the local eval workspace. If you want to verify the full packet/status/index flow before using private media, create a sanitized demo packet: ```bash npm run eval:demo ``` This writes a local demo packet under `results/eval/` using only the sanitized example job and prediction files in `docs/`. It creates a packet, demo labels, status JSON, quality report, promotion gate, and packet index. Re-run with `-- --force` only when you intentionally want to overwrite the local demo. Add real videos with the source helper so paths, packet directories, and job result paths stay consistent: ```bash npm run eval:sources:add -- \ --source-id podcast_001 \ --title "First podcast eval" \ --source-url "C:\path\to\video.mp4" \ --source-type podcast_interview ``` The command defaults to `approved_private`, `collected`, `drafted`, and `tiktok`, and writes default paths under `results/eval//`. Manual CSV editing still works when needed; track `permission_status`, `collection_status`, and `label_status` for every source. Generate the real job request from the source row: ```bash npm run eval:sources:job-request -- podcast_001 ``` If you are not sure which step comes next for a source, use: ```bash npm run eval:source:next -- podcast_001 ``` It returns one next command or the one manual review step that must happen before automation can continue. That command prints a copy-paste `media/find-clips` `/v1/jobs` request from the manifest row. If the row uses a local file path, upload the file first: ```bash npm run eval:sources:upload -- podcast_001 ``` Set `MIQ_KEY` or `MOMENTIQ_API_KEY` to a real MomentIQ API key before running the upload. You can set it in the current terminal, or put `MIQ_KEY=...` in the gitignored `.env.local` file for repeat local eval runs. To validate the file, content type, size, and presign request without contacting the API/R2, run: ```bash npm run eval:sources:upload -- podcast_001 --dry-run ``` The upload helper calls `/v1/uploads/presign`, PUTs the bytes to R2, saves `results/eval//upload.json`, and prints the normal `eval:sources:job-request -- ` command. The job-request helper automatically uses the saved signed `media_url` from `upload.json`. If that URL expires, rerun `eval:sources:upload`. The upload and fetch helpers reject obvious placeholders such as `miq_dev_key` or `miq_live_...` before making a network request. Use a real hosted test/live key in `MIQ_KEY`, `MOMENTIQ_API_KEY`, or gitignored `.env.local`. For the normal first proof packet, use the guarded one-command path: ```bash npm run eval:source:run-hosted -- podcast_001 --dry-run npm run eval:source:run-hosted -- podcast_001 ``` The dry run validates the source file and prints the hosted sequence without network calls. The real run uploads local media if needed, creates the `media/find-clips` job, waits until it completes or fails, saves `job-result.json`, and builds the source eval packet. Use the individual commands below only when you need to inspect or repair a specific step. After upload, create the hosted worker job directly: ```bash npm run eval:sources:create-job -- podcast_001 ``` This calls `POST /v1/jobs`, writes `results/eval//job-create.json`, and prints the exact `eval:sources:fetch-job -- --wait` command. Use `eval:sources:job-request` only when you want to inspect the JSON request before spending credits. After the job completes, fetch the result by job id and save it to that source's manifest `job_result_path`: ```bash npm run eval:sources:fetch-job -- podcast_001 job_123 ``` Add `--wait` to poll until the job completes or fails: ```bash npm run eval:sources:fetch-job -- podcast_001 job_123 --wait ``` Set `MIQ_KEY` or `MOMENTIQ_API_KEY` first. If you already downloaded the job JSON manually, use the fallback: ```bash npm run eval:sources:save-job -- podcast_001 "C:\path\to\downloaded-job-result.json" ``` Then build that source's packet directly from the manifest: ```bash npm run eval:packet:source -- podcast_001 ``` This is the safest first-packet path because it reuses the manifest's source title, media path/URL, source type, platform, job result path, and packet directory instead of asking you to paste a long command by hand. Open the packet's `review-sheet.md` first. It gives a human-friendly watch list, rating guide, and done criteria. After copying those decisions back into `labels-draft.csv`, merge only the reviewed rows into the working labels file: ```bash npm run eval:labels:merge -- results/eval/podcast_001/labels-draft.csv ``` The merge command skips unrated rows by default, blocks duplicate `source_id + clip_label` rows unless you intentionally pass `--replace`, and refuses one-sided drafts that are not ready. A ready draft has at least one `good`/`great` row, at least one `okay`/`bad` row, required clip fields, and review notes. Check it with: ```bash npm run eval:labels:summary -- results/eval/podcast_001/labels-draft.csv ``` The next-step helper checks those same reviewed row keys, not just row counts, so it will still ask you to merge if the working labels file has different clip labels than the reviewed draft. Then mark the source reviewed in the manifest once enough labels exist: ```bash npm run eval:sources:mark-labeled -- podcast_001 ``` This requires 5 reviewed rows, 2 good/great rows, and 2 okay/bad rows by default. Use `--dry-run` before writing if you want to verify the counts. The bad or borderline examples are not busywork; they teach the evaluator what MomentIQ should avoid when tuning the clip finder. Finally, package the full source state and owner/Codex handoff without copy-pasting packet paths: ```bash npm run eval:source:package -- podcast_001 ``` That writes `status.json` and `handoff.md` into the source's eval packet directory by default. It wraps source manifest status, packet readiness, label readiness, prediction run readiness, next actions, and a human-readable handoff report for one source. The command also prints the single recommended next step after the package is written, so if the packet still needs label review, label merge, provider comparison, or owner action, you see that immediately in the terminal and inside the handoff. The lower-level commands still exist when you need custom output paths: ```bash npm run eval:source:status -- podcast_001 --out results/eval/status.json npm run eval:source:handoff -- results/eval/status.json --out results/eval/handoff.md ``` The handoff report summarizes bottom-line status, evidence files, packet checks, promotion-gate status, next actions, and owner review questions. The source status command checks the same source-type mix as `eval:readiness`, and can cross-check that every labeled `source_id` exists in the manifest. If a source has approved permissions and is collected, the command also prints a copy-ready `npm run eval:packet` command. Use `npm run eval:sources:save-job -- ` to save the completed MomentIQ job response to the listed `job_result_path`, then run that packet command to build the repeatable evaluation packet. The optional `--plan-md` file contains copy-ready packet commands plus a per-source checklist for saving job responses, building packets, reviewing `proof-demo.md`, filling `labels-draft.csv`, and copying final labels into your working label CSV. Re-run `eval:sources` after packet generation; it will report whether the job response exists and whether required packet files such as `predictions.json`, `labels-draft.csv`, `review-sheet.md`, `proof-demo.md`, and `README.md` are present. For each source video: 1. Watch or skim the video. 2. Mark 2-5 clips you would actually post, save, or send to someone. 3. Mark 2-5 bad or boring clips the system should avoid. 4. Add the intended platform: `tiktok`, `youtube_shorts`, `reels`, `podcast_promo`, `internal_search`, or `agent_memory`. 5. Score each clip with the rubric below. 6. Note why the clip works or fails in plain English. ## Timestamp Format Use `DD:HH:MM:SS` when writing labels by hand. Examples: - `00:00:03:42` = 3 minutes, 42 seconds - `00:01:02:05` = 1 hour, 2 minutes, 5 seconds - `00:00:00:00` = beginning of media ## Clip Quality Rubric Score each field from `0` to `5`. | Field | 0 means | 5 means | | --- | --- | --- | | hook_score | No reason to keep watching | Immediately interesting | | context_score | Missing setup | Clear enough without more context | | payoff_score | No payoff | Strong ending, laugh, insight, reveal, or answer | | shareability_score | Would not post/send | Would post/send as-is | | audio_quality_score | Hard to hear | Clear enough for publishing | | visual_interest_score | Visually useless | Good faces, action, frame, text, or object | | boundary_score | Starts/ends awkwardly | Clean start and clean end | | titleability_score | Hard to title | Obvious title or hook | Overall labels: - `great`: would publish or use with minimal editing - `good`: useful, but may need boundary/title tweaks - `okay`: understandable, but not exciting - `bad`: should not be returned as a top candidate ## What To Capture Use [clip-label-template.csv](clip-label-template.csv) as the working file. Required fields: - `source_id` - `source_title` - `source_type` - `source_url_or_path` - `clip_label` - `start` - `end` - `platform` - `overall_rating` - `why_good` - `why_bad` - score columns Optional fields: - `speaker` - `topic` - `quote` - `thumbnail_time` - `notes` ## How Codex Will Use This Once labels exist, Codex can help build tests that compare: 1. Random clips 2. Current heuristic candidates 3. `/v1/media/find-clips` heuristic ranking 4. OpenAI transcript-plus-frame judging 5. Gemini video-window judging 6. Twelve Labs video-native judging, if tested The metric that matters first: > Does the top 5 include at least one human-approved clip? Secondary metrics: - average human rating of top returned clips - false positive rate - clean boundary accuracy - cost per source hour - runtime per source hour - explanation usefulness ## Pass/Fail Gate Before Marketing Do not market MomentIQ as "really good at finding clips" until: - Top 5 includes at least one human-approved clip in 80%+ of labeled videos. - AI-reranked candidates beat heuristic-only candidates by human rating. - Cost remains meaningfully cheaper than running a premium model on full video. - Returned reasons match the actual clip content. ## Local Validation After adding labels, run: ```bash npm run eval:clips ``` This validates required fields, timestamp parsing, rubric score ranges, source counts, and approved clip counts. Later, when `/v1/media/find-clips` returns prediction JSON, compare predictions against labels: ```bash node scripts/evaluate-clip-labels.js docs/clip-label-template.csv path/to/predictions.json ``` To compare multiple methods on the same labels, pass named prediction files: ```bash node scripts/evaluate-clip-labels.js docs/clip-label-template.csv \ heuristic=results/heuristic.json \ openai=results/openai-standard.json \ gemini=results/gemini-premium.json ``` The report shows top-1, top-3, and top-5 hit rates, average best top-5 IoU, and average false positives in the top 5. Use this before choosing the default judge provider. The default should be the cheapest method that beats heuristic-only on human-approved clips. Prediction JSON can be either: ```json [ { "source_id": "src_001", "clips": [ { "candidate_id": "cand_001", "start": 222, "end": 268, "score": 0.91 } ] } ] ``` or a flat list of clips with `source_id`, `start`, `end`, and `score`. ## Export Real Job Results When a real hosted or local `media/find-clips` job completes, save the job response JSON and create an evaluation packet: ```bash npm run eval:packet -- results/demo-001/job-result.json \ --source-id podcast_001 \ --title "Podcast interview clip finder" \ --out results/demo-001 ``` This creates: - `results/demo-001/job-result.json` - `results/demo-001/predictions.json` - `results/demo-001/labels-draft.csv` - `results/demo-001/review-sheet.md` - `results/demo-001/proof-demo.md` - `results/demo-001/README.md` If you are using the source manifest, prefer the command printed by `npm run eval:sources`; it fills in `source_id`, title, source type, source URL, platform, and output directory from the manifest row. `review-sheet.md` is the easiest starting point for human review. It shows a watch checklist, plain-English rating guide, and the same candidate windows in a readable format. Copy your decisions back into `labels-draft.csv`, which contains one editable row per returned clip with `start`, `end`, `duration_seconds`, candidate score, and reason in `notes`. After copying reviewed labels, verify the packet before comparing methods: ```bash npm run eval:packet:check -- results/eval/podcast_001 \ --source-id podcast_001 \ --labels docs/clip-label-working.csv ``` To see the whole eval workflow status in one place, run: ```bash npm run eval:status ``` If you are checking custom paths or comparing named prediction runs, pass them explicitly: ```bash npm run eval:status -- \ --manifest docs/clip-source-manifest-working.csv \ --labels docs/clip-label-working.csv \ --packet results/eval/podcast_001 \ --source-id podcast_001 \ heuristic=results/eval/podcast_001/predictions.json \ --out results/eval/status.json ``` This prints the current source-manifest status, packet status, label readiness, prediction-run readiness, and next actions. It also writes a machine-readable JSON snapshot that can later feed an admin view or AI-agent handoff. To scan every local eval packet and build an owner review index, run: ```bash npm run eval:packets -- results/eval \ --out results/eval/packet-index.json \ --md results/eval/packet-index.md ``` This reports which packets are incomplete, ready for human review, or ready for a decision because they include both `quality-report.md` and `promotion-gate.json`. If you already have AI judge smoke outputs, include them in the same packet: ```bash npm run eval:packet -- results/demo-001/job-result.json \ --source-id podcast_001 \ --title "Podcast interview clip finder" \ --compare openai=results/demo-001/openai-standard-smoke.json \ --compare gemini=results/demo-001/gemini-premium-smoke.json \ --labels docs/clip-label-template.csv \ --out results/demo-001 ``` That also writes `predictions-openai.json`, `predictions-gemini.json`, `quality-report.md`, and `promotion-gate.json` when labels are available. Use the README checklist to watch the returned timestamps, add human labels, and run the comparison command. If you only need prediction JSON, convert the job response directly: ```bash node scripts/export-clip-predictions.js results/job.json \ --source-id src_001 \ --run-name heuristic \ --out results/heuristic.json ``` The exporter accepts common MomentIQ shapes: - a full `GET /v1/jobs/{job_id}` response - a direct `media/find-clips` response - an MCP `get_job_result` response - a raw array of clips If you only need a label draft CSV: ```bash npm run eval:labels:draft -- results/job.json \ --source-id podcast_001 \ --source-title "Podcast interview clip finder" \ --source-url "https://example.com/source.mp4" \ --out results/labels-draft.csv ``` Then generate the matching Markdown review checklist: ```bash npm run eval:labels:review -- results/labels-draft.csv \ --out results/review-sheet.md ``` ## Smoke Test The AI Judge Before enabling a live model provider for users, test the exact candidate file that came out of the cheap scan. Free deterministic local check: ```bash npm run smoke:ai-judge -- results/demo-001/job-result.json \ --quality standard \ --mock-provider \ --out results/demo-001/ai-judge-smoke-mock.json ``` Readiness-only check against real environment variables: ```bash npm run smoke:ai-judge -- results/demo-001/job-result.json \ --quality standard \ --out results/demo-001/ai-judge-readiness.json ``` Live provider smoke test, only after provider keys, model name, opt-in flag, budget caps, and data-policy approval are set: ```bash npm run smoke:ai-judge -- results/demo-001/job-result.json \ --quality standard \ --allow-live \ --out results/demo-001/openai-standard-smoke.json ``` The command writes normalized judge output, readiness issues, provider usage, and ranked results. It refuses to spend by default; `--mock-provider` is free, and `--allow-live` is required before a configured provider can be called. Export AI judge smoke results into the same prediction format: ```bash node scripts/export-clip-predictions.js \ results/demo-001/openai-standard-smoke.json \ --source-id podcast_001 \ --run-name openai-standard \ --out results/demo-001/openai-standard.json ``` Then compare the cheap scan and heavier judges against the same human labels: ```bash node scripts/evaluate-clip-labels.js docs/clip-label-template.csv \ heuristic=results/heuristic.json \ openai=results/openai-standard.json \ gemini=results/gemini-premium.json ``` Before spending time or provider budget, check what is still missing: ```bash npm run eval:readiness -- docs/clip-label-template.csv \ heuristic=results/heuristic.json \ openai=results/openai-standard.json ``` This prints the current label count, approved-clip count, okay/bad clip count, prediction runs, gate status if enough runs exist, source-type coverage, and the next concrete actions. Use it as the working status command while building the first labeled dataset. By default, it checks the first dataset mix from the owner action list: 10 podcast/interview sources, 5 comedy/laughter sources, 5 education/webinar sources, and 5 creator/livestream sources. Create a Markdown decision report for review: ```bash npm run eval:report -- docs/clip-label-template.csv \ heuristic=results/heuristic.json \ openai=results/openai-standard.json \ gemini=results/gemini-premium.json \ --out results/clip-quality-report.md ``` Use this report to decide whether `quality=standard` or `quality=premium` should stay in preview, move to more testing, or become a paid default later. It includes an evidence summary, run comparison table, per-source detail, and an owner decision checklist. Use that checklist to confirm clips, reasons, model cost, budget caps, and data-policy approval before enabling a paid heavy-AI quality tier. Evaluation packets with labels and comparison runs also save the strict gate as `promotion-gate.json`. Run the promotion gate before changing product defaults: ```bash npm run eval:gate -- docs/clip-label-template.csv \ heuristic=results/heuristic.json \ openai=results/openai-standard.json \ --candidate openai ``` The default gate expects at least 20 labeled sources, at least 20 human-approved clips, at least 20 okay/bad clips, a candidate top-5 hit rate of 80% or better, and a measurable lift over the heuristic baseline. A failed gate means the heavier AI path should stay in preview/testing. If readiness reports `needs_more_negative_labels`, the dataset already has enough sources and good clips for the current threshold, but it still does not have enough boring, weak, borderline, or bad examples. Do not skip this. Those examples are what teach the scorer and any heavier judge what not to promote. This is the main proof loop: run real jobs, export predictions, compare against human-approved clips, then pick the cheapest judge that actually improves the top returned clips. ## Comparison Rules Use the same source videos, labels, and candidate set when comparing providers. - `heuristic`: output from `quality=cheap`. - `openai`: output after transcript/frame judging. - `gemini`: output after short video-window judging. Do not compare a provider that saw more source material against one that saw less without noting it in the report. If Gemini receives short rendered windows, OpenAI should receive the matching transcript snippets, selected frames, and signal evidence for those same windows.