How to connect Sume MCP with TypeScript

Sume's step-by-step guides for two kinds of work. Read the right one first: each is a SKILL.md that tells you which `generate_*` calls to make and with what settings. 'product-photoshoot' covers product stills of any kind: catalog and packshot photos, styled product scenes, a product with a model, banners, carousels, a set of ad images, try-ons and restyles of an existing product photo. 'ad-video' covers a finished ad clip for a product, brand, app or service: a short vertical spot, a commercial, a teaser, a voiceover ad, or a presenter speaking to camera. Without an argument it returns the guide names and when each applies. With `workflow` it returns that guide's SKILL.md and the paths of its extra files, which `get_workflow_bundle_file` reads. Read-only. It costs nothing.

Returns one extra file, or a folder listing, from a guide that `get_workflow_instructions` loaded. Call it when that SKILL.md points to a file by path. For a folder, `include_contents: true` returns every file in it. Read-only. It costs nothing.

Lists Sume's published presets, which are ready-made recipes for a finished image or video, such as a product commercial, a serum drip, a before-and-after or a virtual try-on. Pass `query` to search names and descriptions. Pass `preset_id` to open one preset. You then get what `execute_preset` needs and the price: `max_usd` is the most one run may spend on generation, and `typical_usd` is what this workspace's recent finished runs of it cost. Browsing never runs anything. Run a preset only with `execute_preset`, and only when the person asks for it. Read-only. It costs nothing.

Returns a published preset's instructions (its SKILL.md) and the files next to it, without running the preset. Pass `preset` as its id (`sume-cream-squeeze`) or as a slash command (`/sume-cream-squeeze`). Leave it out to list the presets with their summaries. Read the instructions before you call `execute_preset`, so you know which inputs the preset needs. Read-only. It costs nothing.

Puts an image, video or audio file into Sume and returns the URL generation tools take. For a file attached in the chat, send it as `file` with a new `idempotency_key`. Sume downloads and stores it, and the answer holds `response.data.asset.url` (https://media.sume.com/...). That needs no PUT and no `media_confirm`. For a local file you can read, send `payload.content_type` and `payload.size_bytes` with a new `idempotency_key`. `filename` and `media_type` are optional. The answer holds `response.data.upload.url`, a short-lived PUT URL, with its method and headers, and `response.data.asset.id`. PUT the exact file bytes to that URL yourself, for example with curl --upload-file. Then call `media_confirm`. If you can do neither, ask the person for a public https link and pass that to `media_import_url`.

Finishes an upload from `media_upload`. Call it after your PUT of the file bytes to `upload.url` succeeded (HTTP 200). Send `asset_id` and a new `idempotency_key`. The answer holds `response.data.asset.url` (https://media.sume.com/...). Pass that URL to generation tools as an input image, video or audio.

Brings a web media URL into Sume so you can generate from it. A TikTok or Instagram post URL is imported. Sume fetches the video into its storage as a paid import job, and `dry_run` shows the price. Wait for the job with `jobs_wait`, then use the import's url as the input. Any other public https image, video or audio URL is registered as an asset, not copied, and you can pass it to generation tools as is. `idempotency_key` is required.

For product photos or an ad video, the workflow names the model and settings to use, so load it with `get_workflow_instructions` first. Lists, searches and describes Sume's image, video and audio models, and recommends models for a task. `action` picks what you get. `list` is the default and lists models by type. `search` matches `query` words in names and descriptions. `get` returns one model's parameters (aspect ratios, resolutions, quality tiers, how many references) and its price. `recommend` returns the default and the candidates for a task. For video it also returns the special tools (motion control, lip-sync, upscale) and when each one fits. If you leave the model out of `generate_image` and `generate_video`, Sume picks it (`sume/auto`), and that is the default. Set a model when the person names one, when a `model_routes` line from `recommend` fits better, or when the request needs a setting Auto does not take. Read-only. It costs nothing.

Returns the connected workspace's balance in US dollars as `available_usd`. Check it before a large or paid batch. Quote dollars, never credits. Read-only.

Lists the workspaces this connection can use: the person's personal workspace and every team they belong to. `is_selected` marks the workspace that generations bill and reads come from right now. Call it when the person names a team or asks where the credits come from. Read-only. It costs nothing.

Switches this connection to another workspace, given a `workspace_id` from `list_workspaces`. From the next call on, generations bill that workspace's balance, and assets, jobs and presets come from it. The switch lasts for this connection. Switch only when the person asks for that workspace. It spends nothing.

Product photos, packshots and other product-led stills need the 'product-photoshoot' workflow. Load it with `get_workflow_instructions` before your first `generate_image` call for them in a conversation. The workflow holds the modes, the one-product identity rule and the 2K settings. For the stills of an ad video, load 'ad-video'. Makes a still image from a prompt. It is not for motion (`generate_video`), audio (`generate_audio`) or removing a background (`rmbg_create`). `payload.n` on `generate_image` makes several takes of one request: same prompt, inputs and settings. When the requests differ (2 to 12 of them), send them in one `generate_image_batch`, `generate_video_batch` or `generate_audio_batch` call. `generate_video` and `generate_audio` make one result per call. Put every field inside `payload` and send a new `idempotency_key`. Send `payload.prompt`, plus any of the optional `aspect_ratio`, `resolution`, `n` and `input_references` (public https image URLs to edit or follow). Leave `payload.model` out and Sume Auto picks the model. Set one when the person names it or a route fits better. `openai/gpt-image-2.5` is the everyday image model: realistic photos, readable text in the image, and edits from reference images. Set `dry_run` to `true` to get the price in dollars without running anything. If a submit times out on the network, it may still have started, so do not send it again with a new key. Retrying with the same `idempotency_key` returns the job already started and charges once. The call is async. The answer brings a job id within seconds. Wait with `jobs_wait`, and the result then has the image URL.

An ad or promo video needs the 'ad-video' workflow. Load it with `get_workflow_instructions` before your first `generate_video` call for one in a conversation. The workflow holds the shot plan, the stills-first product identity, the clip settings and the timeline assembly. Makes a video clip from a prompt, optionally starting from an image. It makes one clip per call. For several clips, whether variants or not, use `generate_video_batch`. Some tasks have their own tool. To swap the people or objects in, restyle, or transfer the moves of one existing video from 1–8 reference images (Sume/Remix), use `remix_video_create`. To make a person in a still image move like a driving video, as in a motion follow, a dance or copying the moves, use `kling_motion_control_create`. To lip-sync a still to a finished voice track, use `avatar_image_to_video_create`. To make a finished clip larger, use `video_upscale_create`. Put every field inside `payload` and send a new `idempotency_key`. Send `payload.prompt`, plus any of the optional `duration`, `resolution`, `aspect_ratio`, `generate_audio`, `frame_images` (`first_frame` or `last_frame` image URLs) and `input_references` (style or character references). Leave `payload.model` out and Sume Auto picks the model. Auto takes `aspect_ratio` 16:9 or 9:16 only. For 1:1 or another ratio, name a model that lists it in `models_explore`. Auto clips always carry their own sound, so leave `generate_audio` out with Auto. `generate_audio` set to `false` is for a named model that lists it. `seedance-2.5` is the everyday video model; `kling-4` handles several shots in one clip or clips with sound; `minimax-h3` takes 2K keyframes and mixed references. To put a different person, product, garment or object into one existing video from 1–8 images, use remix_video_create with `sume-remix-object-swap`. A `generate_video` object-swap model takes one source video plus 1–8 images in `payload.input_references`, with `payload.duration` set to the source's length (4–30 seconds); the result keeps that length and framing. Set `dry_run` to `true` to get the price in dollars. If a submit times out on the network, it may still have started, so do not send it again with a new key. Retrying with the same `idempotency_key` returns the job already started and charges once. The call is async, and video takes 1–5 minutes. Wait with `jobs_wait` and keep waiting. The result then has the clip URL.

Before you make an ad's music or voiceover, load the 'ad-video' workflow with `get_workflow_instructions`. Makes audio: speech from a script, or music from a description. For speech, send `payload.transcript` and a `payload.voice.id` from `tts_router_voices`. Use a family sonic voice with the model left out (Sume TTS 1.0), or a family eleven voice with model `eleven-v4`. For music, send `payload.prompt` only, with the mood, genre, tempo and instruments. Set model `sume/music-1.0` or leave it out. Music has no length field. The track runs about a minute, and `timeline_create` trims it. Put every generation field inside `payload` and send a new `idempotency_key`. Set `dry_run` to `true` to get the price in dollars without running anything. The call is async. The answer has a job id. Wait with `jobs_wait`, and the result then has the audio URL.

Load the 'product-photoshoot' or 'ad-video' workflow with `get_workflow_instructions` before your first `generate_image_batch` call in a conversation for product photos or the stills of an ad. Submits up to 12 independent `generate_image` requests in one call, for example one still per scene. Each request carries its own `index`, `idempotency_key` and `payload`, and runs exactly like `generate_image`. The answer lists the submitted job ids in index order, plus any failed index with its reason. Pass `job_ids` to `jobs_wait`. With `dry_run` set to `true`, every request is priced and nothing runs.

Load the 'ad-video' workflow with `get_workflow_instructions` before your first `generate_video_batch` call in a conversation for an ad or promo video. Submits up to 12 independent `generate_video` requests in one call, for example one clip per scene. Each request carries its own `index`, `idempotency_key` and `payload`, and runs exactly like `generate_video`. The answer lists the submitted job ids in index order, plus any failed index with its reason. Pass `job_ids` to `jobs_wait`. Video takes minutes, so keep waiting. With `dry_run` set to `true`, every request is priced and nothing runs.

Submits up to 12 independent `generate_audio` requests in one call, for example one voice line per scene. Each request carries its own `index`, `idempotency_key` and `payload`, and runs exactly like `generate_audio`. The answer lists the submitted job ids in index order, plus any failed index with its reason. Pass `job_ids` to `jobs_wait`. With `dry_run` set to `true`, every request is priced and nothing runs.

Runs one published Sume preset from `get_presets` and makes its finished image or video. Call it only when the person asks to generate with that preset. Before that, read `get_preset_instructions` and collect the inputs it needs. Send `preset_id`, a new `idempotency_key`, an `instruction` in the person's words, and `attachments` with the public https images the preset should use. The run spends from the workspace balance, up to `generation_spend_cap_usd` if you set one. Set `dry_run` to `true` to run nothing and get the price (`max_usd`, `typical_usd`). The call is async. The answer has a run id (arun_...). In clients that show MCP Apps (Claude), Sume's viewer shows the result. Elsewhere, poll the run with `job_status`, which returns the result once the run finishes.

Waits for jobs to finish and can return their result URLs in the same answer. Pass `job_id`, or `job_ids` (up to 20) from `generate_*` or a batch, and set `include_results` to `true` to get the result URLs. Each call holds for about 50 seconds and returns early once every job is done. When `timed_out` is `true`, call it again in this turn with `wait_slice_expired.pending_job_ids`. Images usually finish in under a minute, and videos in 1–5 minutes. Read-only.

Reads one job's current state without waiting. It takes a generation job id from `generate_*` or a preset run id (arun_...) from `execute_preset`. While the job runs, the answer carries `poll_after_seconds`: the pause to take before the next read. A finished preset run also returns its result: `output`, `primary_output_url` and `artifacts` with media URLs. To check several jobs at once, use `jobs_wait`. Read-only.

Returns public API usage ledger entries (spend/refund-style history) through GET /v1/usage, in a Sume ledger shape. With `thread_id`, `run_id` or `job_id` it returns the exact spend of that one thread, run or job. When the user asks what this thread, this run or this generation cost, pass `thread_id` (in a Studio Agent turn, the current thread), `run_id` (arun_…) or `job_id`, and quote `summary.debited_usd`: what the wallet deducted, in dollars. `summary.held_usd` counts holds that have not settled yet, and `summary.refunded_usd` counts money given back. Neither one is spend. `includes{}` says what the sum covers: LLM turns, sidecars, Browser, and generation jobs. `cap{}` on a run is the generation-cap accounting the receipt prints, never the total cost. `final=false` means a hold is still open. Without a scope, use it to audit recent charges or explain wallet movement. For the live available balance use `balance`. Do not invent credit packs or trial-cancel tools. Never sum rows yourself, because a refunded row keeps its hold amount. The optional `limit` (default 20) caps the rows listed, not the sum. Read-only. Do not echo secrets from ledger metadata into user-facing reports.

Lists recent generation-job status: in a Studio Agent turn, this thread's jobs; with an API key or a sign-in connection, your own jobs in the workspace. It is the history browser: use it and the product .list tools instead of inventing `show_generations`. Use it to recover a `job_id` or live status after a disconnect. For a known id's live status use `job_status/jobs_wait`, for outputs `jobs_result`, to cancel `jobs_cancel`. In a Studio Agent turn it lists this thread by default: every job in it whoever created it (an API-fired Format run's included). `scope=workspace` (interactive chat) lists your own jobs across sibling threads in the same workspace instead. By id you can get/wait/result any job in this thread, plus your own sibling-thread jobs in interactive chat, but other members' sibling-thread jobs and other workspaces stay hidden (404). Identity is the `job_id` you send after lint, so never pick the latest sibling clip by type. Order is recency, not your timeline, so after retries the newest-first page does not match your submit order. Every row carries the `idempotency_key` it was created with. Join your labels on that key, never on array position. Without distinct keys at create, the list cannot tell you which job was which, and you must not guess. It answers 10 jobs a page by default (`limit` up to 100) as summaries: id, type, status, model, idempotency_key, times, output URLs and the error reason. To get the following page, send its `next_cursor` as `cursor`; filter with `type` or `status`, and send `detail: "full"` for whole job records. Responses may include sensitive/signed URLs. Do not echo them in agent reports. Next, call `job_status` or `jobs_wait`, then `jobs_result` once terminal.

Returns one job record by `job_id`, a fuller snapshot than status alone. Use it to inspect a known job after list/create. It is also the only tool that can tell you why a job failed. For tight polling, prefer `job_status` or `jobs_wait`. Do not use it to read outputs before terminal success (`jobs_result`) or to cancel. `job_id` is required. You can read any job in this thread whoever created it, including an API-fired Format run's spine with its voice and model. In interactive chat you can also read your own jobs in sibling threads. Anything else returns 404. The record carries `idempotency_key`, so a job you fetch can be re-attached to the label you created it under. An image, video or music generation carries `input`: the prompt, settings and input media it ran with, next to its `model`. When asked for another one like it, reuse them with that model. A failed job carries the whole error. Read `error.public_reason` rather than `error.code`, which nearly every generation failure shares. Read `error.message` for the sanitized provider/upstream reason (duration, refs, model rejects), and `error.retryable`. The record may include sensitive URLs. Never paste signed/private media URLs into user-visible text. Next, while it still runs, call `jobs_wait` or `job_status`. After terminal success, call `jobs_result`.

Returns the results/artifacts of a job that reached terminal success. It takes `job_id` (single) or `job_ids[]` (1–20, the `jobs_wait` ceiling). Call it only after `jobs_wait/job_status` (or a product get) shows a successful terminal state, and only for ids `jobs_wait` with `include_results` did not already return. After a wave, make ONE batch `jobs_result` call on the ids you just waited on, not one per clip. Do not call it while a job is queued/processing (wait first), as a poll substitute, or to cancel. Send exactly one of `job_id` or `job_ids`. You can read this thread's jobs whoever created them, plus your own sibling-thread jobs in interactive chat. A batch returns `results[]`, one entry per id, each with `ok` plus a `value` or a typed error. A 409 on one id keeps the other results, so read `ok` per entry and re-read only the failed ids. Do not echo signed/private media URLs or secrets in agent reports. On soft-fail package stages the primary outputs may still be usable, so inspect the status fields before you retry paid work. A 409 `job_not_completed` on a queued/processing job means it has not finished yet. It is neither a failure nor a verdict. Repeated 409s are no evidence the job is dead: keep waiting with `jobs_wait` on the same id until its own status is terminal. Dropping a job from your plan after a 409 abandons work that is already billed and still running. A job already in status failed answers with the same 409 and no error fields, so this tool cannot tell you why anything failed. Call `jobs_get` for `error.public_reason`, `error.message` and `error.retryable`. A 409 on a failed job is never a reason to resubmit the paid create.

Cancels a job that has not reached the provider yet, through POST /v1/jobs/:id/cancel. Once provider work has started, it answers 409 `job_generation_already_started` and the job runs on. This is a destructive write. Use it only when the user explicitly asks to stop a live job. Do not use it for routine polling (`job_status` or `jobs_wait`) or to fetch results (`jobs_result`). Send an `idempotency_key` (the legacy `allow_write` is accepted but not required), and confirm with the user before you cancel. Do not resubmit the paid create unless the user asks. Next, call `job_status` or `jobs_get` to confirm the terminal canceled state. Start a new create only after an admission preview, and only with a fresh `idempotency_key`.

Purpose: POST /v1/avatar-1.0/image-to-video is a deprecated REST compatibility alias. Use POST /v1/veed/fabric-1.0 (public model id veed/fabric-1.0), same body. This MCP tool and the alias still work. Send { idempotency_key, model?, payload }. Payload: audio_url, duration_seconds (1–300, audio length) and exactly one visual source, image_url XOR avatar_id / avatar_handle. Never send image_url with an avatar reference. audio_url must be ≤10 MB on the Sume media host (mirror it first), else unsupported_audio_source. Models: model (or payload.model) is veed/fabric-1.0. Never re-cut audio. When to use: lip-sync over audio that must stay as given (user's recording, verified narration spine, saved avatar's line) or an asked lip-sync. New on-camera speech is generate_video seedance-2.5 (still + voice take + lines), not this. When not: wordless beats, B-roll, product motion or a remake Kit scene (generate_video, seedance-2.5). Prefer image_url of a newly generated, inspected posed still. Use avatar_handle only if user-named. Fabric 720p $0.1500/audio s (480p $0.0800/s). Fabric talk clips render at 25 fps. Keep timeline output.fps 25. No provider model ids or endpoints, no lipsync-1.0, and no prompt or script field: audio drives speech. Paid create, billed per audio second via wallet/admission. Send an idempotency_key. Optional: dry_run true for cost. max_spend_usd caps spend. One budget approval per wave, within generation_limits.concurrency_limit. Then jobs_wait, then jobs_result, then video_inspect, then timeline_create with continuous VO.

Submits a Kling 3.0 Motion Control job through POST /v1/kling/3.0/motion-control and returns a job id. The public model id is kling/3.0/motion-control. A caller-supplied motion video drives one still, or one ready avatar still. Output length follows the driving clip. Use it when the user @-named Kling Motion Control or kling/3.0/motion-control, or asked to transfer motion from a driving video onto a still. Do not use it for Kling 3.0 cinematic t2v/i2v, which is `generate_video` with model kling-3. Do not use it for audio-driven Fabric talk clips (`avatar_image_to_video_create`). It is also not an edit-studio motion_control tool, which this server does not have. The payload needs three things. First, `motion_video_url`: the public HTTPS driving clip, max 30s. Second, duration_seconds (1–30): the driving clip's length, used for admission only. Third, exactly one visual source: `image_url`, or `avatar_id` / `avatar_handle`. `image_url` together with an avatar reference is rejected. `keep_original_sound` defaults to `true`. `character_orientation` defaults to `video`, which follows the driving clip framing. The optional `prompt` steers appearance only. Motion follows the driving video. Do not send `model`, `endpoint` or `provider_endpoint`. Do not call `generate_video` with this model id, because it cannot carry a driving video. Do not treat this as another kling-3 i2v preset. Do not invent a motion_control widget. Do not redisplay the @ chip as done. Also, do not refuse it as edit-studio motion_control. This paid create bills per driving-video second via wallet/admission. Send an `idempotency_key`. `dry_run` set to `true` for cost is optional. `max_spend_usd` is optional and enforced when provided. Next, capture the `request_id` or job id, call `jobs_wait`, then `jobs_result` for the media.sume.com `video_url`.

Runs one Sume Remix endpoint: an edit of one existing video, one model per endpoint. `model` is `sume-remix-motion-transfer` (Sume/Remix Motion Transfer), `sume-remix-object-swap` (Sume/Remix Object Swap), `sume-remix-restyle` (Sume/Remix Restyle), `sume-remix-lite-motion-transfer` (Sume/Remix Lite Motion Transfer), `sume-remix-lite-object-swap` (Sume/Remix Lite Object Swap). Motion transfer makes 1–8 image subjects do the clip's moves, object swap (the default) replaces people, products or objects with 1–8 images, restyle turns the whole clip into one of Sume's styles, picked by `payload.preset_id` (required; the list is at GET /v1/sume/remix/restyle/v1.0/presets) and with no images. Send `payload.video_url` (4–15 s, at most 200 MiB, public https), `payload.image_urls` (or `payload.preset_id` on restyle) and optionally `payload.prompt`, `payload.resolution` (480p, 720p default, 1080p) and `payload.duration` (the source's length in seconds; read from the file when left out), with a new `idempotency_key`. The output keeps the source's length and the source's sound. The price counts source seconds; `dry_run` set to `true` returns it in dollars and runs nothing, and `max_spend_usd` caps it. A refusal fails the job and refunds it. The call is async: wait with `jobs_wait`, and the result has the clip URL.

Lists the voices a `generate_audio` take can speak with. family eleven (the default) is for model eleven-v4, and sonic is for the public Sonic voices. Each voice has id, name, description, gender, accent, language and tags. Use it before a one-off take: an ad line, a character, UGC or an emotional read. Match the described voice (gender, age, accent, tone, energy) to one voice, then send its id as `generate_audio` `payload.voice.id` with model eleven-v4. Read-only: it spends nothing. The list is shared and is not the workspace library, so picking a voice here saves nothing. It answers one page of 50 voices. `total` counts the voices that match. For the following page, send that `next_cursor` as `cursor`; a null `next_cursor` means you have them all. Narrow the list with `query`, `gender` and `language` instead of reading every page.

Removes the background from an existing still through POST /v1/rmbg-1.0/remove and returns a cutout PNG with alpha. The public model id is sume/rmbg-1.0. It is not a generator and needs an input `image_url`. Use it for any cutout, remove-background, 배경제거 or no-background ask. That includes a source that already has alpha, so never skip it. Transparent output is not a generation parameter on Sume. Generate the still with `generate_image` first, then cut it out here. Omit `sync_mode` to keep the provider default. Substantial-alpha inputs are auto-flattened onto mid-gray #808080 before background removal. You are billed once. The payload needs `image_url` (public HTTPS). Prefer an attached/prior media.sume.com artifact. Optional fields are `sync_mode`, `metadata` and communication fields. Do not send provider model ids, `endpoint` or `provider_endpoint`. There is no remove_background tool. Do not ask `generate_image` for a transparent background. Do not skip the call when the image is already cut out. This is a paid create. It submits and bills subject to wallet/admission. Send an `idempotency_key`. It serves transport/dedup, not human approval. Optionally set `dry_run` to `true` to preflight the cost. `max_spend_usd` is optional and enforced only when you send it. Legacy allow_write/allow_paid flags are accepted but not required. Check with the user only when the run is costly or would delete work. Do not invent provider model catalogs. Next, capture the job id, call `jobs_wait`, then `jobs_result`. Show the result with the Assets filename BP as alt text: the ASCII source stem plus the transform, for example raymond-input_nobg.png. Never use a localized phrase like 배경 제거 결과 or a bare output.png. It costs $0.02 per image.

Upscales an existing clip through POST /v1/video-upscale-1.0/upscale and returns a job id. The public model id is sume/video-upscale-1.0. It is not a generator and needs a `video_url`. Use it to raise the resolution of short product/B-roll clips after `generate_video`. That includes 업스케일 and 화질 개선 asks on a clip that already exists. Do not use it to generate new video (`generate_video`). Do not use it on stills, because Sume does not upscale stills. Do not regenerate a clip to make it bigger. `resolution` defaults to `1080p`, and `enhancement_tier` defaults to `fast`. Sume reads the clip's length from the file and bills every second of it, at most 600 s; `duration_seconds` does not set the price. `pro` bills 10x the `fast` / `standard` rate. The payload needs `video_url` (public HTTPS). Optional fields are `resolution` (`1080p`, `2K` or `4K`, priced per second at that tier), `enhancement_tier` (`fast`, `standard` or `pro`), `duration_seconds`, `metadata` and communication fields. Do not send provider model ids/endpoints. This is a paid create. It submits and bills subject to wallet/admission. Send an `idempotency_key`. It serves transport/dedup, not human approval. Optionally set `dry_run` to `true` to preflight the cost. `max_spend_usd` is optional and enforced only when you send it. Legacy allow_write/allow_paid flags are accepted but not required. Check with the user only when the run is costly or would delete work. Do not invent provider model catalogs. Next, capture the job id, call `jobs_wait`, then `jobs_result`. A video upscale can take 1–3+ min. It costs $0.0069 per input second.

Extracts stills from a Sume-hosted video at times you choose (POST /v1/video-frames). An async video_frames job returns durable media.sume.com image artifacts. Use it to check a finished render at a specific second, grab a first frame (at: [0]) for a thumbnail, compare two moments, or read on-screen text at a known timestamp. Not for whole-clip evidence (video_inspect) or for changing the video (timeline_*). It gives exact stills only, so reuse suitable frames. Whole-clip evidence comes from video_inspect. Send idempotency_key and payload.video_url (media.sume.com, owned by this workspace). Import a social URL with media_import_url first. Then send exactly one of payload.at (1-24 explicit seconds) or payload.fps (0 < fps <= 2, expanded to mid-bin samples and capped at 24 frames). Optional payload.format jpeg|png (default jpeg) and payload.max_edge (16-2160 long-edge clamp). Omitting max_edge keeps the source size. Never send vf, filter, filter_complex, select, ffmpeg, cmd, codec, crf or preset: they fail with ffmpeg_fields_rejected. You choose the instants and Sume compiles the extract. Never shell out to ffmpeg and never pip install an ffmpeg wheel. By contract, the agent sandbox has no encoder. Limits: 300s source duration cap and 24 frames per call. Every at value must satisfy 0 <= t < duration, or the job fails frame_time_out_of_range naming the probed duration. Call video_frames_create, then jobs_wait on request_id (seconds, not minutes), then video_frames_get for frames[{t,url,width,height}]. A single frame whose extract failed comes back with url null, and that does not fail the job. Paid by run seconds: dry_run:true prices it, max_spend_usd caps it, and idempotency_key keeps a retry from billing twice. Next, jobs_wait on request_id, then video_frames_get. Hand the frame URLs to the user directly: they are durable artifacts.

Reads one frame-extract by id (GET /v1/video-frames/:id). It returns status, plus frames[{t,url,width,height}] once ready. The id is the `request_id` that `video_frames_create` returned. There is no separate resource id. Use it after `video_frames_create` and `jobs_wait`. Not a substitute for `video_inspect`. To re-extract at different times, call `video_frames_create` again. It gives exact stills only, so reuse suitable frames. Whole-clip evidence comes from video_inspect. `resource_status` ready means frames[] is populated. Each url is a media.sume.com image artifact, or null for one instant ffmpeg could not decode. `source_duration_seconds` is what the worker probed. failed surfaces worker codes such as `frame_time_out_of_range`, `invalid_source_url` and `duration_out_of_range`. It is read-only. Never echo temp paths, signed source URLs or provider ids. Next, report or embed the frame URLs. If you need different instants, submit a new `video_frames_create`.

Inspects ONE media.sume.com clip (Video inspect 1.0, POST /v1/video-inspect). Returns a probe (duration, size, rotation, fps, codecs, has_audio, hdr), up to 24 samples in 1–3 artf_ contact-sheet grids (up to 8 tiles each, preserved tile resolution) and an optional STT transcript with word timings. Read-only: never re-encodes, never a new MP4. Requires idempotency_key. Paid; transcribe:true bills the STT. dry_run prices it. The default mode is sync: it waits ≤30 s for the finished inspect, else the queued job goes to jobs_wait. Call, read, decide. Omitted frames give one grid of up to 8 samples. frames:false gives the probe only. Or set frames:{at[]|fps, format jpeg|png, max_edge, seek precise|fast}. fast snaps to the prior keyframe (earlier ≤1 GOP, never later): skimming only. transcribe:false is the default. Request true only when speech is needed and has_audio. Optional language_code. segmentation.mode "sentence" makes segments[] supply timed speech for captions. Use it only for missing duration or visual evidence. Before assembly/filtering, reuse existing results. Samples are not typed scenes. Not video_frames_create (exact frame at t, source size). Sources are media.sume.com, generated in this workspace. Dead: source_not_found. Source ≤1800 s. at values outside [0, duration) fail frame_time_out_of_range, naming the duration. transcribe on a silent clip fails inspect_source_has_no_audio. Never pass -i, vf, af, argv, codec, crf or filtergraph fields (400 ffmpeg_fields_rejected). The sandbox has no ffmpeg, ffprobe, OpenCV or media wheels. Never shell out, never pip a decoder, never curl the media into the box. Result: kind video_inspect with probe{…}, frames[{t,url,width,height,sample_times,columns,rows,tile_width,tile_height}], transcript{text,words[],segments[],audio_url}|null, warnings[]. Tiles read left-to-right, then top-to-bottom. Open each grid once. Then take the sync answer, or jobs_wait then jobs_result.

Lists the Fastlane video models: Fastlane Green Screen (`fastlane/green-screen`), Fastlane Wall of Text (`fastlane/wall-of-text`) and Fastlane Slideshow (`fastlane/slideshow`). Each row has price_usd, aspect_ratios and the inputs it needs. It calls GET /v1/fastlane/models. Read-only and unbilled.

Lists this workspace's Fastlane brands, newest first, then Sume's shared default brand (sume.com, `system: true`), which every workspace can use with no registration fee and cannot delete: id, name, website_url, description, status (analyzing, ready or failed), system and the failure when one failed. A video on the default brand costs the normal video price. It calls GET /v1/fastlane/brands. Call it when the user wants a Fastlane video for a brand, so later calls get a real brand_id; when the list is already in this conversation, reuse its ids. A Fastlane video is always made against a ready brand. Read-only and unbilled.

Reads one Fastlane brand: status analyzing, ready (the brand's fee is charged then) or failed (refunded). Brands finish on their own; reading one only shows where it is. It calls GET /v1/fastlane/brands/:id. Use it to answer whether a brand is ready, when the user asks. Analysis takes a few minutes: do not poll it on a timer or in a loop. Read-only for the caller and unbilled.

Registers a brand from its public website for Fastlane: Fastlane reads the site and builds the brand profile that every video uses. It costs $1.00, held now and charged only when the analysis is ready; a failed or abandoned analysis is refunded. It calls POST /v1/fastlane/brands. payload: `website_url` (required, the brand's own https site), optional `name` and optional `description` (20+ characters help when the site says little). A website this workspace analyzed in the last 30 days comes back as the same brand with `reused: true` and no charge; send `refresh: true` in the payload only when the user asks for a new paid analysis. Quote first with dry_run=true, tell the user it costs $1.00 and get their yes before submitting. The answer has status analyzing. Read fastlane_brands_get when the user asks whether it is ready.

Makes one short marketing video for a ready Fastlane brand. It costs $0.05, held now and charged when the video is stored; a video Fastlane cannot make is refunded. It calls POST /v1/fastlane/videos. payload: `brand_id` (a ready brand from fastlane_brands_list) and `model`: `fastlane/green-screen` (a presenter over a backdrop), `fastlane/wall-of-text` (bold text over brand visuals) or `fastlane/slideshow` (swipeable slides as an MP4, each slide also stored as an image). `format` (green-screen, wall-of-text, slideshow) is accepted in place of `model`. slideshow takes `aspect_ratio` 1:1, 4:5, 3:4 or 9:16 (default 9:16); the video formats are 9:16. Quote first with dry_run=true, tell the user it costs $0.05 and get their yes before submitting. After it is submitted, call jobs_wait, then jobs_result.

Cuts ONE media.sume.com clip to a range (Video trim 1.0, POST /v1/video-trim). An async job returns one NEW artf_ MP4 that holds only [start, end). It is compiled server-side on the hosted encoder. Use it for a requested standalone cut or a required conform. For assembly-only ranges, use timeline source_in without an intermediate encode. The result is a normal clip for `timeline_create` video[] (source_in 0), `timeline_compose` video.url or `video_filter`. It is NOT placement: sequencing, transitions and the audio spine belong to `timeline_create`. Send `idempotency_key`, `payload.video_url`, start and exactly one of end / duration. precision "exact" is the default: a frame-accurate re-encode, with optional output {width,height,fps} to conform once. precision "keyframe" is a stream copy that may start a GOP early, and the result then reports `actual_start_seconds`. audio is keep|drop. Limits: source ≤1800 s and output ≤900 s. Past the source it clamps (`trim_clamped_to_source`). It holds $0.1024 and charges its compute at list, idle time included, never above the hold. Refusals name the fix: `video_trim_range_required`, _conflict or _empty, and `video_trim_output_requires_exact`. Never pass -i, vf, af, argv, codec, crf or filtergraph fields (400 ffmpeg_fields_rejected). The sandbox has no ffmpeg, ffprobe, OpenCV or media wheels. Never shell out, never pip a decoder, never curl the media into the box. The result is kind `video_trim` with `video_url` (new artf_, never the source), `duration_seconds`, `actual_start_seconds`, `precision` and output{…}. Call `video_trim`, then `jobs_wait`, then `jobs_result`.

Extracts the audio of ONE media.sume.com video (Audio detach 1.0, POST /v1/audio-detach). An async job returns one NEW media.sume.com audio artifact: wav pcm_s16le sample-exact by default, or mp3. The video is untouched. Use the file as `timeline_create` audio.url (re-cut B-roll under the original narration), for `timeline_audio` split / concat, or for voice work. Do not use it to slice many ranges. Detach once, then run `timeline_audio` operation split. Spine-grade is the default: wav at the source rate and channels. Never use `video_inspect` transcript.audio_url (a 16 kHz mono STT wav) as `timeline_create` audio.url, or the render warns `audio_spine_low_fidelity`. Send `idempotency_key` and `payload.video_url`. You may add range {start,end} (one range), channels source|mono, sample_rate 16000|44100|48000 (16000 mono is the STT shape) and format wav|mp3. A source with no audio track fails `detach_source_has_no_audio`. Reuse a known has_audio, and inspect with frames:false only if it is unknown. Limits: source ≤1800 s and output ≤900 s, so a whole track past that needs a range. It holds $0.1024 and charges its compute at list, idle time included, never above the hold. Never pass -i, vf, af, argv, codec, crf or filtergraph fields (400 ffmpeg_fields_rejected). The sandbox has no ffmpeg, ffprobe, OpenCV or media wheels. Never shell out, never pip a decoder, never curl the media into the box. The result is kind `audio_detach` with `audio_url` (new artf_), `duration_seconds`, `format`, `channels` and `sample_rate`. Call `audio_detach`, then `jobs_wait`, then `jobs_result`, then use `audio_url`.

Renders one audio track and ordered video[] into one MP4 (Timeline 1.0, POST /v1/timeline-1.0/render), an async timeline_render job. audio.duration_seconds sets output length. audio.source_in trims the spine in-point (no re-TTS), so re-base video[].start. video[].start is authoritative: never pre-shift, Sume compensates xfade. Send idempotency_key, payload.audio {url,duration_seconds} and payload.video[] ({source_url,start,duration}, optional source_in/fit/transition). transition.type: fade|wipeleft|wiperight|slideup|slidedown|dissolve (crossfade is fade). Optional: output, soundtrack, render, metadata.production_id, dry_run, max_spend_usd. ≤$0.2183/started output minute. Limits: ≤200 segments, ≤1800s. video[0] starts at 0 with no transition. Transitions: ≤1s, ≤50% of shorter neighbour, whole frames at output.fps. ≤8 chained fades (too_many_chained_transitions). Sliced VO: audio.parts[] (≤20, url+source_in/duration each) replaces url: gapless concat, no re-TTS. No VO: audio.mode "silence" (no url/gain_db/parts) for silent montage, BGM-only or end card. output.fade_in_seconds/fade_out_seconds: each ≤5s, sum ≤ output. soundtrack {loop, fade_out_seconds ≤10s, duck_db 0-20, spine only}. Stills are static holds (motion ignored). fit: cover|contain|stretch|blur. Omit output.fps to match sources. Another rate repeats/drops frames (judder, output_fps_resamples_sources). Use durable media.sume.com assets generated in this workspace (this thread or a sibling), never external. After remirror, reuse those URLs (do not remirror again). Dead soundtrack: bgm_source_not_found (pick another/omit). Dead clip: source_not_found video[N]. Provider/ffmpeg fields rejected. Dim with video_filter. Prefer dry_run/plan. Auto-chunks past 12 segments. Short sources loop. render {strategy auto|single|chunked, chunk_max_segments 4-32, pad_mode auto|loop|freeze}. Call timeline_create, then jobs_wait, then timeline_get. Paid write: honour max_spend_usd/dry_run.

Combines or splits Sume-hosted audio into durable media.sume.com files (Timeline 1.0, POST /v1/timeline-1.0/audio) as an async job of result kind timeline_audio. To combine, send payload {operation:"concat", parts:[{url, source_in?, duration?}]} with ≤20 ordered parts. You get one audio_url, duration_seconds and segments[] ({index,start,duration_seconds}): the concat offsets to re-base video[].start against. To split, send payload {operation:"split", url, ranges:[{start, end?}]} with ≤20 ranges (omit end for end-of-file). Each segments[] entry has its own audio_url, start (its cut on the source clock) and duration_seconds. Ranges may overlap. The join is sample-domain: no re-TTS, no silence at the seams, no encoder gap. Parts must share one channel layout, else audio_parts_channel_mismatch. Optional payload.output {format}: "wav" is the default, pcm_s16le and sample-exact. "mp3" is smaller but re-adds priming padding at every edge. Keep wav when the file will be joined again or drives lip-sync. Use it when the merged/sliced file must be reusable: driving avatar_image_to_video_create with one file instead of two, or feeding several renders. If only one render needs the joined spine, use timeline_create audio.parts[] and skip this job. Limits: ≤20 parts/ranges, ≤1800s of produced audio, every URL on media.sume.com (mirror first). Provider/ffmpeg fields are rejected. Send idempotency_key and payload.operation. ≤$0.1024/job (compute + idle time at list). Optional: metadata, dry_run, max_spend_usd. Call timeline_audio, then jobs_wait, then jobs_result. Paid write: honour max_spend_usd/dry_run.

Shows one still and one video on screen at ONCE (Timeline 1.0 compose, POST /v1/timeline-1.0/compose). An async job returns one media.sume.com MP4, kind timeline_compose. Use it for 반배너 / half-banner (banner top, avatar video bottom) or a plate over a clip. NOT for image-then-video in sequence: that is two timeline_create video[] slots and no job. Layers are not interchangeable: image.url must probe as a still (compose_image_not_still) and video.url as video (compose_video_not_video). media.sume.com only. provider/ffmpeg fields are rejected. operation "stack" tiles two regions of one frame. "overlay" floats the still over full-frame video. This is operation, not mode: mode is the reserved sync/async/webhook option. Send idempotency_key, payload.operation, image.url and video.url. ≤$0.1024/job. Also takes layout, output, metadata, dry_run, max_spend_usd. Stack layout: split horizontal|vertical, image_region top|bottom|left|right (must match the split axis), ratio 0.1-0.9 as the still share, image_fit/video_fit cover|contain|stretch. Defaults horizontal/top/0.5 give 반배너. The video takes the exact remainder. Overlay layout: position top|center|bottom, width_ratio 0.05-1 of width (default 0.9, keeps aspect), margin_ratio 0-0.45 of height (default 0.05). Mixing stack and overlay keys is a 400. Duration ALWAYS comes from the video layer: video.duration, else the rest of the file from source_in. The held still never sets length. Past the source it clamps (compose_duration_clamped_to_source). Ceiling: 300s. Audio passes through from the video (a mute one gives compose_video_has_no_audio). Inside timeline_create, the spine wins. Set output width/height to the timeline you assemble into (e.g. 720x1280 live commerce), so the clip is not rescaled twice. Omit fps to keep the video rate. Another rate judders (output_fps_resamples_sources). Default: 1080x1920. Call timeline_compose, jobs_wait, jobs_result, then put video_url into timeline_create video[]. Paid write: honour max_spend_usd/dry_run.

Runs a pixel program on ONE Sume-hosted clip (Video filter 1.0, POST /v1/video-filter). An async job returns one NEW media.sume.com artf_ MP4, kind video_filter. Compiled server-side on the hosted encoder: the agent sandbox has no ffmpeg. Run the FREE check first: check_only:true returns diagnostics {code, field, next_action}, no job, no charge. Then encode with the same payload, check_only off and idempotency_key. Encode may still fail on the box as a structured job error. Fix, re-check, resubmit. The program is payload.ops[] and/or payload.filtergraph. ops run in order, ≤8. {op:"dim", amount} multiplies luma by amount in (0, 1]: 0.45 is darker, 1 unchanged, 0 or >1 rejected. {op:"crop", x, y, width, height} takes fractions of the source frame (x+width ≤ 1, y+height ≤ 1). Unknown op: unsupported_filter_op. Extra key: unsupported_filter_op_field. filtergraph is FILTERS ONLY, after ops, e.g. "eq=contrast=1.2,gblur=sigma=2" or "split[a][b];[a]gblur=sigma=4[c];[b][c]blend=all_mode=average". The server injects input and output: no [0:v], no output label, no paths, no -i, no spaces inside a filter. Only allowlisted pixel filters pass. movie, subtitles, drawtext and lut3d fail invalid_filtergraph, which lists the allowlist. Never pass -i, vf, filter_complex, ffmpeg, cmd, codec, preset or crf (400 ffmpeg_fields_rejected). Pixel filters (어둡게, crop, blur, colour) belong here. For assembly, use timeline_create. Still and video in one frame: timeline_compose. Send payload.video_url (media.sume.com, generated in this workspace, never external) and a program. Dead URL: source_not_found video_url. ≤$0.1024/encode, 300s source cap. Also takes metadata, dry_run, max_spend_usd. Returns video_url (new artf_), duration_seconds, ops_applied[], filtergraph, filters[]. filter_source_has_no_audio is soft. HDR or non-YUV fail closed (hdr_source_unsupported, unsupported_pixel_format).

Reads a Timeline 1.0 job result by `job_id` (GET /v1/jobs/:id/result): kind `timeline_render` with `video_url`, `duration_seconds`, `segment_count`, `billable_minutes` and structured warnings[]. Use it after `timeline_create` and `jobs_wait` to fetch the assembled MP4, not for status polling (job_status/jobs_wait) or to invent ffmpeg edits. Success returns one media.sume.com MP4 that matches the audio spine (±0.1s). These warnings[] codes are soft degradations, not hard failures, so proceed: transition_downgraded_to_cut, transition_snapped_to_frame, segment_source_short_padded, segment_source_short_looped, still_source_in_ignored, motion_ignored, timeline_gap_filled, video_coverage_shorter_than_audio, timeline_render_chunked, output_av_drift, output_fps_resamples_sources, soundtrack_shorter_than_spine, soundtrack_source_replaced, soundtrack_source_omitted, edge_fade_shortened. A 409 job_not_completed means the render has not finished yet. It is not a failure and not a verdict. Keep waiting with jobs_wait on the same job_id until the job's own status is terminal. Giving up after a 409 abandons an MP4 that is already billed and still rendering. VO spine starts stay authoritative: never pre-shift starts when you regenerate one short clip. Every input must already be Sume-hosted. Do not ask for ffmpeg commands, filtergraphs, codecs or a compose tool. A whole-clip pixel pass (dim) is `video_filter`, not a render option. It is read-only and returnsSensitiveUrl for the public artifact. Never echo temp paths or provider internals. Deliver `video_url`. If one segment warned, regenerate only that clip and re-run `timeline_create` with the same starts.