Turning music videos into social cutdowns

For several years, a tool I built has been processing every official music video released by a major independent record label. It takes a finished master, finds the musical sections and prepares shorter clips, including vertical versions that follow the performer.

The master is only the start of a release campaign. Short-form platforms need their own cutdowns, and preparing them means repeatedly finding sections of the song, cleaning up the first and last shots, and adjusting the framing. Those decisions are connected: a chorus might start halfway through a camera shot, and the performer might move straight out of a fixed vertical crop.

I built the tool to take that repeated preparation out of the editing timeline. It runs on RunPod, a cloud platform for GPU workloads, and combines music analysis with computer vision. The result is a set of clips to review and choose from, rather than a master that still needs every cutdown assembled by hand.

The section diagram in the overview illustrates the musical analysis. The footage, face boxes and crop positions come from actual tool outputs, with annotations redrawn to make them readable here.

Finding a section worth cutting

The first pass listens to the soundtrack. Demucs separates the audio into stems, and allin1, a music-analysis model, identifies sections such as intros, verses, choruses and bridges. Those timestamps give the tool a musically informed starting point instead of dividing the video into arbitrary chunks.

Music and picture do not always change together, though. Cutting at a musical boundary can leave a brief flash of the next camera angle at the end of a clip. TransNetV2, a shot-boundary detector, maps the visual cuts inside each section. The tool then checks the edge shots: an opening fragment shorter than half a second or a closing fragment shorter than a second and a half can be removed. Audio fades soften the adjusted start and end.

Choosing where the vertical frame should sit

Masters arrive in different aspect ratios, so the tool reads the actual frame dimensions before calculating its 9:16 crop. A YOLO face detector locates faces within each shot. The selection logic favours a larger, centrally placed face initially, then proximity to the previous detection to help maintain continuity.

Detection alone does not make good framing. Moving the crop to every new face coordinate would turn small head movements and detection noise into camera shake. The reframer analyses the shot, smooths the detected positions and decides whether to hold a stable crop or follow the movement. If face detections are inconsistent, it falls back to a centre crop.

The comparison below makes the difference visible. Both vertical panels use the same source frames: one is cropped down the middle, while the other is the tool’s tracked output.

Across these cuts, the performer moves from one side of the composition to the other. A fixed crop pushes their face towards the edge or partly out of view; the tracked crop repositions to keep them framed.

The next example shows continuous movement rather than a change of camera angle. Across a single 52-second take, the crop travels over half the width of the master. It is shown at 4x speed, with the fixed crop alongside it so the effect is easy to see.

Making it useful beyond a demo

A moving face box is only one part of the job. The tool exports the original musical sections, centre-cropped and tracked vertical alternatives, shot-trimmed clips at the source aspect ratio, and debug videos. Keeping alternatives matters: the best framing still depends on the composition, and a person can choose the version that works for the campaign.

The debug output also makes the system maintainable. When a crop looks wrong, I can inspect the detections and the chosen window alongside the clean result. Getting the models to run reliably together on RunPod took its own work, including dependency fixes and startup checks for GPU compatibility. A successful container build was not enough; the analysis had to behave correctly on the worker actually running the job.

That is where the tool earns its place in the release workflow. Each new master arrives with its own edit, aspect ratio and movement. The pipeline handles the repeated analysis and preparation, leaving a set of usable starting points for the next creative decision.