How we built an AI reframe that doesn't cut the action

Every AI clipping tool has to solve the same unglamorous problem: your source is 16:9, but TikTok, Reels and Shorts are 9:16. Going from one to the other means throwing away roughly two-thirds of the width. Get that wrong and the whole clip feels broken, no matter how good the caption or the hook is.

This is the part almost nobody talks about, and it is the part that decides whether a clip looks made by a human or made by a script.

The two easy answers, and why both fail

There are two obvious ways to fit 16:9 into 9:16, and on their own, both are bad:

• Center-crop. Take the middle slice and fill the screen. It looks punchy on a close-up, but on a wide shot it is a disaster. On a football clip the ball swings out to the wing and the crop simply does not show it. You are watching a reaction to an action you cannot see.
• Letterbox. Shrink the whole 16:9 frame and add bars top and bottom. Nothing is lost, but everything is tiny. Do it for a whole clip and it feels cheap and small, like a video of a video.

The real mistake is treating reframing as one global decision. The right framing is not a property of the video. It changes from moment to moment.

Frame each moment on its own terms

A tight close-up wants to fill the screen. A wide shot wants to stay wide enough that you do not lose the action on the edges. A full-screen graphic wants to be shown whole. The right answer is different for each, and it can change every couple of seconds as the footage cuts. So instead of one setting for the whole clip, we choose the framing moment by moment, and hold it steady so it never jitters or drifts.

Talking content adds another layer. One person, two people, or a full panel each want a different frame, and picking the right one automatically, without cropping someone out or squashing everyone into a tiny strip, is most of the work.

The honest part

None of this is one clever model. It is a lot of computer vision and audio, feeding decisions that we tune against real footage, over and over, watching where it looks wrong and fixing it. Most of the progress comes from testing on the videos that break it: a fast football match, a multi-host podcast, a screen recording with no faces at all.

Reframe quality is not a feature you finish. For a clipping tool it is the whole game, so we keep grinding it. If you want to see where it is today, drop a video into Katto and watch how it frames each moment.