Stanford and Adobe Researchers Develop a Video Editor AI

Researchers from Stanford University and Adobe have developed an artificial intelligence that can edit in seconds dialogue-driven scenes of live-action video.

Today’s filmmakers have the liberty to record many versions or takes of a scene thanks in part to digital cameras. With the current frame-based video editing tools, editing these numerous takes is a laborious process, turning creative exploration of different editing styles challenging. Film editors have to review each take, manually piece together each take into clips and then organize these clips on a timeline to tell a story.

In the paper entitled “Computational Video Editing for Dialogue-Driven Scenes”, Stanford and Adobe researchers said the video editing AI that they have developed efficiently edits video of dialogue-driven scenes. For the new video editing system to work, a standard film script and multiple video takes are inputted into the system. The system then picks the most appropriate clip from one of the input takes for each line of dialogue based on a user-specified set of film-editing “idioms”.

An example of a film-editing idiom or preset is starting a clip with a wide angle, ensuring that all the characters in the scene are in the frame, and safeguarding that the speaker of each line of dialogue is visible. Another film-editing preset ensures that the character speaking emotional lines of dialogue has close up shots.

“Our computational video editing system significantly speeds up the editing process by letting editors specify the set of film-editing idioms they would like to enforce and then automatically generating the corresponding edit in a few seconds,” Stanford and Abode researchers said. “This lets editors iteratively adjust the edit by trying different combinations of idioms and testing different parameters to quickly see how their scene would appear using a variety of editing styles.”