My take: a lot of people will immediately claim this is heresy, and threatens the very foundations of cinema as we’ve come to know it over the last one hundred years. And they would be right. And yet, time marches on. I believe some variation of this is the future of ultra-low budget production. Very soon the quality will surpass the shoddy CGI that many multi-million dollar Hollywood productions have been foisting on us lately.
Love it or hate it, as of August 2024, AI Video still has a long way to go.
In this video, AI Samson lays out the current AI Video Pipeline. Although there are a few fledgling story-building tools in development, full-featured “story mode” is not yet available in AI video generators. The current pipeline is:
Create the first and last frames of your clips
Animate the clips between these frames
Create audio and lip-sync the clips
Upscale the clips
Create music and SFX
Edit everything together offline.
It seems new platforms emerge weekly but AI Samson makes these recommendations:
My take: You know, the current pipeline makes me think of an animation pipeline. It’s eerily similar to the Machinima pipeline I used to create films in the sandbox mode of the video game The Movies over ten years ago:
You’ve seen the Sora samples. The Dream Machine videos. How does LTX Studio, touted as “the future of storytelling, transforming imagination into reality,” stand up?
“There are whole bunch of things it does not do, but I love where it’s going and where I hope it’s going to go…. It’s brilliant for keeping track of all of the shots that you really do need to keep track of. It’s brilliant for scene wide settings and project wide settings, something I’ve been craving, and it’s really, really good at that. It’s great for casting. It’s brilliant for allowing you to then kind of just drop those characters in. I love the generative tools that will allow you to erase bits that you don’t need in your starting shot and to add other bits that you need that will help you tidy up the shot…. My two big gripes and I don’t think these are bugs that they’re going to fix, this is just fundamental features that it needs to be in there. One of them is every shot is slow motion…. Secondly, breaking the fourth wall. It drives me out of my mind!”
Here’s a peek at actually using LTX Studio by Riley Brown:
My take: In addition to Haydn’s slo mo and fourth wall gripes, I would add these requirements as well: movement and expression control including blinking and lip-sync. Mid-2024, one has to use each of the many AI tools for what it does best and then bring all the bits together in post. As an early proponent of Machinima (using video games to make movies,) I’m watching this space with interest. My conclusion: advances are being made but we’re nowhere near lucid dreaming.
Record in ProRes Log on an iPhone 15 Pro at 4K 30. He says, “4K 24 especially at ProRes Log just looks kind of choppy.”
In Settings, change to “Most Compatible” from “High Efficiency” and lock the white balance.
Turn on Exposure Adjustment and set it to .7. He says, “If your highlights are blown out it’s going to be a lot harder to actually bring that detail back once you go into color grading.”
Use the main 1X lens under adequate lighting, avoiding top-down noon sunlight. Try angling the light on the opposite side of the subject. He says, “The lighting is probably the most important element actually in making those cameras look good.”
Use the Grid to help create interesting compositions and make sure your camera movement is motivated.
In the edit, convert the Log footage with a Color Space Transform into Rec 709 and colour grade as usual.
Use Halation to lend the footage the characteristic film highlights glow and use a plugin called RSMB to add motion blur.
Hosts Marina Hyde and Richard Osman “pull back the curtain on television, film, journalism and more using their years of knowledge, enviable contact book and wit to bring what’s hot and what’s not in the world of entertainment.”
The episode has them reacting to the first Sora announcement and AI in general. This exchange is particularly insightful:
RO: “…that entire middle of the industry which is where everyone I’ve, you know, worked with my whole life works, you know, it’s going to be really, really tough times I think and there’s no safety net put in place but it’s the middle of culture that will disappear and be replaced by AI I suspect and it’s coming soon and it’ll come sooner.”
MH: “As always the middle in everything is going in the way that mid budget films have fallen away and that kind of middle bankable quality stuff has gone and at the moment I cannot see a pathway to it coming back.”
Richard also says:
“If you’re a new creator in the industry, if you’re a new brain (and by the way it’s quite hard to get into the industry) and you have these tools at your disposal, you would just be using them. I remember a very, very young Edgar Wright when he was starting out and him and his friends would just make these little home movies on videos because that was a new technology that they could suddenly use. If you’re a young person now, a young writer, young creator, young director, a young actor, why would you not be using these AI tools to make incredible content and sharing it with people?”
My take: These comments echo what Jerry Seinfeld recently said about the movie business in general being over. I foresee a continued splintering of audiences into smaller and smaller niches, necessitating smaller budgets as profits also become smaller. Every once in a while something will break through into the general Big C Culture but for the most part that will be left to deep-pocketed streamers chasing smaller and smaller mass audiences. As Seth Godin says:
“Based in Toronto, shy kids are a multimedia production company who utilized Sora for their short film about a balloon man. ‘We now have the ability to expand on stories we once thought impossible,’ shares the trio made up of Walter Woodman, Sidney Leeder and Patrick Cederberg. Walter, who directed Air Head, remarks that ‘as great as Sora is at generating things that appear real, what excites us is its ability to make things that are totally surreal. A new era of abstract expressionism.’ Speaking to the wider industry, ‘people from all over the world with stories ready to burst out of their chests finally have the opportunity to show the world what’s inside.'”
The BTS video is quite illustrative, as it does look like some post work was done on at least one clip:
I love this quote: “Sora is very much a slot machine as to what you get back.”
By the way, these are the folks behind 2013’s brilliant, genre-launching Noah (nsfw):
My take: sure, it’s not prime time yet, but Sora will only get better. Check out these other “text to video” tools on Future Tools. Oh, and I think the contrast in “air head” is too flat.
He begins, “This morning, a really lovely NFS reader reached out to me and wanted my opinion on a program called ScriptReader.AI, which charges $10 for a complete screenplay breakdown.”
He uploads one of his scripts and in two hours gets extensive coverage. It appears the AI’s method is to rate each scene and then average the scores. His script gets a “Recommend.”
However, Jason is not impressed. He concludes:
“Computers cannot think and feel. They have zero sense of nuance. And you cannot take their coverage seriously. No matter how nice it is. AI coverage is worthless!”
My take: the ScriptReader.ai website woefully undersells its capabilities. It certainly looks sketchy. But you can test out three scenes for free.
And then there’s a link in the top right corner of the website labelled “SHALL WE PLAY A GAME?_” I recognized this as a reference to WarGames in which Matthew Broderick unwittingly provokes WWIII.
With some trepidation, I clicked on the link.
This brought me to a page titled “SceneProv_” with two modes, Fun and Training.
I decided to have some fun and generate a scene for a RomCom. The AI, suddenly my writing partner, set the scene and populated it with two characters. My interactions are bolded.
“Genre selected: romance
INT. COFFEE SHOP – DAY
JEN sits at a table with a book, sipping on her latte. She notices AIDEN walk in and their eyes meet.
JEN:
(smiling)
Hey!
AIDEN:
(nervously)
Hi! Can I sit here?
JEN:
(surprised)
Uh, sure!
AIDEN sits across from Jen and they strike up a conversation.
AIDEN: I noticed you’re reading “Do Androids Dream of Sheep?”.
JEN: Yup. It’s my favourite book.
Her eyes glitch.
AIDEN: Do you want to go back to my place?
JEN:
(surprised)
Excuse me?
AIDEN:
(smiling)
Not like that. I have a first edition copy of that book. I’d love to show it to you.
JEN:
(relieved)
Oh, sure. That sounds great.
INT. AIDEN’S PLACE – NIGHT
Aiden and Jen are sitting on the couch, looking at the first edition. Her eyes glitch again.
AIDEN Jen, why do your eyes “glitch?”
JEN What do you mean?
AIDEN Well, every once in a while they kinda spark.
Are you hiding something from me?
Jen looks at Aiden angrily and stands up quickly.
JEN:
(angrily)
No, I’m not hiding anything! It’s just a glitch in my synthetic retinas. I’m an android.
AIDEN:
(surprised)
An android? I had no idea.
JEN:
(sadly)
I know, it’s not exactly a desirable trait in the dating world.
AIDEN:
(smiling)
Well, I’m not like most guys. I think it’s pretty cool, to be honest.
Jen’s eyes glitch again, this time in surprise.
JEN:
(surprised)
Really?
AIDEN:
(nodding)
Really. Plus, I always thought I’d make a great cyborg myself.
Jen smiles and takes Aiden’s hand, feeling grateful for the first time to have found someone who accepts her for who she really is.
They continue to look at the book, their fingers touching.
Aiden turns to Jen, looking into her eyes.
AIDEN:
(softly)
You know, I may not be able to dream like you do, but
I have a feeling this is the beginning of something special.
JEN:
(smiling)
I have that feeling, too.
They lean in and share a kiss, the start of their unconventional love story.”
And then it rated “our” scene a B+. This AI passes the Turing Test for sure.
