The ChatGPT maker said it is blocking particularly damaging forms of abuse, such as child sexual abuse materials and sexual deepfakes.
OpenAI is releasing its AI-creating video tool to millions of users who pay for ChatGPT, marking the next wave of artificial intelligence.
The company on Monday released the latest version of its Sora video generator, Sora Turbo, which will give people the ability to create 20-second video content that looks photorealistic and adds to fears of more deepfake videos.
OpenAI said the product would not be available in the European Economic Area, Switzerland, or the United Kingdom yet, possibly due to regulations, which has been the case for OpenAI’s other products, which reached the European market at a later date.
“We're going to try our hardest to be able to launch there,” OpenAI CEO Sam Altman said on Monday.
The company also wants to prevent the illegal use of the tool, "but we also want to balance that with creative expression," he added.
OpenAI said that it has implemented some safeguards, which include visible watermarks and verification metadata. There will be limits in the beginning on uploading images of people, as the company said it refines its deepfake mitigations.
The company said in a blog post that it was blocking particularly damaging forms of abuse, such as child sexual abuse materials and sexual deepfakes.
Moderation of user content will start “a little conservative," but if it "doesn't quite get it right, just give us that feedback," Altman said.
OpenAI also said Sora still has limitations. "It often generates unrealistic physics and struggles with complex actions over long durations," the company said.
OpenAI first presented Sora in February, but it was only accessible to select film-makers, artists, and safety testers
OpenAI is not the only company to work on AI video tools. Meta is developing Movie Gen and Google has released its Veo video generator. However, both are not available publicly.