Open-Sora 1.1

自2024年3月份Colossal-AI团队发布Open-Sora 1.0，2024年4月又更新了Open-Sora 1.1，它支持 2s~15s、144p 到 720p、任意宽高比的文本到图像、文本到视频、图像到视频、视频到视频、无限时间生成，让我们来看看Open-Sora 1.1的实际视频生成效果。

文生视频

爱因斯坦穿越时空隧道

图生视频

让牛顿手中的苹果飞起来

案例体验

首先注册华为云账号并实名认证，然后进入Notebook页面点击Run in ModelArts一键运行：

🔹 本案例需使用 Pytorch-1.8 GPU-V100 及以上规格运行

🔹 点击Run in ModelArts，将会进入到ModelArts CodeLab中，这时需要你登录华为云账号，如果没有账号，则需要注册一个，且要进行实名认证，参考《ModelArts准备工作_简易版》 即可完成账号注册和实名认证。 登录之后，等待片刻，即可进入到CodeLab的运行环境

🔹 出现 Out Of Memory ，请检查是否为您的参数配置过高导致，修改参数配置，重启kernel或更换更高规格资源进行规避❗❗❗

1. 下载代码和模型

此处运行大约需要2分钟，请耐心等待（警告忽略即可）：

import os
import moxing as mox

if not os.path.exists('Open-Sora'):
    mox.file.copy_parallel('obs://modelbox-course/open-sora_1.1/Open-Sora', 'Open-Sora')
    
if not os.path.exists('/home/ma-user/.cache/huggingface'):
    mox.file.copy_parallel('obs://modelbox-course/open-sora_1.1/huggingface', '/home/ma-user/.cache/huggingface')
    
if not os.path.exists('Open-Sora/opus-mt-zh-en'):
    mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/opus-mt-zh-en', 'Open-Sora/opus-mt-zh-en')
    
if not os.path.exists('/home/ma-user/work/frpc_linux_amd64'):
    mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/frpc_linux_amd64', '/home/ma-user/work/frpc_linux_amd64')

INFO:root:Using MoXing-v2.1.6.879ab2f4-879ab2f4
INFO:root:Using OBS-Python-SDK-3.20.9.1
INFO:root:List OBS time cost: 0.68 seconds.
INFO:root:Copy parallel total time cost: 1.71 seconds.
INFO:root:List OBS time cost: 0.06 seconds.
INFO:root:Copy parallel total time cost: 110.68 seconds.
INFO:root:List OBS time cost: 0.66 seconds.
INFO:root:Copy parallel total time cost: 4.18 seconds.

2. 配置运行环境

按照操作指引一步一步往下运行...

3. 生成视频

使用huggingface镜像源下载数据：

import os

os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'

生成2s高宽比240:426大小的视频：

# text to video
!python scripts/inference.py configs/opensora-v1-1/inference/sample.py --prompt "A beautiful sunset over the city" --num-frames 32 --image-size 240 426

/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/colossalai/shardformer/layer/normalization.py:45: UserWarning: Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel
  warnings.warn("Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel")
Config (path: configs/opensora-v1-1/inference/sample.py): {'num_frames': 32, 'frame_interval': 3, 'fps': 24, 'image_size': [240, 426], 'multi_resolution': 'STDiT2', 'model': {'type': 'STDiT2-XL/2', 'from_pretrained': 'hpcai-tech/OpenSora-STDiT-v2-stage3', 'input_sq_size': 512, 'qk_norm': True, 'enable_flash_attn': True, 'enable_layernorm_kernel': True}, 'vae': {'type': 'VideoAutoencoderKL', 'from_pretrained': 'stabilityai/sd-vae-ft-ema', 'cache_dir': None, 'micro_batch_size': 4}, 'text_encoder': {'type': 't5', 'from_pretrained': 'DeepFloyd/t5-v1_1-xxl', 'cache_dir': None, 'model_max_length': 200}, 'scheduler': {'type': 'iddpm', 'num_sampling_steps': 100, 'cfg_scale': 7.0, 'cfg_channel': 3}, 'dtype': 'fp16', 'prompt_path': './assets/texts/t2v_samples.txt', 'prompt': ['A beautiful sunset over the city'], 'batch_size': 1, 'seed': 42, 'save_dir': './samples/samples/', 'config': 'configs/opensora-v1-1/inference/sample.py', 'prompt_as_path': False, 'reference_path': None, 'loop': 1, 'sample_name': None, 'num_sample': 1}
/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Loading checkpoint shards:   0%|                          | 0/2 [00:00<?, ?it/s]/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:44<00:00, 22.36s/it]
config.json: 762B [00:00, 4.32MB/s]                                             
model.safetensors: 100%|███████████████████| 3.07G/3.07G [03:21<00:00, 15.3MB/s]
100%|█████████████████████████████████████████| 100/100 [02:11<00:00,  1.32s/it]
Prompt: A beautiful sunset over the city
Saved to ./samples/samples/sample_0.mp4

生成的视频保存在Open-Sora/samples文件夹中，随机查看：

import os
import random
from moviepy.editor import *
from IPython.display import Image

# 视频存放目录
video_root = 'samples/samples'
# 列出所有文件
videos = os.listdir(video_root)
# 随机抽取视频
video = random.sample(videos, 1)[0]
# 视频输入路径
video_path = os.path.join(video_root, video)
# 加载原始视频
clip = VideoFileClip(video_path)
# 保存为GIF文件
clip.write_gif("output_animation.gif", fps=10)
# 显示生成结果
Image(open('output_animation.gif','rb').read())

MoviePy - Building file output_animation.gif with imageio.

4. Gradio界面

修改配置文件和Gradio应用...

运行Gradio应用，运行成功后点击 Running on public URL 后的网页链接即可体验！

!python gradio/app-ref.py

/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/transformers/models/marian/tokenization_marian.py:197: UserWarning: Recommended: pip install sacremoses.
  warnings.warn("Recommended: pip install sacremoses.")
/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/colossalai/shardformer/layer/normalization.py:45: UserWarning: Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel
  warnings.warn("Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel")
/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:45<00:00, 22.58s/it]
Running on local URL:  http://127.0.0.1:7860
Running on public URL: https://3e6fa2922141e47ef6.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
['A car on the ocean.']
100%|█████████████████████████████████████████| 100/100 [00:13<00:00,  7.38it/s]
Saved to ./outputs/output_1715846065.774836.png
['A car on the ocean.']
100%|█████████████████████████████████████████| 100/100 [01:14<00:00,  1.34it/s]
Saved to ./outputs/output_1715846157.404953.mp4