【论文笔记】Eidetic 3D LSTM: A Model for Video Prediction and Beyond

Eidetic 3D LSTM: A Model for Video Prediction and Beyond (ICLR 2019)

Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-Fei

代码：https://github.com/google/e3d_lstm

问题：时空预测学习 Spatiotemporal Predictive Learning

如何为其它视频理解和推理任务学习有效的特征表示？

视频预测：输入历史视频帧，预测未来视频帧

• 无监督/自监督：不需要人工标注 → 特征学习

• 未来：确定性（物理规律等），随机性和不确定性

• 两类任务：单帧预测，多帧预测

类似任务

• 视频重构 Reconstruction

• 视频插帧 Interpolation

两类视频建模方法：

• 循环神经网络：在视频预测上效果优异

• 三维卷积神经网络：在动作分类上更胜一筹
  

RNN结合3D卷积

E3D-LSTM序列建模

• 短期信息：3D卷积

• 长期信息：基于自注意力的“回忆”机制

多任务学习：

• 像素级任务：视频预测

• 视频级任务：动作分类

ICLR 2019 Meta-Review

系列工作

Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics

Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, Philip S. Yu

CVPR, 2019

Eidetic 3D LSTM: A Model for Video Prediction and Beyond

Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-Fei

ICLR, 2019

PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, Philip S. Yu

ICML, 2018 

PredCNN: Predictive Learning with Cascade Convolutions

{Ziru Xu, Yunbo Wang}, Mingsheng Long, Jianmin Wang

IJCAI, 2018 

PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs

Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, Philip S. Yu

NIPS, 2017 

Spatiotemporal Pyramid Network for Video Action Recognition

Yunbo Wang, Mingsheng Long, Jianmin Wang, Philip S. Yu

CVPR, 2017