PeterL1n /RobustVideoMatting
Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!
★ 9,607⑂ 1,195👁 133⑃ 24● 260▤ 1
+3
Daily
9,607
stars
1,195
forks
133
watchers
24
prs
260
issues
1
releases
README
Robust Video Matting (RVM)
!Teaser
English | 中文
Official repository for the paper Robust High-Resolution Video Matting with Temporal Guidance. RVM is specifically designed for robust human video matting. Unlike existing neural models that process frames as independent images, RVM uses a recurrent neural network to process videos with temporal memory. RVM can perform matting in real-time on any videos without additional inputs. It achieves 4K 76FPS and HD 104FPS on an Nvidia GTX 1080 Ti GPU. The project was developed at ByteDance Inc.
News
[Nov 03 2021] Fixed a bug in train.py.
[Sep 16 2021] Code is re-released under GPL-3.0 license.
[Aug 25 2021] Source code and pretrained models are published.
[Jul 27 2021] Paper is accepted by WACV 2022.
Showreel
Watch the showreel video (YouTube, Bilibili) to see the model's performance.
All footage in the video are available in Google Drive.
Demo
Webcam Demo: Run the model live in your browser. Visualize recurrent states.
Colab Demo: Test our model on your own videos with free GPU.
Download
We recommend MobileNetv3 models for most use cases. ResNet50 models are the larger variant with small performance improvements. Our model is available on various inference frameworks. See inference documentation for more instructions.
Framework
Download
Notes
PyTorch
rvm_mobilenetv3.pth
rvm_resnet50.pth
Official weights for PyTorch. Doc
TorchHub
Nothing to Download.
Easiest way to use our model in your PyTorch project. Doc
TorchScript
rvm_mobilenetv3_fp32.torchscript
rvm_mobilenetv3_fp16.torchscript
rvm_resnet50_fp32.torchscript
rvm_resnet50_fp16.torchscript
If inference on mobile, consider export int8 quantized models yourself. Doc
ONNX
rvm_mobilenetv3_fp32.onnx
rvm_mobilenetv3_fp16.onnx
rvm_resnet50_fp32.onnx
rvm_resnet50_fp16.onnx
Tested on ONNX Runtime with CPU and CUDA backends. Provided models use opset 12. Doc, Exporter.
TensorFlow
rvm_mobilenetv3_tf.zip
rvm_resnet50_tf.zip
TensorFlow 2 SavedModel. Doc
TensorFlow.js
rvm_mobilenetv3_tfjs_int8.zip
Run the model on the web. Demo, Starter Code
Top issues
#9如何将 rvm_mobilenetv3.pth 转换成 rvm_mobilenetv3.tflite14 comments · 2021-09-07#124请问,关于自己训练的模型,测试视频出来闪的比较厉害,有什么方法避免视频闪烁或者减小闪烁?12 comments · 2022-01-06#22手机移动端开销及其优化问题11 comments · 2021-09-18#40How to run it live?8 comments · 2021-09-25#20C++ sample code available?8 comments · 2021-09-17#16试了一下提供的demo,感觉效果达不到视频里面的啊8 comments · 2021-09-13#139how to enhance the model‘s performence when running on cpu ?7 comments · 2022-02-07#191Models license6 comments · 2022-08-04