From c690d537f1802aadbad34bafe1563712b16fe82b Mon Sep 17 00:00:00 2001 From: Micaela Verucchi Date: Wed, 30 Mar 2022 15:40:25 +0200 Subject: [PATCH] Update READMEs, add README_depth, minors Signed-off-by: Micaela Verucchi --- README.md | 16 +++++++++++- demo/demo/demoDepth.cpp | 2 +- docs/README_depth.md | 54 +++++++++++++++++++++++++++++++++++++++ docs/exporting_weights.md | 12 +++++++++ 4 files changed, 82 insertions(+), 2 deletions(-) create mode 100644 docs/README_depth.md diff --git a/README.md b/README.md index af7c9c5..fa00681 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,10 @@ If you use tkDNN in your research, please cite the [following paper](https://iee - [x] Support 2D/3D Object Detection and Tracking [README](docs/README_2d3dtracking.md) #### 24 November 2021 - [x] Support to sematic segmentation on cuda 11 -- [x] Support to TensorRT8. +- [x] Support to TensorRT8 (tensort8 branch). + +#### 30 March 2022 +- [x] Support to monocular depth esitmation (tensort8 branch) [README](docs/README_depth.md) TensorRT8 (and therefore Jetpack 4.6) is currently supported only on the branch tensort8 due to [performance issue with TensorRT8](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/tensorrt-8.html)). We will merge it to the master as soon as those issues are fixed (probably in future minor releases). @@ -138,6 +141,7 @@ For specific details on how to export weights see [HERE](./docs/exporting_weight For specific details on how to run: - 2D object detection demos, details on FP16, INT8 and batching see [HERE](./docs/demo.md). - segmentation demos see [HERE](./docs/README_seg.md). +- monocular depth estimation see [HERE](./docs/README_depth.md). - 2D/3D object detection and tracking demos see [HERE](./docs/README_2d3dtracking.md). - mAP demo to evaluate 2D object detectors see [HERE](./docs/mAP_demo.md). @@ -185,6 +189,8 @@ For specific details on how to run tkDNN on Windows 10/11 see [HERE](./docs/wind | shelfnet_berkeley | ShelfNet18_realtime11 | [DeepDrive](https://bdd-data.berkeley.edu/) | 20 | 1024x1024 | [weights](https://cloud.hipert.unimore.it/s/m92e7QdD9gYMF7f/download) | | dla34_cnet3d | Centernet3D (DLA34 backend)4 | [KITTI 2017](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) | 1 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/2MDyWGzQsTKMjmR/download) | | dla34_ctrack | CenterTrack (DLA34 backend)12 | [NuScenes 3D](https://www.nuscenes.org/) | 7 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/rjNfgGL9FtAXLHp/download) | +| monodepth2 | Monodepth2 13 | [KITTI DEPTH](http://www.cvlibs.net/datasets/kitti/raw_data.php) | - | 640x192 | [weights-mono](https://cloud.hipert.unimore.it/s/iYw9QwgP6CsqxLR/download) | +| monodepth2 | Monodepth2 13 | [KITTI DEPTH](http://www.cvlibs.net/datasets/kitti/raw_data.php) | - | 640x192 | [weights-stereo](https://cloud.hipert.unimore.it/s/XmwbWNXDfqyQ4EL/download) | ## References @@ -201,3 +207,11 @@ For specific details on how to run tkDNN on Windows 10/11 see [HERE](./docs/wind 10. Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. "Scaled-YOLOv4: Scaling Cross Stage Partial Network." arXiv preprint arXiv:2011.08036 (2020). 11. Zhuang, Juntang, et al. "ShelfNet for fast semantic segmentation." Proceedings of the IEEE International Conference on Computer Vision Workshops. 2019. 12. Zhou, Xingyi, Vladlen Koltun, and Philipp Krähenbühl. "Tracking objects as points." European Conference on Computer Vision. Springer, Cham, 2020. +13. Godard, Clément, et al. "Digging into self-supervised monocular depth estimation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019. + +## Contributors +The main contibutors, in chronological order, are: +- [Francesco Gatti](https://github.com/ceccocats), francesco.gatti@hipert.it +- [Micaela Verucchi](https://github.com/mive93), micaela.verucchi@unimore.it +- [Davide Sapienza](https://github.com/sapienzadavide), davide.sapienza@unimore.it +- [Harshvardhan Chandirasekar](https://github.com/perseusdg), f20180523@goa.bits-pilani.ac.in diff --git a/demo/demo/demoDepth.cpp b/demo/demo/demoDepth.cpp index 41f5dbf..6c7de82 100644 --- a/demo/demo/demoDepth.cpp +++ b/demo/demo/demoDepth.cpp @@ -57,7 +57,7 @@ int main(int argc, char *argv[]) { if(save) { int w = depthNN.output_w; int h = depthNN.output_h; - resultVideo.open("result.mp4", cv::VideoWriter::fourcc('M','J','P','G'), 30, cv::Size(w, h)); + resultVideo.open("result.mp4", cv::VideoWriter::fourcc('M','P','4','V'), 30, cv::Size(w, h)); } if(show) diff --git a/docs/README_depth.md b/docs/README_depth.md new file mode 100644 index 0000000..ac87bb6 --- /dev/null +++ b/docs/README_depth.md @@ -0,0 +1,54 @@ +# Monocular depth estimation with tkDNN + +Currently tkDNN supports only Monodepth2 as monocular depth esitmation network. + + +## Run the demo + +To run the depth estimation demo follow these steps (example with monodepth2): +``` +rm monodepth2_fp32.rt # be sure to delete(or move) old tensorRT files +./test_monodepth2 # run the yolo test (is slow) +./demoDepth monodepth2_fp32.rt ../demo/yolo_test.mp4 +``` +In general the demo program takes the following parameters: +``` +./demoDepth +``` +where +* `````` is the rt file generated by a test +* ```<``` is the path to a video file or a camera input +* `````` if set to 0 the demo will not show the visualization, it will otherwise (default=1) +* `````` if set to 1 the demo will save the video into result.mp4, it won't otherwise (default=1) + +NB) By default it is used FP32 inference + + +![demo](https://user-images.githubusercontent.com/11939259/160845358-0d6ab15d-c5f4-46ae-b9da-bfaf3903389d.gif "Results on yolo_test.mp4") + + + + diff --git a/docs/exporting_weights.md b/docs/exporting_weights.md index 811431d..b4cb366 100644 --- a/docs/exporting_weights.md +++ b/docs/exporting_weights.md @@ -86,6 +86,18 @@ mkdir layer debug python export.py ``` +### 6)Export weights for monodepth2 +To get the weights needed to run Shelfnet tests use [this](https://github.com/perseusdg/monodepth2) fork of a Pytorch implementation of monodepth2 network. + +``` +git clone https://github.com/perseusdg/monodepth2 +cd monodepth2 +mkdir models # Download the official weights and put depth.pth and encorder.pth inside this new folder +conda env create --file monodepth.yaml +conda activate monodepth2 +python exporter.py # you will find the weights inside the tkDNN_bin folder +``` + ## Darknet Parser tkDNN implement and easy parser for darknet cfg files, a network can be converted with *tk::dnn::darknetParser*: ```