Update READMEs, add README_depth, minors

Signed-off-by: Micaela Verucchi <micaelaverucchi@gmail.com>
2022-03-30 15:40:25 +02:00
parent 40266a6c32
commit c690d537f1
4 changed files with 82 additions and 2 deletions
@@ -23,7 +23,10 @@ If you use tkDNN in your research, please cite the [following paper](https://iee
 - [x] Support 2D/3D Object Detection and Tracking [README](docs/README_2d3dtracking.md)
 #### 24 November 2021
 - [x] Support to sematic segmentation on cuda 11
- [x] Support to TensorRT8. 
+- [x] Support to TensorRT8 (tensort8 branch). 
+
+#### 30 March 2022
+- [x] Support to monocular depth esitmation (tensort8 branch) [README](docs/README_depth.md)

 TensorRT8 (and therefore Jetpack 4.6) is currently supported only on the branch tensort8 due to [performance issue with TensorRT8](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/tensorrt-8.html)). We will merge it to the master as soon as those issues are fixed (probably in future minor releases).

@@ -138,6 +141,7 @@ For specific details on how to export weights see [HERE](./docs/exporting_weight
 For specific details on how to run:
 - 2D object detection demos, details on FP16, INT8 and batching see [HERE](./docs/demo.md).
 - segmentation demos see [HERE](./docs/README_seg.md).
+- monocular depth estimation see [HERE](./docs/README_depth.md).
 - 2D/3D object detection and tracking demos see [HERE](./docs/README_2d3dtracking.md).
 - mAP demo to evaluate 2D object detectors see [HERE](./docs/mAP_demo.md).

@@ -185,6 +189,8 @@ For specific details on how to run tkDNN on Windows 10/11 see [HERE](./docs/wind
 | shelfnet_berkeley              | ShelfNet18_realtime<sup>11</sup>                           | [DeepDrive](https://bdd-data.berkeley.edu/)                          | 20        | 1024x1024       | [weights](https://cloud.hipert.unimore.it/s/m92e7QdD9gYMF7f/download)                                                                   |
 | dla34_cnet3d        | Centernet3D (DLA34 backend)<sup>4</sup>         | [KITTI 2017](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)                          | 1        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/2MDyWGzQsTKMjmR/download)     |
 | dla34_ctrack        | CenterTrack (DLA34 backend)<sup>12</sup>         | [NuScenes 3D](https://www.nuscenes.org/)                          | 7        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/rjNfgGL9FtAXLHp/download)     |
+| monodepth2        | Monodepth2 <sup>13</sup>         | [KITTI DEPTH](http://www.cvlibs.net/datasets/kitti/raw_data.php)                          | -        | 640x192       | [weights-mono](https://cloud.hipert.unimore.it/s/iYw9QwgP6CsqxLR/download)     |
+| monodepth2        | Monodepth2 <sup>13</sup>         | [KITTI DEPTH](http://www.cvlibs.net/datasets/kitti/raw_data.php)                          | -        | 640x192       | [weights-stereo](https://cloud.hipert.unimore.it/s/XmwbWNXDfqyQ4EL/download)     |


 ## References
@@ -201,3 +207,11 @@ For specific details on how to run tkDNN on Windows 10/11 see [HERE](./docs/wind
 10. Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. "Scaled-YOLOv4: Scaling Cross Stage Partial Network." arXiv preprint arXiv:2011.08036 (2020).
 11. Zhuang, Juntang, et al. "ShelfNet for fast semantic segmentation." Proceedings of the IEEE International Conference on Computer Vision Workshops. 2019.
 12. Zhou, Xingyi, Vladlen Koltun, and Philipp Krähenbühl. "Tracking objects as points." European Conference on Computer Vision. Springer, Cham, 2020.
+13. Godard, Clément, et al. "Digging into self-supervised monocular depth estimation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
+
+## Contributors
+The main contibutors, in chronological order, are: 
+- [Francesco Gatti](https://github.com/ceccocats), francesco.gatti@hipert.it
+- [Micaela Verucchi](https://github.com/mive93), micaela.verucchi@unimore.it
+- [Davide Sapienza](https://github.com/sapienzadavide), davide.sapienza@unimore.it
+- [Harshvardhan Chandirasekar](https://github.com/perseusdg), f20180523@goa.bits-pilani.ac.in
@@ -57,7 +57,7 @@ int main(int argc, char *argv[]) {
    if(save) {
        int w = depthNN.output_w;
        int h = depthNN.output_h;
-        resultVideo.open("result.mp4", cv::VideoWriter::fourcc('M','J','P','G'), 30, cv::Size(w, h));
+        resultVideo.open("result.mp4", cv::VideoWriter::fourcc('M','P','4','V'), 30, cv::Size(w, h));
    }

    if(show)
@@ -0,0 +1,54 @@
+# Monocular depth estimation with tkDNN
+
+Currently tkDNN supports only Monodepth2 as monocular depth esitmation network.
+
+
+## Run the demo
+
+To run the depth estimation demo follow these steps (example with monodepth2):
+```
+rm monodepth2_fp32.rt        # be sure to delete(or move) old tensorRT files
+./test_monodepth2            # run the yolo test (is slow)
+./demoDepth monodepth2_fp32.rt ../demo/yolo_test.mp4 
+```
+In general the demo program takes the following parameters:
+```
+./demoDepth <network-rt-file> <path-to-video> <show-flag> <save-flag>
+```
+where
+*  ```<network-rt-file>``` is the rt file generated by a test
+*  ```<<path-to-video>``` is the path to a video file or a camera input  
+*  ```<show-flag>``` if set to 0 the demo will not show the visualization, it will otherwise (default=1)
+*  ```<save-flag>``` if set to 1 the demo will save the video into result.mp4, it won't otherwise (default=1)
+
+NB) By default it is used FP32 inference
+
+
+![demo](https://user-images.githubusercontent.com/11939259/160845358-0d6ab15d-c5f4-46ae-b9da-bfaf3903389d.gif "Results on yolo_test.mp4")  
+
+
+<!-- ## FPS Results
+
+Inference FPS of shelfnet with tkDNN, average of 1200 images on:
+  * RTX 2080Ti (CUDA 10.2, TensorRT 7.0.0, Cudnn 7.6.5);
+  * Xavier AGX, Jetpack 4.3 (CUDA 10.0, CUDNN 7.6.3, tensorrt 6.0.1 );
+
+| Platform   | Test                     | Phase   | FP32, ms  | FP32, FPS | FP16, ms  |	FP16, FPS  | INT8, ms |	INT8, FPS | 
+| :------:   | :-----:                  | :-----: | :-----:   | :-----:   | :-----:   |	:-----:    | :-----:  |	:-----:   | 
+| RTX 2080Ti | shelfnet 1024x1024 (B=1) | pre     | 6.11863   |  163.435  |   5.81465 |  171.979   |  5.88699 |   169.866 |
+| RTX 2080Ti | shelfnet 1024x1024 (B=1) | inf     | 11.5464   |  86.6074  |   7.35396 |  135.981   |  6.37623 |   156.832 |
+| RTX 2080Ti | shelfnet 1024x1024 (B=1) | post    | 4.09058   |  244.464  |   3.91961 |  255.128   |  4.07343 |   245.493 |
+| RTX 2080Ti | shelfnet 1024x1024 (B=1) | tot     | 21.7556   |  45.9652  |   17.0882 |  58.5199   |  16.3366 |   61.2121 |
+| RTX 2080Ti | shelfnet 2048x2048 (B=4) | pre     | 25.435    |  39.3158  |   25.2953 |  39.5331   |  25.9303 |   38.565  | 
+| RTX 2080Ti | shelfnet 2048x2048 (B=4) | inf     | 36.5015   |  27.3961  |   17.0534 |  58.6395   |  15.6061 |   64.0773 |  
+| RTX 2080Ti | shelfnet 2048x2048 (B=4) | post    | 17.3917   |  57.4985  |   17.1649 |  58.2583   |  17.5539 |   56.9675 |  
+| RTX 2080Ti | shelfnet 2048x2048 (B=4) | tot     | 79.3283   |  12.6058  |   59.5136 |  16.8029   |  59.0903 |   16.9233 |  
+| AGX Xavier | shelfnet 1024x1024 (B=1) | pre     | 8.0174    |  124.729  |   7.5117  |  133.126   |  7.47333 |   133.809 |
+| AGX Xavier | shelfnet 1024x1024 (B=1) | inf     | 72.4173   |  13.8089  |   37.505  |  26.6631   |  31.3286 |   31.9197 |
+| AGX Xavier | shelfnet 1024x1024 (B=1) | post    | 8.89958   |  112.365  |   8.83576 |  113.176   |  9.42655 |   106.083 |
+| AGX Xavier | shelfnet 1024x1024 (B=1) | tot     | 89.3342   |  11.1939  |   53.8525 |  18.5692   |  48.2285 |   20.7346 |
+| AGX Xavier | shelfnet 2048x2048 (B=4) | pre     | 47.1454   |  21.211   |   21.6475 |  46.1947   |  21.4201 |   46.6851 | 
+| AGX Xavier | shelfnet 2048x2048 (B=4) | inf     | 266.537   |  3.75183  |   128.321 |  7.79293   |  107.621 |   9.29185 |  
+| AGX Xavier | shelfnet 2048x2048 (B=4) | post    | 44.0711   |  22.6906  |   40.1732 |  24.8922   |  39.873  |   25.0796 |  
+| AGX Xavier | shelfnet 2048x2048 (B=4) | tot     | 357.753   |  2.79522  |   190.142 |  5.25922   |  168.914 |   5.92016 |   -->
+
@@ -86,6 +86,18 @@ mkdir layer debug
 python export.py
 ```

+### 6)Export weights for monodepth2
+To get the weights needed to run Shelfnet tests use [this](https://github.com/perseusdg/monodepth2) fork of a Pytorch implementation of monodepth2 network. 
+
+```
+git clone https://github.com/perseusdg/monodepth2
+cd monodepth2
+mkdir models # Download the official weights and put depth.pth and encorder.pth inside this new folder
+conda env create --file monodepth.yaml
+conda activate monodepth2
+python exporter.py # you will find the weights inside the tkDNN_bin folder
+```
+
 ## Darknet Parser
 tkDNN implement and easy parser for darknet cfg files, a network can be converted with *tk::dnn::darknetParser*:
 ```