Update READMEs

Signed-off-by: Micaela Verucchi <micaelaverucchi@gmail.com>
2021-07-23 10:31:34 +02:00
parent 0292e21c33
commit 4f9f27152a
6 changed files with 91 additions and 149 deletions
@@ -19,6 +19,7 @@ If you use tkDNN in your research, please cite the [following paper](https://iee
 ### What's new (20 July 2021)
 - [x] Support to sematic segmentation [README](docs/README_seg.md)
 - [x] Support 2D/3D Object Detection and Tracking [README](docs/README_2d3dtracking.md)
 - [ ] Support to TensorRT8 (WIP)
 ## FPS Results
@@ -70,15 +71,13 @@ Results for COCO val 2017 (5k images), on RTX 2080Ti, with conf threshold=0.001
 - [tkDNN](#tkdnn)
  - [Index](#index)
  - [Dependencies](#dependencies)
  - [About OpenCV](#about-opencv)
  - [How to compile this repo](#how-to-compile-this-repo)
  - [Workflow](#workflow)
-  - [How to export weights](#how-to-export-weights)
+  - [Exporting weights](#exporting-weights)
-  - [Run the demo](#run-the-demo)
+  - [Run the demos](#run-the-demos)
-  - [mAP demo](#map-demo)
+  - [tkDNN on Windows 10 (experimental)](#tkdnn-on-windows-10-experimental)
  - [Existing tests and supported networks](#existing-tests-and-supported-networks)
  - [References](#references)
  - [tkDNN on Windows 10 (experimental)](#tkdnn-on-windows-10-experimental)
 ## Dependencies
@@ -97,7 +96,7 @@ sudo apt install libyaml-cpp-dev curl libeigen3-dev
 ```
-## About OpenCV
+#### About OpenCV
 To compile and install OpenCV4 with contrib us the script ```install_OpenCV4.sh```. It will download and compile OpenCV in Download folder.
 ```
 bash scripts/install_OpenCV4.sh
@@ -125,15 +124,21 @@ Steps needed to do inference on tkDNN with a custom neural network.
 ## Exporting weights
-For specific details on how to export weights see [here](./docs/exporting_weights.md)
+For specific details on how to export weights see [HERE](./docs/exporting_weights.md).
-## Run the demo 
+## Run the demos 
-For specific details on how to run the demos see [here](./docs/demo.md)
+For specific details on how to run:
 - 2D object detection demos, details on FP16, INT8 and batching see [HERE](./docs/demo.md).
 - segmentation demos see [HERE](./docs/README_seg.md).
 - 2D/3D object detection and tracking demos see [HERE](./docs/README_2d3dtracking.md).
 - mAP demo to evaluate 2D object detectors see [HERE](./docs/mAP_demo.md).
-## mAP demo
+![demo](https://user-images.githubusercontent.com/11562617/72547657-540e7800-388d-11ea-83c6-49dfea2a0607.gif)
-For specific details on how to run the mAP demo see [here](./docs/mAP_demo.md)
+## tkDNN on Windows 10 (experimental)
 For specific details on how to run tkDNN on Windows 10 see [HERE](./docs/windows.md).
 ## Existing tests and supported networks
@@ -160,14 +165,20 @@ For specific details on how to run the mAP demo see [here](./docs/mAP_demo.md)
 | resnet101_cnet    | Centernet (Resnet101 backend)<sup>4</sup>     | [COCO 2017](http://cocodataset.org/)                          | 80        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/5BTjHMWBcJk8g3i/download)     |
 | csresnext50-panet-spp    | Cross Stage Partial Network <sup>7</sup>     | [COCO 2014](http://cocodataset.org/)                          | 80        | 416x416       | [weights](https://cloud.hipert.unimore.it/s/Kcs4xBozwY4wFx8/download)     |
 | yolo4             | Yolov4 <sup>8</sup>                           | [COCO 2017](http://cocodataset.org/)                          | 80        | 416x416       | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download)     |
 | yolo4_320         | Yolov4 <sup>8</sup>                           | [COCO 2017](http://cocodataset.org/)                          | 80        | 320x320       | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download)     |
 | yolo4_512         | Yolov4 <sup>8</sup>                           | [COCO 2017](http://cocodataset.org/)                          | 80        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download)     |
 | yolo4_608         | Yolov4 <sup>8</sup>                           | [COCO 2017](http://cocodataset.org/)                          | 80        | 608x608       | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download)     |
 | yolo4_berkeley             | Yolov4 <sup>8</sup>                           | [BDD100K  ](https://bair.berkeley.edu/blog/2018/05/30/bdd/)                          | 10        | 540x320       | [weights](https://cloud.hipert.unimore.it/s/nkWFa5fgb4NTdnB/download)     |
 | yolo4tiny             | Yolov4 tiny <sup>9</sup>                           | [COCO 2017](http://cocodataset.org/)                          | 80        | 416x416       | [weights](https://cloud.hipert.unimore.it/s/iRnc4pSqmx78gJs/download)     |
-| yolo4x             | Yolov4x-mish  <sup>9</sup>                          | [COCO 2017](http://cocodataset.org/)                          | 80        | 640x640       | [weights](https://cloud.hipert.unimore.it/s/5MFjtNtgbDGdJEo/download)     |
+| yolo4x             | Yolov4x-mish  <sup>9</sup>                          | [COCO 2017](http://cocodataset.org/)                          | 
 | yolo4tiny_512           | Yolov4 tiny <sup>9</sup>                           | [COCO 2017](http://cocodataset.org/)                          | 80        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/iRnc4pSqmx78gJs/download)     |
 80        | 640x640       | [weights](https://cloud.hipert.unimore.it/s/5MFjtNtgbDGdJEo/download)     |
 | yolo4x-cps            | Scaled Yolov4 <sup>10</sup>                          | [COCO 2017](http://cocodataset.org/)                          | 80        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/AfzHE4BfTeEm2gH/download)     |
 | shelfnet              | ShelfNet18_realtime<sup>11</sup>                      | [Cityscapes](https://www.cityscapes-dataset.com/)                          | 19        | 1024x1024       | [weights](https://cloud.hipert.unimore.it/s/mEDZMRJaGCFWSJF/download)                                                                   |
 | shelfnet_berkeley              | ShelfNet18_realtime<sup>11</sup>                           | [DeepDrive](https://bdd-data.berkeley.edu/)                          | 20        | 1024x1024       | [weights](https://cloud.hipert.unimore.it/s/m92e7QdD9gYMF7f/download)                                                                   |
 | dla34_cnet3d        | Centernet3D (DLA34 backend)<sup>4</sup>         | [KITTI 2017](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d)                          | 1        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/2MDyWGzQsTKMjmR/download)     |
 | dla34_ctrack        | CenterTrack (DLA34 backend)<sup>12</sup>         | [NuScenes 3D](https://www.nuscenes.org/)                          | 7        | 512x512       | [weights](https://cloud.hipert.unimore.it/s/rjNfgGL9FtAXLHp/download)     |
 ### tkDNN on Windows 10 (experimental)
 For specific details on how to run the demos on Windows 10 see [here](./docs/windows.md)
 ## References
@@ -181,3 +192,5 @@ For specific details on how to run the demos on Windows 10 see [here](./docs/win
 8. Bochkovskiy, Alexey, Chien-Yao Wang, and Hong-Yuan Mark Liao. "YOLOv4: Optimal Speed and Accuracy of Object Detection." arXiv preprint arXiv:2004.10934 (2020).
 9. Bochkovskiy, Alexey, "Yolo v4, v3 and v2 for Windows and Linux" (https://github.com/AlexeyAB/darknet)
 10. Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. "Scaled-YOLOv4: Scaling Cross Stage Partial Network." arXiv preprint arXiv:2011.08036 (2020).
 11. Zhuang, Juntang, et al. "ShelfNet for fast semantic segmentation." Proceedings of the IEEE International Conference on Computer Vision Workshops. 2019.
 12. Zhou, Xingyi, Vladlen Koltun, and Philipp Krähenbühl. "Tracking objects as points." European Conference on Computer Vision. Springer, Cham, 2020.
@@ -1,6 +1,42 @@
 # 2D/3D Object Detection and Tracking
-Currently tkDNN supports only CenterTrack as OD & Tracker network.
+Currently tkDNN supports only CenterTrack as 3DOD & 2D/3D Tracker network.
 ## 3D Object Detection
 To run the 3D object detection demo follow these steps (example with CenterNet based on DLA34):
 ```
 rm dla34_cnet3d_fp32.rt        # be sure to delete(or move) old tensorRT files
 ./test_dla34_cnet3d            # run the yolo test (is slow)
 ./demo3D dla34_cnet3d_fp32.rt ../demo/yolo_test.mp4 NULL c
 ```
 The demo3D program takes the same parameters of the demo program:
 ```
 ./demo3D <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh>
 ```
 where
 * ```<calibration-file>``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
 ## Object Detection and Tracking
 To run the 3D object detection & tracking demo follow these steps (example with CenterTrack based on DLA34):
 ```
 rm dla34_ctrack_fp32.rt  # be sure to delete(or move) old tensorRT files
 ./test_dla34_ctrack      # run the yolo test (is slow)
 ./demoTracker dla34_ctrack_fp32.rt ../demo/yolo_test.mp4 NULL c
 ```
 The demoTracker program takes the same parameters of the demo program:
 ```
 ./demoTracker <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh> <2D/3D-flag>
 ```
 where
 * ```<calibration-file>``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
 *  ```<2D/3D-flag>``` if set to 0 the demo will be in the 2D mode, while if set to 1 the demo will be in the 3D mode (Default is 1 - 3D mode).
 ## FPS Results
@@ -2,19 +2,6 @@
 Currently tkDNN supports only ShelfNet as semantic segmentation network.
 ## Export weights from Shelfnet
 To get the weights needed to run Shelfnet tests use [this](https://git.hipert.unimore.it/mverucchi/shelfnet) fork of a Pytorch implementation of Shelfnet network. 
 ```
 git clone https://git.hipert.unimore.it/mverucchi/shelfnet
 cd shelfnet 
 cd ShelfNet18_realtime
 conda env create --file shelfnet_env.yml
 conda activate shelfnet
 mkdir layer debug
 python export.py
 ```
 ## Run the demo
@@ -46,15 +33,7 @@ NB) The batching is not used to work on more streams, rather to work on more til
 For other demo videos refer to [this playlist](https://www.youtube.com/playlist?list=PLv0nEQYDD45y5EdSiywwCGPBmJVUzIWwe).
-
+NB) The gif and the videos are obtained with Mapillary Vistas weights, that we cannot publicly share due to its license restrictions. However, you can train Shelfnet using Mapillary and [this](https://git.hipert.unimore.it/mverucchi/shelfnet) fork of the original repo.
 ## Existing tests and supported networks
 | Test Name         | Network                                       | Dataset                                                       | N Classes | Input size    | Weights                                                                   |
 | :---------------- | :-------------------------------------------- | :-----------------------------------------------------------: | :-------: | :-----------: | :------------------------------------------------------------------------ |
 | shelfnet              | ShelfNet18_realtime<sup>1</sup>                           | [Cityscapes](https://www.cityscapes-dataset.com/)                          | 19        | 1024x1024       | [weights](https://cloud.hipert.unimore.it/s/mEDZMRJaGCFWSJF/download)                                                                   |
 | shelfnet_berkeley              | ShelfNet18_realtime<sup>1</sup>                           | [DeepDrive](https://bdd-data.berkeley.edu/)                          | 20        | 1024x1024       | [weights](https://cloud.hipert.unimore.it/s/m92e7QdD9gYMF7f/download)                                                                   |
 1. Zhuang, Juntang, et al. "ShelfNet for fast semantic segmentation." Proceedings of the IEEE International Conference on Computer Vision Workshops. 2019.
 ## FPS Results
@@ -1,8 +1,6 @@
-# tkDNN Demo
+# 2D Object Detection with tkDNN
-## Supported Network
+## Supported Networks
 2D Object Detection:
 * Yolo4, Yolo4-csp, Yolo4x, Yolo4_berkeley, Yolo4tiny
 * Yolo3, Yolo3_berkeley, Yolo3_coco4, Yolo3_flir, Yolo3_512, Yolo3tiny, Yolo3tiny_512
@@ -11,28 +9,12 @@
 * Resnet101_cnet, Dla34_cnet
 * Mobilenetv2ssd, Mobilenetv2ssd512, Bdd-mobilenetv2ssd
 3D Object Detection:
 * Dla34_cnet3d
 2D/3D Object Detection and Tracking:
 * Dla34_cnet3d_track
 ## Index
- - [Run the demo](#run-the-demo)
+ - [2D Object Detection](#2d-object-detection)
-    - [2D Object Detection](#2d-object-detection)
+ - [FP16 inference](#fp16-inference)
-    - [3D Object Detection](#3d-object-detection)
+ - [INT8 inference](#int8-inference)
-    - [Object Detection and Tracking](#object-detection-and-tracking)
+ - [Batching](#batching)
    - [FP16 inference](#fp16-inference)
    - [INT8 inference](#int8-inference)
    - [Batching](#batching)
    - [Run the demo on Windows](#run-the-demo-on-windows)
 ## Run the demo 
 N.b. By default it is used FP32 inference
 ### 2D Object Detection
 This is an example using yolov4.
@@ -66,45 +48,11 @@ where
 *  ```<show-flag>``` if set to 0 the demo will not show the visualization but save the video into result.mp4 (if n-batches ==1)
 *  ```<conf-thresh>``` confidence threshold for the detector. Only bounding boxes with threshold greater than conf-thresh will be displayed.
-
+N.B. By default it is used FP32 inference
 ![demo](https://user-images.githubusercontent.com/11562617/72547657-540e7800-388d-11ea-83c6-49dfea2a0607.gif)
 ### 3D Object Detection
 To run the 3D object detection demo follow these steps (example with CenterNet based on DLA34):
 ```
 rm dla34_cnet3d_fp32.rt        # be sure to delete(or move) old tensorRT files
 ./test_dla34_cnet3d            # run the yolo test (is slow)
 ./demo3D dla34_cnet3d_fp32.rt ../demo/yolo_test.mp4 NULL c
 ```
 The demo3D program takes the same parameters of the demo program:
 ```
 ./demo3D <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh>
 ```
 where
 * ```calibration-file``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
 ### Object Detection and Tracking
 To run the 3D object detection & tracking demo follow these steps (example with CenterTrack based on DLA34):
 ```
 rm dla34_ctrack_fp32.rt  # be sure to delete(or move) old tensorRT files
 ./test_dla34_ctrack      # run the yolo test (is slow)
 ./demoTracker dla34_ctrack_fp32.rt ../demo/yolo_test.mp4 NULL c
 ```
 The demoTracker program takes the same parameters of the demo program:
 ```
 ./demoTracker <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh> <2D/3D-flag>
 ```
 where
 * ```calibration-file``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
 *  ```<2D/3D-flag>``` if set to 0 the demo will be in the 2D mode, while if set to 1 the demo will be in the 3D mode (Default is 1 - 3D mode).
 ### FP16 inference
@@ -115,7 +63,7 @@ rm yolo3_fp16.rt        # be sure to delete(or move) old tensorRT files
 ./test_yolo3            # run the yolo test (is slow)
 ./demo yolo3_fp16.rt ../demo/yolo_test.mp4 y
 ```
-N.b. Using FP16 inference will lead to some errors in the results (first or second decimal). 
+N.B. Using FP16 inference will lead to some errors in the results (first or second decimal). 
 ### INT8 inference
@@ -169,49 +117,3 @@ rm yolo3_fp32.rt                   # be sure to delete(or move) old tensorRT fil
 ./test_yolo3                       # build RT file
 ./test_rtinference yolo3_fp32.rt 4 # test with a batch size of 4
 ```
 ### Run the demo on Windows 
 This example uses yolo4_tiny.\
 To run the object detection file create .rt file bu running:
 ```
 .\test_yolo4tiny.exe
 ```
 Once the rt file has been successfully create,run the demo using the following command:
 ```
 .\demo.exe yolo4tiny_fp32.rt ..\demo\yolo_test.mp4 y 
 ```
 For general info on more demo paramters,check Run the demo section on top 
 To run the test_all_tests.sh on windows,use git bash or msys2 
 ### FP16 inference windows 
 This is an untested feature on windows.To run the object detection demo with FP16 interference follow the below steps(example with yolo4tiny):
 ```
 set TKDNN_MODE=FP16
 del /f yolo4tiny_fp16.rt
 .\test_yolo4tiny.exe
 .\demo.exe yolo4tiny_fp16.rt ..\demo\yolo_test.mp4
 ```
 ### INT8 inference windows 
 To run object detection demo with INT8 (example with yolo4tiny):
 ```
 set TKDNN_MODE=INT8
 set TKDNN_CALIB_LABEL_PATH=..\demo\COCO_val2017\all_labels.txt
 set TKDNN_CALIB_IMG_PATH=..\demo\COCO_val2017\all_images.txt
 del /f  yolo4tiny_int8.rt        # be sure to delete(or move) old tensorRT files
 .\test_yolo4tiny.exe           # run the yolo test (is slow)
 .\demo.exe yolo4tiny_int8.rt ..\demo\yolo_test.mp4 y
 ```
 ### Known issues with tkDNN on Windows
 Mobilenet and Centernet demos work properly only when built with msvc 16.7 in Release Mode,when built in debug mode for the mentioned networks one might encounter opencv assert errors
 All Darknet models work properly with demo using MSVC version(16.7-16.9)
 It is recommended to use Nvidia Driver(465+),Cuda unknown errors have been observed when using older drivers on pascal(SM 61) devices.
@@ -8,6 +8,7 @@
    - [3)Export weights for CenterNet](#3export-weights-for-centernet)
    - [4)Export weights for MobileNetSSD](#4export-weights-for-mobilenetssd)
    - [5)Export weights for CenterTrack](#5export-weights-for-centertrack)
    - [6)Export weights for ShelfNet](#6export-weights-for-shelfnet)
 - [Darknet Parser](#darknet-parser)
 ## How to export weights
@@ -30,7 +31,7 @@ make
 mkdir layers debug
 ./darknet export <path-to-cfg-file> <path-to-weights> layers
 ```
-N.b. Use compilation with CPU (leave GPU=0 in Makefile) if you also want debug. 
+N.B. Use compilation with CPU (leave GPU=0 in Makefile) if you also want debug. 
 ### 2)Export weights for DLA34 and ResNet101 
 To get weights and outputs needed to run the tests dla34 and resnet101 use the Python script and the Anaconda environment included in the repository.   
@@ -72,6 +73,19 @@ git clone https://github.com/sapienzadavide/CenterTrack.git
 python demo.py tracking,ddd --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --pre_hm --track_thresh 0.1 --demo /path/to/image/or/folder/or/video/or/webcam --test_focal_length 633 --exp_wo --exp_wo_dim 512 --input_h 512 --input_w 512
 ```
 ### 6)Export weights for ShelfNet
 To get the weights needed to run Shelfnet tests use [this](https://git.hipert.unimore.it/mverucchi/shelfnet) fork of a Pytorch implementation of Shelfnet network. 
 ```
 git clone https://git.hipert.unimore.it/mverucchi/shelfnet
 cd shelfnet 
 cd ShelfNet18_realtime
 conda env create --file shelfnet_env.yml
 conda activate shelfnet
 mkdir layer debug
 python export.py
 ```
 ## Darknet Parser
 tkDNN implement and easy parser for darknet cfg files, a network can be converted with *tk::dnn::darknetParser*:
 ```
@@ -1,8 +1,6 @@
-# mAP demo
+# Run the mAP demo
-## Run the mAP demo
+To compute mAP, precision, recall and f1score to evaluate 2D object detectors, run the map_demo.
 To compute mAP, precision, recall and f1score, run the map_demo.
 A validation set is needed. 
 To download COCO_val2017 (80 classes) run (form the root folder):