mmr/tkDNN
Archived
1
0
Fork 0
Code Issues 46 Pull Requests 12 Releases 4 Activity

Feature-request: YOLOv4-tiny (detector) #59

New Issue
Closed
opened 2020-06-25 16:49:34 +02:00 by AlexeyAB · 24 comments
AlexeyAB commented 2020-06-25 16:49:34 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Feature-request: YOLOv4-tiny (detector)

Many other features from Darknet were added previously.
There is required only 1 feature:

  1. Add groups= and group_id= to the [route] layer.
[route]
layers=-1
groups=2
group_id=1

So if input WxHxC, it divides input by 2 groups WxHx(C/2) (there are 2 groups: 0 and 1), and loads the 2nd group_1 WxHx(C/2).

If there are many layers specified in layers= parameter, then this will be done for each of the input layers specified in layer=, then results will be concatenated across channels.

cmp

Feature-request: YOLOv4-**tiny** (detector) * **source code**: https://github.com/AlexeyAB/darknet * **cfg**: https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny.cfg * **weights**: https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.weights * **discussion**: https://github.com/AlexeyAB/darknet/issues/6067 Many other features from Darknet were added previously. There is required only 1 feature: 1. Add `groups=` and `group_id=` to the `[route]` layer. ```ini [route] layers=-1 groups=2 group_id=1 ``` So if input `WxHxC`, it divides input by 2 groups `WxHx(C/2)` (there are 2 groups: 0 and 1), and loads the 2nd group_1 `WxHx(C/2)`. If there are many layers specified in `layers=` parameter, then this will be done for each of the input layers specified in `layer=`, then results will be concatenated across channels. ![cmp](https://user-images.githubusercontent.com/4096485/85734112-6e366700-b705-11ea-95d1-fcba0de76d72.png)
👍 6 👀 1
mive93 commented 2020-06-29 09:42:40 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @AlexeyAB :)
We'll look into that this week.

Hi @AlexeyAB :) We'll look into that this week.
👍 1 🚀 1
mive93 commented 2020-06-30 19:40:26 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @AlexeyAB ,
Yolov4 tiny is now supported on tkDNN.
Tomorrow I will do some performance tests, today my GPU is busy training.

Hi @AlexeyAB , Yolov4 tiny is now supported on tkDNN. Tomorrow I will do some performance tests, today my GPU is busy training.
👍 7 ❤️ 3 🚀 1
JasonDoingGreat commented 2020-07-01 04:55:44 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@mive93 Thanks for the Yolov4-Tiny impl

I've tested on Jetson Nano with JetPack 4.4, TensorRT v7.1, 416 input size

For FP32, profile results:

Time stats:
Min: 37.3371 ms
Max: 122.952 ms
Avg: 38.0922 ms	26.2521 FPS

For FP16, profile results:

Time stats:
Min: 24.5687 ms
Max: 90.5088 ms
Avg: 25.5292 ms	39.1709 FPS
@mive93 Thanks for the Yolov4-Tiny impl I've tested on Jetson Nano with JetPack 4.4, TensorRT v7.1, 416 input size **For FP32, profile results:** ``` Time stats: Min: 37.3371 ms Max: 122.952 ms Avg: 38.0922 ms 26.2521 FPS ``` **For FP16, profile results:** ``` Time stats: Min: 24.5687 ms Max: 90.5088 ms Avg: 25.5292 ms 39.1709 FPS ```
👍 7
mive93 commented 2020-07-01 10:20:17 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @JasonDoingGreat,

thanks :)
Here the inference results on the RTX 2080Ti (CUDA 10.2, TensorRT 7.0.0, Cudnn 7.6.5); for yolo4tiny 416x416, on 1200 images of size 416x416.

model precision batch avg (ms) min (ms) max(ms) avg FPS
yolo4tiny fp32 1 1,64185 1,57668 1,71029 609,068
yolo4tiny fp32 4 1,0385 1,03024 1,08981 962,926
yolo4tiny fp16 1 1,26474 0,90607 1,4321 790,677
yolo4tiny fp16 4 0,563628 0,556467 0,620871 1774,22
yolo4tiny int8 1 1,03339 0,728739 1,16966 967,69
yolo4tiny int8 4 0,474048 0,467551 0,506916 2109,49

If needed I can test it on the Xavier or tx2

Hi @JasonDoingGreat, thanks :) Here the inference results on the RTX 2080Ti (CUDA 10.2, TensorRT 7.0.0, Cudnn 7.6.5); for yolo4tiny 416x416, on 1200 images of size 416x416. | model | precision | batch | avg (ms) |min (ms) | max(ms) | avg FPS | |:-: |:-: |:-: |:-: |:-: |:-: |:-: | |yolo4tiny | fp32 | 1 | 1,64185 | 1,57668 | 1,71029 | 609,068 |yolo4tiny | fp32 | 4 | 1,0385 | 1,03024 | 1,08981 | 962,926 |yolo4tiny | fp16 | 1 | 1,26474 | 0,90607 | 1,4321 | 790,677 |yolo4tiny | fp16 | 4 | 0,563628 | 0,556467 | 0,620871 | 1774,22 |yolo4tiny | int8 | 1 | 1,03339 | 0,728739 | 1,16966 | 967,69 |yolo4tiny | int8 | 4 | 0,474048 | 0,467551 | 0,506916 | 2109,49 If needed I can test it on the Xavier or tx2
👍 7 🎉 3 🚀 3
AlexeyAB commented 2020-07-01 13:14:29 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@mive93 Thanks! Yes, please test it on AGX or NX with max_N.

@mive93 Thanks! Yes, please test it on AGX or NX with max_N.
👍 1
mive93 commented 2020-07-01 15:33:13 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Here it is.
Results on Xavier AGX, Jetpack 4.3 (CUDA 10.0, CUDNN 7.6.3, tensorrt 6.0.1 ); for yolo4tiny 416x416, on 1200 images of size 416x416.

model precision batch avg (ms) min (ms) max(ms) avg FPS
yolo4tiny fp32 1 6,36684 6,31811 6,48507 157,064
yolo4tiny fp32 4 5,61027 5,58927 5,63641 178,244
yolo4tiny fp16 1 3,48334 3,44269 3,56074 287,081
yolo4tiny fp16 4 2,63374 2,61526 2,65826 379,688
yolo4tiny int8 1 3,13312 3,08334 3,24114 319,17
yolo4tiny int8 4 2,33578 2,32111 2,359 428,122
Here it is. Results on Xavier AGX, Jetpack 4.3 (CUDA 10.0, CUDNN 7.6.3, tensorrt 6.0.1 ); for yolo4tiny 416x416, on 1200 images of size 416x416. |model | precision | batch | avg (ms) | min (ms) | max(ms) | avg FPS| |:--: | :--: | :--: | :--: | :--: | :--: | :--: yolo4tiny | fp32 | 1 | 6,36684 | 6,31811 | 6,48507 | 157,064 yolo4tiny | fp32 | 4 | 5,61027 | 5,58927 | 5,63641 | 178,244 yolo4tiny | fp16 | 1 | 3,48334 | 3,44269 | 3,56074 | 287,081 yolo4tiny | fp16 | 4 | 2,63374 | 2,61526 | 2,65826 | 379,688 yolo4tiny | int8 | 1 | 3,13312 | 3,08334 | 3,24114 | 319,17 yolo4tiny | int8 | 4 | 2,33578 | 2,32111 | 2,359 | 428,122
👍 7 ❤️ 1 🚀 2
CSTEZCAN commented 2020-07-01 17:26:49 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@AlexeyAB @ceccocats @mive93 , single-handedly destroyed the reputation of google, facebook and nvidia. this is extraordinary.

@AlexeyAB @ceccocats @mive93 , single-handedly destroyed the reputation of google, facebook and nvidia. this is extraordinary.
AlexeyAB commented 2020-07-02 18:20:16 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@mive93 Hi,
Does tkDNN work only with converted weights Darknet->tkDNN?
Or can tkDNN work with yolov4.weights file directly without conversion.

@mive93 Hi, Does tkDNN work only with converted weights Darknet->tkDNN? Or can tkDNN work with yolov4.weights file directly without conversion.
mive93 commented 2020-07-06 13:54:38 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @AlexeyAB,
it is necessary to export to our format.

Hi @AlexeyAB, it is necessary to export to our format.
mmaaz60 commented 2020-07-07 14:00:50 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @AlexeyAB,
it is necessary to export to our format.

Hi @mive93 @AlexeyAB,

Is there any accuracy degradation when you convert darknet weights to tkDNN format? What about accuracy loss when inferring in FP16 or INT8 mode? Is there any way to fine-tune the models in FP16 or INT8 mode or perform quantization aware training beforehand? Thanks

> Hi @AlexeyAB, > it is necessary to export to our format. Hi @mive93 @AlexeyAB, Is there any accuracy degradation when you convert darknet weights to tkDNN format? What about accuracy loss when inferring in FP16 or INT8 mode? Is there any way to fine-tune the models in FP16 or INT8 mode or perform quantization aware training beforehand? Thanks
mive93 commented 2020-07-15 19:23:17 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @mmaaz60
The conversion of the weights does not lead to accuracy degradation. Actually the only thing we do is splitting weights layer by layer. However yes, I noticed that there is a very tiny accuracy drop. I checked (almost layer by layer) the weights of darknet and tkDNN and the problem is not there, rather on the output of the network. This is due to the different implementation of the operation (IMHO).

For the FP16 mode, the drop from full precision is negligible, while the drop from full precision to INT8 is heavy.
The problem with INT8 is also the calibration step. We have tried with 100/1000 images. Maybe using more, or using more variance would lead to better results.

I hope I covered all your doubts.

Hi @mmaaz60 The conversion of the weights does not lead to accuracy degradation. Actually the only thing we do is splitting weights layer by layer. However yes, I noticed that there is a very tiny accuracy drop. I checked (almost layer by layer) the weights of darknet and tkDNN and the problem is not there, rather on the output of the network. This is due to the different implementation of the operation (IMHO). For the FP16 mode, the drop from full precision is negligible, while the drop from full precision to INT8 is heavy. The problem with INT8 is also the calibration step. We have tried with 100/1000 images. Maybe using more, or using more variance would lead to better results. I hope I covered all your doubts.
👍 1
mive93 commented 2020-09-11 09:27:41 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Closing for now,
feel free to reopen.

Closing for now, feel free to reopen.
AlexeyAB commented 2020-09-27 15:49:42 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@mive93 Hi,
Have you published a paper with YOLOv4/tiny results on tkDNN or are you planning to do so?

@mive93 Hi, Have you published a paper with YOLOv4/tiny results on tkDNN or are you planning to do so?
mive93 commented 2020-10-09 12:11:49 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @AlexeyAB

Sorry for the late reply.
Actually yes, we submitted the results to a journal, and we are now under review.
However, you can find here some results: https://git.hipert.unimore.it/edbench/edbench

Anyway, if you need some test I am available to do some, I also have a Xavier NX now ;)

Hi @AlexeyAB Sorry for the late reply. Actually yes, we submitted the results to a journal, and we are now under review. However, you can find here some results: https://git.hipert.unimore.it/edbench/edbench Anyway, if you need some test I am available to do some, I also have a Xavier NX now ;)
👍 1
m-kzein commented 2020-10-13 10:32:25 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @mive93
You mention in this thread that you reach 319,17 fps for yolov4tiny (int8) on the xavier; However, on the main readme, you mention that you reach 60.61. What is the difference?

Hi @mive93 You mention in this thread that you reach 319,17 fps for yolov4tiny (int8) on the xavier; However, on the main readme, you mention that you reach 60.61. What is the difference?
mive93 commented 2020-10-13 10:45:40 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @MohammadKassemZein
The difference is that here I'm talking about yolov4Tiny, in the readme of Yolov4 (not tiny).

Hi @MohammadKassemZein The difference is that here I'm talking about yolov4Tiny, in the readme of Yolov4 (not tiny).
👍 1
m-kzein commented 2020-10-13 10:47:48 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@mive93 Nice !
I am going to test it now on the Xavier NX.

Thank you.

@mive93 Nice ! I am going to test it now on the Xavier NX. Thank you.
👍 1
m-kzein commented 2020-10-13 12:09:01 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@mive93 Below are the results on Xavier NX.

model precision batch avg (ms) min (ms) max(ms) avg FPS
yolo4tiny fp32 1 12.3885 10.1343 48.9955 80.7199
yolo4tiny fp32 4 10.2807 9.80685 32.563 97.27
yolo4tiny fp16 1 8.47916 6.8818 38.0609 117.936
yolo4tiny fp16 4 5.01874 4.54197 24.2833 199.253
@mive93 Below are the results on Xavier NX. | model | precision | batch | avg (ms) | min (ms) | max(ms) | avg FPS | |-----------|-----------|-------|----------|----------|---------|---------| | yolo4tiny | fp32 | 1 | 12.3885 | 10.1343 | 48.9955 | 80.7199 | | yolo4tiny | fp32 | 4 | 10.2807 | 9.80685 | 32.563 | 97.27 | | yolo4tiny | fp16 | 1 | 8.47916 | 6.8818 | 38.0609 | 117.936 | | yolo4tiny | fp16 | 4 | 5.01874 | 4.54197 | 24.2833 | 199.253 |
👍 1
mive93 commented 2020-10-13 12:28:32 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Nice @MohammadKassemZein :)

How did you collect those results?

Nice @MohammadKassemZein :) How did you collect those results?
m-kzein commented 2020-10-13 12:44:13 +02:00 (Migrated from github.com)
Copy Link
Copy Source

@mive93 I used your framework (tkDNN) on Jetson NX .

@mive93 I used your framework (tkDNN) on Jetson NX .
mive93 commented 2020-10-13 12:47:44 +02:00 (Migrated from github.com)
Copy Link
Copy Source

yeah I guessed :) Sorry, I was vague.

Have you activated jetson_clocks?
For the measurements, did you use the demo itself, did you wrote your code or did you use this test?

yeah I guessed :) Sorry, I was vague. Have you activated jetson_clocks? For the measurements, did you use the demo itself, did you wrote your code or did you use [this](https://github.com/ceccocats/tkDNN/blob/master/tests/test_rtinference/rtinference.cpp) test?
m-kzein commented 2020-10-13 12:52:03 +02:00 (Migrated from github.com)
Copy Link
Copy Source

I was using MODE 15W 2CORE (which I guess gives the highest clocking for GPU and CPU).
For the measurements, I used demo itself.

I was using MODE 15W 2CORE (which I guess gives the highest clocking for GPU and CPU). For the measurements, I used demo itself.
👍 1
m-kzein commented 2020-10-15 12:11:43 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi again @mive93
Have you considered adding Mask-RCNN implementation to the framework?
I believe it is a very useful network and its benchmark in terms of FPS is not up to the expectations.

Hi again @mive93 Have you considered adding Mask-RCNN implementation to the framework? I believe it is a very useful network and its benchmark in terms of FPS is not up to the expectations.
mive93 commented 2020-12-04 10:48:33 +01:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @MohammadKassemZein
Yes we have considered that, but we haven't had the need in any of your project yet.
Therefore for now, we don't plan to port it, but maybe in the future.
Lately, we have ported a semantic segmentation net.
Probably in the future we'll port also something related to instance/panoptic segmentation.

Hi @MohammadKassemZein Yes we have considered that, but we haven't had the need in any of your project yet. Therefore for now, we don't plan to port it, but maybe in the future. Lately, we have ported a semantic segmentation net. Probably in the future we'll port also something related to instance/panoptic segmentation.
This repo is archived. You cannot comment on issues.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
duplicate
enhancement
explanation
help wanted
invalid
question
wontfix
No Label enhancement
Milestone
No items
No Milestone
Assignees
Clear assignees
eeeck (Erick)
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mmr/tkDNN#59
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?