Recorded FPS vs actual FPS #56

New Issue

2020-06-23T21:27:08+02:00

sambo55 commented

2020-06-23 21:27:08 +02:00

(Migrated from github.com)

I'm struggling to achieve the FPS reported in the command line.

For example when I run inference on a 10 min 30fps video the reported inference fps is 300+.

I would expect that the time taken to run inference on the entire video would be 30x60x10 = 18000 / 300 fps = 60secs = 1min

Yet the code takes at least 3 mins to run. Is there something wrong with my calculation? Why would the reported fps not be the actual?

I'm struggling to achieve the FPS reported in the command line. For example when I run inference on a 10 min 30fps video the reported inference fps is 300+. I would expect that the time taken to run inference on the entire video would be 30x60x10 = 18000 / 300 fps = 60secs = 1min Yet the code takes at least 3 mins to run. Is there something wrong with my calculation? Why would the reported fps not be the actual?

ceccocats commented

2020-06-23 21:33:46 +02:00

(Migrated from github.com)

Hi sambo,
"Inference time" is only inference time, it doesn't count preprocessing, postprocessing and visualisation.
Still the demo is only an example on how to use tkdnn is not the most optimized solution in term of preprocessing, postprocessing and visualisation.

Hi sambo, "Inference time" is only inference time, it doesn't count preprocessing, postprocessing and visualisation. Still the demo is only an example on how to use tkdnn is not the most optimized solution in term of preprocessing, postprocessing and visualisation.

sambo55 commented

2020-06-24 17:01:44 +02:00

(Migrated from github.com)

Thanks. Any pointers on how to optimise those aspects?

ceccocats commented

2020-06-24 19:32:52 +02:00

(Migrated from github.com)

Opencv Is comfortable but slow expecially for visualization. Find an opengl viewer that fit your needs.
For preprocessing and postprocessing ensure that your are compiling opencv with CUDA and cudacodec

Opencv Is comfortable but slow expecially for visualization. Find an opengl viewer that fit your needs. For preprocessing and postprocessing ensure that your are compiling opencv with CUDA and cudacodec

mive93 commented

2020-06-29 09:41:53 +02:00

(Migrated from github.com)

Hi @sambo55,

If you have opencv (4.x) with contrib compiled for CUDA, then you can uncomment line 17 here. The preprocessing will be optimized on GPU.
Another thing you could do is decouple inference and visualization, using different threads.

Hi @sambo55, If you have opencv (4.x) with contrib compiled for CUDA, then you can uncomment line 17 [here](https://github.com/ceccocats/tkDNN/blob/master/include/tkDNN/DetectionNN.h). The preprocessing will be optimized on GPU. Another thing you could do is decouple inference and visualization, using different threads.

rod-hendricks commented

2020-07-03 05:51:12 +02:00

(Migrated from github.com)

I tried running Batchsize=1 vs 4 and I notice that the inference speed is not as fast as I thought it would be on a RTX2070. For size=1 I get ~6.6ms inf time while when i do size=4 I get ~17.7ms (2.6+ slower). Are these numbers correct? I am only using ~1.6Gb of GPU mem and about ~40% processing power even when running size 4.

Would you guys know of a way to optimize this by utilizing more of the GPU power?

On a side note, my pre and post processing times are awful when I do size=4 which are running a total of ~9ms. I will try to check opencv again and ensure compiled on CUDA to see if it significantly improves.

I tried running Batchsize=1 vs 4 and I notice that the inference speed is not as fast as I thought it would be on a RTX2070. For size=1 I get ~6.6ms inf time while when i do size=4 I get ~17.7ms (2.6+ slower). Are these numbers correct? I am only using ~1.6Gb of GPU mem and about ~40% processing power even when running size 4. Would you guys know of a way to optimize this by utilizing more of the GPU power? On a side note, my pre and post processing times are awful when I do size=4 which are running a total of ~9ms. I will try to check opencv again and ensure compiled on CUDA to see if it significantly improves.

rod-hendricks commented

2020-07-13 11:11:20 +02:00

(Migrated from github.com)

I built opencv-4.2.0 with cuda and cudnn enabled and found that there was no substantial improvement gained on the pre and post processing portion of Yolo3Detection.cpp. In fact, it was slower in my end if i enable (uncomment https://github.com/ceccocats/tkDNN/blob/7c2155decfc2d225f523350d55fb6d773d3b3b6c/include/tkDNN/DetectionNN.h#L17) OPENCV_CUDACONTRIB as compared to running it disabled. Am I doing this right?

Also I guess the results I have on inference speed cannot be optimized anymore on my hardware?

I built opencv-4.2.0 with cuda and cudnn enabled and found that there was no substantial improvement gained on the pre and post processing portion of Yolo3Detection.cpp. In fact, it was slower in my end if i enable (uncomment https://github.com/ceccocats/tkDNN/blob/7c2155decfc2d225f523350d55fb6d773d3b3b6c/include/tkDNN/DetectionNN.h#L17) OPENCV_CUDACONTRIB as compared to running it disabled. Am I doing this right? Also I guess the results I have on inference speed cannot be optimized anymore on my hardware?

mive93 commented

2020-07-15 19:15:51 +02:00

(Migrated from github.com)

Hi @rod-hendricks
I checked the problem both on the Xavier and the RTX2080Ti and it is actually true, that enabling that define for Yolo detectors is worse (while is better for others like Mobilenet and Centernet models). The problem with Yolo is that there are too many useless passages between host and device.

If you really want to improve pre and post processing, you could try to implement CUDA kernels for those phases, and having everything on the device. So you could copy the frame at the beginning and copy back the bounding boxes at the end. We have not tried this solution yet, but I have it's on my list to improve those parts.

Hi @rod-hendricks I checked the problem both on the Xavier and the RTX2080Ti and it is actually true, that enabling that define for Yolo detectors is worse (while is better for others like Mobilenet and Centernet models). The problem with Yolo is that there are too many useless passages between host and device. If you really want to improve pre and post processing, you could try to implement CUDA kernels for those phases, and having everything on the device. So you could copy the frame at the beginning and copy back the bounding boxes at the end. We have not tried this solution yet, but I have it's on my list to improve those parts.

👍 2

rod-hendricks commented

2020-07-16 04:45:34 +02:00

(Migrated from github.com)

Thanks for the response and advice @mive93 ! I am not sure yet as to the effort vs gain on doing this so I'll see if I can consider working on this later.

I would like to ask further though about what you meant by Yolo having too many useless passages between host and device. Do you mean that within the yolo inference task, there is still the data passages going on between host and device before the network output is produced?

Thanks for the response and advice @mive93 ! I am not sure yet as to the effort vs gain on doing this so I'll see if I can consider working on this later. I would like to ask further though about what you meant by Yolo having too many useless passages between host and device. Do you mean that within the yolo inference task, there is still the data passages going on between host and device before the network output is produced?

mive93 commented

2020-07-16 09:00:05 +02:00

(Migrated from github.com)

@rod-hendricks if I have updates on any improvements, I will let you know.

No, I meant only on our preprocessing. The code should be cleaned up and fix to remove a useless passage between host and device. When I'll have time I'll fix that :)

@rod-hendricks if I have updates on any improvements, I will let you know. No, I meant only on our preprocessing. The code should be cleaned up and fix to remove a useless passage between host and device. When I'll have time I'll fix that :)

rod-hendricks commented

2020-07-16 09:18:30 +02:00

(Migrated from github.com)

Thanks @mive93 ! Much appreciated. Good work on this repo. Its amazing!

❤️ 1

m-kzein commented

2021-08-13 10:44:03 +02:00

(Migrated from github.com)

Hello @mive93 , any updates on removing the useless passage between host and device?
Thanks.

Hello @mive93 , any updates on removing the useless passage between host and device? Thanks.

mive93 commented

2021-08-19 17:16:23 +02:00