- CUDNN_ACTIVATION_CLIPPED_RELU is now done with ActivationType::kCLIP instead of custom layer. This improved MobildenetSSD network performance in FP16 from 57FPS to 66FPS - that's about 15%.
- ACTIVATION_LOGISTIC was implemented the same as CUDNN_ACTIVATION_SIGMOID - so instead of using custom layers I use the ActivationType::kSIGMOID. This also gave small speed boost.
- ACTIVATION_MISH is now implemented using combination of 3 layers instead of the custom one. In FP32 this doesn't do anything (and in some cases could be slower, because the MISH custom layer is quite optimized), but in FP16 and especially in YOLO networks this is much faster. YOLO4x went from 7FPS to 10FPS. YOLO4-416 went from 23FPS to 33FPS - that's 50% performance boost.
The reason why I wanted to move to TRT layers if possible is that because they are very optimized and all layer fusing operations only happens on NVidia's TRT layers. Custom layers cannot be combined (fused). So even though the MISH change initially added 300 more layers than custom MISH layer, after optimization it had 40 layers less than an optimized network with custom MISH.
An additional place where performance comes from is the fact that none of the custom layers currently support FP16. That can of course be added, but if we use TRT layers then almost all of them support FP16. Not only that gives performance, but that also removed cast layers before and after that do FP32->FP16 and FP16->FP32 conversions.
And lastly, the custom layers didn't support STRIDE operations, so TRT made extra layers that copied input from different layers (especially route concat layers). As the built-in operations support stride, then these copies were almost totally elimitated.
- None of the custom layers support anything than FP32 with default format, so I changed the support function to reflect that.
Many of the layers could be optimized if F16 was actually supported. Especially for big ones like MISH activation.
- Migrated to TRT8 API, but I didn't really try maintaining old compatiblity. Lowest support now could be TRT6.
- Refractored some stuff that are bad C++ practices and made coding really hard, like including headers in namespaces or not using an include guard.
- Needed to move the yolo container outside tkdnn object, which means we now only have one and global. Deserialization in TRT8 doesn't happen in your object, but in the plugin itself so it couldn't access the yolo objects. I think the need to hold onto yolo layers itself is flawed and shouldn't be nessesarry.
Harijs Grinbergs