mmr/tkDNN
Archived
1
0
Fork 0
Code Issues 46 Pull Requests 12 Releases 4 Activity

Performance improvements on NX

Browse Source 
- CUDNN_ACTIVATION_CLIPPED_RELU is now done with ActivationType::kCLIP instead of custom layer. This improved MobildenetSSD network performance in FP16 from 57FPS to 66FPS - that's about 15%.
- ACTIVATION_LOGISTIC was implemented the same as CUDNN_ACTIVATION_SIGMOID - so instead of using custom layers I use the ActivationType::kSIGMOID. This also gave small speed boost.
- ACTIVATION_MISH is now implemented using combination of 3 layers instead of the custom one. In FP32 this doesn't do anything (and in some cases could be slower, because the MISH custom layer is quite optimized), but in FP16 and especially in YOLO networks this is much faster. YOLO4x went from 7FPS to 10FPS. YOLO4-416 went from 23FPS to 33FPS - that's 50% performance boost.

The reason why I wanted to move to TRT layers if possible is that because they are very optimized and all layer fusing operations only happens on NVidia's TRT layers. Custom layers cannot be combined (fused). So even though the MISH change initially added 300 more layers than custom MISH layer, after optimization it had 40 layers less than an optimized network with custom MISH.
An additional place where performance comes from is the fact that none of the custom layers currently support FP16. That can of course be added, but if we use TRT layers then almost all of them support FP16. Not only that gives performance, but that also removed cast layers before and after that do FP32->FP16 and FP16->FP32 conversions.
And lastly, the custom layers didn't support STRIDE operations, so TRT made extra layers that copied input from different layers (especially route concat layers). As the built-in operations support stride, then these copies were almost totally elimitated.
This commit is contained in:
Harijs Grinbergs
2021-11-02 00:09:40 +02:00
parent d411af52fa
commit 18303c06c5
1 changed files with 20 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/NetworkRT.cpp
+20 -13
View File
@@ -419,28 +419,35 @@ ILayer* NetworkRT::convert_layer(ITensor *input, Activation *l) {
IActivationLayer *lRT = networkRT->addActivation(*input, ActivationType::kRELU);
checkNULL(lRT);
return lRT;
} else if(l->act_mode == CUDNN_ACTIVATION_SIGMOID) {
} else if(l->act_mode == CUDNN_ACTIVATION_SIGMOID || l->act_mode == ACTIVATION_LOGISTIC) {
IActivationLayer *lRT = networkRT->addActivation(*input, ActivationType::kSIGMOID);
checkNULL(lRT);
return lRT;
}
else if(l->act_mode == CUDNN_ACTIVATION_CLIPPED_RELU) {
auto *plugin = new ActivationReLUCeiling(l->ceiling);
auto *lRT = networkRT->addPluginV2(&input, 1, *plugin);
IActivationLayer *lRT = networkRT->addActivation(*input, ActivationType::kCLIP);
lRT->setAlpha(0);
lRT->setBeta(l->ceiling);
checkNULL(lRT);
return lRT;
}
else if(l->act_mode == ACTIVATION_MISH) {
auto *plugin = new ActivationMishRT();
auto *lRT = networkRT->addPluginV2(&input, 1, *plugin);
checkNULL(lRT);
return lRT;
}
else if(l->act_mode == ACTIVATION_LOGISTIC) {
auto *plugin = new ActivationLogisticRT();
auto *lRT = networkRT->addPluginV2(&input, 1, *plugin);
checkNULL(lRT);
return lRT;
// Uncomment this to see if you have better performance
// For older TensorRT or for FP32 this might be better
//auto *plugin = new ActivationMishRT();
//auto *lRT = networkRT->addPluginV2(&input, 1, *plugin);
// Assemble MISH using 3 layers that are going to be merged by TensorRT
IActivationLayer *lRT1 = networkRT->addActivation(*input, ActivationType::kSOFTPLUS);
lRT1->setAlpha(1);
lRT1->setBeta(1);
IActivationLayer *lRT2 = networkRT->addActivation(*lRT1->getOutput(0), ActivationType::kTANH);
IElementWiseLayer *lRT3 = networkRT->addElementWise(*input, *lRT2->getOutput(0), ElementWiseOperation::kPROD);
checkNULL(lRT3);
return lRT3;
}
else {
FatalError("this Activation mode is not yet implemented");