mmr/tkDNN
Archived
1
0
Fork 0
Code Issues 46 Pull Requests 12 Releases 4 Activity

Triton: Serialized engine contains plugin, but no plugin factory was provided. #247

New Issue
Open
opened 2021-07-21 19:15:13 +02:00 by GowthamKudupudi · 2 comments
GowthamKudupudi commented 2021-07-21 19:15:13 +02:00 (Migrated from github.com)
Copy Link
Copy Source

I have generated a .rt (.plan) file of a trained yolo4tiny Darknet model by passing the weights, input_bins and out_bins to tests/darknet/yolo4tiny.cpp.

When I try to serve the generated .plan file with Triton server, it fails to load the model with error:

...
...
I0721 04:58:21.624484 27811 backend_factory.h:44] Create TritonBackendFactory
I0721 04:58:21.624551 27811 plan_backend_factory.cc:48] Create PlanBackendFactory
I0721 04:58:21.624582 27811 plan_backend_factory.cc:55] Registering TensorRT Plugins
I0721 04:58:21.624683 27811 logging.cc:52] Registered plugin creator - ::GridAn$hor_TRT version 1
I0721 04:58:21.624707 27811 logging.cc:52] Registered plugin creator - ::NMS_TRT version 1
I0721 04:58:21.624771 27811 logging.cc:52] Registered plugin creator - ::Reorg_TRT version 1
I0721 04:58:21.624801 27811 logging.cc:52] Registered plugin creator - ::Region_TRT version 1
I0721 04:58:21.624819 27811 logging.cc:52] Registered plugin creator - ::Clip_TRT version 1
I0721 04:58:21.624831 27811 logging.cc:52] Registered plugin creator - ::LReLU_TRT version 1
I0721 04:58:21.624868 27811 logging.cc:52] Registered plugin creator - ::PriorBox_TRT version 1
I0721 04:58:21.624903 27811 logging.cc:52] Registered plugin creator - ::Normalize_TRT version 1
I0721 04:58:21.624920 27811 logging.cc:52] Registered plugin creator - ::RPROI_TRT version 1
I0721 04:58:21.624952 27811 logging.cc:52] Registered plugin creator - ::Batche$NMS_TRT version 1
I0721 04:58:21.624971 27811 logging.cc:52] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
I0721 04:58:21.625014 27811 logging.cc:52] Registered plugin creator - ::FlattenConcat_TRT version 1
I0721 04:58:21.625035 27811 logging.cc:52] Registered plugin creator - ::CropAndResize version 1
I0721 04:58:21.625053 27811 logging.cc:52] Registered plugin creator - ::DetectionLayer_TRT version 1
I0721 04:58:21.625069 27811 logging.cc:52] Registered plugin creator - ::Proposal version 1
I0721 04:58:21.625084 27811 logging.cc:52] Registered plugin creator - ::ProposalLayer_TRT version 1
I0721 04:58:21.625108 27811 logging.cc:52] Registered plugin creator - ::PyramidROIAlign_TRT version 1
I0721 04:58:21.625133 27811 logging.cc:52] Registered plugin creator - ::ResizeNearest_TRT version 1
I0721 04:58:21.625156 27811 logging.cc:52] Registered plugin creator - ::Split version 1
I0721 04:58:21.625174 27811 logging.cc:52] Registered plugin creator - ::SpecialSlice_TRT version 1
I0721 04:58:21.625204 27811 logging.cc:52] Registered plugin creator - ::InstanceNormalization_TRT version 1
E0721 04:58:23.444743 27811 logging.cc:43] deserializationUtils.cpp (635) - Serialization Error in load: 0 (Serialized engine contains plugin, but no plugin factory was provided. To deserialize an engine without a factory, please use IPluginV2 instead.)
E0721 04:58:23.446126 27811 logging.cc:43] INVALID_STATE: std::exception
E0721 04:58:23.447053 27811 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed.
...
...

When I try to run Triton with LD_PRELOAD=libkernels.so it throws the below additional errors

E0721 10:11:41.959468 27857 logging.cc:43] coreReadArchive.cpp (32) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
E0721 10:11:41.959607 27857 logging.cc:43] INVALID_STATE: std::exception
E0721 10:11:41.959627 27857 logging.cc:43] INVALID_CONFIG: Deserialize the cudaengine failed.
E0721 10:11:41.961273 27857 logging.cc:43] coreReadArchive.cpp (32) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
E0721 10:11:41.961326 27857 logging.cc:43] INVALID_STATE: std::exception
E0721 10:11:41.961346 27857 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed.

how to create custom plugins .so with tkDNN? is libkernels.so that was created while building the tkDNN, the custom plugins shared object? Can I generate the .rt (.plan) file by using inbuilt TRT plugins mentioned in the above log in place of custom plugins?

I have built and run the tritonserver on the same container in which tkDNN is built and .rt file is generated.

I have generated a `.rt` (`.plan`) file of a trained `yolo4tiny` Darknet model by passing the `weights`, `input_bins` and `out_bins` to `tests/darknet/yolo4tiny.cpp`. When I try to serve the generated `.plan` file with Triton server, it fails to load the model with error: ``` ... ... I0721 04:58:21.624484 27811 backend_factory.h:44] Create TritonBackendFactory I0721 04:58:21.624551 27811 plan_backend_factory.cc:48] Create PlanBackendFactory I0721 04:58:21.624582 27811 plan_backend_factory.cc:55] Registering TensorRT Plugins I0721 04:58:21.624683 27811 logging.cc:52] Registered plugin creator - ::GridAn$hor_TRT version 1 I0721 04:58:21.624707 27811 logging.cc:52] Registered plugin creator - ::NMS_TRT version 1 I0721 04:58:21.624771 27811 logging.cc:52] Registered plugin creator - ::Reorg_TRT version 1 I0721 04:58:21.624801 27811 logging.cc:52] Registered plugin creator - ::Region_TRT version 1 I0721 04:58:21.624819 27811 logging.cc:52] Registered plugin creator - ::Clip_TRT version 1 I0721 04:58:21.624831 27811 logging.cc:52] Registered plugin creator - ::LReLU_TRT version 1 I0721 04:58:21.624868 27811 logging.cc:52] Registered plugin creator - ::PriorBox_TRT version 1 I0721 04:58:21.624903 27811 logging.cc:52] Registered plugin creator - ::Normalize_TRT version 1 I0721 04:58:21.624920 27811 logging.cc:52] Registered plugin creator - ::RPROI_TRT version 1 I0721 04:58:21.624952 27811 logging.cc:52] Registered plugin creator - ::Batche$NMS_TRT version 1 I0721 04:58:21.624971 27811 logging.cc:52] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1 I0721 04:58:21.625014 27811 logging.cc:52] Registered plugin creator - ::FlattenConcat_TRT version 1 I0721 04:58:21.625035 27811 logging.cc:52] Registered plugin creator - ::CropAndResize version 1 I0721 04:58:21.625053 27811 logging.cc:52] Registered plugin creator - ::DetectionLayer_TRT version 1 I0721 04:58:21.625069 27811 logging.cc:52] Registered plugin creator - ::Proposal version 1 I0721 04:58:21.625084 27811 logging.cc:52] Registered plugin creator - ::ProposalLayer_TRT version 1 I0721 04:58:21.625108 27811 logging.cc:52] Registered plugin creator - ::PyramidROIAlign_TRT version 1 I0721 04:58:21.625133 27811 logging.cc:52] Registered plugin creator - ::ResizeNearest_TRT version 1 I0721 04:58:21.625156 27811 logging.cc:52] Registered plugin creator - ::Split version 1 I0721 04:58:21.625174 27811 logging.cc:52] Registered plugin creator - ::SpecialSlice_TRT version 1 I0721 04:58:21.625204 27811 logging.cc:52] Registered plugin creator - ::InstanceNormalization_TRT version 1 E0721 04:58:23.444743 27811 logging.cc:43] deserializationUtils.cpp (635) - Serialization Error in load: 0 (Serialized engine contains plugin, but no plugin factory was provided. To deserialize an engine without a factory, please use IPluginV2 instead.) E0721 04:58:23.446126 27811 logging.cc:43] INVALID_STATE: std::exception E0721 04:58:23.447053 27811 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed. ... ... ``` When I try to run `Triton` with `LD_PRELOAD=libkernels.so` it throws the below additional errors ``` E0721 10:11:41.959468 27857 logging.cc:43] coreReadArchive.cpp (32) - Serialization Error in verifyHeader: 0 (Magic tag does not match) E0721 10:11:41.959607 27857 logging.cc:43] INVALID_STATE: std::exception E0721 10:11:41.959627 27857 logging.cc:43] INVALID_CONFIG: Deserialize the cudaengine failed. E0721 10:11:41.961273 27857 logging.cc:43] coreReadArchive.cpp (32) - Serialization Error in verifyHeader: 0 (Magic tag does not match) E0721 10:11:41.961326 27857 logging.cc:43] INVALID_STATE: std::exception E0721 10:11:41.961346 27857 logging.cc:43] INVALID_CONFIG: Deserialize the cuda engine failed. ``` how to create custom plugins `.so` with tkDNN? is `libkernels.so` that was created while building the `tkDNN`, the custom plugins shared object? Can I generate the `.rt` (`.plan`) file by using inbuilt `TRT` plugins mentioned in the above log in place of custom plugins? I have built and run the `tritonserver` on the same container in which `tkDNN` is built and `.rt` file is generated.
TheExDeus commented 2021-10-28 12:37:15 +02:00 (Migrated from github.com)
Copy Link
Copy Source

The current tkdnn doesn't support plugins registry (which is why it doesn't work on trt8). You can try building the .so in this branch: https://github.com/TheExDeus/tkDNN/tree/feature/tensorrt8_support

The current tkdnn doesn't support plugins registry (which is why it doesn't work on trt8). You can try building the .so in this branch: https://github.com/TheExDeus/tkDNN/tree/feature/tensorrt8_support
mive93 commented 2021-11-25 11:14:14 +01:00 (Migrated from github.com)
Copy Link
Copy Source

TensorRT8 is now supported on tensorrt8 branch.

TensorRT8 is now supported on [tensorrt8 branch](https://github.com/ceccocats/tkDNN/tree/tensorrt8).
This repo is archived. You cannot comment on issues.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
duplicate
enhancement
explanation
help wanted
invalid
question
wontfix
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
eeeck (Erick)
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mmr/tkDNN#247
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?