Converting Neural Network To TensorRT . Part 1 Using Existing Plugins.

Image for post
Image for post
Every 3D bounding box estimation on image above took only 6 milliseconds.

What is TensorRT?

Image for post
Image for post
Inference time in ms

Selecting a neural network

Image for post
Image for post

General workflow

Replacing ambiguous operations

- x = Flatten()(vgg16_model.output)
+ x = Reshape((25088,))(vgg16_model.output)
- orientation = Reshape((bin_num, -1))(orientation)
+ orientation = Reshape((bin_num, 2))(orientation)

Replacing LeakyRelu with TensorRT plugin

node {
name: "leaky_re_lu_2/LeakyRelu"
op: "LeakyRelu"
input: "dense_3/BiasAdd"
attr {
key: "T"
value {
type: DT_FLOAT
}
}
attr {
key: "alpha"
value {
f: 0.10000000149011612
}
}
}
nodes {
id: "leaky_re_lu_2/LeakyRelu"
inputs: "dense_3/BiasAdd"
operation: "_LReLU_TRT"
fields {
key: "negSlope_u_float"
value {
d: 0.1
}
}
}
trt.init_libnvinfer_plugins(G_LOGGER, “”)
nvinfer1::initLibNvInferPlugins(&gLogger, “”)

Environment

Written by

Systems Software Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store