-
Notifications
You must be signed in to change notification settings - Fork 546
Open
Description
I'm new to TensorRT and am using the ONNX-TensorRT project to build inference engine, but I've encountered some issues during implementation: When running the following demo code, I notice that the output shape consistently differs from my expectations, always producing a 1-d shape [32*1280]. I've identified code supporting dynamic dimensions in the ONNX-TensorRT source. My question is: do these outputs require manual reshaping, or is there an issue with the code that I haven't identified? I greatly appreciate your response
version
tensorRT:10.2
onnx-tensort:10.14
torch: 1.16
import torch
import onnx
import onnxruntime
import onnx_tensort.backend as backend
import numpy as np
class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__()
self.linear = torch.nn.Linear(1280, 1280)
def forward(self, embed_x):
concat_hidden_state = self.linear(embed_x)
return concat_hidden_state
embed_x = torch.ones((1, 10, 1280), dtype=torch.float32)
model = Model()
torch.onnx.export(
model,
(embed_x),
"./model.onnx",
input_names=["embed_x"],
output_names=["concat_hidden_state"]
dynamic_axes={
"embed_x": {1: "embed_x_len"},
"concat_hidden_state": {1:"concat_len"}
}
)
onnx_model = onnx.load("./model.onnx")
engine = backend.prepare(onnx_model, device="CUDA:1")
x = np.random.rand(1, 32, 1280).astype(np.float32)
all_output = engine.run(x)
for out in all_output:
print(out.shape)
model = Model()
torch.onnx.export(
model,
(embed_x),
"./model.onnx",
input_names=["embed_x"],
output_names=["concat_hidden_state"]
dynamic_axes={
"embed_x": {1: "embed_x_len"},
"concat_hidden_state": {1:"concat_len"}
}
)
onnx_model = onnx.load("./model.onnx")
engine = backend.prepare(onnx_model, device="CUDA:1")
x = np.random.rand(1, 32, 1280).astype(np.float32)
all_output = engine.run(x)
for out in all_output:
print(out.shape)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels