Skip to content

output to 1d #1047

@Moonmore

Description

@Moonmore

I'm new to TensorRT and am using the ONNX-TensorRT project to build inference engine, but I've encountered some issues during implementation: When running the following demo code, I notice that the output shape consistently differs from my expectations, always producing a 1-d shape [32*1280]. I've identified code supporting dynamic dimensions in the ONNX-TensorRT source. My question is: do these outputs require manual reshaping, or is there an issue with the code that I haven't identified? I greatly appreciate your response

version
tensorRT:10.2
onnx-tensort:10.14
torch: 1.16

import torch 
import onnx
import onnxruntime
import onnx_tensort.backend as backend
import numpy as np

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(1280, 1280)
    def forward(self, embed_x):
        concat_hidden_state = self.linear(embed_x)
        return concat_hidden_state
    
embed_x = torch.ones((1, 10, 1280), dtype=torch.float32)
model = Model()

torch.onnx.export(
    model,
    (embed_x),
    "./model.onnx",
    input_names=["embed_x"],
    output_names=["concat_hidden_state"]
    dynamic_axes={
        "embed_x": {1: "embed_x_len"},
        "concat_hidden_state": {1:"concat_len"}
    }
)

onnx_model = onnx.load("./model.onnx")
engine = backend.prepare(onnx_model, device="CUDA:1")

x = np.random.rand(1, 32, 1280).astype(np.float32)
all_output = engine.run(x)

for out in all_output:
    print(out.shape)
model = Model()

torch.onnx.export(
    model,
    (embed_x),
    "./model.onnx",
    input_names=["embed_x"],
    output_names=["concat_hidden_state"]
    dynamic_axes={
        "embed_x": {1: "embed_x_len"},
        "concat_hidden_state": {1:"concat_len"}
    }
)

onnx_model = onnx.load("./model.onnx")
engine = backend.prepare(onnx_model, device="CUDA:1")

x = np.random.rand(1, 32, 1280).astype(np.float32)
all_output = engine.run(x)

for out in all_output:
    print(out.shape)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions