export rect model

[enhany]

After export detect model with rectangle size (example: 320*640), tensorrt engine model doesn't detect anything (with end2end). Without end2end there will be many-many wrong boxes.
Tensorrt: 10.3

code (imgsz added):

from ultralytics import YOLO
model = YOLO("yolo11n.pt")
model.export(format='onnx', imgsz=[320,640])

and then follow manual

python export.py  -o yolo11n.onnx -e yolov11n.trt --end2end --v8 -p fp32
python trt.py -e yolov11n.trt  -i src/1.jpg -o yolov11-1.jpg --end2end 

[Dartayous]

Hey there enhany!

I tested rectangular TensorRT export separately using the current Ultralytics export pipeline and was able to successfully build a TensorRT engine with:

imgsz=[320, 640]

Environment:
• Ultralytics 8.4.51
• TensorRT 10.16
• CUDA 13
• RTX 2080 SUPER (my strongest graphics card)

Successful engine dimensions:
input: (1, 3, 320, 640)
output: (1, 84, 4200)

So rectangular TensorRT export itself appears to work correctly.

This suggests to me that the issue may instead be related to:
• --end2end postprocessing
• NMS integration
• box decoding
• coordinate scaling after letterbox/resize
• repo-specific ONNX/TensorRT wrapper logic

especially since:
• --end2end --v8 gives no detections
• disabling --end2end produces detections but incorrect boxes

which looks more like a postprocessing mismatch than an engine build failure.

I know this is an old post and you might not even need this help anymore, but I hope this helps.
Also thanks for making this issue report public, because it also helped me in my learning journey.

Take care,
Dartayous