Skip to content

Brian/tensorrt ep#25

Closed
brianzheng206 wants to merge 29 commits intomainfrom
brian/tensorrt_ep
Closed

Brian/tensorrt ep#25
brianzheng206 wants to merge 29 commits intomainfrom
brian/tensorrt_ep

Conversation

@brianzheng206
Copy link
Contributor

@brianzheng206 brianzheng206 commented Dec 6, 2025 •
edited
Loading

tensorrt ep + cuda fallback for yolo inferencing at around 28-32 hz

  • probably gonna change the name of the node later on, its called deep_yolo_inference for now
  • tensorrt engines can be optionally cached, so only the first startup is slow
  • batch size is fixed at 3. can be dynamic but that means multiple trt engines

tested for 12.4.1 and 12.8.0 cudnn runtime on ubuntu22.04

lucasreljic and others added 29 commits October 27, 2025 01:43
@lucasreljic
Initial implementation of deep object detection
039b4ae
@lucasreljic
switched to onnx inferencing
761188b
@lucasreljic
Added onnxruntime gpu to cmake
0c8a8b5
@lucasreljic
pre-commit fixes
a9c84dc
@lucasreljic
Added ort_backend integration, Fixed detection parsing issues, fixed …
15948bb 
…config issues and removed other launch files

TODO: Allow dynamic batch sizes. Optimize!
@lucasreljic
Initial deep_ort_gpu plugin with internal library issues
3c61a40
@lucasreljic
Functioning onnxruntime with CUDA as EP
3bcf8cc
@lucasreljic
Initial deep_ort_gpu plugin with internal library issues
153a467
@lucasreljic
Functioning onnxruntime with CUDA as execution provider
ca1af75
@lucasreljic
Removed requirement for cuda toolkit
e9ad787
@brianzheng206
batch size and other stuff?
a79730e
@brianzheng206
hopefully fixed segfault
43d158e
@brianzheng206
working cuda inference
9cf7b2f
@brianzheng206
working tensorrt ep
6c61d4e
@brianzheng206
new stuff
0dfe8db
@Edwardius
cleaning up cmake
e785893
@brianzheng206
fixed cuda
6d052f3
@brianzheng206
merge onnx_gpu_backend
20e1f6e
@brianzheng206
added libnvinfer, trt ep should work
43130e3
@brianzheng206
caching
db12c23
@brianzheng206
merge main
8b1ee9f
@brianzheng206
should be working
6606449
@brianzheng206
fixed batch size
d916c81
@brianzheng206
optional engine caching
5dfd41b
@brianzheng206
some cleanup
8f08d2e
@brianzheng206
tensorrt version
fb31f7f
@brianzheng206
precommit and cleanup
3cb5dfd
@brianzheng206
added lifecycle node
39936f9
@brianzheng206
loading backend at runtime
ac9a7dd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@brianzheng206 @lucasreljic @Edwardius