|
| 1 | +<!-- |
| 2 | +# Copyright 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. |
| 3 | +# |
| 4 | +# Redistribution and use in source and binary forms, with or without |
| 5 | +# modification, are permitted provided that the following conditions |
| 6 | +# are met: |
| 7 | +# * Redistributions of source code must retain the above copyright |
| 8 | +# notice, this list of conditions and the following disclaimer. |
| 9 | +# * Redistributions in binary form must reproduce the above copyright |
| 10 | +# notice, this list of conditions and the following disclaimer in the |
| 11 | +# documentation and/or other materials provided with the distribution. |
| 12 | +# * Neither the name of NVIDIA CORPORATION nor the names of its |
| 13 | +# contributors may be used to endorse or promote products derived |
| 14 | +# from this software without specific prior written permission. |
| 15 | +# |
| 16 | +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY |
| 17 | +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE |
| 18 | +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR |
| 19 | +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR |
| 20 | +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, |
| 21 | +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, |
| 22 | +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR |
| 23 | +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY |
| 24 | +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
| 25 | +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
| 26 | +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| 27 | +--> |
| 28 | + |
| 29 | +# Release Notes for 2.38.0 |
| 30 | + |
| 31 | +## New Freatures and Improvements |
| 32 | + |
| 33 | +* Triton now has Python bindings for the C API. Please refer to |
| 34 | + [this PR](https://github.com/triton-inference-server/core/pull/265) for |
| 35 | + usage. |
| 36 | + |
| 37 | +* Triton now forwards request parameters to each of the composing models of an |
| 38 | + ensemble model. |
| 39 | + |
| 40 | +* The Filesystem API now supports named temporary cache directories when |
| 41 | + downloading models using the repository agent. |
| 42 | + |
| 43 | +* Added the number of requests currently in the queue to the metrics API. |
| 44 | + Documentation can be found |
| 45 | + [here](https://github.com/triton-inference-server/server/blob/r23.09/docs/user_guide/metrics.md#pending-request-count-queue-size-per-model). |
| 46 | + |
| 47 | +* Python backend models can now respond with error codes in addition to error |
| 48 | + messages. |
| 49 | + |
| 50 | +* TensorRT backend now supports |
| 51 | + [TensortRT version compatibility](https://github.com/triton-inference-server/tensorrt_backend/tree/r23.09#command-line-options) |
| 52 | + across models generated with the same major version of TensorRT. Use the |
| 53 | + `--backend-config=tensorrt,--version-compatible=true` flag to enable this |
| 54 | + feature. |
| 55 | + |
| 56 | +* Triton’s backend API now supports accessing the inference response outputs by |
| 57 | + name or by index. See the new API |
| 58 | + [here](https://github.com/triton-inference-server/core/blob/r23.09/include/triton/core/tritonbackend.h#L1572-L1608). |
| 59 | + |
| 60 | +* The Python backend now supports loading |
| 61 | + [Pytorch models directly](https://github.com/triton-inference-server/python_backend/tree/r23.08#pytorch-platform-experimental). |
| 62 | + This feature is experimental and should be treated as Beta. |
| 63 | + |
| 64 | +* Fixed an issue where if the user didn't call `SetResponseReleaseCallback`, |
| 65 | + canceling a new request could cancel the old response factory as well. Now |
| 66 | + when canceling a request which is being re-used, a new response factory is |
| 67 | + created for each inference. |
| 68 | + |
| 69 | +* Refer to the 23.09 column of the |
| 70 | + [Frameworks Support Matrix](https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html) |
| 71 | + for container image versions on which the 23.09 inference server container is |
| 72 | + based. |
| 73 | + |
| 74 | +## Known Issues |
| 75 | + |
| 76 | +* When using decoupled models, there is a possibility that response order as |
| 77 | + sent from the backend may not match with the order in which these responses |
| 78 | + are received by the streaming gRPC client. Note that this only applies to |
| 79 | + responses from different requests. Any responses corresponding to the same |
| 80 | + request will still be received in their expected order, relative to each |
| 81 | + other. |
| 82 | + |
| 83 | +* The FasterTransformer backend is only officially supported for 22.12, though |
| 84 | + it can be built for Triton container versions up to 23.07. |
| 85 | + |
| 86 | +* The Java CAPI is known to have intermittent segfaults we’re looking for a |
| 87 | + root cause. |
| 88 | + |
| 89 | +* Some systems which implement `malloc()` may not release memory back to the |
| 90 | + operating system right away causing a false memory leak. This can be mitigated |
| 91 | + by using a different malloc implementation. Tcmalloc and jemalloc are |
| 92 | + installed in the Triton container and can be |
| 93 | + [used by specifying the library in LD_PRELOAD](https://github.com/triton-inference-server/server/blob/r22.12/docs/user_guide/model_management.md). |
| 94 | + We recommend experimenting with both `tcmalloc` and `jemalloc` to determine |
| 95 | + which one works better for your use case. |
| 96 | + |
| 97 | +* Auto-complete may cause an increase in server start time. To avoid a start |
| 98 | + time increase, users can provide the full model configuration and launch the |
| 99 | + server with `--disable-auto-complete-config`. |
| 100 | + |
| 101 | +* Auto-complete does not support PyTorch models due to lack of metadata in the |
| 102 | + model. It can only verify that the number of inputs and the input names |
| 103 | + matches what is specified in the model configuration. There is no model |
| 104 | + metadata about the number of outputs and datatypes. Related PyTorch bug: |
| 105 | + https://github.com/pytorch/pytorch/issues/38273 |
| 106 | + |
| 107 | +* Triton Client PIP wheels for ARM SBSA are not available from PyPI and pip |
| 108 | + will install an incorrect Jetson version of Triton Client library for Arm |
| 109 | + SBSA. The correct client wheel file can be pulled directly from the Arm SBSA |
| 110 | + SDK image and manually installed. |
| 111 | + |
| 112 | +* Traced models in PyTorch seem to create overflows when int8 tensor values are |
| 113 | + transformed to int32 on the GPU. Refer to |
| 114 | + https://github.com/pytorch/pytorch/issues/66930 for more information. |
| 115 | + |
| 116 | +* Triton cannot retrieve GPU metrics with MIG-enabled GPU devices (A100 and A30). |
| 117 | + |
| 118 | +* Triton metrics might not work if the host machine is running a separate DCGM |
| 119 | + agent on bare-metal or in a container. |
| 120 | + |
| 121 | +* When cloud storage (AWS, GCS, AZURE) is used as a model repository and a model |
| 122 | + has multiple versions, Triton creates an extra local copy of the cloud model’s |
| 123 | + folder in the temporary directory, which is deleted upon server’s shutdown. |
0 commit comments