-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Add command-line flags to set model memory limit #1561
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
@googlebot I signed it! |
CLAs look good, thanks! ℹ️ Googlers: Go here for more info. |
@ebrevdo since you made the most recent commit, any idea who to ping for this PR or opinions on this? Thanks. |
@@ -59,6 +60,8 @@ class Server { | |||
tensorflow::string batching_parameters_file; | |||
tensorflow::string model_name; | |||
tensorflow::int32 max_num_load_retries = 5; | |||
tensorflow::int64 total_model_memory_limit_megabytes = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
set default to 0
@@ -285,7 +287,9 @@ Status Server::BuildAndStart(const Options& server_options) { | |||
options.flush_filesystem_caches = server_options.flush_filesystem_caches; | |||
options.allow_version_labels_for_unavailable_models = | |||
server_options.allow_version_labels_for_unavailable_models; | |||
|
|||
options.total_model_memory_limit_bytes = std::min( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update total_model_memory_limit_bytes
only when server_options.total_model_memory_limit_megabytes
is not zero (implying someone changed the value via command line). this ensures that default limit continues to be applied as before. your current change assumes that the default is unit64_max.
This PR adds a command-line flag to expose total model memory limit parameter to user. For ease of use limit is expressed in MB units and converted to bytes internally when passed to model server.