Feat/lock free server #236
Conversation
orca-zhang
commented
Feb 27, 2025
orca-zhang
commented
- I have read the contributing guidelines
- Self-reported review complexity:
- Low
- Medium
- High
For a tensor with zero elements ggml_nbytes() was returning uint64_t::max, and this was causing graph allocation failure.
|
Thank you for this PR. LGTM, but as I never use the server and I'm not familiar with the code, I have assigned @saood06 to review it. |
|
Hi Ikawrakow, Please accept my apologies for the accidental PR submission during my preliminary testing phase. I'm currently conducting informal experiments without rigorous benchmarking, and cannot yet confirm the actual utility of these code changes. During my evaluation of DeepSeek-R1-671B performance, I observed occasionnally perceptible latency in Time-to-First-Token (TTFT) measurements within the llama.cpp implementation. This preliminary observation coincided with identifying a potentially prolonged lock duration in the execution flow while reviewing the codebase and profiling results which are early-stage findings requiring further validation. Thank you for your continued dedication to maintaining this exceptional codebase. I'm consistently impressed by the engineering rigor demonstrated in this project. |
examples/server/atomic_hash_map.hpp
Outdated
| /* | ||
| * Copyright 2013-present Facebook, Inc. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ |
There was a problem hiding this comment.
This is Apache, while this project is MIT.
You can set this to be a draft PR until it is ready to be reviewed, but for now I did leave a comment on the license mismatch from some of the code in your PR. |
