Skip to content

Optimizes NN model serialization#298

Open
kazuki wants to merge 1 commit intojubatus:developfrom
kazuki:optimize/nn_model_serde
Open

Optimizes NN model serialization#298
kazuki wants to merge 1 commit intojubatus:developfrom
kazuki:optimize/nn_model_serde

Conversation

@kazuki
Copy link
Contributor

@kazuki kazuki commented Jun 27, 2016

  • Reserves a vector/unordered_map's memory to avoid resize/rehash
  • abstract_columns<bit_vector> serializes to a raw type instead of an array

Performance (lsh, 2048bit, 10000000 records)
size: 3020.78 => 2715.60 [MiB] (10% smaller)
load: 14969.7 => 6967.9 [ms] (x2.15 faster)
save: 4686.22 => 2341.5 [ms] (x2.00 faster)

This change breaks a model compatibility.

* Reserves a vector/unordered_map's memory to avoid resize/rehash
* abstract_columns<bit_vector> serializes to a raw type instead of an array

Performance (lsh, 2048bit, 10000000 records)
  size: 3020.78 => 2715.60 [MiB] (10% smaller)
  load: 14969.7 =>  6967.9 [ms]  (x2.15 faster)
  save: 4686.22 =>  2341.5 [ms]  (x2.00 faster)
@kmaehashi kmaehashi added this to the Near Future milestone Jul 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants