Performance, latency and throughput Issue #1481
Description
I have been testing the latest Scylla Latte for benchmarking a simple 1kb payload key value setup setup. I have noticed that the on a single loader machine with 64 CPU c7i.16xlarge and a 3 node Scylla i4i.16xlarge the client (latte) latencies are extremaly high and the throughput caps at 350k.
If however, I launch 3 latte instances on the same loader, I can easily get to 1M operations per second.
During this time the Scylla side p99 latencies are always good, less than 5ms.
This version of Latte uses the latest 1.4.1 Rust driver.
If I use the very old Latte version 0.28.2 which uses 0.13.2 Rust driver, the performance is amazing and I can get to 1.3Mops with less than 5ms client (latte) latencies.
Finally, when I upgrade the latte version to, still, the very old 0.28.3 which uses Rust 1.0.0, my client (latte) latencies increase to very high numbers.
Here the top output is from 0.28.2 Latte using 0.13.2 Rust driver, while the one on the botton uses Rust 1.0.0 driver.
Going the latest Latte using the latest Rust driver 1.4.x the latencies gets even higher and throughput drops for the exact same setup.
The command used for all of these runs is
latte run /home/ubuntu/latte-profiles/key_value.rn 10.131.0.189 -q -P row_count=1063357473 -P offset=1 -P replication_factor=3 -d=60m --user=scylla_user --password=SecretP@ssw0rd --consistency=LOCAL_QUORUM --threads=60 --concurrency=64 --connections=1 --sampling=5s --retry-number=10 --retry-interval=1s,5s --request-timeout=7 -f standard1_insert:1 -f standard1_select:4
The rune script:
use std::future;
use latte::*;
const TABLETS = latte::param!("tablets", 1);
const ROW_COUNT = latte::param!("row_count", 1000000000);
const OFFSET = latte::param!("offset", 0);
const REPLICATION_FACTOR = latte::param!("replication_factor", 3);
const KEYSPACE = "keyspace1";
const TABLES = #{
"STANDARD1": "standard1",
};
const P_STMT = #{
"STANDARD1": #{
"INSERT": "p_stmt_standard1__insert",
"SELECT": "p_stmt_standard1__select",
},
};
///////////////////////
// SPECIAL FUNCTIONS //
///////////////////////
pub async fn schema(db) {
// Build the CREATE KEYSPACE statement
let ks_stmt =
`CREATE KEYSPACE IF NOT EXISTS ${KEYSPACE}
WITH REPLICATION = {'class': 'NetworkTopologyStrategy',
'replication_factor': ${REPLICATION_FACTOR}}`;
if TABLETS > 0 {
ks_stmt = ks_stmt + " AND tablets = {'enabled': true}";
}
db.execute(ks_stmt).await?;
db.execute(`CREATE TABLE IF NOT EXISTS ${KEYSPACE}.${TABLES.STANDARD1} (
key blob,
val blob,
PRIMARY KEY (key))`).await?;
}
pub async fn erase(db) {
let errors = []; let futures = [];
for table_name in TABLES.values() {
futures.push(db.execute(`TRUNCATE TABLE ${KEYSPACE}.${table_name}`));
}
for result in future::join(futures).await { match result { Err(e) => errors.push(e) } }
if errors.len() < 1 { return Ok(()) }
Err(`Failed to execute 'erase' queries: ${errors.0}`)
}
pub async fn prepare(db) {
db.prepare(P_STMT.STANDARD1.INSERT,
`INSERT INTO ${KEYSPACE}.${TABLES.STANDARD1}(key, val)` +
" VALUES (:key, :val)").await?;
db.prepare(P_STMT.STANDARD1.SELECT,
`SELECT * FROM ${KEYSPACE}.${TABLES.STANDARD1}` +
" WHERE key = :key").await?;
}
pub async fn load(db, i) {
let errors = []; let futures = [];
futures.push(standard1_insert(db, i));
for result in future::join(futures).await { match result { Err(e) => errors.push(e) } }
if errors.len() < 1 { return Ok(()) }
Err(`Failed to execute 'load' queries: ${errors.0}`)
}
//////////////////////////////////
// SPECIFIC PER-TABLE FUNCTIONS //
//////////////////////////////////
pub async fn standard1_insert(db, i) {
let idx = i % ROW_COUNT + OFFSET;
// NOTE: '10' in the 'blob(idx, 10)' is number of bytes
let key = blob(idx, 10);
// NOTE: '1000' in the 'idx+1000' changes the index making the 'blob' func use different 'seed'
let val = blob(idx+1000, 1024);
db.execute_prepared(P_STMT.STANDARD1.INSERT, [key, val]).await?
}
pub async fn standard1_select(db, i) {
let idx = i % ROW_COUNT + OFFSET;
let key = blob(idx, 10);
db.execute_prepared(P_STMT.STANDARD1.SELECT, [key]).await?
}