Don't prepare the statement on all connections

[muzarski]

Similarly to #1289, driver unnecessarily tries to prepare the statement on all connections.

It looks like scylla prepares the statement on every shard when processing PREPARE request: https://github.com/scylladb/scylladb/blob/8f0d0daf53397aa68312571ab9d01d8b75cd1770/transport/server.cc#L1104-L1114.

We should prepare the statement on each node once.

[Lorak-mmk]

See my comment in #1289 : #1289 (comment) , it applies here too - at least partially.

[muzarski]

See my comment in #1289 : #1289 (comment) , it applies here too - at least partially.

AFAIU, the preparation error is not as critical as schema version fetch error. It's because driver will silently try to "reprepare" the statement on the failed node later, correct?

[Lorak-mmk]

It's definitely not as critical. The worst that could happen is unnecessarily returning an error from prepare: Let's say we have a 1-node cluster, and we have a broken connection, and we don't retry on another - we would retry error to the user despite possibly being able to prepare.

[muzarski]

It's definitely not as critical. The worst that could happen is unnecessarily returning an error from prepare: Let's say we have a 1-node cluster, and we have a broken connection, and we don't retry on another - we would retry error to the user despite possibly being able to prepare.

Currently we ignore errors, unless all of the requests failed - only then we return the error to the user (PrepareError::AllAttemptsFailed). I think we can follow the same approach after the fix - return error to the user only if PREPARE request failed for all nodes.

[wprzytula]

It's definitely not as critical. The worst that could happen is unnecessarily returning an error from prepare: Let's say we have a 1-node cluster, and we have a broken connection, and we don't retry on another - we would retry error to the user despite possibly being able to prepare.

Currently we ignore errors, unless all of the requests failed - only then we return the error to the user (PrepareError::AllAttemptsFailed). I think we can follow the same approach after the fix - return error to the user only if PREPARE request failed for all nodes.

Just to make sure:
What follows such a situation when only one node responds correctly to PREPARE request is: the default load balancing policy first tries to execute the prepared statement on a replica. Then, depending on its availability:

1. If it got available, DbError::Unprepared is returned, so silent repreparation is done and the statement is executed on that replica;
2. If replicas are unavailable, the single node that had prepared the statement correctly before serves the request (hopefully it manages to contact the required number of replicas, otherwise a serverside error is returned - probably DbError::{Read,Write}{Error,Timeout}).

Can you confirm?