SNOW-853951: Configurable `highWaterMark` for streaming

[APTy]

Would there be a way to set the highWaterMark for streams? The improvements in #43 are great, but the default of 10 feels a bit low, and would be nice to be able to tweak that value a bit to find something that works for each use case to get optimal throughput.

[sfc-gh-dszmolka]

hi and thank you for raising this issue! rowStreamHighWaterMark seems to be there for this, could you please try setting it in Connection and see if it works for you ? example:

var connection = snowflake.createConnection({
    account: account,
    username: user,
..
    rowStreamHighWaterMark = 20 // default=10
  });

[APTy]

Thanks for the quick response! In my experience, it doesn't seem that this parameter is actually configurable. This is confirmed by @sfc-gh-ext-simba-lf here in #505.

I'd be happy to open a PR to add external: true to this param so that users can configure it, if that's something the team wants to expose to users.

[sfc-gh-dszmolka]

ah, missed that - thank you for pointing it out. definitely something to discuss, i don't see a reason why we would not want to expose it, but i'm checking with the team.

edit: of course the PR is welcome but first I want to make sure we can/want actually expose it. missing the part why we didn't do so in the first place, maybe there was a reason I'm not aware.

[APTy]

Appreciate it! If we indeed want to make it configurable, I guess there's a question of:

• Should it be configured on the connection (createConnection()) or the session (conn.execute()) or the result stream (stmt.streamRows())?
• Or something different entirely, like the CLIENT_RESULT_CHUNK_SIZE param?

(Yes - we may not want to actually expose it, but then it'd be good to know whether 10 is indeed an optimal value, or whether some performance is being left on the table because of it)