Client Affinity by gernest · Pull Request #54 · VoltDB/voltdb-client-go

gernest · 2018-06-19T09:37:20Z

Depends on #53

The struct was not necessary, it only wraps procedureInvocation without adding any new functionality. It is much better just working with procedureInvocation directy

remove unencessary drain channels

- move response processing to one goroutine - rework draining code

* master: Update voltkv benchmark

- fix request cancelation - apply timeouts to @statistics,@SystemCatalog queries - add the wuery string to the timeout error mesage

- add more comments - fallback when client affinity picks a dead node

This seems to be irrelevant

gernest · 2018-06-21T15:03:51Z

var rst = map[string]voltdbclient.procedure{
	"VOTES.insert":            {SinglePartition: true, ReadOnly: false, PartitionParameter: 0, PartitionParameterType: 6},
	"AREA_CODE_STATE.delete":  {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"CONTESTANTS.delete":      {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"CONTESTANTS.insert":      {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"Initialize":              {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"AREA_CODE_STATE.upsert":  {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"CONTESTANTS.upsert":      {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"GetStateHeatmap":         {SinglePartition: false, ReadOnly: true, PartitionParameter: -1, PartitionParameterType: -1},
	"Vote":                    {SinglePartition: true, ReadOnly: false, PartitionParameter: 0, PartitionParameterType: 6},
	"CONTESTANTS.update":      {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"ContestantWinningStates": {SinglePartition: false, ReadOnly: true, PartitionParameter: -1, PartitionParameterType: -1},
	"Results":                 {SinglePartition: false, ReadOnly: true, PartitionParameter: -1, PartitionParameterType: -1},
	"AREA_CODE_STATE.insert":  {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
	"AREA_CODE_STATE.update":  {SinglePartition: false, ReadOnly: false, PartitionParameter: -1, PartitionParameterType: -1},
}

- add more specific assertions

gernest · 2018-06-26T14:41:20Z

TIMESTAMP      HOST_ID  HOSTNAME  SITE_ID  PARTITION_ID  PROCEDURE                       INVOCATIONS  TIMED_INVOCATIONS  MIN_EXECUTION_TIME  MAX_EXECUTION_TIME  AVG_EXECUTION_TIME  MIN_RESULT_SIZE  MAX_RESULT_SIZE  AVG_RESULT_SIZE  MIN_PARAMETER_SET_SIZE  MAX_PARAMETER_SET_SIZE  AVG_PARAMETER_SET_SIZE  ABORTS  FAILURES  TRANSACTIONAL 
-------------- -------- --------- -------- ------------- ------------------------------- ------------ ------------------ ------------------- ------------------- ------------------- ---------------- ---------------- ---------------- ----------------------- ----------------------- ----------------------- ------- --------- --------------
 1530009449115        0 mj               0             2 add_customer                               1                  1             1291353             1291353             1291353               47               47               47                      28                      28                      28       0         0              1
 1530009449115        0 mj               4         16383 org.voltdb.sysprocs.AdHoc_RO_MP            1                  1             6477318             6477318             6477318               84               84               84                    1124                    1124                    1124       0         0              1
 1530009449116        1 mj               2             2 add_customer                               1                  1             1412395             1412395             1412395               47               47               47                      28                      28                      28       0         0              1

(Returned 3 rows in 0.00s)

- add more debugging info - test against topology, no need to query voltdb for procedure stats

- don't sort the affinity logs in test - pass more meaningful parameters to selectedNode callback

Check if the host_id of the nodes are in the initiator table.

The refrence java code uses >>> which go doesnt have, we are using >> which will zero fill when the value is unsigned, by using signed int we are now having similar results to java.

- remove unecessary type cast

- rely on AFFINITY_SERVERS to pickup cluster server - don't generate a topology file - remove hashinator tests, these are already covered

gernest added 30 commits June 11, 2018 14:38

delete networkRequest

e7a463d

The struct was not necessary, it only wraps procedureInvocation without adding any new functionality. It is much better just working with procedureInvocation directy

add procedureInvocation.handleTimeouts

35dddf8

use context to manage goroutines

cb36f3d

cleanup

a6f8b8f

use blocking channel instead of forever loop

d80e7ae

fix connection close

73e80a9

partial working with sync

aeaba29

ensure done is called

443e1e2

refactor 8Conn.Drain

6145405

remove unencessary drain channels

remove unused code

1b54cc9

cleanup

e23976b

update QueryTImeout and ExecTimeout

fa0c3a7

delete closed requests and more cleanups

8014df8

improve error messages

a87dc73

remove pings from nodeConn loop

6f1bd54

fix *Conn.Drain

0572085

correct count queued requests

3bbaa5d

refactoring

0a35342

- move response processing to one goroutine - rework draining code

return early for drainage calls on closed connections

44b1d85

lower number of goroutines to 1 for async

1c4cb79

Merge branch 'master' into timeout

bc9cbba

* master: Update voltkv benchmark

use 1 goroutine for async

ecea333

add more info to error message

3a0aec6

cleanup

c6b28b1

don't panic

449aa75

use a separate connection for results

097c811

improvements

bfe82a4

- fix request cancelation - apply timeouts to @statistics,@SystemCatalog queries - add the wuery string to the timeout error mesage

don't run the benchmark in multiple goroutines

66d2717

properly set context for *Conn.subscribeTopo

80de61e

Update godoc

5ca6fa4

gernest added 9 commits June 20, 2018 13:05

update

0b4c000

- add more comments - fallback when client affinity picks a dead node

fix race condition

8e37d89

update client affinity

687aa7c

fix error handling/reporting

822493d

fix reading topology repicas details

81c46da

cleanup backpressure

8ca6097

This seems to be irrelevant

cleanup

55d2a88

fix getConnByCA api change

5d15d75

cleanup

eeed6cf

gernest added 2 commits June 22, 2018 13:53

add test to ensure we pick the right master

57b7885

dump topology details

69eadf9

gernest changed the title ~~[W.I.P] Client Affinity~~ Client Affinity Jun 24, 2018

gernest added 3 commits June 26, 2018 15:37

add TestVerifyClientAffinity

7944eeb

collect only stats about add_customer procedure

c67154c

update tests

6f6d173

- add more specific assertions

gernest added 12 commits June 26, 2018 18:43

dump more output on test

16a47c9

log nodes used

ed96403

update client affinity tests

2d49010

- add more debugging info - test against topology, no need to query voltdb for procedure stats

add comment about the client affinity test

6c62807

refactor affinity logging

972ee0f

- don't sort the affinity logs in test - pass more meaningful parameters to selectedNode callback

add nil check

46350b2

only run 10 queries

873d9f7

log INITIATOR table in the affinity tests

2bdb529

Update TestVerifyClientAffinity

98bc24b

Check if the host_id of the nodes are in the initiator table.

sign fill the hash value from hashinator

055a297

The refrence java code uses >>> which go doesnt have, we are using >> which will zero fill when the value is unsigned, by using signed int we are now having similar results to java.

cleanup

c6b775c

- remove unecessary type cast

cleanup TestClientAffinity

fb10172

- rely on AFFINITY_SERVERS to pickup cluster server - don't generate a topology file - remove hashinator tests, these are already covered

gernest mentioned this pull request Aug 17, 2018

Help with reconnect with permanent connection #56

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Client Affinity#54

Client Affinity#54
gernest wants to merge 65 commits intoVoltDB:masterfrom
gernest:affinity

gernest commented Jun 19, 2018 •

edited

Loading

Uh oh!

gernest commented Jun 21, 2018

Uh oh!

gernest commented Jun 26, 2018 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

gernest commented Jun 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gernest commented Jun 21, 2018

Uh oh!

gernest commented Jun 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

gernest commented Jun 19, 2018 •

edited

Loading

gernest commented Jun 26, 2018 •

edited

Loading