Skip to content

server/{db,auth}: add a dedicated table for tracking reputation effects #3259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

buck54321
Copy link
Member

Loading reputation history from the orders and matches tables used some gnarly queries that were getting incredibly slow as the table sizes grew. This was causing extremely long connect times that often timed out when the server connection flapped and everybody tried to connect at once. This PR adds a dedicated table for reputation effects that stays pruned, and a rolling upgrade process.

Copy link
Member

@JoeGruffins JoeGruffins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only partial review. I'm testing on simnet and I can't seem to bring reputation down. I make some matches and then don't continue them but score never decreases. Strikes should be going negative here I think.

server log
2025-05-02 19:35:19.991 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:19.991 [INF] SWAP: Sending a 'revoke_match' notification to each client for match abdb2b5052c4bb0a2734a1c2bbe0b8e7e8e7c243e1e63444bf2fd85f12b1be8d
2025-05-02 19:35:19.991 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.991 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match abdb2b5052c4bb0a2734a1c2bbe0b8e7e8e7c243e1e63444bf2fd85f12b1be8d: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.991 [DBG] SWAP: failMatch: swap ab99a57c3e0c5b1cdd3867d8bc7c56fb413359c4b8131d2192af06867eccfb7f failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:19.993 [DBG] AUTH: Recorded order 94586971590ba92767d366ac55c436b36a72a5ecb7938f01ec0e97abf97f33ef that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182119992, target=<nil>
2025-05-02 19:35:19.994 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:19.994 [INF] SWAP: Sending a 'revoke_match' notification to each client for match ab99a57c3e0c5b1cdd3867d8bc7c56fb413359c4b8131d2192af06867eccfb7f
2025-05-02 19:35:19.994 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.994 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match ab99a57c3e0c5b1cdd3867d8bc7c56fb413359c4b8131d2192af06867eccfb7f: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.994 [DBG] SWAP: failMatch: swap f70107549c60c2cfef20f5f6e1ff75ceb41a8d6c17eb103f023af6401c827322 failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:19.995 [DBG] AUTH: Recorded order 59c482fe4704ad0f04e1cd510e90a883016b80cd14d2430cb59837a1cdabf51d that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182119994, target=<nil>
2025-05-02 19:35:19.996 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:19.996 [INF] SWAP: Sending a 'revoke_match' notification to each client for match f70107549c60c2cfef20f5f6e1ff75ceb41a8d6c17eb103f023af6401c827322
2025-05-02 19:35:19.996 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.996 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match f70107549c60c2cfef20f5f6e1ff75ceb41a8d6c17eb103f023af6401c827322: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.996 [DBG] SWAP: failMatch: swap 005a3c9ca592594572302879226567030e25254d79b331cfef773a31038b4665 failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:19.997 [DBG] AUTH: Recorded order 8c6fa51df1502a45f3f65e909eaf97b0bd2c4e1b3c9926de4d59b08d7d3736e8 that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182119996, target=<nil>
2025-05-02 19:35:19.998 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:19.998 [INF] SWAP: Sending a 'revoke_match' notification to each client for match 005a3c9ca592594572302879226567030e25254d79b331cfef773a31038b4665
2025-05-02 19:35:19.998 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.998 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match 005a3c9ca592594572302879226567030e25254d79b331cfef773a31038b4665: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:19.998 [DBG] SWAP: failMatch: swap d0d8642563039321f518704cf8e6635d41ab4e4a2bcc99c0006aad136af34a47 failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:19.999 [DBG] AUTH: Recorded order 1a04639e3710b0d2c67b87bc46e5228d35f9783ed9ebf151e8d865f74bc19df0 that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182119998, target=<nil>
2025-05-02 19:35:20.000 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:20.000 [INF] SWAP: Sending a 'revoke_match' notification to each client for match d0d8642563039321f518704cf8e6635d41ab4e4a2bcc99c0006aad136af34a47
2025-05-02 19:35:20.000 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.000 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match d0d8642563039321f518704cf8e6635d41ab4e4a2bcc99c0006aad136af34a47: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.000 [DBG] SWAP: failMatch: swap f7052880bec0a3c1e0b0de4aaa9e0215bf4184fd231dfd53f79781216078acc6 failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:20.001 [DBG] AUTH: Recorded order b449193d947f059fc26ffa11d95527767385becbc9e3f549cce4e4ccbb9dd4c5 that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182120001, target=<nil>
2025-05-02 19:35:20.002 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:20.002 [INF] SWAP: Sending a 'revoke_match' notification to each client for match f7052880bec0a3c1e0b0de4aaa9e0215bf4184fd231dfd53f79781216078acc6
2025-05-02 19:35:20.002 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.002 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match f7052880bec0a3c1e0b0de4aaa9e0215bf4184fd231dfd53f79781216078acc6: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.002 [DBG] SWAP: failMatch: swap 8a9be5a5ee13aff5762171b699ca6747a42934373fde99fc006ad5c2daa685f9 failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:20.003 [DBG] AUTH: Recorded order 4706a52c6d316a2abed1f79d7fba5dacfdad90dc64926bdefa282eed6cd1dea0 that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182120003, target=<nil>
2025-05-02 19:35:20.004 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:20.004 [INF] SWAP: Sending a 'revoke_match' notification to each client for match 8a9be5a5ee13aff5762171b699ca6747a42934373fde99fc006ad5c2daa685f9
2025-05-02 19:35:20.004 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.004 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match 8a9be5a5ee13aff5762171b699ca6747a42934373fde99fc006ad5c2daa685f9: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.004 [DBG] SWAP: failMatch: swap 2d2f3b9c5ad4022cdad2b42bd54c69372ef3aee13c0ce3f03208099828ac3bc0 failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:20.005 [DBG] AUTH: Recorded order c53b1e1fe298a5eeb1e3d133dda937b8ec1260ac97680b07ef301958f3f78b6f that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182120005, target=<nil>
2025-05-02 19:35:20.006 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:20.006 [INF] SWAP: Sending a 'revoke_match' notification to each client for match 2d2f3b9c5ad4022cdad2b42bd54c69372ef3aee13c0ce3f03208099828ac3bc0
2025-05-02 19:35:20.006 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.006 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match 2d2f3b9c5ad4022cdad2b42bd54c69372ef3aee13c0ce3f03208099828ac3bc0: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.006 [DBG] SWAP: failMatch: swap e84eba96df6f0c068a9d75cf81cf61741294e55a0f87ac3679f3bab8a33c23fa failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:20.007 [DBG] AUTH: Recorded order f3e61417341a426447e534f40aabfc16b4b9cc33888a4133aba89e40db828208 that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182120006, target=<nil>
2025-05-02 19:35:20.009 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11
2025-05-02 19:35:20.009 [INF] SWAP: Sending a 'revoke_match' notification to each client for match e84eba96df6f0c068a9d75cf81cf61741294e55a0f87ac3679f3bab8a33c23fa
2025-05-02 19:35:20.009 [DBG] AUTH: Send requested for disconnected user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.009 [DBG] SWAP: Failed to send 'revoke_match' notification to user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f, match e84eba96df6f0c068a9d75cf81cf61741294e55a0f87ac3679f3bab8a33c23fa: user not connected: 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f
2025-05-02 19:35:20.009 [DBG] SWAP: failMatch: swap d58f4308f60554d34b2a7e7d1a1da81552991b70246e70512c342790871f682f failing at MakerSwapCast (4), user fault = true
2025-05-02 19:35:20.044 [DBG] AUTH: Recorded order bb24bb73dd38b7a68c6d7db70bedaa61ad1bc1a5a95ca17c9aac53c8cad1d2c6 that has finished processing: user=ba6e444e88845e954d4f7b28bfee4f9f6e254612ebbe93f193cb26b241c5485e, time=1746182120017, target=<nil>
2025-05-02 19:35:20.046 [INF] AUTH: Match failure for user 58c371400e97f6209a5f4e47113a992f9ed08dcff9aa7d9d80b50af13d56b20f: '\x04' (badness -11), strikes 0, bond tier 11 => trading tier 11

Comment on lines 737 to 739
if t.Quantity%lotSize != 0 || t.Remaining()%lotSize != 0 {
return fmt.Errorf("market sell order fails lot size requirement %d %% %d = %d", t.Quantity, lotSize, t.Quantity%lotSize)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just moved, but error leaves out possible t.Remaining()%lotSize != 0 being false.

Copy link
Collaborator

@martonp martonp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't tested.. will test after the issue @JoeGruffins found is resolved.

Comment on lines +61 to +67
})
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
})
}
}
})
}
if len(pings) >= pingSz && len(matches) >= matchSz && len(orders) >= orderSz {
break
}
}

Comment on lines +42 to +66
case db.OutcomeClassPreimage:
pimgs = append(pimgs, &db.PreimageOutcome{
DBID: dbID,
OrderID: link,
Miss: outcome == db.OutcomePreimageMiss,
})
case db.OutcomeClassMatch:
var mid order.MatchID
copy(mid[:], link[:])
matches = append(matches, &db.MatchResult{
DBID: dbID,
MatchID: mid,
MatchOutcome: outcome,
})
case db.OutcomeClassOrder:
orders = append(orders, &db.OrderOutcome{
DBID: dbID,
OrderID: link,
Canceled: outcome == db.OutcomeOrderCanceled,
})
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to append these if we already have enough for the class, and then you don't need to shorten the slices below.

CreatePointsTable = `CREATE TABLE IF NOT EXISTS %s (
id BIGSERIAL PRIMARY KEY,
account BYTEA,
link BYTEA, -- Order ID or Match ID
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
link BYTEA, -- Order ID or Match ID
link BYTEA UNIQUE, -- Order ID or Match ID

if o.Miss {
outcome = db.OutcomePreimageMiss
}
if err := stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassPreimage, outcome).Scan(&o.DBID); err != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if err := stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassPreimage, outcome).Scan(&o.DBID); err != nil {
if err := a.queries.insertPoints.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassPreimage, outcome).Scan(&o.DBID); err != nil {

and also below

Comment on lines +561 to +563
OutcomeOrderComplete
OutcomeOrderCanceled
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be confusing.. could do OutcomeCancelledEarly and OutcomeNotCancelledEarly

@@ -1711,7 +1765,7 @@ func (auth *AuthManager) handleConnect(conn comms.Link, msg *msgjson.Message) *m
return nil
}

func (auth *AuthManager) loadRecentFinishedOrders(aid account.AccountID, N int) (*latestOrders, error) {
func (auth *AuthManager) loadRecentFinishedOrders(aid account.AccountID, N int) (*latestOutcomes[*db.OrderOutcome], error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

loadRecentFinishedOrders is unused now.

@JoeGruffins
Copy link
Member

Reputation is going down as expected now.

@JoeGruffins
Copy link
Member

JoeGruffins commented May 8, 2025

On simnet, seeing an error on server whenever connecting with a new user. Seems to be fine after that though.

2025-05-08 16:47:04.261 [ERR] AUTH: failed to load user score: error getting current user reputation version: sql: no rows in result set
2025-05-08 16:47:04.262 [DBG] AUTH: Activating new bond 07ada0694c7da80dfa39d282fd5071be86f7edf489780fcf7872f49c7ef5086c:0 (dcr) committing 5000000000 for user a8db099a3436f30019446c56d2e0ead09158b15df1d9d6e0ec39adde12d2c376
2025-05-08 16:47:04.262 [INF] AUTH: Creating new user account a8db099a3436f30019446c56d2e0ead09158b15df1d9d6e0ec39adde12d2c376 from 127.0.0.1, posted first bond in 07ada0694c7da80dfa39d282fd5071be86f7edf489780fcf7872f49c7ef5086c:0 (dcr)
2025-05-08 16:47:04.262 [INF] AUTH: Bond accepted: acct a8db099a3436f30019446c56d2e0ead09158b15df1d9d6e0ec39adde12d2c376 from 127.0.0.1 locked 5000000000 in 07ada0694c7da80dfa39d282fd5071be86f7edf489780fcf7872f49c7ef5086c:0. Bond total 100, tier 100

if o.Miss {
outcome = db.OutcomePreimageMiss
}
if err := stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassPreimage, outcome).Scan(&o.DBID); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if we are using this pattern elsewhere, but if you return inside the if, this err is in a different scope than the defer err so the tx will not be rolled back.

https://go.dev/play/p/knpoiH3mcmw

Suggested change
if err := stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassPreimage, outcome).Scan(&o.DBID); err != nil {
if err = stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassPreimage, outcome).Scan(&o.DBID); err != nil {

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

}
}
for _, o := range matches {
if err := stmt.QueryRowContext(ctx, user, o.MatchID, db.OutcomeClassMatch, o.MatchOutcome).Scan(&o.DBID); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if err := stmt.QueryRowContext(ctx, user, o.MatchID, db.OutcomeClassMatch, o.MatchOutcome).Scan(&o.DBID); err != nil {
if err = stmt.QueryRowContext(ctx, user, o.MatchID, db.OutcomeClassMatch, o.MatchOutcome).Scan(&o.DBID); err != nil {

if o.Canceled {
outcome = db.OutcomeOrderCanceled
}
if err := stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassOrder, outcome).Scan(&o.DBID); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if err := stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassOrder, outcome).Scan(&o.DBID); err != nil {
if err = stmt.QueryRowContext(ctx, user, o.OrderID, db.OutcomeClassOrder, outcome).Scan(&o.DBID); err != nil {

Comment on lines +877 to +883
// if score != wantScore {
// t.Errorf("wrong score. got %d, want %d", score, wantScore)
// }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we still check this?

buck54321 added 3 commits May 12, 2025 17:56
…effects

Loading reputation history from the orders and matches tables used
some gnarly queries that were getting incredibly slow as the table
sizes grew. This was causing extremly long connect times that often
timed when the server connection flapped and everybody tried to
connect at once. This PR adds a dedicated table for reputation effects
that stays pruned, and a rolling upgrade process.
@JoeGruffins
Copy link
Member

Tested forgiving a user, looks fine. Merging soon...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants