simulate: resource population #6015

joe-p · 2024-06-05T22:59:02Z

Summary

When a user calls simulate with UnnamedResources enabled, simulate should suggest to the user how they can populate the resource arrays in their transactions to properly send the transaction group to the network.

Test Plan

Test ResourcePopulator works with simple local (not group sharing) resources
Test ResourcePopulator with group sharing
Test ResourcePopulator resource limit detection with group sharing (ie. it is able to find the correct transaction to put a resource in)
Test Simulate with ResourcePopulator functionality
Test /simulate endpoint with ResourcePopulator functionality
Write smaller tests for better ledger/simulation/resources.go coverage

codecov · 2024-06-06T11:17:49Z

Codecov Report

Attention: Patch coverage is 89.97494% with 40 lines in your changes missing coverage. Please review.

Project coverage is 51.77%. Comparing base (c95bb50) to head (a253291).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
daemon/algod/api/server/v2/utils.go	0.00%	26 Missing ⚠️
ledger/simulation/resources.go	97.22%	6 Missing and 4 partials ⚠️
ledger/simulation/simulator.go	66.66%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6015      +/-   ##
==========================================
+ Coverage   51.60%   51.77%   +0.17%     
==========================================
  Files         649      649              
  Lines       87048    87447     +399     
==========================================
+ Hits        44917    45277     +360     
- Misses      39269    39302      +33     
- Partials     2862     2868       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joe-p · 2025-03-08T22:54:26Z

~~CI is failing due to an unrelated test: TestP2PEnableGossipService_BothDisable~~

Re-triggered test run passing

jannotti

I still have a lot to look at, but don't want to sit on my current comments.

daemon/algod/api/server/v2/utils.go

ledger/simulation/trace.go

jannotti · 2025-03-14T13:03:28Z

daemon/algod/api/algod.oas2.json

          "description": "The account that needed to sign this transaction when no signature was provided and the provided signer was incorrect.",
          "type": "string",
          "x-algorand-format": "Address"
+        },
+        "populated-resource-arrays": {


I'm basically agnostic on this question: Do you think this should be present if it's empty? I lean toward it not appearing, but don't care too much.

I think mirroring the behavior of unnamed-resources-accessed (not being a required field) makes the most sense

daemon/algod/api/server/v2/utils.go

ledger/simulation/resources.go

jannotti

A couple small utility functions that I had forgotten we had ready to go.

daemon/algod/api/server/v2/utils.go

prefer Sort over SortFunc and use Cmp.or instead of Sprintf

Co-authored-by: John Jannotti <[email protected]>

joe-p · 2025-05-05T18:58:13Z

All comments have been addressed. I also noticed CI failed. I need to step away for a minute but will take a closer look later today.

Also apologies for the long delays between addressing feedback, should be able to focus on this to get it across the finish line now.

Edit: Seemed to have introduced a regression
Edit2: Should be fixed in 0df1e0f. Tests passing locally, will wait for CI

This was erroneously removed in 6353f3b

jannotti · 2025-05-05T23:29:44Z

You probably need to re-merge master and make api to fix the generated conflicts.

jannotti

Some simplistic code comments, and some question about when you add certain things. I think the existing order may be perfectly fine, despite my early comments reservations. But take my confusion as a request to find a good place to describe the whole strategy all at once in a comment.

I became worried that you are not accounting for old avm version apps, which can't use resource sharing. But I think maybe you're ok, because the response from simulate was taken that into account and places the necessary refs in the specific transactions if needed?

jannotti · 2025-05-06T15:17:24Z

daemon/algod/api/server/v2/utils.go

+		Boxes:    &boxes,
+	}
+
+	return &populatedResourceArrays


I'm concerned this will always return in the json even if it's empty. It will always be non-nil, and it will always contain non-nil fields. boxes is always non-nil, and the others probably are too, depending on exactly how convertSlice deals with empty slices.

ledger/simulation/resources.go

jannotti · 2025-05-06T16:07:21Z

ledger/simulation/resources.go

+	return nil
+}
+
+// addBox adds a box to the box array. It does NOT add the app to the app array.


Perhaps it should fail if the app is not there then? I haven't looked to see how it's used yet...

ledger/simulation/resources.go

jannotti · 2025-05-06T16:31:04Z

ledger/simulation/resources.go

+	for box := range groupResourceTracker.Boxes {
+		groupResources.Boxes = append(groupResources.Boxes, box)
+	}
+	slices.SortFunc(groupResources.Boxes, func(a, b logic.BoxRef) int {
+		// NOTE: We intentionally sort in reverse order for apps so appID 0 is last since they can go anywhere
+		return cmp.Or(cmp.Compare(b.App, a.App), cmp.Compare(a.Name, b.Name))
+	})


same, though I now see you essentially do the deferral of the 0 app box refs I was thinking about.

maybe worth commenting to that effect in addBox? I don't know.

ledger/simulation/resources.go

jannotti · 2025-05-06T16:41:42Z

ledger/simulation/resources.go

+		})
+	}
+
+	// Then assign accounts because they have a lower limit than other resources


Ok, interesting. You're doing the single resources last. In some of my reasoning above, I assumed you'd do them first. I think I agree with this though. Your cross-ref types will "find" the best place to land based on the existing references and fields, right? So after that, it is indeed probably better to do these last, because the cross-ref types may make have already put them somewhere.

ledger/simulation/simulator.go

ledger/simulation/resources.go

joe-p · 2025-05-06T22:43:18Z

Some simplistic code comments, and some question about when you add certain things. I think the existing order may be perfectly fine, despite my early comments reservations. But take my confusion as a request to find a good place to describe the whole strategy all at once in a comment.

I will add this in a comment but the general philosophy is to go in order of resources with the most restrictions to least. This is why the order is

Txn-specific resources
Cross-ref resources (because they need two slots and we don't want single slot resources to potentially use up one of the available two-slot transactions)
Boxes, because they may require an app ref in addition to the box ref
Accounts because they have a lower limit than other resources
Standalone resources

I became worried that you are not accounting for old avm version apps, which can't use resource sharing. But I think maybe you're ok, because the response from simulate was taken that into account and places the necessary refs in the specific transactions if needed?

Yeah exactly. The logic for transaction-specific references exists upstream in simulate. By the time we reach the resource populator we know txn-specific resources and can handle those first

joe-p changed the title ~~Feat/populate_resources~~ resource population Jun 5, 2024

joe-p mentioned this pull request Jun 5, 2024

ResourcePopulator joe-p/go-algorand#2

Closed

ResourcePopulator

5ba0a9a

joe-p force-pushed the feat/populate_resources branch from 466fd50 to 5ba0a9a Compare June 5, 2024 23:06

joe-p changed the title ~~resource population~~ simulate: resource population Jun 5, 2024

TestPopulatorWithGlobalResources (arrays only)

028b8c4

joe-p added 14 commits June 6, 2024 07:30

test addBox

941ba6e

use ElementsMatch

a69091c

addHolding

d9e77ef

test variable renaming

4667647

appLocals

cbd1d8a

restore default limits

d79780f

fix rekey field

e6d59ba

populate with static properties and remove zeroAddr

54cd38c

empty boxes

673b03d

ensure duplicates are removed

ceddaa2

overflow txn resources

62213a0

use ConsensusParams

f616449

fix empty box count

6a4f1ee

golangci-lint

0524c3c

algorandskiy added Enhancement external contribution labels Jun 7, 2024

algorandskiy requested review from jasonpaulos and jannotti June 7, 2024 17:13

joe-p added 5 commits June 14, 2024 16:31

Merge branch 'master' into feat/populate_resources

73e5b4c

PopulateResourceArrays in simulate (untested)

6d8c161

populate from ResourceTracker

022c565

initial TestPopulateResources

93dd4e9

group sharing and no group sharing TestPopulateResources

3f78752

add TestGetTotalRefs

b7bda2e

gmalouf linked an issue Mar 11, 2025 that may be closed by this pull request

Given list of necessary resources, automatically populate reference arrays #5616

Open

jannotti reviewed Mar 14, 2025

View reviewed changes

jannotti reviewed Mar 19, 2025

View reviewed changes

daemon/algod/api/server/v2/utils.go Outdated Show resolved Hide resolved

daemon/algod/api/server/v2/utils.go Outdated Show resolved Hide resolved

daemon/algod/api/server/v2/utils.go Outdated Show resolved Hide resolved

joe-p and others added 7 commits March 24, 2025 08:05

fix extra resources check

fc8b37a

refactor: simplify sort functions

6353f3b

prefer Sort over SortFunc and use Cmp.or instead of Sprintf

refactor: move err declaration

d879600

refactor: use slice helper functions

0290ac3

feat: reverse sort box apps and add comments/logic for 0 index

6567445

chore: fix typos

32a51d6

Co-authored-by: John Jannotti <[email protected]>

chore: add additional comment explaining resource array map

eaef8ec

fix: add back errounesouly removed box population

0df1e0f

This was erroneously removed in 6353f3b

jannotti requested review from jannotti and jasonpaulos May 5, 2025 23:31

joe-p added 2 commits May 6, 2025 07:24

Merge branch 'master' into feat/populate_resources

757f205

fix: add reject version to app tests

a253291

jannotti reviewed May 6, 2025

View reviewed changes

ledger/simulation/simulator.go Outdated Show resolved Hide resolved

jannotti reviewed May 6, 2025

View reviewed changes

ledger/simulation/resources.go Outdated Show resolved Hide resolved

joe-p added 7 commits May 9, 2025 10:39

refactor: use two slices.Contains instead of Concat

a5ceee9

refactor: move addr checks before app checks

081e62e

refactor: remove redundant .String()

0d0d481

refactor: rm AppIndex instantiation

dcd2d59

refactor: use slices.Sorted and maps.Keys for assigning values

0f528de

refactor: rm unused anon struct

74c9b8f

refactor: use second iterator value

489d950

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simulate: resource population #6015

simulate: resource population #6015

joe-p commented Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 6, 2024 •

edited

Loading

joe-p commented Mar 8, 2025 •

edited

Loading

jannotti left a comment

jannotti Mar 14, 2025 •

edited

Loading

joe-p May 5, 2025

jannotti left a comment

joe-p commented May 5, 2025 •

edited

Loading

jannotti commented May 5, 2025

jannotti left a comment

jannotti May 6, 2025

jannotti May 6, 2025

jannotti May 6, 2025

jannotti May 6, 2025

joe-p commented May 6, 2025

simulate: resource population #6015

Are you sure you want to change the base?

simulate: resource population #6015

Conversation

joe-p commented Jun 5, 2024 • edited Loading

Summary

Test Plan

codecov bot commented Jun 6, 2024 • edited Loading

Codecov Report

joe-p commented Mar 8, 2025 • edited Loading

jannotti left a comment

Choose a reason for hiding this comment

jannotti Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

joe-p May 5, 2025

Choose a reason for hiding this comment

jannotti left a comment

Choose a reason for hiding this comment

joe-p commented May 5, 2025 • edited Loading

jannotti commented May 5, 2025

jannotti left a comment

Choose a reason for hiding this comment

jannotti May 6, 2025

Choose a reason for hiding this comment

jannotti May 6, 2025

Choose a reason for hiding this comment

jannotti May 6, 2025

Choose a reason for hiding this comment

jannotti May 6, 2025

Choose a reason for hiding this comment

joe-p commented May 6, 2025

joe-p commented Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 6, 2024 •

edited

Loading

joe-p commented Mar 8, 2025 •

edited

Loading

jannotti Mar 14, 2025 •

edited

Loading

joe-p commented May 5, 2025 •

edited

Loading