Draft
Conversation
kramvan1
reviewed
May 6, 2025
rules/options.go
Outdated
|
|
||
| // HighPriority places priority on fields | ||
| // that are associated with a rule during | ||
| // a crawler run |
Collaborator
There was a problem hiding this comment.
Need to mention that normal watcher processing is also done for these rules.
Do we want to consider an uint here, and let the use specific a specific priority order? 1-n, 1 being the highest? Would allow them to "group" priority 1 rules or which ever way they want. getPrioritizedPrefixes could just do a value sort. Rules that overlap, highest priority wins.
kramvan1
reviewed
May 6, 2025
kramvan1
reviewed
May 7, 2025
kramvan1
reviewed
May 7, 2025
kramvan1
reviewed
May 7, 2025
kramvan1
reviewed
May 9, 2025
kramvan1
reviewed
May 9, 2025
rules/rule_manager.go
Outdated
| } | ||
| } | ||
| return out | ||
| return out, sortRulesByPriority(prioritized) |
Collaborator
There was a problem hiding this comment.
Humm, I worry about the overhead of doing this for every call when the result is always the same. Every field value, which can be large, 20 million each crawl bootstrap.
Maybe we need some debug logging or enable Go profiling to see how much of a hit this is going to be.
kramvan1
reviewed
May 9, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue https://github.ibm.com/alchemy-containers/armada-ballast/issues/3593
ENSURE THE FOLLOWING ARE MET:
Below is a brief summary of PR requirements (full list).
By opening this PR for review, the author has agreed that these criteria must be met.
By approving this PR, the reviewers have also agreed these criteria have been met and it is ready to be merged.