probe: probes should follow `soft_probe_prompt_cap`

[leondz]

suffix.GCGCached always runs all stored cached prompts

run.soft_probe_prompt_cap sets a cap on how many distinct prompts to offer per probe

suffix.GCGCached should not issue more prompts than run.soft_probe_prompt_cap

To see the issue, try a config like:

{
	"plugins": {
		"probe_spec": "suffix"
	},
	"run": {
		"soft_probe_prompt_cap": 10,
		"generations": 1,
	}
}

Note that >10 prompts are issued

When addressing this, use a mechanism for randomly selecting prompts like that in probes.base._prune_data()

[jmartin-tech]

There are a number for probes that still ignore soft_probe_prompt_cap for various reasons. In most cases either they have a small number for default prompts. I am all for standardizing all probes on limited caps. Should we expand this issue with a comprehensive list of current probes to be limited?

[jmartin-tech]

This seems connected to #1546

[leondz]

Should we expand this issue with a comprehensive list of current probes to be limited?

to limit, from default active set:

• ansiescape.AnsiEscaped
• ansiescape.AnsiRaw
• atkgen.Tox (?)
• dan.Ablation_Dan_11_0
• dan.AutoDANCached
• divergence.Repeat
• exploitation.JinjaTemplatePythonInjection
• exploitation.SQLInjectionEcho
• goodside.Tag
• goodside.WhoIsRiley
• grandma.Slurs
• grandma.Substances
• grandma.Win10
• grandma.Win11
• latentinjection.LatentWhois
• lmrc.Bullying
• lmrc.Deadnaming
• lmrc.Sexualisation
• malwaregen.Evasion
• malwaregen.Payload
• malwaregen.SubFunctions
• malwaregen.TopLevel
• misleading.FalseAssertion
• realtoxicityprompts.RTPBlank
• snowball.GraphConnectivity
• suffix.GCGCached
• tap.TAPCached
• topic.WordnetControversial
• web_injection.ColabAIDataLeakage
• web_injection.MarkdownImageExfil
• web_injection.MarkdownXSS
• web_injection.PlaygroundMarkdownExfil
• web_injection.StringAssemblyDataExfil

if implemented with comprehensive intent, the param should probably be renamed to probe_prompt_cap