Skip to content

Commit 572bac3

Browse files
committed
feat(test): add enhanced QA content generator for forensics suite
Add comprehensive Python script to generate QA test samples for OSVM CLI test suite with focus on forensics, debugging, and diverse tool usage. The script generates 900+ lines of structured test cases covering: - Account state forensics and suspicious activity detection - Transaction analysis with statistical outlier detection - Multiple difficulty levels (basic to expert) - 70+ unique tools from standard library - Realistic Solana blockchain scenarios with error handling This enhances test coverage for OVSM syntax validation and provides standardized forensics scenarios for CLI testing and validation.
1 parent 1bf0e1f commit 572bac3

File tree

115 files changed

+9393
-6006
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

115 files changed

+9393
-6006
lines changed

scripts/enhance_forensics.py

Lines changed: 926 additions & 0 deletions
Large diffs are not rendered by default.

scripts/generate_qa_complete.rs

Lines changed: 611 additions & 0 deletions
Large diffs are not rendered by default.
File renamed without changes.
File renamed without changes.
Lines changed: 373 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,373 @@
1+
# Transaction Analysis - Advanced Level
2+
3+
## Q1: "Analyze the Cross-Program Invocation (CPI) chain for transaction 1abc...def"
4+
5+
**Expected Plan:**
6+
7+
[TIME: ~5s] [COST: free] [CONFIDENCE: 95%]
8+
9+
**Available Tools:**
10+
From Standard Library:
11+
- getTransaction (Solana RPC)
12+
- MAP, FILTER, FLATTEN (Data Processing)
13+
- COUNT (Collection operations)
14+
15+
**Main Branch:**
16+
$signature = "1abc...def"
17+
18+
TRY:
19+
$tx = getTransaction(signature: $signature)
20+
CATCH FATAL:
21+
RETURN ERROR(message: "Transaction not found")
22+
23+
// Extract inner instructions (CPIs)
24+
$inner_instructions = $tx.meta.innerInstructions
25+
$cpi_count = COUNT(collection: $inner_instructions)
26+
27+
// Map CPI chain
28+
$cpi_chain = MAP(
29+
collection: $inner_instructions,
30+
fn: inner => {
31+
program: $tx.transaction.message.accountKeys[inner.index],
32+
instructions: inner.instructions,
33+
depth: calculateCPIDepth(inner)
34+
}
35+
)
36+
37+
// Calculate metrics
38+
$total_cpi_calls = SUM(
39+
values: MAP(collection: $cpi_chain, fn: item => COUNT(collection: item.instructions))
40+
)
41+
42+
$max_depth = MAX(values: MAP(collection: $cpi_chain, fn: item => item.depth))
43+
44+
**Decision Point:** Analyze CPI complexity
45+
BRANCH A ($max_depth > 4):
46+
$complexity = "very_high"
47+
$warning = "Deep CPI nesting detected - potential security risk"
48+
BRANCH B ($max_depth > 2):
49+
$complexity = "high"
50+
$warning = "Moderate CPI nesting"
51+
BRANCH C ($max_depth <= 2):
52+
$complexity = "normal"
53+
$warning = "Standard CPI usage"
54+
55+
**Action:**
56+
RETURN {
57+
signature: $signature,
58+
cpi_count: $cpi_count,
59+
total_cpi_calls: $total_cpi_calls,
60+
max_depth: $max_depth,
61+
cpi_chain: $cpi_chain,
62+
complexity: $complexity,
63+
warning: $warning,
64+
confidence: 95
65+
}
66+
67+
---
68+
69+
## Q2: "Detect if transaction contains priority fees and calculate optimal fee"
70+
71+
**Expected Plan:**
72+
73+
[TIME: ~4s] [COST: free] [CONFIDENCE: 90%]
74+
75+
**Available Tools:**
76+
From Standard Library:
77+
- getTransaction (Solana RPC)
78+
- getRecentPrioritizationFees (Solana RPC)
79+
- extractPriorityFee (Solana Utilities)
80+
- MEAN, PERCENTILE (Statistical)
81+
82+
**Main Branch:**
83+
$signature = "2def...ghi"
84+
85+
PARALLEL {
86+
$tx = getTransaction(signature: $signature)
87+
$recent_fees = getRecentPrioritizationFees()
88+
}
89+
WAIT_ALL
90+
91+
// Extract priority fee from transaction
92+
$priority_fee = extractPriorityFee(transaction: $tx)
93+
94+
// Calculate optimal fee from recent data
95+
$fee_values = MAP(collection: $recent_fees, fn: fee => fee.prioritizationFee)
96+
$avg_fee = MEAN(data: $fee_values)
97+
$p50_fee = PERCENTILE(data: $fee_values, percentile: 50)
98+
$p75_fee = PERCENTILE(data: $fee_values, percentile: 75)
99+
$p90_fee = PERCENTILE(data: $fee_values, percentile: 90)
100+
101+
**Decision Point:** Analyze fee efficiency
102+
BRANCH A ($priority_fee > $p90_fee):
103+
$efficiency = "overpaid"
104+
$recommendation = "Reduce priority fee to p75 level"
105+
BRANCH B ($priority_fee >= $p50_fee AND $priority_fee <= $p75_fee):
106+
$efficiency = "optimal"
107+
$recommendation = "Fee is in optimal range"
108+
BRANCH C ($priority_fee < $p50_fee):
109+
$efficiency = "underpaid"
110+
$recommendation = "Consider increasing for faster confirmation"
111+
BRANCH D ($priority_fee == 0):
112+
$efficiency = "none"
113+
$recommendation = "No priority fee - may face delays"
114+
115+
**Action:**
116+
RETURN {
117+
signature: $signature,
118+
priority_fee_paid: $priority_fee,
119+
average_market_fee: $avg_fee,
120+
p50: $p50_fee,
121+
p75: $p75_fee,
122+
p90: $p90_fee,
123+
efficiency: $efficiency,
124+
recommendation: $recommendation,
125+
potential_savings: MAX_OF($priority_fee - $p75_fee, 0),
126+
confidence: 90
127+
}
128+
129+
---
130+
131+
## Q3: "Reconstruct the execution flow and identify failed instructions"
132+
133+
**Expected Plan:**
134+
135+
[TIME: ~6s] [COST: free] [CONFIDENCE: 95%]
136+
137+
**Available Tools:**
138+
From Standard Library:
139+
- getTransaction (Solana RPC)
140+
- MAP, FILTER (Data Processing)
141+
- FIND (Collection operations)
142+
143+
**Main Branch:**
144+
$signature = "3ghi...jkl"
145+
146+
TRY:
147+
$tx = getTransaction(signature: $signature)
148+
CATCH FATAL:
149+
RETURN ERROR(message: "Transaction not found")
150+
151+
// Check if transaction failed
152+
$error = $tx.meta.err
153+
154+
**Decision Point:** Transaction status
155+
BRANCH A ($error != null):
156+
// Transaction failed - identify failing instruction
157+
$log_messages = $tx.meta.logMessages
158+
159+
// Parse logs to find error
160+
$error_logs = FILTER(
161+
collection: $log_messages,
162+
predicate: msg => CONTAINS(msg, "failed") OR CONTAINS(msg, "Error")
163+
)
164+
165+
// Find instruction index that failed
166+
$failed_instruction_match = FIND(
167+
collection: $log_messages,
168+
predicate: msg => CONTAINS(msg, "Program") AND CONTAINS(msg, "failed")
169+
)
170+
171+
// Extract program that failed
172+
$failed_program = extractProgramFromLog(log: $failed_instruction_match)
173+
174+
// Map execution flow up to failure
175+
$execution_flow = []
176+
FOR $msg IN $log_messages:
177+
IF CONTAINS($msg, "Program") AND CONTAINS($msg, "invoke"):
178+
$execution_flow = APPEND(array: $execution_flow, item: {
179+
action: "invoke",
180+
message: $msg,
181+
success: NOT CONTAINS($msg, "failed")
182+
})
183+
BREAK IF CONTAINS($msg, "failed")
184+
185+
$status = "failed"
186+
$failure_point = COUNT(collection: $execution_flow)
187+
188+
BRANCH B ($error == null):
189+
// Transaction succeeded
190+
$log_messages = $tx.meta.logMessages
191+
$execution_flow = MAP(
192+
collection: FILTER(
193+
collection: $log_messages,
194+
predicate: msg => CONTAINS(msg, "Program") AND CONTAINS(msg, "invoke")
195+
),
196+
fn: msg => {
197+
action: "invoke",
198+
message: msg,
199+
success: true
200+
}
201+
)
202+
203+
$status = "success"
204+
$failed_program = null
205+
$failure_point = null
206+
207+
**Action:**
208+
RETURN {
209+
signature: $signature,
210+
status: $status,
211+
error: $error,
212+
execution_flow: $execution_flow,
213+
total_instructions: COUNT(collection: $tx.transaction.message.instructions),
214+
executed_instructions: COUNT(collection: $execution_flow),
215+
failed_program: $failed_program,
216+
failure_point: $failure_point,
217+
confidence: 95
218+
}
219+
220+
---
221+
222+
## Q4: "Calculate the effective cost per compute unit and compare to network average"
223+
224+
**Expected Plan:**
225+
226+
[TIME: ~5s] [COST: free] [CONFIDENCE: 88%]
227+
228+
**Available Tools:**
229+
From Standard Library:
230+
- getTransaction (Solana RPC)
231+
- getRecentPrioritizationFees (Solana RPC)
232+
- MEAN, STDDEV (Statistical)
233+
234+
**Main Branch:**
235+
$signature = "4jkl...mno"
236+
237+
PARALLEL {
238+
$tx = getTransaction(signature: $signature)
239+
$recent_fees = getRecentPrioritizationFees()
240+
}
241+
WAIT_ALL
242+
243+
// Calculate this transaction's metrics
244+
$total_fee = $tx.meta.fee
245+
$compute_units = $tx.meta.computeUnitsConsumed
246+
$priority_fee = extractPriorityFee(transaction: $tx)
247+
248+
GUARD $compute_units > 0 ELSE
249+
RETURN ERROR(message: "Compute units not available")
250+
251+
$cost_per_cu = $total_fee / $compute_units
252+
$priority_cost_per_cu = $priority_fee / $compute_units
253+
254+
// Calculate network averages
255+
$network_fees = MAP(collection: $recent_fees, fn: fee => fee.prioritizationFee)
256+
$avg_network_fee = MEAN(data: $network_fees)
257+
$stddev_fee = STDDEV(data: $network_fees)
258+
259+
// Estimate average compute units (use typical value)
260+
$typical_cu = 200000
261+
$estimated_network_cost_per_cu = $avg_network_fee / $typical_cu
262+
263+
**Decision Point:** Compare to network
264+
BRANCH A ($cost_per_cu > $estimated_network_cost_per_cu * 1.5):
265+
$assessment = "expensive"
266+
$note = "Transaction is significantly more expensive than network average"
267+
BRANCH B ($cost_per_cu > $estimated_network_cost_per_cu):
268+
$assessment = "above_average"
269+
$note = "Transaction costs more than network average"
270+
BRANCH C ($cost_per_cu <= $estimated_network_cost_per_cu):
271+
$assessment = "efficient"
272+
$note = "Transaction is cost-efficient"
273+
274+
**Action:**
275+
RETURN {
276+
signature: $signature,
277+
total_fee_lamports: $total_fee,
278+
compute_units: $compute_units,
279+
cost_per_cu: $cost_per_cu,
280+
priority_cost_per_cu: $priority_cost_per_cu,
281+
network_avg_cost_per_cu: $estimated_network_cost_per_cu,
282+
assessment: $assessment,
283+
note: $note,
284+
confidence: 88,
285+
caveats: ["Network average estimated from recent fees and typical compute units"]
286+
}
287+
288+
---
289+
290+
## Q5: "Analyze account write locks and detect potential race conditions"
291+
292+
**Expected Plan:**
293+
294+
[TIME: ~4s] [COST: free] [CONFIDENCE: 92%]
295+
296+
**Available Tools:**
297+
From Standard Library:
298+
- getTransaction (Solana RPC)
299+
- FILTER, MAP (Data Processing)
300+
- UNIQUE (Collection operations)
301+
302+
**Main Branch:**
303+
$signature = "5mno...pqr"
304+
305+
TRY:
306+
$tx = getTransaction(signature: $signature)
307+
CATCH FATAL:
308+
RETURN ERROR(message: "Transaction not found")
309+
310+
$message = $tx.transaction.message
311+
$account_keys = $message.accountKeys
312+
313+
// Extract writable accounts
314+
$num_required_sigs = $message.header.numRequiredSignatures
315+
$num_readonly_signed = $message.header.numReadonlySignedAccounts
316+
$num_readonly_unsigned = $message.header.numReadonlyUnsignedAccounts
317+
318+
// Writable signed accounts
319+
$writable_signed = []
320+
FOR $i IN 0..($num_required_sigs - $num_readonly_signed):
321+
$writable_signed = APPEND(array: $writable_signed, item: $account_keys[$i])
322+
323+
// Writable unsigned accounts
324+
$start_unsigned = $num_required_sigs
325+
$end_unsigned = COUNT(collection: $account_keys) - $num_readonly_unsigned
326+
$writable_unsigned = []
327+
FOR $i IN $start_unsigned..$end_unsigned:
328+
$writable_unsigned = APPEND(array: $writable_unsigned, item: $account_keys[$i])
329+
330+
$all_writable = FLATTEN(collection: [$writable_signed, $writable_unsigned])
331+
$unique_writable = UNIQUE(collection: $all_writable)
332+
333+
// Analyze write patterns
334+
$write_lock_count = COUNT(collection: $unique_writable)
335+
336+
// Check for common patterns that indicate race conditions
337+
$has_token_accounts = ANY(
338+
collection: $unique_writable,
339+
predicate: acc => isTokenAccount(acc)
340+
)
341+
342+
$has_pdas = ANY(
343+
collection: $unique_writable,
344+
predicate: acc => isPDA(acc)
345+
)
346+
347+
**Decision Point:** Assess race condition risk
348+
BRANCH A ($write_lock_count > 10):
349+
$risk_level = "high"
350+
$warning = "Many write locks - high contention risk"
351+
BRANCH B ($write_lock_count > 5 AND $has_token_accounts):
352+
$risk_level = "medium"
353+
$warning = "Multiple token account writes - potential race condition"
354+
BRANCH C ($has_pdas AND $write_lock_count > 3):
355+
$risk_level = "medium"
356+
$warning = "PDA writes with multiple accounts - monitor for conflicts"
357+
BRANCH D (true):
358+
$risk_level = "low"
359+
$warning = "Normal write lock pattern"
360+
361+
**Action:**
362+
RETURN {
363+
signature: $signature,
364+
writable_accounts: $unique_writable,
365+
write_lock_count: $write_lock_count,
366+
writable_signed_count: COUNT(collection: $writable_signed),
367+
writable_unsigned_count: COUNT(collection: $writable_unsigned),
368+
has_token_accounts: $has_token_accounts,
369+
has_pdas: $has_pdas,
370+
risk_level: $risk_level,
371+
warning: $warning,
372+
confidence: 92
373+
}

0 commit comments

Comments
 (0)