Skip to content

Commit b10bfb0

Browse files
Kyu-seokclaude
andauthored
Test: every Java Driver compiles with its Starter Code (#414) (#417)
* test: every Java Driver compiles with its Starter Code (#414) JavaDriversCompileTest assembles each problem's Java Program through ProblemAssets with its own Starter Code and compiles it in memory at --release 13, as Judge0 builds it, with no Judge0 and no database. A failure names the slug, the line in the assembled Program and javac's first error. A starter is a stub, so "missing return statement" on the starter's lines is passed over; javac then runs with should-stop.ifError=FLOW, since by default it stops at the stub's error and never reports a Driver's own flow errors. Fixture Drivers under problem-assets/java-drivers show each breakage failing: a call to a missing method (kth-top-score-stream before #336), a Library above the starter's import (fizz-buzz), a missing return on a Driver line, another error on a starter line, and a Driver with no starter. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * JavaDriversCompileTest: fail when the starter isn't found (#414) A starter the assembly had changed would give a line range from line 1 and pass over errors it shouldn't. CLAUDE.md names the pass-over, and docs/agents/common.md stays a list of main classes. Review follow-up. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
1 parent d98c985 commit b10bfb0

14 files changed

Lines changed: 271 additions & 1 deletion

File tree

‎CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Root `settings.gradle` includes three modules: `common`, `backend`, `worker`. Th
4646
`scripts/check-doc-references.py` (pre-commit and CI) fails when CLAUDE.md, a skill, an agent or `docs/agents/` names a repo path that doesn't exist, or anything cites a glossary term (`… (GLOSSARY.md)`) the glossary lacks, or the add-walkthrough skill's `views.md` has a `## ` heading that carries a ticket number or repeats another: the usual cause of a dangling pointer is a branch citing what another unmerged branch adds.
4747

4848
### Restating a problem
49-
Rewording a problem while its Test Cases stay (Slug, title, description, Starter Code names) is a Restatement: follow `.claude/skills/restate-problem/SKILL.md`. `ProblemSlugsTest` (backend) runs every migration and fails when a Slug and its `drivers/`/`starters/` directories disagree; `MigrationSlugReferencesTest` fails when a migration's `slug = …` names a slug that doesn't exist when it runs. `StatementsMatchTestCasesTest`'s exemption sets (its class comment lists them) are the known statement defects: a fix removes its slug, and the test fails on a listed slug that passes; every run writes one report per check, each problem's mismatch lines with listed ones marked, under `backend/build/reports/statements/`.
49+
Rewording a problem while its Test Cases stay (Slug, title, description, Starter Code names) is a Restatement: follow `.claude/skills/restate-problem/SKILL.md`. `ProblemSlugsTest` (backend) runs every migration and fails when a Slug and its `drivers/`/`starters/` directories disagree; `MigrationSlugReferencesTest` fails when a migration's `slug = …` names a slug that doesn't exist when it runs. `JavaDriversCompileTest` (common) compiles every Java Driver with its Starter Code at Judge0's `--release 13` (a stub's missing return aside), naming the slug and javac's first error. `StatementsMatchTestCasesTest`'s exemption sets (its class comment lists them) are the known statement defects: a fix removes its slug, and the test fails on a listed slug that passes; every run writes one report per check, each problem's mismatch lines with listed ones marked, under `backend/build/reports/statements/`.
5050

5151
### Correcting a problem's Test Cases
5252
`.claude/skills/correct-test-cases/SKILL.md`. `ProblemGuaranteesTest` and `TestCasePositionsTest` (backend) check every migrated case; tests that read migrated rows use `MigratedDatabase`, since H2 keeps `E'…\n…'` escapes as a backslash and an `n`.
Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
package com.codebite.judge.program;
2+
3+
import com.codebite.judge.Language;
4+
import org.junit.jupiter.api.AfterAll;
5+
import org.junit.jupiter.api.Test;
6+
import org.junit.jupiter.params.ParameterizedTest;
7+
import org.junit.jupiter.params.provider.CsvSource;
8+
9+
import javax.tools.Diagnostic;
10+
import javax.tools.DiagnosticCollector;
11+
import javax.tools.FileObject;
12+
import javax.tools.ForwardingJavaFileManager;
13+
import javax.tools.JavaCompiler;
14+
import javax.tools.JavaFileManager;
15+
import javax.tools.JavaFileObject;
16+
import javax.tools.SimpleJavaFileObject;
17+
import javax.tools.ToolProvider;
18+
import java.io.IOException;
19+
import java.io.OutputStream;
20+
import java.net.URI;
21+
import java.util.ArrayList;
22+
import java.util.List;
23+
import java.util.Locale;
24+
import java.util.Optional;
25+
26+
import static org.assertj.core.api.Assertions.assertThat;
27+
28+
/**
29+
* Every Java Driver compiles with its own untouched Starter Code in place of {@code {USER_CODE}},
30+
* assembled as the backend assembles it and built as Judge0 builds it ({@code --release 13}), in
31+
* memory. A Driver that doesn't is a problem no Java submission can pass: one that calls a method
32+
* its types lack, or one that puts a Library class above a starter that opens with an
33+
* {@code import} (#336).
34+
*
35+
* <p>A starter is a stub, so one javac error is expected and passed over: "missing return
36+
* statement" on the starter's own lines. The same error on a Driver line fails.
37+
*/
38+
class JavaDriversCompileTest {
39+
40+
private static final String MISSING_RETURN = "compiler.err.missing.ret.stmt";
41+
42+
/**
43+
* Judge0's release, and every class checked through flow analysis whatever errors came before:
44+
* left to itself javac stops at the stub's missing return and never reports a Driver's own
45+
* missing return, unassigned variable or uncaught checked exception.
46+
*/
47+
private static final List<String> JAVAC_OPTIONS =
48+
List.of("--release", "13", "-proc:none", "-XDshould-stop.ifError=FLOW");
49+
50+
private static final JavaCompiler COMPILER = ToolProvider.getSystemJavaCompiler();
51+
private static final JavaFileManager CLASS_FILES_DISCARDED = COMPILER == null ? null : discardingClassFiles();
52+
53+
@AfterAll
54+
static void closeFileManager() throws IOException {
55+
if (CLASS_FILES_DISCARDED != null) {
56+
CLASS_FILES_DISCARDED.close();
57+
}
58+
}
59+
60+
@Test
61+
void everyJavaDriverCompilesWithItsStarterCode() {
62+
ProblemAssets assets = new ProblemAssets();
63+
List<String> broken = new ArrayList<>();
64+
for (String slug : assets.slugsWithDriver(Language.JAVA)) {
65+
firstError(assets, slug).ifPresent(broken::add);
66+
}
67+
68+
assertThat(assets.slugsWithDriver(Language.JAVA)).hasSizeGreaterThanOrEqualTo(200);
69+
assertThat(broken).as("Java Drivers that do not compile with their Starter Code").isEmpty();
70+
}
71+
72+
// --- What the check catches, on Drivers made for it ---
73+
74+
private static final ProblemAssets FIXTURES = new ProblemAssets("classpath:problem-assets/java-drivers/");
75+
76+
@Test
77+
void aStubStarterInAWorkingDriverPasses() {
78+
assertThat(firstError(FIXTURES, "stub-starter")).isEmpty();
79+
}
80+
81+
@ParameterizedTest
82+
@CsvSource(delimiter = '|', value = {
83+
// kth-top-score-stream before #336: no submission could compile
84+
"driver-calls-a-missing-method | line 22: cannot find symbol | ops.record(\"names\");",
85+
// fizz-buzz before #336: the starter's import lands below the Library's class
86+
"library-above-the-starters-import | line 10: class, interface, or enum expected | import java.util.List;",
87+
// just outside what is passed over: a missing return on a Driver line
88+
"driver-method-returns-nothing | line 22: missing return statement | }",
89+
// and an error on a starter line that is not a missing return
90+
"starter-names-a-type-nothing-imports | line 2: cannot find symbol | public List<String> names(int n) {",
91+
})
92+
void aDriverThatCannotCompileIsNamedWithJavacsFirstError(String slug, String error, String sourceLine) {
93+
assertThat(firstError(FIXTURES, slug)).get().asString()
94+
.startsWith(slug + ": ")
95+
.contains(error)
96+
.endsWith("| " + sourceLine);
97+
}
98+
99+
@Test
100+
void aDriverWithoutAStarterIsNamed() {
101+
assertThat(firstError(FIXTURES, "no-starter")).contains("no-starter: no Java Starter Code");
102+
}
103+
104+
/**
105+
* {@code <slug>: line <n>: <javac's message> | <that line>} for the first error javac reports
106+
* in the slug's Driver with its Starter Code, the line counted in the assembled Program (its
107+
* Libraries are expanded, so it is not the Driver file's line); empty when it compiles.
108+
*/
109+
private static Optional<String> firstError(ProblemAssets assets, String slug) {
110+
assertThat(COMPILER).as("javac: these tests need a JDK, not a JRE").isNotNull();
111+
String starter = assets.starters(slug).get(Language.JAVA);
112+
if (starter == null) {
113+
return Optional.of(slug + ": no Java Starter Code");
114+
}
115+
String source = assets.assemble(slug, Language.JAVA, starter).source();
116+
int starterAt = source.indexOf(starter);
117+
assertThat(starterAt).as(slug + ": its Starter Code, verbatim, in the assembled Program").isNotNegative();
118+
int starterFirstLine = Program.newlines(source, starterAt) + 1;
119+
int starterLastLine = starterFirstLine + Program.newlines(starter, starter.length());
120+
121+
DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<>();
122+
COMPILER.getTask(null, CLASS_FILES_DISCARDED, diagnostics,
123+
JAVAC_OPTIONS, null, List.of(new MainSource(source))).call();
124+
125+
return diagnostics.getDiagnostics().stream()
126+
.filter(d -> d.getKind() == Diagnostic.Kind.ERROR)
127+
.filter(d -> !(MISSING_RETURN.equals(d.getCode())
128+
&& d.getLineNumber() >= starterFirstLine && d.getLineNumber() <= starterLastLine))
129+
.findFirst()
130+
.map(d -> slug + ": line " + d.getLineNumber() + ": " + d.getMessage(Locale.ROOT)
131+
+ " | " + source.lines().skip(d.getLineNumber() - 1).findFirst().orElse("").strip());
132+
}
133+
134+
/** The assembled Program as Judge0 names it, {@code Main.java}. */
135+
private static final class MainSource extends SimpleJavaFileObject {
136+
private final String source;
137+
138+
MainSource(String source) {
139+
super(URI.create("string:///Main.java"), Kind.SOURCE);
140+
this.source = source;
141+
}
142+
143+
@Override
144+
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
145+
return source;
146+
}
147+
}
148+
149+
/** One file manager for every compilation, so the JDK's classes are opened once; no class file is kept. */
150+
private static JavaFileManager discardingClassFiles() {
151+
return new ForwardingJavaFileManager<>(COMPILER.getStandardFileManager(null, null, null)) {
152+
@Override
153+
public JavaFileObject getJavaFileForOutput(Location location, String className,
154+
JavaFileObject.Kind kind, FileObject sibling) {
155+
return new SimpleJavaFileObject(URI.create("discarded:///" + className.replace('.', '/') + kind.extension), kind) {
156+
@Override
157+
public OutputStream openOutputStream() {
158+
return OutputStream.nullOutputStream();
159+
}
160+
};
161+
}
162+
};
163+
}
164+
}
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
import java.util.*;
2+
3+
{USER_CODE}
4+
5+
// {{LIB:Printer}}
6+
7+
public class Main {
8+
public static void main(String[] args) {
9+
List<String> ops = new ArrayList<>();
10+
ops.record("names");
11+
System.out.println(Printer.list(new Solution().names(ops.size())));
12+
}
13+
}
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
import java.util.*;
2+
3+
{USER_CODE}
4+
5+
// {{LIB:Printer}}
6+
7+
public class Main {
8+
static int readCount() {
9+
new Scanner(System.in).nextInt();
10+
}
11+
12+
public static void main(String[] args) {
13+
System.out.println(Printer.list(new Solution().names(readCount())));
14+
}
15+
}
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
class Printer {
2+
static String list(java.util.List<String> list) {
3+
return String.join(",", list);
4+
}
5+
}
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import java.util.*;
2+
3+
// {{LIB:Printer}}
4+
5+
{USER_CODE}
6+
7+
public class Main {
8+
public static void main(String[] args) {
9+
int n = new Scanner(System.in).nextInt();
10+
System.out.println(Printer.list(new Solution().names(n)));
11+
}
12+
}
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{USER_CODE}
2+
3+
public class Main {
4+
public static void main(String[] args) {
5+
System.out.println(new Solution());
6+
}
7+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{USER_CODE}
2+
3+
// {{LIB:Printer}}
4+
5+
public class Main {
6+
public static void main(String[] args) {
7+
System.out.println(Printer.list(new Solution().names(3)));
8+
}
9+
}
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import java.util.*;
2+
3+
{USER_CODE}
4+
5+
// {{LIB:Printer}}
6+
7+
public class Main {
8+
public static void main(String[] args) {
9+
int n = new Scanner(System.in).nextInt();
10+
System.out.println(Printer.list(new Solution().names(n)));
11+
}
12+
}
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import java.util.List;
2+
3+
class Solution {
4+
public List<String> names(int n) {
5+
// Write your solution here
6+
}
7+
}

0 commit comments

Comments
 (0)