[Discussion] Choosing build options

[dector]

@exercism/kotlin

Currently we are using Junit4 tests with @Ignore annotation that students need to remove manually.

Q: What can be used instead?
A: Fail-fast strategy: stop after first failed test - no more manual work for students.

Q: What other options for testing tooling we have?
A:

• JUnit 5 - newer Junit.
• Kotlintest/Spek - better DSL. One step towards data-driven tests.
• Custom test-runner. Can also be an option for that simple tests we have.

[eparovyshnaya]

Most of our tests are data driven. But.

For some reason it is customary to deliver all tests but one in @Ignored state. If we use data packs, a student will not be able to un@Ignore a single row in a data set, but will be forced to jump several stairs in a ladder in one fell swoop.

To be clear - I'm totally OK with bulk un@Ignore as far as we do not expect our student to be a newbie in programming.

Kotlintest/Spek - !

It's definitely worth a shot to involve a framework from Kotlin ecosystem for overwhelming student's experience, especially with a chance to expose a pleasure of inventing a dsl based on Kotlin rich facilities. There are no extraordinary rich cases in these frameworks, but there is a flavour of Kotlin power for DSLs.

A word in favour of JUnit5

It look like due to our build framework pecularities, it's the only framework that allows smooth migration: fix dependencies to junit5, start with all vitage tests -> migrate tests to Jupiter in a regular pace -> exterminate vintage dependencies.

Fundamental framework switch will require build system evolution, which is the thing I'm Ok with, of course :)

[mikegehard]

All great points.

I'd like to get us all on the same page with respect to goals for the test suites, independent of tech/approach used. Once we have that then we can collaborate on a solution.

My understanding, which may be outdated, was that the @Ignore and the ordering was used to guide students towards the solution in a gentle manner. They one had to deal with one red test and could begin to get the feel of the flow of TDD, where you one have one red test as a time.

Question #1: Do we feel that this is still a goal of the test suite?

Question #2: If not, what guiding principles / goals should we use to guide our decision?

Some thoughts on the tech:
I like keeping them in JUnit style because it is a lower barrier to entry for someone coming from the Java world. My hypothesis is that we have a lot of people switching over but I don't have any data what so ever to back that up. Also as @eparovyshnaya pointed out this is less work to transition.

If we don't believe that easing the transition from Java -> Kotlin is a goal, then I like the idea of switching to Kotlintest, Spek or Aspen (https://github.com/dam5s/aspen) (disclaimer, I work with the person who wrote Aspen).

My preference would be Kotlintest because that seems to be most widespread? Again, no data to back that up.

Thoughts?

[dector]

Thanks all for your ideas!

My personal preference is kotlintest and it's very powerful, agile, has awesome DSL, will support KMPP (in new version) and based on JUnit testing platform.

I think we should provide:

• tests from specifications;
• additional tests (I guess we want to move towards semi-automated solution verification).

I think that students should not edit tests unless they want to add their own test-cases. We can achieve this with fixed test order and fail-fast strategy.

---

@eparovyshnaya

To be clear - I'm totally OK with bulk un@Ignore as far as we do not expect our student to be a newbie in programming.

I'm not, tbh. 😸 I think we should avoid adding ignored tests. :)

It look like due to our build framework pecularities, it's the only framework that allows smooth migration: fix dependencies to junit5, start with all vitage tests -> migrate tests to Jupiter in a regular pace -> exterminate vintage dependencies.

Junit5 is powerful because it consists of few components, including JUnit Platform that allows us to use non-JUnit api/runners. Definitely great choice, I think.

---

@mikegehard

Question #1: Do we feel that this is still a goal of the test suite?

"Step-by-step" guiding definitely sounds like something that is nice to have. My answer is: "no for ignored tests, yes for fail-fast strategy".

Question #2: If not, what guiding principles / goals should we use to guide our decision?

Very good question.

When I started mentoring I thought that it would be nice if we can ask a student about his background and reasons why he's learning new language. I would be easier to choose right guiding strategy for her/him.

I like keeping them in JUnit style because it is a lower barrier to entry for someone coming from the Java world. My hypothesis is that we have a lot of people switching over

Definitely a good point. My guess is that people that are migrating from Java are not newbies in programming, our tests are very simple (as they should be simple) and data-driven. So I don't think that it'll be very difficult for them to switch JUnit-classic style to something with more DSL, especially if all test data will be keep in one place.

From the other side, there are lots of people that are new in programming (or they are switching from non-JVM world). I think that Kotlin DSL will work for them as well. Technically they should concentrate on finding solutions of the task in most idiomatic way to get a feel of the language philosophy. KMPP is becoming more and more a thing now and Kotlin has it's own ecosystem that is less JVM-based with every day.

[dector]

@eparovyshnaya moving your post here.

@dector
btw, can we totally avoid all this build infrastructure replication for all exercises in the repo?

I suppose this PR and surrounding activity is dedicated to brushing, while keeping status quo.

But stepping a bit further, we can add a simple task to the root build script that will copy _template-ed wrapper and build scripts in each exercise folder at build time. This simple step will preent us from

• unexpected dependencies (like the ones we faced the other day on this PR)
• necessity of wordy dublicated updates
• attention spreading on build-system related prs review )

You've delt with our build system already. How do you think, is this feasible?

[dector]

I'm not sure that I've got your idea when this build script should be executed.

We should keep gradle files in git because (if I got it right) exercism cli tool is fetching them from the repository.

But in general, I totally agree that lots of tasks like versions update definitely can and should be automated.

[eparovyshnaya]

Regarding exercism cli - this is somehow language-dependent. Not an expert in Go, but can investigate this part. Hope we can satisfy the tool requirements and avoid this enourmous duplication in the same time. Thus the feasibility here is questionable for now.

Kotlin track ci: travis runs our own .sh-scripts. Can we just add copy build-files from _template preude to each exercise build?

[dector]

Kotlin track ci: travis runs our own .sh-scripts. Can we just add copy build-files from _template preude to each exercise build?

I'm sorry. Do you mean these:

• We are updating/creating an exrcise in git.
• When they are merged, CI script updates/creates gradle files in this exercise directory.

Or you mean something different? I'm affraid I could get you idea. :(

[eparovyshnaya]

Sorry for being unclear.

Let's assume we remove all gradle wrapper files and build script (I mean precisely gradle dir, gradlew* scripts and build.gradle* script) from each exercise directory in the repository. There is no building infrastructure anymore, only src, .meta dirs and README.md file. It's just a goal worth fighting for, so let's take it as a starting point for further reasoning.

What happens next?

1. TravisCI will fail all our builds.

What can we do to keep status quo for CI?

I suggest modify bin/journey-test.sh in the place of solve_exercise() function. When it composes a working directory for an exercise at a build agent file system, we could copy all necessary gradle wrapper files and build script not from the exercise source location, but from _template folder. Which, I hope, has been also checked out. Like substitute this row

cp ${track_root}/exercises/${exercise}/build.gradle.kts ${exercism_exercises_dir}/${TRACK}/${exercise}/build.gradle.kts

with something like this

cp ${track_root}/_template/build.gradle.kts ${exercism_exercises_dir}/${TRACK}/${exercise}/build.gradle.kts

and expand this idea for wrapper dir and scripts.

Will it be enough to keep Travis build doing their work?

2. Exercism cli will not find building files

Not going to debate changes in this tool, too huge and unpredictable.
And cannot tell yet if it is feasible at all by a single track source code modification. Investigation is required here.

3. A contributor, say, one who wishes to update an exercise to it's current spec, will not find any building infratructure in the exercise folder.

Here I'd rather

• describe the situation in CONTRIBUTING.md (Add CONTRIBUTING.md - instruction for contributors #353)
• either provide a coping tool for a contributor or rely on manual copying
• extend .gitignore to prevent these files from appearing in the repo again

What do you think?

[dector]

Thanks! Since CI will not commit into our database - sounds like something we can explore, definitely. The main pitfall here is step (2). As we need to provide test runners and building tool for students.

I like the idea of kbuild but it's in the early development stage now. Build-system-as-a-library is something that might work for us otherwise.