Where do you push integration tests ?

[thefasquelle]

I just read your entire website and it does a great job at explaining trunk based development and its various implementations with their advantages and challenges. Congrats and thank you for that.

But there are some things I don't quite understand. Mainly where integrations tests are pushed.
It seems quite common to me that integration tests will be done by different people (testers) than code and unit tests (developers).
In a commit-straight-to-trunk model, devs push code they unit tested as best as they can to make sure their update don't break anything. But things are never that simple and bugs will inevitably be detected by testers.

If we use branch for release, do testers push their tests to the release branch ? That would require to merge the release branch to trunk, which is described as a bad thing to do. But if we don't push code, only tests, it should be fine, right ? Worst case scenario is we did forgot to merge tests back to trunk so testers will notice it on next release and merge it at this moment.

The alternative I found is, instead of doing a branch for release, "freeze" trunk for testers to push tests and devs to push fixes to it directly. Devs who wants to work on new feature do it on a temporary branch. This temporary branch will be merged in trunk after release. So it is still creating a branch for each release, but everything is inverted as pushing to trunk is forbidden and developers all push to a temp branch.

[paul-hammant]

To help me understand where your org is, take a look at https://phdo.paulhammant.com/Trunk-Correlated-Practices-Chart. Your groups practices are most correlated with quarterly planned releases, monthly planned releases, weekly planned releases, daily planned releases, ten releases a day (planned or unplanned) or 100 releases a day (or unplanned). This would be for the same deployable set of applications, apps, services - whether that's a team of five, fifty, or five hundred. And to recap at the slower cadences, planned releases are not the same as unplanned (bug fix releases).

[thefasquelle]

Thank you for your quick response.
While I am indeed interested in advice for my particular situation, I would also wan't to understand the general idea. I feel like the integration test topic was not well explained on your website (unless you consider it should be done by the programmer pair alongside code and unit tests).

We do do embedded systems with quite long validation processes (on real machine). The release cycle is about 1 month.
Teams are rather small, 2-4 devs per project. Code, unit tests and integration tests (automated tests on regular computer/CI) are all in the same svn repo, commited directly to trunk. We release from trunk (tag).
The benefit is devs can easily run integration tests before pushing to check for regression.
The issue is devs may commit features for next release before testers were finished testing the previous versions. Or if a dev breaks the trunk and testers need to rollback to continue testing (leveraging SVN's mixed revision working copy).
Hence why we are trying to set up a better process where testers have time to fully test a version before it is released for validation on real machine.
For reasons unrelated, we are switching to git. This enables some new capabilities like local branching and squashing commits that we may or may not use.

[paul-hammant]

Embedded systems is really hard. Best case is every dev gets an emulator or a actualy device they do not share with anyone. Not only that, but the wire-services the device may avail of are also not shared with others. This is made even more hard as the seller of the hardware (seller, partner, etc) may not authorise more than one live and one non-live device for the company.

I could answer more fully, but given this is free consulting (and I am currently unemployed - no money coming in), would you mind if I asked again for you to pick a cadence from the ones I outline in that long PDF? You're once every 10 days or once every 100 days (I guess).

[thefasquelle]

We are mostly once every 10 days.
You are in no way obligated to help me with my particular situation, I was initially asking about more generic situations like other aspects mentioned on the website (maybe you talk more about it in your book ? I see some test-related entries in the table of contents)

[paul-hammant]

Cool, so one planned release every 7-14 days. Maybe an unplanned release following the planned (big fix) release , but not every time. You mentioned "We release from trunk (tag)" earlier. You might consider doing branch-for-release. You'd still let the devs work on trunk if they were not involved with release preparation itself. You'd consider the release branch as being eligible to be hardened towards release. Nothing committed to that release branch is ever merged back to trunk. Bugs identified in the release cycle are fixed on trunk and cherry picked to the release branch. As often as you cut release branches in a cycle, you're also older ones.

Say your planned cadence was 14 days. The week after a release the QA/UAT env's hardware (recall I said you probably have live hardware targets and a set of coupled non-live for true integ testing. That week after release the QA/ENV is coupled to the CI/CD for commits into trunk. In the week that's running up to the actual release day, the same QA/UAT hardware env is coupled to the CI/CD related to the release branch. It splits duty, cos you only have the one non-live (pre-prod) hard provision.

Say your planned cadence is shorter - 7 days .. as above but 4 days coupled to trunk, and 3 days coupled to the release branch.

Both 7 and 14 days cycles would likely utilize manual QA (also called testers), but I'd prefer QA and QE as the names for those roles myself.

If you're faster than daily planned releases you've moved to much more QE dependence. You're likely using emulators of hardware for much more dev work, and utilized far sooner in the production line that is modern software delivery. The chances of something passing on the emulators and failing one the real hardware should be miniscule. If it wasn't a manager would have insisted that y'all slow down your release cadence and do more checking. I don't think there's a general case when there's hardware involved

Nothing is set in stone though and many of the teleco-industry players have engineered custom high-reliability solutions for all this, that are not easily pigeonhole-able.