Skip to content
Quality

The CRM image build carries the file its analyzers are handed #238

Sign in to view logs

The CRM image build carries the file its analyzers are handed

The CRM image build carries the file its analyzers are handed #238

Workflow file for this run

.github/workflows/quality.yml at cec6eff
name: Quality
on:
push:
branches: [master, dev]
pull_request:
branches: [master, dev]
permissions:
contents: read
env:
DOTNET_NOLOGO: true
DOTNET_CLI_TELEMETRY_OPTOUT: true
jobs:
coverage:
name: Coverage thresholds
runs-on: ubuntu-latest
# `pull-requests: write` for the reporting step below, granted on the job because that
# is the only scope Actions accepts it at — a `permissions:` key on a step is a workflow
# validation error, not an ignored one.
permissions:
contents: read
pull-requests: write
# The same servers ci.yml's build job runs, for the same reason one commit later:
# without them the durability, tenancy and sample suites skip, and every line they
# would have covered is counted as uncovered. A coverage number measured with a third
# of the suite absent is not a smaller number, it is a wrong one.
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 20
redis:
image: redis:7
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 5s
--health-timeout 5s
--health-retries 20
rabbitmq:
image: rabbitmq:3.12
ports:
- 5672:5672
options: >-
--health-cmd "rabbitmq-diagnostics -q ping"
--health-interval 5s
--health-timeout 10s
--health-retries 30
# The Service Bus emulator's own dependency, as a service container so the emulator
# step below can reach it across the bridge. ci.yml §sqledge says the rest.
sqledge:
image: mcr.microsoft.com/azure-sql-edge:latest
env:
ACCEPT_EULA: "Y"
MSSQL_SA_PASSWORD: "Str0ng!Passw0rd"
ports:
- 1433:1433
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
global-json-file: global.json
# Kafka and Service Bus, which ci.yml has had since the two plugins landed and this job
# did not. Their absence here did not lower the coverage number, it made it wrong: every
# line in plugins/FlowX.Kafka and plugins/FlowX.AzureServiceBus was counted as coverable
# and uncovered while the only tests that reach them were skipping for want of a broker.
# That is the exact reading this job's own comment above rejects for PostgreSQL and Redis.
- name: Start Apache Kafka
run: |
curl -sS -O https://archive.apache.org/dist/kafka/4.1.2/kafka_2.13-4.1.2.tgz
tar -xzf kafka_2.13-4.1.2.tgz
cd kafka_2.13-4.1.2
KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid)
bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c config/server.properties --standalone
nohup bin/kafka-server-start.sh config/server.properties > "$RUNNER_TEMP/kafka.log" 2>&1 &
for _ in $(seq 1 30); do
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list > /dev/null 2>&1 && break
sleep 2
done
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
- name: Start the Service Bus emulator
run: |
docker run -d --name flowx-sbemulator \
--add-host host.docker.internal:host-gateway \
-p 5673:5672 \
-v "$GITHUB_WORKSPACE/tests/FlowX.AzureServiceBus.Tests/emulator-config.json:/ServiceBus_Emulator/ConfigFiles/Config.json" \
-e ACCEPT_EULA=Y \
-e SQL_SERVER=host.docker.internal \
-e MSSQL_SA_PASSWORD='Str0ng!Passw0rd' \
mcr.microsoft.com/azure-messaging/servicebus-emulator:latest
for _ in $(seq 1 30); do
docker logs flowx-sbemulator 2>&1 | grep -q "Emulator Service is Successfully Up" && break
sleep 5
done
docker logs flowx-sbemulator 2>&1 | tail -5
- name: Build
# Separately, and before the tests. The architecture gates scan the Release
# output of projects they deliberately do not reference, so under a bare
# `dotnet test` they ran while Ecommerce.dll and Banking.dll were still being
# compiled and failed on a tree that was merely incomplete.
run: dotnet build FlowX.slnx --configuration Release
- name: Test with coverage
# coverlet.runsettings excludes source-generated code. Without it the regex
# generator's output is measured, which reports the generator's branches
# rather than ours and drags the number down for no signal.
env:
FLOWX_POSTGRES_CONNECTION: "Host=localhost;Port=5432;Database=postgres;Username=postgres;Password=postgres"
FLOWX_REDIS_CONNECTION: "localhost:6379"
FLOWX_RABBITMQ_CONNECTION: "amqp://guest:guest@localhost:5672/"
FLOWX_KAFKA_BOOTSTRAP: "localhost:9092"
FLOWX_SERVICEBUS_CONNECTION: "Endpoint=sb://localhost:5673;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;"
# `--logger trx` and the summary below exist because this step has been failing
# without naming a test. The job's conclusion said "Coverage thresholds", the
# thresholds step was skipped because this one failed first, and the console output
# is buried under a thousand lines of RabbitMQ service-container log. A gate whose
# failure a reader cannot attribute is a gate nobody acts on, and this one has been
# red for days for exactly that reason.
run: >
dotnet test FlowX.slnx --configuration Release --no-build
--settings coverlet.runsettings
--collect:"XPlat Code Coverage"
--logger trx
--results-directory ./TestResults
- name: Which test failed
if: failure()
# The one thing the log does not make findable. Reads the trx files rather than the
# console, so the answer is the same whichever assembly drowned it.
#
# Written to the step summary and to a file the next step posts on the pull request,
# because this job cannot report through its log at all. Four service containers dump
# their logs at teardown, and the dump is longer than any tail worth reading: the last
# twelve hundred lines of a failing run are Redis and RabbitMQ starting up, and the
# name of the test that failed is somewhere above them. That is why this job spent
# days reporting "Coverage thresholds" for a failure that was never about coverage.
run: |
python3 - <<'PY' | tee -a "$GITHUB_STEP_SUMMARY" | tee failing-tests.txt
import glob, xml.etree.ElementTree as ET
ns = {'t': 'http://microsoft.com/schemas/VisualStudio/TeamTest/2010'}
found = 0
for path in glob.glob('TestResults/**/*.trx', recursive=True):
for result in ET.parse(path).getroot().iter(f'{{{ns["t"]}}}UnitTestResult'):
if result.get('outcome') != 'Failed':
continue
found += 1
print(f'::error::{result.get("testName")}')
message = result.find(f'.//{{{ns["t"]}}}Message')
if message is not None and message.text:
print(message.text[:2000])
trx = glob.glob('TestResults/**/*.trx', recursive=True)
if found:
print(f'{found} failed test(s) named across {len(trx)} trx file(s).')
else:
# Not "nothing failed" — `dotnet test` exited non-zero. Either an assembly
# never wrote a trx because its test host died, or it wrote one and the run
# ended before the failure was recorded. Naming the count and the assemblies
# is what separates those two, and the first shape is the one that presents
# as "no failed test" while the job is red.
print(f'No failed test in {len(trx)} trx file(s); dotnet test still exited non-zero.')
print('An assembly whose test host died writes no trx at all. Present:')
for path in sorted(trx):
print(f' {path}')
PY
- name: Say so on the pull request
if: failure() && github.event_name == 'pull_request'
# The only channel this job has that a reader can reach in one click. The log is
# unreadable by construction (see above) and an Actions job summary is not exposed
# on the check, so a failure that names its cause has to say so here or nowhere.
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
[ -s failing-tests.txt ] || exit 0
{
echo "**Coverage thresholds — the test step failed, not the thresholds.**"
echo
echo '```'
cat failing-tests.txt
echo '```'
} > comment.md
gh pr comment "${{ github.event.pull_request.number }}" --body-file comment.md
- name: Enforce thresholds
# Thresholds from docs/21-Quality-Gates.md §2.1. Measured on the whole
# assembly here; the diff-scoped gate is Sonar's job below.
run: |
dotnet tool install --global dotnet-reportgenerator-globaltool
reportgenerator \
-reports:'TestResults/**/coverage.cobertura.xml' \
-targetdir:coverage \
-reporttypes:'Cobertura;TextSummary'
cat coverage/Summary.txt
python3 - <<'PY'
import sys, xml.etree.ElementTree as ET
root = ET.parse('coverage/Cobertura.xml').getroot()
line = float(root.get('line-rate', 0)) * 100
branch = float(root.get('branch-rate', 0)) * 100
print(f'line={line:.1f}% branch={branch:.1f}%')
bad = []
if line < 80: bad.append(f'line coverage {line:.1f}% < 80%')
if branch < 75: bad.append(f'branch coverage {branch:.1f}% < 75%')
for b in bad: print(f'::error::{b}')
sys.exit(1 if bad else 0)
PY
- uses: actions/upload-artifact@v4
if: always()
with:
name: coverage
path: coverage/
sonar:
name: Sonar quality gate
runs-on: ubuntu-latest
# Fork PRs have no access to the token. Skipping is correct; failing on a
# missing secret would just teach contributors that red is normal.
if: github.event.pull_request.head.repo.full_name == github.repository || github.event_name == 'push'
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Sonar needs history for new-code detection
- uses: actions/setup-dotnet@v4
with:
global-json-file: global.json
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '17'
- name: Analyze
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: |
if [ -z "$SONAR_TOKEN" ]; then
# A gate that no-ops in silence is indistinguishable from one that passed.
# This job deliberately stays green (see the comment on `if:` above), so the
# only thing that can carry the difference is an annotation loud enough to
# read and a summary that says which rows were not evaluated. `::notice::`
# was not that: it renders as an informational line most reviewers scroll
# past, under a job whose green tick reads as "quality gate passed".
echo "::warning title=Sonar quality gate did NOT run::SONAR_TOKEN is not configured, so the three merge gates in docs/21-Quality-Gates.md §2.1 that depend on it — critical issues, duplicated lines on new code, and security hotspots reviewed — were not evaluated for this change. A green tick on this job means the gate was skipped, not that it passed. See CHECKLIST.md item WP-0."
{
echo "### Sonar quality gate: SKIPPED"
echo
echo '`SONAR_TOKEN` is not configured on this repository, so the scanner never ran.'
echo
echo 'These merge gates were **not** evaluated:'
echo
echo '| Gate (docs/21-Quality-Gates.md §2.1) | Threshold | Evaluated |'
echo '|---|---|---|'
echo '| Critical issues | 0 | no |'
echo '| Duplicated lines on new code | ≤ 3 % | no |'
echo '| Security hotspots reviewed | 100 % | no |'
echo
echo 'The build-class Sonar rules are unaffected: they run from the'
echo '`SonarAnalyzer.CSharp` package at compile time and need no token,'
echo 'so they have already gated this change in every other job.'
} >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
dotnet tool install --global dotnet-sonarscanner
dotnet sonarscanner begin \
/k:"votrongdao_FlowX" \
/d:sonar.token="$SONAR_TOKEN" \
/d:sonar.host.url="https://sonarcloud.io" \
/d:sonar.cs.opencover.reportsPaths='**/coverage.opencover.xml' \
/d:sonar.qualitygate.wait=true
dotnet build FlowX.slnx --configuration Release
dotnet test FlowX.slnx --configuration Release --no-build \
/p:CollectCoverage=true /p:CoverletOutputFormat=opencover
dotnet sonarscanner end /d:sonar.token="$SONAR_TOKEN"
mutation:
name: Mutation score (FlowX.Core)
runs-on: ubuntu-latest
# A job with no timeout inherits six hours. This one ran past thirty-five minutes
# with no output on the first push where the workflow parsed at all, which reads
# as "still working" for the rest of a working day. Twenty-five minutes or it is
# a failure somebody looks at.
timeout-minutes: 25
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
global-json-file: global.json
- name: Stryker
# Applied to FlowX.Core only, on purpose. Line coverage says which lines
# ran; mutation score says whether a test would notice if they were wrong.
# FlowX.Core holds the execution semantics, so that distinction matters there.
run: |
if [ ! -d src/FlowX.Core ]; then
echo "FlowX.Core does not exist yet (WP-2). Nothing to mutate."
exit 0
fi
# Pinned: the flag below was renamed once already, and an unpinned tool
# turns that into a red job on a morning nobody changed anything.
dotnet tool install --global dotnet-stryker --version 4.16.0
# --test-project, or Stryker discovers every test project in the solution and
# runs all ~2700 tests against every mutant. That is the difference between a
# job that finishes and one that does not: FlowX.Core's own suite is what can
# notice a mutation in FlowX.Core.
dotnet stryker --project FlowX.Core.csproj \
--test-project ../../tests/FlowX.Core.Tests/FlowX.Core.Tests.csproj \
--break-at 70 --reporter progress --reporter html
- uses: actions/upload-artifact@v4
if: always()
with:
name: mutation-report
path: StrykerOutput/
debt:
name: Technical-debt policy
runs-on: ubuntu-latest
# The suppression rule of docs/21-Quality-Gates.md §6.1 is NOT here. It is the
# SuppressionsAreAccountable fitness function, run by the "Architecture fitness
# functions" step of ci.yml on this same trigger.
#
# A shell copy of it used to live in this job, and the two were not the same
# rule. It asked whether the *file* contained a FLOWX-DEBT marker anywhere, so
# one accountable suppression at the top licensed every unaccountable one below
# it — demonstrably: a file with a marked suppression on line 7 and a bare one
# on line 16 exited 0. It also never checked that the id cited had a row in
# docs/DEBT.md. The fitness function requires the marker within six lines of the
# suppression and the id to be registered, unexpired and inside the budget.
#
# Two gates for one rule, disagreeing, is worse than one: the weaker is the one
# people meet first, and passing it reads as compliance. Deleted rather than
# repaired, because repairing it would have produced a second implementation of
# a rule that already has one — to be kept in step by hand, forever, with no
# test that they agree.
steps:
- uses: actions/checkout@v4
- name: No unaccountable TODOs
# Every tree that ships or is read as an example, not just src and tests. A
# TODO in samples/ is copied by whoever follows the sample.
# Word-bounded: without \b, XXX matches the branch code every BIC ends in,
# and the banking sample's correspondent table failed this job as debt.
run: |
if grep -rnE '\b(TODO|FIXME|HACK|XXX)\b' --include='*.cs' src tests plugins samples scripts 2>/dev/null | grep -vE 'FLOWX-DEBT|issue #[0-9]+'; then
echo "::error::A TODO/FIXME/HACK must reference a FLOWX-DEBT id or an issue number."
exit 1
fi
echo "No unaccountable markers."