Skip to content

Commit d654099

Browse files
authored
[QT-696] Add enos scenario outline (#128)
* [QT-696] Add `enos scenario outline` Add metadata support to Enos scenarios to allow us to quickly read an outline of a scenario and determine what quality characteristics are verified. We introduce new DSL schema: - `quality`: A new top-level object that describes a quality characteristic. - `quality.description`: An required string attribute in the `quality` block that describes the quality characteristic in detail. - `scenario.description`: An optional string attribute that describes the scenario. - `scenario.step.description`: An optional string attribute that describes the scenario step. - `scenario.step.verifies`: An optional `quality` attribute. Can be either a singular `quality` or a list of `quality`'s. Using these new blocks and attributes a scenario author can now more fully describe the purpose of the scenario, each step, and which quality characteristics are verified by it. To see the generated outline you can use `enos scenario outline <optional-filter>` to generate a scenario outline. The `scenario outline` sub-command supports a new output formatter: `html`. Use the HTML formatter and your browser for a cleaner overview if so desired, e.g.: ``` enos scenario outline --format html > outline.html open outline.html ``` * Fix deprecations in `.golangci.yml` * Fix warnings output by `golangci-lint` * Silence `buf` lint error that is irrelevant to us * Support decoding `quality` blocks * Support decoding `scenario.description` * Support decoding `scenario.step.description` * Support decoding `scenario.verifies` * Add `OutlineScenario()` RPC to the enos service. * Add acceptance test for `enos scenario outline` * Restructure matrix decoder into its own type. This allows us to keep a copy of the original matrix around to use in the outline * Add new decoder target for scenario outlines * Add `ShowOutline()` to `basic` UI * Add `ShowOutline()` to `machine` UI * Create a shim HTML UI * Add `ShowOutline()` to `html` UI * Bump version to 0.0.28 * Add missing documenation for `globals` DSL to `README.md` * Add missing documenation for `sample` DSL to `README.md` * Add documenation for `quality` DSL to `README.md` * Add missing documenation for `enos scenario sample` command to `README.md` * Add documenation for `enos scenario outline` command to `README.md` Signed-off-by: Ryan Cragun <[email protected]>
1 parent e10f3ab commit d654099

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+4605
-2106
lines changed

.github/workflows/build.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ env:
1313

1414
jobs:
1515
product-metadata:
16-
runs-on: linux
16+
runs-on: ubuntu-latest
1717
outputs:
1818
filepath: ${{ steps.generate-metadata-file.outputs.filepath }}
1919
product-version: ${{ steps.product-metadata.outputs.product-version }}
@@ -34,7 +34,7 @@ jobs:
3434
path: ${{ steps.generate-metadata-file.outputs.filepath }}
3535

3636
profile-binary:
37-
runs-on: linux
37+
runs-on: ubuntu-latest
3838
env:
3939
GOPRIVATE: 'github.com/hashicorp/*'
4040
TOKEN: ${{ secrets.ELEVATED_GITHUB_TOKEN }}
@@ -76,7 +76,7 @@ jobs:
7676
needs:
7777
- product-metadata
7878
- profile-binary
79-
runs-on: linux
79+
runs-on: ubuntu-latest
8080
strategy:
8181
matrix:
8282
goos: [linux, darwin]
@@ -147,7 +147,7 @@ jobs:
147147
needs:
148148
- product-metadata
149149
- build
150-
runs-on: linux
150+
runs-on: ubuntu-latest
151151
strategy:
152152
matrix:
153153
arch: ["arm64", "amd64"]

.github/workflows/changelog.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ on:
99
jobs:
1010
# If the PR does not have one of the four designated `changelog/` labels applied, this job will fail.
1111
check-for-changelog-label:
12-
runs-on: linux
12+
runs-on: ubuntu-latest
1313
if: (!contains(github.event.pull_request.labels.*.name, 'changelog/none')) && (!contains(github.event.pull_request.labels.*.name, 'changelog/bug')) && (!contains(github.event.pull_request.labels.*.name, 'changelog/feat')) && (!contains(github.event.pull_request.labels.*.name, 'changelog/other'))
1414
steps:
1515
- name: Changelog label not found

.github/workflows/create_release.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ on:
2929
jobs:
3030
create-release:
3131
name: Create Release
32-
runs-on: linux
32+
runs-on: ubuntu-latest
3333
env:
3434
GOPRIVATE: 'github.com/hashicorp/*'
3535
GITHUB_TOKEN: ${{ secrets.ELEVATED_GITHUB_TOKEN }}

.github/workflows/pr_build.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ env:
1212
jobs:
1313
build-artifact:
1414
name: Build Linux Artifact
15-
runs-on: linux
15+
runs-on: ubuntu-latest
1616
strategy:
1717
matrix:
1818
goos: [linux]

.github/workflows/update_homebrew_formula.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ on:
4949
jobs:
5050
update-formula:
5151
name: "Update Homebrew formula definition"
52-
runs-on: linux
52+
runs-on: ubuntu-latest
5353
env:
5454
# Note: `gh` CLI automatically looks for and uses `env.GH_TOKEN` for authentication.
5555
# This token must have read:org scope in order to authenticate on a different repo.

.github/workflows/validate.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ on:
2020
jobs:
2121
static-analysis:
2222
name: "Format"
23-
runs-on: linux
23+
runs-on: ubuntu-latest
2424
steps:
2525
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
2626
- uses: actions/setup-go@0c52d547c9bc32b1aa3301fd7a9cb496313a4491 # v5.0.0
@@ -48,7 +48,7 @@ jobs:
4848

4949
test:
5050
name: Go tests
51-
runs-on: linux
51+
runs-on: ubuntu-latest
5252
env:
5353
CI: true
5454
steps:

.golangci.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ run:
1111
output:
1212
# colored-line-number|line-number|json|tab|checkstyle|code-climate|junit-xml|github-actions
1313
# default is "colored-line-number"
14-
format: colored-line-number
14+
formats:
15+
- format: colored-line-number
1516
# print lines of code with issue, default is true
1617
print-issued-lines: true
1718
# print linter name in the end of issue text, default is true

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ lint-fix-golang:
6666

6767
.PHONY: lint-proto
6868
lint-proto:
69-
pushd proto && buf lint --error-format=$(BUF_LINT_OUT_FORMAT)
69+
buf lint --error-format=$(BUF_LINT_OUT_FORMAT)
7070

7171
.PHONY: fmt
7272
fmt: fmt-golang fmt-proto fmt-terraform fmt-enos

README.md

Lines changed: 235 additions & 36 deletions
Large diffs are not rendered by default.
Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
// Copyright (c) HashiCorp, Inc.
2+
// SPDX-License-Identifier: MPL-2.0
3+
4+
package acceptance
5+
6+
import (
7+
"context"
8+
"fmt"
9+
"path/filepath"
10+
"testing"
11+
12+
"github.com/stretchr/testify/require"
13+
"google.golang.org/protobuf/encoding/protojson"
14+
15+
"github.com/hashicorp/enos/proto/hashicorp/enos/v1/pb"
16+
)
17+
18+
func TestAcc_Cmd_Scenario_Outline(t *testing.T) {
19+
t.Parallel()
20+
21+
for _, test := range []struct {
22+
dir string
23+
out *pb.OutlineScenariosResponse
24+
fail bool
25+
}{
26+
{
27+
dir: "scenarios/scenario_outline",
28+
out: &pb.OutlineScenariosResponse{
29+
Outlines: []*pb.Scenario_Outline{
30+
{
31+
Scenario: &pb.Ref_Scenario{
32+
Id: &pb.Scenario_ID{
33+
Name: "multiple_verifies",
34+
Description: "This is a multiline description\nof the upgrade scenario.\n",
35+
},
36+
},
37+
Matrix: &pb.Matrix{
38+
Vectors: []*pb.Matrix_Vector{
39+
{Elements: []*pb.Matrix_Element{
40+
{Key: "arch", Value: "amd64"},
41+
{Key: "arch", Value: "arm64"},
42+
}},
43+
{Elements: []*pb.Matrix_Element{
44+
{Key: "distro", Value: "ubuntu"},
45+
{Key: "distro", Value: "rhel"},
46+
}},
47+
},
48+
},
49+
Steps: []*pb.Scenario_Outline_Step{
50+
{
51+
Name: "test",
52+
Description: "This is an indented\nmultiline step description.\n",
53+
Verifies: []*pb.Quality{
54+
{
55+
Name: "inline",
56+
Description: "an inline quality that isn't reused",
57+
},
58+
{
59+
Name: "the_data_is_durable",
60+
Description: "The data is durable\nafter an upgrade.\n",
61+
},
62+
{
63+
Name: "the_tests_pass",
64+
Description: "The tests all pass!",
65+
},
66+
},
67+
},
68+
},
69+
Verifies: []*pb.Quality{
70+
{
71+
Name: "inline",
72+
Description: "an inline quality that isn't reused",
73+
},
74+
{
75+
Name: "the_data_is_durable",
76+
Description: "The data is durable\nafter an upgrade.\n",
77+
},
78+
{
79+
Name: "the_tests_pass",
80+
Description: "The tests all pass!",
81+
},
82+
},
83+
},
84+
{
85+
Scenario: &pb.Ref_Scenario{
86+
Id: &pb.Scenario_ID{
87+
Name: "singular_verifies",
88+
Description: "This is a multiline description\nof the upgrade scenario.\n",
89+
},
90+
},
91+
Matrix: &pb.Matrix{
92+
Vectors: []*pb.Matrix_Vector{
93+
{Elements: []*pb.Matrix_Element{
94+
{Key: "arch", Value: "amd64"},
95+
{Key: "arch", Value: "arm64"},
96+
}},
97+
{Elements: []*pb.Matrix_Element{
98+
{Key: "distro", Value: "ubuntu"},
99+
{Key: "distro", Value: "rhel"},
100+
}},
101+
},
102+
},
103+
Steps: []*pb.Scenario_Outline_Step{
104+
{
105+
Name: "test",
106+
Description: "This is an indented\nmultiline step description.\n",
107+
Verifies: []*pb.Quality{
108+
{
109+
Name: "the_tests_pass",
110+
Description: "The tests all pass!",
111+
},
112+
},
113+
},
114+
},
115+
Verifies: []*pb.Quality{
116+
{
117+
Name: "the_tests_pass",
118+
Description: "The tests all pass!",
119+
},
120+
},
121+
},
122+
},
123+
},
124+
},
125+
} {
126+
t.Run(test.dir, func(t *testing.T) {
127+
t.Parallel()
128+
enos := newAcceptanceRunner(t)
129+
130+
path, err := filepath.Abs(filepath.Join("./", test.dir))
131+
require.NoError(t, err)
132+
cmd := fmt.Sprintf("scenario outline --chdir %s --format json", path)
133+
out, err := enos.run(context.Background(), cmd)
134+
if test.fail {
135+
require.Error(t, err)
136+
137+
return
138+
}
139+
140+
require.NoError(t, err)
141+
got := &pb.OutlineScenariosResponse{}
142+
require.NoError(t, protojson.Unmarshal(out, got))
143+
require.Len(t, got.GetOutlines(), len(test.out.GetOutlines()))
144+
for i := range test.out.GetOutlines() {
145+
require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetName(), got.GetOutlines()[i].GetScenario().GetId().GetName())
146+
require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetDescription(), got.GetOutlines()[i].GetScenario().GetId().GetDescription())
147+
require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetFilter(), got.GetOutlines()[i].GetScenario().GetId().GetFilter())
148+
require.EqualValues(t, test.out.GetOutlines()[i].GetScenario().GetId().GetUid(), got.GetOutlines()[i].GetScenario().GetId().GetUid())
149+
require.EqualValues(t, test.out.GetOutlines()[i].GetMatrix().String(), got.GetOutlines()[i].GetMatrix().String())
150+
require.Len(t, got.GetOutlines()[i].GetVerifies(), len(test.out.GetOutlines()[i].GetVerifies()))
151+
for q := range test.out.GetOutlines()[i].GetVerifies() {
152+
require.EqualValues(t, test.out.GetOutlines()[i].GetVerifies()[q].GetName(), got.GetOutlines()[i].GetVerifies()[q].GetName())
153+
require.EqualValues(t, test.out.GetOutlines()[i].GetVerifies()[q].GetDescription(), got.GetOutlines()[i].GetVerifies()[q].GetDescription())
154+
}
155+
for s := range test.out.GetOutlines()[i].GetSteps() {
156+
require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetName(), got.GetOutlines()[i].GetSteps()[s].GetName())
157+
require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetDescription(), got.GetOutlines()[i].GetSteps()[s].GetDescription())
158+
for q := range test.out.GetOutlines()[i].GetSteps()[s].GetVerifies() {
159+
require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetName(), got.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetName())
160+
require.EqualValues(t, test.out.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetDescription(), got.GetOutlines()[i].GetSteps()[s].GetVerifies()[q].GetDescription())
161+
}
162+
}
163+
}
164+
})
165+
}
166+
}

0 commit comments

Comments
 (0)