Skip to content

Commit d317418

Browse files
authored
Add snap packages to metadata collected in JSON report (new) (#2396)
* Move snap_utils from checkbox_support to checkbox_ng.support * Fix calls to snap_utils * Point checkbox_support.snap_utils to checkbox_ng.support.snap_utils This is required so that scripts that make use of `checkbox_support.snap_utils` can still call it, and are served the modules they expect. * Add list of snap packages to device_info script * Add snap packages to list of collected metadata in JSON report * Update asserts_data location in MANIFEST.in * Black linting * Move requests-unixsocket requirement * Fix checkbox24 build checkbox22 has "python3-requests-unixsocket" as a stage-package in checkbox-support. checkbox24 removed "python3-requests-unixsocket" from checkbox-support stage-packages. When checkbox-ng is built, it needs "requests" library (specified in pyproject.toml). pip tries to upgrade/install requests which depends on urllib3, but urllib3 2.0.7 comes from the Debian package system (not from pip). Without the requests-unixsocket package in stage, pip cannot find the necessary dependencies and tries to uninstall the system-provided urllib3 2.0.7. Adding PIP_IGNORE_INSTALLED should prevent pip from tripping over an already installed system package. * snapcraft: Move requests-unixsocket for series16~22 The python3-requests-unixsocket requirement has moved from checkbox-support to checkbox-ng * snapcraft: undo the hack done in previous commit for series24 * snapcraft: fix series24 using python-packages In Snapcraft documentation about the python plugin[1], it is told that "If the source contains a setup.py or pyproject.toml file, those files are used to install the dependencies specified by the package itself." Because of this, the request_unixsocket2 package is being pulled, and it requires urllib3 2.4 or highed. However, the Debian stage package python3-requests-oauthlib also pulls urllib3, but it's the version from the Debian archives, which is 2.0.7. Because of this, pip tries to uninstall the old version and fails (because it's a Debian install). Replacing the python3-requests-oauthlib stage package with requests-oauthlib python package fixes the build issue. [1]: https://documentation.ubuntu.com/snapcraft/latest/reference/plugins/python_plugin/#how-it-works
1 parent f3e2be5 commit d317418

39 files changed

Lines changed: 638 additions & 601 deletions

checkbox-core-snap/series16/snap/snapcraft.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -207,7 +207,6 @@ parts:
207207
- python3-evdev
208208
- python3-gi
209209
- python3-natsort
210-
- python3-requests-unixsocket
211210
- python3-serial
212211
- python3-setuptools
213212
- python3-yaml
@@ -260,6 +259,7 @@ parts:
260259
- python3-jinja2
261260
- python3-packaging
262261
- python3-requests-oauthlib
262+
- python3-requests-unixsocket
263263
- python3-urwid
264264
- python3-xlsxwriter
265265
- python3-setuptools

checkbox-core-snap/series18/snap/snapcraft.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,6 @@ parts:
204204
source: checkbox-support
205205
source-type: local
206206
stage-packages:
207-
- python3-requests-unixsocket
208207
- libbluetooth3
209208
- libsystemd0
210209
- v4l-utils
@@ -246,6 +245,7 @@ parts:
246245
- python3-jinja2
247246
- python3-packaging
248247
- python3-requests-oauthlib
248+
- python3-requests-unixsocket
249249
- python3-urwid
250250
- python3-xlsxwriter
251251
- python3-yaml

checkbox-core-snap/series20/snap/snapcraft.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,6 @@ parts:
206206
stage-packages:
207207
# actual requirements
208208
- python3-bluez
209-
- python3-requests-unixsocket
210209
- libsystemd0
211210
- v4l-utils
212211
# added to stage python:
@@ -253,6 +252,7 @@ parts:
253252
- python3-packaging
254253
- python3-pyparsing
255254
- python3-requests-oauthlib
255+
- python3-requests-unixsocket
256256
- python3-urwid
257257
- python3-xlsxwriter
258258
# added to stage python:

checkbox-core-snap/series22/snap/snapcraft.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,6 @@ parts:
213213
stage-packages:
214214
# actual requirements
215215
- python3-bluez
216-
- python3-requests-unixsocket
217216
- libsystemd0
218217
- v4l-utils
219218
# added to stage python:
@@ -260,6 +259,7 @@ parts:
260259
- python3-packaging
261260
- python3-pyparsing
262261
- python3-requests-oauthlib
262+
- python3-requests-unixsocket
263263
- python3-urwid
264264
- python3-xlsxwriter
265265
# added to stage python:

checkbox-core-snap/series24/snap/snapcraft.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -244,7 +244,6 @@ parts:
244244
- python3-jinja2
245245
- python3-packaging
246246
- python3-pyparsing
247-
- python3-requests-oauthlib
248247
- python3-urwid
249248
- python3-xlsxwriter
250249
# added to stage python:
@@ -260,6 +259,8 @@ parts:
260259
- python3.12-minimal
261260
- python3-yaml
262261
python-packages:
262+
- requests-oauthlib
263+
- requests-unixsocket
263264
- tqdm
264265
- picamera # p-p-c dep that wouldnt install in another part
265266
after: [checkbox-support]

checkbox-ng/MANIFEST.in

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,3 +51,4 @@ recursive-include checkbox_ng/support/parsers/tests/fixtures *.txt
5151
recursive-include checkbox_ng/support/parsers/tests/pactl_data *.txt
5252
recursive-include checkbox_ng/support/parsers/tests/udevadm_data *.txt *.lsblk.json
5353
recursive-include checkbox_ng/support/parsers/tests/v4l2_compliance_data *.txt
54+
recursive-include checkbox_ng/support/snap_utils/tests/asserts_data *.txt

checkbox-ng/checkbox_ng/support/device_info.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
from checkbox_ng.support.parsers.meminfo import MeminfoParser
99
from checkbox_ng.support.parsers.udevadm import parse_udevadm_output
1010
from checkbox_ng.support.parsers import _json_fallback
11+
from checkbox_ng.support.snap_utils.snapd import Snapd
1112

1213
"""
1314
Device Information Collector
@@ -45,6 +46,10 @@ def get_meminfo():
4546
return MeminfoParser().run()
4647

4748

49+
def get_snap_packages():
50+
return Snapd().list()
51+
52+
4853
def get_uname():
4954
return {
5055
"system": platform.system(),
@@ -75,6 +80,9 @@ def parse_args(argv=None):
7580
help="Return information about the Linux distribution being used",
7681
)
7782
subparsers.add_parser("memory", help="Return memory information")
83+
subparsers.add_parser(
84+
"snaps", help="Return information about installed Snaps"
85+
)
7886
subparsers.add_parser("uname", help="Return uname information")
7987
return parser.parse_args(argv)
8088

@@ -87,6 +95,7 @@ def main(argv=None):
8795
"devices": get_devices,
8896
"debian_packages": get_debian_packages,
8997
"memory": get_meminfo,
98+
"snaps": get_snap_packages,
9099
"uname": get_uname,
91100
}
92101

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# This file is part of Checkbox.
2+
#
3+
# Copyright 2017 Canonical Ltd.
4+
# Written by:
5+
# Jonathan Cave <jonathan.cave@canonical.com>
6+
#
7+
# Checkbox is free software: you can redistribute it and/or modify
8+
# it under the terms of the GNU General Public License version 3,
9+
# as published by the Free Software Foundation.
10+
11+
#
12+
# Checkbox is distributed in the hope that it will be useful,
13+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
14+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15+
# GNU General Public License for more details.
16+
#
17+
# You should have received a copy of the GNU General Public License
18+
# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
19+
20+
21+
"""
22+
:mod:`checkbox_ng.support.snap_utils`
23+
==================================
24+
25+
This module provides tools for testing aspects of systems running snapd.
26+
"""
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Copyright 2020 Canonical Ltd.
2+
# All rights reserved.
3+
#
4+
# Written by:
5+
# Jonathan Cave <jonathan.cave@canonical.com>
6+
7+
import yaml
8+
9+
10+
def decode(assertion_stream):
11+
"""Generate individual assertions in yaml format from a stream.
12+
13+
Multiple assertions are identified in the stream by a double empty line as
14+
per the REST API documentation. Signatures are not verified and removed
15+
so any data returned should be used only for informational purposes.
16+
"""
17+
count = int(assertion_stream.headers["X-Ubuntu-Assertions-Count"])
18+
if count > 0:
19+
# split in to individual assertions
20+
for assertion in assertion_stream.text.split("\n\n\n\n"):
21+
# split to remove signature
22+
content = assertion.split("\n\n")[0]
23+
if int(yaml.__version__.split(".")[0]) < 5:
24+
yield yaml.load(content)
25+
else:
26+
yield yaml.load(content, Loader=yaml.FullLoader)
27+
28+
29+
def model_to_resource(model_assertion):
30+
"""Convert assertion yaml to flat dict for resource output."""
31+
resource = {}
32+
# list keys that can just be copied over
33+
wanted_keys = (
34+
"type",
35+
"authority-id",
36+
"brand-id",
37+
"model",
38+
"architecture",
39+
"base",
40+
"grade",
41+
"sign-key-sha3-384",
42+
"store",
43+
)
44+
for key, val in model_assertion.items():
45+
if key in wanted_keys:
46+
resource[key] = val
47+
# handle other more complicated keys
48+
if "grade" in model_assertion:
49+
# is in UC20 format
50+
resource["grade"] = model_assertion.get("grade")
51+
for snap in model_assertion["snaps"]:
52+
if snap["type"] in ("kernel", "gadget"):
53+
resource[snap["type"]] = snap["name"]
54+
resource["{}_track".format(snap["type"])] = snap[
55+
"default-channel"
56+
]
57+
else:
58+
# older formats
59+
for key in ("kernel", "gadget"):
60+
val = model_assertion.get(key)
61+
if val:
62+
if "=" in val:
63+
snap, track = [x.strip() for x in val.split("=")]
64+
resource[key] = snap
65+
resource["{}_track".format(key)] = track
66+
else:
67+
resource[key] = val
68+
69+
# Check if the model has a store
70+
if "store" not in resource:
71+
resource["store"] = "unknown"
72+
return resource
73+
74+
75+
def serial_to_resource(serial_assertion):
76+
"""Convert assertion yaml to flat dict for resource output."""
77+
resource = {}
78+
# list keys that can just be copied over
79+
wanted_keys = (
80+
"type",
81+
"authority-id",
82+
"brand-id",
83+
"model",
84+
"serial",
85+
"sign-key-sha3-384",
86+
)
87+
for key, val in serial_assertion.items():
88+
if key in wanted_keys:
89+
resource[key] = val
90+
return resource
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Copyright 2017 Canonical Ltd.
2+
# All rights reserved.
3+
#
4+
# Written by:
5+
# Jonathan Cave <jonathan.cave@canonical.com>
6+
# Maciej Kisielewski <maciej.kisielewski@canonical.com>
7+
8+
import configparser
9+
import json
10+
import os
11+
import re
12+
import subprocess as sp
13+
import sys
14+
15+
16+
def get_snapctl_config():
17+
"""Query snapctl for config file variables"""
18+
out = (
19+
sp.check_output(["snapctl", "get", "conf"])
20+
.decode(sys.stdout.encoding)
21+
.strip()
22+
)
23+
if out:
24+
return json.loads(out)
25+
return {}
26+
27+
28+
def get_configuration_set():
29+
"""
30+
Get names and their default values declared in Snap's config_vars.
31+
32+
config_vars should list all the configuration variables in a `key=value`
33+
syntax. The line can list variable name only, if the variable should not
34+
have a default value. All keys should comprise of CAPS, numbers and
35+
undescores (_).
36+
37+
The returned keys are lowercase, as required by snapctl.
38+
"""
39+
config_set_path = os.path.expandvars("$SNAP/config_vars")
40+
config_set = dict()
41+
key_re = re.compile(r"^(?:[A-Z0-9]+_?)*[A-Z](?:_?[A-Z0-9])*$")
42+
try:
43+
for line in open(config_set_path, "rt").readlines():
44+
line = line.strip()
45+
if not line or line.startswith("#"):
46+
continue
47+
k, _, v = line.partition("=")
48+
if not key_re.match(k):
49+
raise ValueError("%s is not a valid configuration key" % k)
50+
# snapd accepts lowercase and dashes only for config names
51+
# so let's "mangle" the names to match the requirement
52+
k = k.replace("_", "-").lower()
53+
config_set[k] = v
54+
except FileNotFoundError:
55+
# silently ignore missing config_vars
56+
pass
57+
return config_set
58+
59+
60+
def write_checkbox_conf(configuration):
61+
"""Write checkbox.conf in $SNAP_DATA dir."""
62+
config = configparser.ConfigParser()
63+
config.optionxform = str
64+
config.add_section("environment")
65+
for key in sorted(configuration.keys()):
66+
val = str(configuration[key])
67+
# unmangle the key
68+
key = key.replace("-", "_").upper()
69+
config.set("environment", key, val)
70+
71+
checkbox_conf_path = os.path.expandvars("$SNAP_DATA/checkbox.conf")
72+
os.makedirs(os.path.dirname(checkbox_conf_path), exist_ok=True)
73+
with open(checkbox_conf_path, "wt") as stream:
74+
config.write(stream)
75+
76+
77+
def print_checkbox_conf():
78+
"""Print the current checkbox.conf in $SNAP_DATA."""
79+
checkbox_conf_path = os.path.expandvars("$SNAP_DATA/checkbox.conf")
80+
config = configparser.ConfigParser()
81+
config.optionxform = str
82+
config.read(checkbox_conf_path)
83+
if config.has_section("environment"):
84+
for key in config["environment"]:
85+
print("{}={}".format(key, config["environment"][key]))
86+
87+
88+
def refresh_configuration():
89+
"""
90+
Read config_vars, write the ones missing in snapd
91+
and call update configuration in both, snapd and checkbox.conf.
92+
93+
This is called from the snap configure hook.
94+
"""
95+
config_vars = get_configuration_set()
96+
if config_vars:
97+
current = get_snapctl_config()
98+
for key in config_vars.keys():
99+
if key not in current.keys() or not current[key]:
100+
current[key] = config_vars[key]
101+
update_configuration(current)
102+
103+
104+
def update_configuration(updated_entries):
105+
"""
106+
Update snapd configuration and write checkbox.conf file
107+
108+
The is called from the configure snap app.
109+
110+
:param updated_entries:
111+
A dict containing the configuration to set.
112+
Keys should contain lowercase letters, dashes and number only.
113+
"""
114+
vars_to_set = []
115+
key_re = re.compile(r"^(?:[a-z0-9]+-?)*[a-z](?:-?[a-z0-9])*$")
116+
for k, v in updated_entries.items():
117+
if not key_re.match(k):
118+
raise ValueError("'%s' is not a valid key" % k)
119+
vars_to_set.append("conf.{}={}".format(k.replace("_", "-").lower(), v))
120+
sp.run(["snapctl", "set"] + sorted(vars_to_set))
121+
write_checkbox_conf(get_snapctl_config())

0 commit comments

Comments
 (0)