Skip to content

Commit 0022f15

Browse files
authored
Move some modules from checkbox_support to checkbox_ng (BugFix) (#2312)
* Move parsers from checkbox_support to a new checkbox_ng.support module In order to include device information in every Checkbox submission without relying on external jobs being present in the test plan run, a bunch of parsers and helpers initially available in checkbox_support need to be migrated to checkbox_ng (since we don't want to import stuff from checkbox_support in checkbox_ng, but the other way around is OK). In addition to these, the following code is also migrated because it is required by the parsers: - checkbox_support.lib modules - checkbox_support.monitor_config module - checkbox_support.helpers release_info and slugify modules In order not to break any providers (internal or third party), all the modules in checkbox_support remain, but they point to the code in checkbox_ng.support. * Modify meminfo resource script The meminfo module was modified when it was migrated from checkbox_support to checkbox_ng.support to decouple the parsed data from the information required by the resource job. In this commit, the resource script is adjusted to match the new code in the parser. * Adjust checkbox-ng tox.ini Some python classes copied over from checkbox_support do not need to be tested even though their class name starts with "Test". * Fix black * Fix checkbox_support tox config checkbox-ng is now a dependency of checkbox_support, so it needs to be installed before running the tests. * Fix unit test in checkbox_support * Move included files in MANIFEST.in * Update pyparsing dependencies pyparsing is used by some modules that have migrated from checkbox_support to checkbox_ng. Therefore, dependencies need updating. * Fix snap recipes Move pyparsing requirements from checkbox-support to checkbox-ng * Add dependency on checkbox-ng for the checkbox-support package * Fix checkbox-support manifest for vendorized libs * Black magic * Fix module import name * Fix memory_compare following modification in meminfo parser In the process of moving the meminfo parser, it was modified because it was mixing code from meminfo_resource.py script in the resource provider. The memory_compare script also needs to be modified, and can be simplified.
1 parent 7857e2d commit 0022f15

224 files changed

Lines changed: 6413 additions & 6387 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

checkbox-core-snap/series16/snap/snapcraft.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -246,7 +246,7 @@ parts:
246246
snapcraftctl build
247247
# install the dependencies of checkbox-support itself because this
248248
# pip version pyproject.toml dependencies
249-
$SNAPCRAFT_PART_INSTALL/usr/bin/python3 -m pip install "pyparsing<3.0.0" "requests<2.26.0" "distro<1.7.0" "requests_unixsocket<=0.3.0" "importlib_metadata<=1.0.0"
249+
$SNAPCRAFT_PART_INSTALL/usr/bin/python3 -m pip install "requests<2.26.0" "distro<1.7.0" "requests_unixsocket<=0.3.0" "importlib_metadata<=1.0.0"
250250
################################################################################
251251
checkbox-ng:
252252
plugin: python
@@ -279,7 +279,7 @@ parts:
279279
snapcraftctl build
280280
# install the dependencies of checkbox-support itself because this
281281
# pip version pyproject.toml dependencies
282-
$SNAPCRAFT_PART_INSTALL/usr/bin/python3 -m pip install "packaging<21.0" "psutil<=5.9.5" "requests<2.26.0" "urwid<=2.1.2" "Jinja2<=2.11.3" "XlsxWriter<=3.0.3" "tqdm<4.65.0" "importlib_metadata<=1.0.0"
282+
$SNAPCRAFT_PART_INSTALL/usr/bin/python3 -m pip install "packaging<21.0" "psutil<=5.9.5" "pyparsing<3.0.0" "requests<2.26.0" "urwid<=2.1.2" "Jinja2<=2.11.3" "XlsxWriter<=3.0.3" "tqdm<4.65.0" "importlib_metadata<=1.0.0"
283283
################################################################################
284284
checkbox-provider-resource:
285285
plugin: dump

checkbox-core-snap/series20/snap/snapcraft.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,6 @@ parts:
206206
stage-packages:
207207
# actual requirements
208208
- python3-bluez
209-
- python3-pyparsing
210209
- python3-requests-unixsocket
211210
- libsystemd0
212211
- v4l-utils
@@ -252,6 +251,7 @@ parts:
252251
- python3-markupsafe
253252
- python3-jinja2
254253
- python3-packaging
254+
- python3-pyparsing
255255
- python3-requests-oauthlib
256256
- python3-urwid
257257
- python3-xlsxwriter

checkbox-core-snap/series22/snap/snapcraft.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,6 @@ parts:
213213
stage-packages:
214214
# actual requirements
215215
- python3-bluez
216-
- python3-pyparsing
217216
- python3-requests-unixsocket
218217
- libsystemd0
219218
- v4l-utils
@@ -259,6 +258,7 @@ parts:
259258
- python3-markupsafe
260259
- python3-jinja2
261260
- python3-packaging
261+
- python3-pyparsing
262262
- python3-requests-oauthlib
263263
- python3-urwid
264264
- python3-xlsxwriter

checkbox-core-snap/series24/snap/snapcraft.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,6 @@ parts:
199199
stage-packages:
200200
# actual requirements
201201
- python3-bluez
202-
- python3-pyparsing
203202
- libsystemd0
204203
- v4l-utils
205204
# added to stage python:
@@ -244,6 +243,7 @@ parts:
244243
- python3-markupsafe
245244
- python3-jinja2
246245
- python3-packaging
246+
- python3-pyparsing
247247
- python3-requests-oauthlib
248248
- python3-urwid
249249
- python3-xlsxwriter

checkbox-ng/MANIFEST.in

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,12 @@ recursive-include docs *.rst *.py *.html *.conf
3737
recursive-include plainbox/test-data *.json *.html *.tar.xz
3838
recursive-exclude debian *
3939
recursive-exclude build *
40+
41+
include checkbox_ng/support/parsers/cputable
42+
recursive-include checkbox_ng/support/parsers/tests/cpuinfo_data *.txt
43+
recursive-include checkbox_ng/support/parsers/tests/dmidecode_data *.txt
44+
recursive-include checkbox_ng/support/parsers/tests/fixtures *.txt
45+
recursive-include checkbox_ng/support/parsers/tests/pactl_data *.txt
46+
recursive-include checkbox_ng/support/parsers/tests/udevadm_data *.txt *.lsblk.json
47+
recursive-include checkbox_ng/support/snap_utils/tests/asserts_data *.txt
48+
recursive-include checkbox_ng/support/parsers/tests/v4l2_compliance_data *.txt
File renamed without changes.
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# This file is part of Checkbox.
2+
#
3+
# Copyright 2024 Canonical Ltd.
4+
# Written by:
5+
# Pierre Equoy <pierre.equoy@canonical.com>
6+
#
7+
# Checkbox is free software: you can redistribute it and/or modify
8+
# it under the terms of the GNU General Public License version 3,
9+
# as published by the Free Software Foundation.
10+
#
11+
# Checkbox is distributed in the hope that it will be useful,
12+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
13+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14+
# GNU General Public License for more details.
15+
#
16+
# You should have received a copy of the GNU General Public License
17+
# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
18+
19+
import string
20+
21+
22+
def slugify(_string):
23+
"""
24+
Slugify a string
25+
26+
Transform any string to one that can be used in filenames and Python
27+
identifers.
28+
"""
29+
if not _string:
30+
return _string
31+
32+
valid_chars = frozenset(
33+
"_{}{}".format(string.ascii_letters, string.digits)
34+
)
35+
# Python identifiers cannot start with a digit
36+
if _string[0].isdigit():
37+
_string = "_" + _string
38+
return "".join(c if c in valid_chars else "_" for c in _string)

checkbox-ng/checkbox_ng/support/lib/__init__.py

Whitespace-only changes.
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# This file is part of Checkbox.
2+
#
3+
# Copyright 2008-2022 Canonical Ltd.
4+
#
5+
# Checkbox is free software: you can redistribute it and/or modify
6+
# it under the terms of the GNU General Public License version 3,
7+
# as published by the Free Software Foundation.
8+
#
9+
# Checkbox is distributed in the hope that it will be useful,
10+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
11+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12+
# GNU General Public License for more details.
13+
#
14+
# You should have received a copy of the GNU General Public License
15+
# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
16+
17+
from struct import calcsize
18+
19+
20+
def get_bitmask(key):
21+
bitmask = []
22+
for value in reversed(key.split()):
23+
value = int(value, 16)
24+
bitmask.append(value)
25+
return bitmask
26+
27+
28+
def get_bitcount(bitmask):
29+
bitcount = 0
30+
for value in bitmask:
31+
while value:
32+
bitcount += 1
33+
value &= value - 1
34+
return bitcount
35+
36+
37+
def test_bit(bit, bitmask, bits=None):
38+
if bits is None:
39+
bits = calcsize("l") * 8
40+
offset = bit % bits
41+
long = int(bit / bits)
42+
if long >= len(bitmask):
43+
return 0
44+
return (bitmask[long] >> offset) & 1
Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# This file is part of Checkbox.
2+
#
3+
# Copyright 2008-2022 Canonical Ltd.
4+
#
5+
# Checkbox is free software: you can redistribute it and/or modify
6+
# it under the terms of the GNU General Public License version 3,
7+
# as published by the Free Software Foundation.
8+
#
9+
# Checkbox is distributed in the hope that it will be useful,
10+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
11+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12+
# GNU General Public License for more details.
13+
#
14+
# You should have received a copy of the GNU General Public License
15+
# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
16+
17+
import re
18+
from datetime import datetime, timedelta
19+
20+
from checkbox_ng.support.lib.tz import tzutc
21+
22+
DATETIME_RE = re.compile(
23+
r"""
24+
^(?P<year>\d\d\d\d)-?(?P<month>\d\d)-?(?P<day>\d\d)
25+
T(?P<hour>\d\d):?(?P<minute>\d\d):?(?P<second>\d\d)
26+
(?:\.(?P<second_fraction>\d{0,6}))?
27+
(?P<tz>
28+
(?:(?P<tz_sign>[-+])(?P<tz_hour>\d\d):(?P<tz_minute>\d\d))
29+
| Z)?$
30+
""",
31+
re.VERBOSE,
32+
)
33+
34+
TYPE_FORMATS = (
35+
(r"(yes|true)", lambda v: True),
36+
(r"(no|false)", lambda v: False),
37+
(r"-?\d+", lambda v: int(v.group(0))),
38+
(r"-?\d+\.\d+", lambda v: float(v.group(0))),
39+
(r"(-?\d+) ?([kmgt]?b?)", lambda v: int(v.group(1))),
40+
(r"(-?\d+\.\d+) ?([kmgt]?b?)", lambda v: float(v.group(1))),
41+
(r"(-?\d+) ?([kmgt]?hz)", lambda v: int(v.group(1))),
42+
(r"(-?\d+\.\d+) ?([kmgt]?hz)", lambda v: float(v.group(1))),
43+
)
44+
TYPE_FORMATS = tuple(
45+
(re.compile(r"^%s$" % pattern, re.IGNORECASE), format)
46+
for pattern, format in TYPE_FORMATS
47+
)
48+
49+
TYPE_MULTIPLIERS = (
50+
(r"b", 1),
51+
(r"kb?", 1024),
52+
(r"mb?", 1024 * 1024),
53+
(r"gb?", 1024 * 1024 * 1024),
54+
(r"tb?", 1024 * 1024 * 1024 * 1024),
55+
(r"hz", 1),
56+
(r"khz?", 1024),
57+
(r"mhz?", 1024 * 1024),
58+
(r"ghz?", 1024 * 1024 * 1024),
59+
(r"thz?", 1024 * 1024 * 1024 * 1024),
60+
)
61+
TYPE_MULTIPLIERS = tuple(
62+
(re.compile(r"^%s$" % pattern, re.IGNORECASE), multiplier)
63+
for pattern, multiplier in TYPE_MULTIPLIERS
64+
)
65+
66+
67+
def datetime_to_string(dt):
68+
"""Return a consistent string representation for a given datetime.
69+
70+
:param dt: The datetime object.
71+
"""
72+
return dt.isoformat()
73+
74+
75+
def string_to_datetime(string):
76+
"""Return a datetime object from a consistent string representation.
77+
78+
:param string: The string representation.
79+
"""
80+
# we cannot use time.strptime: this function accepts neither fractions
81+
# of a second nor a time zone given e.g. as '+02:30'.
82+
match = DATETIME_RE.match(string)
83+
84+
# The Relax NG schema allows a leading minus sign and year numbers
85+
# with more than four digits, which are not "covered" by _time_regex.
86+
if not match:
87+
raise ValueError("Datetime with unreasonable value: %s" % string)
88+
89+
time_parts = match.groupdict()
90+
91+
year = int(time_parts["year"])
92+
month = int(time_parts["month"])
93+
day = int(time_parts["day"])
94+
hour = int(time_parts["hour"])
95+
minute = int(time_parts["minute"])
96+
second = int(time_parts["second"])
97+
second_fraction = time_parts["second_fraction"]
98+
if second_fraction is not None:
99+
milliseconds = second_fraction + "0" * (6 - len(second_fraction))
100+
milliseconds = int(milliseconds)
101+
else:
102+
milliseconds = 0
103+
104+
# The Relax NG validator accepts leap seconds, but the datetime
105+
# constructor rejects them. The time values submitted by the HWDB
106+
# client are not necessarily very precise, hence we can round down
107+
# to 59.999999 seconds without losing any real precision.
108+
if second > 59:
109+
second = 59
110+
milliseconds = 999999
111+
112+
dt = datetime(
113+
year, month, day, hour, minute, second, milliseconds, tzinfo=tzutc
114+
)
115+
116+
tz_sign = time_parts["tz_sign"]
117+
tz_hour = time_parts["tz_hour"]
118+
tz_minute = time_parts["tz_minute"]
119+
if tz_sign in ("-", "+"):
120+
delta = timedelta(hours=int(tz_hour), minutes=int(tz_minute))
121+
if tz_sign == "-":
122+
dt = dt + delta
123+
else:
124+
dt = dt - delta
125+
126+
return dt
127+
128+
129+
def sizeof_bytes(bytes):
130+
for x in ["bytes", "KB", "MB", "GB", "TB"]:
131+
string = "%3.1f%s" % (bytes, x)
132+
if bytes < 1024.0:
133+
break
134+
bytes /= 1024.0
135+
136+
return string
137+
138+
139+
def sizeof_hertz(hertz):
140+
for x in ["Hz", "KHz", "MHz", "GHz"]:
141+
string = "%3.1f%s" % (hertz, x)
142+
if hertz < 1000.0:
143+
break
144+
hertz /= 1000.0
145+
146+
return string
147+
148+
149+
def string_to_type(string):
150+
"""Return a typed representation for the given string.
151+
152+
The result might be a bool, int or float. The string might also be
153+
supplemented by a multiplier like KB which would return an int or
154+
float multiplied by 1024 for example.
155+
156+
:param string: The string representation.
157+
"""
158+
for regex, formatter in TYPE_FORMATS:
159+
match = regex.match(string)
160+
if match:
161+
string = formatter(match)
162+
if len(match.groups()) > 1:
163+
unit = match.group(2)
164+
for regex, multiplier in TYPE_MULTIPLIERS:
165+
match = regex.match(unit)
166+
if match:
167+
string *= multiplier
168+
break
169+
else:
170+
raise ValueError("Unknown multiplier: %s" % unit)
171+
break
172+
173+
return string

0 commit comments

Comments
 (0)