libyang: upgrade to v3 step 1 -- import libyang3 / libyang3-py3, remove stale versions#21679
Conversation
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@ganglyu @qiluo-msft please review ... this needs to be merged before I can submit PRs for the migration of the libyang 1.0.73 code to v3 ... |
|
Hi @bradh352 , could you help me understand why |
ganglyu
requested review from
StormLiangMS,
ganglyu,
liushilongbuaa and
wen587
|
Thank you very much for your contribution! We have encountered numerous issues with libyang 1.0.73, which is currently used by sonic-yang-models. On the other hand, libyang 1.0.184 and libyang 2.1.148, which are used by FRR, have not presented any issues to our knowledge. My suggestion is to install libyang3 in this PR without replacing the existing dependency. |
I honestly believe not having multiple versions of libyang to be extremely important. I think their API is stabilizing. The only thing required was to apply some commits already upstream in FRR 10.2 to support 3.7.8. From the commit message:
I think we'd be foolish to kick the can down the road while we're working on it now. I also haven't verified if libyang2 and libyang3 can be installed simultaneously like libyang1 and libyang3 can. Of course if you tell me there's no way you could possibly accept this PR because of this you leave me little choice but I don't see a sufficient argument at this time against it. |
From the commit message:
So basically, its just to make the review of the upgrade to libyang3 easier. I've actually got libyang3 working and passing these test cases so I wanted a bulk of the test case changes done here to make it easier to review the more complex porting of the libyang1 python swig module to libyang3 python cffi. |
|
Guess I need to figure out python wheel packaging for cross compilation to support marvell armhf (its an architecture-specific package since it uses cffi)? I clearly am not testing such a use case currently in my local environment. Anyone have any pointers for this? |
I will engage the owner of SONiC FRR to review the related changes. |
I'm not an expert, but I believe the Sonic build pipeline uses the armhf system to build the Sonic image for Marvell armhf. Therefore, we shouldn't encounter any cross-compilation issues in the Sonic build pipeline. |
Looks like azure isn't actually cross compiling at all, its running an armhf image that is advertising armv8l support which I think just means the kernel knows its a 64bit cpu ... but the system was compiled for armv7l/armhf, so python is I think determining the system based on I tried using a 64bit arm vm on my macbook to build sonic, but it immediately fails as it thinks I need to use QEMU multiarch ... even hacking the makefile to disable that and it fails later on. Parallels doesn't allow me to directly install an armhf image, so I guess I'm dead in the water trying to test this on my own hardware right now. I might need to pick up an rpi5 or something to build but that seems like it might be really really slow.... So I'll probably try to throw in some test commits to see if a simple 'setarch armhf' works around the issue. |
bradh352
force-pushed
the
bradh352/libyang3-only
branch
2 times, most recently
from
801cc52 to
4a1b620
Compare
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
bradh352
force-pushed
the
bradh352/libyang3-only
branch
from
4a1b620 to
86f10cd
Compare
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
This does the base upgrade from libyang2 to libyang3. It also drops the completely unused libyang1 build (as the version used is in the libyang directory which is 1.0.73). NOTE: libyang3 depends on pkgconf, which supersedes pkg-config so there is some churn to docker image building switching packages but it works for all packages as a replacement.
This patch was rejected upstream as it is considered a bad practice. However, in the sonic-mgmt-common code, it has chosen to bypass using libyang for a lot of validation, so this flag is necessary in this use case. sonic-mgmt-common will likely switch to goyang in the future.
We omit the _UNINSTALLS because it messes up dependency tracking with other packages that depend on libyang3
libyang 1.0.73 provided python bindings via swig. Those have
been removed and libyang3 now requires the use of the cffi bindings
which are in a separate repo.
The binding names themselves are actually different ('yang' vs 'libyang')
so both should be installable simultaneously.
This is built as a debian package rather than a wheel since it contains
binary elements.
CESNET/libyang-python#134 needed for backwards compatibility for existing configurations.
We need to force libyang3-py3 to be built and tested, and confirm it can properly co-exist with libyang1 and its python bindings. Its currently not used, but will be in a followup.
This mostly simplifies the existing test cases and standardizes behavior where they may be incompatible with libyang3. The main goal is to have this in place before the migration to libyang3 to make it easier to evaluate the actual code review for porting aspects for libyang3. Convert some yang model test cases that use `eStr` for failure matching to use `eStrKey` instead to standardize expected failure matching so when messsages change they can be updated in one place instead of all over. Split out some new `eStrKey` values that are needed for libyang3 due to different messages, and document the libyang3 values needed once upgraded but keep libyang1 values for now. Finally, some tests were using both `eStr` and `eStrKey` and the `eStr` portion is not needed to perform a meaningful validation, plus the `eStr` in these cases are not valid for libyang3.
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
I keep getting spurious test failures that aren't related to this PR. I guess I'll just keep on forcing rebuilds all tests pass. |
|
@qiluo-msft looks like all test cases passed on the last run if you could do a final review and hopefully merge this. |
|
/azpw ms_conflict |
1 similar comment
|
/azpw ms_conflict |
|
@qiluo-msft this is merge-ready. You approved in the past but I had upstream a fix to buster builds (not sure why buster is even built as it is EOL). Since this is targeted for 202511 and there are 17 other PRs that depend on this to complete the libyang3 migration it is imperative this be merged sooner rather than later if that goal is going to be accomplished. |
| $(LIBYANG3)_SRC_PATH = $(SRC_PATH)/libyang3 | ||
| SONIC_MAKE_DEBS += $(LIBYANG3) | ||
|
|
||
| LIBYANG3_DEV = libyang-dev_$(LIBYANG3_FULLVERSION)_$(CONFIGURED_ARCH).deb |
There was a problem hiding this comment.
Is there any reason we're breaking naming convention for the dev header package?
There was a problem hiding this comment.
What convention are you referring to? The package is from upstream debian, so they set the naming convention for the generate .debs, so this variable simply sets the appropriate name.
There was a problem hiding this comment.
See also https://packages.debian.org/source/testing/libyang, where Debian chose the name libyang-dev for the dev header package.
Note also that this is because they only support libyang 3 going forward in Trixie, so they don't need to consider providing support for libyang 1 and 2 headers and libraries.
There was a problem hiding this comment.
See also https://packages.debian.org/source/testing/libyang, where Debian chose the name
libyang-devfor the dev header package.Note also that this is because they only support libyang 3 going forward in Trixie, so they don't need to consider providing support for libyang 1 and 2 headers and libraries.
well that and they conflict so both can't be installed.
There was a problem hiding this comment.
Ah typically the "libxxx" portion of the package name stays consistent between the library, header, and debug packages (so I was expecting "libyang3-dev..."). I didn't realize they aren't doing this anymore, my mistake.
| SPATH := $($(LIBYANG3_PY3)_SRC_PATH) | ||
| DEP_FILES := $(SONIC_COMMON_FILES_LIST) rules/libyang3-py3.mk rules/libyang3-py3.dep | ||
| DEP_FILES += $(SONIC_COMMON_BASE_FILES_LIST) | ||
| SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && find . -type f -exec sh -c 'git ls-files --error-unmatch "$0" >/dev/null 2>&1' {} \; -printf '%P\n')) |
There was a problem hiding this comment.
What is the reason for using find . -exec 'git ls-files --error-unmatch' for excluding the files that are not tracked by git?
Could we use git ls-files to achieve the same goal using a single subprocess?
There was a problem hiding this comment.
honestly I didn't evaluate the smdep_files logic at all, I just ripped it from another .dep and it worked the way I needed it to.
11 participants
Why I did it
Upgrade to libyang3
See master tracking ticket #22385
Fixes #22385
Work item tracking
N/A
How I did it
Important: Please review each commit (including commit message) individually. Looking at the patch-set as a whole may cause confusion.
How to verify it
See that the build still succeeds and existing test cases still pass
Which release branch to backport (provide reason below if selected)
N/A
Tested branch (Please provide the tested image version)
master as of 20250613
Description for the changelog
libyang: upgrade to v3 step 1
Link to config_db schema for YANG module changes
N/A
A picture of a cute animal (not mandatory but encouraged)
Signed-off-by: Brad House (@bradh352)