Add build and containerize pipeline for configurable streamer #61
Conversation
ValMobBIllich
had a problem deploying
to
production
GitHub Actions
Failure
ValMobBIllich
had a problem deploying
to
production
GitHub Actions
Failure
ValMobBIllich
had a problem deploying
to
production
GitHub Actions
Failure
|
I'm guessing I should hold on review till the CI passes & you give me a heads up. My assumption though! Let me know. |
Yeah I think we need to merge the other PR first so that the vsomeip stuff works again? The build and push to registry pipeline fails right now because Im using the organization name for the image URL and that has to be lowercase (our is ValtechMobility so its not...) but eclipse-uprotocol should be fine. Anyways the build pipeline wont run on PR but on push to main so it shouldnt block the PR once I change that. |
|
Hey @ValMobBIllich -- looked like CI fell over here. Perhaps need to rebase? I'd like to get CI running and then take a look :) |
|
Hey @ValMobBIllich! I'm going to start reviewing this now. Apologies if I pick over things you did a while back, I'm looking at it with fresh eyes. 🙏 |
There was a problem hiding this comment.
Hey @ValMobBIllich! Thanks for the contribution to containerize. Could you take a look at some of the suggestions I left?
| # SPDX-License-Identifier: Apache-2.0 | ||
| # *******************************************************************************/ | ||
|
|
||
| name: "Setup Rust and Cross" |
There was a problem hiding this comment.
Do we need to use cross-rs? I have the stable Rust compiler installed and a
rustc --print target-listalready shows aarch64-unknown-linux-musl.
It's also listed on the targets page here as supported.
There was a problem hiding this comment.
Hmm cross is just a low-setup way to cross compile. If we dont use cross then we have to provide the build toolchain manually for each target like its done here: https://github.com/eclipse-sdv-blueprints/fleet-management/blob/main/components/Dockerfile.fms-consumer
| push: | ||
| branches: | ||
| - main | ||
| - add_build_and_containerize_pipeline |
There was a problem hiding this comment.
This should be removed prior to merge, right? Just for testing?
There was a problem hiding this comment.
yes! its actually a bit of a problem though. We cant run the pipeline before merging it, so Im not really sure what the process of testing it before merge would be... By having it trigger on PR I could at least test that the github action itself runs through (although only on our forked repo), but not sure how I can test if the access to the main repos ghcr for example works as intended without merging to main.
mayyybe since you have more access rights here, you can manually trigger the workflow from the actions section? I did allow the "workflow_dispatch" trigger after all, but there seems to be some strange behavior in github where it might still not like being run manually...
|
|
||
| name: Containerize uStreamer and Push to Container Registry | ||
|
|
||
| on: |
There was a problem hiding this comment.
Should this be done on each PR? Or only when we have a release? Would like your feedback.
There was a problem hiding this comment.
Ideally I guess only for proper releases? Until we have a real release strategy though maybe I can just set it to "workflow_dispatch" provided that actually works. Then we can manually release whenever we feel like it.
| @@ -0,0 +1,164 @@ | |||
| # ******************************************************************************** | |||
There was a problem hiding this comment.
Most of this I'm taking on faith it's being done correctly, since I'm not familiar enough with Docker and containerizing. 😅
A basic question though -- you've tested the resulting Docker image on your end to make sure it works?
Anyone else that is more familiar with Docker I'd be super appreciative if you could take a look! cc @AnotherDaniel @sophokles73
There was a problem hiding this comment.
Yeah I tried around with the image and container by itself and found no issues yet. Im also using it in our work around the s2s blueprint and so far it behaves exactly like a streamer in a container should.
| # SPDX-License-Identifier: Apache-2.0 | ||
| # *******************************************************************************/ | ||
|
|
||
| [build.env] |
There was a problem hiding this comment.
Is this related to why you chose to use cross-rs? If we don't use cross-rs is this needed?
There was a problem hiding this comment.
Yes this is the config for cross build. Instead of having to provide the whole toolchain directly we can use the default one for each target and just append it via the config here.
ValMobBIllich
changed the title
|
Refresh my memory @ValMobBIllich -- is this ready for review again? |
There was a problem hiding this comment.
Thanks @ValMobBIllich for taking this to the finish line! I've opened #69 to track integrating up-transport-vsomeip-rust when it's ready (or to remove it if it doesn't become better maintained).
ValMobBIllich
force-pushed
the
add_build_and_containerize_pipeline
branch
2 times, most recently
from
dc63b1f to
4ae9522
Compare
- Added configurable streamer with MQTT and Zenoh support - Implemented CI/CD pipeline for container builds - Added example implementations and configurations - Reorganized project structure and utilities - Updated build system and documentation
ValMobBIllich
force-pushed
the
add_build_and_containerize_pipeline
branch
from
bf2b8a1 to
a49aa20
Compare
This PR adds a new package called the configurable streamer. This streamer implementation can be configured with any number of mqtt5 and zenoh transports and any network of message forwarding between them. In the limit of just two transports it recovers the behavior of the previously existing streamer implementations.
With the new implementation comes a pipeline that builds the streamer binary for AMD and ARM, packages it in a docker image and ships it to the github container registry from where it can be pulled without touching the source code.
The new package can be found in the new configurable-streamer/ directory along with example config files for a setup with two zenoh transports and one mqtt transport.
Additionally this PR bumps the dependencies for up-rust, up-transport-zenoh-rust, up-transport-vsomeip-rust and up-transport-mqtt5-rust. Most of the changes to the files in example-streamer-implementations/, example-streamer-uses/ and up-linux-streamer-plugin should be due to that bump (especially from up-transport-mqtt5-rust).
Some changes in example-streamer-uses/ also simplify the UUri patterns of the example entities which makes testing more transparent.
Test setups:
The someip setups currently do not work due to a missing fix in the someip transport library.