-
Notifications
You must be signed in to change notification settings - Fork 71
Support upload image to aws #601
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
tuteng
wants to merge
35
commits into
master
Choose a base branch
from
fixed/support-upload-image-to-aws
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
b9b2eaf
Support upload image to aws
tuteng 92477a8
Fixed file permission
tuteng b87f632
Update action yml file
tuteng 9d4f108
Update image action yml file
tuteng 41dfd0b
Update version check
tuteng 727ee97
Test download image
tuteng e65d751
Update var
tuteng 1587acb
Update upload file
tuteng 59d166f
Fixed upload bucket name
tuteng 3ae1ca6
Debug ali oss
tuteng c379c8c
Fixed command
tuteng 9cbc3d0
Update ali oss config
tuteng 43f7c8c
Fixed bucket
tuteng 53f4efb
Fixed dest path
tuteng 5d3c125
Fixed image save command
tuteng c972c47
Fixed image list
tuteng 2258992
Fixed command
tuteng 926c651
Fixed image upload
tuteng 5018550
Update image, and add trigger action
tuteng 3ae05c2
Update version
tuteng a7e362d
Update oss version
tuteng 0653752
Remove no used image
tuteng f1aec2d
Update image list
tuteng ce90c4f
Fixed images list
tuteng 4131292
Fixed trigger action
tuteng 9756a4b
Fixed echo download info
tuteng 379d61e
Fixed sn-platform info
tuteng d9d6056
trigger image upload
tuteng ce62bbf
Update vault operator image
tuteng 36a1391
Fixed operator image
tuteng c1ca157
Fixed vault operator image
tuteng a7a0024
Fixed image
tuteng 80ce0d2
Echo image list
tuteng 76849da
Push 2.9.2.12 image
tuteng c9098c7
Fixed 2.9.2.13 image
tuteng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| #!/bin/bash | ||
| components=( | ||
| zookeeper | ||
| zookeeper.customTools.backup | ||
| zookeeper.customTools.restore | ||
| bookie | ||
| presto | ||
| autorecovery | ||
| broker | ||
| proxy | ||
| pulsar_detector | ||
| functions | ||
| function_worker | ||
| toolset | ||
| toolset.kafka | ||
| prometheus | ||
| alert_manager | ||
| grafana | ||
| streamnative_console | ||
| node_exporter | ||
| nginx_ingress_controller | ||
| vault | ||
| vault_init | ||
| bank_vaults | ||
| custom_metric_server_prometheus | ||
| custom_metric_server | ||
| pulsar_metadata | ||
| configmapReload | ||
| external_dns | ||
| functionnmesh | ||
| pulsar_operator | ||
| bookkeeper_operator | ||
| zookeeper_operator | ||
| cert_manager_controller | ||
| cert_manager_cainjector | ||
| cert_manager_webhook | ||
| vault_operator) | ||
| sn_platform_tag=$1 | ||
| regexp="^sn-platform-([0-9]{1,3})\.([0-9]{0,3})\.([0-9]{0,3})$" | ||
| mkdir -p $sn_platform_tag | ||
| cd $sn_platform_tag | ||
| # if [[ $sn_platform_tag =~ $regexp ]]; then | ||
| # curl https://raw.githubusercontent.com/streamnative/charts/${sn_platform_tag}/charts/sn-platform/values.yaml -o values.yaml | ||
| curl https://raw.githubusercontent.com/streamnative/charts/fixed/support-upload-image-to-aws/charts/sn-platform/values.yaml -o values.yaml | ||
| image_list="" | ||
| for i in ${components[@]}; do | ||
| repository=$(cat values.yaml | yq .images.$i.repository) | ||
| tag=$(cat values.yaml | yq .images.$i.tag) | ||
| echo "Downloading docker image: $repository:$tag" | ||
| image_list=$image_list" $repository:$tag" | ||
| docker pull $repository:$tag | ||
| done | ||
|
|
||
| uniq_image_list=($(echo ${image_list[*]} | sed 's/ /\n/g'|sort| uniq)) | ||
| echo ${uniq_image_list} | ||
| for j in ${uniq_image_list[@]}; do | ||
| image=$(echo ${j} | sed 's/:/-/g;s/\//-/g'); | ||
| docker save -o ${image}.tar ${j} | ||
| echo https://downloads-streamnative-cloud.oss-cn-beijing.aliyuncs.com/sn-products/$sn_platform_tag/${image}.tar | ||
| done | ||
| # test | ||
| # docker save -o $sn_platform_tag.tar jimmidyson/configmap-reload:v0.3.0 | ||
| # else | ||
| # echo "Please use the correct version number format => sn-platform-x.x.x" | ||
| # echo "Currently only sn-platform images are supported for upload" | ||
| # fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| # curl -X POST https://api.github.com/repos/streamnative/sn-platform/dispatches -d '{"event_type":"upload-image-to-s3-and-oss","tag":"sn-platform-1.2.38"}' | ||
| name: Trigger Upload Image to aws s3 and oss | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: | ||
| - '*' | ||
| # repository_dispatch: | ||
| # types: | ||
| # - upload-image-to-s3-and-oss | ||
|
|
||
| jobs: | ||
| deploy: | ||
| name: Upload sn platform image | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@master | ||
|
|
||
| - name: Setup environments | ||
| run: | | ||
| sudo apt update | ||
| sudo snap install yq | ||
|
|
||
| - name: Download and package all image for sn platform | ||
| run: | # .ci/download_image.sh sn-platform-1.4.0-rc3 | ||
| .ci/download_image.sh sn-platform-1.4.0-rc3 | ||
|
|
||
| - uses: jakejarvis/s3-sync-action@master | ||
| with: | ||
| args: --acl public-read --follow-symlinks --delete | ||
| env: | ||
| AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} | ||
| AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
| AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| AWS_REGION: 'us-east-1' # optional: defaults to us-east-1 | ||
| SOURCE_DIR: sn-platform-1.4.0-rc3 | ||
| DEST_DIR: sn-platform-1.4.0-rc3 | ||
|
|
||
| - name: Install Alibaba Cloud OSSUTIL | ||
| run: wget http://gosspublic.alicdn.com/ossutil/1.7.10/ossutil64 && chmod +x ossutil64 | ||
| - name: Configure Alibaba Cloud OSSUTIL | ||
| run: ./ossutil64 config -i ${ALI_ACCESS_KEY_ID} -k ${ALI_SECRET_ACCESS_KEY} -e ${ALI_OSS_ENDPOINT} -c .ossutilconfig | ||
| env: | ||
| ALI_OSS_ENDPOINT: ${{ secrets.ALI_OSS_ENDPOINT }} | ||
| ALI_ACCESS_KEY_ID: ${{ secrets.ALI_ACCESS_KEY_ID }} | ||
| ALI_SECRET_ACCESS_KEY: ${{ secrets.ALI_SECRET_ACCESS_KEY }} | ||
| - name: Upload the file to the chosen OSS bucket | ||
| run: ./ossutil64 --config-file .ossutilconfig cp ${{ github.workspace }}/sn-platform-1.4.0-rc3 oss://${{ secrets.ALI_OSS_BUCKET }}/sn-products/sn-platform-1.4.0-rc3 -r -f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you should default this or throw a help message & exit if not provided. Otherwise, someone has to open up the file to see what arguments this needs and what they are for.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still working on this test and have some code left to finish, thanks for the suggestion, it seems I should set this pr to draft status