Skip to content

Commit 43046ac

Browse files
wuhuizuopre-commit-ci[bot]gemini-code-assist[bot]
authored
feat(publisher): add image multi arch pusher (#343)
This pull request adds support for a new asynchronous multi-architecture image collection feature in the publisher service, including both API and CLI support. It also updates dependencies and refactors service initialization to accommodate the new functionality. **New multi-architecture image collection feature:** * Adds two new endpoints to the image service: `request-multiarch-collect` (to start collecting a multi-arch image, with async/sync modes) and `query-multiarch-collect-status` (to check the status of an async collection). Updates the service design, HTTP client, and CLI tooling to support these endpoints. [[1]](diffhunk://#diff-662a3768fb9c0121f844e385f0163667e10875a440ffe2bccba86928f71f8af1R143-R190) [[2]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L30-R47) [[3]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8R87-R92) [[4]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8R106-R107) [[5]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8R176-R181) [[6]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8R235-R240) [[7]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8R351-R352) [[8]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L357-R405) [[9]](diffhunk://#diff-84c768b792bed35a7ac29f4a1b66b33428d68ecf953f48bc9e2fd033e9773eafR55-R104) [[10]](diffhunk://#diff-18014c63cbbfca0660ba93571952ac19c83f86197de2a819b1426f7670cf4570R29-R36) [[11]](diffhunk://#diff-18014c63cbbfca0660ba93571952ac19c83f86197de2a819b1426f7670cf4570R59-R60) [[12]](diffhunk://#diff-18014c63cbbfca0660ba93571952ac19c83f86197de2a819b1426f7670cf4570R111-R153) **Service initialization and internal refactoring:** * Refactors the initialization of `imgSvc` in `main.go` to accept the same arguments as other services, removing the previous use of a fixed timeout. [[1]](diffhunk://#diff-6c8650af650b5ae5b805a1959ba3a1d16c2be84e0ffa3bdb78038ff9402c13dcL93-R92) [[2]](diffhunk://#diff-6c8650af650b5ae5b805a1959ba3a1d16c2be84e0ffa3bdb78038ff9402c13dcL13) **Dependency updates:** * Updates several dependencies in `go.mod`, including `github.com/docker/cli`, `github.com/opencontainers/image-spec`, `google.golang.org/grpc`, and others, to newer versions. [[1]](diffhunk://#diff-ed390b3cd3f7051800bee91f419e6a2052876af79b5100fb366922fc00491ab7L26-R32) [[2]](diffhunk://#diff-ed390b3cd3f7051800bee91f419e6a2052876af79b5100fb366922fc00491ab7L46) [[3]](diffhunk://#diff-ed390b3cd3f7051800bee91f419e6a2052876af79b5100fb366922fc00491ab7L64-R65) [[4]](diffhunk://#diff-ed390b3cd3f7051800bee91f419e6a2052876af79b5100fb366922fc00491ab7L82-R84) **CLI improvements:** * Enhances the CLI tool with new commands and usage documentation for the new endpoints, including help text, flag parsing, and example payloads. [[1]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L30-R47) [[2]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L254-R277) [[3]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L269-R289) [[4]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L306-R326) [[5]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L318-R338) [[6]](diffhunk://#diff-e74abd2247ad723a5d1c29161ac2062e50782d3340650fd88d037bc07746bcb8L344-R367) [[7]](diffhunk://#diff-84c768b792bed35a7ac29f4a1b66b33428d68ecf953f48bc9e2fd033e9773eafL27-R27) These changes introduce a robust mechanism for collecting multi-architecture images, supporting both synchronous and asynchronous workflows, and ensure that both API consumers and CLI users can leverage the new functionality. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1 parent 7b63658 commit 43046ac

File tree

36 files changed

+1750
-221
lines changed

36 files changed

+1750
-221
lines changed

‎chatops-lark/pkg/events/handler/devbuild_trigger.go‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import (
44
"context"
55
"flag"
66
"fmt"
7+
"slices"
78
"strings"
89

910
"github.com/go-resty/resty/v2"
@@ -152,6 +153,10 @@ func parseCommandDevbuildTrigger(args []string) (*triggerParams, error) {
152153
return nil, NewInformationError(devBuildHelpText)
153154
}
154155

156+
if ret.engine == "" {
157+
ret.engine = defaultEngineForProduct(ret.product)
158+
}
159+
155160
if err := ret.Verify(); err != nil {
156161
return nil, err
157162
}
@@ -160,3 +165,11 @@ func parseCommandDevbuildTrigger(args []string) (*triggerParams, error) {
160165

161166
return &ret, nil
162167
}
168+
169+
func defaultEngineForProduct(s string) string {
170+
if slices.Contains([]string{"tikv"}, s) {
171+
return "tekton"
172+
}
173+
174+
return ""
175+
}

‎chatops-lark/pkg/events/handler/devbuild_trigger_test.go‎

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -31,20 +31,20 @@ func TestParseCommandDevbuildTrigger(t *testing.T) {
3131
wantErr: false,
3232
},
3333
{
34-
name: "missing product flag",
35-
args: []string{"-version", "v6.1.0", "-gitRef", "master"}, // missing product
34+
name: "missing product flag",
35+
args: []string{"-version", "v6.1.0", "-gitRef", "master"}, // missing product
3636
expected: nil,
3737
wantErr: true,
3838
},
3939
{
40-
name: "missing version flag",
41-
args: []string{"-product", "tidb", "-gitRef", "master"}, // missing version
40+
name: "missing version flag",
41+
args: []string{"-product", "tidb", "-gitRef", "master"}, // missing version
4242
expected: nil,
4343
wantErr: true,
4444
},
4545
{
46-
name: "missing gitRef flag",
47-
args: []string{"-product", "tidb", "-version", "v6.1.0"}, // missing gitRef
46+
name: "missing gitRef flag",
47+
args: []string{"-product", "tidb", "-version", "v6.1.0"}, // missing gitRef
4848
expected: nil,
4949
wantErr: true,
5050
},
@@ -237,6 +237,31 @@ func TestParseCommandDevbuildTrigger(t *testing.T) {
237237
},
238238
wantErr: false,
239239
},
240+
{
241+
name: "tikv default use tekton engine when not hotfix",
242+
args: []string{
243+
"-product", "tikv",
244+
"-version", "v8.5.4",
245+
"-gitRef", "master",
246+
"-e", "community",
247+
"-p", "linux/amd64",
248+
},
249+
expected: &triggerParams{
250+
product: "tikv",
251+
version: "v8.5.4",
252+
gitRef: "master",
253+
edition: "community",
254+
platform: "linux/amd64",
255+
buildEnvs: []string{},
256+
pushGCR: false,
257+
hotfix: false,
258+
dryRun: false,
259+
githubRepo: "",
260+
features: "",
261+
engine: "tekton",
262+
},
263+
wantErr: false,
264+
},
240265
{
241266
name: "no arguments",
242267
args: []string{},

‎publisher/cmd/publisher/main.go‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@ import (
1010
"os/signal"
1111
"sync"
1212
"syscall"
13-
"time"
1413

1514
"github.com/go-redis/redis/v8"
1615
"github.com/rs/zerolog"
@@ -90,7 +89,7 @@ func main() {
9089

9190
tiupSvc = impltiup.NewService(&logger, kafkaWriter, redisClient, config.EventSource)
9291
fsSvc = implfs.NewService(&logger, kafkaWriter, redisClient, config.EventSource)
93-
imgSvc = implimg.NewService(&logger, redisClient, time.Hour)
92+
imgSvc = implimg.NewService(&logger, kafkaWriter, redisClient, config.EventSource)
9493
}
9594

9695
// Wrap the services in endpoints that can be invoked from other services

‎publisher/go.mod‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,10 @@ require (
2626
github.com/docker/cli v28.2.2+incompatible // indirect
2727
github.com/docker/distribution v2.8.3+incompatible // indirect
2828
github.com/docker/docker-credential-helpers v0.9.3 // indirect
29+
github.com/fsnotify/fsnotify v1.7.0 // indirect
2930
github.com/mitchellh/go-homedir v1.1.0 // indirect
3031
github.com/pkg/errors v0.9.1 // indirect
32+
github.com/rogpeppe/go-internal v1.13.1 // indirect
3133
github.com/sirupsen/logrus v1.9.3 // indirect
3234
github.com/vbatts/tar-split v0.12.1 // indirect
3335
github.com/yuin/gopher-lua v1.1.1 // indirect
@@ -43,7 +45,6 @@ require (
4345
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
4446
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
4547
github.com/dimfeld/httppath v0.0.0-20170720192232-ee938bf73598 // indirect
46-
github.com/fsnotify/fsnotify v1.7.0 // indirect
4748
github.com/go-chi/chi/v5 v5.1.0 // indirect
4849
github.com/go-logr/logr v1.4.3 // indirect
4950
github.com/gorilla/websocket v1.5.3 // indirect
@@ -79,7 +80,7 @@ require (
7980
golang.org/x/text v0.26.0 // indirect
8081
golang.org/x/time v0.5.0 // indirect
8182
golang.org/x/tools v0.34.0 // indirect
82-
google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53 // indirect
83-
google.golang.org/grpc v1.67.1 // indirect
83+
google.golang.org/genproto/googleapis/rpc v0.0.0-20241021214115-324edc3d5d38 // indirect
84+
google.golang.org/grpc v1.68.1 // indirect
8485
google.golang.org/protobuf v1.36.3 // indirect
8586
)

‎publisher/go.sum‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,8 @@ github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq
5454
github.com/go-redsync/redsync/v4 v4.13.0 h1:49X6GJfnbLGaIpBBREM/zA4uIMDXKAh1NDkvQ1EkZKA=
5555
github.com/go-redsync/redsync/v4 v4.13.0/go.mod h1:HMW4Q224GZQz6x1Xc7040Yfgacukdzu7ifTDAKiyErQ=
5656
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
57+
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
58+
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
5759
github.com/gomodule/redigo v1.8.9 h1:Sl3u+2BI/kk+VEatbj0scLdrFhjPmbxOc1myhDP41ws=
5860
github.com/gomodule/redigo v1.8.9/go.mod h1:7ArFNvsTjH8GMMzB4uy1snslv2BwmginuMs06a1uzZE=
5961
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
@@ -124,8 +126,8 @@ github.com/redis/go-redis/v9 v9.5.1 h1:H1X4D3yHPaYrkL5X06Wh6xNVM/pX0Ft4RV0vMGvLB
124126
github.com/redis/go-redis/v9 v9.5.1/go.mod h1:hdY0cQFCN4fnSYT6TkisLufl/4W5UIXyv0b/CLO2V2M=
125127
github.com/redis/rueidis v1.0.19 h1:s65oWtotzlIFN8eMPhyYwxlwLR1lUdhza2KtWprKYSo=
126128
github.com/redis/rueidis v1.0.19/go.mod h1:8B+r5wdnjwK3lTFml5VtxjzGOQAC+5UmujoD12pDrEo=
127-
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
128-
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
129+
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
130+
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
129131
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
130132
github.com/rs/zerolog v1.33.0 h1:1cU2KZkvPxNyfgEmhHAz/1A9Bz+llsdYzklWFzgp0r8=
131133
github.com/rs/zerolog v1.33.0/go.mod h1:/7mN4D5sKwJLZQ2b/znpjC3/GQWY/xaDXUM0kKWRHss=
@@ -207,10 +209,10 @@ golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=
207209
golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
208210
golang.org/x/tools v0.34.0 h1:qIpSLOxeCYGg9TrcJokLBG4KFA6d795g0xkBkiESGlo=
209211
golang.org/x/tools v0.34.0/go.mod h1:pAP9OwEaY1CAW3HOmg3hLZC5Z0CCmzjAF2UQMSqNARg=
210-
google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53 h1:X58yt85/IXCx0Y3ZwN6sEIKZzQtDEYaBWrDvErdXrRE=
211-
google.golang.org/genproto/googleapis/rpc v0.0.0-20241015192408-796eee8c2d53/go.mod h1:GX3210XPVPUjJbTUbvwI8f2IpZDMZuPJWDzDuebbviI=
212-
google.golang.org/grpc v1.67.1 h1:zWnc1Vrcno+lHZCOofnIMvycFcc0QRGIzm9dhnDX68E=
213-
google.golang.org/grpc v1.67.1/go.mod h1:1gLDyUQU7CTLJI90u3nXZ9ekeghjeM7pTDZlqFNg2AA=
212+
google.golang.org/genproto/googleapis/rpc v0.0.0-20241021214115-324edc3d5d38 h1:zciRKQ4kBpFgpfC5QQCVtnnNAcLIqweL7plyZRQHVpI=
213+
google.golang.org/genproto/googleapis/rpc v0.0.0-20241021214115-324edc3d5d38/go.mod h1:GX3210XPVPUjJbTUbvwI8f2IpZDMZuPJWDzDuebbviI=
214+
google.golang.org/grpc v1.68.1 h1:oI5oTa11+ng8r8XMMN7jAOmWfPZWbYpCFaMUTACxkM0=
215+
google.golang.org/grpc v1.68.1/go.mod h1:+q1XYFJjShcqn0QZHvCyeR4CXPA+llXIeUIfIe00waw=
214216
google.golang.org/protobuf v1.36.3 h1:82DV7MYdb8anAVi3qge1wSnMDrnKK7ebr+I0hHRN1BU=
215217
google.golang.org/protobuf v1.36.3/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
216218
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

‎publisher/internal/service/design/design.go‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -256,4 +256,52 @@ var _ = Service("image", func() {
256256
Response(StatusOK)
257257
})
258258
})
259+
260+
Method("request-multiarch-collect", func() {
261+
Payload(func() {
262+
Attribute("image_url", String, "The image URL to collect")
263+
Attribute("release_tag_suffix", String, func() {
264+
Description("Suffix for the release tag")
265+
Default("release")
266+
})
267+
Attribute("async", Boolean, func() {
268+
Description("Whether to run the collection asynchronously. If true, returns a request id. If false or omitted, runs synchronously and returns the result directly.")
269+
Default(false)
270+
})
271+
Required("image_url")
272+
})
273+
Result(func() {
274+
Attribute("async", Boolean, func() {
275+
Description("Whether to run the collection asynchronously. If true, returns a request id. If false or omitted, runs synchronously and returns the result directly.")
276+
Default(false)
277+
})
278+
// If async is true, request_id is required; if false, repo and tags are required.
279+
Attribute("repo", String, "Repository of the collected image")
280+
Attribute("tags", ArrayOf(String), "Tags of the collected image")
281+
Attribute("request_id", String, func() {
282+
Description("Request id for async mode (uuidv4 format)")
283+
Format(FormatUUID)
284+
})
285+
Required("async")
286+
})
287+
HTTP(func() {
288+
POST("/collect-multiarch")
289+
Response(StatusOK)
290+
})
291+
})
292+
Method("query-multiarch-collect-status", func() {
293+
Payload(func() {
294+
Attribute("request_id", String, "Request track id", func() {
295+
Format(FormatUUID)
296+
})
297+
Required("request_id")
298+
})
299+
Result(String, "request state", func() {
300+
Enum("queued", "processing", "success", "failed", "canceled")
301+
})
302+
HTTP(func() {
303+
GET("/collect-multiarch/{request_id}")
304+
Response(StatusOK)
305+
})
306+
})
259307
})

‎publisher/internal/service/gen/http/cli/publisher/cli.go‎

Lines changed: 60 additions & 12 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎publisher/internal/service/gen/http/fileserver/client/cli.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)