add AK/SK Certification for uploading snapshot to cloud by liuqinguestc · Pull Request #972 · sodafoundation/api

liuqinguestc · 2019-07-23T02:13:16Z

What this PR does / why we need it:
add AK/SK Certification for uploading snapshot to cloud
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note:

codecov · 2019-07-23T03:58:31Z

Codecov Report

Merging #972 into master will increase coverage by 0.29%.
The diff coverage is 52.52%.

@@            Coverage Diff             @@
##           master     #972      +/-   ##
==========================================
+ Coverage   34.32%   34.61%   +0.29%     
==========================================
  Files          89       92       +3     
  Lines       16608    16858     +250     
==========================================
+ Hits         5700     5835     +135     
- Misses      10096    10197     +101     
- Partials      812      826      +14

Impacted Files	Coverage Δ
contrib/backup/multicloud/client.go	`0% <0%> (ø)`	⬆️
contrib/drivers/lvm/lvm.go	`23.66% <0%> (-0.34%)`	⬇️
...icloud/credentials/keystonecredentials/provider.go	`65.9% <65.9%> (ø)`
...ontrib/backup/multicloud/credentials/credential.go	`69.23% <69.23%> (ø)`
contrib/backup/multicloud/signer/signature.go	`71.85% <71.85%> (ø)`

skdwriting · 2019-07-23T04:49:53Z

Can you please attach the test report for this PR?

wisererik · 2019-07-23T05:50:19Z

@liuqinguestc I don't think your solution is correct when considering the identification when hotpot uploads snapshot to cloud through multi-cloud. IMO, signature should be generated in hotpot which will be passed to multi-cloud and identified in multi-cloud.

liuqinguestc · 2019-07-23T07:06:30Z

@wisererik I just do as you said , I will delete some codes that do not need.

xing-yang · 2019-07-27T21:08:08Z

Please add a release note.

xxwjj · 2019-07-29T02:45:07Z

@@ -0,0 +1,189 @@
+// Copyright 2019 The OpenSDS Authors.


client.go has the same feature code, please use that one.

sorry，I can not use that, because the code is not same, and function is not the same.

xxwjj · 2019-07-29T02:46:16Z

@@ -0,0 +1,37 @@
+// Copyright 2019 The OpenSDS Authors.


fileter is used for api-server client, please remove this code.

sorry，this is used for keystone.go, I can not remove this code.

xxwjj · 2019-07-29T02:49:56Z

  unused-packages = true
+
+[[constraint]]
+  name = "github.com/emicklei/go-restful"


why do we need go-restfull lib?

just for getting signature. the code uses go-restful lib.

xxwjj · 2019-07-29T02:51:01Z

@@ -0,0 +1,61 @@
+// Copyright 2019 The OpenSDS Authors.


please remove this file.

this file is used by auth.go

xxwjj · 2019-07-29T02:52:35Z

+	"strings"
+
+	"github.com/opensds/opensds/contrib/backup/multicloud/auth"
+


remove blank line

xxwjj · 2019-07-29T02:55:25Z

 	Write   = "Write"
 	Execute = "Execute"
+
+	//Signature parameter name


these constants is only used for upload to snapshot to cloud, please move to contrib/backup/multi-cloud directory.

but as erik said. we should use public lib

rhsakarpos · 2019-07-31T03:58:13Z

  version = "2.0.0"

 [[constraint]]
+  branch = "master"


this PR is too huge for a single commit. 16K+ lines of code, please split it up.

rhsakarpos · 2019-07-31T03:58:34Z

@@ -0,0 +1,37 @@
+// Copyright 2019 The OpenSDS Authors.
+//


OpenSDS copyright missing

Is these lines written newly by us?

OpenSDS copyright missing

Its there in the line 1

rhsakarpos · 2019-07-31T03:58:50Z

@@ -0,0 +1,189 @@
+// Copyright 2019 The OpenSDS Authors.
+//


rhsakarpos · 2019-07-31T03:59:17Z

+
+const (
+	ConfFile             = "/etc/opensds/driver/multi-cloud.yaml"
+	DefaultUploadTimeout = 30 // in Seconds


please add to conf

rhsakarpos · 2019-07-31T03:59:42Z

+
+func (k *Keystone) loadConf(p string) (*MultiCloudConf, error) {
+	conf := &MultiCloudConf{
+		Endpoint:      "http://127.0.0.1:8088",


rhsakarpos · 2019-07-31T04:01:32Z

+			return nil, nil, err
+		}
+
+		requestDate := time.Now().UTC().Format("20060102")


why are the dates and times hard-coded ?

rhsakarpos · 2019-07-31T04:04:13Z

+		// and a termination string ("sign_request") in lowercase characters.
+		sign.credentialString,
+		//Step 4: Append the hash of the canonical request created in Task 1
+		hex.EncodeToString(makeSha256([]byte(sign.canonicalString))),


is it secure to store in String ? in JAVA it is recommended to store in char[], since due to GC not being done, the value of the String variable can be seen by a memory profiler

rhsakarpos · 2019-07-31T04:04:39Z

+func (sign *Signature) buildSignature() {
+	// Step 1:  Create the signing key, use the secret access key to create a series of
+	// hash-based message authentication codes (HMACs).
+	kSecret := sign.credValues.SecretAccessKey


error handling ? in case any makeHMAC fails ?

rhsakarpos · 2019-07-31T04:05:15Z

+	// hash-based message authentication codes (HMACs).
+	kSecret := sign.credValues.SecretAccessKey
+	kDate := makeHmac([]byte("OPENSDS"+kSecret), []byte(sign.requestDate))
+	kRegion := makeHmac(kDate, []byte(sign.Region))


key and value not matching here, please check all

rhsakarpos · 2019-07-31T04:06:05Z

+}
+
+// stripExcessSpaces will trim multiple side-by-side spaces.
+func stripExcessSpaces(vals []string) {


can we use strings.TrimSpace(s) ?

rhsakarpos · 2019-07-31T04:06:40Z

+	var canonicalURI string
+
+	if len(url.Opaque) > 0 {
+		canonicalURI = "/" + strings.Join(strings.Split(url.Opaque, "/")[3:], "/")


significance of [3:] ?

skdwriting

Why 142 file changes for this?! Kindly update the points in the PR description as well.

skdwriting · 2019-08-01T11:57:04Z

  version = "2.0.0"

 [[constraint]]
+  branch = "master"


skdwriting · 2019-08-01T11:57:55Z

@@ -0,0 +1,37 @@
+// Copyright 2019 The OpenSDS Authors.
+//


Is these lines written newly by us?

skdwriting · 2019-08-01T11:58:32Z

@@ -0,0 +1,37 @@
+// Copyright 2019 The OpenSDS Authors.
+//


OpenSDS copyright missing

Its there in the line 1

skdwriting · 2019-08-01T12:04:26Z

+
+func (k *Keystone) loadConf(p string) (*MultiCloudConf, error) {
+	conf := &MultiCloudConf{
+		Endpoint:      "http://127.0.0.1:8088",


Why hard-coded?

skdwriting · 2019-08-01T12:09:37Z

+	var r tokens.GetResult
+	// The service token may be expired or revoked, so retry to get new token.
+	err := utils.Retry(2, "verify token", false, func(retryIdx int, lastErr error) error {
+		if retryIdx > 0 {


What are you trying to do here. I see some confusion here!

liuqinguestc added 11 commits July 17, 2019 07:46

add vendor dependences

963e7f4

add multi-cloud dependence

4b83c4b

add vendor dependences

6c00b6a

add snapshot upload function

c5f9eb6

add snapshot upload file

11bcc97

modify multi-cloud.yaml

d256ce8

optimizing the code.

e780435

Merge branch 'master' of https://github.com/liuqinguestc/opensds

530ce6a

modify driver_test error

d69e517

modify the provider_test error

86fbd78

modify signature_test error

7932ac4

liuqinguestc added 2 commits July 23, 2019 16:51

Optimize the codes for uploading snapshot to cloud

3b3207f

Optimize the codes for uploading snapshot to cloud

816dc7f

xing-yang added the feature there is a huge framework change or feature addition label Jul 27, 2019

xxwjj reviewed Jul 29, 2019

View reviewed changes

rhsakarpos reviewed Jul 31, 2019

View reviewed changes

skdwriting reviewed Aug 1, 2019

View reviewed changes

skdwriting requested changes Aug 1, 2019

View reviewed changes

		"strings"

		"github.com/opensds/opensds/contrib/backup/multicloud/auth"

Conversation

liuqinguestc commented Jul 23, 2019

Uh oh!

codecov Bot commented Jul 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

skdwriting commented Jul 23, 2019

Uh oh!

wisererik commented Jul 23, 2019

Uh oh!

liuqinguestc commented Jul 23, 2019

Uh oh!

xing-yang commented Jul 27, 2019

Uh oh!

xxwjj Jul 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skdwriting left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jul 23, 2019 •

edited

Loading

xxwjj Jul 29, 2019 •

edited

Loading