Skip to content

Commit 42cdb3c

Browse files
haha-zwx-ooohaha-zwx-ooo
authored
Merge pull request #108 from haha-zwx-ooo/main
repo-sync-2024-07-16T11:18:14+0800
2 parents 8e8fb47 + 488ba4c commit 42cdb3c

File tree

169 files changed

+4603
-2051
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

169 files changed

+4603
-2051
lines changed

‎.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,10 @@ docs/_build
5858
/db/secretpad.sqlite
5959
/db/secretpad.sqlite-shm
6060
/db/secretpad.sqlite-wal
61+
/secretpad-common/intermediate.crt
62+
/secretpad-common/intermediate.csr
63+
/secretpad-common/intermediate.key
64+
/secretpad-common/root.crt
65+
/secretpad-common/root.key
66+
/secretpad-common/root.srl
67+
/config/ehcache-data/

‎CHANGELOG.md‎

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1414
`Fixed` for any bug fixes.
1515
`Security` in case of vulnerabilities.
1616

17-
## [0.8.1b0] - 2024-06-28
18-
### Fixed
17+
## [0.9.0b0] - 2024-07-15
18+
19+
### Added
20+
21+
- [Feature] Added support for tweedie training in SGD component.
22+
- [Feature] Multi-node architecture restructured with gRPC and logging/monitoring enhancements.
1923

20-
- [Fixed] Fix incompatible sql issue
24+
### Changed
25+
26+
- [Optimization] Single-sided operator support & proto protocol upgrade (dependent on Kuscia).
2127

28+
## [0.8.1b0] - 2024-06-28
29+
30+
### Fixed
2231

32+
- [Fixed] Fix incompatible sql issue
2333

2434
## [0.8.0b0] - 2024-06-15
2535

@@ -39,8 +49,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3949

4050
### Fixed
4151

42-
- [Fixed] Fix the version prompt on the edge node in center deploy mode.
43-
- [Fixed] Fix the official website documentation has some blank sections.
52+
- [Fixed] Fix the version prompt on the edge node in center deploy mode.
53+
- [Fixed] Fix the official website documentation has some blank sections.
4454

4555
## [0.7.0b0] - 2024-05-15
4656

@@ -116,7 +126,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
116126
- Fix graph edge del error
117127
- Fix tee result download error
118128

119-
## [v0.4.1b0] - 2024-1-26
129+
## [v0.4.1b0] - 2024-01-26
120130

121131
### Added
122132

@@ -130,7 +140,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
130140

131141
- Fix the missing SGD prediction tag column
132142

133-
## [v0.4.0b0] - 2024-1-12
143+
## [v0.4.0b0] - 2024-01-12
134144

135145
### Added
136146

@@ -165,7 +175,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
165175

166176
- Overall UI interface display optimization.
167177

168-
## [0.2.0b0] - 2023-9-6
178+
## [0.2.0b0] - 2023-09-06
169179

170180
### Added
171181

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99

1010
SecretPad (official site: [secretflow.org.cn](https://www.secretflow.org.cn/zh-CN/docs/secretpad/))
1111
is a **privacy-preserving computing** web platform based on
12-
the [Kuscia](https://github.com/secretflow/kuscia)
12+
the [Kuscia](https://www.secretflow.org.cn/zh-CN/docs/kuscia/)
1313
framework, designed to provide easy access to privacy-preserving data intelligence and machine learning functions. With
1414
SecretPad:
1515

@@ -102,6 +102,7 @@ For more details, see [Development](./docs/deployment_experience/v0.6.0b0/deploy
102102
103103
| secretpad Version | Kuscia Version | secretflow Version | trustedflow Version | serving Version |
104104
|-------------------|----------------|--------------------|---------------------|-----------------|
105+
| 0.9.0b0 | 0.10.0b0 | 1.8.0b0 | 0.1.1b0 | 0.5.0b0 |
105106
| 0.8.0b0/0.8.1b0 | 0.9.0b0 | 1.7.0b0 | 0.1.1b0 | 0.4.0b0 |
106107
| 0.7.1b0/0.7.2b0 | 0.8.0b0 | 1.6.1b0 | 0.1.1b0 | 0.3.1b0 |
107108
| 0.6.0b0 | 0.7.0b0 | 1.5.0b0 | 0.1.1b0 | 0.2.1b0 |

‎README.zh-CN.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
</p>
99

1010
[SecretPad](https://www.secretflow.org.cn/zh-CN/docs/secretpad/)
11-
是一个基于 [Kuscia](https://github.com/secretflow/kuscia) 的隐私计算的 web 框架，能够方便的使用基于保护隐私的数据智能和机器学习的能力。通过
11+
是一个基于 [Kuscia](https://www.secretflow.org.cn/zh-CN/docs/kuscia) 的隐私计算的 web 框架，能够方便的使用基于保护隐私的数据智能和机器学习的能力。通过
1212
SecretPad：
1313

1414
* 你可以快速创建节点、注册数据、创建项目、进行合作节点间的授权
@@ -92,6 +92,7 @@ sh install.sh p2p -n domainId -s 8099 -g 8092 -k 8091 -p 8090 -q 33801
9292
9393
| secretpad Version | Kuscia Version | secretflow Version | trustedflow Version | serving Version |
9494
|-------------------|----------------|--------------------|---------------------|-----------------|
95+
| 0.9.0b0 | 0.10.0b0 | 1.8.0b0 | 0.1.1b0 | 0.5.0b0 |
9596
| 0.8.0b0/0.8.1b0 | 0.9.0b0 | 1.7.0b0 | 0.1.1b0 | 0.4.0b0 |
9697
| 0.7.1b0/0.7.2b0 | 0.8.0b0 | 1.6.1b0 | 0.1.1b0 | 0.3.1b0 |
9798
| 0.6.0b0 | 0.7.0b0 | 1.5.0b0 | 0.1.1b0 | 0.2.1b0 |

‎config/application-dev.yaml‎

Lines changed: 26 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,12 +19,31 @@ secretpad:
1919
logs:
2020
path: ../log
2121

22-
kusciaapi:
23-
protocol: ${KUSCIA_PROTOCOL:notls} # notls tls mtls install.sh -P
24-
address: 127.0.0.1:18083 # install.sh -g kuscia-api-grpc (kuscia-master) (used in autonomy or master or edge mode)
22+
kuscia:
23+
nodes:
24+
- domainId: ${NODE_ID:kuscia-system}
25+
mode: master
26+
host: ${KUSCIA_API_ADDRESS:root-kuscia-master}
27+
port: ${KUSCIA_API_PORT:8083}
28+
protocol: ${KUSCIA_PROTOCOL:tls}
29+
cert-file: config/certs/client.crt
30+
key-file: config/certs/client.pem
31+
token: config/certs/token
2532

26-
kusciaapi-lite-alice:
27-
address: 127.0.0.1:28083 # install.sh -g kuscia-api-grpc (kuscia-lite-alice) (Only used in master mode default node alice)
33+
- domainId: alice
34+
mode: lite
35+
host: ${KUSCIA_API_LITE_ALICE_ADDRESS:root-kuscia-lite-alice}
36+
port: ${KUSCIA_API_PORT:8083}
37+
protocol: ${KUSCIA_PROTOCOL:tls}
38+
cert-file: config/certs/alice/client.crt
39+
key-file: config/certs/alice/client.pem
40+
token: config/certs/alice/token
2841

29-
kusciaapi-lite-bob:
30-
address: 127.0.0.1:38083 # install.sh -g kuscia-api-grpc (kuscia-lite-bob) (Only used in master mode default node bob)
42+
- domainId: bob
43+
mode: lite
44+
host: ${KUSCIA_API_LITE_BOB_ADDRESS:root-kuscia-lite-bob}
45+
port: ${KUSCIA_API_PORT:8083}
46+
protocol: ${KUSCIA_PROTOCOL:tls}
47+
cert-file: config/certs/bob/client.crt
48+
key-file: config/certs/bob/client.pem
49+
token: config/certs/bob/token

‎config/application-edge.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,17 @@ secretpad:
1313
center: true
1414
p2p: false
1515

16+
kuscia:
17+
nodes:
18+
- domainId: ${NODE_ID}
19+
mode: lite
20+
host: ${KUSCIA_API_ADDRESS:root-kuscia-master}
21+
port: ${KUSCIA_API_PORT:8083}
22+
protocol: ${KUSCIA_PROTOCOL:tls}
23+
cert-file: config/certs/client.crt
24+
key-file: config/certs/client.pem
25+
token: config/certs/token
26+
1627
data:
1728
sync:
1829
- org.secretflow.secretpad.persistence.entity.ProjectDO

‎config/application-p2p.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,17 @@ secretpad:
2222
# retryPolicy : Keep trying until you succeed
2323
retry: fastFailedPolicy
2424

25+
kuscia:
26+
nodes:
27+
- domainId: ${NODE_ID}
28+
mode: p2p
29+
host: ${KUSCIA_API_ADDRESS:root-kuscia-autonomy}
30+
port: ${KUSCIA_API_PORT:8083}
31+
protocol: ${KUSCIA_PROTOCOL:tls}
32+
cert-file: config/certs/client.crt
33+
key-file: config/certs/client.pem
34+
token: config/certs/token
35+
2536
data:
2637
sync:
2738
- org.secretflow.secretpad.persistence.entity.ProjectDO

‎config/application.yaml‎

Lines changed: 33 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@ server:
33
accesslog:
44
enabled: true
55
directory: /var/log/secretpad
6+
servlet:
7+
session:
8+
timeout: 30m
69
http-port: 8080
710
http-port-inner: 9001
811
port: 443
@@ -32,13 +35,14 @@ spring:
3235
properties:
3336
hibernate:
3437
format_sql: false
38+
open-in-view: false
3539
datasource:
3640
driver-class-name: org.sqlite.JDBC
3741
url: jdbc:sqlite:./db/secretpad.sqlite
3842
hikari:
3943
idle-timeout: 60000
4044
maximum-pool-size: 1
41-
connection-timeout: 5000
45+
connection-timeout: 6000
4246
flyway:
4347
baseline-on-migrate: true
4448
locations:
@@ -89,38 +93,38 @@ management:
8993
exposure:
9094
include: health,info,readiness,prometheus
9195
enabled-by-default: false
96+
kusciaapi:
97+
protocol: ${KUSCIA_PROTOCOL:tls}
9298

93-
logging:
94-
level:
95-
root: info
96-
org:
97-
hibernate:
98-
type: info
99+
kuscia:
100+
nodes:
101+
- domainId: ${NODE_ID:kuscia-system}
102+
mode: master
103+
host: ${KUSCIA_API_ADDRESS:root-kuscia-master}
104+
port: ${KUSCIA_API_PORT:8083}
105+
protocol: ${KUSCIA_PROTOCOL:tls}
106+
cert-file: config/certs/client.crt
107+
key-file: config/certs/client.pem
108+
token: config/certs/token
99109

100-
kusciaapi:
101-
address: ${KUSCIA_API_ADDRESS:root-kuscia-master:8083}
102-
protocol: ${KUSCIA_PROTOCOL:tls} # notls tls mtls
103-
tls:
104-
cert-file: config/certs/client.crt
105-
key-file: config/certs/client.pem
106-
ca-file: config/certs/ca.crt
107-
token-file: config/certs/token
110+
- domainId: alice
111+
mode: lite
112+
host: ${KUSCIA_API_LITE_ALICE_ADDRESS:root-kuscia-lite-alice}
113+
port: ${KUSCIA_API_PORT:8083}
114+
protocol: ${KUSCIA_PROTOCOL:tls}
115+
cert-file: config/certs/alice/client.crt
116+
key-file: config/certs/alice/client.pem
117+
token: config/certs/alice/token
108118

109-
kusciaapi-lite-alice:
110-
address: ${KUSCIA_API_LITE_ALICE_ADDRESS:root-kuscia-lite-alice:8083}
111-
tls:
112-
cert-file: config/certs/alice/client.crt
113-
key-file: config/certs/alice/client.pem
114-
ca-file: config/certs/alice/ca.crt
115-
token-file: config/certs/alice/token
119+
- domainId: bob
120+
mode: lite
121+
host: ${KUSCIA_API_LITE_BOB_ADDRESS:root-kuscia-lite-bob}
122+
port: ${KUSCIA_API_PORT:8083}
123+
protocol: ${KUSCIA_PROTOCOL:tls}
124+
cert-file: config/certs/bob/client.crt
125+
key-file: config/certs/bob/client.pem
126+
token: config/certs/bob/token
116127

117-
kusciaapi-lite-bob:
118-
address: ${KUSCIA_API_LITE_BOB_ADDRESS:root-kuscia-lite-bob:8083}
119-
tls:
120-
cert-file: config/certs/bob/client.crt
121-
key-file: config/certs/bob/client.pem
122-
ca-file: config/certs/bob/ca.crt
123-
token-file: config/certs/bob/token
124128

125129
job:
126130
max-parallelism: 1

‎config/components/secretflow.json‎

Lines changed: 45 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -916,7 +916,7 @@
916916
"attrs": [
917917
{
918918
"name": "feature_selects",
919-
"desc": "which features should be binned.",
919+
"desc": "which features should be binned. WARNING: WOE won't be effective for features with enumeration count <=2.",
920920
"colMinCntInclusive": "1"
921921
},
922922
{
@@ -1737,7 +1737,7 @@
17371737
"domain": "ml.train",
17381738
"name": "sgb_train",
17391739
"desc": "Provides both classification and regression tree boosting (also known as GBDT, GBM)\nfor vertical split dataset setting by using secure boost.\n- SGB is short for SecureBoost. Compared to its safer counterpart SS-XGB, SecureBoost focused on protecting label holder.\n- Check https://arxiv.org/abs/1901.08755.",
1740-
"version": "0.0.3",
1740+
"version": "0.0.4",
17411741
"attrs": [
17421742
{
17431743
"name": "num_boost_round",
@@ -1806,7 +1806,8 @@
18061806
"allowedValues": {
18071807
"ss": [
18081808
"linear",
1809-
"logistic"
1809+
"logistic",
1810+
"tweedie"
18101811
]
18111812
}
18121813
}
@@ -1893,8 +1894,15 @@
18931894
"isOptional": true,
18941895
"defaultValue": {},
18951896
"lowerBoundEnabled": true,
1896-
"lowerBound": {},
1897-
"lowerBoundInclusive": true
1897+
"lowerBound": {
1898+
"f": -10.0
1899+
},
1900+
"lowerBoundInclusive": true,
1901+
"upperBoundEnabled": true,
1902+
"upperBound": {
1903+
"f": 10.0
1904+
},
1905+
"upperBoundInclusive": true
18981906
}
18991907
},
19001908
{
@@ -2095,7 +2103,7 @@
20952103
},
20962104
{
20972105
"name": "eval_metric",
2098-
"desc": "Use what metric for monitoring and early stop? Currently support ['roc_auc', 'rmse', 'mse']",
2106+
"desc": "Use what metric for monitoring and early stop? Currently support ['roc_auc', 'rmse', 'mse', 'tweedie_deviance', 'tweedie_nll']",
20992107
"type": "AT_STRING",
21002108
"atomic": {
21012109
"isOptional": true,
@@ -2106,7 +2114,9 @@
21062114
"ss": [
21072115
"roc_auc",
21082116
"rmse",
2109-
"mse"
2117+
"mse",
2118+
"tweedie_deviance",
2119+
"tweedie_nll"
21102120
]
21112121
}
21122122
}
@@ -2161,6 +2171,25 @@
21612171
"lowerBoundInclusive": true
21622172
}
21632173
},
2174+
{
2175+
"name": "tweedie_variance_power",
2176+
"desc": "Parameter that controls the variance of the Tweedie distribution.",
2177+
"type": "AT_FLOAT",
2178+
"atomic": {
2179+
"isOptional": true,
2180+
"defaultValue": {
2181+
"f": 1.5
2182+
},
2183+
"lowerBoundEnabled": true,
2184+
"lowerBound": {
2185+
"f": 1.0
2186+
},
2187+
"upperBoundEnabled": true,
2188+
"upperBound": {
2189+
"f": 2.0
2190+
}
2191+
}
2192+
},
21642193
{
21652194
"name": "save_best_model",
21662195
"desc": "Whether to save the best model on validation set during training.",
@@ -3302,8 +3331,15 @@
33023331
"isOptional": true,
33033332
"defaultValue": {},
33043333
"lowerBoundEnabled": true,
3305-
"lowerBound": {},
3306-
"lowerBoundInclusive": true
3334+
"lowerBound": {
3335+
"f": -10.0
3336+
},
3337+
"lowerBoundInclusive": true,
3338+
"upperBoundEnabled": true,
3339+
"upperBound": {
3340+
"f": 10.0
3341+
},
3342+
"upperBoundInclusive": true
33073343
}
33083344
},
33093345
{

0 commit comments

Comments
 (0)