chore(pegasus-spark): Merge pegasus-spark from pegasus-kv/pegasus-spark #2206
Conversation
…apache#7) * reimplement API using RDD-style model & implement DuplicationVerifier * fix review * add comment * add comment * add comments * fix * fix * add comment
* fix travis * refactor: format codes using scalafmt
ruojieranyishen
changed the title
|
@ruojieranyishen Good job! Please fix the license header issues, see: https://github.com/apache/incubator-pegasus/actions/runs/13805996590/job/38616785844?pr=2206. |
ruojieranyishen
force-pushed
the
merge-pegasus-spark
branch
from
aaaa256 to
ec0de66
Compare
| @@ -0,0 +1,25 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
I think this file is useless because we dodn't use Travis for CI, instead, we use Github actions.
| @@ -76,6 +76,7 @@ Pegasus has support for several languages: | |||
| - [Python](https://github.com/apache/incubator-pegasus/blob/master/python-client) | |||
| - [Node.js](https://github.com/apache/incubator-pegasus/blob/master/nodejs-client) | |||
| - [Scala](https://github.com/apache/incubator-pegasus/blob/master/scala-client) | |||
| - [pegasus-spark](https://github.com/apache/incubator-pegasus/blob/master/pegasus-spark) | |||
There was a problem hiding this comment.
Is pegasus-spark a client driver? Move it to another section if it's not.
| # ================= # | ||
| pegasus-spark/*.class | ||
| # Log file | ||
| pegasus-spark/*.log |
There was a problem hiding this comment.
It would be nice it keep them in lexicographical order, thanks!
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
|
|
||
| <groupId>com.xiaomi.infra</groupId> |
|
|
||
| <groupId>com.xiaomi.infra</groupId> | ||
| <artifactId>pegasus-spark</artifactId> | ||
| <version>1.1.0-mdh-2.3.0-SNAPSHOT</version> |
There was a problem hiding this comment.
Use a completely new version because it's in a new group.
| <repository> | ||
| <id>central</id> | ||
| <name>maven-release-virtual</name> | ||
| <url>https://pkgs.d.xiaomi.net/artifactory/maven-release-virtual</url> |
| </exclusion> | ||
| </exclusions> | ||
| </dependency> | ||
| <!--FDS: Only read fds backup need the dependency--> |
There was a problem hiding this comment.
Let's implement the HDFS target at first, because most users can not use FDS.
| PROJECT_DIR=$(dirname "${SCRIPT_DIR}") | ||
| cd "${PROJECT_DIR}" || exit 1 | ||
|
|
||
| SRC_FILES=(src/main/java/com/xiaomi/infra/pegasus/spark/common/*.java |
There was a problem hiding this comment.
update the paths to org/apache/pegasus/...
| @@ -0,0 +1,43 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
Please use Github action instead, thanks!
#945
Within Xiaomi, there have been many updates to Pegasus-spark. The code needs to be merged into the main branch.
Step: