Translate presto-sdk doc to Chinese #18173
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### What changes are proposed in this pull request? Extract interface out of the only hash provider implementation. Also add one `LocalWorkerPolicy` implementation for testing. See the code comments for suitable use cases. We also extracted the number of virtual nodes in consistent hash algo to a property key so it's possible to configure the virtual node count for large clusters. ### Why are the changes needed? The new policy is for internal testing where all test clients find the local worker for IO. This policy should not be used in real deployments because if all clients find local worker, very few requests can have cache hits. ### Does this PR introduce any user facing changes? No, LocalWorkerPolicy should only be used in internal testing pr-link: Alluxio#17789 change-id: cid-b83e2b6ddb969b67318327c6f72ec47a023f5032
define how to manipulate the assembly jars inside a tarball via configuration file instead of hardcoding the map pr-link: Alluxio#17814 change-id: cid-a9309293edc8c0f1759112ab7b5438b95118ccf4
solves Alluxio#17649 pr-link: Alluxio#17670 change-id: cid-1f0acd1158d68da271007008b836c067562b9498
### What changes are proposed in this pull request? Deleted mount command links because they redirected to main page Added more resources to Get Started page Added more relevant terms to glossary ### Why are the changes needed? Mount command links are not useful. More resources in the Get Started for next steps after running locally More relevant terms to glossary ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17816 change-id: cid-5284e81b2ee2bd57b7a167a0fa54333ea37fa5a7
### What changes are proposed in this pull request? Combined Tecent COS & COSN Docs ### Why are the changes needed? Redundant information can be consolidated via tabs  ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17821 change-id: cid-dc876db94f3192130a8ec20ccca14373514eefb7
Fix Fuse docs pr-link: Alluxio#17815 change-id: cid-1b51e79e4b9fab28fe7c624fab3c7a821f1e862b
### What changes are proposed in this pull request? Remove secondary master ### Why are the changes needed? Secondary Master is no longer used ### Does this PR introduce any user facing changes? - Property key `alluxio.secondary.master.metastore.dir` is removed pr-link: Alluxio#17823 change-id: cid-6eabcd4e53d29aba7cea403ea8172a6af4a1328d
### What changes are proposed in this pull request? Stop shading alluxio-test. ### Why are the changes needed? Shading a module is slow and error-prone. The shading of the `alluxio-test` module was previously introduced in Alluxio#7765 for journal-related tests. There is no longer a need to shade this module. As part of this PR, `listStatusRootTest` (introduced in Alluxio#15361) has been moved to the S3 specific tests. This move was made because the test failed after the shading was removed on my local MacOS (although it passes CI on Linux), likely due to an unusual setting of the root directory on MacOS. I am unsure why removing shading on `alluxio-test` also impacts `alluxio-integration-tools-validation`. However, based on the description of Alluxio#15361, it does seem to fit better in the S3 specific tests. ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#17798 change-id: cid-e7570b42d9dd096af93053f84a3be226b66f9120
### What changes are proposed in this pull request? Remove the experimental feature to fallback async-write when Alluxio space is full. ### Why are the changes needed? This feature has been experimental for a few years, not been able to graduate. Given it adds a good amount of complexity, we plan to remove it. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including - Property key `alluxio.user.file.ufs.tier.enabled` is removed pr-link: Alluxio#17796 change-id: cid-72acdb41ec1164f0ec7abdd5284a580271430855
### What changes are proposed in this pull request? Change for future extension ### Why are the changes needed? Change for future extension ### Does this PR introduce any user facing changes? NA pr-link: Alluxio#17812 change-id: cid-1efc20ddcd97fd4b0d38f507f0bee5f8e883fbf9
remove the accordion , so the `toc` navbar on right side can jump to metric content correctly pr-link: Alluxio#17822 change-id: cid-e0303a9f1a217a1b6a5fc6d5d90ec05d430bf1e5
- don't allow any var args in the entrypoing scripts; specify all configurations via env vars - specify which projects to build via ALLUXIO_DOCKER_MVN_PROJECT_LIST, add -am flag to build dependent projects - specify test prefix via ALLUXIO_DOCKER_MVN_TESTS - remove ALLUXIO_DOCKER_MVN_RUNTOEND - remove "libfuse version = 2" permutation from fuse integration tests - can't pass in an env var that contains a `=`, which was attempted in the format of `ADDITIONAL_ARGS=property=key` and caused parsing failure - combine several integration test permutations to reduce overall time spent, since the permutations are finishing much faster than the slowest builds = unit test and checkstyle builds pr-link: Alluxio#17811 change-id: cid-fca8ed001d33e49867ebb1732e6a180d6726f6b5
### What changes are proposed in this pull request? Add interfaces to allow using custom modules to inject in worker ### Why are the changes needed? The `LocalAlluxioCluster` can be reused outside the repo with custom worker implemenetation. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17829 change-id: cid-6b1d30a95e2eb583cb55d077e5bf88f80cbfae97
### What changes are proposed in this pull request? add data & metadata cache fuse docs to local cache overview ### Why are the changes needed? better user flow ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17828 change-id: cid-6a5682d1efcd5d006ae913c1016c239da4bc6b9e
Monitor was merged in when we were moving stuff from `master-2.x` into `main`. pr-link: Alluxio#17836 change-id: cid-4d1eaaa0174fce8280bb97f7c5e18223d2b457d9
Normalize the HTTP RESTful API: 1) the list files API: `curl -X GET "http://<worker_ip>:<port>/v1/files?path=<path>"`  Now the server will also return the lengths of the files. 2) the get page API: `curl GET "http://<worker_ip>:<port>/v1/file/<file_id>/page/<page_index>"`  pr-link: Alluxio#17833 change-id: cid-567e6e9baf6121b3260c0c0073bbd81d94e1c363
### What changes are proposed in this pull request? 1. Remove all creation and usage of `BaseFileSystem` 2. There are some job-related APIs in `BaseFileSystem`, which are still used in 3.0. Moved those to `UfsBaseFileSystem`. 3. Removed direct users of the `BaseFileSystem`, including `MountCommand/UnmountCommand/UpdateMountCommand`. The `mount()` API on `FileSystem` will be gone. ### Why are the changes needed? `BaseFileSystem` is the main entrance of the 2.x File/Block API on the client side. It links file I/O to AlluxioFileInStream and AlluxioFileOutStream. It also routes metadata RPCs to the master. In 3.0, we introduced `UfsBaseFileSystem` and `DoraCacheFileSystem` (together with some others). We have switched to the new code path in Alluxio#17612, we remove the entrance to the old code path and say goodbye. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 4. addition or removal of property keys 5. webui pr-link: Alluxio#17830 change-id: cid-db9be604dfd12d97173c088cd1914206139de55b
### What changes are proposed in this pull request? Added additional logic for building groups of artifacts via a single command. ### Why are the changes needed? Helpful for internal automation. Allows us to create custom setups for building artifacts ### Does this PR introduce any user facing changes? Adds `dev/scripts/build-artifact.sh` `presets` command pr-link: Alluxio#17834 change-id: cid-82ad6b6a314e1b091eaba9f690facb7d7ce4f167
pr-link: Alluxio#17840 change-id: cid-6fef2b5209574d06267d2b28dbc9e9d308578818
### What changes are proposed in this pull request? move information to better fitting pages update links that were plain text in config properties ### Why are the changes needed? remove redundant/irrelevant information links that are plain text are not user friendly ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17835 change-id: cid-2722135b7789ab72b7f992e02a338dd11fd4a297
this build is failing the compilation step, claiming it can't find dora/tests jar when building dora/microbench. strange that it wasn't happening when validating Alluxio#17811 pr-link: Alluxio#17842 change-id: cid-d339245bb1f66d4dae2ce63fe41c7a77470c3a6e
### What changes are proposed in this pull request? Cleanup top level POM dependencies - Move dependency version definition to <dependencyManagement> section - Remove unused properties - Sort properties and dependencies alphabetically - Remove unused profile ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17826 change-id: cid-7646f3126f9b04e23f239fbd9fd8ccc8c6a4801b
### What changes are proposed in this pull request? Add HDFS UFS integration test using HDFS mini cluster ### Why are the changes needed? To test HDFS UFS related features better in alluxio ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#17832 change-id: cid-7f8df37f7ca6cf4c360b8d3f79edfade858f8e06
### What changes are proposed in this pull request? Rollback to use localcacheInStream for presto local cache ### Why are the changes needed? Fix the oom and out of bound issue in prestodb's local cache ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17845 change-id: cid-42a3ac6c88a44a95e21879b757e984249deee0aa
### What changes are proposed in this pull request? Add metrics for num of request to external storage when cache miss ### Why are the changes needed? For cost saving ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17846 change-id: cid-53374d78e218543bf77b458b04c5ed5b3f633457
### What changes are proposed in this pull request? Add metrics for num of client position read fallback ### Why are the changes needed? counting the number of fallback ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17847 change-id: cid-520d266df7c019a24e43035eff08904149d56f73
### What changes are proposed in this pull request? This PR removes: 1. Load jobs and SetReplica jobs, together with their corresponding `SetReplication` and `DistributedLoad` commands 2. Replication handling logic at the Master 3. JobUtils.loadXXX methods ### Why are the changes needed? These features have no place in 3.x. Replication management will be implemented on the new architecture. And Load feature has been implemented by using the master-side scheduler instead of job service. Another root reason for removing these code is we want to remove `BlockLocationPolicy`. `JobUtils.loadXXX` methods are one of the main users which directly creates `BlockLocationPolicy` objects. This will be a prerequisite for Alluxio#17831 ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 5. webui pr-link: Alluxio#17843 change-id: cid-8cb6d7098fc79761bb4f37819ae3f1cf63d28aa7
### What changes are proposed in this pull request? if catch `InvalidArgumentRuntimeException`, convert to `IllegalArgumentException` in `AbstractFileSystem.java`. if catch other `AlluxioRuntimeException`, convert to `IOException` in `AbstractFileSystem.java`. ### Why are the changes needed? make users clear about the exception type. ### Does this PR introduce any user facing changes? none pr-link: Alluxio#17788 change-id: cid-4017fc9547c0fa80be1efbc5bf5ead45910a6ae3
### What changes are proposed in this pull request? Add metrics counter for the page invalidated by TTL ### Why are the changes needed? ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17849 change-id: cid-c9b2087e31637c64885e95a8d87ca27d31d8754a
This reverts commit 479699b. ### What changes are proposed in this pull request? The PR breaks the tarball build. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#17851 change-id: cid-8cc6108924224cda31045da9bc364b548c426b3e
### What changes are proposed in this pull request? Remove hostname from metrics key ### Why are the changes needed? For easy aggregation on prometheus and grafana side ### Does this PR introduce any user facing changes? Add a flag to disable this for compatibility pr-link: Alluxio#18121 change-id: cid-ba6c2f9fae625747192044168fce7dc026c66b9c
besides the User-CLI.md doc, update other doc files that refer to `bin/alluxio` commands - remove docs on path config - remove starting/stopping job master/worker from contributor docs pr-link: Alluxio#18128 change-id: cid-fc71dd493b16ef3aeeb0b1b190941c43b9af9cab
What changes are proposed in this pull request? I have created a test and create a liststatus test for its function. Why are the changes needed? Please clarify why the changes are needed. For instance, add a unit test for DoraWorkerClientServiceHandler. Does this PR introduce any user facing changes? No. pr-link: Alluxio#18059 change-id: cid-b82706a4419700f017584f3e5579d2ef3410aeb3
Make fuse max reader concurrency configurable. The default value was 64 and it was unchangeable. pr-link: Alluxio#18129 change-id: cid-9c55821622329bd1e608da2e7445e8ab591df38a
Fix typo from alluxio.max.fuse.reader.concurrency to alluxio.fuse.max.reader.concurrency pr-link: Alluxio#18134 change-id: cid-434086cf6ba9e9f8d173e3417fc8518963dfa102
update usages of bin/alluxio, bin/alluxio-start.sh and bin/alluxio-stop.sh to their new counterparts simplify section of CephFS.md and remove sections related to mounting. the ufs must be configured as the root mount via alluxio-site.properties. pr-link: Alluxio#18136 change-id: cid-fa7d0eec00c8fb136680ef6d5a2c7ee78571d123
### What changes are proposed in this pull request? Support accessing OSS through proxy by configuring alluxio properties or system properties. ### Why are the changes needed? When accessing OSS through a proxy, the OSS client cannot recognize the proxy configuration in system property and environment variables. So it has to proactively set proxy-related configurations in the configuration. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18139 change-id: cid-5e30dfd90747d4a1aafe9b2ff985331f05fefec6
### What changes are proposed in this pull request? If don't set oss.proxy.host, the default value should be NULL ### Why are the changes needed? If don't set oss.proxy.host, the default value should be NULL ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio#18142 change-id: cid-bce2790e583445c4ba6720d2f0a64551fb19de20
### What changes are proposed in this pull request? In this change the path conversion logic is extracted to static utility methods for code reuse (because other classes may use the same path resolution logic). The method names are slightly improved, to distinguish the member methods in `DoraCacheFileSystem` (may be inherited) from the static utility methods. pr-link: Alluxio#18140 change-id: cid-557fa148f6daa41f0b296132e5a2ecae6c5d6c22
pr-link: Alluxio#18138 change-id: cid-e9f862385bdfe6cc5e6938eb49907055449deb4a
### What changes are proposed in this pull request? 1. Move the path resolution logic from `DoraCacheFileSystem` to `PathUtils` where it makes more sense 2. Fix the alluxioPathToUfsPath resolution by handling the case where the ufs path may have no matching alluxio path, making the util method more generic pr-link: Alluxio#18146 change-id: cid-ebace1efcf58e385bbf71b599e4b5a15a2199f7e
### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Better error handling when a stream is closed. Log more and don't swallow errors ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#18145 change-id: cid-b1146421ea5bc51e9a5eea5bf11ce1a5d8466912
Use jackson JSON as a standard format for reports.
Output example for `bin/alluxio info report summary`:
```
{
"mSafeMode":false,
"mZookeeper":false,
"mRaftJournal":true,
"version":"304-SNAPSHOT",
"uptime":"0 day(s), 0 hour(s), 3 minute(s), and 7 second(s)",
"rpcPort":19998,
"webPort":19999,
"masterAddress":"Ec2Cluster-masters-0:19998",
"masterVersions":[
{
"version":"304-SNAPSHOT",
"state":"PRIMARY",
"host":"Ec2Cluster-masters-0",
"port":19998
}
],
"started":"08-24-2023 06:43:32:856",
"zookeeperAddress":[
],
"raftJournalAddress":[
"Ec2Cluster-masters-0:19200"
],
"liveWorkers":2,
"lostWorkers":0,
"freeCapacity":"2048.00MB",
"totalCapacityOnTiers":{
"MEM":"2048.00MB"
},
"usedCapacityOnTiers":{
"MEM":"0B"
}
}
```
pr-link: Alluxio#18047
change-id: cid-bf6d54f47390a4d2bd84e4baac2ea2863d4638e1
|
Thank you for your pull request. |
|
Automated checks report:
Some checks failed. Please fix the reported issues and reply 'alluxio-bot, check this please' to re-run checks. |
Rosie-Wang
changed the title
|
Automated checks report:
All checks passed! |
apc999
force-pushed
the
main
branch
2 times, most recently
from
2fec0ec to
b597c61
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Add presto-sdk translation (to Chinese)
Why are the changes needed?
Add presto-sdk translation (to Chinese)
Task here: https://docs.google.com/document/d/1KoR7kPmTFGDwy9ovZThxg1Bj-z2rugH0DjV0NJDNBFw/edit?usp=sharing
Does this PR introduce any user facing changes?
No