fix(sec): upgrade com.google.guava:guava to 32.0.0-jre #18137
+141,105
−202,323
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
replaced by ./dev/scripts/generate-artifacts.sh, which utilizes the code in dev/scripts/src/alluxio.org/build/ pr-link: Alluxio#17807 change-id: cid-9d426410dcd14032fa2c2eff26edc795340ffb85
### What changes are proposed in this pull request? Support multipart upload of S3, OSS, COS and OBS. ### Why are the changes needed? 1. support multipart upload mode of s3. Simple upload refers to [this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html). Multipart upload refer to [this](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) 2. Multipart upload improve streaming upload mode. Multipart upload remove temp file, that optimizes three read and write disk operations into one. Upload each time the input data reaches the `partition size`, and use Netty zero-copy technology for splicing. 3. OSS, COS and OBS are similar as described to the above. **There are some experiments conducted on Mac Laptop and AWS instance. Simple upload (default) and streaming upload are two original upload methods of alluxio.** File Size: 4.8GB MinIO (Mac Laptop): Simple Upload: 28 seconds Streaming Upload: 20 seconds Multipart Upload: 12 seconds AWS same region (r6a.xlarge): Simple Upload: 25 seconds Streaming Upload: 18 seconds Multipart Upload: 12 seconds In an environment with sufficient bandwidth (or an intranet environment), the speed increase is obvious. ### Does this PR introduce any user facing changes? `alluxio.underfs.object.store.multipart.upload.timeout`: Timeout for uploading part when using multipart upload. S3: `alluxio.underfs.s3.multipart.upload.enabled`: Whether to enable multipart upload for S3. If it is `true`, then multipart upload of S3 will be enabled. Defult value is `false`. `alluxio.underfs.s3.multipart.upload.partition.size`: Multipart upload partition size for S3. The default partition size is `64MB`. OSS: `alluxio.underfs.oss.multipart.upload.enabled`: Whether to enable multipart upload for OSS. `alluxio.underfs.oss.multipart.upload.threads`: Thread pool size for OSS multipart upload. `alluxio.underfs.oss.multipart.upload.partition.size`: Multipart upload partition size for OSS. The default partition size is 64MB. COS: `alluxio.underfs.cos.multipart.upload.enabled`: Whether to enable multipart upload for COS. `alluxio.underfs.cos.multipart.upload.threads`: Thread pool size for COS multipart upload. `alluxio.underfs.cos.multipart.upload.partition.size`: Multipart upload partition size for COS. The default partition size is 64MB. OBS: `alluxio.underfs.obs.multipart.upload.enabled`: Whether to enable multipart upload for OBS. `alluxio.underfs.obs.multipart.upload.threads`: Thread pool size for OBS multipart upload. `alluxio.underfs.obs.multipart.upload.partition.size`: Multipart upload partition size for OBS. The default partition size is 64MB. pr-link: Alluxio#17447 change-id: cid-6ba62f951bd232682dd88457ea5bc39517eab3c2
### What changes are proposed in this pull request? - Clients write data to worker and worker write data to local paging cache and UFS. Added a switch to control this. The default way is client writing directly to UFS. - WriteType.THROUGH is handled. - InAlluxioPercentage and inMemoryPercentage is set to 100 when file length is zero. ### Why are the changes needed? So Alluxio workers can do some control, e.g. security, flow, quota, on worker. This also avoid managing UFS access tokens on clients, minimize performance bottles on client. TODO: worker decides writing data to local paging cache or remote UFS based on different write types. For example, for CACHE_THROUGH, data is going to local paging store and UFS. For THROUGH, data is ONLY going to UFS. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio#17748 change-id: cid-596fd0fa730e5c6bbc0efba37b62343040649549
### What changes are proposed in this pull request? Add a new constructor in FileSystemContextReinitializer ### Why are the changes needed? Add a new constructor in FileSystemContextReinitializer, the ConfigHashSync can be created by the caller. ### Does this PR introduce any user facing changes? NA pr-link: Alluxio#17810 change-id: cid-5fe3cfb2eec50f3ab720f1e3606045200f6edf06
the table exceeds its own div on current the page:  fixed example without format:  with format:  pr-link: Alluxio#17803 change-id: cid-00a42d9fe3ec12cabc97efeb545116f785b022b7
…storage integrations ### What changes are proposed in this pull request? Updated code blocks from console to shell for all doc pages for consistency & coloring. Added syntax highlighting for missing code blocks. Added descriptions for storage integrations. Added table formatting on some pages for better UI. ### Why are the changes needed? Some code blocks were missing syntax highlighting and not consistent throughout all of docs. Tables looked odd as plain text, so used table formatting & structure (ie. Documentation Conventions) ### Does this PR introduce any user facing changes? Yes, docs UI pr-link: Alluxio#17808 change-id: cid-0370cc1f0810bbf3733ac3edeecb03874c4c61a9
Handle reserved character for HTTP server. Convert "%2F" to "/" in the path parameter. pr-link: Alluxio#17809 change-id: cid-e09a5c08a6b92d7762774120e87f093bde940ed7
### What changes are proposed in this pull request? Add get functions in S3 underfs ### Why are the changes needed? Add get functions in S3 underfs pr-link: Alluxio#17813 change-id: cid-0771e279854e690aff04476196abb075aefc10ba
### What changes are proposed in this pull request? Extract interface out of the only hash provider implementation. Also add one `LocalWorkerPolicy` implementation for testing. See the code comments for suitable use cases. We also extracted the number of virtual nodes in consistent hash algo to a property key so it's possible to configure the virtual node count for large clusters. ### Why are the changes needed? The new policy is for internal testing where all test clients find the local worker for IO. This policy should not be used in real deployments because if all clients find local worker, very few requests can have cache hits. ### Does this PR introduce any user facing changes? No, LocalWorkerPolicy should only be used in internal testing pr-link: Alluxio#17789 change-id: cid-b83e2b6ddb969b67318327c6f72ec47a023f5032
define how to manipulate the assembly jars inside a tarball via configuration file instead of hardcoding the map pr-link: Alluxio#17814 change-id: cid-a9309293edc8c0f1759112ab7b5438b95118ccf4
solves Alluxio#17649 pr-link: Alluxio#17670 change-id: cid-1f0acd1158d68da271007008b836c067562b9498
### What changes are proposed in this pull request? Deleted mount command links because they redirected to main page Added more resources to Get Started page Added more relevant terms to glossary ### Why are the changes needed? Mount command links are not useful. More resources in the Get Started for next steps after running locally More relevant terms to glossary ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17816 change-id: cid-5284e81b2ee2bd57b7a167a0fa54333ea37fa5a7
### What changes are proposed in this pull request? Combined Tecent COS & COSN Docs ### Why are the changes needed? Redundant information can be consolidated via tabs  ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17821 change-id: cid-dc876db94f3192130a8ec20ccca14373514eefb7
Fix Fuse docs pr-link: Alluxio#17815 change-id: cid-1b51e79e4b9fab28fe7c624fab3c7a821f1e862b
### What changes are proposed in this pull request? Remove secondary master ### Why are the changes needed? Secondary Master is no longer used ### Does this PR introduce any user facing changes? - Property key `alluxio.secondary.master.metastore.dir` is removed pr-link: Alluxio#17823 change-id: cid-6eabcd4e53d29aba7cea403ea8172a6af4a1328d
### What changes are proposed in this pull request? Stop shading alluxio-test. ### Why are the changes needed? Shading a module is slow and error-prone. The shading of the `alluxio-test` module was previously introduced in Alluxio#7765 for journal-related tests. There is no longer a need to shade this module. As part of this PR, `listStatusRootTest` (introduced in Alluxio#15361) has been moved to the S3 specific tests. This move was made because the test failed after the shading was removed on my local MacOS (although it passes CI on Linux), likely due to an unusual setting of the root directory on MacOS. I am unsure why removing shading on `alluxio-test` also impacts `alluxio-integration-tools-validation`. However, based on the description of Alluxio#15361, it does seem to fit better in the S3 specific tests. ### Does this PR introduce any user facing changes? n/a pr-link: Alluxio#17798 change-id: cid-e7570b42d9dd096af93053f84a3be226b66f9120
### What changes are proposed in this pull request? Remove the experimental feature to fallback async-write when Alluxio space is full. ### Why are the changes needed? This feature has been experimental for a few years, not been able to graduate. Given it adds a good amount of complexity, we plan to remove it. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including - Property key `alluxio.user.file.ufs.tier.enabled` is removed pr-link: Alluxio#17796 change-id: cid-72acdb41ec1164f0ec7abdd5284a580271430855
### What changes are proposed in this pull request? Change for future extension ### Why are the changes needed? Change for future extension ### Does this PR introduce any user facing changes? NA pr-link: Alluxio#17812 change-id: cid-1efc20ddcd97fd4b0d38f507f0bee5f8e883fbf9
remove the accordion , so the `toc` navbar on right side can jump to metric content correctly pr-link: Alluxio#17822 change-id: cid-e0303a9f1a217a1b6a5fc6d5d90ec05d430bf1e5
- don't allow any var args in the entrypoing scripts; specify all configurations via env vars - specify which projects to build via ALLUXIO_DOCKER_MVN_PROJECT_LIST, add -am flag to build dependent projects - specify test prefix via ALLUXIO_DOCKER_MVN_TESTS - remove ALLUXIO_DOCKER_MVN_RUNTOEND - remove "libfuse version = 2" permutation from fuse integration tests - can't pass in an env var that contains a `=`, which was attempted in the format of `ADDITIONAL_ARGS=property=key` and caused parsing failure - combine several integration test permutations to reduce overall time spent, since the permutations are finishing much faster than the slowest builds = unit test and checkstyle builds pr-link: Alluxio#17811 change-id: cid-fca8ed001d33e49867ebb1732e6a180d6726f6b5
### What changes are proposed in this pull request? Add interfaces to allow using custom modules to inject in worker ### Why are the changes needed? The `LocalAlluxioCluster` can be reused outside the repo with custom worker implemenetation. ### Does this PR introduce any user facing changes? no pr-link: Alluxio#17829 change-id: cid-6b1d30a95e2eb583cb55d077e5bf88f80cbfae97
### What changes are proposed in this pull request? add data & metadata cache fuse docs to local cache overview ### Why are the changes needed? better user flow ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17828 change-id: cid-6a5682d1efcd5d006ae913c1016c239da4bc6b9e
Monitor was merged in when we were moving stuff from `master-2.x` into `main`. pr-link: Alluxio#17836 change-id: cid-4d1eaaa0174fce8280bb97f7c5e18223d2b457d9
Normalize the HTTP RESTful API: 1) the list files API: `curl -X GET "http://<worker_ip>:<port>/v1/files?path=<path>"`  Now the server will also return the lengths of the files. 2) the get page API: `curl GET "http://<worker_ip>:<port>/v1/file/<file_id>/page/<page_index>"`  pr-link: Alluxio#17833 change-id: cid-567e6e9baf6121b3260c0c0073bbd81d94e1c363
### What changes are proposed in this pull request? 1. Remove all creation and usage of `BaseFileSystem` 2. There are some job-related APIs in `BaseFileSystem`, which are still used in 3.0. Moved those to `UfsBaseFileSystem`. 3. Removed direct users of the `BaseFileSystem`, including `MountCommand/UnmountCommand/UpdateMountCommand`. The `mount()` API on `FileSystem` will be gone. ### Why are the changes needed? `BaseFileSystem` is the main entrance of the 2.x File/Block API on the client side. It links file I/O to AlluxioFileInStream and AlluxioFileOutStream. It also routes metadata RPCs to the master. In 3.0, we introduced `UfsBaseFileSystem` and `DoraCacheFileSystem` (together with some others). We have switched to the new code path in Alluxio#17612, we remove the entrance to the old code path and say goodbye. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 4. addition or removal of property keys 5. webui pr-link: Alluxio#17830 change-id: cid-db9be604dfd12d97173c088cd1914206139de55b
### What changes are proposed in this pull request? Added additional logic for building groups of artifacts via a single command. ### Why are the changes needed? Helpful for internal automation. Allows us to create custom setups for building artifacts ### Does this PR introduce any user facing changes? Adds `dev/scripts/build-artifact.sh` `presets` command pr-link: Alluxio#17834 change-id: cid-82ad6b6a314e1b091eaba9f690facb7d7ce4f167
pr-link: Alluxio#17840 change-id: cid-6fef2b5209574d06267d2b28dbc9e9d308578818
### What changes are proposed in this pull request? move information to better fitting pages update links that were plain text in config properties ### Why are the changes needed? remove redundant/irrelevant information links that are plain text are not user friendly ### Does this PR introduce any user facing changes? Web UI pr-link: Alluxio#17835 change-id: cid-2722135b7789ab72b7f992e02a338dd11fd4a297
this build is failing the compilation step, claiming it can't find dora/tests jar when building dora/microbench. strange that it wasn't happening when validating Alluxio#17811 pr-link: Alluxio#17842 change-id: cid-d339245bb1f66d4dae2ce63fe41c7a77470c3a6e
### What changes are proposed in this pull request? Cleanup top level POM dependencies - Move dependency version definition to <dependencyManagement> section - Remove unused properties - Sort properties and dependencies alphabetically - Remove unused profile ### Does this PR introduce any user facing changes? No pr-link: Alluxio#17826 change-id: cid-7646f3126f9b04e23f239fbd9fd8ccc8c6a4801b
Reverts Alluxio#18070 revert per request on backward compatibility pr-link: Alluxio#18119 change-id: cid-aeecbff3bc9d6bff41eec25405e87a2f2b79079f
### What changes are proposed in this pull request? Refactor the Netty read handler of worker. ### Why are the changes needed? The previous implementation creates a state machine per read request, instead of per channel. This implies that if two read requests are sent over the same channel, the worker would possibly use one channel to send data of different files or regions. This can lead to data corruption. This PR proposes to use a state machine per channel, and handles channel events throughout the whole lifecycle of the channel. Things like a faulty client sending a second request over the same channel before the first request is completed, is handled gracefully with a client error. The state transitions look like the following:  ### Does this PR introduce any user facing changes? No. ### Tests Tests have been done with basic Alluxio CLI tools, as well as automated PrestoDB and TPC-DS tests. pr-link: Alluxio#17479 change-id: cid-bb8b2c70f0bf0bd84e73d9e858bf6e80706427aa
Refactor Netty read handler to allow subclassing. pr-link: Alluxio#18120 change-id: cid-d0d4125459644d8e92c04dd9decdb622f14cbdb6
### What changes are proposed in this pull request? Implement per-thread cache context ### Why are the changes needed? Enable the fine-grained cache admission ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18029 change-id: cid-839bc71b2df158a4aeaedf22c5c7fb40dfd769e8
Fuse is supported on K8s. Remove the outdated limitation. pr-link: Alluxio#18122 change-id: cid-99cacf64405867a720a5304ab5604b778c0b2127
### What changes are proposed in this pull request? Change `runClass` in Benchmark to `exec class` ### Why are the changes needed? Since new alluxio cli change from `runClass` to `exec class`, related benchmark code should be change as well to keep it work ### Does this PR introduce any user facing changes? no pr-link: Alluxio#18079 change-id: cid-2e8b23a7fc272ed0266c81f5e816442fdb9cd25b
`uname -m` on arm == `arm64` pr-link: Alluxio#18123 change-id: cid-c10f5760052ff7cc3757843df86710200d70b090
### What changes are proposed in this pull request? Remove hostname from metrics key ### Why are the changes needed? For easy aggregation on prometheus and grafana side ### Does this PR introduce any user facing changes? Add a flag to disable this for compatibility pr-link: Alluxio#18121 change-id: cid-ba6c2f9fae625747192044168fce7dc026c66b9c
besides the User-CLI.md doc, update other doc files that refer to `bin/alluxio` commands - remove docs on path config - remove starting/stopping job master/worker from contributor docs pr-link: Alluxio#18128 change-id: cid-fc71dd493b16ef3aeeb0b1b190941c43b9af9cab
What changes are proposed in this pull request? I have created a test and create a liststatus test for its function. Why are the changes needed? Please clarify why the changes are needed. For instance, add a unit test for DoraWorkerClientServiceHandler. Does this PR introduce any user facing changes? No. pr-link: Alluxio#18059 change-id: cid-b82706a4419700f017584f3e5579d2ef3410aeb3
Make fuse max reader concurrency configurable. The default value was 64 and it was unchangeable. pr-link: Alluxio#18129 change-id: cid-9c55821622329bd1e608da2e7445e8ab591df38a
Fix typo from alluxio.max.fuse.reader.concurrency to alluxio.fuse.max.reader.concurrency pr-link: Alluxio#18134 change-id: cid-434086cf6ba9e9f8d173e3417fc8518963dfa102
update usages of bin/alluxio, bin/alluxio-start.sh and bin/alluxio-stop.sh to their new counterparts simplify section of CephFS.md and remove sections related to mounting. the ufs must be configured as the root mount via alluxio-site.properties. pr-link: Alluxio#18136 change-id: cid-fa7d0eec00c8fb136680ef6d5a2c7ee78571d123
|
Thank you for your pull request. |
apc999
force-pushed
the
main
branch
2 times, most recently
from
2fec0ec to
b597c61
Compare
Xenorith
reviewed
Nov 13, 2023
pom.xml
Outdated
| @@ -137,7 +137,7 @@ | |||
| <findbugs.skip>false</findbugs.skip> | |||
| <grpc.version>1.54.1</grpc.version> | |||
| <gson.version>2.8.9</gson.version> | |||
| <guava.version>31.0.1-jre</guava.version> | |||
| <guava.version>32.0.0-jre</guava.version> | |||
There was a problem hiding this comment.
according to the guava release notes https://github.com/google/guava/releases/tag/v32.0.1 and the linked CVE description, this upgrade should be to version 32.0.1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What happened？
There are 1 security vulnerabilities found in com.google.guava:guava 31.0.1-jre
What did I do？
Upgrade com.google.guava:guava from 31.0.1-jre to 32.0.0-jre for vulnerability fix
What did you expect to happen？
Ideally, no insecure libs should be used.
How can we automate the detection of these types of issues?
By using the GitHub Actions configurations provided by murphysec, we can conduct automatic code security checks in our CI pipeline.
The specification of the pull request
PR Specification from OSCS