Features/optimize startup shell #5479
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- adapter 默认提供prometheus暴露JVM指标 - 指定 maven-antrun-plugin 未指定版本号，默认是 3.0.0，但是配置中使用 <tasks> 标签，编译报错。 - 降级版本号，显式指定 1.8 版本 - 或者把配置中的 <tasks> 标签改成 <target>
- 提升30%以上的性能 - 顺便解决protobuf历来版本的安全漏洞
- FastJSON2 内存保护机制，默认64MB，超过则抛出OutOfMemory异常
- 见：JSONWriter.ensureCapacity
- String.format is lower than StringBuilder. Benchmark like below:
code snippet:
String str = String.format("%s-%s-%s", 0, 1, 10);
Benchmark Mode Cnt Score Error Units
StringBenchmark.append thrpt 46431458.255 ops/s
StringBenchmark.format thrpt 985724.313 ops/s
StringBenchmark.append avgt ≈ 10⁻⁸ s/op
StringBenchmark.format avgt ≈ 10⁻⁶ s/op
StringBenchmark.append sample 364232 ≈ 10⁻⁷ s/op
StringBenchmark.append:p0.00 sample ≈ 10⁻⁸ s/op
StringBenchmark.append:p0.50 sample ≈ 10⁻⁷ s/op
StringBenchmark.append:p0.90 sample ≈ 10⁻⁷ s/op
StringBenchmark.append:p0.95 sample ≈ 10⁻⁷ s/op
StringBenchmark.append:p0.99 sample ≈ 10⁻⁷ s/op
StringBenchmark.append:p0.999 sample ≈ 10⁻⁷ s/op
StringBenchmark.append:p0.9999 sample ≈ 10⁻⁵ s/op
StringBenchmark.append:p1.00 sample 0.001 s/op
StringBenchmark.format sample 336220 ≈ 10⁻⁶ s/op
StringBenchmark.format:p0.00 sample ≈ 10⁻⁶ s/op
StringBenchmark.format:p0.50 sample ≈ 10⁻⁶ s/op
StringBenchmark.format:p0.90 sample ≈ 10⁻⁶ s/op
StringBenchmark.format:p0.95 sample ≈ 10⁻⁶ s/op
StringBenchmark.format:p0.99 sample ≈ 10⁻⁶ s/op
StringBenchmark.format:p0.999 sample ≈ 10⁻⁵ s/op
StringBenchmark.format:p0.9999 sample ≈ 10⁻⁴ s/op
StringBenchmark.format:p1.00 sample 0.001 s/op
StringBenchmark.append ss ≈ 10⁻⁶ s/op
StringBenchmark.format ss ≈ 10⁻⁵ s/op
- canal server logback.xml 配置增加 CanalServerWithEmbedded 单独日志配置，流量非常大的情况下，getMessages/ack等日志量非常大 日志刷盘太快也影响性能，占磁盘空间。用户可以根据需要单独调整日志级别。默认是INFO - TableMetaCache 锁的范围太大，针对这块做了代码逻辑优化。以下场景影响性能： - 当用户关闭tsdb.enable=false, 分库分表数量非常多回查次数很多； - 每张表有四五百列甚至更多，里面的集合容器提前初始化，减少扩容的次数； - DatabaseTableMeta 标记一个内存不安全的地方，以后看情况优化掉 使用MemoryTableMeta在内存区记录表元数据，当分库分表数量非常多，表结构非常复杂非常吃内存，容易OOM
- canal server 启动脚本会把日志重定向到 canal_stdout.log 线上运行时会把GC相关日志持续输出到该文件，直到把磁盘打爆；我们使用JVM原生的gc日志配置 开启日志滚动，最大日志文件32MB，最多存放5个文件，日志调优相关的参数根据业务情况进行±开闭 启动后标准输出流重定向到 /dev/null 不再输出到文件
- 重点升级JVM参数 - 重构SessionHandler代码, 处理错误请求会响应两次包；
Member
|
改动这么大？建议拆分成多个MR，单独针对startup脚本独立提交一个 |
Contributor
Author
Files changed 26. 代码变更比较清晰。上述所有的改动都已经过生产发布验证过 : ） |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
refactor(canal): 优化启动脚本JVM参数
线上运行时会把GC相关日志持续输出到该文件，直到把磁盘打爆；我们使用JVM原生的gc日志配置
开启日志滚动，最大日志文件32MB，最多存放5个文件，日志调优相关的参数根据业务情况进行±开闭
启动后标准输出流重定向到 /dev/null 不再输出到文件