Skip to content

Commit e2fe088

Browse files
YaSuenagYaSuenag
committed
Migrate from YaSuenag repos
1 parent 165cc53 commit e2fe088

13 files changed

+1370
-0
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/target/

‎README.md

+114
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# HSLoader
2+
3+
HSLoader is data loader for HeapStats snapshot and resource log to Elasticsearch.
4+
This tool can be run for Elasticsearch 5.0 or later.
5+
6+
7+
## Library dependencies
8+
9+
* heapstats-core.jar
10+
* ```heapstats-core.jar``` in HeapStats Analyzer
11+
* This repository contains this library.
12+
13+
14+
## How to use
15+
16+
```
17+
$ java -jar hsloader.jar [options] file1 file2 ...
18+
```
19+
20+
21+
## Options
22+
23+
* --help
24+
* Help message
25+
* --mode
26+
* Parser mode. ```snapshot``` or ```log``` .
27+
* snapshot is by default
28+
* --host
29+
* Hostname of Elasticsearch
30+
* localhost is by default
31+
* --port
32+
* Port number of Elasticsearch
33+
* 9300 is by default
34+
* --bulk
35+
* Number of bulk transport
36+
* 1000 is by default
37+
* --timezone
38+
* Timezone of snapshot
39+
* System default is by default
40+
* --timeout
41+
* Timeout in second
42+
* 60 is by default
43+
44+
45+
## Sample dashboard
46+
47+
```kibana-dashboard.json``` in this repository provides sample dashboard of HSLoader. You can use it on Kibana 5 or later.
48+
If you use this dashboard, you need to define indices as below:
49+
50+
* HeapStats snapshots
51+
* heapstats-snapshot-summary-*
52+
* heapstats-snapshot-objects-*
53+
* heapstats-snapshot-refs-*
54+
* HeapStats resources
55+
* heapstats-resource-log-*
56+
* heapstats-resource-diff-*
57+
58+
------------
59+
60+
HSLoader は HeapStats のスナップショットとリソースログを Elasticsearch にロードするためのアプリケーションです。
61+
Elasticsearch 5.0 以降に対応しています。
62+
63+
64+
## 依存ライブラリ
65+
66+
* heapstats-core.jar
67+
* HeapStats Analyzer に同梱される ```heapstats-core.jar```
68+
* 本リポジトリに HeapStats 2.0.2 の JAR を登録しています
69+
70+
71+
## 使用方法
72+
73+
```
74+
$ java -jar hsloader.jar [options] file1 file2 ...
75+
```
76+
77+
78+
## オプション
79+
80+
* --help
81+
* ヘルプメッセージ
82+
* --mode
83+
* パーサーモード。 ```snapshot``` または ```log``` 。
84+
* デフォルト値は snapshot
85+
* --host
86+
* Elasticsearch のホスト名
87+
* デフォルト値は localhost
88+
* --port
89+
* Elasticsearch のポート番号
90+
* デフォルト値は 9300
91+
* --bulk
92+
* 一度に投入するデータ量（バルク転送）
93+
* デフォルト値は 1000
94+
* --timezone
95+
* スナップショット取得時間のタイムゾーン指定
96+
* デフォルトはシステムのタイムゾーン
97+
* --timeout
98+
* タイムアウト（秒）
99+
* デフォルトは 60
100+
101+
102+
## サンプルダッシュボード
103+
104+
ソースに含まれる ```kibana-dashboard.json``` を Kibana 5 にインポートすることで HSLoader のサンプルダッシュボードが利用可能です。
105+
その際、Kibana で以下のインデックスを定義するようにしてください。
106+
107+
* HeapStats snapshots
108+
* heapstats-snapshot-summary-*
109+
* heapstats-snapshot-objects-*
110+
* heapstats-snapshot-refs-*
111+
* HeapStats resources
112+
* heapstats-resource-log-*
113+
* heapstats-resource-diff-*
114+

‎kibana-dashboard.json

+204
Large diffs are not rendered by default.

‎lib/heapstats-core.jar

81.1 KB
Binary file not shown.

‎pom.xml

+88
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<groupId>jp.dip.ysfactory</groupId>
5+
<artifactId>hsloader</artifactId>
6+
<version>0.2.0</version>
7+
<packaging>jar</packaging>
8+
<name>HSLoader</name>
9+
<properties>
10+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
11+
<maven.compiler.source>1.8</maven.compiler.source>
12+
<maven.compiler.target>1.8</maven.compiler.target>
13+
</properties>
14+
<organization>
15+
<name>Yasumasa Suenaga</name>
16+
</organization>
17+
<dependencies>
18+
<dependency>
19+
<groupId>org.elasticsearch.client</groupId>
20+
<artifactId>transport</artifactId>
21+
<version>5.1.1</version>
22+
</dependency>
23+
24+
<!--
25+
We have to add dependency to Log4j 2
26+
https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_java_api_changes.html#_elasticsearch_will_no_longer_detect_logging_implementations
27+
-->
28+
<dependency>
29+
<groupId>org.apache.logging.log4j</groupId>
30+
<artifactId>log4j-api</artifactId>
31+
<version>2.6.2</version>
32+
</dependency>
33+
<dependency>
34+
<groupId>org.apache.logging.log4j</groupId>
35+
<artifactId>log4j-core</artifactId>
36+
<version>2.6.2</version>
37+
</dependency>
38+
<dependency>
39+
<groupId>jp.co.ntt.oss.heapstats</groupId>
40+
<artifactId>heapstats-core</artifactId>
41+
<version>2.0</version>
42+
<scope>system</scope>
43+
<systemPath>${pom.basedir}/lib/heapstats-core.jar</systemPath>
44+
</dependency>
45+
</dependencies>
46+
<build>
47+
<plugins>
48+
<plugin>
49+
<groupId>org.apache.maven.plugins</groupId>
50+
<artifactId>maven-jar-plugin</artifactId>
51+
<version>2.6</version>
52+
<configuration>
53+
<archive>
54+
<manifest>
55+
<mainClass>jp.dip.ysfactory.heapstats.hsloader.HSLoader</mainClass>
56+
<addClasspath>true</addClasspath>
57+
<classpathPrefix>lib</classpathPrefix>
58+
</manifest>
59+
<manifestEntries>
60+
<Class-Path>lib/heapstats-core.jar</Class-Path>
61+
</manifestEntries>
62+
</archive>
63+
<finalName>hsloader</finalName>
64+
</configuration>
65+
</plugin>
66+
<plugin>
67+
<groupId>org.apache.maven.plugins</groupId>
68+
<artifactId>maven-assembly-plugin</artifactId>
69+
<version>2.5.5</version>
70+
<configuration>
71+
<descriptors>
72+
<descriptor>src/main/assembly/distribution.xml</descriptor>
73+
</descriptors>
74+
<attach>false</attach>
75+
</configuration>
76+
<executions>
77+
<execution>
78+
<id>make-assembly</id>
79+
<phase>package</phase>
80+
<goals>
81+
<goal>single</goal>
82+
</goals>
83+
</execution>
84+
</executions>
85+
</plugin>
86+
</plugins>
87+
</build>
88+
</project>

‎src/main/assembly/distribution.xml

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<assembly
3+
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
4+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
5+
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2
6+
http://maven.apache.org/xsd/assembly-1.1.2.xsd">
7+
8+
<id>bin</id>
9+
<formats>
10+
<format>dir</format>
11+
<format>zip</format>
12+
</formats>
13+
<includeBaseDirectory>false</includeBaseDirectory>
14+
<fileSets>
15+
<fileSet>
16+
<directory>${project.build.directory}</directory>
17+
<outputDirectory>hsloader</outputDirectory>
18+
<includes>
19+
<include>*.jar</include>
20+
</includes>
21+
</fileSet>
22+
<fileSet>
23+
<directory>${pom.basedir}/lib</directory>
24+
<outputDirectory>hsloader/lib</outputDirectory>
25+
<includes>
26+
<include>*.jar</include>
27+
</includes>
28+
</fileSet>
29+
</fileSets>
30+
<dependencySets>
31+
<dependencySet>
32+
<unpack>false</unpack>
33+
<scope>runtime</scope>
34+
<useProjectArtifact>false</useProjectArtifact>
35+
<outputDirectory>hsloader/lib</outputDirectory>
36+
</dependencySet>
37+
</dependencySets>
38+
39+
</assembly>

‎src/main/java/jp/dip/ysfactory/heapstats/hsloader/HSLoader.java

+133
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
/*
2+
* HSLoader.java
3+
*
4+
* Copyright (C) 2015-2016 Yasumasa Suenaga
5+
*
6+
* This program is free software; you can redistribute it and/or
7+
* modify it under the terms of the GNU General Public License
8+
* as published by the Free Software Foundation; either version 2
9+
* of the License, or (at your option) any later version.
10+
*
11+
* This program is distributed in the hope that it will be useful,
12+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
13+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14+
* GNU General Public License for more details.
15+
*
16+
* You should have received a copy of the GNU General Public License
17+
* along with this program; if not, write to the Free Software
18+
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
19+
*
20+
*/
21+
package jp.dip.ysfactory.heapstats.hsloader;
22+
23+
import java.net.InetAddress;
24+
import java.util.Objects;
25+
import java.util.concurrent.TimeUnit;
26+
import jp.dip.ysfactory.heapstats.hsloader.log.LogProcessor;
27+
import jp.dip.ysfactory.heapstats.hsloader.snapshot.SnapShotProcessor;
28+
import org.elasticsearch.action.bulk.BulkProcessor;
29+
import org.elasticsearch.action.bulk.BulkRequest;
30+
import org.elasticsearch.action.bulk.BulkResponse;
31+
import org.elasticsearch.client.transport.TransportClient;
32+
import org.elasticsearch.common.settings.Settings;
33+
import org.elasticsearch.common.transport.InetSocketTransportAddress;
34+
import org.elasticsearch.common.unit.TimeValue;
35+
import org.elasticsearch.transport.client.PreBuiltTransportClient;
36+
37+
38+
/**
39+
* Main class of HSLoader.
40+
*
41+
* @author Yasumasa Suenaga
42+
*/
43+
public class HSLoader {
44+
45+
private static class BulkProcessListener implements BulkProcessor.Listener{
46+
47+
private final int timeout;
48+
49+
public BulkProcessListener(int timeout){
50+
this.timeout = timeout;
51+
}
52+
53+
@Override
54+
public void beforeBulk(long l, BulkRequest br) {
55+
br.timeout(TimeValue.timeValueSeconds(timeout));
56+
}
57+
58+
@Override
59+
public void afterBulk(long l, BulkRequest req, BulkResponse res) {
60+
if(res.hasFailures()){
61+
throw new RuntimeException(res.buildFailureMessage());
62+
}
63+
}
64+
65+
@Override
66+
public void afterBulk(long l, BulkRequest br, Throwable thrwbl) {
67+
throw new RuntimeException(thrwbl);
68+
}
69+
70+
}
71+
72+
private static class ExceptionHandler implements Thread.UncaughtExceptionHandler{
73+
74+
@Override
75+
public void uncaughtException(Thread thread, Throwable thrwbl) {
76+
Throwable cause = thrwbl;
77+
while(cause.getCause() != null){
78+
cause = cause.getCause();
79+
}
80+
81+
System.err.println(cause.getLocalizedMessage());
82+
83+
if(Boolean.getBoolean("debug")){
84+
thrwbl.printStackTrace();
85+
}
86+
87+
}
88+
89+
}
90+
91+
/**
92+
* @param args the command line arguments
93+
* @throws java.lang.InterruptedException
94+
*/
95+
public static void main(String[] args) throws Exception{
96+
Option opt;
97+
try{
98+
opt = new Option(args);
99+
}
100+
catch(Exception e){
101+
Option.printOptions();
102+
return;
103+
}
104+
105+
Thread.setDefaultUncaughtExceptionHandler(new ExceptionHandler());
106+
107+
InetSocketTransportAddress addr = new InetSocketTransportAddress(InetAddress.getByName(opt.getHost()), opt.getPort());
108+
109+
try(TransportClient client = (new PreBuiltTransportClient(Settings.EMPTY)).addTransportAddress(addr);
110+
BulkProcessor bulkProcessor = BulkProcessor.builder(client, new BulkProcessListener(opt.getTimeout()))
111+
.setName("HSLoader bulk processor")
112+
.setBulkActions(opt.getBulkLevel())
113+
.setConcurrentRequests(Runtime.getRuntime().availableProcessors())
114+
.build()){
115+
116+
Processor processor = null;
117+
118+
if(opt.getParserMode() == Option.ParserMode.snapshot){
119+
processor = new SnapShotProcessor(opt, bulkProcessor);
120+
}
121+
else if(opt.getParserMode() == Option.ParserMode.log){
122+
processor = new LogProcessor(opt, bulkProcessor);
123+
}
124+
125+
Objects.requireNonNull(processor, "Processor must not be null");
126+
processor.process();
127+
128+
bulkProcessor.awaitClose(opt.getTimeout(), TimeUnit.SECONDS);
129+
}
130+
131+
}
132+
133+
}

0 commit comments

Comments
 (0)