Usage Examples
Table of Contents
For writes, TickTock supports three protocols, TCP, UDP, and HTTP, at the same time. We will provide examples in the rest of this wiki.
- TCP:
A reliable streaming protocol providing once and only once semantics. You don't need to worry if your data will be lost or received more than once in server side. However, keep in mind that TCP write requests will be returned immediately without waiting for server responses. The write requests must fail to be applied to disks even though they were received by TickTock server. So we call TCP write async write. You can update port in config,
tcp.server.port = 6181
- UDP:
An unreliable connectionless protocol. Compared with TCP, UDP is faster but less reliable without once and only once semantics, which means that requests might be received multiple times or lost by TickTock server. As TCP, UDP requests will be returned immediately without waiting for response.
UDP is disabled by default. To enable UDP, you need to set explicitly in config,udp.server.enabled = true. It uses same port as TCP in config,tcp.server.port = 6181
- HTTP
HTTP is built on top of TCP. Unlike TCP, HTTP requests will not be returned until TickTock server finishes applying the requests, successfully or failed. We call HTTP writes sync writes. It is slower than TCP as it is blocking. Default port: 6182, as controlled by a setting,
http.server.port = 6182.
For reads, TickTock only supports HTTP. Please refer to examples below in this wiki.
If you are interested in send data to TickTock yourself, the TCP Server inside TickTock accepts
data in the following format by default (default configuration option: --http.request.format=plain):
put <metric-name> <timestamp> <value> <tag1>=<val1> <tag2>=<val2> ...
One data point per line (Note: must be ended with \n), where <timestamp> is Unix timestamp
(number of seconds/milliseconds since Epoch. Default is second, as controlled by a setting,tsdb.timestamp.resolution = second), and <value> can be either integer or
floating point number. Supported floating point number formats include,
- Decimal floating point numbers, e.g.
88,+1.23,-0.45,15e16- Hexdecimal floating point numbers, e.g.
0x12,-0x1afp-2
Do not put quotes anywhere, even around metric names, tag names, or tag values. Metric name, tag names, and tag values all cannot contain spaces.
Metric names, tag names, and tag values all has to be ASCII encoded. TickTock does not support unicode, yet.
If you want to use json format in PUT requests as Opentsdb APIs, you can run TickTock with configuration --http.request.format=json. Please refer to User Guide section 2.3 above.
data point metric name: testM1, tag: host=foo, timestamp: 1633412175, value=123
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 123 host=foo'
Note:
- The Data point will be stored in data files located in a data directory specified as
tsdb.data.dir(default: /tmp) in conf/tt.conf.- There is a data file (1633392000.1633478400.0) and a metadata file (1633392000.1633478400.meta) in /tmp for the date of the written data point at 1633412175. Each file has a prefix as ., where:
- fromSecond: the first second of a day
- toSecond: the first second of the next day
- There may be more than 1 data file for a day if there are more data points than a data file size. They will be postfixed as ".1", ".2", etc. The data file size is determined by
tsdb.page.count(each page 4kB) in conf/tt.conf. There will be only meta file for a day no matter how may data files in a day.- You can specify the granularity you want to start a new meta file as
tsdb.rotation.frequency(default: 1d).
Data point metric name: test.cpu.usr, tag: host=foo cpu=1, timestamp: 1633412175, value=123
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put test.cpu.usr 1633412175 123 host=foo cpu=1'
curl -v -XPOST 'http://localhost:6182/api/put' --data-binary 'put test.cpu.usr 1633412175 123 host=foo cpu=1\nput test.cpu.sys 1633412175 123 host=foo cpu=1\n'
curl -v 'http://localhost:6182/api/query?start=1600000000&m=avg:testM1'
Note the avg, time series aggregator. It will aggregate all data points of the related time series at each timestamp to a single data point. In this example, testM1 has only 1 time series (with tag host=foo), so you won't see the difference. But let say, if we insert two data points:
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 123 host=foo'
curl -v -XPOST 'http://localhost:6182/api/put' -d 'put testM1 1633412175 125 host=goo'
Metric testM1 will have two data points from two time series (host=foo and host=goo) at timestamp 1633412175. The above query m=avg:testM1 will return the average value as 124 (i.e, (123+125)/2).
The aggregator options are:
- avg
- sum
- min
- max
- p50
- p90
- p99
- p999
- count
curl -v 'http://localhost:6182/api/query?start=1600000000&end=1633412176&m=avg:testM1{host=foo}'
Note the end=1633412176. It can be a future epoch time. If you don't specify end (as in Example 2.4 above), it will be set to the current time by default.
curl -v 'http://localhost:6182/api/query?start=1600000000&end=1600000060=avg:1m-avg:cpu.usr'
Note the 1m-avg, a downsample aggregator. It will aggregate all data points of the related time series in 1 minute to a single data point. It is different from the time series aggregator avg as explained in Example 2.4. Let say there are 4 data points within 1 minute (from 1600000000 to 1600000060) of cpu.usr:
- 2 data points from host=foo, one at 1600000000 with value=50 and the other at 1600000030 with value=70,
- 2 data points from host=goo, one at 1600000000 with value=30 and the other at 1600000030 with value=50,
The query avg:1m-avg:cpu.usr will return a single value 50 at timestamp 1600000000. It will downsample 1m-avg of
-
host=fooas (1600000000, 60) in which 60==(50+70)/2, and -
host=gooas (1600000000, 40) in which 40=(30+50)/2.
Then it will calculate the avg of host=foo and host=goo as (1600000000, 50) in which 50=(60+40)/2.
You can use, e.g.,
-
1h-avg, average of all data points in 1 hour -
1d-avg, average of all data points in 1 day -
1w-avg, average of all data points in 1 week -
10m-avg, average of all data points in 10 minutes -
10m-count, count of all data points in 10 minutes -
10m-max, max value of all data points in 10 minutes
To query existing metric names starting with 'test':
curl -v 'http://localhost:6189/api/suggest?type=metrics&q=test'
It will return, e.g.,
["test.cpu.sys","test.cpu.sys1","test.cpu.usr","test.cpu.usr1"]
We provide several simple python examples below. If you want to use Python to talk with TickTock, we recommend tcollector, OpenTSDB official collector. If you use Python3, please use our forked version in https://github.com/ylin30/tcollector. The official TCollector has bugs with Python3 and doesn't work.
This example writes two data points in plain format with TCP request. The code is checked in source code dir as ticktock/api-examples/python/tcp_plain_writer.py.
#!/usr/bin/env python3
import socket
import sys
import time
if len(sys.argv) != 3:
print("Usage: python tcp_plain_writer.py <host> <port>")
sys.exit(1);
else:
HOST = sys.argv[1]
PORT = int(sys.argv[2])
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM);
s.connect((HOST, PORT))
print('connect to', HOST, PORT)
put_data_point1 = 'put tcp.cpu.usr 1633412275 20 host=foo cpu=1';
put_data_point2 = 'put tcp.cpu.sys 1633412275 20 host=foo cpu=1';
# Note each PUT request must be ended with '\n'
req = put_data_point1 +'\n'+put_data_point2+'\n';
s.sendall(req.encode('utf-8'));
# We need to sleep a few seconds before close the socket in this example.
# Otherwise TickTock server might not be able to read data as the socket is closed too early.
time.sleep(5)
print("Done sending two put reqeuests:\n"+req);
s.close();
except socket.error as e:
print("Exception: %s", e)
This example writes two data points in plain format with TCP request. The code is checked in source code dir as ticktock/api-examples/python/udp_plain_writer.py. Note that you need to enable UDP write in TickTock config, udp.server.enabled = true
#!/usr/bin/env python3
import socket
import sys
import time
if len(sys.argv) != 3:
print("Usage: python udp_plain_writer.py <host> <port>")
sys.exit(1);
else:
HOST = sys.argv[1]
PORT = int(sys.argv[2])
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM);
s.connect((HOST, PORT))
print('connect to', HOST, PORT)
put_data_point1 = 'put udp.cpu.usr 1633412275 20 host=foo cpu=1';
put_data_point2 = 'put udp.cpu.sys 1633412275 20 host=foo cpu=1';
# Note each PUT request must be ended with '\n'
req = put_data_point1 +'\n'+put_data_point2+'\n';
s.sendall(req.encode('utf-8'));
# We need to sleep a few seconds before close the socket in this example.
# Otherwise TickTock server might not be able to read data as the socket is closed too early.
time.sleep(5)
print("Done sending two put reqeuests:\n"+req);
s.close();
except socket.error as e:
print("Exception: %s", e)
This example writes two data points in plain format with HTTP request. The code is checked in source code dir as ticktock/api-examples/python/http_plain_writer.py.
import sys
PY3 = sys.version_info[0] > 2
if PY3:
from urllib.request import Request, urlopen # pylint: disable=maybe-no-member,no-name-in-module,import-error
from urllib.error import HTTPError, URLError # pylint: disable=maybe-no-member,no-name-in-module,import-error
else:
from urllib2 import Request, urlopen, HTTPError, URLError # pylint: disable=maybe-no-member,no-name-in-module,import-error
if len(sys.argv) != 3:
print("Usage: python http_plain_writer.py <host> <port>")
sys.exit(1);
else:
HOST = sys.argv[1]
PORT = sys.argv[2]
try:
url = "http://%s:%s/api/put" % (HOST, PORT)
req = Request(url)
req.add_header('Content-type', 'application/text')
put_data_point1 = 'put http.cpu.usr 1633412175 20 host=foo cpu=1';
put_data_point2 = 'put http.cpu.sys 1633412175 20 host=foo cpu=1';
put_req = put_data_point1 +'\n'+put_data_point2+'\n';
response = urlopen(req, put_req.encode())
print("Received response:", response.getcode())
except HTTPError as e1:
print("HTTP Exception:", e1)
except URLError as e2:
print("URL Exception:", e2)
This example kicks off several threads to stress-test TickTock. Each thread writes data points continually to TickTock with TCP protocol. Data points are in plain format. The code is checked in source code dir as ticktock/tools/client_tcp.c.
This example inspects TickTock data files and metadata files. The code is checked in source code dir as ticktock/tools/inspect.cpp. If you build with 'make all', there will be a binary in <ticktock>/bin/inspect. Examples:
./inspect /var/ticktock/data/1633392000.1633478400.0
./inspect -a /var/ticktock/data/1633392000.1633478400.0
BTW, metadata files are plain text files. You can open them with, e.g., vim 1633392000.1633478400.meta.
This example backfills data to TickTock by reading append log files. The code is checked in source code dir as ticktock/tools/backfill.cpp. If you build with 'make all', there will be a binary in <ticktock>/bin/backfill.