This guide demonstrates how to integrate TDengine, a high-performance time-series database, with Perspective, a powerful data visualization library. By combining these technologies, you can create a real-time data visualization platform that streams data from TDengine to an interactive real-time Perspective dashboard.
TDengine is a purpose-built time-series database optimized for Industrial IoT (IIoT) applications. TDengine excels in handling massive datasets generated by sensors, devices, and other time-series sources. Its efficient architecture supports real-time analytics, predictive maintenance, and AI/ML workflows, making it an ideal choice for industries requiring scalable and reliable data solutions. With features like data compression, high throughput, and seamless integration with modern tools, TDengine is one of the dominant choices for time-series data.
Perspective is a powerful data visualization library that enables interactive, real-time data analysis in web applications. Developed by Prospective.co, Perspective leverages WebAssembly and Web Workers to provide high-performance data visualization capabilities directly in the browser. With Perspective, you can create dynamic dashboards, charts, and tables that update in real-time, allowing users to explore and interact with data seamlessly. Perspective's flexibility, speed, and ease of use make it an excellent choice for building data-driven applications that require real-time data visualization and analysis.
Together with TDengine and Perspective users can build high-performance -- milliseconds response time -- real-time data analysis and visualization applications on IoT data sources.
The architecture is as follow:
- Install TDengine python client libs (taospy).
- Instantiate a TDengine docker container.
- Start a producer script (
producer.py) which simulates real-time data ingestion into TDengine. - Start a perspective-python server (
perspective_server.py) which reads data periodically from TDengine and publishes it out to Perspective real-time Table via a Tornado Websocket. - Embed a Perspective viewer in HTML and connect to the backend Server.
- Visualize and interact with TDengine data in real-time
- Install and Configure TDengine client: Set up the TDengine python client.
- Docker Setup: Deploy a TDengine Docker container and populate it with benchmark data.
- Virtual Environment: Create and configure a virtual environment with necessary Python dependencies.
- Data Producer: Implement a script to simulate real-time data ingestion into TDengine.
- Perspective Server: Develop a server to stream data from TDengine to a Perspective viewer.
- Perspective Viewer: Embed and configure a Perspective viewer in an HTML page for data visualization.
By following these steps, you can create a scalable and efficient platform for real-time data analysis and monitoring. Customize and extend the provided examples to fit your specific use case, whether it's monitoring stock prices, IoT sensor data, or other time-series data.
We hope this guide has been helpful in getting you started with TDengine and Perspective. For further information and advanced features, please refer to the helpful resources provided. Happy coding!
Run the install.sh script to download and install the TDengine client libraries locally. This is necessary for the TDengine Python SDK (taospy) to function.
For more information on installing TDengine's client, please refer to install client library docs.
./install.shAfter the install script runs, please verify if the everything is setup correctly.
You should see a symlink for libtaos.so in:
ls -l tdengine-client/driver/Output:
total 68488
lrwxrwxrwx 1 warthog warthog 18 Jan 7 16:08 libtaos.so -> libtaos.so.3.3.5.0
-rwxr-xr-x 1 warthog warthog 59186032 Dec 31 03:42 libtaos.so.3.3.5.0
-rwxr-xr-x 1 warthog warthog 10937480 Dec 31 03:42 libtaosws.so
-rw-r--r-- 1 warthog warthog 8 Dec 31 03:42 vercomp.txtCheck if the client lib folder is correctly added to $LD_LIBRARY_PATH:
echo $LD_LIBRARY_PATHLD_LIBRARY_PATH should have been added to your bash profile file. Please check to ensure that it is set properly.
If you don't see this line at the end of your ~/.bashrc or ~/.bash_profile, please add it:
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:***YOUR PATH***/tdengine-client/driver"Run the docker.sh script to start a TDengine container. This script will also wait for the database to initialize.
./docker.shIf you with to pre-populate the TDengine container with benchmark data. Run the script with the following flag:
./docker --benchmarkFor complete information on running TDengine docker engine, please refer to Get Started with TDengine Using Docker docs.
install.sh already sets up a virtual environment for you and installs the TDengine taospy client. If you need to activate it manually, use the following commands:
source venv/bin/activate
pip install --upgrade pip
pip install --upgrade -r requirements.txtRun the producer.py script to periodically insert data into TDengine. This script simulates real-time data ingestion by generating random data points and inserting them into the TDengine database.
python producer.pyRun the perspective_server.py script to start a Perspective server (on a new terminal). This server will pull data from TDengine and stream it into a Tornado WebSocket.
python perspective_server.pyNOTE: Don't forget to activate your virtual environment before running the script.
Open the prsp-viewer.html file in your browser to view the Perspective Table. This table will display the real-time data streamed from TDengine.
open prsp-viewer.htmlThe docker.sh script starts a TDengine Docker container. It waits for the database to initialize before returning. You can run this script with the following flags:
--benchmark: Pre-populates the TDengine container with benchmark data.--no-pull: Skips pulling the TDengine Docker image.
./docker.sh --benchmark --no-pullThe producer.py script connects to the TDengine database and inserts data at regular intervals.
Here's how it works:
- Connecting to TDengine:
import taosws
TAOS_HOST = "localhost"
TAOS_PORT = 6041
TAOS_USER = "root"
TAOS_PASSWORD = "taosdata"
conn = taosws.connect(host=TAOS_HOST, port=TAOS_PORT, user=TAOS_USER, password=TAOS_PASSWORD)- Creating a table:
create_table = """
CREATE TABLE IF NOT EXISTS stocks_values (
timestamp TIMESTAMP,
ticker NCHAR(10),
client NCHAR(10),
open FLOAT,
high FLOAT,
low FLOAT,
close FLOAT,
volume INT UNSIGNED,
date TIMESTAMP
)
"""
conn.execute(create_table)- Inserting data:
The gen_data() method generates a series of random stock trades on every call:
import random
from datetime import datetime, date, timezone as tz
def gen_data():
modifier = random.random() * random.randint(1, 50)
return [{
"timestamp": datetime.now(tz=tz.utc),
"ticker": random.choice(["AAPL.N", "AMZN.N", "QQQ.N", "NVDA.N", "TSLA.N", "FB.N", "MSFT.N", "TLT.N", "XIV.N", "YY.N", "CSCO.N", "GOOGL.N", "PCLN.N"]),
"client": random.choice(["Homer", "Marge", "Bart", "Lisa", "Maggie", "Moe", "Lenny", "Carl", "Krusty"]),
"open": random.uniform(0, 75) + random.randint(0, 9) * modifier,
"high": random.uniform(0, 105) + random.randint(1, 3) * modifier,
"low": random.uniform(0, 85) + random.randint(1, 3) * modifier,
"close": random.uniform(0, 90) + random.randint(1, 3) * modifier,
"volume": random.randint(10_000, 100_000),
"date": date.today(),
} for _ in range(250)]The insert_data() method uses prepared statements and batch inserts to enhance performance. By generating a batch of records at a time and using a prepared SQL statement, the method minimizes the overhead associated with multiple individual insert operations. This approach ensures efficient data insertion into the TDengine database.
def insert_data(conn):
records = gen_data()
sql = "INSERT INTO stocks_values VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"
stmt = conn.statement()
stmt.prepare(sql)
timestamps = [int(record['timestamp'].timestamp() * 1000) for record in records]
tickers = [record['ticker'] for record in records]
clients = [record['client'] for record in records]
opens = [record['open'] for record in records]
highs = [record['high'] for record in records]
lows = [record['low'] for record in records]
closes = [record['close'] for record in records]
volumes = [record['volume'] for record in records]
dates = [int(datetime.combine(record['date'], datetime.min.time()).timestamp() * 1000) for record in records]
stmt.bind_param([
taosws.millis_timestamps_to_column(timestamps),
taosws.nchar_to_column(tickers),
taosws.nchar_to_column(clients),
taosws.floats_to_column(opens),
taosws.floats_to_column(highs),
taosws.floats_to_column(lows),
taosws.floats_to_column(closes),
taosws.ints_to_column(volumes),
taosws.millis_timestamps_to_column(dates),
])
stmt.add_batch()
stmt.execute()
while True:
insert_data(conn)
time.sleep(0.25)The perspective_server.py script starts a Perspective server that reads data from TDengine and streams it to a Perspective Table via a Tornado WebSocket.
Here's how it works:
- Connecting to TDengine:
import taosws
TAOS_HOST = "localhost"
TAOS_PORT = 6041
TAOS_USER = "root"
TAOS_PASSWORD = "taosdata"
conn = taosws.connect(host=TAOS_HOST, port=TAOS_PORT, user=TAOS_USER, password=TAOS_PASSWORD)- Reading data from TDengine:
The read_tdengine() function queries the TDengine database and retrieves the latest stock data:
def read_tdengine(conn):
sql = """
SELECT `timestamp`, ticker, client, open, high, low, close, volume, date
FROM stocks_values
WHERE `timestamp` >= NOW() - 1s
ORDER BY `timestamp` DESC
LIMIT 1000
"""
res = conn.query(sql)
data = [
{
"timestamp": convert_ts(row[0]),
"ticker": row[1],
"client": row[2],
"open": row[3],
"high": row[4],
"low": row[5],
"close": row[6],
"volume": row[7],
"date": convert_ts(row[8]),
}
for row in res
]
return data- Updating Perspective Table:
The perspective_thread() function creates a Perspective table and updates it with new data from TDengine every 250 milliseconds:
def perspective_thread(perspective_server, tdengine_conn):
client = perspective_server.new_local_client()
schema = {
"timestamp": datetime,
"ticker": str,
"client": str,
"open": float,
"high": float,
"low": float,
"close": float,
"volume": int,
"date": datetime,
}
table = client.table(schema, limit=1000, name="stock_values")
def updater():
data = read_tdengine(tdengine_conn)
table.update(data)
callback = tornado.ioloop.PeriodicCallback(callback=updater, callback_time=250)
callback.start()- Starting Tornado WebSocket Server:
The make_app() function sets up a Tornado application with a WebSocket handler to serve the Perspective table:
def make_app(perspective_server):
return tornado.web.Application([
(
r"/websocket",
perspective.handlers.tornado.PerspectiveTornadoHandler,
{"perspective_server": perspective_server},
),
])- Running the server:
The main block initializes the Perspective server, TDengine connection, and starts the Tornado IOLoop:
if __name__ == "__main__":
perspective_server = perspective.Server()
tdengine_conn = create_tdengine_connection()
app = make_app(perspective_server)
app.listen(8080, address='0.0.0.0')
loop = tornado.ioloop.IOLoop.current()
loop.call_later(0, perspective_thread, perspective_server, tdengine_conn)
loop.start()The prsp-viewer.html file embeds a Perspective Table in an HTML page. It connects to the Perspective server via a WebSocket and displays the real-time data streamed from TDengine.
Here's how it works:
- HTML Component:
The HTML file includes the necessary Perspective libraries and sets up a <perspective-viewer> element within a container. This custom HTML component, written in WebAssembly, provides easily embeddable and highly interactive real-time data visualization on top of TDengine data. The viewer is configured to connect to the Perspective server via WebSocket and load the stock_values table, allowing for dynamic data visualization.
- Styling:
CSS styles are applied to ensure the viewer occupies the full viewport and has a dark background.
- JavaScript Initialization:
A script is included to load the Perspective viewer and connect it to the Perspective server via WebSocket. The viewer is bound to the stock_values table on the server, allowing real-time data updates to be displayed.
<script type="module">
import perspective from "https://cdn.jsdelivr.net/npm/@finos/perspective@3.1.3/dist/cdn/perspective.js";
document.addEventListener("DOMContentLoaded", function() {
async function load_viewer() {
const table_name = "stock_values";
const viewer = document.getElementById("prsp-viewer");
const websocket = await perspective.websocket("ws://localhost:8080/websocket");
const server_table = await websocket.open_table(table_name);
await viewer.load(server_table);
}
load_viewer();
});
</script>- Viewer Configuration:
The perspective-viewer element is configured with the "Pro Dark" theme to match the dark background and provide a consistent visual appearance.
In this guide, we have walked through the process of integrating TDengine with Perspective to create a real-time data visualization platform. By following the steps outlined, you have learned how to:
- Install and configure the TDengine client and server.
- Set up a Docker container for TDengine and populate it with benchmark data.
- Create a virtual environment and install necessary Python dependencies.
- Implement a data producer to simulate real-time data ingestion into TDengine.
- Develop a Perspective server to stream data from TDengine to a web-based Perspective viewer.
- Embed and configure a Perspective viewer in an HTML page to visualize the data.
This integration allows you to leverage the high-performance time-series database capabilities of TDengine with the powerful data visualization features of Perspective. The combination of these technologies provides a robust solution for real-time data analysis and monitoring.
By following the provided examples and scripts, you can customize and extend this setup to fit your specific use case. Whether you are monitoring stock prices, IoT sensor data, or any other time-series data, this integration offers a scalable and efficient way to visualize and interact with your data in real-time.
We hope this guide has been helpful in getting you started with TDengine and Perspective. For further information and advanced features, please refer to the helpful resources provided. Happy coding!
-
TDengine client library examples including python and node.js: Download TDengine's client library tar file and unpack it. Look inside the examples directory for a comprehensive list of examples.
Next steps: