Skip to content

Commit 92bd13a

Browse files
phillipleblancphillipleblanc
authored
Add task history documentation (#408)
* Add task history documentation * Review feedback * Add cli command
1 parent b8de5a0 commit 92bd13a

4 files changed

Lines changed: 104 additions & 46 deletions

File tree

‎spiceaidocs/docs/api/adbc/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ DBAPI interface:
5252
(1, 2.0, 'Hello, world!')
5353
>>> cursor.execute("SHOW TABLES")
5454
>>> cursor.fetchall()
55-
[('spice', 'public', 'messages', 'BASE TABLE'), ('spice', 'runtime', 'query_history', 'BASE TABLE'), ('spice', 'information_schema', 'tables', 'VIEW'), ('spice', 'information_schema', 'views', 'VIEW'), ('spice', 'information_schema', 'columns', 'VIEW'), ('spice', 'information_schema', 'df_settings', 'VIEW'), ('spice', 'information_schema', 'schemata', 'VIEW')]
55+
[('spice', 'public', 'messages', 'BASE TABLE'), ('spice', 'runtime', 'task_history', 'BASE TABLE'), ('spice', 'information_schema', 'tables', 'VIEW'), ('spice', 'information_schema', 'views', 'VIEW'), ('spice', 'information_schema', 'columns', 'VIEW'), ('spice', 'information_schema', 'df_settings', 'VIEW'), ('spice', 'information_schema', 'schemata', 'VIEW')]
5656
```
5757

5858
Arrow:

‎spiceaidocs/docs/getting-started/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,7 @@ sql> show tables
115115
| table_catalog | table_schema | table_name | table_type |
116116
+---------------+--------------+---------------+------------+
117117
| spice | public | taxi_trips | BASE TABLE |
118-
| spice | runtime | query_history | BASE TABLE |
118+
| spice | runtime | task_history | BASE TABLE |
119119
| spice | runtime | metrics | BASE TABLE |
120120
+---------------+--------------+---------------+------------+
121121

‎spiceaidocs/docs/reference/query_history.md‎

Lines changed: 0 additions & 44 deletions
This file was deleted.

‎spiceaidocs/docs/reference/task_history.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
---
2+
title: 'Task History'
3+
sidebar_label: 'Task History'
4+
pagination_prev: 'reference/index'
5+
pagination_next: null
6+
sidebar_position: 4
7+
---
8+
9+
The Spice runtime stores information about completed tasks in the `spice.runtime.task_history` table. A task is a single unit of execution within the runtime, such as a SQL query or an AI chat completion (see [Task Types](#task-types) below). Tasks can be nested, and the runtime will record the parent-child relationship between tasks.
10+
11+
Each task executed has a row in this table, and by default the data is retained for 8 hours. Use a `SELECT` query to return information about each task as shown in this example:
12+
13+
```sql
14+
SELECT
15+
*
16+
FROM
17+
spice.runtime.task_history
18+
LIMIT
19+
100;
20+
```
21+
22+
Output:
23+
24+
```console
25+
+----------------------------------+------------------+----------------+---------------------+----------------------------------------------+-----------------+----------------------------+----------------------------+-----------------------+--------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
26+
| trace_id | span_id | parent_span_id | task | input | captured_output | start_time | end_time | execution_duration_ms | error_message | labels |
27+
+----------------------------------+------------------+----------------+---------------------+----------------------------------------------+-----------------+----------------------------+----------------------------+-----------------------+--------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
28+
| f94dba6b89de98c6e54b074f2353a897 | 4eb243d9b5347762 | | accelerated_refresh | runtime.metrics | | 2024-09-23T23:17:39.907789 | 2024-09-23T23:17:39.917777 | 9.988 | | {sql: SELECT * FROM runtime.metrics} |
29+
| 1f1f8305520e15ea7ad9b0a43e5d2c7e | 6aadf7c91caea3c4 | | accelerated_refresh | runtime.task_history | | 2024-09-23T23:17:39.907873 | 2024-09-23T23:17:39.917797 | 9.924000000000001 | | {sql: SELECT * FROM runtime.task_history} |
30+
| 1432e30c5ed7764f4ef35f6508dfd56c | fbb31c60d41d8232 | | accelerated_refresh | logs_file | | 2024-09-23T23:17:40.143699 | 2024-09-23T23:17:40.271678 | 127.97900000000001 | | {sql: SELECT * FROM logs_file} |
31+
| fd0b909b789938384d99f0e4e6f4b68b | 624ea4751bb6727a | | accelerated_refresh | logs | | 2024-09-23T23:17:40.676838 | 2024-09-23T23:17:42.345932 | 1669.0939999999998 | | {sql: SELECT * FROM "logs"} |
32+
| 3db5488039408825ac0829a3feb49b05 | e3e5ac928b497eef | | accelerated_refresh | decimal | | 2024-09-23T23:17:41.592359 | 2024-09-23T23:17:43.781699 | 2189.34 | | {sql: SELECT * FROM "decimal"} |
33+
| 5c5ddd481d1e19df823da74fe33f261f | 6afcfd1e65385a16 | | sql_query | select * from runtime.task_history limit 100 | | 2024-09-23T23:17:48.305649 | 2024-09-23T23:17:48.307369 | 1.72 | | {runtime_query: true, query_execution_duration_ms: 1.429375, protocol: FlightSQL, datasets: runtime.task_history, rows_produced: 5} |
34+
| 4c3dd314b874aa63fcd15023e67fc645 | cab3cdc2d31c1b6a | | sql_query | select block_number from logs_file limit 5 | | 2024-09-23T23:18:00.267218 | 2024-09-23T23:18:00.269278 | 2.06 | | {datasets: logs_file, rows_produced: 5, query_execution_duration_ms: 1.940291, accelerated: true, protocol: FlightSQL} |
35+
| f135c00df3aecd68dfa4d2360eff78f5 | db3474855449715c | | sql_query | select * from foobar | | 2024-09-23T23:18:12.865122 | 2024-09-23T23:18:12.865196 | 0.074 | Error during planning: table 'spice.public.foobar' not found | {protocol: FlightSQL, error_code: QueryPlanningError, rows_produced: 0, query_execution_duration_ms: 0.126959, datasets: } |
36+
+----------------------------------+------------------+----------------+---------------------+----------------------------------------------+-----------------+----------------------------+----------------------------+-----------------------+--------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
37+
```
38+
39+
## Task Types
40+
41+
The following top-level task types are currently recorded:
42+
43+
| Task Type | Description | CLI Command |
44+
| --------------------- | ----------------------------- | -------------- |
45+
| `sql_query` | SQL Query | `spice sql` |
46+
| `nsql_query` | Natural Language to SQL Query | |
47+
| `ai_chat` | AI Chat Completion | `spice chat` |
48+
| `vector_search` | Vector Search | `spice search` |
49+
| `accelerated_refresh` | Accelerated Table Refresh | |
50+
| `text_embed` | Text Embedding | |
51+
52+
## Configuration
53+
54+
Set the following parameters in the `runtime.task_history` section of your `config.yaml` file to configure task history:
55+
56+
- `enabled`: Enable or disable task history. Default: `true`.
57+
- `retention_period`: The duration for which task history data is retained. Default: `8h`.
58+
- `retention_check_interval`: The interval at which the task history retention is checked. Default: `1m`.
59+
- `captured_output`: The level of output captured for tasks. `none` or `truncated`. Default: `truncated`.
60+
61+
### Examples
62+
63+
Adjust the retention period for task history:
64+
65+
```yaml
66+
runtime:
67+
task_history:
68+
retention_period: 1h # Keep tasks for 1 hour
69+
retention_check_interval: 1m # Check for expired tasks every minute
70+
```
71+
72+
Disable task history:
73+
74+
```yaml
75+
runtime:
76+
task_history:
77+
enabled: false
78+
```
79+
80+
Disable capturing output from tasks:
81+
82+
```yaml
83+
runtime:
84+
task_history:
85+
captured_output: none # none or truncated
86+
```
87+
88+
## Table Schema
89+
90+
| Column Name | Data Type | Is Nullable | Description |
91+
|-----------------------|---------------------------|-------------|----------------------------------------------------------------------------------|
92+
| trace_id | Utf8 | NO | Unique identifier for the entire trace this task happened in |
93+
| span_id | Utf8 | NO | Unique identifier for this specific task within the trace |
94+
| parent_span_id | Utf8 | YES | Identifier of the parent task, if any |
95+
| task | Utf8 | NO | Name or description of the task being performed (e.g. `sql_query`) |
96+
| input | Utf8 | NO | Input data or parameters for the task |
97+
| captured_output | Utf8 | YES | Output or result of the task, if available |
98+
| start_time | Timestamp(Nanosecond, None) | NO | Time when the task started |
99+
| end_time | Timestamp(Nanosecond, None) | NO | Time when the task ended |
100+
| execution_duration_ms | Float64 | NO | Duration of the task execution in milliseconds |
101+
| error_message | Utf8 | YES | Error message if the task failed, otherwise null |
102+
| labels | Map(Utf8, Utf8) | NO | Key-value pairs for additional metadata or attributes associated with the task |

0 commit comments

Comments
 (0)