You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/blueql/1.overview.md
+11-3Lines changed: 11 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -6,6 +6,7 @@ id: overview
6
6
In this document we explore some of the meta parts of BlueQL. If you want to look at how you can use BlueQL, consider looking at the sections that follow.
7
7
8
8
Design principles:
9
+
9
10
-**Simplicity and clarity**: The language shouldn't be overwhelming to understand
10
11
-**Security with mandatory parameterization**: We want to reduce the surface of injection attacks. For this reason, [parameterization is mandatory](#parameters).
11
12
@@ -18,17 +19,18 @@ Just like SQL, BlueQL has three categories of commands/queries inside it:
18
19
Jump to [differences from SQL](#differences-from-sql).
19
20
20
21
:::info
21
-
This text is *not* a detailed, formal guide. It's meant for developers and users who want to work with
22
-
Skytable. If you need a more formal specification, like a grammar definition, please ask us, and we'll create
22
+
This text is *not* a detailed, formal guide. It's meant for developers and users who want to work with
23
+
Skytable. If you need a more formal specification, like a grammar definition, please ask us, and we'll create
23
24
it. We haven't published it yet because no one has requested it.
24
25
:::
25
26
26
27
## Identifiers
28
+
27
29
Can begin with any ASCII alphabet or an underscore (`_`) and then have any number of alphanumeric characters and/or underscores.
28
30
29
31
## Keywords
30
32
31
-
Keywords are identifiers with special meanings and hence can't be used as identifiers in other places. Here's a full-list of
33
+
Keywords are identifiers with special meanings and hence can't be used as identifiers in other places. Here's a full-list of
32
34
keywords:
33
35
34
36
```ts
@@ -45,6 +47,7 @@ keywords:
45
47
## Data types
46
48
47
49
### Boolean
50
+
48
51
A boolean value, either `true` or `false`
49
52
50
53
### Unsigned integers
@@ -75,9 +78,11 @@ A boolean value, either `true` or `false`
75
78
76
79
-`list`: a list of any of the data types, including nested lists
77
80
- A list is represented as: `[]` with values inbetween. For example, a `list { type:string }` would be represented as:
81
+
78
82
```sql
79
83
["sayan", "loves", "dogs"]
80
84
```
85
+
81
86
-**Lists cannot contain nullvalues**
82
87
-**List can be nested**: You can have heavily nested lists like: `[[[]], [["another one"]]]`
83
88
-**List can only have one base type**: This means that if you have a list like`[[[string]]]` each element must either be the same nested list, or an empty list
@@ -100,6 +105,7 @@ New data types are frequently added, so treat this list as non-exhaustive.
100
105
101
106
:::warning Literals are not available everywhere
102
107
It is very important for you to know that literals are not allowed everywhere. The only literals allowed everywhere are:
108
+
103
109
- Lists
104
110
- Dictionaries
105
111
@@ -150,6 +156,7 @@ On a final note, BlueQL doesn't support comments of any form also for security r
150
156
## DDL
151
157
152
158
Queries include:
159
+
153
160
- Spaces:
154
161
-`CREATE SPACE myspace [WITH { property: value, ... }]`
155
162
-`ALTER SPACE myspace [WITH { property: updated_value, ... }]`
@@ -172,6 +179,7 @@ Queries include:
172
179
## DCL
173
180
174
181
Queries include:
182
+
175
183
-`SYSCTL REPORT STATUS`: returns the status of the system. (Not a control query per se)
176
184
-`SYSCTL CREATE USER "username" WITH { password: ... }`: create a new user
177
185
-`SYSCTL DROP USER "username"`: removes the user in question
Copy file name to clipboardExpand all lines: docs/c.architecture.md
+19-15Lines changed: 19 additions & 15 deletions
Original file line number
Diff line number
Diff line change
@@ -4,18 +4,18 @@ title: Architecture
4
4
---
5
5
6
6
Skytable is a modern NoSQL database that prioritises performance, scalability and reliability while providing a rich and powerful querying interface.
7
-
We are generally targetting an audience that wants to build high performance, large-scale, low latency applications, such as social networking services,
7
+
We are generally targetting an audience that wants to build high performance, large-scale, low latency applications, such as social networking services,
8
8
auth services, adtech and such. Skytable is designed to work with both **structured and semi-structured data**.
9
9
10
-
Our goal is to provide you with a powerful and solid foundation for your application with no gimmicks — just a solid core. That's why, every component in
10
+
Our goal is to provide you with a powerful and solid foundation for your application with no gimmicks — just a solid core. That's why, every component in
11
11
Skytable has been engineered from the ground up, from scratch.
12
12
13
13
And all of that, without you having to be an expert, and with the least maintenance that you can expect.
14
14
15
15
## Fundamental differences from relational systems
16
16
17
-
BlueQL kind of looks and feels like using SQL with a relational database but that doesn't make Skytable's internals the same, with the most important
18
-
distinction being the fact that Skytable has a NoSQL engine! But Skytable's evaluation and execution of queries is fundamentally different from SQL
17
+
BlueQL kind of looks and feels like using SQL with a relational database but that doesn't make Skytable's internals the same, with the most important
18
+
distinction being the fact that Skytable has a NoSQL engine! But Skytable's evaluation and execution of queries is fundamentally different from SQL
19
19
counterparts and even NoSQL engines. Here are some key differences:
20
20
21
21
- All DML queries are point queries and **not** range queries:
@@ -26,18 +26,18 @@ counterparts and even NoSQL engines. Here are some key differences:
26
26
-**Remember, in NoSQL systems we denormalize.** Hence, no `JOIN`s or foreign keys as with many other NoSQL systems
27
27
- A different transactional model:
28
28
- All DDL and DCL queries are ACID transactions
29
-
- However, DML transactions are not ACID and instead are efficiently batched and are automatically made part of a batch
30
-
transaction. The engine decides *when* it will execute them, for example based on the pressure on cache. That's because our
29
+
- However, DML transactions are not ACID and instead are efficiently batched and are automatically made part of a batch
30
+
transaction. The engine decides *when* it will execute them, for example based on the pressure on cache. That's because our
31
31
focus is to maximize throughput
32
32
- All these differences mean that **DDL and DCL transactions are ACID transactions** while **DML queries are ACI and *eventually* D** (we call it a *delayed durability transaction*). This delay however can be adjusted to make sure that the DML
33
33
queries *emulate* ACID transactions but that defeats the point of the eventually durable system which aims to heavily increase throughput.
34
-
- The idea of eventually durable transactions relies on the idea that hardware failure even though prominent is still rare,
34
+
- The idea of eventually durable transactions relies on the idea that hardware failure even though prominent is still rare,
35
35
thanks to the extreme hard work that cloud vendors put into reliability engineering. If you plan to run on unreliable hardware, then the delay setting (reliability service) is what you need to change.
36
36
- For extremely unreliable hardware on the other hand, we're working on a new storage driver `rtsyncblock` that is expected to be released in Q1'24
37
37
- The transactional engine powering DDL and DCL queries might often choose to demote a transaction to a virtual transaction which makes sure that the transaction is obviously durable but not necessarily actually executed but is eventually executed. If the system crashes, the engine will still be able to actually execute the transaction (even if it crashed halfway)
38
38
39
39
:::tip
40
-
We believe you can now hopefully see how Skytable's workings are fundamentally different from traditional engines. And, we know
40
+
We believe you can now hopefully see how Skytable's workings are fundamentally different from traditional engines. And, we know
41
41
that you might have a lot of questions! If you do, please reach out. We're here to help.
42
42
:::
43
43
@@ -52,18 +52,19 @@ While a `MODEL` is the only data container for now, many more are set to come. N
52
52
53
53
### A `space` is like a `database`
54
54
55
-
A `space` in Skytable is a collection of `model`s and other objects, and settings. It is different from a traditional SQL
55
+
A `space` in Skytable is a collection of `model`s and other objects, and settings. It is different from a traditional SQL
56
56
Database (that is created with SQL's `CREATE DATABASE`) because it is not meant for tables only, but many other things.
57
57
58
58
Spaces have "space local" settings that can be set for the space (and will be respected by all its `models` and other items). You'll learn more about this in the section on DDL.
59
59
60
60
### A `model` is like a `table`
61
61
62
62
A `model` in Skytable is like a `table` in SQL but is vastly different because of certain concepts such as:
63
+
63
64
- DML queries are point and not range queries by default
64
65
- Restrictions on indexes
65
66
- Settings (which can't be created in traditional `table`s)
66
-
- Semi-structured data: with collection types in columns such as lists and dictionaries that violates some of the ideas of
67
+
- Semi-structured data: with collection types in columns such as lists and dictionaries that violates some of the ideas of
67
68
traditional schema enforcement
68
69
69
70
## Query language
@@ -75,13 +76,13 @@ For example, Skytable's BlueQL<sup>TM</sup> *only* allows the parameterization o
75
76
## Transactions
76
77
77
78
Skytable's DDL and DCL queries are all ACID transactions. However, DML queries use an AOF-style storage driver that periodically
78
-
records, analyses and then intelligently syncs the changes to disk. We're working on making ACID transactions widely available
79
+
records, analyses and then intelligently syncs the changes to disk. We're working on making ACID transactions widely available
79
80
across DML queries as well.
80
81
81
82
## Storage
82
83
83
-
Skytable's storage engine is collectively called the Skytable Disk Storage Subsystem or SDSS for short. The storage engine uses
84
-
several different storage drivers, using ones appropriate for the task. We do not use RocksDB or any other engine but we
84
+
Skytable's storage engine is collectively called the Skytable Disk Storage Subsystem or SDSS for short. The storage engine uses
85
+
several different storage drivers, using ones appropriate for the task. We do not use RocksDB or any other engine but we
85
86
implement everything in house, engineering them piece by piece.
86
87
87
88
:::info Features on track
@@ -103,8 +104,9 @@ Skytable will use atleast as many threads as the number of logical CPUs present
103
104
104
105
## Networking
105
106
106
-
Skytable uses its own in-house Skyhash protocol for client-server communication. It is built on top of TCP, enabling any programming language that has a
107
+
Skytable uses its own in-house Skyhash protocol for client-server communication. It is built on top of TCP, enabling any programming language that has a
107
108
TCP client to use it without issues. There are three phases in the connection:
109
+
108
110
- Handshake: All auth data, compatibility information and other data is exchanged at this step
109
111
- Connection mode selection: based on the handshake parameters a connection mode is chosen and the server responds with the chosen exchange mode
110
112
- Data exchange: This is where the real querying happens
@@ -115,14 +117,16 @@ You can [read more about the protocol here](protocol).
115
117
## Backwards compatibility
116
118
117
119
We make the promise to you that no matter what changes in Skytable, you will always be able to:
120
+
118
121
- Upgrade from one version to another without data loss or too many hoops
119
122
- Export data from Skytable at any time
120
123
121
124
More technically:
125
+
122
126
-**For minor/patch releases**: The minor/patch is just in the name but it indicates that no data migration effort is needed. **No minor releases ever need data migration, and any migration is done automatically**
123
127
-**For major releases**: Major releases generally introduce breaking changes (just like the upgrade from `0.7.x` to `0.8.0` is a largely breaking change). **Major releases will either automatically upgrade the data files or require you to use a migration tool that is shipped with the bundle**.
124
128
- Definitions (closely following semantic versioning):
125
-
-**A major release** is something like `1.0.0` to `2.0.0` or `0.8.0` to `0.9.0` (in development versions, 0.8.0 to 0.9.0 is considered a major version
129
+
-**A major release** is something like `1.0.0` to `2.0.0` or `0.8.0` to `0.9.0` (in development versions, 0.8.0 to 0.9.0 is considered a major version
126
130
bump)
127
131
-**A minor release** is something like `1.0.0` to `1.1.0` or `0.8.0` to `0.8.1`
128
132
-**A patch release** is something like `1.0.0` to `1.0.1` or `0.8.0` to `0.8.1` (note that in development versions there is no distinction between a minor and patch release)
0 commit comments