docs: create table update syntax (part 1)

doc/user/assets/sass/_content.scss

@@ -749,9 +749,7 @@ p+p {
         padding: 0;
         border-bottom: 1px solid #9c86e0;
         display: flex;
-        overflow-x: scroll;
-
-        padding-bottom: var(--xx-small);
+        overflow-x: auto;
 
         @media(max-width: 850px) {}
 
@@ -760,13 +758,15 @@ p+p {
             margin: 0 rem(0.1);
             padding: 0;
             position: relative;
-            bottom: -1px;
+
             background: var(--gray-lightest);
+            border-radius: 8px 8px 0 0;
+
 
             a {
                 color: var(--body);
                 display: block;
-                padding: rem(0.8) rem(1.6);
+                padding: rem(0.8) rem(1.5);
                 font-size: rem(1.4);
                 text-decoration: none;
                 font-weight: 500;
@@ -787,7 +787,6 @@ p+p {
 
             &.active {
                 background: var(--bg);
-                border-radius: 2px 2px 0 0;
                 border: 1px solid #9c86e0;
                 border-bottom-color: var(--bg);
 

doc/user/config.toml

@@ -198,6 +198,11 @@ weight = 30
 # allow <a name="link-target">, the old syntax no longer works
 unsafe = true
 
+[markup]
+  [markup.highlight]
+    noClasses = false
+    style = "monokai"
+
 [[deployment.targets]]
 name = "production"
 url = "s3://materialize-website?region=us-east-1"

doc/user/content/sql/create-table.md

@@ -1,6 +1,6 @@
 ---
 title: "CREATE TABLE"
-description: "`CREATE TABLE` creates a table that is persisted in durable storage."
+description: "Reference page for `CREATE TABLE`. `CREATE TABLE` creates a table that is persisted in durable storage."
 pagerank: 40
 menu:
   # This should also have a "non-content entry" under Reference, which is
@@ -9,50 +9,239 @@ menu:
     parent: 'commands'
 ---
 
-`CREATE TABLE` defines a table that is persisted in durable storage and can be
-written to, updated and seamlessly joined with other tables, views or sources.
+`CREATE TABLE` defines a table that is persisted in durable storage. In
+Materialize, you can create:
 
-Tables in Materialize are similar to tables in standard relational databases:
-they consist of rows and columns where the columns are fixed when the table is
-created but rows can be added to at will via [`INSERT`](../insert) statements.
+- User-populated tables. User-populated tables can be written to (i.e.,
+  [`INSERT`]/[`UPDATE`]/[`DELETE`]) by the user.
 
-{{< warning >}}
-At the moment, tables have many [known limitations](#known-limitations). In most
-situations, you should use [sources](/sql/create-source) instead.
-{{< /warning >}}
+- [Source-populated](/concepts/sources/) tables. Source-populated tables cannot
+  be written to by the user; they are populated through data ingestion from a
+  source.
 
-[//]: # "TODO(morsapaes) Bring back When to use a table? once there's more
-clarity around best practices."
+Tables can be joined with other tables, materialized views, and views. Tables in
+Materialize are similar to tables in standard relational databases: they consist
+of rows and columns where the columns are fixed when the table is created.
 
 ## Syntax
 
-{{< diagram "create-table.svg" >}}
+{{< tabs >}}
 
-### `col_option`
+{{< tab "User-populated tables" >}}
 
-{{< diagram "col-option.svg" >}}
+To create a table that users can write to (i.e., perform
+[`INSERT`](/sql/insert/)/[`UPDATE`](/sql/update/)/[`DELETE`](/sql/delete/)
+operations):
 
-Field | Use
-------|-----
-**TEMP** / **TEMPORARY** | Mark the table as [temporary](#temporary-tables).
-_table&lowbar;name_ | A name for the table.
-_col&lowbar;name_ | The name of the column to be created in the table.
-_col&lowbar;type_ | The data type of the column indicated by _col&lowbar;name_.
-**NOT NULL** | Do not allow the column to contain _NULL_ values. Columns without this constraint can contain _NULL_ values.
-*default_expr* | A default value to use for the column in an [`INSERT`](/sql/insert) statement if an explicit value is not provided. If not specified, `NULL` is assumed.
+```mzsql
+CREATE [TEMP|TEMPORARY] TABLE <table_name> (
+  <column_name> <column_type> [NOT NULL][DEFAULT <default_expr>]
+  [, ...]
+)
+[WITH (
+  PARTITION BY (<column_name> [, ...]) |
+  RETAIN HISTORY [=] FOR <duration>
+)]
+;
+```
+
+{{% yaml-table data="syntax_options/create_table_options_user_populated" %}}
+
+{{</ tab >}}
+
+{{< tab "Source-populated tables (DB source)" >}}
 
-### `with_options`
+To create a table from a [source](/sql/create-source/), where the source maps to
+an external database system:
 
-{{< diagram "with-options.svg" >}}
+{{< note >}}
 
-| Field                                    | Value               | Description                                                                                                                                                       |
-|------------------------------------------|---------------------| ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **PARTITION BY** _columns_               | `(ident [, ident]*)` | The key by which Materialize should internally partition this durable collection. See the [partitioning guide](/transform-data/patterns/partition-by/) for restrictions on valid values and other details.
-| **RETAIN HISTORY FOR** _retention_period_ | `interval`          | ***Private preview.** This option has known performance or stability issues and is under active development.* Duration for which Materialize retains historical data, which is useful to implement [durable subscriptions](/transform-data/patterns/durable-subscriptions/#history-retention-period). Accepts positive [interval](/sql/types/interval/) values (e.g. `'1hr'`). Default: `1s`.
+Users cannot write to source-populated tables; i.e., users cannot perform
+[`INSERT`](/sql/insert/)/[`UPDATE`](/sql/update/)/[`DELETE`](/sql/delete/)
+operations on source-populated tables.
 
+{{</ note >}}
+
+```mzsql
+CREATE TABLE <table_name> FROM SOURCE <source_name> (REFERENCE <ref_object>)
+[WITH (
+    TEXT COLUMNS (<fq_column_name> [, ...])
+  | EXCLUDE COLUMNS (<fq_column_name> [, ...])
+  | PARTITION BY (<column_name> [, ...])
+  [, ...]
+)]
+;
+```
+
+{{% yaml-table data="syntax_options/create_table_options_source_populated_db" %}}
+
+<a name="supported-db-source-types" ></a>
+
+{{< tabs >}}
+{{< tab "Supported MySQL types">}}
+
+Materialize natively supports the following MySQL types:
+
+<ul style="column-count: 3">
+<li><code>bigint</code></li>
+<li><code>binary</code></li>
+<li><code>bit</code></li>
+<li><code>blob</code></li>
+<li><code>boolean</code></li>
+<li><code>char</code></li>
+<li><code>date</code></li>
+<li><code>datetime</code></li>
+<li><code>decimal</code></li>
+<li><code>double</code></li>
+<li><code>float</code></li>
+<li><code>int</code></li>
+<li><code>json</code></li>
+<li><code>longblob</code></li>
+<li><code>longtext</code></li>
+<li><code>mediumblob</code></li>
+<li><code>mediumint</code></li>
+<li><code>mediumtext</code></li>
+<li><code>numeric</code></li>
+<li><code>real</code></li>
+<li><code>smallint</code></li>
+<li><code>text</code></li>
+<li><code>time</code></li>
+<li><code>timestamp</code></li>
+<li><code>tinyblob</code></li>
+<li><code>tinyint</code></li>
+<li><code>tinytext</code></li>
+<li><code>varbinary</code></li>
+<li><code>varchar</code></li>
+</ul>
+
+Replicating tables that contain **unsupported data types** is
+possible via the [`TEXT COLUMNS` option](#text-columns) for the
+following types:
+
+<ul style="column-count: 1">
+<li><code>enum</code></li>
+<li><code>year</code></li>
+</ul>
+
+The specified columns will be treated as `text`, and will thus not offer the
+expected MySQL type features. For any unsupported data types not listed above,
+use the [`EXCLUDE COLUMNS`](#exclude-columns) option.
+
+{{</ tab >}}
+
+{{< tab "Supported PostgreSQL types">}}
+Materialize natively supports the following PostgreSQL types (including the
+array type for each of the types):
+
+<ul style="column-count: 3">
+<li><code>bool</code></li>
+<li><code>bpchar</code></li>
+<li><code>bytea</code></li>
+<li><code>char</code></li>
+<li><code>date</code></li>
+<li><code>daterange</code></li>
+<li><code>float4</code></li>
+<li><code>float8</code></li>
+<li><code>int2</code></li>
+<li><code>int2vector</code></li>
+<li><code>int4</code></li>
+<li><code>int4range</code></li>
+<li><code>int8</code></li>
+<li><code>int8range</code></li>
+<li><code>interval</code></li>
+<li><code>json</code></li>
+<li><code>jsonb</code></li>
+<li><code>numeric</code></li>
+<li><code>numrange</code></li>
+<li><code>oid</code></li>
+<li><code>text</code></li>
+<li><code>time</code></li>
+<li><code>timestamp</code></li>
+<li><code>timestamptz</code></li>
+<li><code>tsrange</code></li>
+<li><code>tstzrange</code></li>
+<li><code>uuid</code></li>
+<li><code>varchar</code></li>
+</ul>
+
+Replicating tables that contain **unsupported data types** is possible via the
+[`TEXT COLUMNS` option](#text-columns). When decoded as `text`, the specified
+columns will not have the expected PostgreSQL type features. For example:
+
+* [`enum`]: When decoded as `text`, the resulting `text` values will
+  not observe the implicit ordering of the original PostgreSQL `enum`; instead,
+  Materialize will sort the values as `text`.
+
+* [`money`]: When decoded as `text`, the resulting `text` value
+  cannot be cast back to `numeric` since PostgreSQL adds typical currency
+  formatting to the output.
+
+[`enum`]: https://www.postgresql.org/docs/current/datatype-enum.html
+[`money`]: https://www.postgresql.org/docs/current/datatype-money.html
+
+{{</ tab >}}
+{{</ tabs >}}
+
+See also [Materialize SQL data types](/sql/types/).
+
+{{</ tab >}}
+
+{{< tab "Source-populated tables (Kafka/Redpanda source)" >}}
+
+To create a table from a source, where the source maps to an external
+Kafka/Redpanda system:
+
+{{< note >}}
+
+Users cannot write to source-populated tables; i.e., users cannot perform
+[`INSERT`](/sql/insert/)/[`UPDATE`](/sql/update/)/[`DELETE`](/sql/delete/)
+operations on source-populated tables.
+
+{{</  note >}}
+
+```mzsql
+CREATE TABLE <table_name> FROM SOURCE <source_name> [(REFERENCE <ref_object>)]
+[FORMAT <format> | KEY FORMAT <format> VALUE FORMAT <format>]
+   -- <format> can be:
+   -- AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION <conn_name>
+   --     [KEY STRATEGY
+   --       INLINE <schema> | ID <schema_registry_id> | LATEST ]
+   --     [VALUE STRATEGY
+   --       INLINE <schema> | ID <schema_registry_id> | LATEST ]
+  -- | PROTOBUF USING CONFLUENT SCHEMA REGISTRY CONNECTION <conn_name>
+  -- | PROTOBUF MESSAGE <msg_name> USING SCHEMA <encoded_schema>
+  -- | CSV WITH HEADER ( <col_name>[, ...]) [DELIMITED BY <char>]
+  -- | CSV WITH <num> COLUMNS DELIMITED BY <char>
+  -- | JSON | TEXT | BYTES
+]
+[INCLUDE
+    KEY [AS <name>] | PARTITION [AS <name>] | OFFSET [AS <name>]
+  | TIMESTAMP [AS <name>] | HEADERS [AS <name>] | HEADER <key_name> AS <name> [BYTES]
+  [, ...]
+]
+[ENVELOPE
+    NONE  --  Default.  Uses the append-only envelope.
+  | DEBEZIUM
+  | UPSERT [(VALUE DECODING ERRORS = INLINE [AS name])]
+]
+;
+```
+
+{{% yaml-table data="syntax_options/create_table_options_source_populated_kafka"
+%}}
+
+
+{{</ tab >}}
+
+{{</ tabs >}}
 
 ## Details
 
+### Table names and column names
+
+Names for tables and column(s) must follow the [naming
+guidelines](/sql/identifiers/#naming-restrictions).
+
 ### Known limitations
 
 Tables do not currently support:
@@ -109,4 +298,9 @@ The privileges required to execute this statement are:
 ## Related pages
 
 - [`INSERT`](../insert)
+- [`CREATE SOURCE`](/sql/create-source/)
 - [`DROP TABLE`](../drop-table)
+
+[`INSERT`]: /sql/insert/
+[`UPDATE`]: /sql/update/
+[`DELETE`]: /sql/delete/

doc/user/content/transform-data/patterns/partition-by.md

@@ -26,7 +26,7 @@ If you want to return results in a specific order, use an `ORDER BY` clause on y
 
 ## Syntax
 
-The option `PARTITION BY <column list>` declares that a [materialized view](/sql/create-materialized-view/#with_options) or [table](/sql/create-table/#with_options) should be partitioned by the listed columns.
+The option `PARTITION BY <column list>` declares that a [materialized view](/sql/create-materialized-view/#with_options) or [table](/sql/create-table/#partition-by) should be partitioned by the listed columns.
 For example, a table that stores an append-only collection of events may want to partition the data by time:
 
 ```mzsql

doc/user/data/syntax_options/create_table_options_source_populated_db.yml

@@ -0,0 +1,45 @@
+columns:
+  - column: "Parameter"
+  - column: "Description"
+rows:
+  - "Parameter": "`<table_name>`"
+    "Description": |
+
+      The name of the table to create. Names for tables must follow the [naming
+      guidelines](/sql/identifiers/#naming-restrictions).
+
+  - "Parameter": "`<source_name>`"
+    "Description": |
+
+      The name of the [source](/sql/create-source/) associated with the
+      reference object from which to create the table.
+
+  - "Parameter": "**(REFERENCE <ref_object>)**"
+    "Description": |
+
+      The name of the reference object from which to create the table. Reference
+      objects are the names of the tables in the upstream database. You can
+      create multiple tables from the same reference object.
+
+      To find the reference objects available in your
+      [source](/sql/create-source/), you can use the following query,
+      substituting your source name for `<source_name>`:
+
+      <br>
+
+      ```mzsql
+      SELECT refs.*
+      FROM mz_internal.mz_source_references refs, mz_sources s
+      WHERE s.name = '<source_name>' -- substitute with your source name
+      AND refs.source_id = s.id;
+      ```
+
+  - "Parameter": "**WITH (<with_option>[,...])**"
+    "Description": |
+      The following `<with_option>`s are supported:
+
+      | Option | Description |
+      |--------|-------------|
+      | <a name="text-columns"></a>`TEXT COLUMNS (<fq_column_name> [, ...])` | *Optional.* If specified, decode data as `text` for the listed column(s), such as for unsupported data types. Use fully qualified column names. See also [supported types](#supported-db-source-types). |
+      | <a name="exclude-columns"></a>`EXCLUDE COLUMNS (<fq_column_name> [, ...])` | *Optional.* If specified, exclude the listed column(s) from the table. Use fully qualified column names. |
+      | <a name="partition-by"></a>`PARTITION BY (<column> [, ...])` | *Optional.* The key by which Materialize should internally partition the table. See the [partitioning guide](/transform-data/patterns/partition-by/) for restrictions on valid values and other details. |