1.49.1

Merged PRs

dolt

• 8831: Expand ItemAccess::itemWidth to 32 bits
  ItemAccess is a class used to read data out of prolly tree nodes. Because the itemWidth field was limited to 16 bits, reading any value larger than 2^16 bytes would result in silent truncation.
  We don't usually store values this large, although it should be safe to do so.
  This issue was discovered because the new JSON chunker (introduced in #7912) always stores embedded strings as a single chunk, so a document containing a string larger than 32KB would result in a node with a single value whose length didn't fit in 16 bits.
  While we were investigating this issue, we created #8723 to disable the new JSON chunker in the presence of these long strings. This PR partially reverts that one, resuming the smart chunking of JSON even in the presence of large embedded strings.
• 8817: Bug fix for encoding extended types in keys
  This PR allows certain Doltgres extended types to be correctly serialized as part of keys

go-mysql-server

• 2845: Added external function provider
  This adds an external function provider, currently needed for Doltgres to get function creation working. This is intended to be a temporary measure until a more permanent solution is developed (which may involve modifying Dolt's DatabaseProvider).
• 2844: Correctly skip skipped assertions in transaction tests
• 2843: Replace golang.org/x/exp with stdlib
  These experimental packages are now available in the Go standard library.
  1. golang.org/x/exp/slices -> slices (https://go.dev/doc/go1.21#slices)
  2. golang.org/x/exp/maps -> maps (https://go.dev/doc/go1.21#maps)
  3. golang.org/x/exp/constraints -> cmp (https://go.dev/doc/go1.21#cmp)
  4. golang.org/x/exp/rand -> math/rand/v2 (https://go.dev/doc/go1.22#math_rand_v2)
• 2842: fix panic when comparing system set type against other types
  Comparison between systemSetTypes and other types is still not correct.
  It appears that MySQL actually treats @@sql_mode as just a string.
  This PR only fixes the panic
• 2839: fix case insensitivity and return type for str_to_date
  This PR fixes an issue with the str_to_date function where we wouldn't match string literals in the date with literals in the format, because we were improperly converting them to lowercase.
  Additionally, this PR has it so the str_to_date function returns a time.Time instead of a string. This gets us closer to MySQL behavior over the server.
  fixes: #8807
• 2833: refactor create procedure and call procedure
  This PR refactors a ton of the stored procedure behavior to more closely match MySQL.
  Changes:
  • properly error when creating procedures/triggers/events in other stored routines
  • allow create procedure statements that reference symbols (tables, views, procedures, etc) that don't exist (yet).
  • remove unnecessary analysis and building work
  • simplify building plan.Call nodes
    Partially addresses: #8053

Closed Issues

• 8807: Issue with "T" when using str_to_date

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.65	0.34
groupby_scan	13.22	18.61	1.41
index_join	1.47	2.48	1.69
index_join_scan	1.44	1.47	1.02
index_scan	34.33	31.37	0.91
oltp_point_select	0.18	0.27	1.5
oltp_read_only	3.49	5.37	1.54
select_random_points	0.33	0.61	1.85
select_random_ranges	0.37	0.64	1.73
table_scan	34.33	33.72	0.98
types_table_scan	75.82	123.28	1.63
reads_mean_multiplier			1.33

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	9.06	6.32	0.7
oltp_insert	4.1	3.13	0.76
oltp_read_write	9.06	11.65	1.29
oltp_update_index	4.18	3.19	0.76
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.77	6.32	1.1
types_delete_insert	8.43	6.67	0.79
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.45	39.6	2.44
tpcc_tps_multiplier			2.44

Overall Mean Multiple	1.55

1.49.0

Backwards Incompatible Changes

The --user and --password arguments to dolt sql-server have been deprecated. This change affects the command line arguments as well as the configuration file parameters for dolt sql-server. These options previously created a temporary superuser with the specified name and optional password; however, since support for SQL user and privilege management was added, these arguments introduced edge cases that caused issues for customers. Instead, customers should use the standard SQL support for managing users and grants. For more information on this change, including instructions on how to manage users and grants with SQL, see our announcement blog post for this change.

Per Dolt’s versioning policy, this is a minor version bump because these changes may impact existing applications. Please reach out to us on GitHub or Discord if you have questions or need help with any of these changes.

Merged PRs

dolt

• 8812: [kvexec] fix left join nullability bug
  We were using a stale variable for a nullability comparison.
  test here: dolthub/go-mysql-server#2840
• 8809: go: sqle/binlogreplication: Add some waits for more reliable observations of IO_error results.
• 8808: [kvexec] literal types cast to lookup expr type
• 8806: replace cli printErrf with printf in server startup logging
• 8804: Include deleteErr message in returned error
• 8801: go: store/types: Fix dolt_gc on databases that use vector indexes.
• 8800: Remove user and password options for sql-server
  The --user and --password options to sql-server (whether specified on the command line or in a configuration file) allow users to specify an ephemeral superuser to use for the life of the sql-server process. This is a convenient way to temporarily create a superuser, but has several edge cases that can cause unexpected behavior.
  This change removes support for the --user and --password options and instead customers should now explicitly create that user using standard SQL statements for managing users, such as:

  CREATE USER myUser@'%' IDENTIFIED BY 'myPassw0rd';
  GRANT ALL ON *.* TO myUser@'%';

  If user or password is specified in a config file, the sql-server will still startup, but will log a warning. If --user or --password is specified as a CLI argument the sql-server will error out with an error message explaining the change. The reason for the difference in handling is that it may be harder for customers to update config files.
• 8783: Avoid loading DB for commands where it's not necessary.
  This PR avoids loading dolt DBs at startup, instead waiting for either DoltEnv.ReloadDB or MultiRepoEnv.ReloadDBs to be called.
  Based on the specific subcommand being run:
  • MultiRepoEnv.ReloadDBs is called prior to command execution if the command will always need DB access.
  • Commands that conditionally require a DB can choose whether to call DoltEnv.ReloadDB
  • Commands that never need the DB will never load it.
    Commands that make use of CliContext don't need to load the DB, as the underlying query engine will load the DB if it needs database access. As a result, commands that are implemented purely in terms of the CliContext will load the DB when Dolt is runnning an embedded SQL engine, and will skip loading the DB when Dolt is connecting to a running server.
• 8774: /go/libraries/doltcore/env/actions: make iter resolved tags paginated sort in lexicographical order
• 8732: Tidy up journal writer code comments and error handling
  Before this change, the code that iterates over journal records would silently stop processing journal records once it encountered any invalid record. This change turns those invalid records into errors so that data isn't silently ignored. Customers can opt-in to the previous behavior by setting the DOLT_SKIP_INVALID_JOURNAL_RECORDS environment variable to any value.

go-mysql-server

• 2835: Added the concept of an interpreter expression
  This adds the concept of an interpreter expression, which is primarily used by integrators.
• 2828: return error for ddl in create proc statements
  This PR moves the logic for validateStoredProcedure analyzer rule into the planbuilder, allowing for proper error messages.
  Related: #8762

Closed Issues

• 2830: Dump data to file
• 2829: Will there be a postgres version of this?

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.64	0.34
groupby_scan	13.22	17.32	1.31
index_join	1.47	2.48	1.69
index_join_scan	1.42	1.44	1.01
index_scan	35.59	30.26	0.85
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.49	5.28	1.51
select_random_points	0.34	0.58	1.71
select_random_ranges	0.37	0.62	1.68
table_scan	36.24	32.53	0.9
types_table_scan	75.82	114.72	1.51
reads_mean_multiplier			1.27

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.21	0.7
oltp_insert	4.1	3.07	0.75
oltp_read_write	8.9	11.45	1.29
oltp_update_index	4.18	3.13	0.75
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.77	6.21	1.08
types_delete_insert	8.43	6.55	0.78
writes_mean_multiplier			0.87

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	95.73	40.13	2.39
tpcc_tps_multiplier			2.39

Overall Mean Multiple	1.51

1.48.0

Backwards incompatible changes in this release:

The dolt_branches system table has changed. The table now has a ninth column to indicate that a branch is dirty. Some software may depend on the dolt_branches table having exactly eight columns, and will require updates to work with this latest version.

Per Dolt’s versioning policy, this is a minor version bump because these changes may impact existing applications. Please reach out to us on GitHub or Discord if you have questions or need help with any of these changes.

Merged PRs

dolt

• 8793: Add the dirty column to the dolt_branches table
  This change allows users, particularly on servers, to quickly determine which branches have a dirty working set.

go-mysql-server

• 2828: return error for ddl in create proc statements
  This PR moves the logic for validateStoredProcedure analyzer rule into the planbuilder, allowing for proper error messages.
  Related: #8762
• 2825: Bug fixes for serializing certain types in column default expressions
  Fixed a couple bugs in column default normalization: 1) editing the expression tree in place so the change is invisible to debug logging, 2) normalizing some literals that should manage their own round-trip serialization
• 2824: implement drop temporary table ... and display temporary in show create table statements
  In-memory database doesn't support temporary tables, so tests are here: #8781
  fixes: #7992

Closed Issues

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.93	0.64	0.33
groupby_scan	15.27	17.32	1.13
index_join	1.47	2.48	1.69
index_join_scan	1.42	1.44	1.01
index_scan	34.33	30.26	0.88
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.43	5.18	1.51
select_random_points	0.34	0.59	1.74
select_random_ranges	0.37	0.62	1.68
table_scan	34.33	32.53	0.95
types_table_scan	75.82	116.8	1.54
reads_mean_multiplier			1.26

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.21	0.7
oltp_insert	4.1	3.07	0.75
oltp_read_write	8.9	11.24	1.26
oltp_update_index	4.18	3.13	0.75
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.21	1.1
types_delete_insert	8.28	6.55	0.79
writes_mean_multiplier			0.87

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.53	39.63	2.44
tpcc_tps_multiplier			2.44

Overall Mean Multiple	1.52

1.47.2

Merged PRs

dolt

• 8788: Bug fix: ref conflict can cause tags to be temporarily removed on a replica
  The read replica code that compares remote refs and local refs to determine which local refs need to be deleted when synchronizing a read replica from a remote had a problem where certain patterns of ref names could cause the code to select the wrong local refs to remove. These removed refs would not be present on the replica until a future synchronization run recreated them.
  The fix for this is to use the Ref.String() method (instead of Ref.GetPath()) to get a unique identifier for the ref (e.g. refs/tags/t1) and to ensure that the remote and local ref lists are both sorted by that unique identifier.
• 8787: better nil transaction checking for commits
• 8781: implement and test drop temporary table ... and show create table statements
  addresses the related issue here: #8762
• 8780: go/store/types: Move to a safepoint controller which will allow a caller better control over when to take actions while the GC is running.
• 8777: Bump undici from 5.28.4 to 5.28.5 in /.github/actions/ses-email-action
  Bumps undici from 5.28.4 to 5.28.5.
  Release notes

  Sourced from undici's releases.

  v5.28.5

  ⚠️ Security Release ⚠️

  Fixes CVE CVE-2025-22150 GHSA-c76h-2ccp-4975 (embargoed until 22-01-2025).

  Full Changelog: nodejs/undici@v5.28.4...v5.28.5

  Commits

  • 6139ed2 Bumped v5.28.5
  • 711e207 Backport of c2d78cd
  • See full diff in compare view
  
  [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
  Dependabot commands and options

  
  You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/dolthub/dolt/network/alerts).
• 8773: Fix failing MacOS BATS tests
  Testing Dolt's caching_sha2_password auth support with the mysql client on MacOS recently started failing with: mysql: command not found
  These tests were previously working, but the mysql client seems to no longer be available. This change adds a new step to ensure the current LTS mysql client is installed.
  These tests normally run nightly, but they passed in an earlier CI run on this PR that triggered them.
• 8767: implement implicit commits and fix import behavior
  This PR implements the implicit commit logic introduced in this PR dolthub/go-mysql-server#2818
  On error, dolt table import -c does not create the table, and we did that by simply rolling back the existing transaction.
  Since DDL statements now implcitly COMMIT, we need to start a new transaction, and possibly drop any tables created.
  fixes #7485
  maybe fixes: #8716
• 8739: Create dolt_help system table
  Created the dolt_help system table. This table is meant to store documentation for system tables, procedures, functions, and variables. Currently dolt_help is only populated with documentation for procedures, and only procedures that have equivalent CLI commands.
  Part of #7984

go-mysql-server

• 2824: implement drop temporary table ... and display temporary in show create table statements
  In-memory database doesn't support temporary tables, so tests are here: #8781
  fixes: #7992
• 2822: Bug fix: record last_insert_id when 0 is specified for the auto_increment column value
  fixes: #8769
• 2819: Don't force a table rewrite when appending extra values to the end of an enum.
  Adding extra strings to the end of an enum type doesn't change the values for any of the existing strings. A table rewrite isn't necessary in this case.
  If a specific table implementation does need to be rewritten when an enum type changes this way, they can still implement ShouldRewriteTable in order to force a rewrite anyway.
• 2818: implicit commit on ddl statements
  Certain queries ignore autocommit, and will implicitly commit the transaction.

vitess

• 395: syntax for drop temporary table ...
• 394: parse more partition options in ALTER TABLE statements
  parses more partition options as no-ops
  fixes: #8744

Closed Issues

• 8716: Multiple Users report "table not found" errors when using Dolt under concurrency
• 7980: Create a config.yaml file when database server is first run
• 7992: SHOW CREATE TABLE doesn't include TEMPORARY for temporary tables
• 7485: MySQL DDL ignores autocommit transactionality
• 8769: last_insert_id gives 0 when 0 is explicitly specified for an auto_increment primary key in an insertion

1.47.1

Merged PRs

dolt

• 8770: Speed up fetch when there are many tags which haven't changed
  A user remarked that dolt pull took 2 hours to pull changes. This was the result of wasting time for every tag which had not changed. This change alters the tag iteration code to defers the loading of metadata until it's actually required. Testing against user takes less that 1min now.
• 8753: make autoincrement tracker load async
• 8752: go/store/{nbs,types}: GC: Move the reference walk from types to nbs.
  Make the ChunkStore itself responsible for the reference walk, being given handles for walking references and excluding chunks as part of the GC process. This is an incremental step towards adding dependencies on read chunks during the GC process. The ChunkStore can better distinguish whether the read is part of the GC process itself or whether it came from the application layer. It also allows better management of cache impact and the potential for better memory usage.
  This transformation gets rid of parallel reference walking and some manual batching which was present in the ValueStore implementation of reference walking. The parallel reference walking was necessary for reasonable performance in format LD_1, but it's actually not necessary in DOLT. For some use cases it's a slight win, but the simplification involved in getting rid of it is worth it for now.
• 8747: go/libraries/doltcore/sqle/dprocedures: dolt_gc.go: Retry canceling running queries when waiting for safepoint establishment.
  This allows call dolt_gc() to more quickly and realibly establish a safepoint if the call to safepointF() races with a new query beginning and being registered for the connection in the process list.

go-mysql-server

• 2820: Don't close ctx prematurely for single row results
  User reported cancelled context error which is caused by prematurely closing the iterator when the ctx is still needed.
• 2819: Don't force a table rewrite when appending extra values to the end of an enum.
  Adding extra strings to the end of an enum type doesn't change the values for any of the existing strings. A table rewrite isn't necessary in this case.
  If a specific table implementation does need to be rewritten when an enum type changes this way, they can still implement ShouldRewriteTable in order to force a rewrite anyway.
• 2817: Use vector index when the SELECT cause has a projection.
  Due to some overly strict pattern matching in the vector index selection, we weren't always using the index when there was a projection involved: we were only applying the index in the presence of a TopN node, but we also weren't generating TopN nodes in the case we had a Limit -> Project -> Sort node structure.
  I was hoping that dolthub/go-mysql-server#2813 would fix this, and I suspect there's improvements to GMS that would make this unnecessary. But for now, we should allow the pattern matching in replaceIdxOrderByDistance to apply a vector index lookup in this case.

Closed Issues

• 8769: last_insert_id gives 0 when 0 is explicitly specified for an auto_increment primary key in an insertion

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.65	0.34
groupby_scan	13.22	17.63	1.33
index_join	1.47	2.48	1.69
index_join_scan	1.42	1.44	1.01
index_scan	34.95	30.81	0.88
oltp_point_select	0.18	0.27	1.5
oltp_read_only	3.49	5.37	1.54
select_random_points	0.34	0.61	1.79
select_random_ranges	0.37	0.63	1.7
table_scan	34.95	32.53	0.93
types_table_scan	77.19	116.8	1.51
reads_mean_multiplier			1.29

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.74	6.32	0.72
oltp_insert	4.1	3.07	0.75
oltp_read_write	8.9	11.45	1.29
oltp_update_index	4.18	3.13	0.75
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.32	1.11
types_delete_insert	8.28	6.67	0.81
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	95.28	39.62	2.4
tpcc_tps_multiplier			2.4

Overall Mean Multiple	1.52

1.47.0

Vector indexes and search are supported in this release.

Merged PRs

dolt

• 8749: Allow importing parquet fields containing repeated elements.
  NOTE: This still needs tests. I'm looking for a good tool for generating parquet. We can't use dolt table export to generate the parquet because we can't generate composite types that way.
  This PR adds support for importing specific composite parquet types into Dolt. Specifically, we're now able to import a compose parquet field if:
  • There is exactly one leaf column in the field.
  • There is at most one repeated tag in the field.
    We flatten these composite values into a single primitive value (if there are no repeated tags) or an array of primitive values (if there's exactly one repeated tag.)
    There's more work to be done here (multidimensional arrays, objects, etc), but this allows us to import vector embedding stored in parquet files.

  Why do we flatten the type?

  We want to be able to import parquet files from HuggingFace, and store embedding sequences as arrays. Embedding sequences in HuggingFace exports are an optional field containing a single repeated child field, which itself contains a single optional field containing the sequence element. Flattening this into a single array is more usable and doesn't lose any data.
• 8686: Proximity Map implementation with support for incremental edits.
  Based on #8408, now with additional functionality for incremental changes to indexes.
  This is a large-scale PR merging several features into main, all designed for supporting vector indexes.

  Vector Index Nodes

  1defec9 adds a new message/node type: the vector index node. This message stores a node in a Merkle tree index whose structure is based on some distance measure in a multi-dimensional space: at each level, keys are arranged such that a key is closer to its parent key than any other key in the parent node.
  One consequence of this design is that it's not possible to put a hard limit on the number of keys contained in each node. We can control the mean node size, but there's always a non-zero chance that a node will be large enough to break our usual encoding scheme (which uses 16-bit ints to store message offsets). To address this, the vector index node uses 32-bit ints to store message offsets instead of the 16 bits used by other node types.

  Proximity Map

  A ProximityMap is a new implementation of Dolt's Map, a data structure built on Merkle trees that maps key bytestrings to value bytestrings. The ProximityMap is backed by a tree of vector index nodes, allowing it to perform an approximate nearest neighbor search.
  Proximity Maps resemble other Prolly Maps, but have the following invariants:
  • Each key must be convertible to a vector. Typically, the key is a val.Tuple, and the vector is the first value in that tuple.
  • The keys are arranged in the tree such that, for each of a key's parent keys (the keys that appear on the path from the root to the key), the key is closer to that parent key than any of the parent key's siblings.
  • The keys in a node are sorted lexographically (note that this is not necessarily the same ordering as the tuple that the key represents), except for the first key which matches its direct parent.
    Notably, while the keys of an individual node are sorted, walking all of a vector indexes keys in standard iteration order will not be sorted.
    28b7065 and 6b91635 contain the bulk of the ProximityMap implementation.
    The bulk of the changes are in these three commits. Each of the other commits is a smaller self-contained change necessary to support vector indexes.

go-mysql-server

• 2817: Use vector index when the SELECT cause has a projection.
  Due to some overly strict pattern matching in the vector index selection, we weren't always using the index when there was a projection involved: we were only applying the index in the presence of a TopN node, but we also weren't generating TopN nodes in the case we had a Limit -> Project -> Sort node structure.
  I was hoping that dolthub/go-mysql-server#2813 would fix this, and I suspect there's improvements to GMS that would make this unnecessary. But for now, we should allow the pattern matching in replaceIdxOrderByDistance to apply a vector index lookup in this case.
• 2816: Allow using vector index when the queried vector is provided in a user variable.
  Right now, vector indexes are very narrowly applied. One of the inputs to the DISTANCE function needs to be a constant. Before we required it to be a Literal expression, but UserVar expressions should also work.
• 2797: Persist and load superusers
  Previously, superusers were persisted to disk, but never loaded back again when the database was restarted. This essentially made all superusers ephemeral, since they only lasted for the duration of a SQL server process.
  This change loads persisted superusers from disk, and also adds a new function to create ephemeral superusers that do not get persisted to disk.
  This also includes a fix for the event scheduler to use a privileged account so that it can load events from all databases.

Closed Issues

• 8734: Can't delete remote branch refs that no longer exist in origin

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.68	0.36
groupby_scan	12.98	17.32	1.33
index_join	1.44	2.48	1.72
index_join_scan	1.42	1.44	1.01
index_scan	34.33	30.81	0.9
oltp_point_select	0.18	0.27	1.5
oltp_read_only	3.49	5.37	1.54
select_random_points	0.34	0.6	1.76
select_random_ranges	0.37	0.63	1.7
table_scan	34.95	33.12	0.95
types_table_scan	75.82	118.92	1.57
reads_mean_multiplier			1.3

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	9.06	6.32	0.7
oltp_insert	4.1	3.13	0.76
oltp_read_write	9.06	11.65	1.29
oltp_update_index	4.18	3.19	0.76
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.77	6.32	1.1
types_delete_insert	8.43	6.67	0.79
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.45	39.71	2.43
tpcc_tps_multiplier			2.43

Overall Mean Multiple	1.54

1.46.0

Backwards incompatible changes in this release:

• The default root superuser is now persisted to the privileges database and is scoped to localhost, instead of %. Previously, the root superuser only existed when no other accounts had been created. Creating accounts, then restarting the sql-server would cause the root superuser to disappear. The root@localhost superuser is now created the first time a sql-server is started, as the privileges database is initialized.
• For Docker customers – note that the default root superuser is now scoped to localhost, instead of any host. This change is made for security and to better match MySQL's default security posture. To connect to a Dolt sql-server from outside the container the sql-server is running on, you need to supply the -e DOLT_ROOT_HOST=<host> argument. For more details and examples, see the dolthub/sql-server Docker readme, our Docker documentation, or our blog post covering this change.

Per Dolt’s versioning policy, this is a minor version bump because these changes may impact existing applications. Please reach out to us on GitHub or Discord if you have questions or need help with any of these changes.

Merged PRs

dolt

• 8746: Allow dolt sql to always log in as the root superuser
  From the command line, when a sql-server is not running, dolt sql implicitly uses the root account to log, but if the root account exists with a password, dolt sql will fail to log in. Since the user has access to the host and the database data directory, we should allow dolt sql to log into the SQL shell, even if the root user has a password set. This change also makes this behavior match when a sql-server is running, and we allow superuser login through the __dolt_local_user__ account (which only exists while a sql-server is running).
• 8745: Add --prune option to dolt_pull procedure
  Expose in CLI and test too.
  See: #8734
• 8742: Don't panic when attempting to update workspace table
  Previously a panic was very likely if any update to dolt_workspace_* involved a schema change. This change restricts the updates to the workspace tables only in cases where the schemas have not changed.
• 8740: /go/libraries/doltcore/sql/dsess: parallelize sql.NewDatabase work
• 8690: Initialize persisted root superuser on SQL server startup
  Previously, Dolt would only create a root superuser on sql-server startup when no other user accounts had been created. This resulted in a behavior where users would run dolt sql-server, create user accounts, then the next time they restart the sql-server, the root account would no longer be present. This behavior has surprised several customers (see #5759) and is different from MySQL's behavior, which creates a persistent root superuser as part of initialization.
  This change modifies this behavior so that a root superuser is created, and persisted, the first time a SQL server is started for a database, unless the --skip-root-user-initialization flag is specified, or if an ephemeral super user is requested with the --user option. Subsequent runs of dolt sql-server do not automatically create the root superuser – only the first time dolt sql-server is started when there is no privileges database yet, will trigger the root user to be created and the privileges database to be initialized
  Internally, this is implemented by detecting the presence of any user account and privilege data stored to disk (by default, in the .doltcfg/privileges.db file). When no user account and privilege data exists, the root superuser initialization logic will run. This means the privileges.db data is now always created on the first run of dolt sql-server, even if the data is empty.
  As part of this change, the root superuser is now scoped to localhost, instead of % (i.e. any host). This improves the default security posture of a Dolt sql-server and better aligns with MySQL's behavior. Customers who rely on using the root account to connect from non-localhost hosts, will need to either log in and alter the root account to allow connections from the hosts they need, or they can specify the DOLT_ROOT_HOST and/or DOLT_ROOT_PASSWORD environment variables to override the default host (localhost) and password ("") for the root account when it is initialized the first time a sql-server is launched.
  One side effect of this change is that dolt sql -u <user> may work differently for some uses. Previously, if there was no user account and privilege data persisted to disk yet (i.e. the .doltcfg/privileges.db file), then users could specify any username and password to dolt sql (e.g. dolt sql -u doesnotexist) and they would still be logged in – user authentication was ignored since no user account and privilege data existed. Now that the user account and privilege data is always initialized when running dolt sql-server, customers may no longer use dolt sql --user <user> to log in with unknown user accounts. The workaround for this is to simply run dolt sql without the --user option, and Dolt will use the default local account.
  Fixes: #5759
  Depends on: dolthub/go-mysql-server#2797
  Related to: dolthub/doltgresql#1113
  Documentation updates: dolthub/docs#2460

go-mysql-server

• 2814: [rowexec] full outer join rightIter exhaust
  Full join should exhaust right side, not return as soon as we EOF the left iterator.
  fixes: #8735
• 2813: [binder] hoist projections in certain cases where we can combine with top-level projection
  This is a bit unintuitive, but hoisting projections above sorts in the binder seems to uniformly improve projection pruning because we will always combine it with the top-level return projection.
  fixes: #8736
• 2812: Fix cte naming conflict
  fixes: #8724
  Distinct CTE references need unique column and table ids.
• 2811: Reset BytesBuffer after each rowBatch
  Once we spool a batch of rows to client, there's no reason to keep them in memory.
  Fixes #8718
• 2797: Persist and load superusers
  Previously, superusers were persisted to disk, but never loaded back again when the database was restarted. This essentially made all superusers ephemeral, since they only lasted for the duration of a SQL server process.
  This change loads persisted superusers from disk, and also adds a new function to create ephemeral superusers that do not get persisted to disk.
  This also includes a fix for the event scheduler to use a privileged account so that it can load events from all databases.

vitess

• 394: parse more partition options in ALTER TABLE statements
  parses more partition options as no-ops
  fixes: #8744
• 393: fix starting by and terminated by order
  the starting by and terminated by clauses in load data statements can appear in any order and any number of times.

Closed Issues

• 5759: Dolt's disappearing root user is confusing
• 8744: Parser support for adding/removing partition

1.45.6

Merged PRs

dolt

• 8730: Make show command more resilient when resolving references
  Currently the show command can print internal objects, which requires a local environment. This goes against the sql migration expectations that there is no environment. This change only makes the situation less bad. Splitting out the admin operations into another command is the right approach.
  Fixes: #8727
• 8726: Replace min/max helpers with built-in min/max
  We can use the built-in min and max functions since Go 1.21.
  Reference: https://go.dev/ref/spec#Min_and_max
• 8719: Replace min/max helpers with built-in min/max
  We can use the built-in min and max functions since Go 1.21.
  Reference: https://go.dev/ref/spec#Min_and_max
• 8644: Generate config.yaml when running sql-server
  If sql-server is ran without a specified config file and there is no config.yaml in the database directory, one will be generated.
  The rules for how the default config.yaml file is generated are as follows:
  • If a field has a value defined by the sql-server execution (such as through CLI args), then that value will be used.
  • If a field has no set value but has a default value, then that default will be used.
  • If a field has no set value and no default, a commented-out line setting the field to null will be included as a placeholder.
    Part of #7980

go-mysql-server

• 2811: Reset BytesBuffer after each rowBatch
  Once we spool a batch of rows to client, there's no reason to keep them in memory.
  Fixes #8718

Closed Issues

• 8736: Sort/Alias dependency conflict preventing prune
• 8735: [Bug] FULL OUTER JOIN not commutative, and not giving correct results
• 8724: [Bug] Incorrect SUM calculation in CTE with correlated subquery
• 8727: \show in SQL shell panics when arg isn't a commit
• 8718: mydumper OOMs in deterministic fashion for large database

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.89	0.65	0.34
groupby_scan	13.22	17.32	1.31
index_join	1.44	2.43	1.69
index_join_scan	1.42	1.44	1.01
index_scan	34.33	30.81	0.9
oltp_point_select	0.18	0.26	1.44
oltp_read_only	3.49	5.37	1.54
select_random_points	0.37	0.6	1.62
select_random_ranges	0.4	0.63	1.57
table_scan	34.33	33.12	0.96
types_table_scan	74.46	114.72	1.54
reads_mean_multiplier			1.27

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.21	0.7
oltp_insert	4.1	3.07	0.75
oltp_read_write	9.06	11.45	1.26
oltp_update_index	4.18	3.13	0.75
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.32	1.11
types_delete_insert	8.43	6.55	0.78
writes_mean_multiplier			0.87

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	95.72	39.79	2.41
tpcc_tps_multiplier			2.41

Overall Mean Multiple	1.52

1.45.5

Merged PRs

dolt

• 8725: More information about types in int conversions

Closed Issues

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	1.93	0.67	0.35
groupby_scan	13.22	17.32	1.31
index_join	1.47	2.48	1.69
index_join_scan	1.44	1.47	1.02
index_scan	34.33	30.81	0.9
oltp_point_select	0.18	0.27	1.5
oltp_read_only	3.49	5.37	1.54
select_random_points	0.34	0.6	1.76
select_random_ranges	0.37	0.63	1.7
table_scan	34.33	33.12	0.96
types_table_scan	75.82	114.72	1.51
reads_mean_multiplier			1.29

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.9	6.32	0.71
oltp_insert	4.1	3.13	0.76
oltp_read_write	8.9	11.45	1.29
oltp_update_index	4.18	3.19	0.76
oltp_update_non_index	4.18	3.07	0.73
oltp_write_only	5.67	6.32	1.11
types_delete_insert	8.43	6.67	0.79
writes_mean_multiplier			0.88

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	96.35	39.77	2.42
tpcc_tps_multiplier			2.42

Overall Mean Multiple	1.53

1.45.4

Merged PRs

dolt

• 8723: If a JSON document contains strings that can't fit in a single chunk, use the naive Blob chunker instead of the smart JSON chunker.
  The JSON chunker never creates a chunk boundary inside of a string.
  Originally, this PR added functionality to allow the JSON chunker to split JSON document inside a string. This was supposed to be safe and backwards compatible, because older versions of Dolt reading documents written by newer versions of Dolt are supposed to fall back on ignoring JSON document metadata if they don't understand it and treat the document like a blob.
  However, tests revealed that older clients were not checking for this in enough places and would hang when trying to read documents written with this fix. This PR also contains fixes to check the JSON metadata in more places... but this doesn't do anything for existing Dolt servers running older versions.
  So instead, this PR detects when a document contains strings that exceed some limit, and instead the writer falls back on writing the document as a plain blob without metadata. The limit is currently 32KB, but can be raised in the future.
  I chose to keep the logic for splitting JSON documents inside a string, although the chunker doesn't currently use it, since we may decide to enable it in the future.

Closed Issues