Skip to content

Commit b901fe8

Browse files
theorytheory
committed
support for YugabyteDB
Works great with the Postgres engine except for locks. So detect that it's Yugabyte and don't create the locks if it is. Change the handling of the `lock_sql` option in the database tests to take a code reference instead of a hash reference, and only return the locking instructions in `t/pg.t` if the engine is not Yugabyte. Mention the Yugabyte support here and there. Alway sort the current state test tags for Yugabyte, as previously done only for Oracle and MySQL. The Postgres aggregate docs say that the ordering of `array_agg()` is undefined, though it has been reliable in all testing. Not so for Yugabyte. So we force the sort for Yugabyte, but also added #636 to propose a better path forward for aggregate sorting Add GitHub Actions testing for Yugabyte 2.6 and higher, and link it in the README. While at it, clean up the README, properly listing all supported engines and linking to them.
1 parent 2dea22d commit b901fe8

12 files changed

Lines changed: 135 additions & 36 deletions

File tree

‎.github/workflows/pg.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
# This workflow tests Sqitch's PostgreSQL engine on all supported versions of
22
# Postgres. It runs for pushes and pull requests on the `main`, `develop`,
3-
# `**postgres**`, and `**engine**` branches.
3+
# `**postgres**`, `**yugabyte**`, and `**engine**` branches.
44
name: 🐘 Postgres
55
on:
66
push:
7-
branches: [main, develop, "**engine**", "**postgres**" ]
7+
branches: [main, develop, "**engine**", "**postgres**", "**yugabyte**" ]
88
pull_request:
9-
branches: [main, develop, "**engine**", "**postgres**" ]
9+
branches: [main, develop, "**engine**", "**postgres**", "**yugabyte**" ]
1010
jobs:
1111
Postgres:
1212
strategy:

‎.github/workflows/yugabyte.yml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# This workflow tests Sqitch's PostgreSQL engine on all supported versions of
2+
# YugabyteDB. It runs for pushes and pull requests on the `main`, `develop`,
3+
# `**postgres**`, `**yugabyte**`, and `**engine**` branches.
4+
name: 💫 YugabyteDB
5+
on:
6+
push:
7+
branches: [main, develop, "**engine**", "**postgres**", "**yugabyte**" ]
8+
pull_request:
9+
branches: [main, develop, "**engine**", "**postgres**", "**yugabyte**" ]
10+
jobs:
11+
Yugabyte:
12+
strategy:
13+
matrix:
14+
include:
15+
- { version: '2.13', tag: 2.13.2.0-b135 }
16+
- { version: '2.12', tag: 2.12.5.0-b24 }
17+
- { version: '2.8', tag: 2.8.6.0-b12 }
18+
- { version: '2.6', tag: 2.6.18.0-b3 }
19+
name: 💫 YugabyteDB ${{ matrix.version }}
20+
runs-on: ubuntu-latest
21+
steps:
22+
- name: Setup YugabyteDB cluster
23+
id: yugabyte
24+
uses: yugabyte/yugabyte-db-action@master
25+
with:
26+
yb_image_tag: "${{ matrix.tag }}"
27+
- uses: actions/checkout@v2
28+
- name: Setup Perl
29+
id: perl
30+
uses: shogo82148/actions-setup-perl@v1
31+
with: { perl-version: latest }
32+
- name: Cache CPAN Modules
33+
uses: actions/cache@v2
34+
with:
35+
path: local
36+
key: perl-${{ steps.perl.outputs.perl-hash }}
37+
- run: cpm install --verbose --show-build-log-on-failure --no-test --with-recommends --cpanfile dist/cpanfile
38+
- run: cpm install --verbose --show-build-log-on-failure --no-test --with-recommends DBD::Pg
39+
- name: prove
40+
env:
41+
PERL5LIB: "${{ github.workspace }}/local/lib/perl5"
42+
LIVE_PG_REQUIRED: true
43+
SQITCH_TEST_PG_URI: db:pg://yugabyte@localhost:${{ steps.yugabyte.outputs.ysql_port }}/
44+
run: prove -lvr t/pg.t

‎Changes‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
Revision history for Perl extension App::Sqitch
22

33
1.2.2
4-
54
- Fixed an issue when testing Firebird on a host with Firebird installed
65
but no `isql`, and when using a local Firebird (e.g., the Engine12
76
provider), which allows only one connection at a time. Thanks to Slaven
87
Rezić for the the reproducible configuration (#597).
8+
- Tweaked the Postgres engine to support Yugabyte. The only unsupported
9+
features are explicit locks, so users need to manually ensure that only
10+
one instance of Sqitch is updating the cluster at a time.
911

1012
1.2.1 2021-12-05T19:59:45Z
1113
- Updated all the live engine tests, aside from Oracle, to test with

‎README.md‎

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,20 @@ App/Sqitch version v1.2.2-dev
77
| [![Docker]][🐳] | [![Perl]][🧅] | [![Firebird]][🔥] | [![Snowflake]][❄️] |
88
| [![Homebrew]][🍺] | [![Coverage]][📈] | [![MySQL]][🐬] | [![SQLite]][💡] |
99
| [![Debian]][🍥] | | [![Postgres]][🐘] | [![Vertica]][🔺] |
10+
| | | [![Yugabyte]][🚀] | |
1011

11-
[Sqitch] is a database change management application. It currently supports
12-
PostgreSQL 8.4+, SQLite 3.7.11+, MySQL 5.0+, Oracle 10g+, Firebird 2.0+, Vertica
13-
6.0+, Exasol 6.0+ and Snowflake.
12+
[Sqitch] is a database change management application. It currently supports:
13+
14+
* [PostgreSQL] 8.4+
15+
* [YugabyteDB] 2.6+
16+
* [SQLite][lite] 3.7.11+
17+
* [MySQL][my] 5.0+
18+
* [MariaDB] 10.0+
19+
* [Oracle][orcl] 10g+,
20+
* [Firebird][bird] 2.0+
21+
* [Vertica][vert] 6.0+
22+
* [Exasol][exa] 6.0+
23+
* [Snowflake][flake]
1424

1525
What makes it different from your typical migration approaches? A few things:
1626

@@ -24,7 +34,7 @@ What makes it different from your typical migration approaches? A few things:
2434

2535
Changes are implemented as scripts native to your selected database engine.
2636
Writing a [PostgreSQL] application? Write SQL scripts for [`psql`]. Writing
27-
an [Oracle]-backed app? Write SQL scripts for [SQL\*Plus].
37+
an [Oracle][orcl]-backed app? Write SQL scripts for [SQL\*Plus].
2838

2939
* Dependency resolution
3040

@@ -176,13 +186,23 @@ SOFTWARE.
176186
[🍥]: https://packages.debian.org/stable/sqitch "Latest version on Debian"
177187
[Postgres]: https://github.com/sqitchers/sqitch/actions/workflows/pg.yml/badge.svg
178188
[🐘]: https://github.com/sqitchers/sqitch/actions/workflows/pg.yml "Tested with PostgreSQL 9.3–14"
189+
[Yugabyte]: https://github.com/sqitchers/sqitch/actions/workflows/yugabyte.yml/badge.svg
190+
[🚀]: https://github.com/sqitchers/sqitch/actions/workflows/yugabyte.yml "Tested with YugabyteDB 2.6–2.13"
179191
[Vertica]: https://github.com/sqitchers/sqitch/actions/workflows/vertica.yml/badge.svg
180192
[🔺]: https://github.com/sqitchers/sqitch/actions/workflows/vertica.yml "Tested with Vertica 7.1–11.0"
181193

182194
[Sqitch]: https://sqitch.org/
183195
[PostgreSQL]: https://postgresql.org/
196+
[YugabyteDB]: https://www.yugabyte.com/yugabytedb/
197+
[lite]: https://sqlite.org/
198+
[my]: https://dev.mysql.com/
199+
[MariaDB]: https://mariadb.org
184200
[`psql`]: https://www.postgresql.org/docs/current/static/app-psql.html
185-
[Oracle]: https://www.oracle.com/database/
201+
[orcl]: https://www.oracle.com/database/
202+
[bird]: https://www.firebirdsql.org/
203+
[vert]: https://www.vertica.com/
204+
[exa]: https://www.exasol.com/
205+
[flake]: https://www.snowflake.net/
186206
[SQL\*Plus]: https://www.orafaq.com/wiki/SQL*Plus
187207
[Merkle tree]: https://en.wikipedia.org/wiki/Merkle_tree "Wikipedia: “Merkle tree”"
188208
[gitmerkle]: https://stackoverflow.com/a/18589734/

‎lib/App/Sqitch/Command/upgrade.pm‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@ use utf8;
77
use Moo;
88
use App::Sqitch::Types qw(URI Maybe Str Bool HashRef);
99
use Locale::TextDomain qw(App-Sqitch);
10-
use Type::Utils qw(enum);
1110
use App::Sqitch::X qw(hurl);
1211
use List::Util qw(first);
1312
use namespace::autoclean;

‎lib/App/Sqitch/Engine/pg.pm‎

Lines changed: 31 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ use Locale::TextDomain qw(App-Sqitch);
1111
use App::Sqitch::Plan::Change;
1212
use List::Util qw(first);
1313
use App::Sqitch::Types qw(DBH ArrayRef);
14+
use Type::Utils qw(enum);
1415
use namespace::autoclean;
1516

1617
extends 'App::Sqitch::Engine';
@@ -101,6 +102,13 @@ sub name { 'PostgreSQL' }
101102
sub driver { 'DBD::Pg 2.0' }
102103
sub default_client { 'psql' }
103104

105+
has _provider => (
106+
is => 'rw',
107+
isa => enum([qw( postgres yugabyte )]),
108+
default => 'postgres',
109+
lazy => 1,
110+
);
111+
104112
has dbh => (
105113
is => 'rw',
106114
isa => DBH,
@@ -135,6 +143,12 @@ has dbh => (
135143
# https://www.nntp.perl.org/group/perl.dbi.dev/2013/11/msg7622.html
136144
$dbh->set_err(undef, undef) if $dbh->err;
137145
};
146+
# Determine the provider. Yugabyte says this is the right way to do it.
147+
# https://yugabyte-db.slack.com/archives/CG0KQF0GG/p1653762283847589
148+
my $v = $dbh->selectcol_arrayref(
149+
q{SELECT split_part(version(), ' ', 2)}
150+
)->[0] // '';
151+
$self->_provider('yugabyte') if $v =~ /-YB-/;
138152
return;
139153
},
140154
},
@@ -167,6 +181,10 @@ sub _char2ts { $_[1]->as_string(format => 'iso') }
167181

168182
sub _listagg_format {
169183
q{ARRAY(SELECT * FROM UNNEST( array_agg(%s) ) a WHERE a IS NOT NULL)}
184+
185+
# Consider dropping 8.4 and passing the sort by column to ensure that list
186+
# is always in a well-defined order:
187+
# q{ARRAY(SELECT * FROM UNNEST( array_agg(%s ORDER BY %s) ) a WHERE a IS NOT NULL)}
170188
}
171189

172190
sub _regex_op { '~' }
@@ -259,7 +277,10 @@ sub begin_work {
259277

260278
# Start transaction and lock changes to allow only one change at a time.
261279
$dbh->begin_work;
262-
$dbh->do('LOCK TABLE changes IN EXCLUSIVE MODE');
280+
$dbh->do('LOCK TABLE changes IN EXCLUSIVE MODE')
281+
unless $self->_provider eq 'yugabyte';
282+
# Yugabyte does not yet support EXCLUSIVE MODE.
283+
# https://docs.yugabyte.com/preview/api/ysql/the-sql-language/statements/txn_lock/#lockmode-1
263284
return $self;
264285
}
265286

@@ -274,6 +295,14 @@ sub try_lock {
274295
# until timeout.
275296
sub wait_lock {
276297
my $self = shift;
298+
299+
# Yugabyte and Cockroach do not support advisory locks.
300+
# https://github.com/yugabyte/yugabyte-db/issues/3642
301+
# https://github.com/cockroachdb/cockroach/issues/13546
302+
# Use pessimistic locking when it becomes available.
303+
# https://github.com/yugabyte/yugabyte-db/issues/5680
304+
return 1 if $self->_provider ne 'postgres';
305+
277306
# Asynchronously request a lock with an indefinite wait.
278307
my $dbh = $self->dbh;
279308
$dbh->do(
@@ -478,7 +507,7 @@ App::Sqitch::Engine::pg - Sqitch PostgreSQL Engine
478507
=head1 Description
479508
480509
App::Sqitch::Engine::pg provides the PostgreSQL storage engine for Sqitch. It
481-
supports PostgreSQL 8.4.0 and higher as well as Postgres-XC 1.2 and higher.
510+
supports PostgreSQL 8.4.0 and higher, Postgres-XC 1.2 and higher, and YugabyteDB.
482511
483512
=head1 Interface
484513

‎lib/sqitch-config.pod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -437,7 +437,7 @@ The database engine to use. Supported engines include:
437437

438438
=over
439439

440-
=item * C<pg> - L<PostgreSQL|https://postgresql.org/> and L<Postgres-XC|https://sourceforge.net/>
440+
=item * C<pg> - L<PostgreSQL|https://postgresql.org/>, L<Postgres-XC|https://sourceforge.net/projects/postgres-xc/>, and L<YugabyteDB|https://www.yugabyte.com/yugabytedb/>
441441

442442
=item * C<sqlite> - L<SQLite|https://sqlite.org/>
443443

‎lib/sqitch-init.pod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ include:
3737

3838
=over
3939

40-
=item * C<pg> - L<PostgreSQL|https://postgresql.org/> and L<Postgres-XC|https://sourceforge.net/>
40+
=item * C<pg> - C<pg> - L<PostgreSQL|https://postgresql.org/>, L<Postgres-XC|https://sourceforge.net/projects/postgres-xc/>, and L<YugabyteDB|https://www.yugabyte.com/yugabytedb/>
4141

4242
=item * C<sqlite> - L<SQLite|https://sqlite.org/>
4343

‎t/lib/DBIEngineTest.pm‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -961,10 +961,10 @@ sub run {
961961
is $engine->latest_change_id(3), $change->id, 'Should get "users" offset 3 from latest';
962962

963963
$state = $engine->current_state;
964-
# MySQL's group_concat() and Oracle's collect() do not by default sort
965-
# by row order, alas.
964+
# MySQL's group_concat(), Oracle's collect(), and Yugabyte's array_agg()
965+
# do not by default sort by row order, alas.
966966
$state->{tags} = [ sort @{ $state->{tags} } ]
967-
if $class =~ /::(?:mysql|oracle)$/;
967+
if $class =~ /::(?:mysql|oracle)$/ || try { $engine->_provider eq 'yugabyte' };
968968
is_deeply $state, {
969969
project => 'engine',
970970
change_id => $barney->id,
@@ -1724,7 +1724,7 @@ sub run {
17241724

17251725
######################################################################
17261726
# Test try_lock() and wait_lock().
1727-
if (my $sql = $p{lock_sql}) {
1727+
if (my $sql = ($p{lock_sql} || sub {})->($engine)) {
17281728
ok !$engine->dbh->selectcol_arrayref($sql->{is_locked})->[0],
17291729
'Should not be locked';
17301730
ok $engine->try_lock, 'Try lock';

‎t/mysql.t‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -520,19 +520,19 @@ DBIEngineTest->run(
520520
like $sql_mode, qr/\b\Q$mode\E\b/i, "sql_mode should include $mode";
521521
}
522522
},
523-
lock_sql => {
524-
is_locked => q{SELECT is_used_lock('sqitch working')},
525-
try_lock => q{SELECT get_lock('sqitch working', 0)},
526-
wait_time => 1, # get_lock() does not support sub-second precision, apparently.
527-
async_free => 1,
528-
free_lock => 'SELECT ' . ($dbh ? do {
529-
# MySQL 5.5-5.6 and Maria 10.0-10.4 prefer release_lock(), while
530-
# 5.7+ and 10.5+ prefer release_all_locks().
531-
$dbh->selectrow_arrayref('SELECT version()')->[0] =~ /^(?:5\.[56]|10\.[0-4])/
532-
? q{release_lock('sqitch working')}
533-
: 'release_all_locks()'
534-
} : ''),
535-
},
523+
lock_sql => sub { return {
524+
is_locked => q{SELECT is_used_lock('sqitch working')},
525+
try_lock => q{SELECT get_lock('sqitch working', 0)},
526+
wait_time => 1, # get_lock() does not support sub-second precision, apparently.
527+
async_free => 1,
528+
free_lock => 'SELECT ' . ($dbh ? do {
529+
# MySQL 5.5-5.6 and Maria 10.0-10.4 prefer release_lock(), while
530+
# 5.7+ and 10.5+ prefer release_all_locks().
531+
$dbh->selectrow_arrayref('SELECT version()')->[0] =~ /^(?:5\.[56]|10\.[0-4])/
532+
? q{release_lock('sqitch working')}
533+
: 'release_all_locks()'
534+
} : ''),
535+
} },
536536
);
537537

538538
done_testing;

0 commit comments

Comments
 (0)