Improve formatting of dumped schema

[derekprior]

Over time, our dumped schema formatting has become less pretty than I would like. Way back in 2016, Caleb and I talked about adopting a SQL formatter for Scenic, but we never found a gem we liked and never took it on ourselves. Enter: niceql.

niceql has zero dependencies of its own, is implemented in a single (large) file, and has over 1.6 million downloads. It has had no releases since November 2022 but I think it's safe to consider it complete rather than abandoned. It's required Ruby version is permissive (>= 2.5), so unless there are breaking changes to Ruby basics, I don't think it will be a problem.

I've tried it on a few basic views of my own and it produces compact, readable SQL. It's maybe not how I would have formatted things by hand, but it's better than the format we're getting from Postgres. The most common formatter for Postgres is pgFormatter but as best I can tell there is no Ruby implementation for it today.

In addition to formatting the SQL, this change also adds a newline between create_view statements and any indexes associated with the view.

Before these changes:

  create_view "searches", materialized: true, sql_definition: <<-SQL
      SELECT posts.content,
      posts.user_id
     FROM posts
  UNION
   SELECT posts.title AS content,
      posts.user_id
     FROM posts
  UNION
   SELECT comments.content,
      comments.user_id
     FROM comments;
  SQL
  add_index "searches", ["content"], name: "index_searches_on_content"
  add_index "searches", ["user_id"], name: "index_searches_on_user_id"

After these changes:

  create_view "searches", materialized: true, sql_definition: <<-SQL
    SELECT posts.content, posts.user_id
    FROM posts UNION
      SELECT posts.title AS content, posts.user_id
      FROM posts UNION
        SELECT comments.content, comments.user_id
        FROM comments
  SQL

  add_index "searches", ["content"], name: "index_searches_on_content"
  add_index "searches", ["user_id"], name: "index_searches_on_user_id"

[derekprior]

I'm not sure we should take a dependency for this, but I do like the output better. What do you think @calebhearth?

I figure our release with tsort is already going to change folks' schema.rb quite a bit, so we might as well clean this up while we are at it, if we can.

[derekprior]

This is the diff generated when I tried this out on Mastodon. I think it's better but I'm not sure it's worth it. It does some odd things. pgFormatter is still what we really want.

The newline before indexes is a keeper and I think I can probably make some minor improvements futzing with current indentation.

Thoughts?

[calebhearth]

I'd be interested to see, maybe in Mastodon for a more complex/real example, what the difference is between the current and new formatting styles when diffing a new and old version of a view.

It looks like the current version has the column list for SELECT as one per line in at least one dump--that seems like it would produce a cleaner diff. I wonder if there are other places where niceql or Postgres are doing a better job producing diffable dumps.

[calebhearth]

If we're going to get this in, we probably want to do it as part of 2.0. Breaking the view dumping format isn't breaking per-se, but since I think we're targeting that anyhow I think we may as well lump it in.

First off, it looks like there's a new gem for activerecord. Not sure if we should prefer that but it's there.

ATTENTION: After ver 0.5.0 the ActiveRecord integration is provided via standalone gem: rails_sql_prettifier!

Niceql has configuration and I'd be interested to know whether this change supports that out of the box. If so, we should probably document it.