feat(db-postgres): optionally allow to store blocks as JSON and add support for "in" operator within richText

[r1tsuu]

Description

This PR adds option to store blocks as JSON column in Postgres with storeBlocksAsJSON property for the adapter.

While i like that you guys choose a hard, traditional way in implementing blocks with SQL, i think this may be not for everyone.

I like the strict approach to a schema when we talk about things like Users, Products, Orders etc.

But when it comes to a sites content it's different, not only that we can have too much things here (resulting into long table / constraints names, many joins on SELECT), it could also be changed frequently and you need to deal with a big migrations.

With Lexical blocks feature - they are stored in the same way, so i guess it's fine to allow that here too?

This PR also adds support for querying with in operator on richText and as well blocks when it's JSON.
This was done in order to pass as many tests as i can with storeBlocksAsJSON. And here's the result for "fields -> blocks"

Yes, not great here with querying on richtext, which is nested to already json blocks.

If this is something that will block this PR to merge, i have an idea on how we can improve overall our JSON queries, it could work for any nested structure. And JSON blocks, blocks in Lexical will work too.
This in theory could be achieved with this query:

CASE
WHEN jsonb_typeof(data) = 'array'  THEN

Why? we don't know in runtime, is current key object or array, because we don't know the structure when dealing with JSON field type. But with this Postgres JSON query way we can determine that.

In any way whatever separated PR or here i want to try implement this, because for now JSON / RichText querying isn't great in Payload.

p.s wrong base branch caused first commit, i reverted it.

• I have read and understand the CONTRIBUTING.md document in this repository.

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist:

• I have added tests that prove my fix is effective or that my feature works
• Existing test suite passes locally with my changes // with passing storeBlocksAsJSON: true 3 tests are failing.
• I have made corresponding changes to the documentation

[github-actions]

This PR is stale due to lack of activity.

To keep the PR open, please indicate that it is still relevant in a comment below.

[maximseshuk]

When can we expect this PR? This feature is really useful when there are around 50 blocks in one collection. Right now, I'm using a workaround that works great, but I'd love to have native support.

[rilrom]

When can we expect this PR? This feature is really useful when there are around 50 blocks in one collection. Right now, I'm using a workaround that works great, but I'd love to have native support.

Hi @maximseshuk, would you mind sharing what your workaround is? I'm running into issues with large numbers of blocks in a collection.

[maximseshuk]

When can we expect this PR? This feature is really useful when there are around 50 blocks in one collection. Right now, I'm using a workaround that works great, but I'd love to have native support.

Hi @maximseshuk, would you mind sharing what your workaround is? I'm running into issues with large numbers of blocks in a collection.

Sure. This workaround creates a dual-field system where the original field becomes virtual (exists only in the admin interface) while a hidden JSON field handles the actual database storage. The main field displays normally in the admin but doesn't get stored directly in the database.
The synchronization happens through hooks that automatically copy data between these fields during read and write operations. When saving, the virtual field data gets transferred to the JSON field for storage, and when reading, the JSON data gets copied back to the virtual field for display in the admin interface.

import type {
  Block,
  CollapsibleField,
  CollectionAfterReadHook,
  CollectionBeforeChangeHook,
  CollectionBeforeReadHook,
  Field,
  GlobalAfterReadHook,
  GlobalBeforeChangeHook,
  GlobalBeforeReadHook,
  RowField,
  TabsField,
  UIField,
} from 'payload'

import toSnakeCase from 'to-snake-case'

const convertToGeoJSON = (coordinates: any[]): any => {
  if (Array.isArray(coordinates) && coordinates.length === 2 &&
      typeof coordinates[0] === 'number' && typeof coordinates[1] === 'number') {
    return { type: 'Point', coordinates }
  }
  return coordinates
}

const findBlocksByType = (obj: any, type: string): any[] => {
  if (!obj) return []

  if (Array.isArray(obj)) {
    return obj.filter(item =>
      item && typeof item === 'object' &&
      (item.blockType === type || item.type === type))
  }

  if (obj.blockType === type || obj.type === type) {
    return [obj]
  }

  return []
}

const findPointFields = (fieldConfig: any, path: string[] = []): string[][] => {
  let paths: string[][] = []

  if (fieldConfig.type === 'point') {
    paths.push([...path, fieldConfig.name])
  }

  if (fieldConfig.fields && Array.isArray(fieldConfig.fields)) {
    fieldConfig.fields.forEach((field: any) => {
      if (field.name) {
        const nestedPaths = findPointFields(field, [...path, field.name])
        paths = [...paths, ...nestedPaths]
      }
    })
  }

  if (fieldConfig.blocks && Array.isArray(fieldConfig.blocks)) {
    fieldConfig.blocks.forEach((block: any) => {
      if (block.fields && Array.isArray(block.fields)) {
        block.fields.forEach((blockField: any) => {
          if (blockField.name) {
            const blockPaths = findPointFields(blockField, [...path, block.slug, blockField.name])
            paths = [...paths, ...blockPaths]
          }
        })
      }
    })
  }

  return paths
}

const processFieldConfig = (
  fieldConfig: Field | Block,
  mainFieldName: string,
  jsonFieldName: string,
): any => {
  if ('type' in fieldConfig && fieldConfig.type === 'point') {
    return {
      ...fieldConfig,
      hooks: {
        ...fieldConfig.hooks,
        afterRead: [
          ...(fieldConfig.hooks?.afterRead || []),
          ({
            originalDoc,
            path,
            value,
          }: {
            originalDoc: Record<string, any>
            path: (string | number)[]
            value: any
          }) => {
            if (!originalDoc) return value

            const jsonPath = [...path]
            const fieldIndex = jsonPath.findIndex(segment => segment === mainFieldName)
            if (fieldIndex !== -1) jsonPath[fieldIndex] = jsonFieldName

            let jsonValue = originalDoc
            for (const key of jsonPath) {
              if (!jsonValue) return value
              jsonValue = jsonValue[key]
            }

            return jsonValue !== undefined ? jsonValue : value
          },
        ],
      },
    }
  }

  if ('fields' in fieldConfig) {
    return {
      ...fieldConfig,
      fields: fieldConfig.fields.map(field => processFieldConfig(field, mainFieldName, jsonFieldName)),
    }
  }

  if ('blocks' in fieldConfig) {
    return {
      ...fieldConfig,
      blocks: fieldConfig.blocks.map(block => processFieldConfig(block, mainFieldName, jsonFieldName)),
    }
  }

  return fieldConfig
}

const transformPointsToGeoJSON = (data: any, pointPaths: string[][]): any => {
  if (!data) return data

  const result = JSON.parse(JSON.stringify(data))

  const simplifiedPaths = pointPaths.map(path => {
    const simplified = []
    for (let i = 0; i < path.length; i++) {
      if (i === 0 || path[i] !== path[i - 1]) {
        simplified.push(path[i])
      }
    }
    return simplified
  })

  simplifiedPaths.forEach(pathTemplate => {
    if (pathTemplate.length < 2) return

    const blockType = pathTemplate[0]
    let blocks = findBlocksByType(result, blockType)

    if (blocks.length === 0 && Array.isArray(result)) {
      result.forEach(item => {
        blocks = blocks.concat(findBlocksByType(item, blockType))
      })
    }

    blocks.forEach(block => {
      let currentObj = block
      let pathIndex = 1

      while (pathIndex < pathTemplate.length - 1 && currentObj) {
        const currentKey = pathTemplate[pathIndex]

        if (!currentObj[currentKey]) break

        if (Array.isArray(currentObj[currentKey])) {
          currentObj[currentKey].forEach((item: any) => {
            let subCurrent = item
            let subIndex = pathIndex + 1

            while (subIndex < pathTemplate.length - 1 && subCurrent) {
              const subKey = pathTemplate[subIndex]
              if (!subCurrent[subKey]) break
              subCurrent = subCurrent[subKey]
              subIndex++
            }

            const coordField = pathTemplate[pathTemplate.length - 1]
            if (subCurrent && Array.isArray(subCurrent[coordField])) {
              subCurrent[coordField] = convertToGeoJSON(subCurrent[coordField])
            }
          })

          break
        } else {
          currentObj = currentObj[currentKey]
        }

        pathIndex++
      }

      const coordField = pathTemplate[pathTemplate.length - 1]
      if (currentObj && Array.isArray(currentObj[coordField])) {
        currentObj[coordField] = convertToGeoJSON(currentObj[coordField])
      }
    })
  })

  return result
}

export const vjsonField = <T extends 'global' | 'collection' = 'collection'>(
  field: Exclude<Field, CollapsibleField | RowField | TabsField | UIField>,
  jsonFieldName?: string,
): {
  fields: Field[]
  hooks: {
    afterRead: T extends 'global' ? GlobalAfterReadHook : CollectionAfterReadHook
    beforeChange: T extends 'global' ? GlobalBeforeChangeHook : CollectionBeforeChangeHook
    beforeRead: T extends 'global' ? GlobalBeforeReadHook : CollectionBeforeReadHook
  }
} => {
  if (!('name' in field)) {
    throw new Error('Field must have a name')
  }

  jsonFieldName = jsonFieldName || `_${toSnakeCase(field.name)}`

  const pointPaths = findPointFields(field)

  const fields: Field[] = [
    {
      ...processFieldConfig(field, field.name, jsonFieldName),
      required: 'required' in field ? field.required : false,
      virtual: true,
    },
    {
      name: jsonFieldName,
      type: 'json',
      access: field.access,
      admin: { hidden: true, readOnly: true },
      hidden: true,
      localized: field.localized,
    },
  ]

  return {
    fields,
    hooks: {
      afterRead: async ({
        doc,
        req,
      }: Parameters<CollectionAfterReadHook>[0] | Parameters<GlobalAfterReadHook>[0]) => {
        if (req.context[field.name]) doc[field.name] = req.context[field.name]
        delete doc[jsonFieldName]
        return doc
      },
      beforeChange: async ({
        data,
        req,
      }: Parameters<CollectionBeforeChangeHook>[0] | Parameters<GlobalBeforeChangeHook>[0]) => {
        if (
          data[field.name] && (
            Array.isArray(data[field.name]) && data[field.name].length > 0 ||
            (typeof req.data === 'object' && field.name in req.data)
          )
        ) {
          const jsonData = data[field.name]
          const transformedJsonData = transformPointsToGeoJSON(jsonData, pointPaths)
          data[jsonFieldName] = transformedJsonData
          req.context[field.name] = jsonData
        }
        return data
      },
      beforeRead: async ({
        doc,
      }: Parameters<CollectionBeforeReadHook>[0] | Parameters<GlobalBeforeReadHook>[0]) => {
        if (doc[jsonFieldName]) {
          doc[field.name] = doc[jsonFieldName]
        }
        return doc
      },
    },
  }
}

Usage example:

export const layoutField = vjsonField({
  name: 'layout',
  type: 'blocks',
  access: access.collection.field('collections', 'layout'),
  blocks: [
    ...Object.values(blocks),
  ],
  label: false,
  localized: true,
  required: true,
})

[rilrom]

@maximseshuk this is impressive! Thanks so much for sharing. I'll try this out on my end and see how it goes.

Edit: unfortunately after trying a number of different things I could only get it working in certain scenarios. Hooks that called payload.update or using the 'Duplicate' functionality in the admin panel caused the data to go missing. Nonetheless thank you for sharing!

Hopefully this PR can get revisited soon, it solves a very real problem.

[DanRibbens]

I'm still hesitant to introduce a separate mode for managing schema in our DB adapters because of the complexity of maintaining multiple. Imagine we have to do a breaking change in the future and need to write a migration script for existing projects for example. This could be problematic if we have not only all the different DB adapters to worry about but a multitude of schema shapes to consider.

[r1tsuu]

@maximseshuk I opened a new PR #12750 and we release a canary version with this change 3.43.0-internal.c5bbc84 if you also want to try it out.

[r1tsuu]

Closing this old one.