[Bug]: "Cannot convert undefined or null to object" when trying to create an SQLite Source Query

[jamietanna]

Describe the bug

I'm testing out evidence.dev to provide visualisation on an SQLite database produced by dependency-management-data, which contains a number of tables including dependency data or information about source code epositories those dependencies came from.

As part of this, one of the tables I rely on heavily fails to be imported into Evidence, with an error:

repository_metadata ✖ Error: "Cannot convert undefined or null to object"

This is triggered from the following Source Query:

select * from repository_metadata;

CREATE TABLE for the repository_metadata table

-- the below has had comments removed - see https://gitlab.com/tanna.dev/dependency-management-data/-/blob/1c8bc50d7de06be0b7cfb2665d0f8897382c68f0/internal/repositorymetadata/db/schema.sql for more details
CREATE TABLE IF NOT EXISTS repository_metadata (
  platform TEXT NOT NULL,
  organisation TEXT NOT NULL,
  repo TEXT NOT NULL,

  is_monorepo boolean not null,

  is_fork boolean not null,

  repository_type text not null,

  repository_usage text,

  visibility TEXT NOT NULL
    CHECK (
      visibility IN (
        'PUBLIC',
        'PRIVATE',
        'INTERNAL'
      )
    ),

  description TEXT,

  additional_metadata TEXT,

  UNIQUE (platform, organisation, repo) ON CONFLICT REPLACE
);

I've also found that this seems to trigger on a simpler table, too:

select * from external_licenses;

CREATE TABLE for the external_licenses table

-- the below has had comments removed - see https://gitlab.com/tanna.dev/dependency-management-data/-/blob/1c8bc50d7de06be0b7cfb2665d0f8897382c68f0/internal/externaldata/db/schema.sql for more details
CREATE TABLE IF NOT EXISTS external_licenses (
  package_name TEXT NOT NULL,
  version TEXT NOT NULL,
  package_manager TEXT NOT NULL,

  license TEXT NOT NULL,

  UNIQUE (package_name, version, package_manager, license) ON CONFLICT REPLACE,
  CHECK(package_name <> ''),
  CHECK(version <> ''),
  CHECK(package_manager <> ''),
  CHECK(license <> '')
);

Steps to Reproduce

1. Create an SQLite database with the CREATE TABLE definition for external_licenses
2. Load it as a Data Source
3. Add a Source Query as above
4. Notice Evidence fails with error

Logs

[Processing] dmd
  dmd ℹ Skipped
  metadata ℹ Skipped
  renovate ℹ Skipped
  repository_metadata ℹ Skipped
  [Skipping]: needful_things
-----
  1 source were not run due to filters
  Updating schema 'dmd'
  | Schema exists already
  | 4 queries found
  |   dmd
  |   metadata
  |   renovate
  |   repository_metadata
  | 0 queries are new
  | 2 queries already exists
  |   static/data/dmd/metadata/metadata.parquet
  |   static/data/dmd/renovate/renovate.parquet
  | 2 queries to be rendered
  |   static/data/dmd/metadata/metadata.parquet
  |   static/data/dmd/renovate/renovate.parquet
  Updating schema 'needful_things'
  | Schema exists already
  | No queries were located, but schema exists already
  |   Previous files will be kept
✔ Loading plugins & sources
-----
  [Processing] dmd
  dmd ℹ Skipped
  metadata ℹ Skipped
  renovate ℹ Skipped
  repository_metadata ✖ Error: "Cannot convert undefined or null to object"

System Info

System:
    OS: Linux 6.15 Arch Linux
    CPU: (20) x64 12th Gen Intel(R) Core(TM) i7-12800H
    Memory: 4.65 GB / 31.01 GB
    Container: Yes
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.16.0 - /usr/bin/node
    Yarn: 1.22.22 - /usr/bin/yarn
    npm: 11.4.2 - /usr/bin/npm
    pnpm: 10.12.1 - /usr/bin/pnpm
  Browsers:
    Chromium: 137.0.7151.103
  npmPackages:
    @evidence-dev/bigquery: ^2.0.10 => 2.0.10 
    @evidence-dev/core-components: ^5.2.2 => 5.2.2 
    @evidence-dev/csv: ^1.0.14 => 1.0.14 
    @evidence-dev/databricks: ^1.0.8 => 1.0.8 
    @evidence-dev/duckdb: ^1.0.13 => 1.0.13 
    @evidence-dev/evidence: ^40.1.2 => 40.1.2 
    @evidence-dev/motherduck: ^1.0.4 => 1.0.4 
    @evidence-dev/mssql: ^1.1.2 => 1.1.2 
    @evidence-dev/mysql: ^1.1.4 => 1.1.4 
    @evidence-dev/postgres: ^1.0.7 => 1.0.7 
    @evidence-dev/snowflake: ^1.2.2 => 1.2.2 
    @evidence-dev/source-javascript: ^0.0.3 => 0.0.3 
    @evidence-dev/sqlite: ^2.0.7 => 2.0.7 
    @evidence-dev/trino: ^1.0.9 => 1.0.9

Severity

blocking all usage of Evidence

Additional Information, or Workarounds

As it's not possible to use this table, this blocks a number of key pieces of metadata I can use to visualise

[jamietanna]

Per discussion on Slack, it appears to be down to the CHECK constraints on external_licenses - I'll see if I can get a PR raised to ignore them 🤔

It's unclear on repository_metadata though

[jamietanna]

Hmm, this seems to still occur with:

DROP table repository_metadata;
CREATE TABLE IF NOT EXISTS repository_metadata (
  platform TEXT NOT NULL,
  organisation TEXT NOT NULL,
  repo TEXT NOT NULL
);

Then with a Source Query repository_metadata.sql:

select * from repository_metadata;

Will return the same error 🤔

[jamietanna]

It looks like

evidence/packages/lib/db-commons/index.cjs

Line 133 in f461d6d

const firstRow = await iterator.next().then((x) => x.value);

is receiving a row with:

{ x: { value: undefined, done: true } }

[jamietanna]

OK, it looks like this is actually an issue which happens when there's 0 rows in the table