Collections regression 3.5.0 > 3.5.1

[thobson]

Describe the bug
I have two entities, Tenancy and ApiKey, a service and collection. tenancyId acts as a common pk for both the Tenancy and ApiKey. The ApiKey uses its token as the sort key. A collection, settings pulls back the tenancy and associated api keys. This was working in 3.5.0, however in 3.5.1 and 3.5.2 the collection only fetches the "child" ApiKey, not the parent Tenancy.

ElectroDB Version
3.5.1

ElectroDB Playground Link
link here

Entity/Service Definitions
Include your entity model (or a model that sufficiently recreates your issue) to help troubleshoot.

const tenancy = new Entity(
  {
    model: {
      entity: "tenancy",
      version: "1",
      service: "tenancy"
    },
    attributes: {
      tenancyId: {
        type: "string",
        required: true
      },
      name: {
        type: "string",
        required: true
      }
    },
    indexes: {
      primary: {
        pk: {
          field: "pk",
          composite: ["tenancyId"]
        },
        sk: {
          field: "sk",
          // create composite keys for partial sort key queries
          composite: []
        }
      },
      settings: {
        // collections allow for queries across multiple entities
        collection: "settings",
        index: "gsi1pk-gsi1sk-index",
        pk: {
          // map to your GSI Hash/Partition key
          field: "gsi1pk",
          composite: ["tenancyId"]
        },
        sk: {
          // map to your GSI Range/Sort key
          field: "gsi1sk",
          composite: []
        }
      },
    }
  },
  { table }
);

const apiKey = new Entity(
  {
    model: {
      entity: "apiKey",
      service: "tenancy",
      version: "1"
    },
    attributes: {
      tenancyId: {
        type: "string",
        required: true
      },
      apiKeyId: {
        type: "string",
        required: true
      },
      token: {
        type: "string",
        required: true
      }
    },
    indexes: {
      primary: {
        pk: {
          composite: ["tenancyId"],
          field: "pk"
        },
        sk: {
          composite: ["token"],
          field: "sk"
        }
      },
      settings: {
        collection: "settings",
        index: "gsi1pk-gsi1sk-index",
        pk: {
          composite: ["tenancyId"],
          field: "gsi1pk"
        },
        sk: {
          field: "gsi1sk",
          composite: []
        }
      }
    }
  },
  { table }
);

const tenancyId = "myTenancyId"
app.collections.settings({ tenancyId }).go()

Expected behavior
Should return the tenancy and api key

Errors
3.5.1 only returns the apiKey, the tenancy array is empty

Additional context
Tenancy pk & sk:
$tenancy#tenancyid_h83imegk8jb6
$tenancy_1

Api Key pk & sk:
$tenancy#tenancyid_h83imegk8jb6
$apikey_1#token_42wecu3oj-iztdtwhtn-ad3fankya

[tywalch]

Thank you for reporting this, I will prioritize this today!

[tywalch]

Hey @thobson 👋

I tried to recreate this issue, but was having trouble doing so. Here is a gist of what I ran: https://gist.github.com/tywalch/321e033a97026b4893bcff7e8117eebb. Could you take a look and see if you can reproduce your issue?

The gist basically:

1. Force creates a new table at the start (assumes you have dynamodb-local running)
2. Initializes the entities you shared with the following versions: 3.5.0, 3.5.1, 3.5.2
3. For each set of initialized entities, creates the entity with one version and queries it with the other versions.
4. Evaluates that each result matches between versions.

It's possible the items you're seeing issues with were created with an even older version (before 3.5.0).

Another vector for you to compare: If you can retrieve the raw items (as they are stored in DynamoDB), how do those items compare to one created with >3.5.0?

I will be reverting the change that likely caused this issue, but it will still be very helpful to recreate your issue.

[tywalch]

Rollback PR: #541

[thobson]

Hi @tywalch

Thanks for looking into this!

I don't think it's related to old items, as I was seeing the same issue when starting out with a clean table, i.e. creating and fetching in 3.5.0 worked, but creating and fetching in 3.5.1 resulting in the issue.

I'll take a look at the Gist and see if I can get some more info...

[thobson]

Hi @tywalch

I think I've got to the bottom of this. I examined the generated Dynamo queries and noticed something.

3.5.0 generates this collection query:

{
  KeyConditionExpression: '#pk = :pk and begins_with(#sk1, :sk1)',
  TableName: 'passlock-toby-ElectroTableTable-xadcnhkr',
  ExpressionAttributeNames: {
    '#pk': 'gsi1pk',
    '#sk1': 'gsi1sk',
    '#__edb_e__': '__edb_e__',
    '#__edb_v__': '__edb_v__'
  },
  ExpressionAttributeValues: {
    ':pk': '$tenancy#tenancyid_8rebd69yzmse',
    ':sk1': '$settings',
    ':__edb_e___apiKeys': 'apiKey',
    ':__edb_v___apiKeys': '1',
    ':__edb_e___tenancies': 'tenancy',
    ':__edb_v___tenancies': '1'
  },
  IndexName: 'gsi1pk-gsi1sk-index',
  FilterExpression: '((#__edb_e__ = :__edb_e___apiKeys AND #__edb_v__ = :__edb_v___apiKeys) OR (#__edb_e__ = :__edb_e___tenancies AND #__edb_v__ = :__edb_v___tenancies))'
}

whereas 3.5.1 generates this

{
  KeyConditionExpression: '#pk = :pk and begins_with(#sk1, :sk1)',
  TableName: 'passlock-toby-ElectroTableTable-xadcnhkr',
  ExpressionAttributeNames: {
    '#pk': 'gsi1pk',
    '#sk1': 'gsi1sk',
    '#__edb_e__': '__edb_e__',
    '#__edb_v__': '__edb_v__'
  },
  ExpressionAttributeValues: {
    ':pk': '$tenancy#tenancyid_tkimxj5fe86f',
    ':sk1': '$settings#apikeyid_', 
    ':__edb_e___apiKeys': 'apiKey',
    ':__edb_v___apiKeys': '1',
    ':__edb_e___tenancies': 'tenancy',
    ':__edb_v___tenancies': '1'
  },
  IndexName: 'gsi1pk-gsi1sk-index',
  FilterExpression: '((#__edb_e__ = :__edb_e___apiKeys AND #__edb_v__ = :__edb_v___apiKeys) OR (#__edb_e__ = :__edb_e___tenancies AND #__edb_v__ = :__edb_v___tenancies))'
}

The gsi1 sort key is different, ':sk1': '$settings' vs ':sk1': '$settings#apikeyid_'

I guess that would explain why I'm only getting the child ApiKey, not the parent Tenancy?

[tywalch]

Two things:

1. I rolled back the change made in 3.5.0 out of an abundance of caution. Try 3.5.3, let me know how that goes for you.
2. The params you shared above definitely track with the change made in 3.5.0, but I'm not seeing it with the models I used in the example. Is your model slightly different? I want to make a more targeted patch, but unless I can recreate it, I'm shooting in the dark.

[thobson]

@tywalch I managed to create a reproducible test case (see attached). By the way 3.5.3 is working for me 👌

tenancyEntity.test.ts

[tywalch]

@thobson thank you for putting that together! I see the area that needs improvement 💪. I'll keep this ticket open until I can get the change in.