Skip to content

Long make calls lock table metadata #1170

@ethho

Description

@ethho

Bug Report

Description

A client locks table metadata for the entire duration of a make function call. When other clients attempt to drop or declare child tables, the call is blocked until the first client finishes make. This approach scales poorly with number of clients and number of child tables.

Reproducibility

Include:

Proposed Solution

As an alternative to writing a Computed.make function, allow user to write three functions:

  1. make_fetch for reading inputs
  2. make_compute, which is not run in a transaction, and is passed the return value of make_fetch
  3. make_insert, which inserts computed values using the same transaction semantics as make.

In pseudocode, these three functions will be used in the following routine as such:

if hasattr(table, "make"):
    return make()
else:
    assert hasattr(table, "make_fetch")
    assert hasattr(table, "make_compute")
    assert hasattr(table, "make_insert")
    input = make_fetch()
    conn.disconnect() # I assume this disconnect step is to ensure that make_compute cannot insert?
    result = make_compute(input)
    tx = conn.start_transaction()
    input2 = make_fetch()
    if hash(serialize(input2)) == hash(serialize(input)):
        result = make_insert(result)
        tx.commit()
        return result
    else:
        print("ERROR: inputs have changed")
        tx.abort()
        return None

Additional Research and Context

Related Issues


cc: @dimitri-yatsenko @ttngu207 @CBroz1 @samuelbray32 @peabody124

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions