Describe the bug:
When calling dynamic_bucket on a data pipeline, I am getting an "incompatible function arguments" error.
TypeError: dynamic_bucket(): incompatible function arguments. The following argument types are supported:
1. (self: fairseq2n.bindings.data.data_pipeline.DataPipelineBuilder, threshold: float, fn: Callable[[Any], float], bucket_creation_fn: Optional[Callable[[List[Any]], Tuple[List[Any], List[Any]]]] = None, min_num_examples: Optional[int] = None, max_num_examples: Optional[int] = None, drop_remainder: bool = False) -> fairseq2n.bindings.data.data_pipeline.DataPipelineBuilder
Invoked with: <fairseq2n.bindings.data.data_pipeline.DataPipelineBuilder object at 0x7f76c41daaf0>; kwargs: threshold=5.0, cost_fn=<function my_cost_fn at 0x7f76a7f703a0>
Describe how to reproduce:
from typing import Any
from fairseq2.data import read_sequence
def my_cost_fn(item: Any) -> float:
return float(item)
items = list(range(10))
builder = read_sequence(items)
builder.dynamic_bucket(threshold=5.0, cost_fn=my_cost_fn)
pipeline = builder.and_return()
bucketed = list(pipeline)
print(bucketed)
assert bucketed == [[0, 1, 2, 3], [4, 5], [6], [7], [8], [9]]Describe the expected behavior:
I expected no error to happen and assertions to pass.
Environment:
I am using fairseq2==0.4.4, but the error seems to persist in the main branch.
Additional Context:
What is confusing is that I am using exactly the signature described in the Python interface (
|
cost_fn: Callable[[Any], float], |
) and in the documentation (
https://facebookresearch.github.io/fairseq2/nightly/basics/data_pipeline.html#bucketing). But the underlying C++ code (
) expects the argument to be called just
fn.
Please reconcile these interfaces. Having to consult the native code for writing simple stuff is annoying.
Describe the bug:
When calling
dynamic_bucketon a data pipeline, I am getting an "incompatible function arguments" error.Describe how to reproduce:
Describe the expected behavior:
I expected no error to happen and assertions to pass.
Environment:
I am using
fairseq2==0.4.4, but the error seems to persist in the main branch.Additional Context:
What is confusing is that I am using exactly the signature described in the Python interface (
fairseq2/src/fairseq2/data/_data_pipeline.py
Line 238 in 19aeff1
fairseq2/native/src/fairseq2n/data/dynamic_bucket_data_source.h
Line 24 in 19aeff1
fn.Please reconcile these interfaces. Having to consult the native code for writing simple stuff is annoying.