Skip to content

Use threading and unordered stream to increase speed #6

Open
@pnadolny13

Description

@pnadolny13

This PR makes it explicit that we sort the stream records and that its safe to resume if failed. This could be much faster if we dont care about order. Potentially adding a feature that you can config to make is_sorted false and stop using an ordered queue for keeping records in order, potentially using threads and just yield them as they are received.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions