Skip to content

How to save modified dataframe with write_fst when only a subset of columns are read? #283

@jlambvo

Description

@jlambvo

I'm using FST + data.tables to manage and analyze large panel data generated by simulation experiments. My tasks involve some cleaning and auxiliary variable creation from the original data.

Due to memory constraints, I read in only a subset of columns for some tables. The workflow is so seamless and fast it did not occur to me that write_fst() apparently overwrites the original FST with the subset.

Is there a way to save changes to modified columns (and bind new ones) to an existing FST without loading the entire dataset, or is it necessary to create new FSTs for any changes and join results manually?

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions