I'm using FST + data.tables to manage and analyze large panel data generated by simulation experiments. My tasks involve some cleaning and auxiliary variable creation from the original data.
Due to memory constraints, I read in only a subset of columns for some tables. The workflow is so seamless and fast it did not occur to me that write_fst() apparently overwrites the original FST with the subset.
Is there a way to save changes to modified columns (and bind new ones) to an existing FST without loading the entire dataset, or is it necessary to create new FSTs for any changes and join results manually?
Thanks!