Add option to control if the index record should be pre-allocated

Summary:
By default, during filter-copy operations, when the output file is created locally, vrs pre-allocates space for the index record early in the file. While this usually makes the file more streaming-friendly if it's later uploaded to the cloud, in some situations, the index can be significantly smaller or bigger than anticipated, causing some file size "surprises". This is usually not a problem (the files are fine and the "surprises" are at most 10 to 20 MB large when dealing with large indexes), but it can create confusion, in particular when trying experimenting with data size.
This new option gives control, so we can avoid this condition altogether.

Note that these conditions never happen when a file is created in the cloud, because of how the index can be prepended to the file, at the end of the file's head.

Differential Revision: D77634889

fbshipit-source-id: bec9ad38d4dcdc02ec26c08398a6895367c9e1b4

Author: georges-berenger

vrs/utils/FilterCopy.cpp

@@ -93,7 +93,7 @@ int filterCopy(
   if (copyOptions.tagOverrider) {
     copyOptions.tagOverrider->overrideTags(writer);
   }
-  if (throttledFileDelegate->shouldPreallocateIndex()) {
+  if (copyOptions.preallocateIndex && throttledFileDelegate->shouldPreallocateIndex()) {
     writer.preallocateIndex(filteredReader.buildIndex());
   }
   int copyResult = throttledFileDelegate->createFile(pathToCopy);
@@ -276,7 +276,7 @@ int filterMerge(
   }
   sort(records.begin(), records.end());
 
-  if (throttledFileDelegate->shouldPreallocateIndex()) {
+  if (copyOptions.preallocateIndex && throttledFileDelegate->shouldPreallocateIndex()) {
     // Build preliminary index
     auto preliminaryIndex = make_unique<deque<IndexRecord::DiskRecordInfo>>();
     deque<IndexRecord::DiskRecordInfo>& index = *preliminaryIndex;

vrs/utils/FilterCopyHelpers.h

@@ -72,6 +72,9 @@ struct CopyOptions {
   unique_ptr<TagOverrider> tagOverrider;
   // For merge operations only: tell if streams with the same RecordableTypeId should be merged.
   bool mergeStreams = false;
+  // Control if the index record should be pre-allocated early in the file (good for streaming).
+  bool preallocateIndex = true;
+
   // Count the number of records copied. Set during the copy/merge operation.
   mutable uint32_t outRecordCopiedCount = 0;
   // Maybe: output URI if the destination's storage system decides where to write the file

vrs/utils/cli/CliParsing.cpp

@@ -110,7 +110,8 @@ void printCopyOptionsHelp() {
           "  [ --chunk-size <nb>[M|G] ]: chunk output file every <nb> number of MiB or GiB.\n"
           "    Use 'M' for MiB (default), or 'G' for GiB.\n"
           "  [ --compression={none|default|fast|tight|zfast|zlight|zmedium|ztight|zmax} ]:"
-          " set compression setting.\n";
+          " set compression setting.\n"
+          "  [ --late-index ]: write the index at the end of the file.\n";
 }
 
 bool parseTagOverrideOptions(