You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+19-13Lines changed: 19 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -330,24 +330,27 @@ For S3, Azure and GCS, without a cloud profile, FastBCP will try to use the defa
330
330
You can improve performance by using a method for parallelize the export (`-m` or `--method`) and a degree of parallelism (-p or --paralleldegree).
331
331
The method (`-m` or `--method`) can take several values :
332
332
-`None` (No parallelism)
333
+
-`DataDriven` : this method will use all the values of a column (or a given list provided by the `--datadrivenquery` parameter) to split the export to different files. If the number of values is greater than the dop, the dop will be used like a throttling. You can use an expression in the distributekeycolumn parameter instead a column ==> eg : YEAR(o_orderdate).
334
+
-`Ntile` : this method use the distributed column field and ntile values to retrieve evenlly distributed chunk of data. Each parallel thread will export a portion of the data based on a range build using the distributed column values. The unicity of the distributed column is **not** mandatory.
335
+
-`RangeId` : this method use the distributed column field and it's min and max value to retrieve chunk of data. Each parallel thread will export a portion of the data based on a range build using de distributed column values. Best if table is clustered by this field. The unicity of the distributed column is **not** mandatory.
333
336
-`Random` : this method is linked to a distribution column that must be an integer/bigint and that should have many values (at least as many as the dop)
334
-
-`DataDriven` : this method will use all the values of a column to split the export to different files. If the number of values is greater than the dop, the dop will be used like a throttling. You can use an expression in the distributekeycolumn parameter instead a column ==> eg : YEAR(o_orderdate).
337
+
338
+
-`Ctid` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the CTID range. PostgreSQL like databases only (and some compatibles PostgreSQL Databases).
339
+
-`Physloc` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the Physloc range. SQL Server like databases only.
335
340
-`Rowid` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the ROWID range. Oracle like databases only.
336
-
-`Ctid` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the CTID range. PostgreSQL like databases only.
337
-
-`RangeId` : this method use the distributed column field and it's min and max value to retrieve chunk of data. Each parallel thread will export a portion of the data based on a range build using de distributed column values. Best if table is clustered by this field. The unicity of the distributed column is **not** mandatory.
338
-
-`Ntile` : this method use the distributed column field and ntile values to retrieve evenlly distributed chunk of data. Each parallel thread will export a portion of the data based on a range build using the distributed column values. The unicity of the distributed column is **not** mandatory.
339
341
340
342
Table of Distribution Method :
341
343
342
-
| method | need a distributed column | database source type |
### Distribute Key Column (`-c` or `--distributekeycolumn`)
@@ -358,6 +361,9 @@ the degree of parallelism could be 0. In this case the dop we be allign with the
358
361
If the dop is greater than the number of cores (or threads if HT is on) then it will be downscale to the number of cores/threads of the machine.
359
362
If the dop is less than 0 it will be computed as the number of cores/(abs(dop)). For exemple if you have 16 cores and you set dop to -2 then the dop will computed and set to 8.
360
363
364
+
### Data Driven Query (`--datadrivenquery`)
365
+
When using the DataDriven method, you can provide a query that will return the list of values that will be used to split the data. You can thus filter the values that will be exported and used to split the data.
366
+
361
367
### Merge (`-M` or `--merge`)
362
368
You can specify if the "temporary" files generated for the parallel export should be merge to the final ouput file and deleted or if you prefer to keep the distributed files without merging them (faster export and later faster import).
363
369
**Warning** : Current version allow valid merge for CSV and Parquet only.
0 commit comments