Skip to content

Commit 5b126ac

Browse files
committed
update documentation with datadrivenquery and physloc that was missing
1 parent 83e41f8 commit 5b126ac

1 file changed

Lines changed: 19 additions & 13 deletions

File tree

README.md

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -330,24 +330,27 @@ For S3, Azure and GCS, without a cloud profile, FastBCP will try to use the defa
330330
You can improve performance by using a method for parallelize the export (`-m` or `--method`) and a degree of parallelism (-p or --paralleldegree).
331331
The method (`-m` or `--method`) can take several values :
332332
- `None` (No parallelism)
333+
- `DataDriven` : this method will use all the values of a column (or a given list provided by the `--datadrivenquery` parameter) to split the export to different files. If the number of values is greater than the dop, the dop will be used like a throttling. You can use an expression in the distributekeycolumn parameter instead a column ==> eg : YEAR(o_orderdate).
334+
- `Ntile` : this method use the distributed column field and ntile values to retrieve evenlly distributed chunk of data. Each parallel thread will export a portion of the data based on a range build using the distributed column values. The unicity of the distributed column is **not** mandatory.
335+
- `RangeId` : this method use the distributed column field and it's min and max value to retrieve chunk of data. Each parallel thread will export a portion of the data based on a range build using de distributed column values. Best if table is clustered by this field. The unicity of the distributed column is **not** mandatory.
333336
- `Random` : this method is linked to a distribution column that must be an integer/bigint and that should have many values (at least as many as the dop)
334-
- `DataDriven` : this method will use all the values of a column to split the export to different files. If the number of values is greater than the dop, the dop will be used like a throttling. You can use an expression in the distributekeycolumn parameter instead a column ==> eg : YEAR(o_orderdate).
337+
338+
- `Ctid` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the CTID range. PostgreSQL like databases only (and some compatibles PostgreSQL Databases).
339+
- `Physloc` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the Physloc range. SQL Server like databases only.
335340
- `Rowid` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the ROWID range. Oracle like databases only.
336-
- `Ctid` : this method use internal hidden field to retrieve chunk of data. Each parallel thread will export a portion of the data based on the CTID range. PostgreSQL like databases only.
337-
- `RangeId` : this method use the distributed column field and it's min and max value to retrieve chunk of data. Each parallel thread will export a portion of the data based on a range build using de distributed column values. Best if table is clustered by this field. The unicity of the distributed column is **not** mandatory.
338-
- `Ntile` : this method use the distributed column field and ntile values to retrieve evenlly distributed chunk of data. Each parallel thread will export a portion of the data based on a range build using the distributed column values. The unicity of the distributed column is **not** mandatory.
339341

340342
Table of Distribution Method :
341343

342-
| method | need a distributed column | database source type |
343-
|:------- | :------------------------:| :-------------------:|
344-
| None | No | Any |
345-
| Random | Yes | Any |
346-
| DataDriven | Yes | Any |
347-
| Ctid | No | PostgreSQL (pgsql/pgcopy) |
348-
| Rowid | No | Oracle (oraodp) |
349-
| RangeId | Yes | Any |
350-
| Ntile | Yes | Any |
344+
| method | Parallel | need a distributed column | database source type |
345+
|:------- | :--:| :------------------------: | :-------------------:|
346+
| None | No | No | Any |
347+
| Random | Yes | Yes | Any |
348+
| DataDriven | Yes | Yes | Any |
349+
| RangeId | Yes | Yes | Any |
350+
| Ntile | Yes | Yes | Any |
351+
| Ctid | Yes | No | PostgreSQL (pgsql/pgcopy) |
352+
| Physloc | Yes | No | Sql Server (mssql) |
353+
| Rowid | Yes | No | Oracle (oraodp) |
351354

352355

353356
### Distribute Key Column (`-c` or `--distributekeycolumn`)
@@ -358,6 +361,9 @@ the degree of parallelism could be 0. In this case the dop we be allign with the
358361
If the dop is greater than the number of cores (or threads if HT is on) then it will be downscale to the number of cores/threads of the machine.
359362
If the dop is less than 0 it will be computed as the number of cores/(abs(dop)). For exemple if you have 16 cores and you set dop to -2 then the dop will computed and set to 8.
360363

364+
### Data Driven Query (`--datadrivenquery`)
365+
When using the DataDriven method, you can provide a query that will return the list of values that will be used to split the data. You can thus filter the values that will be exported and used to split the data.
366+
361367
### Merge (`-M` or `--merge`)
362368
You can specify if the "temporary" files generated for the parallel export should be merge to the final ouput file and deleted or if you prefer to keep the distributed files without merging them (faster export and later faster import).
363369
**Warning** : Current version allow valid merge for CSV and Parquet only.

0 commit comments

Comments
 (0)