Skip to content

README.md

Latest commit

 

History

History
34 lines (26 loc) · 1.48 KB

README.md

File metadata and controls

34 lines (26 loc) · 1.48 KB
Raw

Teradiff

A tool which compares 2 large scale datasets based on Spark

Runner script

  • Locate "run.sh" file under directory "src/scripts/"
  • Deploy "run.sh" file to the same level of teradiff jar file
  • Create properties file and put it in the same directory as teradiff jar file
    • Example: src/main/resources/teradiff.properties
  • Tune parameters of "--num-executors", "--executor-memory", etc based on cluster resources
  • Execute ./run.sh

Arguments explained

  • leftSchema: Comma seperated list of columns of the left source
  • rightSchema: Comma seperated list of columns of the right source
  • leftDelimiter: Only applies to csv, default is comma
  • rightDelimiter: Only applies to csv, default is comma
  • leftWithHeader: Only applies to csv, whether schema header is included in the file
  • rightWithHeader: Only applies to csv, whether schema header is included in the file
  • leftKey: Comma seperated list of columns to be the unique ID of the left source
  • rightKey: Comma seperated list of columns to be the unique ID of the right source
  • leftValue: Comma seperated list of columns to be compared for the left source
  • rightValue: Comma seperated list of columns to be compared for the right source
  • leftIgnores: Comma seperated list of columns to be ignored for left
  • rightIgnores: Comma seperated list of columns to be ignored for the right
  • List of columns between left and right must follow the same order

Example delimiter options:

Pipe: --leftDelimiter '|' Ctrl-A: --leftDelimiter $'\001'