-
Notifications
You must be signed in to change notification settings - Fork 27
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Currently we have code like
autoCreateTables = config.getBoolean(BigQuerySinkConfig.TABLE_CREATE_CONFIG);
upsertDelete = config.getBoolean(BigQuerySinkConfig.UPSERT_ENABLED_CONFIG)
|| config.getBoolean(BigQuerySinkConfig.DELETE_ENABLED_CONFIG);
useCredentialsProjectId = config.getBoolean(BigQuerySinkConfig.USE_CREDENTIALS_PROJECT_ID_CONFIG);
useStorageApi = config.getBoolean(BigQuerySinkConfig.USE_STORAGE_WRITE_API_CONFIG);
useStorageApiBatchMode = useStorageApi && config.getBoolean(BigQuerySinkConfig.ENABLE_BATCH_MODE_CONFIG);
retry = config.getInt(BigQuerySinkConfig.BIGQUERY_RETRY_CONFIG);
retryWait = config.getLong(BigQuerySinkConfig.BIGQUERY_RETRY_WAIT_CONFIG);
allowNewBigQueryFields = config.getBoolean(BigQuerySinkConfig.ALLOW_NEW_BIGQUERY_FIELDS_CONFIG);
allowRequiredFieldRelaxation = config.getBoolean(BigQuerySinkConfig.ALLOW_BIGQUERY_REQUIRED_FIELD_RELAXATION_CONFIG);
Where the configuration logic is scattered across the system in various places.
These should be implemented within the configuration class.
autoCreateTables = config.autoCreateTables();
upsertDelete = config.upsertDelete();
useCredentialsProjectId = config.useCredentialsFromProjectId();
useStorageApi = config.useStorageApi();
useStorageApiBatchMode = config.useStorageApiBatchMode();
RetryConfig retryCfg = config.getRetryConfig();
allowNewBigQueryFields = config.allowNewBigQueryFields();
allowRequiredFieldRelaxation = config.allowRequiredFieldRelaxation();
Where RetryConfig is
Interface RetryConfig {
int getCount();
long getDelay()
}
That will encapsulate the conversions and logic gymnastics into the config class keep the code cleaner.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request