Document scope of library

[hknutsen]

This library is currently in an early development phase, as highlighted in the README:

This allows us to experiment with the library, adding and removing breaking changes as we like without actually having to mark them as breaking changes just yet.

Before the first full v1.0.0 release and the removal of this disclaimer in the README, we should document a clear scope and structure for this library. The pros and cons of the different scopes and structures should be discussed before a final decision is made.

Example 1: Single package

This repository contains a single package (dataorc) with smaller modules that contain classes and functions for a specific scope, for example:

• A config module that contains generic pipeline configuration classes and functions
• A databricks module that contains Databricks specific classes and functions
• An azure module that contains Azure specific classes and functions

Installation:

pip install dataorc

Imports:

from dataorc.config import CorePipelineConfig
from dataorc.azure import Logger

Example 2: Multiple packages

This repository contains multiple packages that contain modules for a specific scope, for example:

• A dataorc-utils package that contains generic utility modules, for example the config module from example 1.
• A dataorc-logging package that contains logging modules, for example an azure module that contains classes and functions for logging to Azure Log Analytics.

Each package will be a separate entry in pypi, adding some administrative overhead.

Installation:

pip install dataorc-utils dataorc-logging

Imports:

from dataorc_utils.config import CorePipelineConfig
from dataorc_logging.azure import Logger

[onoedland]

Unless there are strong arguments for why we need option 2, personally I prefer the simplicity of option 1 (only one package name to remember).