diff --git a/.github/workflows/lint-readme.yaml b/.github/workflows/lint-readme.yaml index 0849261..a585ec9 100644 --- a/.github/workflows/lint-readme.yaml +++ b/.github/workflows/lint-readme.yaml @@ -36,7 +36,10 @@ jobs: uses: actions/checkout@v4 - name: Install dependencies - run: pip install setuptools docutils pygments + run: pip install build twine - - name: Lint Readme - run: python setup.py check --restructuredtext --strict + - name: Build package + run: python -m build + + - name: Run twine check + run: twine check dist/* diff --git a/README.md b/README.md index 8e093ad..a746e13 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,26 @@ in your code using the builder API: spark = DataprocSparkSession.builder.projectId('my-project').location('us-central1').dataprocSessionConfig(session_config).getOrCreate() ``` +### Builder Configuration + +The `DataprocSparkSession.builder` provides a fluent API to configure the session. Below is a list of available methods: + +| Method | Description | +|--------|-------------| +| `config(key, value)` | Sets a Spark configuration property. | +| `dataprocSessionConfig(dataproc_config)` | Sets the [Dataproc Session](https://docs.cloud.google.com/python/docs/reference/dataproc/latest/google.cloud.dataproc_v1.types.Session) configuration object. | +| `dataprocSessionId(session_id)` | Sets a custom session ID for creating or reusing sessions. | +| `idleTtl(duration)` | Sets the idle time-to-live (idle TTL) for the session using a `datetime.timedelta` object. | +| `label(key, value)` | Adds a single label to the session. | +| `labels(labels)` | Adds multiple labels to the session. | +| `location(location)` | Sets the Google Cloud region. | +| `projectId(project_id)` | Sets the Google Cloud project ID. | +| `runtimeVersion(version)` | Sets the Dataproc runtime version (e.g., "3.0"). | +| `serviceAccount(account)` | Sets the service account for the session. | +| `sessionTemplate(template)` | Sets the session template to use. | +| `subnetwork(subnet)` | Sets the subnetwork URI for the session. | +| `ttl(duration)` | Sets the time-to-live (TTL) for the session using a `datetime.timedelta` object. | + ### Reusing Named Sessions Across Notebooks Named sessions allow you to share a single Spark session across multiple notebooks, improving efficiency by avoiding repeated session startup times and reducing costs. diff --git a/setup.py b/setup.py index 3aa2426..539e50e 100644 --- a/setup.py +++ b/setup.py @@ -23,6 +23,7 @@ version="1.0.2", description="Dataproc client library for Spark Connect", long_description=long_description, + long_description_content_type="text/markdown", author="Google LLC", url="https://github.com/GoogleCloudDataproc/dataproc-spark-connect-python", license="Apache 2.0",