Skip to content

docs: Add Scala tutorial for LightGBM Quantile Regression in Drug Discovery (#731) - #2701

Open
Venkata Surya Mahendra Mula (MahendraMula) wants to merge 16 commits into
microsoft:masterfrom
MahendraMula:doc/731-scala-quantile-tutorial
Open

docs: Add Scala tutorial for LightGBM Quantile Regression in Drug Discovery (#731)#2701
Venkata Surya Mahendra Mula (MahendraMula) wants to merge 16 commits into
microsoft:masterfrom
MahendraMula:doc/731-scala-quantile-tutorial

Conversation

@MahendraMula

@MahendraMula Venkata Surya Mahendra Mula (MahendraMula) commented Sep 7, 2026

Copy link
Copy Markdown

Description

This pull request introduces a comprehensive Scala tutorial for performing Quantile Regression using LightGBM in drug discovery workflows, addressing community request #731.

Related Issue

Closes #731

Changes Proposed

  • New Tutorial Guide: Added step-by-step Scala documentation demonstrating:
    • Spark session initialization and synthetic molecular descriptor dataset creation.
    • Feature assembly using Spark's VectorAssembler.
    • Training multi-quantile models (20th, 50th median, 80th percentiles) using LightGBMRegressor with .setObjective("quantile") and .setAlpha(...).
    • Sequential transformation for uncertainty envelope prediction.
    • Model evaluation with RegressionEvaluator (RMSE).
  • PySpark vs. Spark Scala Comparison: Included an explanatory callout table highlighting key syntax differences (fluent setters vs. keyword arguments, typing, arrays).

Testing & Verification

  • Verified against SynapseML's internal Scala test suites (LightGBMSuite.scala) and API specifications.
  • Ensured dataset is self-contained with zero missing external dependencies.

Checklist

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

Hey Venkata Surya Mahendra Mula (@MahendraMula) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

  • fix: Fix LightGBM crashes with empty partitions
  • feat: Make HTTP on Spark back-offs configurable
  • docs: Update Spark Serving usage
  • build: Add codecov support
  • perf: improve LightGBM memory usage
  • refactor: make python code generation rely on classes
  • style: Remove nulls from CNTKModel
  • test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new tutorial markdown currently has escaped Markdown/broken code formatting, an outdated Maven coordinate, and stops after imports despite the PR description claiming a complete end-to-end walkthrough.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds a Scala-focused tutorial document under notebooks/features/lightgbm/ intended to demonstrate LightGBM quantile regression workflows for drug discovery (QSAR) use cases, complementing the existing PySpark example requested in #731.

Changes:

  • Adds a new Scala tutorial markdown file for LightGBM quantile regression in drug discovery.
  • Introduces a PySpark vs Scala syntax comparison table and initial environment/import setup section.
File summaries
File Description
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md New Scala tutorial document for quantile regression (currently contains Markdown rendering issues and appears incomplete vs. the PR description).
Review details

Suppressed comments (3)

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:37

  • The PySpark vs Scala table is separated by blank lines, which breaks Markdown table rendering in most parsers. Also, several code spans are unnecessarily escaped (e.g., \[ ... \], \_), so they’ll display backslashes literally.
| Feature | Python (PySpark) | Scala (Spark) | Explanation |

| :--- | :--- | :--- | :--- |

| \*\*Parameter Configuration\*\* | `LightGBMRegressor(alpha=0.5, objective="quantile")` | `new LightGBMRegressor().setAlpha(0.5).setObjective("quantile")` | Scala uses the \*\*fluent setter pattern\*\* (`.setParam()`) instead of constructor keyword arguments. |

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:59

  • The Maven coordinate is pinned to 0.11.4 and also escapes the underscore inside an inline code span (synapseml\_2.12), which will render with a literal backslash. This should match the repo’s current installation guidance (e.g., README shows com.microsoft.azure:synapseml_2.12:1.1.3 for master).
\* \*\*Maven Coordinate:\*\* `com.microsoft.azure:synapseml\_2.12:0.11.4`

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:79

  • The Scala code fence is not closed, and functions.\_ will render a literal backslash and isn’t valid Scala. Either close the fenced block and use functions._, or switch to an indented code block so the snippet is copy/paste runnable.
```scala

import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.functions.\_

import org.apache.spark.ml.feature.VectorAssembler

import org.apache.spark.ml.evaluation.RegressionEvaluator

import com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor


  • Files reviewed: 1/1 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@MahendraMula

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new tutorial contains several copy/paste-breaking documentation issues (notably incorrect sbt dependency notation and inconsistent Maven repository guidance) that should be corrected before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:42

  • This spark-shell example uses the azureedge Maven repository, which doesn’t match the repository URL recommended in README.md (blob.core.windows.net). Update it for consistency and to avoid resolution issues.
spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3 \
            --repositories https://mmlspark.azureedge.net/maven

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:145

  • Step 4 always applies a VectorAssembler over descriptor columns and transforms qsarDf. If the user chose Option B (LibSVM), qsarDf already contains a features vector and won’t have the listed descriptor columns, so this transform path will break. Consider skipping assembly when features already exists.
val assembler = new VectorAssembler()
  .setInputCols(featureCols)
  .setOutputCol("features")

val assembledDf = assembler.transform(qsarDf)

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:268

  • This resolver points to https://mmlspark.azureedge.net/maven, but the repo’s current installation guidance recommends https://mmlspark.blob.core.windows.net/maven (README.md:91-93). Keeping these consistent reduces copy/paste failures.
resolvers += "SynapseML Maven Repo" at "https://mmlspark.azureedge.net/maven"

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:370

  • The spark-submit command uses the azureedge Maven repository, which doesn’t match the repository URL recommended in README.md (blob.core.windows.net). This can cause dependency resolution failures depending on the runtime.
  --packages com.microsoft.azure:synapseml_2.12:1.1.3 \
  --repositories https://mmlspark.azureedge.net/maven \
  target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar
  • Files reviewed: 1/1 changed files
  • Comments generated: 4
  • Review effort level: Lite

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new tutorial contains several concrete configuration inaccuracies (repository URL and Spark/Scala version pins) that are likely to cause dependency resolution or runtime compatibility problems for readers.

Review details

Suppressed comments (5)

Previously missed (1) — in code that hasn't changed since the last review.

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:62

  • Hard-coding .master("local[*]") in the SparkSession builder will force local mode and is likely to break or behave unexpectedly when readers run this on Databricks/Synapse/cluster deployments. It’s safer to let spark-submit / the notebook environment set the master and just call getOrCreate().

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:36

* **Maven Coordinate:** `com.microsoft.azure:synapseml_2.12:1.1.3`
* **Spark Packages:** `com.microsoft.azure:synapseml_2.12:1.1.3`
* **Repository:** `https://mmlspark.azureedge.net/maven`

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:42

spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3 \
            --repositories https://mmlspark.azureedge.net/maven

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:270

  • The standalone build.sbt snippet pins Spark 3.4.1 and Scala 2.12.18, which doesn’t match this repo’s current master line (Spark 3.5.x / Scala 2.12.17). Using Spark 3.4.x with the 1.1.3 artifacts built for Spark 3.5 can lead to binary incompatibilities at runtime (e.g., NoSuchMethodError).
scalaVersion := "2.12.18"

resolvers += "SynapseML Maven Repo" at "https://mmlspark.azureedge.net/maven"

val sparkVersion = "3.4.1"

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:370

  --packages com.microsoft.azure:synapseml_2.12:1.1.3 \
  --repositories https://mmlspark.azureedge.net/maven \
  target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new documentation includes a couple of concrete copy/paste hazards/inconsistencies (hard-coded local master and Scala version mismatch with the repo) that should be corrected to avoid misleading users.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:62

  • The SparkSession snippet hard-codes .master("local[*]"), which can accidentally force local mode when users copy/paste into a real Spark cluster or managed notebook environment (Databricks/Synapse) and may ignore the configured cluster master. For docs, it’s safer to omit .master(...) and rely on the deployment configuration.
    notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:56
  • The import list includes LightGBMRegressionModel, but the tutorial code never references it. Removing unused imports makes the snippet cleaner for copy/paste and avoids unused-import warnings in IDEs.
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@MahendraMula

Venkata Surya Mahendra Mula (MahendraMula) commented Sep 7, 2026

Copy link
Copy Markdown
Author

All Copilot feedback has been addressed across the latest commits. Kindly requesting a maintainer review when you get a chance. Thank you!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The spark-submit section’s wording is inconsistent with the provided --master yarn commands and should be corrected to avoid misleading readers about the intended cluster manager/runtime.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:395

  • This section labels the second command as "standalone Spark cluster" but still uses --master yarn, which is a YARN cluster manager (not Spark Standalone). Either adjust the wording to YARN, or switch the --master value to a Spark Standalone URL.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

# Package the application
sbt package

# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The crossed-quantile diagnostic section states crossed rows must be explicitly flagged, but the snippet currently only prints a count while still computing interval metrics for crossed rows, which can produce misleading negative widths and coverage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment on lines +227 to +246
// Calculate prediction interval width (uncertainty) and interval coverage
val predictionsWithInterval = predictions.withColumn(
"uncertainty_width",
$"pred_q80" - $"pred_q20"
).withColumn(
"within_interval",
$"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80"
)

// ── Crossed-Quantile Diagnostic ───────────────────────────────────────────────
// Each quantile model (q20, q50, q80) is trained *independently*, so there is
// no mathematical guarantee that q20 ≤ q50 ≤ q80 holds for every compound.
// When the ordering is violated the resulting "interval" has a negative width or
// a reversed median, making it meaningless as an uncertainty estimate. These
// rows must be flagged explicitly rather than absorbed silently by taking absolute
// values or sorting the bounds — doing so would hide a real model quality signal.
val crossingCount = predictions.filter(
$"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()
println(s"Rows with crossed quantiles: $crossingCount")

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The tutorial contains runnable Scala snippets with a crossed-quantile count recomputation/variable reuse issue and spark-submit examples that inaccurately describe --master usage for different cluster modes.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:429

  • The spark-submit examples label the target as “Databricks / Azure Synapse” and “standalone Spark cluster”, but both commands use --master yarn. yarn is appropriate only for YARN deployments; for Spark Standalone mode the master URL is spark://..., and managed platforms typically don’t use spark-submit this way. Adjust the wording and the second command’s --master so the example is accurate.
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
  --class com.example.drugdiscovery.QSARQuantileApp \
  --master yarn \
  --deploy-mode client \
  --packages com.microsoft.azure:synapseml_2.12:1.1.3 \
  --repositories https://mmlspark.blob.core.windows.net/maven \
  target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar

# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
spark-submit \
  --class com.example.drugdiscovery.QSARQuantileApp \
  --master yarn \
  --deploy-mode client \
  --packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 \
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment on lines +289 to +305
val crossingCount = predictions.filter(
$"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()

val totalCount = predictionsWithInterval.count()
val coverageCount = predictionsWithInterval.filter($"within_interval" === true).count()
val empiricalCoverage = (coverageCount.toDouble / totalCount.toDouble) * 100.0

// Coverage restricted to rows where quantile ordering is correct
val validRows = predictionsWithInterval.filter($"uncertainty_width" >= 0)
val validTotal = validRows.count()
val validCoverageCount = validRows.filter($"within_interval" === true).count()
val validCoverage = if (validTotal > 0) (validCoverageCount.toDouble / validTotal.toDouble) * 100.0 else 0.0

println(f"Rows with crossed quantiles : $crossingCount (out of $totalCount)")
println(f"Empirical Coverage (all rows) : $empiricalCoverage%.2f%% — includes $crossingCount crossed row(s); interpret with caution")
println(f"Empirical Coverage (valid rows only): $validCoverage%.2f%% (Nominal target: 60.00%%)")
…n for Drug Discovery (Scala).md

Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The tutorial has a copy/paste-breaking Scala redeclaration (crossingCount) and a couple of misleading spark-submit labeling/details that should be corrected for accuracy.

Review details

Suppressed comments (4)

Previously missed (1) — in code that hasn't changed since the last review.

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:240

  • predictionsWithInterval is used in Step 7 for multiple actions (count(), filtered count()s). Without caching, Spark may recompute the full transformation lineage (including the three model transforms) several times. Caching the DataFrame once makes the evaluation section much faster on non-trivial datasets.

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:295

  • In Step 7 the code redeclares val crossingCount even though it was already defined in Step 6. If readers run the tutorial top-to-bottom in a single Spark shell / worksheet, Scala will fail with “crossingCount is already defined”. Rename the Step 7 variable (or reuse the earlier one) to keep the tutorial copy/paste friendly.
val crossingCount = predictions.filter(
  $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:423

  • This spark-submit example is labeled as “Databricks / Azure Synapse”, but it hard-codes --master yarn. Since --master yarn specifically targets YARN, the label is misleading for Databricks (and for Spark Standalone). Consider renaming the comment to match the command (or adjusting the --master value).
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
  --class com.example.drugdiscovery.QSARQuantileApp \
  --master yarn \
  --deploy-mode client \

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:435

  • This second spark-submit command is described as “standalone Spark cluster”, but it also uses --master yarn (YARN). If the intent is Spark’s standalone cluster manager, --master spark://... would be the right form; otherwise, rename the comment to “YARN cluster” to match the flags.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
spark-submit \
  --class com.example.drugdiscovery.QSARQuantileApp \
  --master yarn \
  --deploy-mode client \
  --packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 \
  --repositories https://mmlspark.blob.core.windows.net/maven \
  target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The added tutorial contains a Scala val redefinition that will fail when executed sequentially and has misleading spark-submit environment wording that should be corrected for accuracy.

Review details

Suppressed comments (4)

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:295

  • The tutorial defines crossingCount in Step 6 and then defines crossingCount again in Step 7. In a typical Scala REPL / notebook session, re-defining the same val will fail and prevents readers from running the tutorial top-to-bottom.
val crossingCount = predictions.filter(
  $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:240

  • Step 7 triggers multiple Spark actions (count() several times) on predictionsWithInterval, which can recompute the same lineage repeatedly and slow down the tutorial noticeably. Consider caching predictionsWithInterval once after it’s constructed (and optionally unpersisting later).
val predictionsWithInterval = predictions
  .withColumn("is_crossed", $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80")
  .withColumn("uncertainty_width", $"pred_q80" - $"pred_q20")
  .withColumn("within_interval", $"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80")

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:423

  • The spark-submit example labels the first command as “Databricks / Azure Synapse” but uses --master yarn, which is specific to Spark-on-YARN and is not representative of Databricks/Synapse submission. This can confuse readers; it’s clearer to describe this as a YARN example and note that --master/--deploy-mode should be adjusted for the user’s environment.
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
  --class com.example.drugdiscovery.QSARQuantileApp \
  --master yarn \
  --deploy-mode client \

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:432

  • The second spark-submit example says “standalone Spark cluster” but still uses --master yarn. “Standalone” is a specific Spark master type (spark://...) and differs from YARN, so the wording here is misleading.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
spark-submit \
  --class com.example.drugdiscovery.QSARQuantileApp \
  --master yarn \
  --deploy-mode client \
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

MahendraMula

This comment was marked as low quality.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The tutorial currently contains a Scala REPL-breaking val redefinition and has a confusing spark-submit “standalone” label while using --master yarn, which should be corrected for copy/paste correctness and clarity.

Review details

Suppressed comments (2)

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:428

  • This comment says “standalone Spark cluster”, but the command still uses --master yarn (YARN). Either clarify the text (Spark on YARN) or change the master example to a Spark standalone master URL to avoid confusing readers.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:234

  • In spark-shell / Scala REPL, val names cannot be redefined later in the same session. Step 7 defines val crossingCount again (line 293), which will fail if the reader ran Step 6 first. Rename the Step 6 variable to avoid a REPL compilation error while keeping Step 7 self-contained.
val crossingCount = predictions.filter(
  $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()
println(s"Rows with crossed quantiles: $crossingCount")
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new doc has a correctness issue in the “valid rows only” quantile-crossing logic and needs Docusaurus frontmatter plus clearer spark-submit cluster-manager guidance to avoid misleading readers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:327

  • The snippet assumes any quantile-ordering violation implies a negative uncertainty_width, but q20 > q50 or q50 > q80 can still occur while q80 >= q20 (positive width). As a result, validRows should be filtered using the is_crossed flag (or the explicit ordering predicate), otherwise the “valid rows only” coverage can still include crossed quantiles.
// Rows where that ordering is violated have a negative uncertainty_width and
// must NOT be presented as valid uncertainty intervals. Coverage computed over
// all rows (including crossed ones) is therefore misleading — both figures are
// reported here so the reader can make an informed judgement.
// Taking an absolute value or sorting the bounds is NOT a valid fix: it does

docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:455

  • The spark-submit examples label Databricks/Azure Synapse and a “standalone Spark cluster”, but both commands hardcode --master yarn. This is misleading for readers not using YARN (and Databricks isn’t a YARN deployment); either clarify that the example assumes YARN or instruct users to adjust --master for their cluster manager.
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
  --class com.example.drugdiscovery.QSARQuantileApp \
  --master yarn \
  --deploy-mode client \
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

@@ -0,0 +1,496 @@
# LightGBM - Quantile Regression for Drug Discovery (Scala)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Request: Spark Scala standalone app example for LightGBM

3 participants