docs: Add Scala tutorial for LightGBM Quantile Regression in Drug Discovery (#731) - #2701
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey Venkata Surya Mahendra Mula (@MahendraMula) 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
There was a problem hiding this comment.
🟡 Changes recommended
The new tutorial markdown currently has escaped Markdown/broken code formatting, an outdated Maven coordinate, and stops after imports despite the PR description claiming a complete end-to-end walkthrough.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds a Scala-focused tutorial document under notebooks/features/lightgbm/ intended to demonstrate LightGBM quantile regression workflows for drug discovery (QSAR) use cases, complementing the existing PySpark example requested in #731.
Changes:
- Adds a new Scala tutorial markdown file for LightGBM quantile regression in drug discovery.
- Introduces a PySpark vs Scala syntax comparison table and initial environment/import setup section.
File summaries
| File | Description |
|---|---|
| notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md | New Scala tutorial document for quantile regression (currently contains Markdown rendering issues and appears incomplete vs. the PR description). |
Review details
Suppressed comments (3)
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:37
- The PySpark vs Scala table is separated by blank lines, which breaks Markdown table rendering in most parsers. Also, several code spans are unnecessarily escaped (e.g.,
\[ ... \],\_), so they’ll display backslashes literally.
| Feature | Python (PySpark) | Scala (Spark) | Explanation |
| :--- | :--- | :--- | :--- |
| \*\*Parameter Configuration\*\* | `LightGBMRegressor(alpha=0.5, objective="quantile")` | `new LightGBMRegressor().setAlpha(0.5).setObjective("quantile")` | Scala uses the \*\*fluent setter pattern\*\* (`.setParam()`) instead of constructor keyword arguments. |
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:59
- The Maven coordinate is pinned to
0.11.4and also escapes the underscore inside an inline code span (synapseml\_2.12), which will render with a literal backslash. This should match the repo’s current installation guidance (e.g., README showscom.microsoft.azure:synapseml_2.12:1.1.3formaster).
\* \*\*Maven Coordinate:\*\* `com.microsoft.azure:synapseml\_2.12:0.11.4`
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:79
- The Scala code fence is not closed, and
functions.\_will render a literal backslash and isn’t valid Scala. Either close the fenced block and usefunctions._, or switch to an indented code block so the snippet is copy/paste runnable.
```scala
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.\_
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.evaluation.RegressionEvaluator
import com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@microsoft-github-policy-service agree |
There was a problem hiding this comment.
🟡 Changes recommended
The new tutorial contains several copy/paste-breaking documentation issues (notably incorrect sbt dependency notation and inconsistent Maven repository guidance) that should be corrected before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (4)
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:42
- This
spark-shellexample uses theazureedgeMaven repository, which doesn’t match the repository URL recommended in README.md (blob.core.windows.net). Update it for consistency and to avoid resolution issues.
spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3 \
--repositories https://mmlspark.azureedge.net/maven
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:145
- Step 4 always applies a
VectorAssemblerover descriptor columns and transformsqsarDf. If the user chose Option B (LibSVM),qsarDfalready contains afeaturesvector and won’t have the listed descriptor columns, so this transform path will break. Consider skipping assembly whenfeaturesalready exists.
val assembler = new VectorAssembler()
.setInputCols(featureCols)
.setOutputCol("features")
val assembledDf = assembler.transform(qsarDf)
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:268
- This resolver points to
https://mmlspark.azureedge.net/maven, but the repo’s current installation guidance recommendshttps://mmlspark.blob.core.windows.net/maven(README.md:91-93). Keeping these consistent reduces copy/paste failures.
resolvers += "SynapseML Maven Repo" at "https://mmlspark.azureedge.net/maven"
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:370
- The
spark-submitcommand uses theazureedgeMaven repository, which doesn’t match the repository URL recommended in README.md (blob.core.windows.net). This can cause dependency resolution failures depending on the runtime.
--packages com.microsoft.azure:synapseml_2.12:1.1.3 \
--repositories https://mmlspark.azureedge.net/maven \
target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar
- Files reviewed: 1/1 changed files
- Comments generated: 4
- Review effort level: Lite
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The new tutorial contains several concrete configuration inaccuracies (repository URL and Spark/Scala version pins) that are likely to cause dependency resolution or runtime compatibility problems for readers.
Review details
Suppressed comments (5)
Previously missed (1) — in code that hasn't changed since the last review.
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:62
- Hard-coding
.master("local[*]")in the SparkSession builder will force local mode and is likely to break or behave unexpectedly when readers run this on Databricks/Synapse/cluster deployments. It’s safer to letspark-submit/ the notebook environment set the master and just callgetOrCreate().
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:36
- The canonical SynapseML Maven repository in this repo’s install docs is https://mmlspark.blob.core.windows.net/maven; using the azureedge URL here may fail to resolve artifacts (especially for newer Spark lines).
* **Maven Coordinate:** `com.microsoft.azure:synapseml_2.12:1.1.3`
* **Spark Packages:** `com.microsoft.azure:synapseml_2.12:1.1.3`
* **Repository:** `https://mmlspark.azureedge.net/maven`
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:42
- This example should use the same SynapseML Maven repository URL recommended in the main install docs (https://mmlspark.blob.core.windows.net/maven) to avoid dependency resolution issues.
spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3 \
--repositories https://mmlspark.azureedge.net/maven
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:270
- The standalone
build.sbtsnippet pins Spark 3.4.1 and Scala 2.12.18, which doesn’t match this repo’s currentmasterline (Spark 3.5.x / Scala 2.12.17). Using Spark 3.4.x with the 1.1.3 artifacts built for Spark 3.5 can lead to binary incompatibilities at runtime (e.g.,NoSuchMethodError).
scalaVersion := "2.12.18"
resolvers += "SynapseML Maven Repo" at "https://mmlspark.azureedge.net/maven"
val sparkVersion = "3.4.1"
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:370
- This
spark-submitexample uses the azureedge repository URL, but the repo’s install guide recommends https://mmlspark.blob.core.windows.net/maven. Aligning these avoids confusion and resolution failures.
--packages com.microsoft.azure:synapseml_2.12:1.1.3 \
--repositories https://mmlspark.azureedge.net/maven \
target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
The new documentation includes a couple of concrete copy/paste hazards/inconsistencies (hard-coded local master and Scala version mismatch with the repo) that should be corrected to avoid misleading users.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:62
- The SparkSession snippet hard-codes
.master("local[*]"), which can accidentally force local mode when users copy/paste into a real Spark cluster or managed notebook environment (Databricks/Synapse) and may ignore the configured cluster master. For docs, it’s safer to omit.master(...)and rely on the deployment configuration.
notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md:56 - The import list includes
LightGBMRegressionModel, but the tutorial code never references it. Removing unused imports makes the snippet cleaner for copy/paste and avoids unused-import warnings in IDEs.
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
All Copilot feedback has been addressed across the latest commits. Kindly requesting a maintainer review when you get a chance. Thank you! |
There was a problem hiding this comment.
🟡 Changes recommended
The spark-submit section’s wording is inconsistent with the provided --master yarn commands and should be corrected to avoid misleading readers about the intended cluster manager/runtime.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:395
- This section labels the second command as "standalone Spark cluster" but still uses
--master yarn, which is a YARN cluster manager (not Spark Standalone). Either adjust the wording to YARN, or switch the--mastervalue to a Spark Standalone URL.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
| # Package the application | ||
| sbt package | ||
|
|
||
| # Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed) |
There was a problem hiding this comment.
🟡 Changes recommended
The crossed-quantile diagnostic section states crossed rows must be explicitly flagged, but the snippet currently only prints a count while still computing interval metrics for crossed rows, which can produce misleading negative widths and coverage.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
| // Calculate prediction interval width (uncertainty) and interval coverage | ||
| val predictionsWithInterval = predictions.withColumn( | ||
| "uncertainty_width", | ||
| $"pred_q80" - $"pred_q20" | ||
| ).withColumn( | ||
| "within_interval", | ||
| $"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80" | ||
| ) | ||
|
|
||
| // ── Crossed-Quantile Diagnostic ─────────────────────────────────────────────── | ||
| // Each quantile model (q20, q50, q80) is trained *independently*, so there is | ||
| // no mathematical guarantee that q20 ≤ q50 ≤ q80 holds for every compound. | ||
| // When the ordering is violated the resulting "interval" has a negative width or | ||
| // a reversed median, making it meaningless as an uncertainty estimate. These | ||
| // rows must be flagged explicitly rather than absorbed silently by taking absolute | ||
| // values or sorting the bounds — doing so would hide a real model quality signal. | ||
| val crossingCount = predictions.filter( | ||
| $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" | ||
| ).count() | ||
| println(s"Rows with crossed quantiles: $crossingCount") |
There was a problem hiding this comment.
🟡 Changes recommended
The tutorial contains runnable Scala snippets with a crossed-quantile count recomputation/variable reuse issue and spark-submit examples that inaccurately describe --master usage for different cluster modes.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:429
- The
spark-submitexamples label the target as “Databricks / Azure Synapse” and “standalone Spark cluster”, but both commands use--master yarn.yarnis appropriate only for YARN deployments; for Spark Standalone mode the master URL isspark://..., and managed platforms typically don’t usespark-submitthis way. Adjust the wording and the second command’s--masterso the example is accurate.
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
--class com.example.drugdiscovery.QSARQuantileApp \
--master yarn \
--deploy-mode client \
--packages com.microsoft.azure:synapseml_2.12:1.1.3 \
--repositories https://mmlspark.blob.core.windows.net/maven \
target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
spark-submit \
--class com.example.drugdiscovery.QSARQuantileApp \
--master yarn \
--deploy-mode client \
--packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 \
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
| val crossingCount = predictions.filter( | ||
| $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" | ||
| ).count() | ||
|
|
||
| val totalCount = predictionsWithInterval.count() | ||
| val coverageCount = predictionsWithInterval.filter($"within_interval" === true).count() | ||
| val empiricalCoverage = (coverageCount.toDouble / totalCount.toDouble) * 100.0 | ||
|
|
||
| // Coverage restricted to rows where quantile ordering is correct | ||
| val validRows = predictionsWithInterval.filter($"uncertainty_width" >= 0) | ||
| val validTotal = validRows.count() | ||
| val validCoverageCount = validRows.filter($"within_interval" === true).count() | ||
| val validCoverage = if (validTotal > 0) (validCoverageCount.toDouble / validTotal.toDouble) * 100.0 else 0.0 | ||
|
|
||
| println(f"Rows with crossed quantiles : $crossingCount (out of $totalCount)") | ||
| println(f"Empirical Coverage (all rows) : $empiricalCoverage%.2f%% — includes $crossingCount crossed row(s); interpret with caution") | ||
| println(f"Empirical Coverage (valid rows only): $validCoverage%.2f%% (Nominal target: 60.00%%)") |
…n for Drug Discovery (Scala).md Co-authored-by: Rana Singh <ranadeep.dtu@gmail.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The tutorial has a copy/paste-breaking Scala redeclaration (crossingCount) and a couple of misleading spark-submit labeling/details that should be corrected for accuracy.
Review details
Suppressed comments (4)
Previously missed (1) — in code that hasn't changed since the last review.
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:240
predictionsWithIntervalis used in Step 7 for multiple actions (count(), filteredcount()s). Without caching, Spark may recompute the full transformation lineage (including the three model transforms) several times. Caching the DataFrame once makes the evaluation section much faster on non-trivial datasets.
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:295
- In Step 7 the code redeclares
val crossingCounteven though it was already defined in Step 6. If readers run the tutorial top-to-bottom in a single Spark shell / worksheet, Scala will fail with “crossingCount is already defined”. Rename the Step 7 variable (or reuse the earlier one) to keep the tutorial copy/paste friendly.
val crossingCount = predictions.filter(
$"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:423
- This spark-submit example is labeled as “Databricks / Azure Synapse”, but it hard-codes
--master yarn. Since--master yarnspecifically targets YARN, the label is misleading for Databricks (and for Spark Standalone). Consider renaming the comment to match the command (or adjusting the--mastervalue).
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
--class com.example.drugdiscovery.QSARQuantileApp \
--master yarn \
--deploy-mode client \
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:435
- This second spark-submit command is described as “standalone Spark cluster”, but it also uses
--master yarn(YARN). If the intent is Spark’s standalone cluster manager,--master spark://...would be the right form; otherwise, rename the comment to “YARN cluster” to match the flags.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
spark-submit \
--class com.example.drugdiscovery.QSARQuantileApp \
--master yarn \
--deploy-mode client \
--packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 \
--repositories https://mmlspark.blob.core.windows.net/maven \
target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
The added tutorial contains a Scala val redefinition that will fail when executed sequentially and has misleading spark-submit environment wording that should be corrected for accuracy.
Review details
Suppressed comments (4)
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:295
- The tutorial defines
crossingCountin Step 6 and then definescrossingCountagain in Step 7. In a typical Scala REPL / notebook session, re-defining the samevalwill fail and prevents readers from running the tutorial top-to-bottom.
val crossingCount = predictions.filter(
$"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:240
- Step 7 triggers multiple Spark actions (
count()several times) onpredictionsWithInterval, which can recompute the same lineage repeatedly and slow down the tutorial noticeably. Consider cachingpredictionsWithIntervalonce after it’s constructed (and optionally unpersisting later).
val predictionsWithInterval = predictions
.withColumn("is_crossed", $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80")
.withColumn("uncertainty_width", $"pred_q80" - $"pred_q20")
.withColumn("within_interval", $"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80")
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:423
- The
spark-submitexample labels the first command as “Databricks / Azure Synapse” but uses--master yarn, which is specific to Spark-on-YARN and is not representative of Databricks/Synapse submission. This can confuse readers; it’s clearer to describe this as a YARN example and note that--master/--deploy-modeshould be adjusted for the user’s environment.
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
--class com.example.drugdiscovery.QSARQuantileApp \
--master yarn \
--deploy-mode client \
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:432
- The second
spark-submitexample says “standalone Spark cluster” but still uses--master yarn. “Standalone” is a specific Spark master type (spark://...) and differs from YARN, so the wording here is misleading.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
spark-submit \
--class com.example.drugdiscovery.QSARQuantileApp \
--master yarn \
--deploy-mode client \
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Lite
Venkata Surya Mahendra Mula (MahendraMula)
left a comment
There was a problem hiding this comment.
done
There was a problem hiding this comment.
🔵 Needs a closer look
The tutorial currently contains a Scala REPL-breaking val redefinition and has a confusing spark-submit “standalone” label while using --master yarn, which should be corrected for copy/paste correctness and clarity.
Review details
Suppressed comments (2)
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:428
- This comment says “standalone Spark cluster”, but the command still uses
--master yarn(YARN). Either clarify the text (Spark on YARN) or change the master example to a Spark standalone master URL to avoid confusing readers.
# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support)
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:234
- In spark-shell / Scala REPL,
valnames cannot be redefined later in the same session. Step 7 definesval crossingCountagain (line 293), which will fail if the reader ran Step 6 first. Rename the Step 6 variable to avoid a REPL compilation error while keeping Step 7 self-contained.
val crossingCount = predictions.filter(
$"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80"
).count()
println(s"Rows with crossed quantiles: $crossingCount")
- Files reviewed: 1/1 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
The new doc has a correctness issue in the “valid rows only” quantile-crossing logic and needs Docusaurus frontmatter plus clearer spark-submit cluster-manager guidance to avoid misleading readers.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:327
- The snippet assumes any quantile-ordering violation implies a negative
uncertainty_width, butq20 > q50orq50 > q80can still occur whileq80 >= q20(positive width). As a result,validRowsshould be filtered using theis_crossedflag (or the explicit ordering predicate), otherwise the “valid rows only” coverage can still include crossed quantiles.
// Rows where that ordering is violated have a negative uncertainty_width and
// must NOT be presented as valid uncertainty intervals. Coverage computed over
// all rows (including crossed ones) is therefore misleading — both figures are
// reported here so the reader can make an informed judgement.
// Taking an absolute value or sorting the bounds is NOT a valid fix: it does
docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md:455
- The
spark-submitexamples label Databricks/Azure Synapse and a “standalone Spark cluster”, but both commands hardcode--master yarn. This is misleading for readers not using YARN (and Databricks isn’t a YARN deployment); either clarify that the example assumes YARN or instruct users to adjust--masterfor their cluster manager.
# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed)
spark-submit \
--class com.example.drugdiscovery.QSARQuantileApp \
--master yarn \
--deploy-mode client \
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
| @@ -0,0 +1,496 @@ | |||
| # LightGBM - Quantile Regression for Drug Discovery (Scala) | |||
Description
This pull request introduces a comprehensive Scala tutorial for performing Quantile Regression using LightGBM in drug discovery workflows, addressing community request #731.
Related Issue
Closes #731
Changes Proposed
VectorAssembler.LightGBMRegressorwith.setObjective("quantile")and.setAlpha(...).RegressionEvaluator(RMSE).Testing & Verification
LightGBMSuite.scala) and API specifications.Checklist