From a4d0647247bba66685d8e2bd3bc3e54a52eed952 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 10:38:31 +0530 Subject: [PATCH 01/16] docs: add Scala tutorial for LightGBM quantile regression in drug discovery (#731) --- ...e Regression for Drug Discovery (Scala).md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md diff --git a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md new file mode 100644 index 0000000000..d54d8cd548 --- /dev/null +++ b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -0,0 +1,78 @@ +\# LightGBM - Quantile Regression for Drug Discovery (Scala) + + + +\## Overview \& Background + + + +In pharmaceutical research and drug discovery, predicting the biological activity or potency of chemical compounds (Quantitative Structure-Activity Relationship, or \*\*QSAR\*\*) is a foundational task. + + + +Traditional machine learning regression models optimize for \*\*Mean Squared Error (MSE)\*\*, producing a single point estimate representing the \*average\* activity. However, in drug development, high uncertainty can result in costly laboratory failures. + + + +\*\*Quantile Regression\*\* solves this by estimating conditional percentiles (e.g., 20th percentile, 50th percentile/median, and 80th percentile) of the response variable. This creates an \*\*uncertainty envelope\*\* (confidence interval) around each prediction, allowing medicinal chemists to quantify risk and prioritize stable drug candidates. + + + +\--- + + + +\## Key Syntax Differences: PySpark vs. Spark Scala + + + +If you are migrating from the Python SynapseML tutorial, keep these key differences in mind: + + + +| Feature | Python (PySpark) | Scala (Spark) | Explanation | + +| :--- | :--- | :--- | :--- | + +| \*\*Parameter Configuration\*\* | `LightGBMRegressor(alpha=0.5, objective="quantile")` | `new LightGBMRegressor().setAlpha(0.5).setObjective("quantile")` | Scala uses the \*\*fluent setter pattern\*\* (`.setParam()`) instead of constructor keyword arguments. | + +| \*\*Variable Immutability\*\* | `model = ...` | `val model = ...` | Scala uses `val` for immutable variables and `var` for mutable ones. | + +| \*\*Array Definitions\*\* | `\[0.7, 0.3]` | `Array(0.7, 0.3)` | Scala requires explicit `Array(...)` collections for methods like `randomSplit`. | + +| \*\*Lambdas / Filtering\*\* | `\[c for c in cols if c != "label"]` | `cols.filter(\_ != "label")` | Scala uses concise underscore `\_` syntax for anonymous lambda functions. | + +| \*\*Imports\*\* | `import synapse.ml.lightgbm...` | `import com.microsoft.azure.synapse.ml.lightgbm...` | Scala follows full JVM package hierarchy namespaces. | + + + +\--- + + + +\## Step 1: Environment Setup and Imports + + + +To use LightGBM in Spark Scala, ensure the `synapseml` Maven package is attached to your Spark cluster or session: + +\* \*\*Maven Coordinate:\*\* `com.microsoft.azure:synapseml\_2.12:0.11.4` + + + +Import the required Spark and SynapseML classes: + + + +```scala + +import org.apache.spark.sql.SparkSession + +import org.apache.spark.sql.functions.\_ + +import org.apache.spark.ml.feature.VectorAssembler + +import org.apache.spark.ml.evaluation.RegressionEvaluator + +import com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor + From 970fdab3bff9d2967e7cc81f854bfec7a77e32b1 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 12:18:29 +0530 Subject: [PATCH 02/16] docs: complete LightGBM quantile regression Scala tutorial for drug discovery (#731) --- ...e Regression for Drug Discovery (Scala).md | 354 ++++++++++++++++-- 1 file changed, 330 insertions(+), 24 deletions(-) diff --git a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md index d54d8cd548..3b26e6bf6b 100644 --- a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -1,78 +1,384 @@ -\# LightGBM - Quantile Regression for Drug Discovery (Scala) +# LightGBM - Quantile Regression for Drug Discovery (Scala) +## Overview & Background +In pharmaceutical research and drug discovery, predicting the biological activity or potency of chemical compounds (Quantitative Structure-Activity Relationship, or **QSAR**) is a foundational task. -\## Overview \& Background +Traditional machine learning regression models optimize for **Mean Squared Error (MSE)**, producing a single point estimate representing the *conditional mean* activity. However, in lead optimization and drug candidate selection, point estimates alone can be misleading: +* Experimental assays have intrinsic measurement noise. +* Novel chemical scaffolds often reside in sparse regions of chemical space (out-of-domain), where model confidence is naturally lower. +* High variance and uncertainty can lead to costly laboratory synthesis and in vitro assay failures. +**Quantile Regression** addresses this challenge by estimating conditional percentiles (e.g., 20th percentile, 50th percentile / median, and 80th percentile) of the response distribution. Fitting models across multiple quantiles produces an **uncertainty envelope** (prediction interval) for every candidate compound. This empowers medicinal chemists to quantify risk, prioritize high-confidence candidates, and flag compounds requiring further experimental validation. +--- -In pharmaceutical research and drug discovery, predicting the biological activity or potency of chemical compounds (Quantitative Structure-Activity Relationship, or \*\*QSAR\*\*) is a foundational task. +## Key Syntax Differences: PySpark vs. Spark Scala +If you are transitioning from the Python SynapseML tutorial, keep these key differences in mind: +| Feature | Python (PySpark) | Scala (Spark) | Explanation | +| :--- | :--- | :--- | :--- | +| **Parameter Configuration** | `LightGBMRegressor(alpha=0.5, objective="quantile")` | `new LightGBMRegressor().setAlpha(0.5).setObjective("quantile")` | Scala uses the **fluent setter pattern** (`.setParam()`) instead of constructor keyword arguments. | +| **Variable Immutability** | `model = ...` | `val model = ...` | Scala uses `val` for immutable bindings and `var` for mutable variables. | +| **Array Definitions** | `[0.8, 0.2]` | `Array(0.8, 0.2)` | Scala uses typed collections (`Array(...)`, `Seq(...)`). | +| **Anonymous Functions** | `[c for c in cols if c != "label"]` | `cols.filter(_ != "label")` | Scala uses concise underscore `_` syntax for lambdas. | +| **Imports & Namespaces** | `import synapse.ml.lightgbm...` | `import com.microsoft.azure.synapse.ml.lightgbm...` | Scala follows full JVM package hierarchy namespaces. | -Traditional machine learning regression models optimize for \*\*Mean Squared Error (MSE)\*\*, producing a single point estimate representing the \*average\* activity. However, in drug development, high uncertainty can result in costly laboratory failures. +--- +## Step 1: Environment Setup and Dependencies +To use LightGBM in Spark Scala, attach the SynapseML Maven coordinate to your Spark cluster or include it in your build configuration: -\*\*Quantile Regression\*\* solves this by estimating conditional percentiles (e.g., 20th percentile, 50th percentile/median, and 80th percentile) of the response variable. This creates an \*\*uncertainty envelope\*\* (confidence interval) around each prediction, allowing medicinal chemists to quantify risk and prioritize stable drug candidates. +* **Maven Coordinate:** `com.microsoft.azure:synapseml_2.12:1.1.3` +* **Spark Packages:** `com.microsoft.azure:synapseml_2.12:1.1.3` +* **Repository:** `https://mmlspark.azureedge.net/maven` +### Spark Shell / Databricks / Synapse Configuration +When launching `spark-shell` or `spark-submit`, include the package: +```bash +spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3 \ + --repositories https://mmlspark.azureedge.net/maven +``` +--- -\--- +## Step 2: Spark Session and Imports +Import the necessary classes from Spark SQL, Spark ML, and SynapseML: +```scala +import org.apache.spark.sql.SparkSession +import org.apache.spark.sql.functions._ +import org.apache.spark.ml.feature.VectorAssembler +import org.apache.spark.ml.evaluation.RegressionEvaluator +import com.microsoft.azure.synapse.ml.lightgbm.{LightGBMRegressor, LightGBMRegressionModel} -\## Key Syntax Differences: PySpark vs. Spark Scala +// Initialize or retrieve the active SparkSession +val spark = SparkSession.builder() + .appName("LightGBM-QSAR-QuantileRegression") + .master("local[*]") // Use cluster master when deploying in production + .getOrCreate() +import spark.implicits._ +``` +--- -If you are migrating from the Python SynapseML tutorial, keep these key differences in mind: +## Step 3: Dataset Preparation +In QSAR modeling, compounds are typically represented by physicochemical descriptors or molecular fingerprints (e.g., Molecular Weight, LogP, Hydrogen Bond Donors/Acceptors, Topological Polar Surface Area, Rotatable Bonds) mapped to a biological potency target (such as $pIC_{50} = -\log_{10}(IC_{50})$). +### Option A: Self-Contained Synthetic QSAR Dataset +To run this tutorial immediately without external network dependencies, generate a synthetic QSAR dataset: -| Feature | Python (PySpark) | Scala (Spark) | Explanation | +```scala +case class CompoundDescriptor( + compound_id: String, + molecular_weight: Double, + logP: Double, + hbd_count: Double, + hba_count: Double, + tpsa: Double, + rotatable_bonds: Double, + pIC50: Double // Bioactivity potency target +) + +// Generate sample molecular descriptor data with heteroscedastic noise +val random = new scala.util.Random(42) +val sampleCompounds = (1 to 500).map { i => + val mw = 200.0 + random.nextDouble() * 350.0 // Molecular Weight (Da) + val logP = -0.5 + random.nextDouble() * 5.5 // Octanol-water partition coefficient + val hbd = random.nextInt(6).toDouble // Hydrogen Bond Donors + val hba = random.nextInt(10).toDouble // Hydrogen Bond Acceptors + val tpsa = 20.0 + random.nextDouble() * 120.0 // Topological Polar Surface Area (Ų) + val rotBonds = random.nextInt(8).toDouble // Rotatable Bonds + + // Synthetic QSAR response with scaffold-dependent variance (heteroscedasticity) + val latentPotency = 4.0 + (0.005 * mw) + (0.4 * logP) - (0.15 * hbd) - (0.01 * tpsa) + val noiseScale = 0.2 + 0.1 * (logP.abs) // Uncertainty increases with extreme logP + val noise = random.nextGaussian() * noiseScale + val potency = latentPotency + noise + + CompoundDescriptor(s"CMPD-$i", mw, logP, hbd, hba, tpsa, rotBonds, potency) +} + +val qsarDf = sampleCompounds.toDF() +qsarDf.show(5, truncate = false) +``` + +### Option B: Public Triazines Benchmark Dataset (LibSVM) +SynapseML also hosts the classic benchmark Triazines QSAR dataset (predicting inhibition of dihydrofolate reductase by pyrimidines): -| :--- | :--- | :--- | :--- | +```scala +// Load benchmark Triazines QSAR dataset (requires cluster network connectivity) +val triazinesDf = spark.read + .format("libsvm") + .load("wasbs://publicwasb@mmlspark.blob.core.windows.net/triazines.scale.svmlight") -| \*\*Parameter Configuration\*\* | `LightGBMRegressor(alpha=0.5, objective="quantile")` | `new LightGBMRegressor().setAlpha(0.5).setObjective("quantile")` | Scala uses the \*\*fluent setter pattern\*\* (`.setParam()`) instead of constructor keyword arguments. | +println(s"Total records in Triazines dataset: ${triazinesDf.count()}") +triazinesDf.printSchema() +``` -| \*\*Variable Immutability\*\* | `model = ...` | `val model = ...` | Scala uses `val` for immutable variables and `var` for mutable ones. | +--- -| \*\*Array Definitions\*\* | `\[0.7, 0.3]` | `Array(0.7, 0.3)` | Scala requires explicit `Array(...)` collections for methods like `randomSplit`. | +## Step 4: Feature Assembly & Train/Test Split -| \*\*Lambdas / Filtering\*\* | `\[c for c in cols if c != "label"]` | `cols.filter(\_ != "label")` | Scala uses concise underscore `\_` syntax for anonymous lambda functions. | +Assemble the molecular descriptor columns into a single Spark ML feature vector: -| \*\*Imports\*\* | `import synapse.ml.lightgbm...` | `import com.microsoft.azure.synapse.ml.lightgbm...` | Scala follows full JVM package hierarchy namespaces. | +```scala +val featureCols = Array( + "molecular_weight", + "logP", + "hbd_count", + "hba_count", + "tpsa", + "rotatable_bonds" +) +val assembler = new VectorAssembler() + .setInputCols(featureCols) + .setOutputCol("features") +val assembledDf = assembler.transform(qsarDf) -\--- +// Split into training (80%) and testing (20%) datasets +val Array(trainData, testData) = assembledDf.randomSplit(Array(0.8, 0.2), seed = 1234L) +trainData.cache() +testData.cache() +println(s"Training set: ${trainData.count()} compounds") +println(s"Testing set: ${testData.count()} compounds") +``` +--- -\## Step 1: Environment Setup and Imports +## Step 5: Training Multi-Quantile LightGBM Models +To construct an **uncertainty envelope**, train three separate `LightGBMRegressor` estimators configured with `setObjective("quantile")`: +* **$\alpha = 0.20$**: 20th percentile (conservative lower bound of compound potency). +* **$\alpha = 0.50$**: 50th percentile (median prediction, robust to outliers). +* **$\alpha = 0.80$**: 80th percentile (optimistic upper bound of compound potency). +```scala +// Helper method to create and configure a quantile regressor +def createQuantileRegressor(alpha: Double, predCol: String): LightGBMRegressor = { + new LightGBMRegressor() + .setObjective("quantile") + .setAlpha(alpha) + .setLabelCol("pIC50") + .setFeaturesCol("features") + .setPredictionCol(predCol) + .setNumLeaves(31) + .setNumIterations(100) + .setLearningRate(0.05) + .setMinDataInLeaf(10) + .setSeed(42) +} -To use LightGBM in Spark Scala, ensure the `synapseml` Maven package is attached to your Spark cluster or session: +println("Training 20th percentile (Lower Bound) model...") +val modelQ20 = createQuantileRegressor(0.20, "pred_q20").fit(trainData) -\* \*\*Maven Coordinate:\*\* `com.microsoft.azure:synapseml\_2.12:0.11.4` +println("Training 50th percentile (Median) model...") +val modelQ50 = createQuantileRegressor(0.50, "pred_q50_median").fit(trainData) +println("Training 80th percentile (Upper Bound) model...") +val modelQ80 = createQuantileRegressor(0.80, "pred_q80").fit(trainData) +``` +--- -Import the required Spark and SynapseML classes: +## Step 6: Generating the Uncertainty Envelope +Transform the test data sequentially through all three models, then calculate the **uncertainty width** ($q_{80} - q_{20}$): +```scala +val predictions = modelQ80.transform( + modelQ50.transform( + modelQ20.transform(testData) + ) +) + +// Calculate prediction interval width (uncertainty) and interval coverage +val predictionsWithInterval = predictions.withColumn( + "uncertainty_width", + $"pred_q80" - $"pred_q20" +).withColumn( + "within_interval", + $"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80" +) + +// Display sample predictions with uncertainty bounds +predictionsWithInterval + .select("compound_id", "pIC50", "pred_q20", "pred_q50_median", "pred_q80", "uncertainty_width", "within_interval") + .show(10, truncate = false) +``` + +### Interpretation for Medicinal Chemists +* **Small `uncertainty_width`:** The model is confident; the molecule's chemical features lie in well-sampled chemical space. +* **Large `uncertainty_width`:** High epistemic or assay uncertainty; proceed with caution before ordering synthesis. +* **High `pred_q20`:** Even under pessimistic estimation, the molecule exhibits strong potency—ideal for prioritization. + +--- + +## Step 7: Model Evaluation & Validation + +Evaluate the median model using standard regression metrics (RMSE and MAE) via `RegressionEvaluator`, and compute empirical coverage: ```scala +// 1. Evaluate Median Model RMSE +val rmseEvaluator = new RegressionEvaluator() + .setLabelCol("pIC50") + .setPredictionCol("pred_q50_median") + .setMetricName("rmse") -import org.apache.spark.sql.SparkSession +val rmse = rmseEvaluator.evaluate(predictionsWithInterval) +println(f"Median Model RMSE: $rmse%.4f") -import org.apache.spark.sql.functions.\_ +// 2. Evaluate Median Model MAE +val maeEvaluator = new RegressionEvaluator() + .setLabelCol("pIC50") + .setPredictionCol("pred_q50_median") + .setMetricName("mae") -import org.apache.spark.ml.feature.VectorAssembler +val mae = maeEvaluator.evaluate(predictionsWithInterval) +println(f"Median Model MAE: $mae%.4f") -import org.apache.spark.ml.evaluation.RegressionEvaluator +// 3. Empirical Interval Coverage (Nominal target: 80% - 20% = 60%) +val coverageCount = predictionsWithInterval.filter($"within_interval" === true).count() +val totalCount = predictionsWithInterval.count() +val empiricalCoverage = (coverageCount.toDouble / totalCount.toDouble) * 100.0 +println(f"Empirical Coverage: $empiricalCoverage%.2f%% (Nominal target: 60.00%%)") +``` + +--- + +## Step 8: Standalone Spark Scala Application (`spark-submit`) + +To package this workflow into a standalone Scala application as requested in [#731](https://github.com/microsoft/SynapseML/issues/731), organize the project with sbt: + +### 1. `build.sbt` +```scala +name := "synapseml-lightgbm-qsar-standalone" +version := "1.0.0" +scalaVersion := "2.12.18" + +resolvers += "SynapseML Maven Repo" at "https://mmlspark.azureedge.net/maven" + +val sparkVersion = "3.4.1" + +libraryDependencies ++= Seq( + "org.apache.spark" %% "spark-core" % sparkVersion % "provided", + "org.apache.spark" %% "spark-sql" % sparkVersion % "provided", + "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided", + "com.microsoft.azure" %% "synapseml_2.12" % "1.1.3" +) +``` + +### 2. Standalone Application (`QSARQuantileApp.scala`) +```scala +package com.example.drugdiscovery + +import org.apache.spark.sql.SparkSession +import org.apache.spark.ml.feature.VectorAssembler +import org.apache.spark.ml.evaluation.RegressionEvaluator import com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor +object QSARQuantileApp { + def main(args: Array[String]): Unit = { + val spark = SparkSession.builder() + .appName("QSAR-Quantile-Regression-Standalone") + .getOrCreate() + + import spark.implicits._ + + println("=== Running SynapseML LightGBM Quantile Regression Pipeline ===") + + // 1. Generate Synthetic Data + val random = new scala.util.Random(42) + val data = (1 to 1000).map { i => + val mw = 150.0 + random.nextDouble() * 400.0 + val logP = -1.0 + random.nextDouble() * 6.0 + val hbd = random.nextInt(5).toDouble + val hba = random.nextInt(9).toDouble + val potency = 5.0 + 0.004 * mw + 0.35 * logP - 0.1 * hbd + random.nextGaussian() * 0.25 + (s"MOL_$i", mw, logP, hbd, hba, potency) + }.toDF("id", "mw", "logP", "hbd", "hba", "potency") + + // 2. Assemble Features + val assembler = new VectorAssembler() + .setInputCols(Array("mw", "logP", "hbd", "hba")) + .setOutputCol("features") + + val assembled = assembler.transform(data) + val Array(train, test) = assembled.randomSplit(Array(0.8, 0.2), 42L) + + // 3. Train Quantile Models (10th, 50th, 90th percentiles for an 80% confidence band) + val quantiles = Seq( + (0.10, "pred_lower_10"), + (0.50, "pred_median_50"), + (0.90, "pred_upper_90") + ) + + var scoredTest = test + for ((alpha, predCol) <- quantiles) { + val model = new LightGBMRegressor() + .setObjective("quantile") + .setAlpha(alpha) + .setLabelCol("potency") + .setFeaturesCol("features") + .setPredictionCol(predCol) + .setNumLeaves(31) + .setNumIterations(50) + .setLearningRate(0.05) + .fit(train) + + scoredTest = model.transform(scoredTest) + } + + // 4. Compute Metrics + val evaluator = new RegressionEvaluator() + .setLabelCol("potency") + .setPredictionCol("pred_median_50") + .setMetricName("rmse") + + val rmse = evaluator.evaluate(scoredTest) + println(f"Median Model RMSE: $rmse%.4f") + + scoredTest.select("id", "potency", "pred_lower_10", "pred_median_50", "pred_upper_90") + .show(5, truncate = false) + + spark.stop() + } +} +``` + +### 3. Execution via `spark-submit` +```bash +# Package the application +sbt package + +# Submit to Spark cluster +spark-submit \ + --class com.example.drugdiscovery.QSARQuantileApp \ + --master yarn \ + --deploy-mode client \ + --packages com.microsoft.azure:synapseml_2.12:1.1.3 \ + --repositories https://mmlspark.azureedge.net/maven \ + target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar +``` + +--- + +## Summary + +In this guide, you learned how to: +1. Configure SynapseML LightGBM in Apache Spark Scala using current coordinates (`1.1.3`). +2. Translate PySpark syntax to idiomatic Scala using fluent setter methods (`.setParam()`). +3. Model biological activity ($pIC_{50}$) with Quantile Regression to estimate uncertainty intervals ($q_{20}, q_{50}, q_{80}$). +4. Calculate empirical coverage and evaluate prediction accuracy with Spark ML's `RegressionEvaluator`. +5. Package and submit a standalone Spark Scala LightGBM application using `sbt` and `spark-submit`. + + From 3d48245c35b656fab84d7a7f0aa7fbf9d93ae437 Mon Sep 17 00:00:00 2001 From: Venkata Surya Mahendra Mula Date: Mon, 7 Sep 2026 14:07:58 +0530 Subject: [PATCH 03/16] Fix dependency declaration for synapseml Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- ...LightGBM - Quantile Regression for Drug Discovery (Scala).md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 3b26e6bf6b..2469956017 100644 --- a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -273,7 +273,7 @@ libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion % "provided", "org.apache.spark" %% "spark-sql" % sparkVersion % "provided", "org.apache.spark" %% "spark-mllib" % sparkVersion % "provided", - "com.microsoft.azure" %% "synapseml_2.12" % "1.1.3" + "com.microsoft.azure" % "synapseml_2.12" % "1.1.3" ) ``` From 3545e5abc875a8bea301ef2377a30b419113cbc3 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 14:16:35 +0530 Subject: [PATCH 04/16] docs: align Maven repo URL, Spark version, and LibSVM dataset path in Scala tutorial (#731) --- ...e Regression for Drug Discovery (Scala).md | 27 ++++++++++++++----- 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 2469956017..f8eb975dd8 100644 --- a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -33,13 +33,13 @@ To use LightGBM in Spark Scala, attach the SynapseML Maven coordinate to your Sp * **Maven Coordinate:** `com.microsoft.azure:synapseml_2.12:1.1.3` * **Spark Packages:** `com.microsoft.azure:synapseml_2.12:1.1.3` -* **Repository:** `https://mmlspark.azureedge.net/maven` +* **Repository:** `https://mmlspark.blob.core.windows.net/maven` ### Spark Shell / Databricks / Synapse Configuration When launching `spark-shell` or `spark-submit`, include the package: ```bash spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3 \ - --repositories https://mmlspark.azureedge.net/maven + --repositories https://mmlspark.blob.core.windows.net/maven ``` --- @@ -109,7 +109,9 @@ qsarDf.show(5, truncate = false) ``` ### Option B: Public Triazines Benchmark Dataset (LibSVM) -SynapseML also hosts the classic benchmark Triazines QSAR dataset (predicting inhibition of dihydrofolate reductase by pyrimidines): +SynapseML also hosts the classic benchmark Triazines QSAR dataset (predicting inhibition of dihydrofolate reductase by pyrimidines). + +> **Note on LibSVM Schema:** The LibSVM format pre-assembles molecular features into a single Vector column (`features`) and maps the target property to `label`. As shown below, it does not require an intermediate `VectorAssembler` step and can be fed directly to `LightGBMRegressor`: ```scala // Load benchmark Triazines QSAR dataset (requires cluster network connectivity) @@ -119,8 +121,21 @@ val triazinesDf = spark.read println(s"Total records in Triazines dataset: ${triazinesDf.count()}") triazinesDf.printSchema() + +// Direct training on LibSVM's native columns without VectorAssembler: +val Array(triazinesTrain, triazinesTest) = triazinesDf.randomSplit(Array(0.8, 0.2), seed = 1234L) +val triazinesModel = new LightGBMRegressor() + .setObjective("quantile") + .setAlpha(0.5) + .setLabelCol("label") + .setFeaturesCol("features") + .fit(triazinesTrain) + +triazinesModel.transform(triazinesTest).select("label", "prediction").show(5) ``` +> **Tutorial Flow:** The subsequent sections (Steps 4 through 7) follow **Option A (`qsarDf`)** to demonstrate how to perform custom feature engineering with `VectorAssembler`, multi-quantile uncertainty envelope modeling, and domain-specific bioactivity metric evaluation. + --- ## Step 4: Feature Assembly & Train/Test Split @@ -265,9 +280,9 @@ name := "synapseml-lightgbm-qsar-standalone" version := "1.0.0" scalaVersion := "2.12.18" -resolvers += "SynapseML Maven Repo" at "https://mmlspark.azureedge.net/maven" +resolvers += "SynapseML Maven Repo" at "https://mmlspark.blob.core.windows.net/maven" -val sparkVersion = "3.4.1" +val sparkVersion = "3.5.0" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % sparkVersion % "provided", @@ -366,7 +381,7 @@ spark-submit \ --master yarn \ --deploy-mode client \ --packages com.microsoft.azure:synapseml_2.12:1.1.3 \ - --repositories https://mmlspark.azureedge.net/maven \ + --repositories https://mmlspark.blob.core.windows.net/maven \ target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar ``` From 02b389fd5e6e3f56f5ade4e6b3e175d57977b6e8 Mon Sep 17 00:00:00 2001 From: Venkata Surya Mahendra Mula Date: Mon, 7 Sep 2026 14:21:55 +0530 Subject: [PATCH 05/16] Update Scala version from 2.12.18 to 2.12.17 Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- ...ightGBM - Quantile Regression for Drug Discovery (Scala).md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md index f8eb975dd8..3c0cadef6f 100644 --- a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -278,8 +278,7 @@ To package this workflow into a standalone Scala application as requested in [#7 ```scala name := "synapseml-lightgbm-qsar-standalone" version := "1.0.0" -scalaVersion := "2.12.18" - +scalaVersion := "2.12.17" resolvers += "SynapseML Maven Repo" at "https://mmlspark.blob.core.windows.net/maven" val sparkVersion = "3.5.0" From 37d11d1adbdf1e6fc751f3c9203fda45f96eb28a Mon Sep 17 00:00:00 2001 From: Venkata Surya Mahendra Mula Date: Mon, 7 Sep 2026 14:33:50 +0530 Subject: [PATCH 06/16] Remove unused import and update SparkSession config --- ...ightGBM - Quantile Regression for Drug Discovery (Scala).md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 3c0cadef6f..8d402ff20a 100644 --- a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -53,12 +53,11 @@ import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.ml.feature.VectorAssembler import org.apache.spark.ml.evaluation.RegressionEvaluator -import com.microsoft.azure.synapse.ml.lightgbm.{LightGBMRegressor, LightGBMRegressionModel} +import com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor // Initialize or retrieve the active SparkSession val spark = SparkSession.builder() .appName("LightGBM-QSAR-QuantileRegression") - .master("local[*]") // Use cluster master when deploying in production .getOrCreate() import spark.implicits._ From b7fdb545c035e03caa6753ee2256ec0a4a9bfe66 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 17:17:27 +0530 Subject: [PATCH 07/16] docs: move Scala tutorial to published docs tree --- .../LightGBM - Quantile Regression for Drug Discovery (Scala).md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {notebooks/features/lightgbm => docs/Explore Algorithms/LightGBM}/LightGBM - Quantile Regression for Drug Discovery (Scala).md (100%) diff --git a/notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md similarity index 100% rename from notebooks/features/lightgbm/LightGBM - Quantile Regression for Drug Discovery (Scala).md rename to docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md From d10339d446e42af2c01220df0e033f67f1000f77 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 17:39:42 +0530 Subject: [PATCH 08/16] docs: document hadoop-azure:3.3.4 requirement for standalone Spark wasbs:// access On standard standalone Apache Spark 3.5.0 setups with only SynapseML included, reading Option B's wasbs:// URL throws: ClassNotFoundException: org.apache.hadoop.fs.azure.NativeAzureFileSystem Managed platforms (Databricks, Azure Synapse) pre-install the Azure Hadoop connector. Standalone Spark users must explicitly include hadoop-azure. Changes: - Step 1: Add standalone-specific spark-shell command with com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 - Option B: Add warning callout explaining the wasbs:// connector requirement and inline code comment referencing Step 1 - Step 8 spark-submit: Add a second command variant for standalone Spark that includes hadoop-azure:3.3.4 alongside synapseml_2.12:1.1.3 Fixes P2 issue raised by Rana Singh in PR #2701. --- ...e Regression for Drug Discovery (Scala).md | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 8d402ff20a..831636d3f5 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -42,6 +42,14 @@ spark-shell --packages com.microsoft.azure:synapseml_2.12:1.1.3 \ --repositories https://mmlspark.blob.core.windows.net/maven ``` +> **Note for Standalone Spark users (Option B — `wasbs://` dataset):** If you intend to use **Option B** (reading the public LibSVM dataset over Azure Blob Storage via `wasbs://`), you must also include the `hadoop-azure` connector. Managed cloud platforms (Databricks, Azure Synapse) pre-install this driver, but **standalone Apache Spark does not include it by default**. Add `org.apache.hadoop:hadoop-azure:3.3.4` to `--packages`: +> ```bash +> spark-shell \ +> --packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 \ +> --repositories https://mmlspark.blob.core.windows.net/maven +> ``` +> Without this, Spark will throw `ClassNotFoundException: org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure`. + --- ## Step 2: Spark Session and Imports @@ -112,8 +120,13 @@ SynapseML also hosts the classic benchmark Triazines QSAR dataset (predicting in > **Note on LibSVM Schema:** The LibSVM format pre-assembles molecular features into a single Vector column (`features`) and maps the target property to `label`. As shown below, it does not require an intermediate `VectorAssembler` step and can be fed directly to `LightGBMRegressor`: +> **⚠️ Standalone Spark Requirement:** The URL below uses the `wasbs://` scheme to read from Azure Blob Storage. On **standard standalone Apache Spark 3.5.0** (with only SynapseML included), this will throw `ClassNotFoundException: org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure` because the Azure Hadoop file system driver is **not bundled by default**. Managed platforms (Databricks, Azure Synapse) pre-install this connector automatically. +> +> To resolve this on standalone Spark, add `org.apache.hadoop:hadoop-azure:3.3.4` to your `--packages` flag (see Step 1 above). + ```scala // Load benchmark Triazines QSAR dataset (requires cluster network connectivity) +// NOTE: wasbs:// requires hadoop-azure on standalone Spark — see Step 1 for the correct --packages flag val triazinesDf = spark.read .format("libsvm") .load("wasbs://publicwasb@mmlspark.blob.core.windows.net/triazines.scale.svmlight") @@ -373,7 +386,7 @@ object QSARQuantileApp { # Package the application sbt package -# Submit to Spark cluster +# Submit to Spark cluster (Databricks / Azure Synapse — hadoop-azure is pre-installed) spark-submit \ --class com.example.drugdiscovery.QSARQuantileApp \ --master yarn \ @@ -381,6 +394,15 @@ spark-submit \ --packages com.microsoft.azure:synapseml_2.12:1.1.3 \ --repositories https://mmlspark.blob.core.windows.net/maven \ target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar + +# Submit to standalone Spark cluster (hadoop-azure must be added explicitly for wasbs:// support) +spark-submit \ + --class com.example.drugdiscovery.QSARQuantileApp \ + --master yarn \ + --deploy-mode client \ + --packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 \ + --repositories https://mmlspark.blob.core.windows.net/maven \ + target/scala-2.12/synapseml-lightgbm-qsar-standalone_2.12-1.0.0.jar ``` --- From 9950a244e6697fe0ed48989b31f1f9b58a0192f2 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 17:48:59 +0530 Subject: [PATCH 09/16] docs: apply contributor exact wording for Option B wasbs note (PR #2701) --- ...GBM - Quantile Regression for Drug Discovery (Scala).md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 831636d3f5..b7da10dee5 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -118,15 +118,12 @@ qsarDf.show(5, truncate = false) ### Option B: Public Triazines Benchmark Dataset (LibSVM) SynapseML also hosts the classic benchmark Triazines QSAR dataset (predicting inhibition of dihydrofolate reductase by pyrimidines). -> **Note on LibSVM Schema:** The LibSVM format pre-assembles molecular features into a single Vector column (`features`) and maps the target property to `label`. As shown below, it does not require an intermediate `VectorAssembler` step and can be fed directly to `LightGBMRegressor`: - -> **⚠️ Standalone Spark Requirement:** The URL below uses the `wasbs://` scheme to read from Azure Blob Storage. On **standard standalone Apache Spark 3.5.0** (with only SynapseML included), this will throw `ClassNotFoundException: org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure` because the Azure Hadoop file system driver is **not bundled by default**. Managed platforms (Databricks, Azure Synapse) pre-install this connector automatically. +> LibSVM supplies a `features` vector and a `label` column, so this path does not need `VectorAssembler`. > -> To resolve this on standalone Spark, add `org.apache.hadoop:hadoop-azure:3.3.4` to your `--packages` flag (see Step 1 above). +> Standalone Apache Spark also needs Hadoop's Azure connector to read `wasbs://` URLs. For Spark 3.5.0 with Hadoop 3.3.4, use `--packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4` with the Maven repository from Step 1. On managed clusters, use the connector supplied by the runtime or match the connector to the runtime's Hadoop version. ```scala // Load benchmark Triazines QSAR dataset (requires cluster network connectivity) -// NOTE: wasbs:// requires hadoop-azure on standalone Spark — see Step 1 for the correct --packages flag val triazinesDf = spark.read .format("libsvm") .load("wasbs://publicwasb@mmlspark.blob.core.windows.net/triazines.scale.svmlight") From 93864c9ab3e49aafb1242ae6d2be950a53589f53 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 18:18:19 +0530 Subject: [PATCH 10/16] docs: add crossed quantiles diagnostic to Scala LightGBM tutorial --- ...uantile Regression for Drug Discovery (Scala).md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index b7da10dee5..0c240e95b5 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -233,6 +233,19 @@ val predictionsWithInterval = predictions.withColumn( $"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80" ) +// ── Crossed-Quantile Diagnostic ─────────────────────────────────────────────── +// Each quantile model (q20, q50, q80) is trained *independently*, so there is +// no mathematical guarantee that q20 ≤ q50 ≤ q80 holds for every compound. +// When the ordering is violated the resulting "interval" has a negative width or +// a reversed median, making it meaningless as an uncertainty estimate. These +// rows must be flagged explicitly rather than absorbed silently by taking absolute +// values or sorting the bounds — doing so would hide a real model quality signal. +val crossingCount = predictions.filter( + $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" +).count() +println(s"Rows with crossed quantiles: $crossingCount") +// ───────────────────────────────────────────────────────────────────────────── + // Display sample predictions with uncertainty bounds predictionsWithInterval .select("compound_id", "pIC50", "pred_q20", "pred_q50_median", "pred_q80", "uncertainty_width", "within_interval") From e6ae72aab1912dd496b59a0ba554c9e9bf7e9959 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 18:26:32 +0530 Subject: [PATCH 11/16] docs: flag crossed quantiles in displayed output and report alongside coverage --- ...e Regression for Drug Discovery (Scala).md | 60 ++++++++++++------- 1 file changed, 38 insertions(+), 22 deletions(-) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 0c240e95b5..3ad4d6b76c 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -224,31 +224,24 @@ val predictions = modelQ80.transform( ) ) -// Calculate prediction interval width (uncertainty) and interval coverage -val predictionsWithInterval = predictions.withColumn( - "uncertainty_width", - $"pred_q80" - $"pred_q20" -).withColumn( - "within_interval", - $"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80" -) - // ── Crossed-Quantile Diagnostic ─────────────────────────────────────────────── -// Each quantile model (q20, q50, q80) is trained *independently*, so there is -// no mathematical guarantee that q20 ≤ q50 ≤ q80 holds for every compound. -// When the ordering is violated the resulting "interval" has a negative width or -// a reversed median, making it meaningless as an uncertainty estimate. These -// rows must be flagged explicitly rather than absorbed silently by taking absolute -// values or sorting the bounds — doing so would hide a real model quality signal. +// Independent quantile fits do not guarantee monotonic ordering (q20 <= q50 <= q80). +// When crossed, subtracting predictions produces negative widths and invalid intervals. +// See maintainer explanation in lightgbm-org/LightGBM#3447. val crossingCount = predictions.filter( $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" ).count() println(s"Rows with crossed quantiles: $crossingCount") -// ───────────────────────────────────────────────────────────────────────────── -// Display sample predictions with uncertainty bounds +// Flag crossed quantiles and calculate prediction interval width & coverage +val predictionsWithInterval = predictions + .withColumn("is_crossed", $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80") + .withColumn("uncertainty_width", $"pred_q80" - $"pred_q20") + .withColumn("within_interval", $"pIC50" >= $"pred_q20" && $"pIC50" <= $"pred_q80") + +// Display sample predictions with uncertainty bounds and crossed flags predictionsWithInterval - .select("compound_id", "pIC50", "pred_q20", "pred_q50_median", "pred_q80", "uncertainty_width", "within_interval") + .select("compound_id", "pIC50", "pred_q20", "pred_q50_median", "pred_q80", "uncertainty_width", "within_interval", "is_crossed") .show(10, truncate = false) ``` @@ -256,12 +249,13 @@ predictionsWithInterval * **Small `uncertainty_width`:** The model is confident; the molecule's chemical features lie in well-sampled chemical space. * **Large `uncertainty_width`:** High epistemic or assay uncertainty; proceed with caution before ordering synthesis. * **High `pred_q20`:** Even under pessimistic estimation, the molecule exhibits strong potency—ideal for prioritization. +* **`is_crossed` flag:** Compounds with reversed or crossed endpoints (`is_crossed == true`) have negative widths or inconsistent medians. They must not be treated as valid uncertainty intervals; taking absolute values or sorting does not establish advertised coverage. --- ## Step 7: Model Evaluation & Validation -Evaluate the median model using standard regression metrics (RMSE and MAE) via `RegressionEvaluator`, and compute empirical coverage: +Evaluate the median model using standard regression metrics (RMSE and MAE) via `RegressionEvaluator`. The **crossed-quantile count and empirical coverage are printed together** so the reader can judge whether the coverage figure is reliable: ```scala // 1. Evaluate Median Model RMSE @@ -282,12 +276,34 @@ val maeEvaluator = new RegressionEvaluator() val mae = maeEvaluator.evaluate(predictionsWithInterval) println(f"Median Model MAE: $mae%.4f") -// 3. Empirical Interval Coverage (Nominal target: 80% - 20% = 60%) +// 3. Crossed-quantile count reported alongside empirical coverage +// ── IMPORTANT ──────────────────────────────────────────────────────────────── +// Because each quantile model is trained independently, there is no guarantee +// that q20 ≤ q50 ≤ q80 holds for every compound (see lightgbm-org/LightGBM#3447). +// Rows where that ordering is violated have a negative uncertainty_width and +// must NOT be presented as valid uncertainty intervals. Coverage computed over +// all rows (including crossed ones) is therefore misleading — both figures are +// reported here so the reader can make an informed judgement. +// Taking an absolute value or sorting the bounds is NOT a valid fix: it does +// not establish the advertised 60 % nominal coverage. +val crossingCount = predictions.filter( + $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" +).count() + +val totalCount = predictionsWithInterval.count() val coverageCount = predictionsWithInterval.filter($"within_interval" === true).count() -val totalCount = predictionsWithInterval.count() val empiricalCoverage = (coverageCount.toDouble / totalCount.toDouble) * 100.0 -println(f"Empirical Coverage: $empiricalCoverage%.2f%% (Nominal target: 60.00%%)") +// Coverage restricted to rows where quantile ordering is correct +val validRows = predictionsWithInterval.filter($"uncertainty_width" >= 0) +val validTotal = validRows.count() +val validCoverageCount = validRows.filter($"within_interval" === true).count() +val validCoverage = if (validTotal > 0) (validCoverageCount.toDouble / validTotal.toDouble) * 100.0 else 0.0 + +println(f"Rows with crossed quantiles : $crossingCount (out of $totalCount)") +println(f"Empirical Coverage (all rows) : $empiricalCoverage%.2f%% — includes $crossingCount crossed row(s); interpret with caution") +println(f"Empirical Coverage (valid rows only): $validCoverage%.2f%% (Nominal target: 60.00%%)") +// ───────────────────────────────────────────────────────────────────────────── ``` --- From 2c988e498a855168890ee7f0542927c2d4ae802d Mon Sep 17 00:00:00 2001 From: Venkata Surya Mahendra Mula Date: Mon, 7 Sep 2026 18:32:54 +0530 Subject: [PATCH 12/16] Update docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md Co-authored-by: Rana Singh --- ...- Quantile Regression for Drug Discovery (Scala).md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 3ad4d6b76c..2d199b6bba 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -246,9 +246,13 @@ predictionsWithInterval ``` ### Interpretation for Medicinal Chemists -* **Small `uncertainty_width`:** The model is confident; the molecule's chemical features lie in well-sampled chemical space. -* **Large `uncertainty_width`:** High epistemic or assay uncertainty; proceed with caution before ordering synthesis. -* **High `pred_q20`:** Even under pessimistic estimation, the molecule exhibits strong potency—ideal for prioritization. +For correctly ordered quantiles: + +* A small `uncertainty_width` means the estimated 20th and 80th percentiles are close. It does not establish model confidence or show that a compound is inside the training domain. +* A large `uncertainty_width` means the estimated response interval is wide. These models do not separate assay noise from uncertainty in the fitted model. +* A high `pred_q20` is a high estimated lower response quantile, not a guaranteed minimum potency. Check held-out interval coverage and applicability to new compounds before using it for prioritization. + +This synthetic example demonstrates the API, not a validated predictor of compound activity. * **`is_crossed` flag:** Compounds with reversed or crossed endpoints (`is_crossed == true`) have negative widths or inconsistent medians. They must not be treated as valid uncertainty intervals; taking absolute values or sorting does not establish advertised coverage. --- From 10fc4c314634171b9e9417babaa9fb9c7ccb24e5 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 19:59:03 +0530 Subject: [PATCH 13/16] docs(lightgbm): add tested runtime matrix and table of contents --- ...e Regression for Drug Discovery (Scala).md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 2d199b6bba..3a9d9a6fcf 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -1,5 +1,23 @@ # LightGBM - Quantile Regression for Drug Discovery (Scala) +## Contents + +- [Overview & Background](#overview--background) +- [Tested runtime / compatibility matrix](#tested-runtime--compatibility-matrix) +- [Key Syntax Differences](#key-syntax-differences-pyspark-vs-spark-scala) +- [Step 1: Environment Setup and Dependencies](#step-1-environment-setup-and-dependencies) +- [Step 2: Spark Session and Imports](#step-2-spark-session-and-imports) +- [Step 3: Dataset Preparation](#step-3-dataset-preparation) +- [Step 4: Feature Assembly & Train/Test Split](#step-4-feature-assembly--traintest-split) +- [Step 5: Training Multi-Quantile LightGBM Models](#step-5-training-multi-quantile-lightgbm-models) +- [Step 6: Generating the Uncertainty Envelope](#step-6-generating-the-uncertainty-envelope) +- [Step 7: Model Evaluation & Validation](#step-7-model-evaluation--validation) +- [Step 8: Standalone Spark Scala Application (`spark-submit`)](#step-8-standalone-spark-scala-application-spark-submit) +- [Troubleshooting & common runtime errors](#troubleshooting--common-runtime-errors) +- [Summary](#summary) + +--- + ## Overview & Background In pharmaceutical research and drug discovery, predicting the biological activity or potency of chemical compounds (Quantitative Structure-Activity Relationship, or **QSAR**) is a foundational task. @@ -11,6 +29,15 @@ Traditional machine learning regression models optimize for **Mean Squared Error **Quantile Regression** addresses this challenge by estimating conditional percentiles (e.g., 20th percentile, 50th percentile / median, and 80th percentile) of the response distribution. Fitting models across multiple quantiles produces an **uncertainty envelope** (prediction interval) for every candidate compound. This empowers medicinal chemists to quantify risk, prioritize high-confidence candidates, and flag compounds requiring further experimental validation. +## Tested runtime / compatibility matrix + +| Component | Version (tested) | Notes | +|---|---:|---| +| Scala | 2.12.17 | Use the 2.12 SynapseML build (`synapseml_2.12`) | +| Spark | 3.5.0 | Examples were run on Spark 3.5.0 | +| SynapseML | 1.1.3 | Verify runtime supports this coordinate; managed runtimes may have different preinstalled versions | +| Hadoop connector (if using wasbs://) | org.apache.hadoop:hadoop-azure:3.3.4 | Required only for standalone clusters reading wasbs:// blobs | + --- ## Key Syntax Differences: PySpark vs. Spark Scala From 3cd4f15a8e19a4df84baca0ca37b90ba45ecedee Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 20:01:29 +0530 Subject: [PATCH 14/16] docs(lightgbm): unify target column name to pIC50 across examples --- ...e Regression for Drug Discovery (Scala).md | 30 ++++++++++++------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index 3a9d9a6fcf..e937342e94 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -104,6 +104,11 @@ import spark.implicits._ In QSAR modeling, compounds are typically represented by physicochemical descriptors or molecular fingerprints (e.g., Molecular Weight, LogP, Hydrogen Bond Donors/Acceptors, Topological Polar Surface Area, Rotatable Bonds) mapped to a biological potency target (such as $pIC_{50} = -\log_{10}(IC_{50})$). +> Note: this tutorial uses a canonical target column name `pIC50` across examples. +> - Option A (synthetic) uses `pIC50`. +> - Option B (LibSVM) renames the incoming `label` to `pIC50` for consistency. +> - The standalone app uses `pIC50` as the target column too. + ### Option A: Self-Contained Synthetic QSAR Dataset To run this tutorial immediately without external network dependencies, generate a synthetic QSAR dataset: @@ -145,7 +150,7 @@ qsarDf.show(5, truncate = false) ### Option B: Public Triazines Benchmark Dataset (LibSVM) SynapseML also hosts the classic benchmark Triazines QSAR dataset (predicting inhibition of dihydrofolate reductase by pyrimidines). -> LibSVM supplies a `features` vector and a `label` column, so this path does not need `VectorAssembler`. +> LibSVM supplies a `features` vector and a `label` column (renamed to `pIC50` below), so this path does not need `VectorAssembler`. > > Standalone Apache Spark also needs Hadoop's Azure connector to read `wasbs://` URLs. For Spark 3.5.0 with Hadoop 3.3.4, use `--packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4` with the Maven repository from Step 1. On managed clusters, use the connector supplied by the runtime or match the connector to the runtime's Hadoop version. @@ -158,16 +163,19 @@ val triazinesDf = spark.read println(s"Total records in Triazines dataset: ${triazinesDf.count()}") triazinesDf.printSchema() -// Direct training on LibSVM's native columns without VectorAssembler: -val Array(triazinesTrain, triazinesTest) = triazinesDf.randomSplit(Array(0.8, 0.2), seed = 1234L) +// Rename LibSVM's default 'label' column to the canonical target column 'pIC50' +// so column names match the rest of this tutorial +val triazinesDfRenamed = triazinesDf.withColumnRenamed("label", "pIC50") + +val Array(triazinesTrain, triazinesTest) = triazinesDfRenamed.randomSplit(Array(0.8, 0.2), seed = 1234L) val triazinesModel = new LightGBMRegressor() .setObjective("quantile") .setAlpha(0.5) - .setLabelCol("label") + .setLabelCol("pIC50") .setFeaturesCol("features") .fit(triazinesTrain) -triazinesModel.transform(triazinesTest).select("label", "prediction").show(5) +triazinesModel.transform(triazinesTest).select("pIC50", "prediction").show(5) ``` > **Tutorial Flow:** The subsequent sections (Steps 4 through 7) follow **Option A (`qsarDf`)** to demonstrate how to perform custom feature engineering with `VectorAssembler`, multi-quantile uncertainty envelope modeling, and domain-specific bioactivity metric evaluation. @@ -386,9 +394,9 @@ object QSARQuantileApp { val logP = -1.0 + random.nextDouble() * 6.0 val hbd = random.nextInt(5).toDouble val hba = random.nextInt(9).toDouble - val potency = 5.0 + 0.004 * mw + 0.35 * logP - 0.1 * hbd + random.nextGaussian() * 0.25 - (s"MOL_$i", mw, logP, hbd, hba, potency) - }.toDF("id", "mw", "logP", "hbd", "hba", "potency") + val pIC50 = 5.0 + 0.004 * mw + 0.35 * logP - 0.1 * hbd + random.nextGaussian() * 0.25 + (s"MOL_$i", mw, logP, hbd, hba, pIC50) + }.toDF("id", "mw", "logP", "hbd", "hba", "pIC50") // 2. Assemble Features val assembler = new VectorAssembler() @@ -410,7 +418,7 @@ object QSARQuantileApp { val model = new LightGBMRegressor() .setObjective("quantile") .setAlpha(alpha) - .setLabelCol("potency") + .setLabelCol("pIC50") .setFeaturesCol("features") .setPredictionCol(predCol) .setNumLeaves(31) @@ -423,14 +431,14 @@ object QSARQuantileApp { // 4. Compute Metrics val evaluator = new RegressionEvaluator() - .setLabelCol("potency") + .setLabelCol("pIC50") .setPredictionCol("pred_median_50") .setMetricName("rmse") val rmse = evaluator.evaluate(scoredTest) println(f"Median Model RMSE: $rmse%.4f") - scoredTest.select("id", "potency", "pred_lower_10", "pred_median_50", "pred_upper_90") + scoredTest.select("id", "pIC50", "pred_lower_10", "pred_median_50", "pred_upper_90") .show(5, truncate = false) spark.stop() From b10808ef919a11c6fa21421c42ae6cc0bd12ec10 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 20:01:47 +0530 Subject: [PATCH 15/16] docs(lightgbm): remove duplicate crossingCount and use single diagnostic variable --- ...Quantile Regression for Drug Discovery (Scala).md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index e937342e94..fa39ed6023 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -263,10 +263,10 @@ val predictions = modelQ80.transform( // Independent quantile fits do not guarantee monotonic ordering (q20 <= q50 <= q80). // When crossed, subtracting predictions produces negative widths and invalid intervals. // See maintainer explanation in lightgbm-org/LightGBM#3447. -val crossingCount = predictions.filter( +val numCrossedRows = predictions.filter( $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" ).count() -println(s"Rows with crossed quantiles: $crossingCount") +println(s"Rows with crossed quantiles: $numCrossedRows") // Flag crossed quantiles and calculate prediction interval width & coverage val predictionsWithInterval = predictions @@ -325,10 +325,6 @@ println(f"Median Model MAE: $mae%.4f") // reported here so the reader can make an informed judgement. // Taking an absolute value or sorting the bounds is NOT a valid fix: it does // not establish the advertised 60 % nominal coverage. -val crossingCount = predictions.filter( - $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" -).count() - val totalCount = predictionsWithInterval.count() val coverageCount = predictionsWithInterval.filter($"within_interval" === true).count() val empiricalCoverage = (coverageCount.toDouble / totalCount.toDouble) * 100.0 @@ -339,8 +335,8 @@ val validTotal = validRows.count() val validCoverageCount = validRows.filter($"within_interval" === true).count() val validCoverage = if (validTotal > 0) (validCoverageCount.toDouble / validTotal.toDouble) * 100.0 else 0.0 -println(f"Rows with crossed quantiles : $crossingCount (out of $totalCount)") -println(f"Empirical Coverage (all rows) : $empiricalCoverage%.2f%% — includes $crossingCount crossed row(s); interpret with caution") +println(f"Rows with crossed quantiles : $numCrossedRows (out of $totalCount)") +println(f"Empirical Coverage (all rows) : $empiricalCoverage%.2f%% — includes $numCrossedRows crossed row(s); interpret with caution") println(f"Empirical Coverage (valid rows only): $validCoverage%.2f%% (Nominal target: 60.00%%)") // ───────────────────────────────────────────────────────────────────────────── ``` From dd50dce840d402ecbba96e8bdb9ce7d7bdb05887 Mon Sep 17 00:00:00 2001 From: MahendraMula Date: Mon, 7 Sep 2026 20:02:26 +0530 Subject: [PATCH 16/16] docs(lightgbm): add troubleshooting for hadoop-azure and link to LightGBM issue --- ...e Regression for Drug Discovery (Scala).md | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md index fa39ed6023..d8e58de7cf 100644 --- a/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md +++ b/docs/Explore Algorithms/LightGBM/LightGBM - Quantile Regression for Drug Discovery (Scala).md @@ -262,7 +262,7 @@ val predictions = modelQ80.transform( // ── Crossed-Quantile Diagnostic ─────────────────────────────────────────────── // Independent quantile fits do not guarantee monotonic ordering (q20 <= q50 <= q80). // When crossed, subtracting predictions produces negative widths and invalid intervals. -// See maintainer explanation in lightgbm-org/LightGBM#3447. +// See maintainer explanation in [LightGBM issue #3447](https://github.com/LightGBM/LightGBM/issues/3447). val numCrossedRows = predictions.filter( $"pred_q20" > $"pred_q50_median" || $"pred_q50_median" > $"pred_q80" ).count() @@ -318,7 +318,8 @@ println(f"Median Model MAE: $mae%.4f") // 3. Crossed-quantile count reported alongside empirical coverage // ── IMPORTANT ──────────────────────────────────────────────────────────────── // Because each quantile model is trained independently, there is no guarantee -// that q20 ≤ q50 ≤ q80 holds for every compound (see lightgbm-org/LightGBM#3447). +// that q20 ≤ q50 ≤ q80 holds for every compound. See maintainer explanation in +// [LightGBM issue #3447](https://github.com/LightGBM/LightGBM/issues/3447). // Rows where that ordering is violated have a negative uncertainty_width and // must NOT be presented as valid uncertainty intervals. Coverage computed over // all rows (including crossed ones) is therefore misleading — both figures are @@ -468,6 +469,23 @@ spark-submit \ --- +## Troubleshooting & common runtime errors + +- `ClassNotFoundException: org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure` + - Cause: missing Hadoop Azure connector on standalone Spark clusters when reading `wasbs://`. + - Fix: add `org.apache.hadoop:hadoop-azure:3.3.4` to `--packages` (or match your runtime's Hadoop version). + - Example: + ```bash + spark-shell \ + --packages com.microsoft.azure:synapseml_2.12:1.1.3,org.apache.hadoop:hadoop-azure:3.3.4 \ + --repositories https://mmlspark.blob.core.windows.net/maven + ``` + +- Quantile crossing ($q_{20} > q_{50}$ or $q_{50} > q_{80}$): + - Explanation: independent quantile fits can cross because each quantile regression model is trained separately without joint monotonic constraints. See maintainer explanation in [LightGBM issue #3447](https://github.com/LightGBM/LightGBM/issues/3447). + +--- + ## Summary In this guide, you learned how to: @@ -476,5 +494,3 @@ In this guide, you learned how to: 3. Model biological activity ($pIC_{50}$) with Quantile Regression to estimate uncertainty intervals ($q_{20}, q_{50}, q_{80}$). 4. Calculate empirical coverage and evaluate prediction accuracy with Spark ML's `RegressionEvaluator`. 5. Package and submit a standalone Spark Scala LightGBM application using `sbt` and `spark-submit`. - -