Feat: CanonicalDataDownloaderDecorator for chain support (#9299)

* feat: add CanonicalDataDownloaderDecorator for chain support

Introduced CanonicalDataDownloaderDecorator to enable automatic resolution and parallel downloading of all contracts for canonical symbols (options and futures chains). Updated Program.cs and DownloaderDataProvider to use this decorator, ensuring seamless data retrieval for both canonical and non-canonical symbols. Refactored initialization logic to handle chain providers within the decorator and removed redundant code from Program.cs.

* feat: improve canonical symbol error handling in data downloader

* refactor: data downloader selection in DownloaderDataProvider

* refactor: CanonicalDataDownloaderDecorator construction

* feat: limit parallelism and improve contract data error handling

* refactor: canonical data downloader and provider usage

* remove: CanonicalNotSupportedException and refactor handling

* refactor: data downloader selection with selector class

Introduce DataDownloaderSelector to choose the correct IDataDownloader implementation based on data type, using CanonicalDataDownloaderDecorator only when needed. Update Program.cs and DownloaderDataProvider to use the selector, remove redundant initialization logic, and ensure proper resource disposal. This improves flexibility, correctness, and resource management for data downloads, especially for custom and canonical data types.

* fix: wrong selector condition

* refactor: change default log handler; add parallelism config to downloader

- Set ConsoleLogHandler as the default log handler in code and config
- Make contract download parallelism configurable via downloader-thread-count (default 4)
- Track and log number of processed contracts in CanonicalDataDownloaderDecorator
- Add error logging for missing universe data
- Improve logging clarity and code readability

* feat: prevent duplicate contract downloads in canonical chains

Introduce ContractDownloadParameters to uniquely identify contract/tick type/resolution combinations and cache them in CanonicalDataDownloaderDecorator, avoiding redundant downloads when contracts are shared across canonical symbol chains. Add date range optimization for contract downloads and unit tests for the new class.

* Revert "feat: prevent duplicate contract downloads in canonical chains"

This reverts commit 44386c4.

* feat: clamp contract date ranges, deduplicate downloads

Added AdjustDateRangeForContract to CanonicalDataDownloaderDecorator to clamp start/end dates based on contract expiry and security type. Introduced _contractsCache to prevent duplicate downloads across canonical chains and _processedContracts for logging. Updated contract retrieval logic for deduplication. Added NUnit tests to verify date range adjustments for futures and options.

* refactor: canonical contract date range adjustment logic

* test:feat: add test for non-option/future contract date adjustment

* feat: make look-back periods for canonical symbols configurable

* refactor: remove Lazy from CanonicalDataDownloaderDecorator

Revert "refactor: remove Lazy from CanonicalDataDownloaderDecorator"

This reverts commit 909c129.

refactor: Inject IMapFileProvider into downloader selectors

Revert "refactor: Inject IMapFileProvider into downloader selectors"

This reverts commit 0add952.

Reapply "refactor: remove Lazy from CanonicalDataDownloaderDecorator"

This reverts commit 9e3c4be.

* refactor: data downloader dependency injection

* refactor: provider initialization in data downloader classes

* refactor: DataDownloaderSelector dependencies, add tests

- Require explicit IDataProvider in DataDownloaderSelector constructor, removing default and fallback logic
- Change default log handler to ConsoleLogHandler in Program.cs
- Pass DownloaderDataProvider as IDataProvider to DataDownloaderSelector
- Add DataDownloaderSelector unit tests to verify correct downloader selection and decorator usage

* feat: ensure factorFileProvider is initialized when defaulted

* fix: missed dataProvider in DataDownloaderSelector ctor

* refactor: improve IFactorFileProvider resolution in DataDownloaderSelector

Author: Romazes

DownloaderDataProvider/Program.cs

@@ -21,14 +21,14 @@
 using QuantConnect.Interfaces;
 using QuantConnect.Securities;
 using QuantConnect.Configuration;
-using QuantConnect.Lean.Engine.DataFeeds;
 using QuantConnect.Data.UniverseSelection;
 using DataFeeds = QuantConnect.Lean.Engine.DataFeeds;
+using QuantConnect.Lean.Engine.DataFeeds.DataDownloader;
 using QuantConnect.DownloaderDataProvider.Launcher.Models;
 using QuantConnect.DownloaderDataProvider.Launcher.Models.Constants;
-using QuantConnect.Lean.Engine.HistoricalData;
 
 namespace QuantConnect.DownloaderDataProvider.Launcher;
+
 public static class Program
 {
     /// <summary>
@@ -63,30 +63,29 @@ public static void Main(string[] args)
             Config.MergeCommandLineArgumentsWithConfiguration(DownloaderDataProviderArgumentParser.ParseArguments(args));
         }
 
-        InitializeConfigurations();
+        var dataDownloaderSelector = InitializeConfigurations();
 
-        var dataDownloader = Composer.Instance.GetExportedValueByTypeName<IDataDownloader>(Config.Get(DownloaderCommandArguments.CommandDownloaderDataDownloader));
         var commandDataType = Config.Get(DownloaderCommandArguments.CommandDataType).ToUpperInvariant();
 
+        var dataDownloadConfig = default(BaseDataDownloadConfig);
         switch (commandDataType)
         {
             case "UNIVERSE":
-                RunUniverseDownloader(dataDownloader, new DataUniverseDownloadConfig());
+                dataDownloadConfig = new DataUniverseDownloadConfig();
+                RunUniverseDownloader(dataDownloaderSelector.GetDataDownloader(dataDownloadConfig.DataType), dataDownloadConfig);
                 break;
             case "TRADE":
             case "QUOTE":
             case "OPENINTEREST":
-                RunDownload(dataDownloader, new DataDownloadConfig(), Globals.DataFolder, _dataCacheProvider);
+                dataDownloadConfig = new DataDownloadConfig();
+                RunDownload(dataDownloaderSelector.GetDataDownloader(dataDownloadConfig.DataType), dataDownloadConfig, Globals.DataFolder, _dataCacheProvider);
                 break;
             default:
                 Log.Error($"QuantConnect.DownloaderDataProvider.Launcher: Unsupported command data type '{commandDataType}'. Valid options: UNIVERSE, TRADE, QUOTE, OPENINTEREST.");
                 break;
         }
 
-        if (dataDownloader is BrokerageDataDownloader brokerageDataDownloader)
-        {
-            brokerageDataDownloader.DisposeSafely();
-        }
+        dataDownloaderSelector.Dispose();
     }
 
     /// <summary>
@@ -98,7 +97,7 @@ public static void Main(string[] args)
     /// <param name="dataCacheProvider">The provider used to cache history data files</param>
     /// <param name="mapSymbol">True if the symbol should be mapped while writing the data</param>
     /// <exception cref="ArgumentNullException">Thrown when <paramref name="dataDownloader"/> is null.</exception>
-    public static void RunDownload(IDataDownloader dataDownloader, DataDownloadConfig dataDownloadConfig, string dataDirectory, IDataCacheProvider dataCacheProvider, bool mapSymbol = true)
+    public static void RunDownload(IDataDownloader dataDownloader, BaseDataDownloadConfig dataDownloadConfig, string dataDirectory, IDataCacheProvider dataCacheProvider, bool mapSymbol = true)
     {
         if (dataDownloader == null)
         {
@@ -167,7 +166,7 @@ public static void RunDownload(IDataDownloader dataDownloader, DataDownloadConfi
     /// </summary>
     /// <param name="dataDownloader">The data downloader instance.</param>
     /// <param name="dataUniverseDownloadConfig">The universe download configuration.</param>
-    private static void RunUniverseDownloader(IDataDownloader dataDownloader, DataUniverseDownloadConfig dataUniverseDownloadConfig)
+    private static void RunUniverseDownloader(IDataDownloader dataDownloader, BaseDataDownloadConfig dataUniverseDownloadConfig)
     {
         foreach (var symbol in dataUniverseDownloadConfig.Symbols)
         {
@@ -206,28 +205,19 @@ private static (DateTimeZone dataTimeZone, DateTimeZone exchangeTimeZone) GetDat
     /// <seealso cref="IDataProvider"/>
     /// <seealso cref="IMapFileProvider"/>
     /// <seealso cref="IFactorFileProvider"/>
-    public static void InitializeConfigurations()
+    public static DataDownloaderSelector InitializeConfigurations()
     {
         Log.DebuggingEnabled = Config.GetBool("debug-mode", false);
-        Log.LogHandler = Composer.Instance.GetExportedValueByTypeName<ILogHandler>(Config.Get("log-handler", "CompositeLogHandler"));
+        Log.LogHandler = Composer.Instance.GetExportedValueByTypeName<ILogHandler>(Config.Get("log-handler", "ConsoleLogHandler"));
 
         var dataProvider = Composer.Instance.GetExportedValueByTypeName<IDataProvider>("DefaultDataProvider");
         var mapFileProvider = Composer.Instance.GetExportedValueByTypeName<IMapFileProvider>(Config.Get("map-file-provider", "LocalDiskMapFileProvider"));
         var factorFileProvider = Composer.Instance.GetExportedValueByTypeName<IFactorFileProvider>(Config.Get("factor-file-provider", "LocalDiskFactorFileProvider"));
 
-        var optionChainProvider = Composer.Instance.GetPart<IOptionChainProvider>();
-        if (optionChainProvider == null)
-        {
-            var historyManager = Composer.Instance.GetExportedValueByTypeName<HistoryProviderManager>(nameof(HistoryProviderManager));
-            historyManager.Initialize(new HistoryProviderInitializeParameters(null, null, dataProvider, _dataCacheProvider,
-                mapFileProvider, factorFileProvider, _ => { }, false, new DataPermissionManager(), null, new AlgorithmSettings()));
-            var baseOptionChainProvider = new LiveOptionChainProvider();
-            baseOptionChainProvider.Initialize(new(mapFileProvider, historyManager));
-            optionChainProvider = new CachingOptionChainProvider(baseOptionChainProvider);
-            Composer.Instance.AddPart(optionChainProvider);
-        }
-
         mapFileProvider.Initialize(dataProvider);
         factorFileProvider.Initialize(mapFileProvider, dataProvider);
+
+        var dataDownloader = Composer.Instance.GetExportedValueByTypeName<IDataDownloader>(Config.Get(DownloaderCommandArguments.CommandDownloaderDataDownloader));
+        return new DataDownloaderSelector(dataDownloader, mapFileProvider, dataProvider, factorFileProvider);
     }
 }

DownloaderDataProvider/config.example.json

@@ -1,7 +1,7 @@
 {
   "map-file-provider": "LocalDiskMapFileProvider",
   "data-provider": "DefaultDataProvider",
-  "log-handler": "CompositeLogHandler",
+  "log-handler": "ConsoleLogHandler",
   "factor-file-provider": "LocalDiskFactorFileProvider",
 
   // The root directory of the data folder for this application