Add CPU Affinity Group option

.gitignore

@@ -131,3 +131,4 @@ dmypy.json
 # Custom
 config.yaml
 *.bak
+.idea

README.RU.md

@@ -208,6 +208,7 @@ Each job must have unique temporary directories.
 * `max_plots` - Максимальное количество заданий, выполняемых за один запуск менеджера. При любом перезапуске диспетчера эта переменная будет сброшена. Он здесь только для того, чтобы помочь в краткосрочном планировании засева.
 * [ОПЦИЯ]`farmer_public_key` - Ваш публичный ключ фермера. Если не указан, менеджер не будет передавать эту переменную исполняемому файлу chia, что приведет к использованию ваших ключей по умолчанию. Этот параметр необходим только в том случае, если на компьютере нет ваших учетных данных chia.
 * [ОПЦИЯ]`pool_public_key` - Ваш публичный ключ пула. Аналогично как и выше. 
+* [ОПЦИЯ] `pool_contract_address` Your Pool's Contract Address. Refer to https://github.com/Chia-Network/chia-blockchain/wiki/Pooling-FAQ#how-do-i-assign-portable-plots-to-a-pool
 * `temporary_directory` - Временное место для засева. Может иметь одно или несколько значений. Обычно размещается на SSD диске. These directories must be unique from one another.
 * [ОПЦИЯ]`temporary2_directory` - Может иметь одно или несколько значений. Это необязательный параметр для использования второго временного каталога засева полей Chia.
 * `destination_directory` - Может иметь одно или несколько значений. Указывает на финальную директорию куда будет помещено готовое поле. Если вы укажете несколько, готовые поля будут размещаться по одному на каждый следующий диск поочереди.

README.md

@@ -5,7 +5,7 @@
 
 ![The view of the manager](https://i.imgur.com/hIhjXt0.png "View")
 
-##### Development Version: v0.1.0
+##### Development Version: v0.1.1
 
 This is a cross-platform Chia Plot Manager that will work on the major operating systems. This is not a plotter. The purpose of this library is to manage your plotting and kick off new plots with the settings that you configure. Everyone's system is unique so customization is an important feature that was engraved into this library.
 
@@ -21,6 +21,7 @@ This library has been tested for Windows and Linux.
 * Utilize temporary space to its maximum potential by starting a new plot early.
 * Run a maximum number of plots concurrently to avoid bottlenecks or limit resource hogging.
 * More in-depth active plot screen.
+* Support MadMax and the new pooling protocol
 
 
 ## Sponsor / Support this Library
@@ -147,6 +148,10 @@ This is a single variable that should contain the location of your chia executab
 * Linux Example: `/usr/lib/chia-blockchain/resources/app.asar.unpacked/daemon/chia`
 * Another Linux Example: `/home/swar/chia-blockchain/venv/bin/chia`
 
+### backend
+
+This is where you signify if you are using Chia or the MadMax binary files for plotting.
+
 ### manager
 
 These are the config settings that will only be used by the plot manager.
@@ -207,7 +212,8 @@ Check for more details on the Chia CLI here: https://github.com/Chia-Network/chi
 * `name` - This is the name that you want to give to the job.
 * `max_plots` - This is the maximum number of jobs to make in one run of the manager. Any restarts to manager will reset this variable. It is only here to help with short term plotting.
 * [OPTIONAL]`farmer_public_key` - Your farmer public key. If none is provided, it will not pass in this variable to the chia executable which results in your default keys being used. This is only needed if you have chia set up on a machine that does not have your credentials.
-* [OPTIONAL]`pool_public_key` - Your pool public key. Same information as the above. 
+* [OPTIONAL] pool_public_key: Your pool public key. DONT Use IF YOU ARE USING A POOL CONTRACT ADDRESS! Othewise same information as above.
+* [OPTIONAL] pool_contract_address:  Your Pool's Contract Address. Refer to https://github.com/Chia-Network/chia-blockchain/wiki/Pooling-FAQ#how-do-i-assign-portable-plots-to-a-pool
 * `temporary_directory` - Can be a single value or a list of values. This is where the plotting will take place. If you provide a list, it will cycle through each drive one by one. These directories must be unique from one another.
 * [OPTIONAL]`temporary2_directory` - Can be a single value or a list of values. This is an optional parameter to use in case you want to use the temporary2 directory functionality of Chia plotting.
 * `destination_directory` - Can be a single value or a list of values. This is the final directory where the plot will be transferred once it is completed. If you provide a list, it will cycle through each drive one by one.  
@@ -234,4 +240,4 @@ Check for more details on the Chia CLI here: https://github.com/Chia-Network/chi
 	* 128   `HIGH_PRIORITY_CLASS`
 	* 256   `REALTIME_PRIORITY_CLASS`
 * `enable_cpu_affinity` - Enable or disable cpu affinity for plot processes. Systems that plot and harvest may see improved harvester or node performance when excluding one or two threads for plotting process.
-* `cpu_affinity` - List of cpu (or threads) to allocate for plot processes. The default example assumes you have a hyper-threaded 4 core CPU (8 logical cores). This config will restrict plot processes to use logical cores 0-5, leaving logical cores 6 and 7 for other processes (6 restricted, 2 free).
+* `cpu_affinity` - List of cpu (or threads) to allocate for plot processes. The default example assumes you have a hyper-threaded 4 core CPU (8 logical cores). This config will restrict plot processes to use logical cores 0-5, leaving logical cores 6 and 7 for other processes (6 restricted, 2 free). You have to specify all logical cores/threads that you wanted to use in this format [ 0, 1, 2, 3 ] and NOT [ 0-3 ] if you intend to use the first 4 logical cores.

config.yaml.default

@@ -6,6 +6,11 @@
 #  MAC OS EXAMPLE: /Applications/Chia.app/Contents/Resources/app.asar.unpacked/daemon/chia
 chia_location:
 
+# This variable defines which plotter binary you're using. It can be one of the following options:
+#
+# - chia (default if none provided)
+# - madmax
+backend: chia
 
 manager:
   # These are the config settings that will only be used by the plot manager.
@@ -19,7 +24,7 @@ manager:
 
 log:
   # folder_path: This is the folder where your log files for plots will be saved.
-  folder_path: S:\Chia\Logs\Plotter
+  folder_path: C:\Chia\Logs\Plotter
 
 
 view:
@@ -86,10 +91,20 @@ progress:
   #                 information with the existing log file to calculate a progress percent.
   #   phase_weight: These are the weight to assign to each phase in the progress calculations. Typically, Phase 1 and 3
   #                 are the longest phases so they will hold more weight than the others.
-  phase1_line_end: 801
-  phase2_line_end: 834
-  phase3_line_end: 2474
-  phase4_line_end: 2620
+  #
+  # If you're using "madmax" backend, you should use the following config instead:
+  # phase1_line_end: 20
+  # phase2_line_end: 34
+  # phase3_line_end: 48
+  # phase4_line_end: 53
+  # phase1_weight: 33.4
+  # phase2_weight: 20.43
+  # phase3_weight: 42.29
+  # phase4_weight: 3.88
+  phase1_line_end: 803
+  phase2_line_end: 836
+  phase3_line_end: 2476
+  phase4_line_end: 2622
   phase1_weight: 33.4
   phase2_weight: 20.43
   phase3_weight: 42.29
@@ -124,7 +139,9 @@ jobs:
   # [OPTIONAL] farmer_public_key: Your farmer public key. If none is provided, it will not pass in this variable to the
   #                               chia executable which results in your default keys being used. This is only needed if
   #                               you have chia set up on a machine that does not have your credentials.
-  # [OPTIONAL] pool_public_key: Your pool public key. Same information as the above.
+  # [OPTIONAL] pool_public_key: Your pool public key. DONT Use IF YOU ARE USING A POOL CONTRACT ADDRESS! Othewise same information as above.
+  #
+  # [OPTIONAL] pool_contract_address:  Your Pool's Contract Address. Refer to https://github.com/Chia-Network/chia-blockchain/wiki/Pooling-FAQ#how-do-i-assign-portable-plots-to-a-pool
   #
   # temporary_directory: Can be a single value or a list of values. This is where the plotting will take place. If you
   #                      provide a list, it will cycle through each drive one by one.
@@ -133,12 +150,16 @@ jobs:
   # destination_directory: Can be a single value or a list of values. This is the final directory where the plot will be
   #                        transferred once it is completed. If you provide a list, it will cycle through each drive
   #                        one by one.
+  #    *** MadMax requires trailing '/' on directories! ***
   #
   # size: This refers to the k size of the plot. You would type in something like 32, 33, 34, 35... in here.
   # bitfield: This refers to whether you want to use bitfield or not in your plotting. Typically, you want to keep
   #           this as true.
-  # threads: This is the number of threads that will be assigned to the plotter. Only phase 1 uses more than 1 thread.
-  # buckets: The number of buckets to use. The default provided by Chia is 128.
+  # threads: This is the number of threads that will be assigned to the plotter. Chia only uses more than 1 thread in phase 1.
+  #           Madmax is multithreaded.
+  # [OPTIONAL] threadX_p2: [madmax only] This is a *MULTIPLIER* (integer) for the number of threads the madmax plotter will use for phase 2. Blank defaults to 1.
+  # buckets: The number of buckets to use. The default provided by Chia is 128. (applies to phases 1 and 2 in madmax when used in conjunction with 'buckets_p3' parameter)
+  # [OPTIONAL] buckets_p3: [madmax only] The number of buckets to use in Phases 3 (and 4). Leave blank to use the same value for all phases (from 'buckets' parameter, default 128)
   # memory_buffer: The amount of memory you want to allocate to the process.
   # max_concurrent: The maximum number of plots to have for this job at any given time.
   # max_concurrent_with_start_early: The maximum number of plots to have for this job at any given time including
@@ -184,7 +205,9 @@ jobs:
     size: 32
     bitfield: true
     threads: 8
+    threadX_p2:
     buckets: 128
+    buckets_p3:
     memory_buffer: 4000
     max_concurrent: 6
     max_concurrent_with_start_early: 7

plotmanager/library/commands/plots.py

@@ -1,14 +1,55 @@
-def create(size, memory_buffer, temporary_directory, destination_directory, threads, buckets, bitfield,
-           chia_location='chia', temporary2_directory=None, farmer_public_key=None, pool_public_key=None,
-           exclude_final_directory=False):
+import logging
+
+
+def create(*args, chia_location='chia', backend='chia', **kwargs):
+    backend_command = _get_backend_command(backend, chia_location)
+    backend_flags = _get_backend_flags(backend, *args, **kwargs)
+
+    for key, value in backend_flags.items():
+        flag = f'-{key}'
+        backend_command.append(flag)
+        if value == '':
+            continue
+        backend_command.append(str(value))
+
+    return backend_command
+
+
+def _get_backend_command(backend, chia_location):
+    logging.debug(f'Parsing backend commands for {backend}')
+
+    backend_commands = dict(
+        chia=[chia_location, 'plots', 'create'],
+        madmax=[chia_location],
+    )
+
+    return backend_commands.get(backend)
+
+
+def _get_backend_flags(backend, *args, **kwargs):
+    logging.debug(f'Parsing backend flags for {backend}')
+
+    backend_parsers = dict(
+        chia=_get_chia_flags,
+        madmax=_get_madmax_flags,
+    )
+    backend_parser = backend_parsers.get(backend)
+
+    return backend_parser(*args, **kwargs)
+
+
+def _get_chia_flags(size, memory_buffer, temporary_directory, destination_directory, threads, buckets, bitfield,
+                    temporary2_directory=None, farmer_public_key=None, pool_public_key=None,
+                    exclude_final_directory=False, pool_contract_address=None, **kwargs):
     flags = dict(
         k=size,
         b=memory_buffer,
         t=temporary_directory,
         d=destination_directory,
         r=threads,
-        u=buckets,
+        u=buckets,       
     )
+
     if temporary2_directory is not None:
         flags['2'] = temporary2_directory
     if farmer_public_key is not None:
@@ -19,12 +60,32 @@ def create(size, memory_buffer, temporary_directory, destination_directory, thre
         flags['e'] = ''
     if exclude_final_directory:
         flags['x'] = ''
+    if pool_contract_address is not None:
+        flags['c'] = pool_contract_address
 
-    data = [chia_location, 'plots', 'create']
-    for key, value in flags.items():
-        flag = f'-{key}'
-        data.append(flag)
-        if value == '':
-            continue
-        data.append(str(value))
-    return data
+    return flags
+
+
+def _get_madmax_flags(temporary_directory, destination_directory, threads, buckets,
+                      buckets_p3=None, threadX_p2=None, temporary2_directory=None, farmer_public_key=None, pool_public_key=None, pool_contract_address=None, **kwargs):
+    flags = dict(
+        r=threads,
+        t=temporary_directory,
+        d=destination_directory,
+        u=buckets,
+    )
+
+    if temporary2_directory is not None:
+        flags['2'] = temporary2_directory
+    if farmer_public_key is not None:
+        flags['f'] = farmer_public_key
+    if pool_public_key is not None:
+        flags['p'] = pool_public_key
+    if pool_contract_address is not None:
+        flags['c'] = pool_contract_address
+    if buckets_p3 is not None:
+        flags['v'] = buckets_p3
+    if threadX_p2 is not None:
+        flags['K'] = threadX_p2
+
+    return flags

plotmanager/library/parse/configuration.py

@@ -134,9 +134,14 @@ def _check_parameters(parameter, expected_parameters, parameter_type):
                                          f'{", ".join(failed_checks)}')
 
 
+def _get_backend_settings(config):
+    return config.get('backend', 'chia')
+
+
 def get_config_info():
     config = _get_config()
     chia_location = _get_chia_location(config=config)
+    backend = _get_backend_settings(config=config)
     manager_check_interval, log_level = _get_manager_settings(config=config)
     log_directory = _get_log_settings(config=config)
     if not os.path.exists(log_directory):
@@ -150,4 +155,4 @@ def get_config_info():
 
     return chia_location, log_directory, jobs, manager_check_interval, max_concurrent, max_for_phase_1, \
         minimum_minutes_between_jobs, progress_settings, notification_settings, log_level, view_settings, \
-        instrumentation_settings
+        instrumentation_settings, backend

plotmanager/library/utilities/commands.py

@@ -32,7 +32,7 @@ def start_manager():
 
     chia_location, log_directory, config_jobs, manager_check_interval, max_concurrent, max_for_phase_1, \
         minimum_minutes_between_jobs, progress_settings, notification_settings, debug_level, view_settings, \
-        instrumentation_settings = get_config_info()
+        instrumentation_settings, backend = get_config_info()
 
     load_jobs(config_jobs)
 
@@ -80,7 +80,7 @@ def stop_manager():
 def json_output():
     chia_location, log_directory, config_jobs, manager_check_interval, max_concurrent, max_for_phase_1, \
         minimum_minutes_between_jobs, progress_settings, notification_settings, debug_level, view_settings, \
-        instrumentation_settings = get_config_info()
+        instrumentation_settings, backend = get_config_info()
 
     system_drives = get_system_drives()
 
@@ -107,11 +107,11 @@ def json_output():
 
     jobs = load_jobs(config_jobs)
     jobs, running_work = get_running_plots(jobs=jobs, running_work=running_work,
-                                           instrumentation_settings=instrumentation_settings)
+                                           instrumentation_settings=instrumentation_settings, backend=backend)
     check_log_progress(jobs=jobs, running_work=running_work, progress_settings=progress_settings,
                        notification_settings=notification_settings, view_settings=view_settings,
-                       instrumentation_settings=instrumentation_settings)
-    print_json(jobs=jobs, running_work=running_work, view_settings=view_settings)
+                       instrumentation_settings=instrumentation_settings, backend=backend)
+    print_json(jobs=jobs, running_work=running_work, view_settings=view_settings, backend=backend)
 
     has_file = False
     if len(running_work.values()) == 0:
@@ -131,7 +131,7 @@ def json_output():
 def view(loop=True):
     chia_location, log_directory, config_jobs, manager_check_interval, max_concurrent, max_for_phase_1, \
         minimum_minutes_between_jobs, progress_settings, notification_settings, debug_level, view_settings, \
-        instrumentation_settings = get_config_info()
+        instrumentation_settings, backend = get_config_info()
     view_check_interval = view_settings['check_interval']
     system_drives = get_system_drives()
     analysis = {'files': {}}
@@ -160,13 +160,13 @@ def view(loop=True):
             analysis = analyze_log_dates(log_directory=log_directory, analysis=analysis)
             jobs = load_jobs(config_jobs)
             jobs, running_work = get_running_plots(jobs=jobs, running_work=running_work,
-                                                   instrumentation_settings=instrumentation_settings)
+                                                   instrumentation_settings=instrumentation_settings, backend=backend)
             check_log_progress(jobs=jobs, running_work=running_work, progress_settings=progress_settings,
                                notification_settings=notification_settings, view_settings=view_settings,
-                               instrumentation_settings=instrumentation_settings)
+                               instrumentation_settings=instrumentation_settings, backend=backend)
             print_view(jobs=jobs, running_work=running_work, analysis=analysis, drives=drives,
                        next_log_check=datetime.now() + timedelta(seconds=view_check_interval),
-                       view_settings=view_settings, loop=loop)
+                       view_settings=view_settings, loop=loop, backend=backend)
             if not loop:
                 break
             time.sleep(view_check_interval)
@@ -190,5 +190,5 @@ def view(loop=True):
 def analyze_logs():
     chia_location, log_directory, config_jobs, manager_check_interval, max_concurrent, max_for_phase_1, \
         minimum_minutes_between_jobs, progress_settings, notification_settings, debug_level, view_settings, \
-        instrumentation_settings = get_config_info()
-    analyze_log_times(log_directory)
+        instrumentation_settings, backend = get_config_info()
+    analyze_log_times(log_directory, backend)