Archive loop do not send out plots

[koninka]

I'm running plotman in a cycles (plotman plot and plotman archive). So, archiving only works when i'm running interactive mode and it works fine. But when I run the archive cycle, the plots are not sent.

[rafaelsteil]

What are the logs when you run plotman archive? Does it show "Starting archive: rsync [rest of the command]"? If so, what happens if you try to run that rsync command manually? (stop archive first)

[altendky]

@koninka, I'm confused. You run all three of plot, archive, and interactive? But don't leave them running, but rather just run them each briefly and cycle through them? If I am understanding correctly, this isn't really the intended usage.

interactive shows the TUI and launches plotting and archiving processes. plot just launches plotting jobs with a basic log output. archive just launches archiving jobs with a basic log output. All three are intended to be left running as long as you want your plotter operating. Given the overlap in their functionality, you aren't generally expected to run interactive and the others at the same time. Though, odds are you only occasionally get some confusion with double-launched jobs. Also, only either interactive or archive would log the starting of the archival process, except in the case they double launch which we don't really want.

Long term I do intend to have plotman setup for a clean separation of actions by a service and monitor when you are interested. We're not quite there yet.

If this doesn't clarify what is going on, perhaps you could share some more details include commands and outputs and any scripts you use to help me understand what you are doing.

[koninka]

@rafaelsteil @altendky Hey,

Yes, rsync works. Cause I've checked in on interactive mode. For me archive job works only on interactive mode and not in loop plotman archive. When i run the backup loop i see endless messages sleeping.

And I do not run both interactive and plot cycles.

[altendky]

When running both interactive and archive at the same time you have two things that consider starting archiving jobs. It's kind of chance which one chooses to start the next job first.

[koninka]

@altendky when I run the archive loop (no/without interactive mode) rsync doesn't work. I see a lot sleeping messages. That's all (no archive)

[altendky]

Can you share that output and while it is running also collect ps aux | grep -P '(plotman|rsync)', share the full config file, and whatever evidence you think shows it should be starting a new archiving job. Please share the full prompts, commands, and outputs.

[koninka]

@altendky Hey, I ran two loops (plot and archive) and I had 5 plots that were not archived.

In interactive mode, archiving works.

Output of ps aux | grep -P '(plotman|rsync)':

mark      267293  0.1  0.0  51292 12792 pts/0    S+   01:31   1:02 /home/mark/chia-blockchain/venv/bin/python /home/mark/chia-blockchain/venv/bin/plotman plot
mark      267296  0.0  0.0  51144 12364 pts/1    S+   01:31   0:32 /home/mark/chia-blockchain/venv/bin/python /home/mark/chia-blockchain/venv/bin/plotman archive
mark      297221  0.0  0.0  17604  2932 pts/2    S+   14:05   0:00 grep --color=auto -P (plotman|rsync)

Plotman config (I've changed rsync parameters for publishing):

# Default/example plotman.yaml configuration file

# Options for display and rendering
user_interface:
        # Call out to the `stty` program to determine terminal size, instead of
        # relying on what is reported by the curses library.   In some cases,
        # the curses library fails to update on SIGWINCH signals.  If the
        # `plotman interactive` curses interface does not properly adjust when
        # you resize the terminal window, you can try setting this to True. 
        use_stty_size: True

# Where to plot and log.
directories:
        # One directory in which to store all plot job logs (the STDOUT/
        # STDERR of all plot jobs).  In order to monitor progress, plotman
        # reads these logs on a regular basis, so using a fast drive is
        # recommended.
        log: /home/mark/.chia/mainnet/plotter/log

        # One or more directories to use as tmp dirs for plotting.  The
        # scheduler will use all of them and distribute jobs among them.
        # It assumes that IO is independent for each one (i.e., that each
        # one is on a different physical device).
        #
        # If multiple directories share a common prefix, reports will
        # abbreviate and show just the uniquely identifying suffix.
        tmp:
                # - /mnt/plots-temp/ssd2tb/01
                # - /mnt/plots-temp/ssd2tb/02
                # - /mnt/plots-temp/ssd2tb/03
                - /mnt/plots-temp/ssd2tb02
                - /mnt/plots-temp/ssd2tb02/01

                # - /mnt/plots-temp/ssdnvme1tb/01
                # - /mnt/plots-temp/ssdnvme1tb/02
                # - /mnt/plots-temp/ssdnvme1tb/03

        # Optional: Allows overriding some characteristics of certain tmp
        # directories. This contains a map of tmp directory names to
        # attributes. If a tmp directory and attribute is not listed here,
        # it uses the default attribute setting from the main configuration.
        #
        # Currently support override parameters:
        #     - tmpdir_max_jobs
        tmp_overrides:
                # In this example, /mnt/tmp/00 is larger than the other tmp
                # dirs and it can hold more plots than the default.
                "/mnt/tmp/00":
                        tmpdir_max_jobs: 5

        # Optional: tmp2 directory.  If specified, will be passed to
        # chia plots create as -2.  Only one tmp2 directory is supported.
        # tmp2: /mnt/tmp/a

        # One or more directories; the scheduler will use all of them.
        # These again are presumed to be on independent physical devices,
        # so writes (plot jobs) and reads (archivals) can be scheduled
        # to minimize IO contention.
        dst:
                - /mnt/plots-temp/ssd2tb02
                # - /mnt/plots-storage/500gb01

        # Archival configuration.  Optional; if you do not wish to run the
        # archiving operation, comment this section out.
        #
        # Currently archival depends on an rsync daemon running on the remote
        # host, and that the module is configured to match the local path.
        # See code for details.
        archive:
               rsyncd_module: module
               rsyncd_path: /path/to
               rsyncd_bwlimit: 150000  # Bandwidth limit in KB/s
               rsyncd_host: local.rsync
               rsyncd_user: user
                # Optional index.  If omitted or set to 0, plotman will archive
                # to the first archive dir with free space.  If specified,
                # plotman will skip forward up to 'index' drives (if they exist).
                # This can be useful to reduce io contention on a drive on the
                # archive host if you have multiple plotters (simultaneous io
                # can still happen at the time a drive fills up.)  E.g., if you
                # have four plotters, you could set this to 0, 1, 2, and 3, on
                # the 4 machines, or 0, 1, 0, 1.
                #   index: 0


# Plotting scheduling parameters
scheduling:
        # Run a job on a particular temp dir only if the number of existing jobs
        # before tmpdir_stagger_phase_major tmpdir_stagger_phase_minor
        # is less than tmpdir_stagger_phase_limit.
        # Phase major corresponds to the plot phase, phase minor corresponds to
        # the table or table pair in sequence, phase limit corresponds to
        # the number of plots allowed before [phase major, phase minor]
        tmpdir_stagger_phase_major: 2
        tmpdir_stagger_phase_minor: 1
        # Optional: default is 1
        tmpdir_stagger_phase_limit: 1

        # Don't run more than this many jobs at a time on a single temp dir.
        tmpdir_max_jobs: 3

        # Don't run more than this many jobs at a time in total.
        global_max_jobs: 6

        # Don't run any jobs (across  all temp dirs) more often than this, in minutes.
        global_stagger_m: 90

        # How often the daemon wakes to consider starting a new plot job, in seconds.
        polling_time_s: 30


# Plotting parameters.  These are pass-through parameters to chia plots create.
# See documentation at
# https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference#create
plotting:
        k: 32
        e: False             # Use -e plotting option
        n_threads: 8         # Threads per job
        n_buckets: 128       # Number of buckets to split data into
        job_buffer: 5000     # Per job memory
        # If specified, pass through to the -f and -p options.  See CLI reference.
        #   farmer_pk: ...
        #   pool_pk: ...

I do not know what's the problem. I don't like this as I have two plotting computers (ubuntu 20.04) and both have the same problems. I have to open the interactive mode one by one and rsync the plots.

[koninka]

@altendky ping

[altendky]

Sorry, I have no theories yet. Could you install the latest development version and see if it makes any difference? If not we may have to add extra diagnostic code. You can following the regular instructions at https://github.com/ericaltendorf/plotman/tree/1876d4cbda73fb38584f95113a00f01af1e21704#installation but specify @development instead of @main.

[koninka]

@altendky Ok, I'll check and write later about the results. Thank you

[koninka]

@altendky Hey, archive mode work fine on the development branch. And finally I'm happy:)