Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 30 additions & 18 deletions docs/source/pipelines/yaml.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,62 +10,74 @@ HTTomo mainly targets GPU computations, therefore the use of :ref:`tutorials_pl_
preferable. However, when the GPU device is not available or a GPU method is not implemented, the use of
:ref:`tutorials_pl_templates_cpu` is possible.

.. note:: The combination of both GPU and CPU methods is possible. If one expects to achieve the faster performance, please use the GPU methods provided, where possible.

.. _tutorials_pl_templates_gpu:

GPU Pipeline templates
-----------------------
Pipelines using HTTomo libraries
--------------------------------

The GPU-pipelines consist of methods from httomolibgpu (GPU) and httomolib (CPU) backend :ref:`backends_list`. Those libraries are supported directly by the HTTomo development team.
Those pipelines consist of methods from HTTomolibgpu (GPU) and HTTomolib (CPU) backends :ref:`backends_list`. Those libraries are supported directly by the HTTomo development team and pipelines are built in computationally efficient way.

.. dropdown:: GPU pipeline with auto-centering and the `FBP3d_tomobar` reconstruction method.
.. dropdown:: Using :code:`find_center_vo` auto-centering and :code:`FBP3d_tomobar` reconstruction method, then save the result into images.

.. literalinclude:: ../pipelines_full/FBP3d_tomobar.yaml
:language: yaml

.. dropdown:: GPU pipeline as above and Total Variation denoising on the result of the FBP reconstruction.
.. dropdown:: Using :code:`find_center_pc` auto-centering, FBP reconstruction and downsampling the result before saving the images.

.. literalinclude:: ../pipelines_full/titaren_center_pc_FBP3d_resample.yaml
:language: yaml

.. dropdown:: Using :code:`find_center_pc` auto-centering, FBP reconstruction and downsampling the result before saving the images.

.. literalinclude:: ../pipelines_full/titaren_center_pc_FBP3d_resample.yaml
:language: yaml

.. dropdown:: Using :code:`LPRec3d_tomobar` reconstruction, which is the fastest from all available reconstruction methods.

.. literalinclude:: ../pipelines_full/LPRec3d_tomobar.yaml
:language: yaml

.. dropdown:: Applying Total Variation denoising :code:`total_variation_PD` to the result of the FBP reconstruction.

.. literalinclude:: ../pipelines_full/FBP3d_tomobar_denoising.yaml
:language: yaml

.. _tutorials_pl_templates_cpu:

CPU Pipeline templates
-----------------------
Pipelines using TomoPy library
------------------------------

The CPU-pipelines mostly use TomoPy methods. They are executed solely on the CPU and therefore expected to be slower than the GPU pipelines.
One can build CPU-only pipelines by using mostly TomoPy methods. They are expected to be slower than the pipelines above.

.. dropdown:: CPU pipeline using auto-centering and the gridrec reconstruction method on the CPU (TomoPy).
.. dropdown:: CPU pipeline using auto-centering and the gridrec reconstruction method from TomoPy.

.. literalinclude:: ../pipelines_full/tomopy_gridrec.yaml
:language: yaml


.. _tutorials_pl_templates_dls:

DLS-specific templates
DLS-specific pipelines
----------------------

These pipelines are specific to Diamond Light Source processing strategies and can vary between different tomographic beamlines.

.. dropdown:: Pipeline for 360-degrees data with automatic CoR/overlap finding and stitching to 180-degrees data. Paganin filter is applied to the data.
.. dropdown:: Reconstructing 360-degrees data with automatic CoR/overlap finding and stitching to 180-degrees data. Paganin filter is applied to the data.

.. literalinclude:: ../pipelines_full/deg360_paganin_FBP3d_tomobar.yaml
:language: yaml

.. dropdown:: Using a distortion correction module as a part of the pipeline with the 360-degrees data.
.. dropdown:: Using distortion correction module as a part of the pipeline with 360-degrees data.

.. literalinclude:: ../pipelines_full/deg360_distortion_FBP3d_tomobar.yaml
:language: yaml

.. _tutorials_pl_templates_sweeps:

Parameter Sweeps templates
--------------------------
Pipelines with parameter sweeps
-------------------------------

Here we demonstrate how to perform a sweep across multiple values of a
single parameter (see :ref:`parameter_sweeping` for more details).
Here we demonstrate how to perform a sweep across multiple values of a single parameter (see :ref:`parameter_sweeping` for more details).

.. note:: There is no need to add image saving plugin for sweep runs as it will be added automatically. It is also preferable to keep the `preview` small as the time of computation can be substantial.

Expand Down
11 changes: 11 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,11 @@ def FBP3d_tomobar_denoising():
return "docs/source/pipelines_full/FBP3d_tomobar_denoising.yaml"


@pytest.fixture
def titaren_center_pc_FBP3d_resample():
return "docs/source/pipelines_full/titaren_center_pc_FBP3d_resample.yaml"


@pytest.fixture
def deg360_paganin_FBP3d_tomobar():
return "docs/source/pipelines_full/deg360_paganin_FBP3d_tomobar.yaml"
Expand Down Expand Up @@ -378,6 +383,12 @@ def pipeline_paganin_sweep_recon_images_i12_119647_tiffs():
return "tests/test_data/raw_data/i12/sweep/images_sweep_FBP3d_tomobar16bit_tif/"


@pytest.fixture
def pipeline_parallel_titaren_center_pc_FBP3d_resample_i12_119647_tiffs():
# 200 downsampled to 512 x 512 tiff files from reconstruction of i12_119647
return "tests/test_data/raw_data/i12/tiffs/images8bit_tif/"


# ---------------------END------------------------#

# TODO: deprecate when loader is generalised
Expand Down
53 changes: 53 additions & 0 deletions tests/test_parallel_pipeline_big.py
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,59 @@ def test_parallel_pipe_360deg_distortion_FBP3d_tomobar_i13_179623_preview(
assert res_norm < 1e-4


# %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%5
@pytest.mark.full_data_parallel
def test_parallel_pipe_titaren_center_pc_FBP3d_resample_i12_119647_preview(
get_files: Callable,
cmd_mpirun,
i12_119647,
titaren_center_pc_FBP3d_resample,
pipeline_parallel_titaren_center_pc_FBP3d_resample_i12_119647_tiffs,
output_folder,
):
change_value_parameters_method_pipeline(
titaren_center_pc_FBP3d_resample,
method=[
"standard_tomo",
"data_resampler",
"remove_stripe_ti",
],
key=[
"preview",
"newshape",
"beta",
],
value=[
{"detector_y": {"start": 900, "stop": 1100}},
[512, 512],
0.01,
],
)

cmd_mpirun.insert(9, i12_119647)
cmd_mpirun.insert(10, titaren_center_pc_FBP3d_resample)
cmd_mpirun.insert(11, output_folder)

process = Popen(
cmd_mpirun, env=os.environ, shell=False, stdin=PIPE, stdout=PIPE, stderr=PIPE
)
output, error = process.communicate()
print(output)

files = get_files(output_folder)
files_references = get_files(
pipeline_parallel_titaren_center_pc_FBP3d_resample_i12_119647_tiffs
)

# recurse through output_dir and check that all files are there
files = get_files(output_folder)
assert len(files) == 204

#: check the number of the resulting tif files
check_tif(files, 200, (512, 512))
compare_tif(files, files_references)


# ########################################################################
@pytest.mark.full_data_parallel
def test_parallel_pipe_sweep_FBP3d_tomobar_i13_177906(
Expand Down
Loading