Encountered an error while using MultiApp on the Alliance Canada cluster

[mahdipasaie]

Hi ,

I am currently running a multi-app simulation on the Alliance Canada cluster. The setup involves one application solving the Navier-Stokes equations and another solving the phase-field equation.

Initially, I attempted to follow the this but I encountered errors during the process. Since Docker and Conda are not permitted on the cluster, my only viable alternative was to use Apptainer. When attempting to convert a Docker image to an Apptainer container, I ran into a similar issue as described in here(#27377) I also considered installing Conda inside an Apptainer container and using it to install MOOSE, but that approach was equally challenging.

Ultimately, I sought assistance from cluster support. They installed MOOSE via Apptainer for me, and I now source MOOSE following this . However, when running my simulation, I encountered the following error:

---

Error Output:

filesystem error: cannot check if file is empty: No such file or directory [PF_out_cp/1048-restart-0.rd]
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor
srun: error: nc31413: task 24: Exited with exit code 1
libMesh terminating:
filesystem error: cannot check if file is empty: No such file or directory [PF_out_cp/1573-restart-0.rd]
...

This error appears to be related to file system issues with the restart files.

---

Additional Context:

Here is my working directory and its contents:

[pasha77@narval3 ghosh2]$ ls
ghosh_test.err  NS.i       simulation.e       simulation.e-s004  simulation.e-s007  simulation.e-s010  simulation.e-s013  simulation.e-s016  simulation.e-s019
ghosh_test.out  PF.i       simulation.e-s002  simulation.e-s005  simulation.e-s008  simulation.e-s011  simulation.e-s014  simulation.e-s017  simulation.e-s020
job.sh          PF_out_cp  simulation.e-s003  simulation.e-s006  simulation.e-s009  simulation.e-s012  simulation.e-s015  simulation.e-s018
[pasha77@narval3 ghosh2]$ cd PF_out_cp/
[pasha77@narval3 PF_out_cp]$ ls
0522-mesh.cpa.gz   0530-restart-0.rd  0768-mesh.cpa.gz   0775-restart-0.rd  1047-mesh.cpa.gz   1585-restart-0.rd  4420-mesh.cpa.gz   9046-restart-0.rd  9260-mesh.cpa.gz   9371-restart-0.rd
0522-restart-0.rd  0533-mesh.cpa.gz   0768-restart-0.rd  0777-mesh.cpa.gz   1047-restart-0.rd  1828-mesh.cpa.gz   4420-restart-0.rd  9110-mesh.cpa.gz   9260-restart-0.rd  9462-mesh.cpa.gz
0525-mesh.cpa.gz   0533-restart-0.rd  0770-mesh.cpa.gz   0777-restart-0.rd  1293-mesh.cpa.gz   1828-restart-0.rd  7611-mesh.cpa.gz   9110-restart-0.rd  9292-mesh.cpa.gz   9462-restart-0.rd
0525-restart-0.rd  0763-mesh.cpa.gz   0770-restart-0.rd  0779-mesh.cpa.gz   1293-restart-0.rd  2099-mesh.cpa.gz   7611-restart-0.rd  9126-mesh.cpa.gz   9292-restart-0.rd  9496-mesh.cpa.gz
0527-mesh.cpa.gz   0763-restart-0.rd  0774-mesh.cpa.gz   0779-restart-0.rd  1310-mesh.cpa.gz   2099-restart-0.rd  7873-mesh.cpa.gz   9126-restart-0.rd  9353-mesh.cpa.gz   9496-restart-0.rd
0527-restart-0.rd  0765-mesh.cpa.gz   0774-restart-0.rd  0783-mesh.cpa.gz   1310-restart-0.rd  4158-mesh.cpa.gz   7873-restart-0.rd  9226-mesh.cpa.gz   9353-restart-0.rd
0530-mesh.cpa.gz   0765-restart-0.rd  0775-mesh.cpa.gz   0783-restart-0.rd  1585-mesh.cpa.gz   4158-restart-0.rd  9046-mesh.cpa.gz   9226-restart-0.rd  9371-mesh.cpa.gz
[pasha77@narval3 PF_out_cp]$ ls

Given these circumstances, could you confirm if the error is likely due to the Apptainer installation or another aspect of the setup? I would greatly appreciate your insights or guidance to resolve this issue.

Thank you for your time and support.

Best regards,
Mahdi

[GiudGiud]

Hello

Can you find this file?

filesystem error: cannot check if file is empty: No such file or directory [PF_out_cp/1048-restart-0.rd]

Same for the other file.

It's very odd that it's look for restart files at different time steps at the same time?

[GiudGiud]

could you confirm if the error is likely due to the Apptainer installation or another aspect of the setup

Does it run in a regular setup? Apptainer works fine for us on INL HPC

[mahdipasaie]

I checked on my laptop, and it was running, So I think it should be an installation issue. Also, the support mentions that it doesn't seem very easy to build Moose from the source that is why they send me an Apptainer image.

Also this is the files I am using:

Click to expand the code for File1.i

[Mesh]
    type = GeneratedMesh
    dim = 2
    nx = 25
    ny = 100 # 
    xmin = 0.0
    xmax = 500.0
    ymin = 0.0
    ymax = 2000.0
[]

[Variables]
    # Variables for the two equations
    [./phi]
        family = LAGRANGE
        order = FIRST
    [../]
  
    [./u]
        family = LAGRANGE
        order = FIRST
    [../]
[]

[AuxVariables]
    [./ux_op]
    [../]
    [./uy_op]
    [../]
[]

[Kernels]
    #phi
    [./phi_time_derivative]
        type = ADTimeop
        variable = phi
    [../]
    [./phi_terms1]
        type = ADkarma1
        variable = phi
    [../]
    [./phi_src]
        type = ADsrc
        variable = phi
        UH = u
    [../]
    # U
    [./U_time_derivative]
        type = E1
        variable = u
        op = phi
    [../]
    [./Div_term]
        type = E2
        variable = u
        op = phi
    [../]
    [./u_term_3]
        type = E3
        variable = u
        op = phi
        
    [../]
[]

[ICs]
    [./phiIC]
        type = init  
        variable = phi
        dy = 0.5                 
        y_solid = 5.0           # Height of the solid region at the bottom
        amplitude = 1.0          # Amplitude of the sinusoidal perturbation
        wavelength = 4.0         # Wavelength of the sinusoidal perturbation
    [../]
    [./uIC]
        type = ConstantIC
        variable = u
        value = -1
    [../]
[]

[Materials]
    [./Theta]
        type = ADtheta
        op = phi
    [../]
[]

[Adaptivity]
    initial_steps = 5
    initial_marker = refine_region
    max_h_level = 5
    marker = err_phi
    interval = 10

    [./Markers]
        [./err_phi]
            type = ErrorFractionMarker
            coarsen = 0.3
            refine = 0.95
            indicator = ind_phi
        [../]
        [./refine_region]
            type = BoxMarker
            bottom_left = '0 0 0'  
            top_right = '600 40 0'  # Set top_y to match y_solid
            inside = refine         
            outside = do_nothing  
        [../]
    [../]
    [./Indicators]
        [./ind_phi]
            type = GradientJumpIndicator
            variable = phi
        [../]
    [../]
[]

[Preconditioning]
    [./advanced_smp_precond]
        type = SMP
        full = true
    [../]
[]

[Executioner]
    type = Transient
    solve_type = PJFNK                     # Krylov-Newton approach for nonlinear solves
    nl_abs_tol = 1e-5                      # Absolute tolerance for nonlinear solver
    nl_rel_tol = 1e-4                      # Relative tolerance for nonlinear convergence
    l_max_its = 200                        # Maximum linear solver iterations
    nl_max_its = 60                        # Maximum nonlinear solver iterations
    petsc_options_iname = '-ksp_type -pc_type -pc_hypre_type -ksp_gmres_restart -pc_hypre_boomeramg_strong_threshold'
    petsc_options_value = 'gmres hypre boomeramg 30 0.7'
    dt = 0.13                               # Initial time step for stability
    start_time = 0                         # Start time
    end_time = 75000                       # End time for simulation
    automatic_scaling = true               # Enable automatic scaling for better conditioning
[]

[MultiApps]
    [./fluid]
      type = TransientMultiApp
      positions = '0 0 0'
      input_files = 'NS.i'
    [../]
[]

[Transfers]
    [./utrans]
      type = MultiAppGeometricInterpolationTransfer
      from_multi_app = fluid
      source_variable = ux
      variable = ux_op
    [../]
    [./vtrans]
        type = MultiAppGeometricInterpolationTransfer
        from_multi_app = fluid
        source_variable = uy
        variable = uy_op
    [../]
    [./Ttrans]
      type = MultiAppGeometricInterpolationTransfer
      to_multi_app = fluid
      source_variable = phi
      variable = phif 
    [../]
[]


[Outputs]
    [./exodus]
      type = Exodus
      interval = 100     
      file_base = simulation     
      execute_on = 'initial timestep_end' 
    [../]
[]

Click to expand the code for File2.i

[Mesh]
    [gen]
    type = GeneratedMeshGenerator
    dim = 2
    nx = 25
    ny = 100 
    xmin = -100.0
    xmax = 600.0
    ymin = 0.0
    ymax = 2000.0
    elem_type = QUAD8
    [../]
    [./corner_node]
        type = ExtraNodesetGenerator
        new_boundary = 'pinned_node'
        nodes = '0'
        input = gen
    [../]
[../]

[Variables]
    [./ux]
        family = LAGRANGE
        order = SECOND
    [../]
    [./uy]
        family = LAGRANGE
        order = SECOND
    [../]
    [./p]
        family = LAGRANGE
        order = FIRST
    [../]
[]

[AuxVariables]
    [./phif]
        family = LAGRANGE
        order = FIRST
    [../]
[]

[Materials]
    [./rho]
      type = ADGenericConstantMaterial
      prop_names = 'rho'
      prop_values = '1' 
    [../]
    [./mu]
        type = ADGenericConstantMaterial
        prop_names = 'mu'
        prop_values = '0.001' 
    [../]
[]

[Kernels]
    [./T_x] 
        type = CoefTimeDerivative
        variable = ux
        Coefficient = 1
    [../]
    [./T_y] 
        type = CoefTimeDerivative
        variable = uy
        Coefficient = 1
    [../]
    [./vis_x]
        type = f1
        variable = ux
        p = p
        rho = rho
        mu = mu 
    [../]
    [./vis_y]
        type = f2
        variable = uy
        p = p
        rho = rho
        mu = mu 
    [../]
    [./adv_x] 
        type = f3
        variable = ux
        uy = uy
    [../]
    [./adv_y] 
        type = f4
        variable = uy
        ux = ux
    [../]
    [./cont]
        type = f5
        variable = p
        ux = ux
        uy = uy
    [../] 
    [./p_x] 
        type = f6
        variable = ux
        p = p
    [../] 
    [./p_y] 
        type = f7
        variable = uy
        p = p
    [../] 
    [./penalty_x]
        type = f8
        variable = ux
        uy = uy
        op = phif
        penalty = 1e6
    [../]
    [./penalty_y]
        type = f9
        variable = uy
        ux = ux
        op = phif
        penalty = 1e6
    [../]
[]

[BCs]
    [./left_x]
        type = DirichletBC
        boundary = 'left'
        variable = ux
        value = 1.0
    [../]
    [./left_y]
        type = DirichletBC
        boundary = 'left'
        variable = uy
        value = 0.0
    [../]
    [./right_p]
        type = DirichletBC
        boundary = 'right'
        variable = p
        value = 0.0
    [../]
    [./bottom_x]
        type = DirichletBC
        boundary = 'bottom'
        variable = ux
        value = 0.0
    [../]
    [./bottom_y]
        type = DirichletBC
        boundary = 'bottom'
        variable = uy
        value = 0.0
    [../]
    [./lid_x]
        type = DirichletBC
        boundary = 'top'
        variable = ux
        value = 0
    [../]
    [./lid_y]
        type = DirichletBC
        boundary = 'top'
        variable = uy
        value = 0.0
    [../]
[]

[ICs]
    [ux_ic]
      type = FunctionIC
      variable = 'ux'
      function = zerofunc
    []
    [uy_ic]
        type = FunctionIC
        variable = 'uy'
        function = zerofunc
    []
    [p_ic]
        type = FunctionIC
        variable = 'p'
        function = zerofunc
    []
[]


[Preconditioning]
    [./advanced_smp_precond]
        type = SMP
        full = true
    [../]
[]

[Functions]
    [zerofunc]
        type = ParsedFunction
        expression = '0'
    [../]
[]

[Adaptivity]
    initial_steps = 5
    initial_marker = refine_region
    max_h_level = 5
    marker = err_phi
    interval = 10

    [./Markers]
        [./err_phi]
            type = ErrorFractionMarker
            coarsen = 0.3
            refine = 0.95
            indicator = ind_phi
        [../]
        [./refine_region]
            type = BoxMarker
            bottom_left = '-100 0 0'  
            top_right = '600 40 0'  
            inside = refine         
            outside = do_nothing  
        [../]
    [../]
    [./Indicators]
        [./ind_phi]
            type = GradientJumpIndicator
            variable = phif
        [../]
    [../]
[]
 

[Executioner]
    type = Transient
    solve_type = PJFNK                   
    nl_abs_tol = 1e-4                     
    nl_rel_tol = 1e-3                    
    l_max_its = 200                        
    nl_max_its = 60                        
    # petsc_options_iname = '-ksp_type -pc_type -pc_hypre_type -ksp_gmres_restart -pc_hypre_boomeramg_strong_threshold'
    # petsc_options_value = 'gmres hypre boomeramg 30 0.7'
    petsc_options = ' -snes_converged_reason -ksp_converged_reason '
    petsc_options_iname = ' -pc_type -pc_factor_shift_type -pc_factor_mat_solver_package '
    petsc_options_value = ' lu NONZERO superlu_dist '
    dt = 0.13
    start_time = 0                 
    end_time = 75000             
    automatic_scaling = true  
[]

[GiudGiud]

could it be that the files are not being written because you ran out of memory on your allocation of the cluster?

I see a lot of files in that folder, that's what's making me think that.

This looks like an error from the automatic checkpointing. You can suppress that in the meantime
https://mooseframework.inl.gov/source/actions/AutoCheckpointAction.html
set wall_time_checkpoint = false

[mahdipasaie]

The memory is not an issue I have checked that. Also, I will test set wall_time_checkpoint = false.