Merge branch 'dev' into update_perftests

Author: MaxThevenet

.gitignore

@@ -7,3 +7,6 @@ o/
 tmp_build_dir/
 AMReX_buildInfo.cpp
 Backtrace.*
+*.vth
+*.sw[opqrs]
+*.png

Docs/source/latex_theory/Boosted_frame/Boosted_frame.tex

@@ -142,8 +142,7 @@ \subsection{Numerical Stability and alternate formulation in a Galilean frame}
 For instance, NCI-specific corrections include periodically smoothing
 the electromagnetic field components \cite{Martinscpc10},
 using a special time step \cite{VayAAC2010,Vayjcp2011} or
-applying a wide-band smoothing of the current components \cite{
-  VayAAC2010,Vayjcp2011,VayPOPL2011}. Another set of mitigation methods
+applying a wide-band smoothing of the current components \cite{VayAAC2010,Vayjcp2011,VayPOPL2011}. Another set of mitigation methods
 involve scaling the deposited
 currents by a carefully-designed wavenumber-dependent factor
 \cite{GodfreyJCP2014_FDTD,GodfreyIEEE2014} or slightly modifying the

Docs/source/running_cpp/parameters.rst

@@ -339,16 +339,20 @@ Laser initialization
     ``laser.profile_focal_distance`` in the laboratory frame, and use ``warpx.gamma_boost``
     to automatically perform the conversion to the boosted frame.
 
+* ``laser.stc_direction`` (`3 floats`) optional (default `1. 0. 0.`)
+    Direction of laser spatio-temporal couplings.
+  	See definition in Akturk et al., Opt Express, vol 12, no 19 (2014).
+
 * ``laser.zeta`` (`float`; in meters.seconds) optional (default `0.`)
-    Spatial chirp at focus in the ``x`` direction. See definition in
+    Spatial chirp at focus in direction ``laser.stc_direction``. See definition in
     Akturk et al., Opt Express, vol 12, no 19 (2014).
 
 * ``laser.beta`` (`float`; in seconds) optional (default `0.`)
-    Angular dispersion (or angular chirp) at focus in the ``x`` direction.
+    Angular dispersion (or angular chirp) at focus in direction ``laser.stc_direction``.
     See definition in Akturk et al., Opt Express, vol 12, no 19 (2014).
 
 * ``laser.phi2`` (`float`; in seconds**2) optional (default `0.`)
-    Temporal chirp at focus in the ``x`` direction.
+    Temporal chirp at focus.
     See definition in Akturk et al., Opt Express, vol 12, no 19 (2014).
 
 Numerics and algorithms

Docs/source/visualization/sensei.rst

@@ -0,0 +1,77 @@
+In situ Visualization with SENSEI
+=================================
+SENSEI is a light weight framework for in situ data analysis. SENSEI's data
+model and API provide uniform access to and run time selection of a diverse set
+of visualization and analysis back ends including VisIt Libsim, ParaView
+Catalyst, VTK-m, Ascent, ADIOS, Yt, and Python.
+
+SENSEI uses an XML file to select and configure one or more back ends at run
+time. Run time selection of the back end via XML means one user can access
+Catalyst, another Libsim, yet another Python with no changes to the code.
+
+Compiling with GNU Make
+-----------------------
+For codes making use of AMReX's build system add the following variable to the
+code's main :code:`GNUmakefile`.
+
+.. code-block:: bash
+
+   USE_SENSEI_INSITU = TRUE
+
+When set, AMReX's make files will query environment variables for the lists of
+compiler and linker flags, include directories, and link libraries. These lists
+can be quite elaborate when using more sophisticated back ends, and are best
+set automatically using the :code:`sensei_config` command line tool that should
+be installed with SENSEI. Prior to invoking make use the following command to
+set these variables:
+
+.. code-block:: bash
+
+   source sensei_config
+
+Typically, the :code:`sensei_config` tool is in the users PATH after loading
+the desired SENSEI module. After configuring the build environment with
+:code:`sensei_config`, proceed as usual.
+
+.. code-block:: bash
+
+   make -j4 -f GNUmakefile
+
+ParmParse Configuration
+-----------------------
+Once an AMReX code has been compiled with SENSEI features enabled, it will need
+to be enabled and configured at runtime. This is done using ParmParse input file.
+The following 3 ParmParse parameters are used:
+
+.. code-block:: python
+
+   insitu.int = 2
+   insitu.start = 0
+   insitu.config = render_iso_catalyst_2d.xml
+
+:code:`insitu.int` turns in situ processing on or off and controls how often
+data is processed. :code:`insitu.start` controls when in situ processing
+starts. :code:`insitu.config` points to the SENSEI XML file which selects and
+configures the desired back end.
+
+Obtaining SENSEI
+-----------------
+SENSEI is hosted on Kitware's Gitlab site at https://gitlab.kitware.com/sensei/sensei
+It's best to checkout the latest release rather than working on the master branch.
+
+To ease the burden of wrangling back end installs SENSEI provides two platforms
+with all dependencies pre-installed, a VirtualBox VM, and a NERSC Cori
+deployment. New users are encouraged to experiment with one of these.
+
+
+SENSEI VM
+~~~~~~~~~
+The SENSEI VM comes with all of SENSEI's dependencies and the major back ends
+such as VisIt and ParaView installed. The VM is the easiest way to test things
+out. It also can be used to see how installs were done and the environment
+configured.
+
+NERSC Cori
+~~~~~~~~~~
+SENSEI is deployed at NERSC on Cori. The NERSC deployment includes the major
+back ends such as ParaView Catalyst, VisIt Libsim, and Python.

Docs/source/visualization/visualization.rst

@@ -7,3 +7,4 @@ Visualizing the simulation results
    yt
    visit
    picviewer
+   sensei

Examples/Tests/subcycling/inputs.2d

@@ -0,0 +1,154 @@
+# stop_time = 1.12966840205e-10
+amrex.v=1
+max_step = 2500
+amr.n_cell = 64 256
+
+amr.max_grid_size = 4096
+amr.blocking_factor = 16
+
+# Maximum level in hierarchy (for now must be 0, i.e., one level in total)
+amr.max_level = 1
+amr.plot_file = "plotfiles/plt"
+amr.plot_int = 200
+
+warpx.fine_tag_lo = -2.e-6   -15.e-6 
+warpx.fine_tag_hi =  2.e-6    -7.e-6
+
+
+# Geometry
+geometry.coord_sys   = 0                  # 0: Cartesian
+geometry.is_periodic = 0 0      # Is periodic?
+geometry.prob_lo     = -30.e-6   -20.e-6      # physical domain
+geometry.prob_hi     =  30.e-6     0.e-6
+
+# Verbosity
+warpx.verbose = 1
+warpx.plot_raw_fields = 1
+warpx.do_dive_cleaning = 0
+warpx.use_filter = 1
+warpx.do_pml = 1
+warpx.do_subcycling = 0
+warpx.refine_plasma = 0
+warpx.plot_raw_fields = 1 
+warpx.plot_raw_fields_guards = 1 
+warpx.plot_finepatch = 1 
+warpx.plot_crsepatch = 1
+warpx.n_current_deposition_buffer = 0
+warpx.n_field_gather_buffer = 0
+
+# Algorithms
+algo.current_deposition = 0
+algo.charge_deposition = 0
+algo.field_gathering = 0
+algo.particle_pusher = 0
+algo.maxwell_fdtd_solver = "ckc"
+
+# CFL
+warpx.cfl = .9999
+particles.nspecies = 4
+particles.species_names = driver beam plasma_e plasma_p
+
+# interpolation
+interpolation.nox = 3
+interpolation.noy = 3
+interpolation.noz = 3
+
+#
+# The driver species information
+#
+
+driver.charge = -q_e
+driver.mass = m_e
+driver.injection_style = "gaussian_beam"
+driver.x_rms = 3.e-6
+driver.y_rms = 3.e-6
+driver.z_rms = .2e-6
+driver.x_m = 0.
+driver.y_m = 0.
+driver.z_m = -3.e-6
+driver.npart = 10000
+driver.q_tot = -1.e-10
+driver.profile = "constant"
+driver.density = 5.e24                   # number of particles per m^3
+driver.momentum_distribution_type = "gaussian"
+driver.ux_m = 0.0
+driver.uy_m = 0.0
+driver.uz_m = 1e6
+driver.uz_th = 0.
+#driver.ux_th = 2.
+#driver.uy_th = 2.
+#driver.uz_th = 20000.
+driver.zinject_plane = 0.
+driver.rigid_advance = true
+driver.projected = true
+driver.focused = false
+
+#
+# The beam species information
+#
+
+beam.charge = -q_e
+beam.mass = m_e
+beam.injection_style = "gaussian_beam"
+beam.x_rms = .1e-6
+beam.y_rms = .1e-6
+beam.z_rms = .2e-6
+beam.x_m = 0.
+beam.y_m = 0.
+beam.z_m = -11.e-6
+beam.npart = 10000
+beam.q_tot = -1.e-12
+beam.profile = "constant"
+beam.density = 1.e24
+beam.momentum_distribution_type = "gaussian"
+beam.ux_m = 0.0
+beam.uy_m = 0.0
+beam.uz_m = 20.
+beam.u_th = 0.
+
+#
+# The plasma species information
+#
+
+plasma_e.charge = -q_e
+plasma_e.mass = m_e
+plasma_e.injection_style = "NUniformPerCell"
+plasma_e.xmin = -15.e-6
+plasma_e.xmax =  15.e-6
+plasma_e.ymin = -15.e-6
+plasma_e.ymax =  15.e-6
+plasma_e.zmin = 0.e-6
+plasma_e.profile = "constant"
+plasma_e.density = 1.e25
+plasma_e.num_particles_per_cell_each_dim = 2 2
+plasma_e.momentum_distribution_type = "constant"
+plasma_e.ux = 0.0
+plasma_e.uy = 0.0
+plasma_e.uz = 0.0
+
+
+plasma_p.charge = q_e
+plasma_p.mass = m_p
+plasma_p.injection_style = "NUniformPerCell"
+plasma_p.xmin = -15.e-6
+plasma_p.xmax =  15.e-6
+plasma_p.ymin = -15.e-6
+plasma_p.ymax =  15.e-6
+plasma_p.zmin = 0.e-6
+plasma_p.profile = "constant"
+plasma_p.density = 1.e25
+plasma_p.num_particles_per_cell_each_dim = 2 2
+plasma_p.momentum_distribution_type = "constant"
+plasma_p.ux = 0.0
+plasma_p.uy = 0.0
+plasma_p.uz = 0.0
+
+# Moving window
+warpx.do_moving_window = 1
+warpx.moving_window_dir = z
+warpx.moving_window_v = 1. # in units of the speed of light
+
+# Particle Injection
+warpx.do_plasma_injection = 1
+warpx.num_injected_species = 2
+warpx.injected_plasma_species = 2 3

GNUmakefile

@@ -24,6 +24,8 @@ EBASE     = main
 
 USE_PYTHON_MAIN = FALSE
 
+USE_SENSEI_INSITU = FALSE
+
 WarpxBinDir = Bin
 
 USE_PSATD = FALSE

Regression/WarpX-tests.ini

@@ -251,7 +251,7 @@ compareParticles = 0
 [LaserAcceleration]
 buildDir = .
 inputFile = Examples/Physics_applications/laser_acceleration/inputs.3d
-runtime_params = warpx.do_dynamic_scheduling=0 amr.n_cell=32 32 256 max_step=100 electrons.zmin=0.e-6
+runtime_params = warpx.do_dynamic_scheduling=0 amr.n_cell=32 32 256 max_step=100 electrons.zmin=0.e-6 warpx.serialize_ics=1
 dim = 3
 restartTest = 0
 useMPI = 1
@@ -261,12 +261,26 @@ numthreads = 2
 compileTest = 0
 doVis = 0
 compareParticles = 1
-particleTypes = electrons beam
+particleTypes = electrons
+
+[subcyclingMR]
+buildDir = .
+inputFile = Examples/Tests/subcycling/inputs.2d
+runtime_params = warpx.serialize_ics=1 warpx.do_dynamic_scheduling=0
+dim = 2
+restartTest = 0
+useMPI = 1
+numprocs = 2
+useOMP = 1
+numthreads = 2
+compileTest = 0
+doVis = 0
+compareParticles = 0
 
 [LaserAccelerationMR]
 buildDir = .
 inputFile = Examples/Physics_applications/laser_acceleration/inputs.2d
-runtime_params = amr.max_level=1 max_step=100
+runtime_params = amr.max_level=1 max_step=100 warpx.serialize_ics=1
 dim = 2
 restartTest = 0
 useMPI = 1
@@ -281,7 +295,7 @@ particleTypes = electrons beam
 [PlasmaAccelerationMR]
 buildDir = .
 inputFile = Examples/Physics_applications/plasma_acceleration/inputs.2d
-runtime_params = amr.max_level=1 amr.n_cell=32 512 max_step=100 plasma_e.zmin=-200.e-6
+runtime_params = amr.max_level=1 amr.n_cell=32 512 max_step=100 plasma_e.zmin=-200.e-6 warpx.serialize_ics=1 warpx.do_dynamic_scheduling=0
 dim = 2
 restartTest = 0
 useMPI = 1

Source/LaserParticleContainer.H

@@ -52,6 +52,8 @@ private:
     amrex::Vector<amrex::Real> position;
     amrex::Vector<amrex::Real> nvec;
     amrex::Vector<amrex::Real> p_X;
+    amrex::Vector<amrex::Real> stc_direction;
+
     long                      pusher_algo = -1;
     amrex::Real               e_max       = std::numeric_limits<amrex::Real>::quiet_NaN();
     amrex::Real               wavelength  = std::numeric_limits<amrex::Real>::quiet_NaN();
@@ -72,6 +74,7 @@ private:
     amrex::Real zeta = 0.;
     amrex::Real beta = 0.;
     amrex::Real phi2 = 0.;
+    amrex::Real theta_stc = 0.;
 
     // parse_field_function profile
     int parser_instance_number = 0;

Source/LaserParticleContainer.cpp

@@ -57,6 +57,8 @@ LaserParticleContainer::LaserParticleContainer (AmrCore* amr_core, int ispecies)
 	   pp.get("profile_duration", profile_duration);
 	   pp.get("profile_t_peak", profile_t_peak);
 	   pp.get("profile_focal_distance", profile_focal_distance);
+	   stc_direction = p_X;
+	   pp.queryarr("stc_direction", stc_direction);
 	   pp.query("zeta", zeta);
 	   pp.query("beta", beta);
 	   pp.query("phi2", phi2);
@@ -113,6 +115,22 @@ LaserParticleContainer::LaserParticleContainer (AmrCore* amr_core, int ispecies)
 
 	p_Y = CrossProduct(nvec, p_X);   // The second polarization vector
 
+	s = 1.0/std::sqrt(stc_direction[0]*stc_direction[0] + stc_direction[1]*stc_direction[1] + stc_direction[2]*stc_direction[2]);
+	stc_direction = { stc_direction[0]*s, stc_direction[1]*s, stc_direction[2]*s };    
+	dp = std::inner_product(nvec.begin(), nvec.end(), stc_direction.begin(), 0.0);
+        AMREX_ALWAYS_ASSERT_WITH_MESSAGE(std::abs(dp) < 1.0e-14, 
+            "stc_direction is not perpendicular to the laser plane vector");
+    
+	// Get angle between p_X and stc_direction
+	// in 2d, stcs are in the simulation plane
+#if AMREX_SPACEDIM == 3
+	theta_stc = acos(stc_direction[0]*p_X[0] + 
+			 stc_direction[1]*p_X[1] + 
+			 stc_direction[2]*p_X[2]);
+#else
+	theta_stc = 0.;
+#endif
+
 #if AMREX_SPACEDIM == 3
 	u_X = p_X;
 	u_Y = p_Y;
@@ -419,7 +437,7 @@ LaserParticleContainer::Evolve (int lev,
 		warpx_gaussian_laser( &np, plane_Xp.data(), plane_Yp.data(),
 				      &t_lab, &wavelength, &e_max, &profile_waist, &profile_duration,
 				      &profile_t_peak, &profile_focal_distance, amplitude_E.data(),
-				      &zeta, &beta, &phi2 );
+				      &zeta, &beta, &phi2, &theta_stc );
 	    }
 
             if (profile == laser_t::Harris) {