Restart does not work or is not reliable

[rjarvinen]

RHybrid's restart is not working properly or at all. It is not clear if the problem is in the writing or reading of restart files or both or in the initialization after a restart. Restart has not been used/needed for a long time.

[rjarvinen]

Related to fmihpc/corsair#2

[rjarvinen]

writeRestart(...) in corsair/src/kernel/restart_writer.cpp calls ParGrid::getDynamicUserDataInfo:

simClasses.pargrid.getDynamicUserDataInfo(arrayDataIDs,arrayNames,arraySizes,arrayDatatypes,arrayByteSizes,arrayDynamicPointers);

Here arrayDatatypes returns unknown:

std::cout << arrayDatatypes[0] << std::endl << std::flush;

getDynamicUserDataInfo in pargrid/pargrid.h assigns arrayDatatypes as follows:

for (typename std::map<DataID,UserDataDynamic<ParGrid<C> >*>::const_iterator it=userDataDynamic.begin(); it!=userDataDynamic.end(); ++it) {
 if (it->second == NULL) { continue; }
 ...
 datatypes.push_back(it->second->getDatatype());
 ...
}

where unknown comes from ParGrid:

template<typename T> inline
std::string getDatatype() { return "unknown"; }

Known datatypes are:

std::string getDatatype<bool>() { return "int"; }
std::string getDatatype<char>() { return "int"; }
std::string getDatatype<int8_t>() { return "int"; }
std::string getDatatype<int16_t>() { return "int"; }
std::string getDatatype<int32_t>() { return "int"; }
std::string getDatatype<int64_t>() { return "int"; }
std::string getDatatype<uint8_t>() { return "uint"; }
std::string getDatatype<uint16_t>() { return "uint"; }
std::string getDatatype<uint32_t>() { return "uint"; }
std::string getDatatype<uint64_t>() { return "uint"; }
std::string getDatatype<float>() { return "float"; }
std::string getDatatype<double>() { return "float"; }
std::string getDatatype<long double>() { return "float"; }

[rjarvinen]

Why the datatype is unknown?

getDatatype in rhybrid/particle_definition.h creates the particle datatype for MPI:

template<typename REAL> inline
void Particle<REAL>::getDatatype(MPI_Datatype& datatype) {
   MPI_Type_contiguous(particle::SIZE,MPI_Type<REAL>(),&datatype);
}

corsair/src/include/particle_list_skeleton.h calls this and commits the datatype to use it in communications:

PARTICLE::getDatatype(particleType);
MPI_Type_commit(&particleType);

I tested and MPI_Type_commit returns MPI_SUCCESS. PARTICLE is defined in rhybrid/register_objects.cpp as:

typedef Particle<Real> PARTICLE;

MPI_Type_commit assigns the value for particleType in the class ParticleListBase in corsair/src/include/particle_list_base.h:

MPI_Datatype particleType; /**< MPI datatype used to transfer particles.*/

particleType is used only in ParticleListBase::finalize() in corsair/src/kernel/particle_list_base.cpp to finalize the program:

// Free MPI datatype that was used to transfer the particles:
if (particleType != MPI_DATATYPE_NULL) MPI_Type_free(&particleType);

[iljah]

One possibility is that corsair doesn't see rhybrid's definition of getDatatype for Particle. I suspect it would if e.g. particle was given as argument to getDatatype (https://en.wikipedia.org/wiki/Argument-dependent_name_lookup), or if corsair was calling particle's getDatatype().

Can corsair handle array types, i.e. more than one item of integer/floating point? I've always returned a tuple of void*, int, MPI_Datatype from these type of functions since that's what MPI_Send etc require anyway and handling contiguous arrays is also trivial (https://github.com/fmihpc/dccrg/blob/6ca02ed9937b4294925cee867a159b2801e3c40e/dccrg_get_cell_datatype.hpp#L217).

[rjarvinen]

Arto mentioned long time ago that unknown datatypes are not a problem per se:
fmihpc/corsair#2

[rjarvinen]

I think this has got to do with Corsair's interaction with VLSV. I've been studying these old topics, which partly worked back then, but haven't yet found a solution or a workaround for the current codes:

fmihpc/vlsv#31
fmihpc/vlsv#32
fmihpc/vlsv#37

[iljah]

This might be relevant https://stackoverflow.com/q/119123 although looks like rhybrid's particle is simple and large enough that padding shouldn't affect this.

[iljah]

I guess this is a problem also in 1 process run, in which case it might be easiest to debug just by sprinkling couts of MPI_Type_size etc into saving/loading code to figure out how much particle data each cell has at which memory offset(s) and how much it's writing/reading at which file offset(s). At best you might be able to reproduce this in like 3x3x3 cell simulation with couple of particles.

[rjarvinen]

Coming back to this issue

• restart works without particles
• not sure if this is important but when writing particles
  ...only this is called: const std::string& UserDataDynamic<PARGRID>::getDatatype()
  ...while this is never called:
  /** Get a derived MPI datatype that sends or receives given cell data
  • with a single transfer. Note that array sizes must have been transferred,
  • i.e. the amount of data sent or received, and reallocate function called,
  • before getDatatype is called. Otherwise this function may reference
  • invalid memory areas. This function commits the newly created datatype.
  • @param globalIDs List of transferred cells.
  • @param datatype Variable in which the derived datatype is to be written.*/
    void UserDataDynamic<PARGRID>::getDatatype(const std::set<CellID>& globalIDs,MPI_Datatype& datatype)