investigate zero-copy read API for Ioss

[utkarshayachit]

With Catalyst support, it is possible to use Ioss for in situ processing. Currently, however, when the data is read in on ParaView side, it gets deep-copied. This proposal is intended to avoid that deep copy.

Proposal

Ioss::GroupingEntity::get_field_data(...) is used to read data. Current API requires that the target memory is already allocated by the caller code. To support zero-copy, we can add an overload that returns references to internally held memory instead.

  int64_t get_field_data(const std::string &field_name, void **data, size_t* data_size) const;

Issues

1. Not every database needs to support zero-copy. Nor is every field zero-copyable e.g. if a field has transforms defined, it's not zero-copyable.
   This will require that we have some mechanism to query each Field if it's zero-copyable. This could be added a member-variable on Ioss::Field which can be set by the database when creation to indicate if a field is zero-copyable. Readers can check this to determine whether to use zero-copy API or standard API.

2. The zero-copy would return a reference to the actual data being stored somewhere; if the calling code wants to, it can modify the data. Maybe change to const void * const* data or somehow disallow modifying the underlying copy of the data

[gsjaardema]

Thank you for the proposal. This will be a good location to discuss this.

I editted the above comment and change the first issue to 'Not every database needs to support...'

[tjotaha]

Thanks for opening this issue, Utkarsh. Below are some of my thoughts on the current and proposed state of memory usage when a simulation code calls Catalyst through IOSS using API 1, API 2, and the proposed zero-copy interface (descriptions may or may not be entirely accurate).

Current IOSS (Catalyst API 1)

1. Simulation (allocates and frees buffers for IOSS interaction)
2. Call IOSS Write Catalyst API 1 Database
3. Catalyst Database deep copies IOSS buffers to create VTK Multiblock mesh
4. Call Catalyst and pass a reference to VTK Multiblock mesh

Current IOSS (Catalyst API 2)

1. Simulation (allocates and frees buffers for IOSS interaction)
2. Call IOSS Write Catalyst API 2 Database
3. Catalyst Database deep copies IOSS buffers to create Conduit mesh
4. Call Catalyst
5. Call IOSS Read Catalyst API 2 Database in ParaView implementation of Catalyst API 2
6. ParaView allocates buffers to read from Catalyst API 2 Database through IOSS

Proposed IOSS zero-copy (Catalyst API 2)

1. Simulation (allocates and frees buffers for IOSS interaction)
2. Call IOSS Write Catalyst API 2 Database
3. Catalyst Database deep copies IOSS buffers to create Conduit mesh
4. Catalyst Database indicates which IOSS::Fields() are zero copyable.
5. Call Catalyst
6. Call IOSS Read Catalyst API 2 Database in ParaView implementation of Catalyst API 2
7. ParaView only allocates buffers to read from Catalyst API 2 Database for IOSS::Fields() that are not zero copyable

The current IOSS Catalyst API 2 implementation uses about twice the memory as Catalyst API 1?

The proposed zero-copy interface will use some amount of memory more than the
Catalyst API 1 implementation, depending on how many IOSS::Fields() are zero copyable.

Would it be possible for the simulation code to indicate that a field is zero-copyable on write to IOSS? In that case, the simulation would need to persist the buffer through the call to Catalyst or give a reference to some persistent data in the correct ordering for IOSS. This might realize even better memory savings than current Catalyst API 1.

[utkarshayachit]

Current IOSS (Catalyst API 2)
..
Catalyst Database deep copies IOSS buffers to create Conduit mesh
Would it be possible for the simulation code to indicate that a field is zero-copyable on write to IOSS?

Iocatalyst:: DatabaseIO already supports a property that let's you indicate that all fields are shallow-copied on put_field.

[gsjaardema]

Issue:

• If a field is zero_copy enabled and a transform is added to the field is that an error, or is the transform added and the field is changed to non-zero copy enabled and is a warning/message output?
• Current prototype implementation throws an error.

[tjotaha]

It makes sense that transforms should not be allowed when zero_copy is enabled. I vote for the current implementation.

It is an unexpected side effect of setting zero_copy enabled that using transform() related methods will cause an error throw. A more extensive redesign would involve creating an abstract class for Ioss::Field and then subclassing a zero_copy field class and a non-zero copy field class that has the transform methods.