Allow properties/multi_properties to be created from non-filesystem sources

[drbenmorgan]

A use case identified in SuperNEMO's use of Bayeux is the ability to store properties/multi_properties (I'll just use properties from now on to refer to both for simplicity) in non-filesystem based stores like databases. As implemented, properties and the underlying Bayeux kernel and associated tools always assume that paths (where to find the data), and data (the content to be read) are files on the local filesystem.

The issue is not directly with the lowest level interfaces such properties::config::_read_, which use istream as the data source:

Bayeux/source/bxdatatools/src/properties.cc

Line 3191 in b9395b4

void properties::config::_read_(std::istream & in_, properties & props_)

so have no hard requirement to use ifstream/FILE (for example). However the higher level read/write interfaces do make this assumption:

Bayeux/source/bxdatatools/src/properties.cc

Lines 3142 to 3160 in b9395b4

	void properties::config::read(const std::string & filename_, properties & props_)
	{
	std::string filename = filename_;
	if (_resolve_path_) {
	DT_THROW_IF(!fetch_path_with_env(filename),
	std::logic_error,
	"Cannot resolve filename '" + filename_ + "'!");
	}
	std::ifstream fin(filename.c_str());
	DT_THROW_IF(!fin,
	std::logic_error,
	"Cannot open filename '" + filename_ + "' (resolved as '" + filename + "'!");
	set_reader_input(filename, -1);
	_read_(fin, props_);
	fin.close();
	_current_filename_.clear();
	_current_line_number_ = 0;
	return;
	}

as do the helper classes like file_include:

Bayeux/source/bxdatatools/src/properties.cc

Lines 3621 to 3629 in b9395b4

	std::tuple<bool, std::string> fi_result = _fi_.resolve_err(include_config_file);
	DT_PROP_CFG_READ_THROW_IF(! std::get<0>(fi_result),
	std::logic_error,
	_current_filename_,
	_section_name_,
	_section_start_line_number_,
	_current_line_number_,
	"Cannot resolve include file path for '" << include_config_file << "'!");
	std::string resolved_path = std::get<1>(fi_result);

The problem is thus to try and hide this assumption of paths representing files on the filesystem, and that the data/content will come as an ifstream. Pull Requests #65, #66, and #67 provide a first step for investigating this by decoupling the reading/writing of properties from the in-memory representation. That's generally useful, but the key thing here is that across Bayeux it reduces the construction of properties from a file to a canonical form:

datatools::fetch_path_with_env(someFilePath);
datatools::properties someProps;
datatools::read_config(someFilePath, someProps);

I think what's needed is a "content_resolver" object that would take a path and return the content at that path:

class content_resolver {
...
  content_type get(path_type path) const;
};

with a datatools::read_config implementation then looking something like:

namespace datatools {
  void read_config(path_type& path, content_resolver& resolver, properties& props)
  {
    properties_config reader(resolver); // hands down resolver to other things that need it
    reader.read(path, props); // populate as needed
  }
}

and properties_config (as shown in #67) could then implement read with little change as:

void properties_config::read(path_type& path, properties& props)
{
  content_type cont = resolver.get(path); // uses the constructed with content_resolver
  // make an istream from cont, e.g. istringstream if it's string or similar  
  this->_read_(cont, props); // reuse existing implementation!
}

Different implementations of the resolver would handle any needed resolution mechanism, e.g. local filesystem, remote web, Git, SQL etc. They could also be organised in a PATH like structure to allow "overlays", e.g use database, but overlay local filesystem to allow testing. The content_type could be as simple as std::string for properties as these are always text data. A basic example of a Git backend is shown in https://github.com/SuperNEMO-DBD/SNGitCondDB, but it can be seen from that that extension to, e.g. SQL or similar would be straightforward.

There's quite a bit of inspiration here from Fermilab's cetlib and fhicl-cpp libraries, from which code could be used if needed. See

This is just a quick sketch though, so wanted to raise the issue to start discussion on design/implementation, and bring in @robobre, @pfranchini, @lemiere, @emchauve, @cherylepatrick from the SuperNEMO AB as this will impact/benefit the experiment more broadly. There're also more locations through Bayeux to consider, so obviously needs discussion and thought on the design.