Location of public APIs

[johnmcfarlane]

I've started writing a Conan recipe in order to cleanly integrate wide-integer into CNL. The plumbing and the conclusions I've come up with are hopefully applicable beyond CNL's concerns so I though I'd share the following observations/suggestions:

• There are two public include directories (the thing that GCC receives as a header search path with options like -I or -isystem):
  • ./math/wide-integer/, and
  • ./util/utility/.
• There are two public global namespaces:
  • util, and
  • math/wide_integer.
• Two of the four headers exposed in public include directories are only necessary for testing, not integration with dependent packages:
  • uintwide_t_examples.h, and
  • uintwide_t_test.h.

In the CMake or Conan scripts, it is possible to shuffle some of the files around in order to omit the test headers from the install destination. However, because of this line

#include <util/utility/util_dynamic_array.h>

two separate install directories are needed and neither -- at their root -- do very much to indicate that they contain headers specific to the wide-integer library. The namespaces suffer from a similar problem: they are one or two levels deep and given very open-ended names at their root: 'util' and 'math'.

Like namespaces, directories (under usr/include at least) are not for creating taxonomies. (That's a great video to watch generally but that nugget of advice translates very well to C++.) By using 'util' and 'math', you're more likely to risk collisions and it's less likely for users to be able to find and remember where you library is on their system. So please consider some minor rearrangement to your source files and their location -- even if you don't go with the following suggestions...

My recommended changes (which are just one possible solution and which I'd be happy to submit in a PR) are to:

• move test-only headers out of the public search path;
• decide on a namespace and a directory for the library (e.g. ::wide_integer and /wide-integer/);
• move public library headers together and away from the rest of the project files, e.g.:
  • ./include/wide-integer/uintwide_t.h, and
  • ./include/wide-integer/dynamic_array.h;
• move the public definitions into this top-level namespace, e.g. ::wide_integer::uintwide_t;
• put non-public (but header-exposed) definitions in a detail sub-namespace, e.g. ::wide_integer::detail;
• move util::dynamic_array and its comparison operators to the detail sub-namespace.

I suggest this with virtually no understanding of:

• the other libraries that might share dynamic_array (there are ways to keep it hidden in detail for wide-integer but inject it into another, public namespace elsewhere), and
• the users of wide-integer who would be disrupted by this change. A less disruptive change is entirely possible but you may find that they appreciate the added clarity -- especially if it is accompanied by build and package management facilities which are simple and idiomatic.

[ckormanyos]

the following observations/suggestions

Thanks for these interesting inputs, John. I think we can find some common compromises, but we need to discuss these just a bit.

Some of these ideas I have also had in similar form(s). These kinds of chasnges could potentially percolate also into my embedded projects extend to some of my other numeric types as well.

I believe the namespace change might be a rather small but breaking change. In the past, I have announced these kinds of breaking changes about a month up front and then implemented them, with no problem really after such things.

[ckormanyos]

Let me wrap my mind around these suggestions and soon get back to you. Then we can decide if the PR is the right place to draft a version adhering to these suggestions, as i agree it probably will be. Then we can see how a working model, let's say, plays out.

[johnmcfarlane]

@ckormanyos no problem. I can work with the multiple directories, headers, namespaces with straight-forward workarounds for now, so no hurry. If CNL comes to depend on wide-integer, they will end up being installed on systems together wherever CNL goes but that's a ways off.

A thought in that direction: while the option of absorbing the algorithms into CNL (as I think you mentioned in another thread) would avoid a lot of complexity, I'm interested in exploring paths which involve maintaining links between the various projects so that fixes and improvements which benefit one project benefit them all.

The application of a build system like CMake and a package system like Conan to a project is one way of making it less painful to plug projects together. Particularly if you're using util_dynamic_array.h in other projects, it's worth knowing that there are ways to reuse this work in a more automated way.

See this PR for latest progress on tieing everything together. This branch is a loooong way from merging!

[ckormanyos]

Well I kind of have to do something. When I first started collecting a few numeric types (independently from my work in Boost), I found myself gathering algorithms and code sequences from as alo as last century.

At first, I put some of them together in rather coherent, yet independent, forms. I now have several nice numeric types (with maybe a few more to come over the years) and several key projects (both published as well as unpublished) that use them.

But it's getting silly with redundancies. I believe my wide-decimal header is used by myself in at least 3 redundant published copies. Also wide-integer is on the same redundant path. Redundancy can be good, it has the advantage that you get only what you need, lightweight, independent and non-dependent on some kind of UNIX philosophy or similar. The disadvantage is redundancy itself, multiple copies, synchronization woes, error propagation and the like.

So I will indeed be working on reducing redundancies and finding more let's say standard storage/distro modes. The tried and true *nix usr/include and urs/lib is a classic --- with local instances of /include and /lib for a given project.

I am a bit stuck on the namespace. I like having the namespace math. But this has come into strong conflict with the popular namespace boost::math so I often end up needing to use my own ::math namespace being resolved with the scope resolution operator. This is kind of inconvenient.

[johnmcfarlane]

Unfortunately, math is not a good choice for the reason you mention and the ones I gave above: just because it's maths-related doesn't lead to wanting it in a "math" namespace. Namespaces are not for taxonomy. It would be better to think up something that's a proper noun.

I started CNL with the name fixed_point, which fails to differentiate it from all the other fixed-point projects. Similarly, I know of a few wide integer projects and I'm sure there are others. By choosing an arbitrary name (like Hana, Clara, Spirit or Boost) or an acronym (like CNL or CTRE) you reduce the likelihood of name collisions (the main reason you'd ever want to use a namespace) and increase usefulness of the name for users.

[ckormanyos]

Hi @johnmcfarlane a very first level simplification could:

• Remove uintwide_t_examples.h and uintwide_t_test.h and place these in the relevant text or examples folder(s).
• Integrate util/utility/util_dynamic_array directly as a file-local detail. This project has no need for a separate utility folder.

I do not think this would influence existing clients of wide-integer. This might not even really help your points in this issue, but does this help?

The potentially ambigous namespace(s) would remain, but single-header only would be the benefit...

[johnmcfarlane]

That sounds like a good plan. TBH I haven't looked at the integration since I started to write my error handling document. I hope to turn back to it soon.

[ckormanyos]

a good plan

Thanks John. OK @johnmcfarlane I took some initial steps, essentially those having no impact on current library use.

• The test/example headers which are, in fact, solely needed for testing have been removed from the main public directory.
• The utility header file <util/utility/util_dynamic_array.h> and its former directory have been removed.
• uintwide_t is now single-header only, dependency-free.

Namespaces and header uintwide_t.h location, however, remain unchanged. The latter would be easier to address in a future change. For the moment, I am a bit reticent to change the existing namespace(s).

[johnmcfarlane]

OK, I'm up to speed with the changes. We can probably resume this conversation. Much has been addressed since the original post. Would you like me to start a new issue with outstanding suggestions or maybe strike through what no longer applies?

Another item I don't think I mentioned is that _t is generally reserved for typedefs. As alluded to in the CompilerExplorer example here, my recommendation would be for two types: wide_integer and integer. These are the latest suggestions from the committee. Certainly uintwide_t now has a misleading u at the start and closely resembles the typedefs from <cstdint> which isn't what that type is about. If they remained a single type, perhaps multiword_integer or something that calls out the multiple limbs.

What are your latest thoughts?

[ckormanyos]

OK, I'm up to speed with the changes. We can probably resume this conversation. Much has been addressed since the original post. Would you like me to start a new issue with outstanding suggestions or maybe strike through what no longer applies?

Yes. I believe we have done well moving forward. Thanks for your help.

Another item I don't think I mentioned is that _t is generally reserved for typedefs. As alluded to in the CompilerExplorer example here, my recommendation would be for two types: wide_integer and integer. These are the latest suggestions from the committee.

Certainly uintwide_t now has a misleading u at the start and closely resembles the typedefs from <cstdint> which isn't what that type is about. If they remained a single type, perhaps multiword_integer or something that calls out the multiple limbs.

What are your latest thoughts?

My unfiltered gut-reaction is to leave the preliminary work of uintwide_t closely as-it-is. A lot of people are using it. Big name changes and redesings of this scale might almost be better done in a new, mostly-independent derivative work that more closely parallels the progress in SG6. I'm just not sure if I would like to metamorphosize uintwide_t --- with all its attributes --- to try to adhere to where we it's going in the committee.

In order to at least add some conformant-look to uintwide_t, we could add some (potentially templated) typedefs that more closely resemble wide_integer and integer.

But if you ask my honest opinion, I feel like the scope of changes being discussed at this point almost need a new derivative work that is more closely resembling a partial reference implementation for wide_integer and integer, which might heavily borrow code or gain inspiration from ckormanyos/wide-integer.

I'm pretty sure the committe needs a reference implementation when dealing with numerics on this complexity level for sure. I would need and expect one. Now just one small part would/could be types like those in my project. I think the drive forward can and should be more closely in line with progress in SG6 --- fully knwoing, ... that is or would be a lot of work.

[johnmcfarlane]

My unfiltered gut-reaction is to leave the preliminary work of uintwide_t closely as-it-is. A lot of people are using it. Big name changes and redesings of this scale might almost be better done in a new, mostly-independent derivative work that more closely parallels the progress in SG6. I'm just not sure if I would like to metamorphosize uintwide_t --- with all its attributes --- to try to adhere to where we it's going in the committee.

If ckormanyos/wide-integer formed the basis for a new type or library, how would support for that work? Would it be a snapshot of wide-integer which was then maintained by me, or by both of us? And would you wish to continue supporting / advancing wide-integer separately?

From CNL's PoV, what happens to ckormanyos/wide-integer is not as important as ensuring we're not expending duplicate effort and that CNL can continue to benefit from the expertise and effort you put into numerics. If I fork wide-integer or snapshot it inside CNL and then wide-integer continues to enjoy bug fixes and improvements from you, then I've created the burden of porting those over. That's really the thing I'm trying to avoid here.

In order to at least add some conformant-look to uintwide_t, we could add some (potentially templated) typedefs that more closely resemble wide_integer and integer.

That would work well except that anyone using those aliases (including CNL users) would also get ::math and ::util added into their global namespace. We could flip your idea on its head and turn uintwide_t into an alias of those new types:

#include <ckormanyos/integer.h>
#include <ckormanyos/wide_integer.h>

namespace math::wide_integer {
  template<const size_t Width2,
           typename LimbType,
           typename AllocatorType,
           const bool IsSigned>
  using uintwide_t = std::conditional_t<
      std::is_same_v<AllocatorType, void>, 
      ::ckormanyos::wide_integer<std::array<LimbType, (Width2 + width_v<LimbType> - 1) / width_v<LimbType>>>, 
      ::ckormanyos::integer<LimbType, std::vector<LimbType, AllocatorType>>>;
}

In this way, wide-integer should remain backward compatible but it would be drawing on the new library. (Note I think I could reduce the number of wide_integer template parameters.)

This is a good choice if you want to keep uintwide_t in active development. But if you're happy to draw a line under it (call it V1) and move on to a new, incompatible interface (call it V2), then it's not necessary because uintwide_t stops being actively worked on.

It's up to you.