Fork overview, and thoughts to improve basis for chance of upstreaming

[mara004]

See below for an overview of this fork. Note, this writeup is a non-exhaustive work in progress.

This information may be valuable for working towards a basis that could be merged back into upstream at some point, though this seems fairly hypothetical for the near term, given time constraints, and mismatched design intents (e.g. relating to backwards compatibility).

However, this fork of ctypesgen may be a good starting point for any active future development, with a significantly overhauled code base that should be nicer to work with.

Selection of improvements from this fork

• Removal of bloated old string classes that imply technical debt.
  Enforcement of explicit string encoding/decoding. (We might want to add back implicit string handling as opt-in in the future, see below.)
  Note, the old string classes are incompatible with some python releases of the 3.7/3.8 branches.
  See also Incompatibility with python 3.8.1 pypdfium2#76, String seems incorrect ctypesgen/ctypesgen#77, bpo-16575: Add checks for unions passed by value to functions. python/cpython#16799, Make string auto-conversion optional & use leaner strings classes ctypesgen/ctypesgen#177
• Bloated old library loader replaced with new lean library loader that is more explicit/controllable.
  See also Are library loader classes actually necessary? ctypesgen/ctypesgen#176
  Resolve . to the module directory, not the caller's CWD. Don't add compile libdirs to runtime. ¹
• Preventing the assignment of invalid/non-existent struct fields by correction of __slots__ declaration. This fix should be fairly easy for upstream to pick. See also Correct definition of __slots__ on structs (must be defined in class body) ctypesgen/ctypesgen#183
• Implemented relative imports with --link-modules, and library handle sharing with --no-embed-preamble, Removed incorrect POINTER override. This properly fixes Recognize structs from common header files in different wrapper modules ctypesgen/ctypesgen#86 (shared headers), and allows to divide bindings to a library in multiple outputs (e.g. translate each header to a separate python file).
• More powerful/flexible means of control over symbol inclusion via --symbol-rules.
• Pre-processor auto-detection and significant improvements to call style (see 7559e81).
• Removed questionable UNCHECKED wrapper from preamble.
• Do not bypass c_void_p -> int auto-conversion (see Changelog or commit for background).
• Propagate exception if no output members were found. (Previously would have been a warning, but the if-check was incorrect.)
• New style-related printer options that allow to disable symbol if-guards² and macro guards.
• Proper newline concept for the python printer, see a538742.
• Free library handles after use, to allow for in-session deletion of DLLs. This allows to activate a formerly skipped test case on Windows.
• Internal code cleanups and test suite improvements.

small, self-contained fixes have usually been submitted upstream and may have been merged

• Fixed conflicting names resolver never actually being called. 105a3c6, Fix conflicting names resolver being never called ctypesgen/ctypesgen#193. But what the code does is still poor, unfortunately.

Points to consider

• Restoring implicit UTF-8 string encoding/decoding as optional?

  ctypesgen originally did implicit UTF-8 encoding/decoding of in/out strings.
  While that tends to be bad practice and callers had better handle strings explicitly instead, it would seem reasonable to retain an optional backward compatibility layer for existing callers.
  I also imagine it might be convenient for a library that consistently uses UTF-8 for everything.

  Adding the old string classes back is certainly not an option for us. However, it may be possible to create a lean replacement. See Make string auto-conversion optional & use leaner strings classes ctypesgen/ctypesgen#177 for a suggestion (copy below), or String seems incorrect ctypesgen/ctypesgen#77 (comment) for an alternative draft by @olsonse.
  Note that in/out must be handled in a single class.

• Mixed calling conventions

  We do not currently support mixed calling conventions (both cdecl and windows-only stdcall functions in one binary). Our fork dropped this while rewriting the library loader, because it does not match the API layer of ctypes and would need a template to handle, which the assumably low relevance did not seem to justify. The previous implementation was a bit dirty, too: In case of a pure stdcall library, a cdecl handle would be opened anyway, and binary variables would be taken from the cdecl one.
  If there is a real-world use case, this could be added back.

• Removal of support for multiple libraries in one bindings file

  This feature was a significant complexity burden in some code areas, including pollution around symbols in printer/output code. For now we decided to remove it - callers can use --no-embed-preamble and --link-modules to create separate bindings files. This also encourages individual/explicit rather than unified loader config.

  See also Recognize structs from common header files in different wrapper modules ctypesgen/ctypesgen#86 (comment) for some alternative considerations.

Other notes

• Shifts in design intent: We would prefer to stick with plain ctypes as much as possible and avoid cluttering the bindings with custom wrappers.

• CLI: We changed the command-line interface from action=append to action=extend and nargs=+/*. This implied switching headers from positional to flag argument to avoid confusion/interference with flags that take multiple arguments. There are more CLI changes not listed here, see diff for details.

Done tasks

• Restored test suite usability by adapting to fork changes.
• Restored macro guards as opt-out

Footnotes

1. Quoting from the commit in question "Fun fact: It seems like ctypesgen prior to this change added the complie_libdirs to the runtime library loader. Further, relative paths (I think) were never correctly interpreted relative to the file's directory, but to the user's CWD, which is useless and could even cause unexpected behavior. So runtime_libdirs = ["."] didn't work, but the fallback below [commented with] 'then we search the directory where the generated python interface is stored' captured." ↩

2. Note, this is meant for use with inherently ABI correct packaging only ↩

[mara004]

Dumping our lean string class replacement draft below as it's not very well visible in the PR diff.

class _wraps_c_char_p:
    def __init__(self, raw, value):
        self.raw = raw
        self.value = value

    # provided for clarity, not actually necessary due to __getattr__ wrapper below
    def decode(self, encoding="utf-8", errors="strict"):
        return self.value.decode(encoding, errors=errors)

    def __str__(self):
        return self.decode()

    def __getattr__(self, attr):
        return getattr(self.value, attr)


class String(ctypes.c_char_p):
    @classmethod
    def _check_retval_(cls, result):
        value = result.value
        return value if value is None else _wraps_c_char_p(result, value)

    @classmethod
    def from_param(cls, obj):
        if isinstance(obj, str):
            obj = obj.encode("utf-8")
        return super().from_param(obj)

[mara004]

Another improvement that comes to my mind for autostrings would be making the kind of encoding configurable.

e.g. pdfium mostly uses UTF16LE, so autostrings with this might actually be convenient for pypdfium2, though formally a default encoding remains a problem - it would still be a concern with any APIs that use other encodings, like UTF-8 or ASCII.