Adopt a shared specification for api structure

[tristanlatr]

Hello there.

In a general effort to make the python api documentation ecosystem a bit more standardized, I think documentation generators should adopt a common standard for serializable api structure of a collection of modules. This will serve the python libraries developers at large since the separation in between the backend parser and the front end renderer would become standard accros generators. This way developers will be able to choose their preferred backend/frontend combination that fits their package the best.

Currently mkdocstring is working with this infrastructure since the backend can be changed.
Whereas pydoctor (but I’ll definitely work on that in the coming months).
The sphinx-apidocs project is also pretty close.

These three packages covers the API documentation needs for more or less any project: using Sphinx, Mkdocs or simply standalone API.

So I wanted to start this discussion with maintainers of these three packages to think and settle on a common standard.
The docspec standard exists but I believe it needs to be extended a little bit to fit everyone usage.

Tell me what’s you think :)

[pawamoy]

Hey @tristanlatr!

Yes, it's a good idea to share about this between maintainers 🙂 Thanks for reaching out!

Indeed, the docspec specification is too limited for Griffe right now. I think the most important thing to standardize is the serialized data, as you mentioned it. It must provide all the necessary information to inflate/hydrate back into any language. Then each language and respective library can choose to provide more methods, or different methods.

Griffe's own API is serialized and exposed here: https://mkdocstrings.github.io/griffe/griffe.json. It's updated everytime I deploy its documentation. If others want to share a similar JSON file, we can start comparing and maybe suggest incremental standardizations 🙂

Due to the discussion nature of this issue, I'll convert it to a discussion.

[tristanlatr]

Thanks for your answer,

I totally agree with your approach! Before I rush into generating a JSON from the pydoctor models, I believe we should better define the general structure of the data. Also in order for document generators to support multiple backend, we should settle on an interface for generating the data, with a set of standards options that might be supported by a parser.

What are the principals missing features from the docspec standard for it to cover the griffe informations ?

Also the responsibilities of the parser should be written down so we know what is to be implemented on the backend side.
For instance, my point of view is that the docstring parsing belongs to the frontend. What do you think ?

[pawamoy]

we should settle on an interface for generating the data, with a set of standards options that might be supported by a parser.

👍

What are the principals missing features from the docspec standard for it to cover the griffe informations ?

I'll have to check docspec again, let me save this notification.

For instance, my point of view is that the docstring parsing belongs to the frontend. What do you think ?

I think we can tolerate both. Maybe we don't want to ask all front-end tools to reimplement Python docstring style parsers. But if they want to, they're free to do so, also to support their own custom styles maybe. Same for backends: if they want to support parsing docstrings, they should be free to do so, but they should allow users to opt-out (or rather opt-in). Of course that raises the question of how to standardize parsed docstrings data, which is probably a hot but invaluable topic 🙂

[pawamoy]

What are the principals missing features from the docspec standard for it to cover the griffe informations ?

I will go through the spec and write down the differences I notice. Tagging @NiklasRosenstein too 😄

---

Struct Location

• filename: Griffe provides both the relative path from the current working directory, and from the top-level package path (independently of the CWD). It could probably try to provide the relative path from the Git repository root too, by iterating on the file-system to find the closest .git folder 🤔
• lineno is required, but objects loaded from builtin or compiled modules do not have line numbers

Struct Docstring

• Griffe stores line numbers of the docstring too. This can be useful to front-end if they want to render the source of a function or class without the docstring.
• Griffe stores a reference to the parent (function, class, module, attribute) in the docstring, but I suppose it's more an implementation thing than a spec one.
• Griffe allows each docstring to have its own style by assigning a parser name/instance. I suppose this can be delegated to front-ends or implementations.

Struct Indirection

• Griffe calls that Alias. I actually like Indirection better 🤣
• not specific to indirections themselves, but general to any "API object", Griffe stores a runtime boolean that tells if the object or indirection is available at runtime. Objects/indirections that are type-guarded behind if TYPE_CHECKING for example will have a runtime value set to False. Picking up these objects/indirections not available at runtime is useful for the cross-reference feature in front-ends: even though the object is not available, it is used as a type annotation, which the front-end might want to display with a link pointing at the actual documented object.
• Griffe reuses the concept of indirection - lets use alias for this paragraph - for members inherited from parent classes. I suppose it's quite specific to the implementation, but here is what Griffe does: each Class object has properties/methods to compute inherited members by iterating on base classes, using Method Resolution Order and C3Linearization (that I stole from you by the way @tristanlatr ❤️). Each inherited member is returned as an Alias pointing to its actual parent member. Since aliases can be both imported objects and inherited members, they have an inherited attribute to tell them apart. This attribute is also used by front-ends to tell regular members from inherited members apart.

Struct Variable

• Griffe calls them Attribute. Digression: I don't think either Variable or Attribute are great names, but I can't think of a better one. In Python, everything is a variable, and almost everything is an attribute. Sometimes, Python won't let us re-assign this variable/attribute. But it's one nonetheless. Then these variables/attributes specify into different kinds of objects. I chose to have only Module, Class, and Function kinds, but other implementations could choose to have many more. Griffe uses "labels" for further specification.

• datatype: I find the description unclear:

  The name of the type of the variable or constant.

  I would need examples here to understand what this means. Attributes/variables can be annotated with complex types.

• value is a string, like datatype by the way, but in practice, if you want powerful cross-referencing abilities in front-ends, you have to store some kind of AST (or even CST) for values and types.

• modifiers: Not Python specific. I would prefer following a spec that is Python specific, and only later, if more similar specs are created for other languages, and we actually see that they can converge to a unified model, then follow this unified spec. In Griffe, these modifiers would be labels.

• semantic_hints are labels too in Griffe

Struct Argument

• Called Parameter in Griffe. The difference IIUC is that the parameter is the "slot", while the argument is the passed value.
• location: we don't store the location of parameters, because the location is often used to retrieve the source, but the source of just one parameter will often produce invalid syntax. Typo in the description by the way? It mentions "decoration".
• type: we're compatible, we just use different names, and we call it kind
• datatype: a bit confusing, maybe value_type would be less confusing. In Griffe we store it as annotation.
• again, for the type annotation and the default value, simple strings are not enough for proper cross-references
• Griffe has an additional wrapper class called Parameters, which allows access using both index and name of parameters, as well as iterating on them.

Struct Decoration

• Griffe doesn't have such structs. Decorators are parsed as "expressions" (a kind of AST) and stored like that, in a decorators attribute on classes and functions. Being an expression, we have lots of powerful methods on decorators to get their corresponding callable canonical path, their arguments, etc.

Struct Function

• we're quite compatible. Same remarks as above for strings vs. expressions for cross-references, and for modifiers and semantic hints vs. labels.
• Griffe stores property setters and deleters, as well as overloads, in attributes of the same name

Struct Class

• Griffe doesn't store the metaclass 🤔 maybe it should. I think it doesn't differentiate between base classes and metaclass. They are all stored as expressions again.

Struct Module

• we're quite compatible.

---

I don't see anywhere in the spec where Indirections are stored? I was expecting to find them in the accepted types for module and class members.

Griffe also stores a public attribute on each kind of object. It's not set directly by Griffe, but rather expected to be used by extensions that want to support different ways of marking objects as public than the conventions we have in Python.

[NiklasRosenstein]

Hi there, sorry for the late response. An apt analysis @pawamoy! From a glance this all seems quite accurate.

Docspec was originally intended to be language agnostic, but it likely falls short because I've never actually tried to parse another language and fit it into the model. There's no need for it to be language agnostic at this time really, so I don't mind focusing on reaching stable and consistent Python compatibility before anything else. I'd be happy to have us update the docspec specification/package to be more generalized and aligned with the needs for tools such as Griffe!

---

Re the unclarities you pointed out:

• I agree neither Variable nor Attribute are great names for what the struct represents though I still don't have a better suggestion from the top of my head
• datatype is effectively a synonym for "annotation" in Python
• In docspec, expressions (e.g. default values) are a string as the parsing into an AST would be left to the frontend (e.g. you can just call ast.parse() on the string if your frontend is implemented in Python). I chose this for two reasons (1) simplicity for its purpose in Pydoc-Markdown and (2) it would have made it extremely complex to cover Python expressions alone (re-modeling the entire AST) and nigh impossible to generalize for multiple languages.
• modifies were intended for syntactic elements that modify the object (e.g. async in Python, or private/public in other languages)
• Agreed that Arguments should be Parameters
• I chose to store "locations" on each kind of API object including parameters, because you could imagine docspec data being used as data source for an IDE for example and you might want precise locations for a parameter when jumping to its definition for UX. Very niche of course, but nothing really seemed to speak against storing it as the information is readily available to the parser.
• Decoration could totally be an Expression instead
• Indirections are stored as members on Modules for example:

  _Members = t.Union[Variable, Function, Class, Indirection]
  

[pawamoy]

Thanks @NiklasRosenstein!

Indirections are stored as members on Modules for example:

Thanks, that's what I was expecting indeed, and Griffe does the same thing, storing instances of Alias in members too. The spec doesn't explain that, that's why I was confused.

By the way, the type you show above doesn't match the spec, which shows List[Class | Variable | Function | Module] for module members. And that's quite a sensitive point: Griffe also store submodules in members. It can be a bit counter-intuitive, and doesn't really match what Python does behind the scenes, and it actually was an early design flaw, but I later decided to make this flaw "official".

By storing submodules as members too, Griffe is not able to distinguish between a member and a submodule of the same name, which Python supports. See the context in #124. We actually could support distinguishing members and submodules with the same name, by moving submodules out of members. But...

⚠️ wall of text below 😅 ⚠️

But there's the issue of location identifiers. What I call an identifier here, or path in Griffe, is for example a.b.C.d where a and b are modules, C a class and d an attribute. How do you distinguish a.b.c where c is a submodule, from a.b.c where c is a member (for example, a function)? Usually, we use a colon: a.b.c for the submodule, and a.b:c for the member.

But this syntax quickly shows its limitations. What about indirections? If a.b.c.d (module) is imported into a as d, and d has both a member and submodule e, how do you identify e through the a.d indirection? a.d.e for the submodule, and a.d:e for the member? But d is not actually a submodule of a, it's an indirection, and as such is stored in members. So shouldn't it be a:d.e for the submodule, and a:d:e for the member?

Why not, that would work, members are delimited with :, submodules are delimited with .. But then, when someone gives you the following identifier: a:d.e, and you are supposed to import it dynamically, how do you do that? You'd have to first import a, then get d (with getattr), then get the fully qualified name of d, add e to it and finally import e.

OK, that would work. Griffe's dynamic import code only supports first importing modules, then getting attributes, in this order, and nothing more. We could change it to support importing modules, then getting attributes, then resolve to fully qualified names, then importing modules again, etc.

However a:d.e doesn't really make much sense. Python won't let you import from a.d, so if submodule e is not exposed as a member in d (by being imported), it won't be accessible. But there's already a member named e in there...

This whole thing is very confusing (at least to me) and requires complex code, logic and reasoning to be supported, and there are no standard or official specification to help us here. All this just to support the case where both a member and a submodule have the same name.

In the end, I decided to recommend against having submodules and members with the same name, to keep things simple and only use dots in identifiers, and documented it here: https://mkdocstrings.github.io/griffe/best_practices/#avoid-member-submodule-name-shadowing.

[pawamoy]

I recently announced Griffe on discuss.python.org: https://discuss.python.org/t/griffe-signatures-for-entire-python-programs/66990, and linked to this discussion, which could be interesting to have there too ☺️

[AWhetter]

I'm not sure if this discussion is still active but I wanted to jump in and say that as the maintainer of sphinx-autoapi I'm also interested in a specification for API descriptions.
I don't have the capacity to contribute much to this initiative at the moment though.
In case it's useful I'll highlight that AutoAPI makes an internal mostly-JSON representation of the API that it's documenting currently. It's very specific to the needs of AutoAPI but maybe it could provide some inspiration for ideas.

[pawamoy]

Yes, please share any docs you have on your own format 😊 This will at least be one more data point to compare and find common ground if we ever want to tackle this.

[tristanlatr]

it's here: https://sphinx-autoapi.readthedocs.io/en/latest/reference/templates.html#

[tristanlatr]

Hi friends,

I had some energy to gather my thoughts into a more or less coherent proposal (very not exhaustive yet).
But first, a note about the process of adopting a common spec. What we are trying to do here is hard.
Mainly because we have to agree on a lot of things to move forward together. So in order not to loose
ourselves in details, I've listed the steps that we should be taking to succeed.

1/ Discussions:

• settle on a specification of the data structure (the output)
• settle on the specification of the messages send to the indexer/backend (the input)
• settle on some behavioural specification
  - what is required when a property is encountered ?
  - what to do when some names clashes ?
  - inherited members display as indirection on subclasses ?
  - etc...
• settle on a name for our great specification
• settle on a code of conduct, review process and license

2/ Production:

• create a repository to hold the information regarding the specification
• give the maintainers write permissions
• adopt the code of conduct and review process
• start sending PR to formalize and document the specification, respecting the review process
• make a json-schema so we can validate generated data
• write some tests that all indexers must pass
  • these checks the behavioural specification

So, now let's continue with the first step:
here is a proposal for a graph based approach for the API specification
of a collection of python modules. The graph based approach is best suited
to represent higher level relations in between any objects in the system; and will
enable further development by adding new kind of edges that would be otherwise harder
to represent in a tree-like structure.

It is freely inspired by griffe, docspec, pydoctor, sphinx-autoapi,
lsif and others.

It's totally a work in progress and I've probably forget some stuff. Don’t mind the naming, it’s more or less arbitrary.

Tell me what you think about this kind of graph bases approach

Thanks

Tristan

Vertexes types

• Idexable

  • uid, obscure type

• Symbol extends Idexable

  • name

• Location extends Idexable

  • filename
  • modulename
  • linenumber
  • end_linenumber
  • source_href

• Docstring extends Idexable

  • contents

• AST maybe Idexable - like standard library ast, but with optional uid property
  to link to their definitions. Serializable
  with something like a fork of json2ast and ast2json that would handle the uid attribute.

• Deprecation extends Idexable

  • replacement
  • since

Enumerations types:

• Modifier:

  • TYPE
  • ASYNC

• Kind:

  • MODULE
  • CLASS
  • FUNCTION
  • PARAMETER
  • ATTRIBUTE
  • INDIRECTION

• Semantics

  • Generic semantics

    • SPECIAL
    • WILDCARD_EXPOSED - EXPORTED
    • HIGHLY_DYNAMIC (when getattr is defined for instance or vars() is used as a setitem target)
    • TYPE_CHECK_ONLY (when a symbol is defined under TYPE_CHECKING blocks)

  • Module semantics

    • PACKAGE
    • NAMESPACE PACKAGE

  • Class semantics

    • INTERFACE
    • FINAL
    • ENUM
    • ABSTRACT
    • DATACLASS
    • EXCEPTION

  • Function semantics

    • METHOD
    • CLASS METHOD
    • STATIC METHOD
    • FINAL
    • GETTER / SETTER / DELETER
    • ABSTRACT
    • COROUTINE
    • NO_RETURN

  • Parameter semantics

    • SELF
    • CLS

  • Attribute semantics

    • PROPERTY
    • INSTANCE_ATTRIBUTE
    • CLASS_ATTRIBUTE
    • CONSTANT
    • FINAL
    • DESCRIPTOR
    • ALIAS
    • TYPE ALIAS
    • TYPE VARIABLE

• Visibility

  • HIDDEN
  • PRIVATE
  • PUBLIC

• ParameterKind

  • POSITIONAL_ONLY
  • POSITIONAL
  • POSITIONAL_REMAINDER
  • KEYWORD_ONLY
  • KEYWORD_REMAINDER

Edges types

Edges: An edge mental model should answer to
the question: "what is the {{edge name}} of this object".

Edges that can apply to any vertex:

• Locations - type: list of locations, required

Edges that can apply too all kinds of symbols:

• Kind - type: kind, required

• Visibility - type: visibility, required

• Semantics - type: list of semantics, optional

• Children - Parents - type: list of symbols, optional

• Modifiers - type: list of modifiers, optional

• Docstring - type: docstring, optional

• Decorators - type: list of ast, optional

• Deprecation - type: deprecation, optional

• Docsources - type: list of symbols, optional, useful for class members

(higher level edges)

• Overriden by - type: list of symbols, optional
• Implements - Implemented by - type: list of symbols, optional
• References - type: list of AST | Indirection | Attribute

Module edges:

• Submodules - type: list of module symbols, optional

Class edges:

• Raw Bases - type: list of ast, required if there are bases
• Bases - Subclasses - type: list of symbols, required if there are bases
• MRO - type: list of symbols, required
• Constructors - type: list of symbols, optional
• Raw Metaclass - type: ast, required if a metaclass is detected
• Metaclass - type: symbol, optional (of the metaclass can be resolved)

Class/Attribute edges:

• Type variables - type: list of symbols, optional
• Raw Type variables - type: list of ast, optional

Function edges:

• Overloads - type: list of function symbols (edge from implementation function to it's overloads)
• Primary - type: function symbol (edge from a overload to the implementation function - the last one defined )
• Return type - type: ast
• Parameters - type list of symbols

Parameter edges:

• Parameter kind - type: parameter kind
• Default value - type: ast

Parameter/Attribute edges:

• Annotation - type: ast

Attribute edges:

• Value - type: ast
• Getter - Setter - Deleter - type: symbol

AST/Indirection/Attribute edges:

• Definitions - type: list of symbols

Indirection edges:

• Origin module - type: module symbol
• Import code - type: ast, optional
• Inherited from - type: symbol, optional

[pawamoy]

Awesome @tristanlatr, thansk! I'll try to reply soon 🙂