Inconsistent plugins development documentation

[speller]

I'm trying to add a custom plugin to the postgraphile:http:handler event.
Accordint to the https://www.graphile.org/postgraphile/plugins/#enabling-via-postgraphilercjs section, I'm adding it to the plugins rc file section.
According to the https://www.graphile.org/postgraphile/plugins/#writing-your-own-plugins section, I've created my plugin stub:

const CorsPlugin = {
  ['postgraphile:http:handler'] (req, p2) {
    console.log(req);
    console.log(p2);
    return req;
  }
}
module.exports = CorsPlugin

But when running, I'm getting the following error log:

/postgraphile/build-turbo/postgraphile/cli.js:300
throw new Error(`No plugin found matching spec '${name}' - expected function, found '${typeof plugin}'`);
^
Error: No plugin found matching spec '/postgraphile/CorsPlugin' - expected function, found 'object' at /postgraphile/build-turbo/postgraphile/cli.js:300:19

Please update these docs on how to write own plugins.

Also, there's no documentation on how to use the existing CORS plugin example with the rc file to be able to adapt it to my needs. Plugins documentation is overall messy and it's hard to understand what's going on and how to catch anything without a long time googling, discord discussions, or github issues.

My main pain points during all my Postgraphile experience are:

1. What type of plugins exists?
2. What's the difference between plugin types?
3. How plugins and hooks relate?
4. What types of hooks exist?
5. Which hooks can be set up from which plugins?
6. Where to see plugins and hooks parameters documentation?
7. How to convert plugins and hooks declaration/connection between different types (middleware, library, rc file, CLI)? If I found an example of a plugin/hook I spend a lot of time trying to adapt it to rc file.
8. When I see a comment like use postgraphile:http:handler to intercept the result if it's for us used by Benjie everywhere I see a wall of hours of investigations how to implement it in the reality. Especially in my deployment way (with an rc file).
9. A single entrypoint of plugins/hooks documentation with a clear table of contents would be very useful. Like this: https://www.easypost.com/docs/api . Examples of plugins with the different types of connection (library/rc file/cli) like code examples in different languages also would be super useful (here https://www.easypost.com/docs/api it is implemented as a dropdown list, some sites use tabs: https://cloud.google.com/tasks/docs/reference/rest/v2/projects.locations.queues.tasks/run).

I like Postgraphile a lot, but every time I need to do something by my own plugins I feel pain... :(

[benjie]

Please share your .postgraphilerc file; the error you have indicated is related to the appendPlugins/etc options (which are for schema plugins), however the documentation you've linked to is for server plugins and uses the plugins option without any prefix. I suspect you've put appendPlugins rather than plugins in your postgraphilerc. I believe the documentation is consistent on this front (certainly the places you linked to seem to be), but if you find an inconsistency please point it out.

Plugins documentation is overall messy and it's hard to understand what's going on and how to catch anything without a long time googling, discord discussions, or github issues.

As noted in the Server Plugins documentation, server plugins are experimental; this is why there is not complete documentation for them yet - you need to dive into the code to get a feel for what they do and what arguments they accept. Plugins are for fiddling with some of the deepest internals of PostGraphile's server/CLI; most people don't need to write their own server plugins so it's not a part of the API/documentation that's received much attention, it's currently expected that people writing them are intimately familiar with PostGraphile's internals. Here's another example of a server plugin: https://github.com/graphile/persisted-operations

[speller]

Hi Benjie, thank you for your answer. You're right, I accidentally put my plugin to appendPlugins. Docs are correct about this matter.

Regarding everything else: when I want to implement my custom CORS plugin - should I know anything about Graphile internals? It is a simple web server plugin. But it has a lot of blind spots and it's not easy to implement an easy plugin.

I knew what's the difference between Schema and Server plugins. But after a few months, I saw on the config parameters' names plugins and append_plugins and had no idea what are each of them.

You're asking for deep diving into the sources, but deep-diving into a very new, large, and difficult project is nonsense because you don't understand anything. You need to spend months to get a minimum familiarity. Just for a plugin? The project is split into many packages and you can't just clone and explore, instead, build is required. Building the whole project for learning how to create a simple plugin, really? A mix of TypeScript and plain JavaScript also doesn't bring more understanding. The code is made from millions of callbacks which are the worst ever for reading and understanding anything. I don't mean the code is wrong, just its reading and understanding is a pain. I also just found that there are two types of server plugin hooks declaration: ["cli:library:options"](options, { config, cliOptions }) and "postgraphile:options"(options) - what does it mean? Too many questions.

I'm not blaming you for anything, I really appreciate your efforts in developing this amazing tool. But, again, every time I touch plugins, it's a big pain and time waste. And reading sources is not a replacement for the documentation. I just vote for better and clearer documentation.

[speller]

Returning back to plugins. I need a health-check endpoint. I google this ticket: graphile/crystal#1098.
You mentioned there that it is possible to implement it using v4 plugins. I followed your comment and made this plugin:

const HealthPlugin = {
  ['postgraphile:http:handler'] (req, {options, res, next}) {
    if (req.path === '/pg-health') {
      res.send('HTTP Server: OK')
      return null
    }
    return req
  }
}

But it doesn't work, Express responds with the following:

Cannot GET /pg-health

So how to implement a health-check endpoint using a plugin?

[benjie]

Regarding everything else: when I want to implement my custom CORS plugin - should I know anything about Graphile internals? It is a simple web server plugin. But it has a lot of blind spots and it's not easy to implement an easy plugin.

This is much better served by having a (e.g. Express) middleware before PostGraphile rather than writing a PostGraphile plugin. That's the intended development pattern.

I also just found that there are two types of server plugin hooks declaration: ["cli:library:options"](options, { config, cliOptions }) and "postgraphile:options"(options) - what does it mean? Too many questions.

They're the same; that's just JavaScript syntax differences.

A mix of TypeScript and plain JavaScript also doesn't bring more understanding.

The v5 branch already has everything standardized on TypeScript: https://github.com/graphile/postgraphile/tree/v5 however it's a bit behind v4 right now in other ways. Moving over to TypeScript everywhere would be a breaking change so it has to wait for a major release I'm afraid.

I need a health-check endpoint. I google this ticket: graphile/crystal#1098. You mentioned there that it is possible to implement it using v4 plugins.

This probably better served by using an Express (or Koa, etc) middleware, unless you really need it to represent PostGraphile internal states.

Express responds with the following: Cannot GET /pg-health

Are you mounting PostGraphile via a subpath or something? e.g. app.use("/something", postgraphile(...)). Maybe you should log out req.path in that plugin to see why it's not matching /pg-health as you'd expect.

[speller]

This probably better served by using an Express (or Koa, etc) middleware, unless you really need it to represent PostGraphile internal states.

Is there any complete example? Because I use Postgraphile Docker image and then add my rc file tuning the existing working build. I expect that Docker image could be tuned almost freely without rewriting entrypoints.

Are you mounting PostGraphile via a subpath or something? e.g. app.use("/something", postgraphile(...)). Maybe you should log out req.path in that plugin to see why it's not matching /pg-health as you'd expect.

I'm using Docker image and don't change default paths. When I add this line: console.log('req.path:', req.path) it logs the following: req.path: undefined while this field should be provided by the express framework: http://expressjs.com/en/api.html#req.path .

[speller]

Using logs, I found that the req and res objects have no relation to Express, they are purely internal. I can use res.url but it returns the full URL with query parameters. I think there should be a way to get a path without query parameters. Also, the res.send() is undefined. How to return custom content then?

[speller]

Threr's no Express under the hood on Postgraphile, I was misled by my previous experience that I was working with Express only in Node. It's the standard Node ServerResponse object.

Such things should be described in the documentation. Plugins creation is simple when you know what you do with the documentation. I'm not sure it's the best way to require a strong Postgraphile experience from plugin writers.

Is it possible to make such a good documentation in the future to allow plugin writers to get all the information right in-place instead of spending days of googling and filling issues on GitHub?

[benjie]

Threr's no Express under the hood on Postgraphile, [...] Such things should be described in the documentation.

I'm not sure how to make this clearer. At no point have I said that PostGraphile or the PostGraphile docker image bundles express, and I explain the various frameworks that can be used at the very top of the library usage page: https://www.graphile.org/postgraphile/usage-library/ but the docker image effectively uses the CLI which does not use/need any framework. Suggestions of specific edits are welcome, but I really don't know how to address your feedback here.

Using logs, I found that the req and res objects have no relation to Express, they are purely internal.

That's not strictly true; if you are using Express then they will be the Express req/res objects. If you're using Fastify they'll be the Fastify req/res objects. It happens that you're not using any framework, so they're the Node HTTP req/res objects.

I expect that Docker image could be tuned almost freely without rewriting entrypoints.

The Docker image is like the CLI, once you want to do something more advanced we recommend that you roll your own with the library - see the usage documentation where we make this recommendation, and the Running PostGraphile as a library in Docker Guide for how to use PostGraphile as a library in Docker.

Is it possible to make such a good documentation in the future to allow plugin writers to get all the information right in-place instead of spending days of googling and filling issues on GitHub?

I hope so! Filing a pull request against the documentation is the best way to help in this effort - it doesn't have to be perfect, I always edit it anyway. You'll find a "Suggest improvements to this page" link at the top of each website page, clicking this will lead you through everything you need to do, all without leaving GitHub's web interface. When you find information that's misleading (for example implying that the Express framework exists in the Docker image?), please send a PR that fixes that text.

Other than the thin documentation around Server Plugins themselves (which I've already noted is because they're still experimental, and they're not intended for most users to use; even your use cases are much better served via standard Node.js middleware) I don't know how to address your other feedback. Pointing out specific places in the documentation where things should be noted, or where text is confusing/ambiguous would be very helpful.

[benjie]

As a first step to addressing this, I've added a bold note to recommend users use middleware instead of server plugins where possible, and added another couple plugin examples. d110740

[speller]

Thanks!

On another note, I would suggest remaking the documentation page structure. Currently, it's difficult to quickly get where I am if I came from Google:

The issues are:

• First-level navigation is on the right. It is often expected on the left.
• Second level navigation is seen before the first one and visually separated. Moreover, it looks like first-level navigation.
• After seeing the brighter second-level navigation and perceive it as a first-level one, I see the real first-level navigation on the right from it and my mind is blowing, I'm lost in the navigation.
• From looking on the page top, it's impossible to understand where I am in the documentation, I can't see a path. This also leads to difficulty in searching for other relevant topics.

All of these make the information from the docs in my mind unstructured and messy. I can't easily orient there and quickly pull required information. I feel resistance when I think that I need to go back to docs and find something. I can't easily remember where to go to find a page that I visited.

I would suggest implementing something organized similar to Google documentation: https://cloud.google.com/tasks/docs/reference/rest/v2/projects.locations.queues.tasks/run