Keep internal documents out of the PGXN search index and document the PGXN install (#373)

* Keep internal documents out of the PGXN search index and document the PGXN install

Addresses items 2 and 3 of \#359 together, because they interlock: the README is
what PGXN renders as the distribution landing page AND is itself an indexed
document, so adding install docs and changing what gets indexed have to be
verified against each other.

## no_index (item 3)

PGXN indexes documentation for full-text search. Both published releases index
69 documents, 19 of which are internal: all seven files under prompts/
(including the release runbook that describes our own PGXN publishing
procedure), .github/copilot-instructions, the .agents SQL-generation skill,
three open-problem notes, the repo TODO, the website sources, and two
requirements.txt pip manifests, which Text::Markup treats as documents because
of the .txt extension.

Nothing here is secret; the repository is public. The problem is signal: a
reader searching PGXN for pg_durable currently finds our internal AI
development prompts and a list of our open bugs presented as product
documentation.

This compounds. A PGXN release is immutable, so 0.2.6 and 0.2.7 keep their
69-document indexes permanently, and every further release published without
no_index adds another.

The PGXN Meta Spec has a field for exactly this, and it is implemented in
pgxn-api lib/PGXN/API/Indexer.pm find_docs(). It affects only indexing and
search, so the release archive and the GitHub source assets stay byte-identical.
.gitattributes export-ignore would also have suppressed these files, but it
removes them from the tarball entirely and would silently change the GitHub
release archives built by package-release.yml as well.

Two matching rules worth knowing before editing the list:

  - file entries are exact string equality INCLUDING the extension, while the
    API's docs keys are extension-stripped. Copying a key from
    api.pgxn.org/dist/pg_durable.json does not work: docs/dep_issues must be
    written docs/dep_issues.md.
  - directory entries are an anchored literal prefix match (/^\Q$_/), not a
    path-component match, so "docs" would also match a hypothetical docs2/.
    The trailing slash is deliberate.

provides.pg_durable.docfile is collected before the skip checks run, so
USER_GUIDE.md cannot be excluded by accident. README has no such protection --
its skip check runs before PGXN's README special case -- so it is deliberately
absent from the list.

Verified by reimplementing find_docs() against the tracked file list: the
indexed set drops from 69 to 50, all 19 internal entries are excluded, zero
residual prompts/ .github/ .agents/ or docs/website/ entries remain, and
README, USER_GUIDE, CHANGELOG, LICENSE, SECURITY, CONTRIBUTING and
CODE_OF_CONDUCT all survive.

Left indexed deliberately: the docs/spec-* set, ARCHITECTURE, api-reference,
grammar, and all 13 examples/ documents. The security review, the design and
proposal documents, the test plans, and docs/pg_durable_mvp are arguable either
way and are left alone for now; unlike a release, this list is not immutable
and can be tightened later at no cost.

## Install documentation (item 2)

PGXN renders README.md as the distribution landing page, so a PGXN visitor is
already reading it. Until now the entire treatment was one sentence at the end
of the Packages section saying PGXN carries the source distribution "built and
installed exactly as described above" -- which never showed the pgxn command,
and pointed at the tarball recipe, a different entry point.

The new section states plainly that PGXN ships source rather than a binary,
lists the prerequisites including the compile time, and names the pgxn install
and pgxn uninstall commands. It also documents the auto-init behaviour added in
\#370 and the make pgrx-init / make install-pgrx targets that go with it.

All links in the new section are absolute. Relative links do not resolve when
the README is rendered on pgxn.org -- pgxn.org/dist/pg_durable/USER_GUIDE.md is
a 404 -- so a relative link would be broken for exactly the readers this section
is written for.

USER_GUIDE.md is the provides docfile and PGXN links it as "Documentation", but
its Prerequisites began after installation, with shared_preload_libraries. It
now starts by naming the install channels and linking to Packages, with an
absolute URL for the same reason.

Refs \#359 (items 2 and 3)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Correct the pgxn install command: --sudo and the -- separator are both required

Review of \#373 caught that the documented `pgxn install pg_durable` fails on the
exact platform the section names. Confirmed three ways.

pgxnclient elevates only when told to. In commands/__init__.py the --sudo option
is declared with const='sudo' and nargs='?' but NO default=, so argparse leaves
opts.sudo as None when the flag is absent, and SudoInstallUninstall.run() then
raises before building:

    ERROR: PostgreSQL library directory (/usr/lib/x86_64-linux-gnu) not
    writable: you should run the program as superuser, or specify a 'sudo'
    program

Reproduced as a non-root user against a PGDG PostgreSQL 17: exit 1, no build
attempted. Upstream docs/usage.rst says the same.

The obvious correction, `pgxn install --sudo pg_durable`, is also broken. Because
--sudo takes an optional PROG, argparse consumes pg_durable as that argument and
leaves no distribution to install:

    $ pgxn install --sudo pg_durable
    usage: pgxn install [--help] ... [--sudo [PROG] | --nosudo]
    exit 2

The separator is therefore mandatory here, not optional as upstream's note
suggests, because --sudo is always the last option before the distribution name:

    pgxn install --sudo -- pg_durable

Verified end to end as a non-root user with PostgreSQL 17 from PGDG and no
~/.pgrx: exit 0, pg_durable.so installed root-owned into
/usr/lib/postgresql/17/lib, while ~/.pgrx stays owned by the building user --
confirming the build runs unprivileged and only the install step is elevated,
which is what the text now says. uninstall takes the same flags.

Also replaces the cargo-pgrx prerequisite. It pointed at `make install-pgrx`,
which needs a source checkout that a PGXN user does not have -- `pgxn install`
downloads the source itself -- and never named the pinned version. It now gives
the direct cargo install command, and mentions the make targets afterwards as
the checkout-based alternative.

Why the original text was wrong: Install._inun calls
run_make('install', sudo=self.get_sudo_prog()), which reads as unconditional
elevation. get_sudo_prog() returns opts.sudo, whose default is None. The
end-to-end validation missed it because it ran as root, where
is_libdir_writable() is true and the sudo branch never executes.

Refs \#359 (item 2)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Pino de Candia <pinod@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: 3 people

META.json.in

@@ -47,5 +47,23 @@
       "background worker",
       "pgrx",
       "rust"
-   ]
+   ],
+   "no_index": {
+      "directory": [
+         ".agents/",
+         ".github/",
+         "prompts/",
+         "docs/website/"
+      ],
+      "file": [
+         "TODO.md",
+         "docs/CODESPACES_PREBUILDS.md",
+         "docs/DUROXIDE_PG_DEADLOCK_ISSUE.md",
+         "docs/dep_issues.md",
+         "docs/http_problems.md",
+         "docs/loop_rework_problems.md",
+         "examples/azure-functions/function-app/requirements.txt",
+         "examples/invoice-approval/function-app/requirements.txt"
+      ]
+   }
 }

README.md

@@ -159,7 +159,58 @@ on the build command to enable an HTTP policy feature. `DESTDIR` may be set on
 `sudo make uninstall PG_CONFIG="$PG_CONFIG"` removes the installed files again.
 It needs no build, so it also works from an unbuilt source tree.
 
-The extension is also distributed on [PGXN](https://pgxn.org/dist/pg_durable/), the PostgreSQL Extension Network. PGXN carries the **source** distribution, built and installed exactly as described above, so it needs the Rust toolchain and cargo-pgrx. For prebuilt binaries use the Debian packages or the Docker image.
+### Installing from PGXN
+
+The extension is listed on [PGXN](https://pgxn.org/dist/pg_durable/), the
+PostgreSQL Extension Network. **PGXN carries the source distribution, not a
+binary**: `pgxn install` downloads the source and compiles it on your machine,
+so it needs the same toolchain as a source-archive build and takes several
+minutes. For prebuilt binaries use the Debian packages or the Docker image
+above.
+
+Prerequisites:
+
+- PostgreSQL 17 or 18, including development headers and `pg_config`
+  (`postgresql-server-dev-17` on Debian/Ubuntu)
+- A Rust toolchain — see [rustup](https://rustup.rs)
+- [pgxnclient](https://pgxn.github.io/pgxnclient/) (`pip install pgxnclient`)
+- cargo-pgrx, matching the `pgrx` version pinned in `Cargo.toml`:
+
+  ```bash
+  cargo install --locked cargo-pgrx --version 0.16.1
+  ```
+
+Then, for a PostgreSQL installed from a package:
+
+```bash
+pgxn install --sudo -- pg_durable
+```
+
+Both parts of `--sudo --` are load-bearing. `pgxn install` elevates only when
+told to, so without `--sudo` it stops before building:
+
+```
+ERROR: PostgreSQL library directory (...) not writable: you should run the
+program as superuser, or specify a 'sudo' program
+```
+
+The build itself still runs as your user; only the install step is elevated. The
+`--` separator is required because `--sudo` takes an optional program name and
+would otherwise swallow `pg_durable` as that argument, leaving no distribution to
+install. If `pg_config --libdir` is writable by your user — a PostgreSQL you
+built yourself, for instance — plain `pgxn install pg_durable` works.
+
+`make package` registers your PostgreSQL with cargo-pgrx automatically the first
+time, so no separate `cargo pgrx init` step is needed. From a source checkout you
+can also run `make install-pgrx` to install the pinned cargo-pgrx, or
+`make pgrx-init PG_CONFIG="$PG_CONFIG"` to register PostgreSQL explicitly; set
+`PGRX_AUTO_INIT=0` to make the build report the command to run instead of
+initializing on its own.
+
+Afterwards, add `pg_durable` to `shared_preload_libraries`, restart PostgreSQL,
+and run `CREATE EXTENSION pg_durable` as described above.
+
+`pgxn uninstall --sudo -- pg_durable` removes the installed files again.
 
 ## Development Installation
 

USER_GUIDE.md

@@ -68,9 +68,13 @@ pg_durable enables you to define and execute **durable SQL functions** entirely
 ### Prerequisites
 
 pg_durable requires:
-1. **PostgreSQL configuration**: Add `pg_durable` to `shared_preload_libraries` in `postgresql.conf`
-2. **Server restart**: Required after modifying `shared_preload_libraries`
-3. **Extension creation**: Run `CREATE EXTENSION pg_durable` in your database
+1. **An installed extension**: a Debian package, the Docker image, a source
+   build, or `pgxn install pg_durable`. See
+   [Packages](https://github.com/microsoft/pg_durable#packages) for the
+   available channels and their prerequisites.
+2. **PostgreSQL configuration**: Add `pg_durable` to `shared_preload_libraries` in `postgresql.conf`
+3. **Server restart**: Required after modifying `shared_preload_libraries`
+4. **Extension creation**: Run `CREATE EXTENSION pg_durable` in your database
 
 ### Enable the Extension