docs: warn that connector configs are stored in plaintext, and document secrets provider behaviour

[david-streamlio]

Rewritten. This PR originally documented secrets, custom_runtime_options and the remaining source/sink arguments as README property tables. Commit f8319dc has since replaced those tables with links to the generated docs, which made the original diff both conflicting and largely redundant. See "What changed and why" below.

Motivation

secrets has been supported on pulsar_source and pulsar_sink since v0.2.0 (#96 / #97), but nothing in the provider explains why it exists.

The generated docs say what it is — "The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider" — and stop. A user reading that has no way to learn the thing that actually matters: credentials placed in configs are stored and returned in plaintext, through the function metadata topic, pulsar-admin sinks get / sources get output, and Terraform state. The natural thing to do — put the password in configs, because that is where the connector's other settings go — is the wrong thing, and nothing says so.

There is a second, sharper problem. {"path": ..., "key": ...} references are resolved only by a runtime whose secrets provider understands them, such as the Kubernetes runtime's KubernetesSecretsProviderConfigurator. Under the default ClearTextSecretsProvider — process and standalone runtimes, including the cluster this repository's own acceptance tests start via make run-pulsar-in-docker — terraform apply succeeds and the connector silently receives no value for the secret. A user who follows the advice to move credentials into secrets, and tests it locally, gets a green apply and a connector that cannot authenticate, with nothing anywhere to explain it.

Neither fact is documented anywhere in the repository today: grep -riE "plaintext|ClearTextSecretsProvider|secrets provider" docs/ README.md returns only the two auto-generated description lines above.

Modifications

• Extend the schema Description strings for configs and secrets on both pulsar_source and pulsar_sink to carry the plaintext warning and the ClearTextSecretsProvider caveat. Putting the guidance in the schema rather than in prose means it reaches the generated docs automatically and cannot drift away from the code.
• Regenerate docs/resources/sink.md and docs/resources/source.md with go generate ./....
• Add a worked example to examples/sinks/main.tf showing the password moved out of configs and into secrets, with the runtime caveat noted inline.

What changed and why

The original diff added README property-table rows for 21 arguments across the two resources. All 21 are now published from the schema in docs/resources/{sink,source}.md — I checked each one — so re-targeting those rows would have duplicated content that already exists, in a file that no longer holds tables.

What was not covered anywhere was the security guidance, which was the reason this PR was opened. That is all this now contains.

Verifying this change

• Make sure that the change passes the CI checks.

This change is documentation only and carries no test coverage of its own. Verified as follows:

• go generate ./... reproduces the committed docs/ output; the only diff is the four description strings.
• make test and golangci-lint pass.
• terraform fmt -check -recursive examples/ is clean.

Documentation

Check the box below.

Need to update docs?

• doc-required

• no-need-doc

• doc

  Schema descriptions updated and docs/resources/{sink,source}.md regenerated.

[github-actions]

@david-streamlio:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

[github-actions]

@david-streamlio:Thanks for providing doc info!

[david-streamlio]

@maxsxu @freeznet Friendly ping when you have a moment — this is a docs-only PR (README + examples) documenting secrets and the other already-supported source/sink arguments. All checks are green and the doc label is set. Happy to adjust anything.

[freeznet]

Thanks for this — the table gaps were real and the secrets-over-plaintext-configs guidance is the right thing to document.

I verified the PR's central claim mechanically rather than by eye, and it holds:

• Completeness ✅ — extracted every resourceSource*Key / resourceSink*Key constant from pulsar/resource_pulsar_source.go and pulsar/resource_pulsar_sink.go and diffed against the two README tables on this branch: zero missing, zero extra top-level arguments in either (after excluding the 5 nested input_specs sub-keys).
• Descriptions ✅ — all new rows match the schema Description strings verbatim, so README and generated docs/resources/*.md agree. Agreed no tfplugindocs run needed.
• Required column ✅ — every new row is correctly False. The only Required: true attributes are tenant/namespace/name/archive/destination_topic_name (source) and tenant/namespace/name/cleanup_subscription/archive/auto_ack (sink).
• Secret shape ✅ — the documented { "path": ..., "key": ... } object matches what the acceptance tests already exercise (resource_pulsar_source_test.go:254, resource_pulsar_sink_test.go:237).
• No perpetual-diff risk ✅ — I ran the snippets through terraform validate/apply: jsonencode emits compact key-sorted JSON, which matches what the read path writes back via json.Marshal(map[string]interface{}). Both the README (:) and examples (=) forms parse fine.

Two things I'd like changed before merge, both in the examples rather than the tables:

1. The gsaKey secret in the source example doesn't correspond to anything the connector reads. Both source examples use pulsar-io-file with configs = {"inputDirectory":"opt"} — a filesystem source with no credentials at all. There was no plaintext credential to move, and the file connector never reads a gsaKey. Details inline.
2. examples/ should stay runnable against the repo's own local cluster. hack/pulsar-docker.sh starts a plain Pulsar standalone (default DefaultSecretsProviderConfigurator → ClearTextSecretsProvider, process/thread runtime). The {path, key} indirection is KubernetesSecretsProviderConfigurator-specific — as your README note says. terraform apply still succeeds (Pulsar stores the map without resolving it at submit time), so this fails silently: the JDBC sink just gets no password at runtime. Details inline.

The rest are nits — take or leave.

One follow-up worth a separate issue, since this PR raises the security framing: configs isn't marked Sensitive: true in either schema, so its value is echoed in plan output and stored readably in state regardless of the secrets guidance. Closing that is a code change, and I'd rather not widen a docs PR into it — but the note here would be strictly stronger once it lands.

[freeznet]

Nit — style drift from the convention already in the tree. examples/functions/main.tf:61 writes this as:

  secrets = jsonencode(
  {
    "SECRET1": {
       "path": "sectest"
       "key": "hello"
    }
  })

i.e. the paren on its own line and : separators — which is what your README snippets use. The two new blocks in examples/sinks/main.tf and examples/sources/main.tf use inline jsonencode({ with = instead. Both parse (I checked with terraform validate), so purely cosmetic, but worth matching the functions example so all three read the same.

[freeznet]

Nit, purely cosmetic: the long new keys (max_pending_messages_across_partitions, use_thread_local_producers, crypto_key_reader_classname, and negative_ack_redelivery_delay_ms in the sink table) overflow the padded column widths the rest of the table uses, so the raw source goes ragged. Renders identically on GitHub — only worth a reflow if you want future diffs to stay clean.

[david-streamlio]

Thanks for the thorough review! Addressed in 068acd5: dropped the gsaKey secrets block from the source examples (the file connector reads no credentials), restored the working password in both sink examples so they run against the local standalone cluster, and moved the secrets form to a commented-out alternative with a note that {path, key} references require a resolving SecretsProvider (Kubernetes runtime). Also picked up the type-hint nits and strengthened the credentials note. Skipped the table-padding reflow to keep this diff clean — happy to do it as a follow-up.

Agreed Sensitive: true on configs deserves its own issue.

[david-streamlio]

Filed #213 to track marking configs as Sensitive: true, per the review discussion above.

[david-streamlio]

Rewritten and rebased onto current master — this is now a fresh diff on the same branch, so it is worth re-reading rather than skimming the previous review context.

The original version documented secrets, custom_runtime_options and the remaining source/sink arguments as README property tables. Commit f8319dc replaced those tables with links to the generated docs, which left this PR both conflicting and mostly redundant: I checked all 21 arguments it documented, and every one is now published from the schema in docs/resources/{sink,source}.md.

What was never covered anywhere is the security guidance this PR was opened for — that credentials in configs are stored and returned in plaintext, and that {path, key} secret references are silently ignored under the default ClearTextSecretsProvider, so apply succeeds while the connector gets no value. That second one bites anyone who follows the advice and tests it against a standalone cluster, including the one make run-pulsar-in-docker starts.

So the guidance now lives in the schema Description strings, where tfplugindocs picks it up and it cannot drift from the code, plus a worked example in examples/sinks/main.tf. Net diff is 5 files, +32/-8, down from the previous README-heavy version.

Now MERGEABLE.