BIP draft: Binary Output Descriptors

[seedhammer]

A BIP 174 PSBT may contain an extended key for deriving input and output
addresses. This document proposes an additional field for PSBTs to represent
arbitrary BIP 380 output script descriptors.

To support transfer of output descriptors outside signing flows, the proposal
makes the unsigned transaction optional.

[achow101]

I don't think PSBT is the right way to be doing this. Descriptors are not necessary in PSBT, and relaxing the invariant that a PSBT represents a transaction (by allowing a PSBT to include no transaction data) would break many parsers and also just is antithetical to PSBT. It's not a general purpose format for sending data around - you could use protobuf if you wanted that.

If the goal is to have a machine parsable representation of descriptors, then by all means, do that. But it doesn't need to (nor do I think it should) interact with PSBT, other than having a different header that identifies it.

It's also unnecessary to have a descriptor in a PSBT. A PSBT can contain all of the data to derive it from the scriptPubKey at a later time. If an updater has the descriptor in order to put it in the PSBT, then it can also put the rest of the data in the PSBT so that the descriptor can be inferred. I just don't see how this proposal is at all useful.

Also, descriptors are a textual format, and encoding text in binary is just not very efficient. There are better ways to do that if you really wanted to.

[seedhammer]

I don't think PSBT is the right way to be doing this. Descriptors are not necessary in PSBT, and relaxing the invariant that a PSBT represents a transaction (by allowing a PSBT to include no transaction data) would break many parsers and also just is antithetical to PSBT. It's not a general purpose format for sending data around - you could use protobuf if you wanted that.

If the goal is to have a machine parsable representation of descriptors, then by all means, do that. But it doesn't need to (nor do I think it should) interact with PSBT, other than having a different header that identifies it.

FWIW, the first draft was indeed PSBT with a different header: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-November/022184.html. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-November/022186.html suggested we add descriptors to PSBT proper, which seemed like a good idea.

Would you support an alternative BIP that changed the header to make it intentionally different from PSBT files?

It's also unnecessary to have a descriptor in a PSBT. A PSBT can contain all of the data to derive it from the scriptPubKey at a later time. If an updater has the descriptor in order to put it in the PSBT, then it can also put the rest of the data in the PSBT so that the descriptor can be inferred.

How can the scriptPubKey infer a descriptor that contains, say, a key expression that ends in '/*'?

I just don't see how this proposal is at all useful.

The point of the BIP is to transfer descriptors between devices, in particular between a coordinator and (resource constrained) signing devices. It can be viewed as a codec for wallet policies.

Today, there exists a plethora of formats for exchanging descriptors, all with downsides:

• Raw xpubs are underspecified (missing script type)
• The BlueWallet textual format is inefficient (base58 xpubs) and doesn't support arbitrary descriptors.
• Raw BIP-380 descriptors are also inefficient and lack metadata.
• BCR-2023-010 are efficient but rely on the more complex (d)CBOR encoding.
• JSON encoded textual descriptors are also inefficient and require a JSON implementation.

This proposal addresses them all:

• Based on PSBT, signing devices need not spend additional code space on a separate codec.
• Efficient encoding of extended keys.
• Metadata (title, birth block) supported.
• Every current or future bip-380 descriptor can be represented.

Also, descriptors are a textual format, and encoding text in binary is just not very efficient. There are better ways to do that if you really wanted to.

We believe it's hopeless to keep a binary representation of output descriptors up to date with the moving target that is textual bip-380 descriptors (BCR-2020-010 attempted but failed). This BIP (along with wallet policies) address the worst offender by encoding extended keys efficiently.

[achow101]

Would you support an alternative BIP that changed the header to make it intentionally different from PSBT files?

That would be fine with me.

How can the scriptPubKey infer a descriptor that contains, say, a key expression that ends in '/*'?

Although /* cannot be directly inferred, you can still make an educated guess. It only affects the ends of derivation paths from xpubs, so the last derivation index of a key expression with an xpub could just be replaced with /*. Although descriptors allow for xpubs without a /*, it seems unlikely to me that that would actually be used in practice.

But I don't understand why you'd need to have the parent descriptor at all. You'll have enough key origin information to derive any private keys and be able to sign.

[pythcoiner]

But I don't understand why you'd need to have the parent descriptor at all. You'll have enough key origin information to derive any private keys and be able to sign.

By example in the current ledger miniscript signing flow, the hardware device produce a Proof of Registration (PoR) against the parent descriptor that is stored by the softwarewallet (in order to keep stateless) the first time the descriptor is checked by user on signer side, then every time the software wallet send a psbt to sign to ledger device, it should also supply also the parent descriptor and PoR.

we previously discuss about this here

[pythcoiner]

cc @bigspider

[bigspider]

Hardware signing devices need knowledge of the full descriptor/wallet policy in order to:

• recognize change addresses
• recognize from which of your different accounts (sometimes called wallets) the inputs are spending from.

(The second could perhaps be considered a strengthening of the traditional guarantees of hardware signers, not sure if it was ever formalized before wallet policies; I think it's essential in practice for anything that is not single-user)

While it's true that PSBTs today have enough info to sign, they do not have enough info to do it securely - the descriptor is also needed. Therefore, I agree that it would be a great improvement to be able to pass this info via PSBTs; currently, the Ledger app's signing flow is sign(PSBT, wallet_policy), and quite some work in the app goes into figuring out change/address_index for each input and change output based on the derivations in the PSBT.

However, I think that's a separate discussion, and it's unclear that recycling PSBTs for a purpose other than representing transactions brings any benefit over than directly designing a space-efficient encoding specifically for descriptors or wallet policies – if the <30% space saving that can be obtained with a re-encoding is deemed important in practice.

[jonatack]

@seedhammer, do you plan to update here based on the review feedback?

[seedhammer]

Thanks for the prod, @jonatack . I've updated the proposal so it's no longer a PSBT extension but a separate format merely based on PSBT key-value maps.

I'd like to add proof-of-registration (PoR), but I'm not sure how it's best done. There's some discussion at wizardsardine/liana#539 (comment), where @bigspider suggests a HMAC scheme and in a later post @pythcoiner suggest that the format of PoR field should be vendor defined. I'm still of the position that to be maximally useful and interoperable, the PoR field should have a defined format and be a signature, not a HMAC.

[pythcoiner]

I'd like to add proof-of-registration (PoR), but I'm not sure how it's best done. There's some discussion at wizardsardine/liana#539 (comment), where @bigspider suggests a HMAC scheme and in a later post @pythcoiner suggest that the format of PoR field should be vendor defined. I'm still of the position that to be maximally useful and interoperable, the PoR field should have a defined format and be a signature, not a HMAC.

As i already said in previous discussion, i do not have strong opinion about wether the PoR should be standardized or not, but i do not think it should be standardized in this BIP.

And anyway if i get it well both sheme (HMAC vs Signature) will be 32 bytes of data?

[pythcoiner]

it should be interesting to have insight from other signing devices teams.

cc @benma @scgbckbone @stepansnigirev @stepansnigirev @JamieDriver

[seedhammer]

Also @kdmukai. Maybe others?

However, I think that's a separate discussion, and it's unclear that recycling PSBTs for a purpose other than representing transactions brings any benefit over than directly designing a space-efficient encoding specifically for descriptors or wallet policies – if the <30% space saving that can be obtained with a re-encoding is deemed important in practice.

I want to point out that while saving space is important to us, the ability to add metadata and eventually proof-of-registration is the more interesting feature in general.

[benma]

I am in agreement with @bigspider's comment above.

BitBox02 libraries have functions like sign_psbt(psbt, wallet_policy) (example), and the policy/descriptor is required for anything but simple single-sigs.

It would be helpful if the way the policy is described and transferred to signers was standardized so apps don't have to implement different ways to interface with different signers.

PSBT seems like the obvious location to put this information. Another format that contains both the PSBT and the policy, which is interoperable with all software, seems difficult to achieve and impractical.

[seedhammer]

Given

• PSBTs are useful without descriptors, and
• descriptors are useful without PSBTs (backups, registration), and
• the combination of a PSBT and descriptor is useful for more secure signing flows,

Perhaps it's feasible to keep the formats separate, but concatenate them at the transport layer (QR code, files), in a backwards compatible way? Say, append the decriptor at the end of the PSBT, where devices without knowledge of descriptors may parse the PSBT and ignore the rest?

[murchandamus]

I have only lightly reviewed this document.

It sounds to me that there is still some discussion what solution is exactly to be pursued for this document. I’m going to change this to a draft PR for the moment, please set it ready to merge when the proposal is a bit more mature.

Please note that you will need a Backwards Compatibility section, and without going into the content in detail, it seems to me that the Rationale section could be a bit more extensive.

[seedhammer]

Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.

It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.

Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.

[pythcoiner]

Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.

It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.

Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.

To be more accurate, i think how is processed the PoR should de defined in a separate BIP, but having an optional key-value pair defined in this BIP should be useful imho.

[seedhammer]

Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.
It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.
Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.

To be more accurate, i think how is processed the PoR should de defined in a separate BIP, but having an optional key-value pair defined in this BIP should be useful imho.

Good point. Done.

[scgbckbone]

I'm very much PRO descriptors directly IN PSBTs. Even tho I understand @achow101 comments.

Regardless of the format here (which i haven't studied) it is useful to have descriptor (or at least policy) in PSBT. Example from Coldcard is when you want to have multiple miniscripts registered on device that have identical keys in miniscript but policy/script is different. Or same keys same policy but order of keys in policy is different. Here you just cannot infer solely from PSBT which wallet to use because keys match for all wallets...

why not just use PSBT_GLOBAL_PROPRIETARY key and send descriptor in there?

[seedhammer]

I'm very much PRO descriptors directly IN PSBTs. Even tho I understand @achow101 comments.

Descriptors in PSBTs are useful, but descriptors are also useful outside PSBTs: transferring descriptors between wallets, (long term) backups. It would be unfortunate to specify a descriptor format that only solved the signing use-cases.

why not just use PSBT_GLOBAL_PROPRIETARY key and send descriptor in there?

That would hijack PSBT_GLOBAL_PROPRIETARY, leaving it no longer available for vendor-specific uses.

[jonatack]

A few review suggestions follow for the BIP text.

[murchandamus]

Hey, wanted to check in to ask what the status with this proposal is. Was there progress on determining the direction it is being developed? Is it ready for an editor review? Please let us know if we can help.

[seedhammer]

We're (still) interested in pushing this proposal forward, given enough interest from one or more signing device vendors.

[benma]

I am very interested in this, and my previous comment is still my opinion today.

[fslmultiservice22]

Sono di accordo Sono di accordo per eventuale sviluppo, credo che sia una buona descrizione.

[benma]

An error in which stage? It seems like this should still be optional. For example, for regular single-sig transactions (and even multisig if the global xpub field is populated), one can infer the descriptor.

[seedhammer]

The descriptor field is mandatory if we assume the format is separate from PSBT (albeit similar structure). See previous comment.

[benma]

What's the purpose of this?

[fslmultiservice22]

Avere un buon sviluppo, consenti una buona gestione del prodotto!!!

[seedhammer]

It serves the same purpose as PSBT_GLOBAL_PROPRIETARY. Again, assuming separate formats.

[benma]

It would be beneficial if we could specify a list of output descriptors, not just one, for two reasons:

• it should be possible to securely sign a transaction where inputs/changes belong to different descriptors
• a signer may want to inform a user which account/wallet (registered descriptor) they are sending funds to if they send funds to a different account/wallet, even if the inputs all belong to a different descriptor. For this, the signer needs access to the full descriptor

To achieve this, the global field could contain a list of all descriptors, and each input and output can reference one of them, e.g. by index.

[seedhammer]

Your comment seem to assume that this proposal expands signing PSBT with extra information.

As per #1548 (comment), the high-level issue blocking this proposal is whether to expand the PSBT format, or to create a new format (that happens to have similar structure). I don't think we can move forward without a decision either way.

FWIW, I lean towards separating the formats., given @achow101 et al insisting on PSBT being kept for signing use cases only (for good reasons).

[benma]

Indeed that was my assumption, apologies. I should have read the full BIP again before commenting.

I lean strongly towards extending the PSBT format instead of creating a new format.

1. Descriptors are essential for secure signing as @bigspider explained here, so including them in the PSBT format is appropriate.
2. @achow101 commented that one could infer the descriptor. It is not quite clear to me if descriptors can really be inferred from other existing PSBT fields in all cases, but even if, it would add significant complexity, performance problems and code bloat - especially for hardware signers that don't have many resources
3. There is a lot of software that interacts with PSBTs. Getting all of them to support a parallel format would take a long time and adds needless friction. Wallets handling PSBTs already simply ignore fields unknown to them, so having the descriptors in the PSBT is automatically compatible with all existing software.
4. UX for users who directly use PSBTs (importing/exporting) would become tricky: how would a user export and transfer both formats between wallets? Doing it twice (once per format) is cumbersome and error prone.

[benma]

Have you considered referencing BIP-388 instead? BIP-388 has similarities to the format you describe below, e.g. placeholder for keys (@0, @1, ...). Mentioning it here for awareness, but it is probably better to keep the descriptors in this BIP general, without the restrictions of BIP-388.

[fslmultiservice22]

Sono di accordo.

[seedhammer]

BIP-388 is mentioned in the "acknowledgements" section, but I can add it here if you think it brings value.

[seedhammer]

Thank you for the review. As noted in the comments below, I believe the immediate next step for this proposal is to decide whether to extend PSBT or create a new format.

[seedhammer]

BIP-388 is mentioned in the "acknowledgements" section, but I can add it here if you think it brings value.

[seedhammer]

Your comment seem to assume that this proposal expands signing PSBT with extra information.

As per #1548 (comment), the high-level issue blocking this proposal is whether to expand the PSBT format, or to create a new format (that happens to have similar structure). I don't think we can move forward without a decision either way.

FWIW, I lean towards separating the formats., given @achow101 et al insisting on PSBT being kept for signing use cases only (for good reasons).

[seedhammer]

The descriptor field is mandatory if we assume the format is separate from PSBT (albeit similar structure). See previous comment.

[seedhammer]

It serves the same purpose as PSBT_GLOBAL_PROPRIETARY. Again, assuming separate formats.

[craigraw]

I'm in favour of a separate format to PSBT, as is currently defined by this PR. There is nothing here that refers to a transaction, and wallet setup and registration is an entirely different process to transaction creation. Having a separate format simplifies how this format is detected and how it can evolve to serve future needs.

One thing to consider is whether it is worth expanding this format to also be used for collecting keys during setup, similar to how a PSBT is used to collect signatures during signing. This would allow a coordinator to specify at minimum a derivation path, and for signers to add keys to the format until it is complete. AFAIK there is currently no single format which allows for this, which makes geographically distributed multisig setup more difficult or vendor specific.

Another consideration is adding a BOD_GLOBAL_VERSION global type to specify the version, allowing for future iterations.

I also suggest adding an Encoding section which includes the file extension when serialized to disk, as in BIP174.

[jonatack]

@seedhammer, I updated the PR title to that of the current "Title" field in this draft. Could you please verify if the PR description is current, and update it, if not?