[Question] Interrupt IN Endpoint On the MCTP USB Interface?

[santoshpuranik]

Hi Jeremy,

There is a proposal on the DMTF PMCI WG to add an additional interrupt IN endpoint to the two BULK endpoints that exist on the MCTP USB interface. The idea there is that since interrupt transfers get guaranteed bus bandwidth, it would improve latency by reducing the need for host side implementations to always issue bulk INs.

The proposed flow would then be something like:

1. Host sends a req over the bulk OUT
2. Host queues an interrupt IN transfer (The HCD will handle the repeated polling per the bInterval)
3. When the endpoint has data to send on the bulk IN, it will first send the number of bytes it has to send over the interrupt IN endpoint.
4. The host reads the number of bytes available and submits a bulk IN. (Along with resubmitting the interrupt IN).

In my view, since bulk INs are guaranteed bandwidth on the bus, we will avoid the repeated NAKs for bulk INs, but I see little benefits of implementing the proposed changes:

1. For a bus with many endpoints, the repeated interrupt INs can hog the bus bandwidth.
2. There is added latency now between the driver receiving the number of bytes and it submitting the bulk IN (as opposed to before where it could just submit one or more bulk INs for each endpoint)
3. They add complexity to the host side driver by forcing state into the IN transfers.

What are your thoughts on this proposal from a Linux driver implementation perspective? (Can add you to the DMTF PMCI WG stream if that helps)

Thanks,
Santosh

[jk-ozlabs]

Just a minor correction:

1. Host sends a req over the bulk OUT
2. Host queues an interrupt IN transfer (The HCD will handle the repeated polling per the bInterval)
3. When the endpoint has data to send on the bulk IN, it will first send the number of bytes it has to send over the interrupt IN endpoint.
4. The host reads the number of bytes available and submits a bulk IN. (Along with resubmitting the interrupt IN).

The interrupt IN transfer needs to be queued on driver init, not after (1) there.

In my view, since bulk INs are guaranteed bandwidth on the bus, we will avoid the repeated NAKs for bulk INs, but I see little benefits of implementing the proposed changes:

The benefit is in better transfer scheduling, rather than fast repeated NAKs on the bulk IN EP. However, you're correct in that the latency will be greater, and that the fundamental issue of repeated polling is still present (unavoidable for anything USB-based, really), just that we're now polling an interrupt endpoint rather than a bulk one.

I'm fine to implement the new spec, we'll just need to ensure it can coexist with the bulk-only implementation.

[santoshpuranik]

The interrupt IN transfer needs to be queued on driver init, not after (1) there.

Yes, at the start rather than after sending the request!

In my view, since bulk INs are guaranteed bandwidth on the bus, we will avoid the repeated NAKs for bulk INs, but I see little benefits of implementing the proposed changes:

The benefit is in better transfer scheduling, rather than fast repeated NAKs on the bulk IN EP. However, you're correct in that the latency will be greater, and that the fundamental issue of repeated polling is still present (unavoidable for anything USB-based, really), just that we're now polling an interrupt endpoint rather than a bulk one.

+1. If the spec. (in the future) moves to support USB 3.0, I think the INs (bulk or interrupt) can be made truly somewhat asynchronous without the need for repeated host polling.

I'm fine to implement the new spec, we'll just need to ensure it can coexist with the bulk-only implementation.

I think the proposal is to keep the usage of the new interrupt endpoint optional (even when the endpoint advertises it). If the host side driver chooses not to use the interrupt IN (i.e, never submit a xfer to it), do you know if the HCD will still poll it?

[jk-ozlabs]

even when the endpoint advertises it

that might be tricky, I'll do some reading.

If the host side driver chooses not to use the interrupt IN (i.e, never submit a xfer to it), do you know if the HCD will still poll it?

no, shouldn't do. it cannot do anything with the data if the attempted transfer is successful anyway.

[moriol]

I'm collecting input for the new spec discussion. what would be the penalty of getting size in Bytes from interrupt IN and then scheduling bulk IN from driver perspective?
to my knowledge simple use of Bulk IN is fully offloaded to Host controller and does not need any kernel operation.

and for USB 3.0 the NRDY/ Async ERDY will completely eliminate polling.

Can I get confirmation that use of scheduling from MCTP driver a set of pre-allocated URB's will allow to read full MCTP message within same microframe (many reads until ZLP or shorter than max read). do you see any value with providing size in interrupt IN ?

[jk-ozlabs]

Reordering a bit for context:

to my knowledge simple use of Bulk IN is fully offloaded to Host controller and does not need any kernel operation.

That's correct - with any sensible controller & device hardware, the software/firmware implementation only needs to handle submission and completion; the NAKed transfers do not cause any CPU overhead on either host or device.

I'm collecting input for the new spec discussion. what would be the penalty of getting size in Bytes from interrupt IN and then scheduling bulk IN from driver perspective?

Separating the size bytes concept from the interrupt concept here, but the penalty of an interrupt means twice the number of transfers (which do involve interactions from the CPU, vs NAKed bulk transfers, which do not). The"twice" could be amortised over multiple transfers maybe, depending on spec.

do you see any value with providing size in interrupt IN ?

I don't see any need for the size field - at least within the semantics of the current transfer layouts (which may change as part of a new spec? you mention introduction of a ZLP, which implies new framing).

On receiving a completion of the IN interrupt, the hcd would submit the IN bulk transfer. Given:

• we need to support a max-sized USB transfer (and so have a suitable buffer allocated); and
• state (on the endpoint) may have changed between the interrupt completion and bulk transfer submission

The host-side driver probably wouldn't be able to do anything useful with the size data anyway.

I'd be keen to review the proposed spec, if it's in a state to do so.

[moriol]

Reordering a bit for context:

to my knowledge simple use of Bulk IN is fully offloaded to Host controller and does not need any kernel operation.

That's correct - with any sensible controller & device hardware, the software/firmware implementation only needs to handle submission and completion; the NAKed transfers do not cause any CPU overhead on either host or device.

I'm collecting input for the new spec discussion. what would be the penalty of getting size in Bytes from interrupt IN and then scheduling bulk IN from driver perspective?

Separating the size bytes concept from the interrupt concept here, but the penalty of an interrupt means twice the number of transfers (which do involve interactions from the CPU, vs NAKed bulk transfers, which do not). The"twice" could be amortised over multiple transfers maybe, depending on spec.

do you see any value with providing size in interrupt IN ?

I don't see any need for the size field - at least within the semantics of the current transfer layouts (which may change as part of a new spec? you mention introduction of a ZLP, which implies new framing).

the plan is to support packet spanning so we need ZLP or packet shorter than max bulk size - this is already within a USB spec.

On receiving a completion of the IN interrupt, the hcd would submit the IN bulk transfer. Given:

• we need to support a max-sized USB transfer (and so have a suitable buffer allocated); and
• state (on the endpoint) may have changed between the interrupt completion and bulk transfer submission

The host-side driver probably wouldn't be able to do anything useful with the size data anyway.

I'd be keen to review the proposed spec, if it's in a state to do so.

I'll ask. for now you need to be DMTF member and also join PMCI workgroup. for now I'm reaching you to collect full feedback from HOST side.

[jk-ozlabs]

Hi,

the plan is to support packet spanning so we need ZLP or packet shorter than max bulk size - this is already within a USB spec.

OK, so we have multiple framing mechanisms already in place; I'll take a more detailed look, but I still don't see any reason for including size data in the interrupt transfer.

I'll ask. for now you need to be DMTF member and also join PMCI workgroup.

Yep, we're in both - as per the recent email thread.

for now I'm reaching you to collect full feedback from HOST side.

We also have device side implementations, for Linux systems as well as embedded

[moriol]

Hi,

the plan is to support packet spanning so we need ZLP or packet shorter than max bulk size - this is already within a USB spec.

OK, so we have multiple framing mechanisms already in place; I'll take a more detailed look, but I still don't see any reason for including size data in the interrupt transfer.

I'll ask. for now you need to be DMTF member and also join PMCI workgroup.

Yep, we're in both - as per the recent email thread.

feel free to review https://members.dmtf.org/wg/pmci/document/61470 I uploaded. it is critical to provide feedback. Also I plan to discuss on Tue PMCU meeting - if time will allow. I'm looking at simplicity and high performance for MCTP over USB. feel free to join (see https://members.dmtf.org/wg/pmci/calendar for PMCI WG Tue 3PM CET )

for now I'm reaching you to collect full feedback from HOST side.

We also have device side implementations, for Linux systems as well as embedded

You can also create your own "response" or even add to https://members.dmtf.org/wg/pmci/document/61470.