I/O ordering requirements

[kdockser]

As I understand it, current RISC-V Linux expects all I/O regions to be on Channel 0 (point-to-point strong ordering). Will there be a server requirement that all I/O regions be on Channel 0. Or perhaps a requirement that I/O regions must support Channel 0?

[jones-drew]

Hi Ken,

While the Server Platform spec doesn't yet have a requirement like that, it is based on the Server Soc spec which does require
"Not cacheable, non-idempotent, coherent, strongly-ordered (I/O ordering)" in:

• IIC_070 for IMSIC memory regions (and here it even explicitly states "channel 0")
• ECM_010 for ECAM
• MMS_030 for PCIe memory space

If we need each of those regions to explicitly state "channel 0" then we can suggest a patch for ECAM and PCIe. If we need other regions in addition to those then we can discuss adding them to the Server Platform spec.

It does look like there's a long standing expectation that this spec will require strongly ordered I/O. Volume I A.4.2 "I/O Ordering" has this paragraph:

"For implementations that conform to the RISC-V Unix Platform Specification, I/O devices and DMA
operations are required to access memory coherently and via strongly ordered I/O channels.
Therefore, accesses to regular main memory regions that are concurrently accessed by external
devices can also use the standard synchronization mechanisms. Implementations that do not conform
to the Unix Platform Specification and/or in which devices do not access memory coherently will
need to use mechanisms (which are currently platform-specific or device-specific) to enforce
coherency."

(RISC-V Unix Platform Specification is the old name for this spec and we should send a PR to volume I to get it updated.)

[kdockser]

Hi Drew,

Thanks for your reply.

As you point out, Appendix A of the The RISC-V Instruction Set Manual Volume I states that strong ordering I/O is required. Unfortunately this statement is based on a non-ratified specification. Nevertheless, I believe it to be true.

The Linux RISC-V mmio.h file that I was referencing states:

/*

• Relaxed I/O memory access primitives. These follow the Device memory
• ordering rules but do not guarantee any ordering relative to Normal memory
• accesses. These are defined to order the indicated access (either a read or
• write) with all other I/O memory accesses to the same peripheral. Since the
• platform specification defines that all I/O regions are strongly ordered on
• channel 0, no explicit fences are required to enforce this ordering.
  */

This is also a bit circular as it is quoting a newer yet now-defunct draft specification.

As you point out, there is a channel 0 reference in the Server SoC Spec (which is required by the Server Platform), but it is limited to the IMSIC interrupt files. Since each each IMSIC is (typically) only written by its own hart, the point to point strong ordering should be sufficient.

We need to clearly specify the I/O requirements for Servers such that compliant processors will work with OSs written for RISC-V. It seems like this should be channel 0. If there was a reason for omitting this in the specifications, and that OSs will run properly on implementations that don't have all I/O use channel 0, this should be clearly stated in a non-normative note - probably in the Server SoC specification.

[jones-drew]

I've sent a note to @ved-rivos to see if he'd like to chime in.

[ved-rivos]

The intent was for the MMIO and ECAM regions to be use channel 0 for ordering. This looks like an omission in the Server SoC specification that we should fix.

[jones-drew]

Hi Ken,

Ved has pushed [1] to the server-soc spec. Does that resolve this issue?

[1] riscv-non-isa/server-soc@ffdf34d

Thanks,
drew

[kdockser]

Thanks, this looks much better.

One issue though, it appears that these references are limited to PCIe usage. Since PCIe support is not required, it would be good to generalize them or repeat them for non-PCIe usage.