On the ARM Cortex-A9 that comprises part of the Zynq SoC I’m using, regions of memory are labelled as “normal”, “device” or “strongly ordered”. This is described in the Zynq technical reference manual, but I understand it is a property of ARMs more generally. Obviously, the ability to have strongly ordered memory accesses for memory mapped devices (which includes many in FPGA fabric) should simplify the software somewhat, so is desirable to set up.
I’m using the UIO driver for mapping the device memory into userspace, in which the bulk of the driver runs. According to this reference the UIO driver sets up its mapped memory as “device/strongly ordered”. Unfortunately, this is the only reference I can find to this, and before I start ripping out memory fences from my code, I’d like to have a little more confidence about what is going on.
It’s not clear to me currently how the Linux kernel denotes memory regions of a particular type. It seems to me that the MT_* properties denote something along these lines, but I can’t find the definitions of each type. Nor can I work out how the UIO driver specifies the particular memory.
Any pointers about how the memory properties are set in Linux, either in general terms or ideally with reference to UIO would be exceptionally helpful. I’m happy to have that in the form of a pointer to documentation.
Advertisement
Answer
There are a few parts to this.
On ARMv7, which includes the Zynq-7000, memory is denoted as a given type by the memory region attributes configured through the translation table descriptors. There are various ways to configure these and the mechanisms are described in section B3.8 of the ARM Architecture Reference Manual ARMv7-A. Also useful is the Zynq technical reference manual, which is less complete as regards the ARM, but easier to process.
Broadly, the bits of interest are the B (bufferable) bit, the C (cacheable) bit and the 3 TEX (type extension) bits. These may be set directly, or through a redirection when SCTLR.TRE bit is set (which effectively allows a custom remapping using the PRRR and NMRR registers – there may be more to it than that, but I can’t immediately see what).
The translation table descriptors are set up in Linux in the memory-management unit (MMU) subsystem. This is obviously very architecture specific, and the relevant ARM bits are found in arch/arm/mm. It’s interesting to look through mmu.c to see how different memory attributes are configured on different types.
What follows is a little more speculative, but I think is accurate.
The core UIO driver sets up its relevant memory protections on a physical device by calling pgprot_noncached(). Now, I think this is delegated to the architecture specific implementation, which in the case of ARMv7 is a macro defined in arch/arm/include/asm/pgtable.h, and resolves to setting the L_PTE_MT_UNCACHED flag.
The L_PTE_MT_UNCACHED constant is in turn set in arch/arm/include/asm/pgtable-2level.h. There is a nice bit of documentation in that file that describes what the various constants represent. The value for each type is remapped to the B, C and TEX bits, either through the TRE redirection or through a look-up table configured in arch/arm/mm/proc-macros.S. The TRE redirection registers (PRRR and NMRR) I think are configured in arch/arm/mm/proc-v7-2level.S. If you track those through, you get the same values as the look-up table (which references constants defined in arch/arm/include/asm/pgtable-2level-hwdef.h – note that those constants are for a small page table descriptor, distinct to the ones used in mmu.c)
Where does this leave us? The UIO driver configuring a piece of memory as pgprot_noncached() implies it is L_PTE_MT_UNCACHED, which in turn implies TEX = 000, B = 0 and C = 0. Looking those settings up in the reference manual, we see this corresponds to an unbuffered, strongly ordered, shareable memory region (denoted “Strongly-typed”).
It’s apparent, that to change the device (perhaps to allow buffered writes), we’d need to modify the driver configuration of memory to use e.g. pgprot_writecombine() which would set the memory type to be normal but with the cache off. There is also a pgprot_device() macro, which also sets up device memory type (bufferable) and cache off, but with some additional flags I haven’t understood properly yet (I think they are for configuring a software “Linux” version of the page table entry for the case where the hardware doesn’t support it, so aren’t relevant when it does).