PCIe 6.0 Address Translation Services: Verification Challenges And Strategies

Address Translation Services (ATS) is a mechanism in PCIe that allows devices to request address translations from the Input/Output Memory Management Unit (IOMMU). This is particularly important where devices need to access virtual memory. ATS enhances performance by enabling devices to cache translations, reducing the latency associated with memory access. This blog delves into the semantics of ATS request-completion protocol and Invalidation protocol. It also addresses key verification challenges that verification engineers might face while covering the error scenarios at various stages of verification.

Semantics of ATS protocol

ATS uses a request-completion protocol between a device and a Root Complex (RC).

• The device function generates an ATS translation request that is sent upstream through the PCIe hierarchy to the RC which then forwards it to the Translation Agent (TA)
• ATS translation requests are processed for each traffic class that follows the ordering rules and issues the associated ATS translation completion

ATS uses invalidate request-completion protocol between a device and an RC and ensures that the cached address translations are not stale.

• TA (or RC) sends the invalidate request downstream to the device Address Translation Cache (ATC)
• Devices like endpoint (EP) issue one or more invalidate completion responses to TA, indicating that EP has cleared the ATC

Translation request format

The translation request format is like a memory read in both Non-Flit Mode (NFM) or Flit Mode (FM), where Address Type (AT) bits identify that it is going to be a translation request. In FM, the Orthogonal Header Content (OHC) encoding adds OHC-A1 Double Word (DW) in the packet header, including information like No Write (NW), explicit byte enables and/or Process Address Space ID (PASID) fields. These translation requests are not Direct Memory Access (DMA) transactions.

Translation completion format

The translation completions format is like the data completions. For a translation completion, the Relaxed Ordering (RO) bit is applicable and permitted to be set if the corresponding translation request RO bit was set. Completion data contains:

• Translated address
• Size (or pages in terms of Smallest Translation Unit (STU))
• Memory protection attributes (Global, Priv, Exe, U, W, and R)

Completion can have multiple translated addresses. If the TA was not able to perform the requested translation, the completion with no data (Cpl) must be returned. The OHC-A5 DW in the packet header is required for unsuccessful completions and for completions with lower address [1:0] not equal to 00b.

Invalidate request

An invalidate request clears a specific subset of the address range from the ATC when translation is changed in TA. The invalidate request is a message with a data transaction with 64 bits of data. The message payload contains the untranslated address range to be invalidated, and the ITag field is used by the TA to uniquely identify invalidate requests. In FM, PASID is included when OHC-A4 DW is present and the PV bit is set.

Invalidate completion

An invalidate completion is a message transaction that must contain the requester ID used in the associated invalidate request. All fragments of an invalidate completion must have an identical request ID, Completion Count (CC), and ITag vector fields.

Key verification challenges

The verification challenges mentioned below need to be taken care of by the verification engineer by building the randomized test scenarios, designing the automated checkers, and assertions that can seamlessly be integrated into the verification environment.

Consistency

• Ensuring that changes to the page tables (like updates or invalidations) are correctly reflected in the address translation cache
• In request translations or invalidate, ensure that the memory access is the same as is mentioned in the request

Coherency

• Maintaining coherence between multiple translation caches in a multi-processor system

Correctness of translations

• Ensuring that the virtual addresses are correctly translated to a physical address in the page table mapping
• Verifying that the translations respect all access permissions and protections defined in the page tables
• In FM, OHCs are included as per the request. Ensure the desired values are added correctly

Performance

• Ensuring that the system can recover from translation errors
• Verifying that the address translation mechanisms do not impact the overall throughput of the system

Scalability

• Maintaining the correctness of caches when system sizes and complexities grow

Security

• If PASID-related attributes are included in the translation request, ensure that the completion comes with the same attributes to prevent unauthorized access
• If the address translation table has an overlapping translation address entry, ensure that the entry with the latest entry is considered valid

Summary

The verification challenges of ATS in PCIe systems highlight the need to ensure consistency, correctness, and coherency of address translations. It addresses the importance of maintaining access permissions, managing performance impacts, and handling security concerns such as OHC and PASID attributes. By employing randomization test scenarios, one can ensure that the system can handle unexpected address patterns or workload distributions without compromising performance and security.

Further information

• PCI Express Base Specification Revision 6.2, Version 1.0
• To learn more about Cadence PCIe Verification IP, see VIP for PCI Express