When Is Robustness Verification Complete?

Understandably, hardware designed for an aircraft, or indeed any safety critical application, must be robust. I also believe that all engineers wish to verify their designs as thoroughly as possible, anyway. However, there are limiting factors; most notably the high complexity of most designs. Since we are unable to discover and verify the design against all abnormal conditions, the main question is: when is robustness verification truly complete?

Random nature of robustness
Test scenarios for robustness verification always contain many input stability issues, such as erroneous values, lack or loss of value, unexpected timing and unpredicted toggling. Certainly, there is a significant random factor but it should not lead us to oversimplify this part of verification by applying less or more advanced randomization methods only. The robustness of any design, and especially for projects where human lives are potentially at risk, cannot be achieved by inspecting the results of randomly generated scenarios. It must be part of the original design.

RTCA/DO-254
The “Design Assurance Guidance for Airborne Electronic Hardware” document does not explicitly address robustness testing. However, two supplements – “FAA Order 8110.105A” and the “EASA Certification Memorandum” – clarify that to demonstrate robustness, the applicant should also define the requirements-based testing. In other words, it is expected that abnormal operating conditions be captured and documented as derived requirements.

How many “robustness requirements” do we need?
Having extra requirements for robustness verification does not solve the problem stated above, so the question really is: how many “robustness requirements” will be enough?

To capture the requirements, the system specification must be analyzed. We need to look deeper into the non-functional, quality, and performance requirements. The second important source of “robustness requirements” is the system design where any unpredicted behavior that might result in functional degradation should be considered and documented as derived requirements.

From robustness test cases to higher quality of requirements
Test cases for robustness verification will not derive from the “robustness requirements” alone. They also need to come from the regular functional requirements; and it is this combination that can reveal inadequacies in the functional specification. This is one of the situations where I really appreciate the RTCA/DO-254 guidance and the defined processes, because the product cannot be safe without having the highest quality of requirements.

Robustness hardware verification
DO-254 requires that the hardware requirements are verified in real hardware and should be done in the operating environment (i.e. LRU, CCA). However, it may be very difficult or even impossible to verify some robustness test cases in the operating environment.

Fortunately, we may streamline this task by using the standalone test equipment like DO-254/CTS. This equipment allows us to verify all functional requirements (including robustness) in the target device and at real speed. All can be done without struggling with test procedures by reusing test benches prepared for the simulation. In addition, DO-254/CTS provides the features to perform the verification with normal and abnormal power supply and timing conditions.