Having a device that works at time zero is no longer a guarantee of reliability over its lifetime.
Key Takeaways:
The AI revolution is significantly outpacing the IC industry’s ability to sufficiently test multi-chip systems for all necessary failure mechanisms at probe, final test, and system-level test.
The wider adoption of 2.5D systems, along with ongoing efforts to develop full 3D-ICs, depends on in-system testing and the engineer’s ability to monitor, detect, and repair failures in the field. This is especially important in mission-critical situations where a chiplet or interconnect failure can be catastrophic. However, effective resource orchestration is required to contain test costs while ensuring full monitor, test, and repair capabilities.
The idea behind in-system testing is to electrically detect upcoming failures inside chiplets — or more likely the thousands of bumps, through-silicon vias, RDL lines and vias, or hybrid-bonded interconnections — before disaster strikes. Proactive maintenance in the field means having the redundancy to swap a bad lane for a good one, or to divert operations from a failing core as soon as needed.
“Ensuring initial quality is no longer sufficient. Devices must also maintain their integrity throughout their operational lifespan,” said Peter Orlando, DFT product director for Siemens EDA’s Tessent Silicon Lifecycle Solutions. “In-field testing is therefore essential to quickly detect emerging defects, whether caused by silicon aging, environmental conditions, or unexpected stress, especially in mission-critical applications such as hyperscale data centers and automotive systems where uptime and safety are paramount.”
Having visibility into the behavior of these complex devices can help pinpoint weaknesses and remediation options before they turn into failures. “Some chips can repair themselves in the field based on feedback from silicon lifecycle management, DFT, and other in-system test methods,” said Adam Cron, distinguished architect at Synopsys. “Many recent standards for interfaces and buses, including UCIe, AIB (Advanced Interface Bus), and HBM4, provide spare lanes or pins. If SLM data indicates a current or impending failure, swaps can be performed automatically to bypass the problem. Some chips also duplicate cores for dual modular redundancy or provide three registers for triple modular redundancy, with a failing flop “outvoted” by the other two.”
But first, the in-system test needs to detect the impending failure, which is where on-chip monitors are really coming into their own. On-chip monitors and analytics can predict impending failures. “Time-to-failure modeling has been applied in data center environments to correlate degradation trends with usage profiles, enabling predictive maintenance and improved reliability planning,” said Eidan Mendelsohn, vice president of engineering at proteanTecs. “Predictive analytics are already delivering tangible results across the lifecycle.”
In addition to these on-chip monitors, design-for-test techniques are rising to the challenge of monitoring multi-chiplet health in advanced packages, which is performed by DFT methods utilizing the available AXI bus.
In-system test is needed to catch what manufacturing testing cannot. “In-system testing is becoming more and more prevalent in data centers,” said Nilanjan Mukherjee, VP of engineering at Siemens EDA’s Tessent Division. “We’re seeing silent data errors, silent data corruption because of the extensive use of those ICs over a long period of time, for example, doing AI training. Many latent defects are escaping manufacturing test, and the reason is that we are not stressing the transistors enough, and they are failing in the field. The failures show up as small delay-type defects. Secondly, intermittent defects show up because of the conditions that those ICs are subjected in the field, like the power, the voltage, the temperature, which is very difficult to replicate in manufacturing test.”
Using embedded deterministic test (EDT) patterns during in-system test enables efficient and targeted in-field testing to detect latent defects that may arise during device operation. “These could include issues related to silicon aging, voltage fluctuations, thermal stress, or workload-induced degradation,” said Siemens’ Orlando.
It’s also important that in-system testing takes place not just while the system is running but also during power-up and power-down transition times when failures can occur. “These moments are often overlooked but can expose vulnerabilities that are not detected during standard runtime conditions,” said Orlando.
High-speed interfaces for big data environments
Foundries face relentless pressure to deliver multi-chip systems with the highest yield, while maintaining high reliability in the field of data center, robotics, autonomous vehicles, and so much more. While system-level test and in-system testing has been around for some time, it’s only in recent years that companies have had the technology to transfer data in the ways needed to begin lifecycle tracking of chips from first silicon to end-of-life.
In recent years, one key technology development was the introduction of high-speed interfaces to enable in-field testing through high-speed I/O interfaces such as PCIe (Peripheral Component Interconnect Express) or USB. PCIe was originally rolled out in 2005 as a general-purpose expansion bus for computers, but it has become the universal high-speed interconnect for graphics cards, solid-state drives, AI accelerators, and CPU-to-GPU connections in data centers, replacing slow, general-purpose I/O.
PCIe has since been repurposed for testing. “The root of the test challenge is the ever-widening ratio between silicon capacity and available pin count,” explained Ash Patel, director of product marketing at Synopsys. “Chips have scaled dramatically in capacity as geometries have shrunk, but the physical parameters associated with off-chip I/O connections have not kept pace. The traditional approach of using slow general-purpose I/O pins for test access no longer suffices. It is impractical to add many GPIO pins to increase bandwidth and test the huge amount of functionality within large SoC devices.”
This is where USB and PCIe shine. “The latest generations of USB and PCI Express can transfer high volumes of data quickly, greatly reducing time spent on automated test equipment (ATE) during chip manufacturing,” Patel said. “Re-use of functional pins opens the possibility of running chip tests during in-field deployment to check for degradation and aging as part of an effective silicon lifecycle management (SLM) strategy.”
This repurposing of the PCIe interface for IC test means it now forms the high-speed communications backbone connecting test hardware, instruments, and the device under test (DUT). That enables engineers to rapidly inject test patterns and diagnostic data, which effectively slashes test times.
“If you compare the bandwidth of GPIO, which depending on the GPIO, the streaming scan fabric can reach 100, 200 or even 400 MHz, if you use the SerDes, like the DFT, PCIe, USB or UCIe, there is a huge gain in bandwidth, which reduces test times by 10X to 20X,” said Sandeep Goel, senior director at TSMC, during a recent webinar. [1]
But before the high-speed interface can be used, it must be verified and tested. First, at design verification, simulations ensure the logic functions correctly before manufacture. During ATE and other insertions, the PCIe is tested using scan testing, ATPG patterns, and built-in self-test (BiST). While these tests ensure the interface was manufactured properly, only high-speed testing with adequate power supplies and a cooling platform can determine whether the PHY can communicate at the speed level dictated by that generation of PCIe.
Because power and heat are becoming so problematic during test, power-aware ATPG is gaining steam. “Power-aware ATPG is an approach designed to minimize power consumption during the critical scan shift and scan capture operations. This is achieved through a combination of hardware insertion and pattern generation techniques,” said Quoc Phan, technology enablement manager of 3DIC DFT and yield at Siemens EDA. “It actively strives to minimize switching activity during the scan shift operations and keep as many scan cells as possible in their previous state or a fixed state during capture. This strategic reduction in switching activity significantly lowers both dynamic power and peak power.”
For AI accelerators, PCIe testing is particularly important because the interface connects the accelerator to the host CPU and system memory. Any defect can prevent the AI accelerator from being recognized by the host PCIe controller.
Once the PCIe is tested and is known good, the test challenge becomes one of leveraging the interface to access “hidden” dies that cannot be accessed directly.
Inside in-system testing
A recent project between Synopsys and TSMC demonstrated a multi-die test strategy, including monitor, test, and repair mechanisms. They built a two-die chiplet system on a silicon interposer in a 3NP process, connected using a UCIe bus, and demonstrated monitor, test, and repair capabilities. “[For multi-chiplet systems], how you transfer the test data at very high speeds in and out of the stack becomes a bigger problem, and that is one of the problems we are solving in this particular joint work,” said Goel.
The more dies, the more difficult the access to a given die in the middle of a stack. “Accessibility becomes an important thing, while also exercising the different test, repair, and monitoring capabilities of a hidden die,” said Yervant Zorian, chief architect and fellow of Synopsys. “We have to find a way to reach it in manufacturing and also in-system, because it impacts manufacturing results, manufacturing quality, manufacturing yield, but also in-field reliability, in-field RAS, and the performance of the system.”
Fig. 1: The project demonstrated monitor, test, and repair capabilities (simplified block diagram shown) via UCIe die-to-die communications, signal integrity monitors, and UCIe controllers, while leveraging the IEEE 1838 standard. Source: Synopsys
Zorian described one challenge the teams encountered that is not covered in the UCIe standard. “The UCIe die-to-die solution, including the PHY that originally started with 16, then 32 and is now moving to 64 Gbps, is moving fast. One of the challenges is health monitoring during mission mode, which is not part of the main standards. Therefore, it has to come as an add-on capability.”
To monitor the health of signals, a signal integrity monitor (SIM) runs parallel to the incoming UCIe, sampling the signal and providing eye diagram openings for the health of the signal. These SIMs reside on the receiving end of the PHY on both sides of the UCIe. “The SIM data will tell us if that eye opening is sufficiently large, and if there is degradation over time, it will tell us by what percentage the signal has degraded, and that data gets passed along to the control engine. This is especially important in mission-critical applications like automotive and AI/HPC,” said Zorian.
Fig. 2: The main band of the UCIe is used for functional data while health monitoring is performed on the slower side band. Multiple signal integrity monitors (black boxes) measure signal degradation on a sampling basis and report it to the control engine. Source: Synopsys
The latest standard specifies two redundant bits for every 32 bits, enabling a shifting of data lanes when faults are identified. Zorian notes, however, that repair will not only be happening in-field but also after assembly of multiple chip packages. “We have to start doing it for manufacturing when we find out that there are defectivities in manufacturing, such that when we send the package to our users, it is already repaired for the known defects that are there already in the package between the dies. We do this using a fuse box, an OTP, or some non-volatile memory that permanently reports defective lanes.”
Siemens EDA’s Mukherjee describes how repairs are performed when defective components in the chip are found. “When a faulty part of the chip is identified by in-system testing, various steps can be taken. Let’s assume that you are running this testing in the field, and you figure that one of the memory BiST controllers has failed. The in-system test allows you to figure out whether the memory is repairable or not, provided there are some repair resources there, and if the memory is repairable, then the in-system test allows you to do an on-chip incremental, either soft repair or hard repair, where someone has to make sure that the OTP can be programmed by raising the voltages or whatever else is needed.”
It’s not unusual that test patterns must be delivered during in-system test. “DFT is expanding beyond manufacturing defect identification to support continuous reliability testing and in-system testing,” said Mukharjee. “Continuous reliability testing and in-system testing involves inserting on-chip sensors and silicon lifecycle management (SLM) to generate early warnings without prohibitive overhead. In-system test controllers allow manufacturing test patterns to be applied in a functional environment, enabling the detection of faults that may develop slowly over time.”
Synopsys showed the path that ATPG pattern loading follows. “When a logic block on the hidden die requires ATPG patterns, the data follows a route from the PCIe through die 1, into its UCIe controller, across the UCIe bridge connecting the two dies, into its UCIe controller, then into IP that decodes the patterns, then into the logic block,” said Zorian.
Synopsys and TSMC demonstrated the health monitoring in mission mode, as well as loading of ATPG patterns to the second die in a two-die chiplet process on a 65nm interposer. Also, they measured memory access times and performance using its clock and delay monitor IP. They further performed memory BiST test and repair through the IEEE 1838 parallel port (IJTAG). Zorian noted that the IEEE 1838 standard allows chips to communicate with one another without requiring a PHY interface.
TSMC’s Goel further elaborated on the test strategy. “The stack-level test is the key part where you want to exercise all the functions through the stack, so whatever function you have at the die level, which you could exercise through the die, now you want to exercise them through the stack level, which we applied through the C4 bumps. If you’re going to do a repair in manufacturing test, you have to generate certain bridge patterns that run right between those C4 bumps. So we built a physical-aware die-to-die test — and this is now internal language and part of the IEEE standard — that uses the bump map file to see whether it physically makes sense to have a bridge there or not. This reduced the number of bridges by 95% to 99%. And, by the way, it reduced the number of test patterns that you would generate for the bridges by half.”
The decision was made not to perform direct microbump testing (20µm bumps, 25µm pitch) due to the significant bump size and pitch requirements, but probing was performed on multiple 50 x 50µm sacrificial pads. Both optical inspection and continuity testing of the 65nm interposer ensured no known defects.
On-chip monitors with silicon lifecycle analytics
On-chip monitors rely on an analytics platform that is capable of connecting data across different test insertions. “Enabling effective feed-forward and feed-back requires predictive intelligence built on correlated data across multiple post-silicon stages,” said proteanTecs’ Mendelsohn. “We achieve this by deploying ML models trained on deep data collected from on-chip agents across multiple post-silicon. These models learn the expected behavior of silicon and can predict deviations for each individual die, regardless of where it is in the distribution. That enables a personalized approach to quality and power/performance management.
On-chip monitors, or Agents, also play a critical role in outlier detection. “There is a need to “shift-left” in testing, especially for chiplet-based designs. You need to be able to do smart testing and outlier detection at wafer sort to avoid detecting a defective chiplet device after assembly due to its cost. Also, there’s the additional challenge of testing the die-to-die interconnect that can only be done after assembly, where the ATE no longer has access to these interconnects from the outside,” said Nir Sever, senior director of business development at proteanTecs. “By leveraging our on-chip agents, we monitor signal timing, per lane, in mission mode to identify marginal lanes which could be candidates for lane repair on the ATE and in-field.”
Conclusion
In-system testing for automotive and data centers is considered essential, and some of the fine details are still being developed. Through on-die monitors and DFT technologies, companies are implementing predictive methods for test and repair that cannot be underestimated given the vast and growing number of interconnects in today’s advanced packages.
An example of a multi-die monitor, test and repair strategy reveals the challenges associated with in-system testing even when only two dies are involved. Nonetheless, the components for success, including high-speed interfaces like PCIe, UCIe, and USB, on-chip test controllers, DFT, and various supporting IP and methods, enable thorough monitoring of device signals, memory performance, and overall device health needed for mission-critical applications.
Reference
1. Yervant Zorian and Sandeep Goel, “Synopsys and TSMC Discuss Multi-die Monitoring, Embedded Test & Repair Flows,” February 2026.
Related Articles
AI Accelerators Usher In New Era For IC Test
The number and variety of test interfaces, coupled with increased packaging complexity, are adding a slew of new challenges.
AI Accelerator Testing Depends On DFT Innovations
Multi-die assemblies greatly increase the number of things that can go wrong, and the difficulty of finding them.
Wafer Probe Struggles To Adapt To Multi-Die Assemblies
Delicate features, uneven surfaces, and extreme density make it difficult to manage probe force and ensure reliability.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Leave a Reply