Deep Learning Delivers Fast, Accurate Solutions For Object Detection In The Automated Optical Inspection Of Electronic Assemblies

When automated optical inspection (AOI) works, it is almost always preferable to human visual inspection. It can be faster, more accurate, more consistent, less expensive, and it never gets tired. However, some tasks that are very simple for humans are quite difficult for machines. Object detection is an example. For example, shown an image containing a cat, a dog, and a duck, a human can instantly confirm which objects are present, even when they overlap, and tell exactly what points in the image are included in each object. This seemingly simple task can be very challenging for AOI. In electronic assembly, a manufacturer may want to confirm the presence or absence, size, and location of a component. If all components are identical and change very little in vacation, this is a relatively simple task for AOI. If there is variability in the object’s appearance or location, the task is much more difficult. Corner fill is material used to secure an integrated circuit (IC) to a substrate. Though the location of the fill is relatively constant, its shape and size may vary from instance to instance. This variability makes detection much more complicated.

Machine learning uses software algorithms that automatically learn to improve a machine’s performance in a certain task based on feedback from past performance. Deep learning is a subset of machine learning. Deep learning runs on artificial neural networks, which learn using processes modeled on biological networks, i.e., brains. The “deep” in deep learning refers to the many layered architecture of the neural networks it uses. The system may be trained using explicit examples of the concept being learned or it may be left to discover patterns in the training data on its own. Labeled examples make training faster, but self-discovery opens the possibility that the system may find previously unknown relationships within the data.

In the case of corner fill, a memory manufacturer approached CyberOptics looking for an inspection solution that could detect the presence or absence of fill and measure its length. Traditional methods of corner fill inspection, such as blob analysis, are challenged by the lack of gray level specificity in this application. Blob analysis attempts to find a continuous blob within a certain intensity or contrast range and sometimes breaks larger blobs into separate smaller blobs. This customer’s results from inspection with blob analysis were inconsistent and unreliable, with many false negatives.

Research in the use of deep learning for object detection has made dramatic progress in recent years, driven by demand across a variety of applications, including facial recognition and autonomous driving. Autonomous driving shares some requirements with the corner fill application. It needs to be fast, i.e., it needs to detect objects, such as pedestrians and other cars, in nearly real-time. It needs to determine how big the object is and where it is in the field of view with enough accuracy to avoid a collision. It does not need to precisely define the edges of the object.

We proposed a deep learning solution based on new approach that that has gained wide acceptance in autonomous driving applications. Earlier approaches to object detection repurpose classifiers to perform detection. This approach applies a single neural network to an image, dividing it into regions and predicting bounding boxes that are weighted by class probabilities, all in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. It shares the same network architecture across all classes, which simplifies programming and speeds inferencing. The network can be trained on a personal computer with a single GPU (graphics processor). Once trained, inferencing can run on a device as simple as a mobile phone.

Corner fill is located below the corners of a relatively flat, rectangular package where it is not easy to see from any one point of view. Some corner fill inspection systems use a top-down camera and a mirror to view all sides of the package as it rotates – an approach that adds time and complexity to the data acquisition process. We used our SQ3000 Multi-Function System powered by Multi-Reflection Suppression (MRS) sensor technology, which incorporates a unique optical sensor originally designed for three-dimensional inspection and metrology using phase shift profilometry. The sensor views the inspection target simultaneously through four side-view cameras positioned off the normal axis at azimuths of 0°, 90°, 180°, and 270°. For corner fill inspection, the side-view cameras can instantly acquire images of all four sides, without mirrors or rotating the sample.

We trained and tested the system with a set of 72 images. 62 images were used for training and 10 for validation. Corner fill was labeled in the training images and the training was run on a standard personal computer with dual GPUs.

The results confirmed robust performance in detecting the presence or absence of corner fill and measuring its length. One simplified model was able to perform all necessary tasks. The bounding box proved to be sufficiently accurate for corner fill measurements. The system was easy to train and ran readily on a standard PC. Future integration of the deep learning algorithm into standard system software would allow factory engineers to train networks that could then be run locally for inspection. There are many more potentially valuable applications for deep learning object detection in SMT and semiconductor applications that we are actively pursuing.

For more information about this application of deep learning please visit: Case Study – Deep Learning for Corner Fill Inspection and Metrology on Integrated Circuits.