New Data Processing Module Makes Deep Neural Networks Smarter (Attentive Normalization)
Source: North Carolina State University. Authors: Xilai Li, Wei Sun, and Tianfu Wu
Abstract: “In state-of-the-art deep neural networks, both feature normalization and feature attention have become ubiquitous. They are usually studied as separate modules, however. In this paper, we propose a light-weight integration between the two schema and present Attentive Normalization (AN). Instead of learning a single affine transformation, AN learns a mixture of affine transformations and utilizes their weighted sum as the final affine transformation applied to re-calibrate features in an instance-specific way. The weights are learned by leveraging channel-wise feature attention.
In experiments, we test the proposed AN using four representative neural architectures in the ImageNet-1000 classification benchmark and the MS-COCO 2017 object detection and instance segmentation benchmark. AN obtains consistent performance improvement for different neural architectures in both benchmarks with absolute increase of top-1 accuracy in ImageNet-1000 between 0.5% and 2.7%, and absolute increase up to 1.8% and 2.2% for bounding box and mask AP in MS-COCO respectively. We observe that the proposed AN provides a strong alternative to the widely used Squeeze-and-Excitation (SE) module. The source codes are publicly available at the ImageNet Classification Repo (https://github.com/iVMCL/AOGNet-v2) and the MS-COCO Detection and Segmentation Repo (https://github.com/iVMCL/AttentiveNorm_Detection)”
Technical paper link can be found here.
An upbeat industry at the start of the year met one of its biggest challenges, but instead of being a headwind, it quickly turned into a tailwind.
Continuous design innovation adds to verification complexity, and pushes more companies to actually do it.
The semiconductor industry will look and behave differently this year, and not just because of the pandemic.
Experts at the Table: Any chip can be reverse-engineered, so what can be done to minimize the damage?
More than $1.5B in funding for 26 startups; December was a big month for AI hardware.
Experts at the Table: The open-source ISA is gaining ground in multiple markets, but the tool suite is incomplete and the business model is uncertain.
Taiwan and Korea are in the lead, and China could follow.
An upbeat industry at the start of the year met one of its biggest challenges, but instead of being a headwind, it quickly turned into a tailwind.
New data suggests that more chips are being forced to respin due to analog issues.
New horizontal technologies and vertical markets are fueling the opportunities for massive innovation throughout an expanding ecosystem.
Rising costs, complexity, and fuzzy delivery schedules are casting a cloud over next-gen lithography.
Experts at the Table: The current state of open-source tools, and what the RISC-V landscape will look like by 2025.
Nvidia-Arm is just the beginning; more acquisitions are on the horizon.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
Leave a Reply