Compatibility Ratio as a Guideline for Hardware/DNN Co-Design of Embedded Accelerators

ACM Transactions on Embedded Computing Systems. Increasingly large deep neural networks (DNNs) pose a significant challenge regarding required compute capabilities and energy consumption, especially at the edge. This challenge generally necessitates dedicated inference hardware accelerators. Hardware/software co-design can improve overall system performance by jointly optimizing the hardware micro-architecture and DNN workload mapping. It is however not easy to quantify how well such an accelerator generalizes to workloads not considered during hardware/software co-design. In response to this, we introduce the Compatibility Ratio (CR) as a simple guideline for evaluating performance tradeoffs between optimal hardware micro-architecture configurations across different workloads. CR allows us to quantify the performance tradeoffs of deploying a workload on an accelerator optimized for a different workload. We demonstrate CR through two case studies on a systolic array-based accelerator. First, we apply CR to explore the design space of the accelerator across 13 DNN workloads. In this case study, CR analysis showed that the choice of representative workload during co-design can implicitly increase the normalized area-latency cost of unconsidered workloads by more than 30% in the evaluated design space. Furthermore, we use CR to analyze how well our systolic array-based accelerator template can generalize beyond a single DNN workload to cover a family of DNN workloads. Our findings show that, for the considered accelerator, a DNN model-family optimized configuration might occupy an effective middle ground between highly targeted single- and general-purpose configurations. Second, we use CR as a guideline for a practical memory-retargeting decision in a specialized variant of our accelerator template. In this case study, CR quantifies whether a memory-retargeted accelerator derivative is justified under the selected memory-area and latency objective. For this two-configuration memory-retargeting case, analytical CR differs from implementation-level CR by 0.01, corresponding to one percentage point on the normalized CR scale.