Diss. ETH No. 30735
Architectural and
Microarchitectural Solutions
for Multi-Precision Hardware
Acceleration
A thesis submitted to attain the degree of
DOCTOR OF SCIENCES
(Dr. sc. ETH Zurich)
presented by
LUCA BERTACCINI
MSc EE, University of Bologna
born on 25.09.1995
accepted on the recommendation of
Prof. Dr. Luca Benini, examiner
Prof. Dr. Mauro Olivieri, co-examiner
2024
Acknowledgments
Working towards the Ph.D. degree has been one of the most challenging and rewarding journeys in my education. At the beginning of
this path, I could not have imagined how much I would have learned
from a professional and personal point of view. For all this, I want
to thank the many people who have accompanied and supported me
throughout the past years.
First, I want to thank my Ph.D. advisor, Luca. Thank you for
welcoming me into such a stimulating research group and guiding me
throughout this journey. I highly appreciate the exceptional opportunities you enable for this research group, while providing dedicated
mentorship for all team members. I would also like to express my
gratitude to Mauro for promptly accepting to co-referee my Ph.D.
thesis and providing valuable insights and feedback.
Special thanks go to Frank for his mentorship over the past years
and for always finding the time to give me advice.
Thanks to Francesco, who supervised my master’s thesis and introduced me to hardware acceleration and the research group. Thank
you also for the fruitful collaborations and advice throughout the past
years. Who knows if I had entered this journey without your guidance
and supervision in the last step of my master’s studies.
I would also like to thank Stefan, who guided me in the first months
of my Ph.D., introducing me to floating-point hardware, and Gianna
for the many suggestions, discussions, and collaborations throughout
these years.
With the long time horizon of a Ph.D. journey, the everyday
atmosphere plays a crucial role. For this, I cannot thank enough
all the people who shared the J78 office with me, creating a great,
iii
iv
ACKNOWLEDGMENTS
easy-going, productive environment. A special thank you to Robert,
Sergei, Michael, Luca V., Alfio, Andrea, and Gamze.
I would also like to thank Alessandro and Samuel for the many
collaborations and chats, Matteo for the first joint project of my
Ph.D., as well as Nils, Luca C., Tim, Victor, Arpan, Matheus, Angelo,
Yvan, Sergio, Florian, Marco, and everyone else at IIS and UNIBO
for the great time we shared.
I want to thank all my friends for all the amazing time we spent
together, which helped me relax and recharge. Finally, I am deeply
thankful to my whole family. Thanks to my parents and my brother,
who have always encouraged and supported me throughout my whole
life.
Abstract
Domain-specialized hardware accelerators are currently revolutionizing the entire spectrum of computing systems, from large data center
platforms to small Internet of Things (IoT) devices. These accelerators remarkably increase the energy efficiency of hardware systems,
which is especially important today, with the technology scaling benefits gradually diminishing and the ever-higher compute requirements
introduced by neural network (NN) applications.
These specialized units are, however, inherently less flexible than
general-purpose cores. As a consequence, they are often found in
heterogeneous systems-on-chip (SoCs), where a versatile host processor is coupled with domain-specialized engines. This approach
combines the best of both worlds: the flexibility and programmability
of general-purpose cores and the high energy efficiency of hardware
accelerators.
At the same time, the full customizability of hardware accelerators
presents many degrees of freedom to the designers. Depending on the
design, an accelerator can be more or less programmable and more or
less specialized. It can be conceived as a standalone hardware block or
designed to be integrated into a more general-purpose system or even
directly into the processor pipeline. It can provide wider or narrower
bandwidths to communicate with the other components of the system.
It can be a tiny unit up to a massive system, built by scaling up or
scaling out specific modules.
This thesis explores various architectural, microarchitectural, and,
in general, specialization solutions for hardware accelerators. First, we
v
vi
ABSTRACT
focus on low-cost area-constrained devices and compare different approaches to support floating-point (FP) operations: a fully softwarebased solution, a small multi-cycle floating-point unit (FPU), and a
high-performance pipelined FPU. Second, we analyze two domainspecific hardware accelerators. We concentrate on fast Fourier transform (FFT) engines and compare a standalone buffered module and
a buffer-less implementation sharing the memory with other cores.
Third, we consider a different avenue for specialization, extending a
cluster of specialized cores with support for low-precision FP formats
and mixed-precision FP operations. Fourth, we elaborate on this
hardware platform, identifying some of the limitations introduced by
low-precision formats and further enhancing the system to make it
more resilient to destructive events.
Riassunto
Gli acceleratori hardware stanno attualmente rivoluzionando l’intero
spettro dei sistemi computazionali, da grandi piattaforme per data
center a piccoli sistemi per l’Internet delle cose (IoT). Questi acceleratori aumentano notevolmente l’efficienza energetica delle piattaforme hardware. Ciò è particolarmente importante nello scenario
odierno, che vede i vantaggi della scalabilità tecnologica gradualmente
diminuire e i sempre più elevati requisiti computazionali introdotti
dalle applicazioni per reti neurali (NN).
Tuttavia, queste unità hardware altamente specializzate sono
intrinsecamente meno flessibili di processori per uso generale. Conseguentemente, si trovano spesso in sistemi su chip (SoC) eterogenei,
dove un processore versatile è accoppiato con acceleratori specializzati
per un dominio specifico. Questo approccio combina le caratteristiche
migliori di entrambe le soluzioni: la flessibilità e la programmabilità
dei processori per uso generale e l’elevata efficienza energetica degli
acceleratori hardware.
Allo stesso tempo, la completa adattabilità degli acceleratori
hardware per uno specifico scopo presenta molti gradi di libertà ai
progettisti. A seconda del progetto, un acceleratore può essere più o
meno programmabile e più o meno specializzato. Può essere concepito
come un blocco hardware autonomo o progettato per essere integrato
in un sistema più generico o anche direttamente nella pipeline di un
processore. Può fornire bande più o meno ampie per comunicare
con gli altri componenti del sistema. Può essere un’unità molto
piccola fino a un sistema particolarmente grande, costruito replicando
o espandendo moduli specifici.
vii
viii
RIASSUNTO
Questa tesi esplora diverse soluzioni architetturali, microarchitetturali e in generale relative alla specializzazione degli acceleratori
hardware. In primo luogo, ci focalizziamo su sistemi a basso costo
e con vincoli in termini di area e compariamo diversi approcci per
supportare operazioni a virgola mobile (FP): una soluzione completamente realizzata a livello software, una piccola unità hardware per
calcolare operazioni FP (FPU) iterativamente e una FPU contenente
una pipeline e specializzata per alte prestazioni. In secondo luogo,
analizziamo due acceleratori hardware specializzati per un dominio
specifico. Ci concentriamo su acceleratori per la trasformata di
Fourier veloce (FFT) e compariamo un modulo contenente un buffer
interno e un’implementazione dello stesso acceleratore senza buffer
che condivide la memoria con altri processori. In terzo luogo,
consideriamo un’altra strada per specializzare le unità hardware,
estendendo un sistema formato da un gruppo di processori con
supporto per formati FP a bassa precisione e per operazioni FP a
precisione mista. In quarto luogo, identifichiamo alcuni dei limiti
introdotti dai formati a bassa precisione ed estendiamo ulteriormente
il sistema hardware appena introdotto per renderlo più resiliente a
eventi distruttivi.
ix
In reference to IEEE copyrighted material which is used with permission in this
thesis, the IEEE does not endorse any of ETH Zurich’s products or services. Internal
or personal use of this material is permitted. If interested in reprinting/republishing
IEEE copyrighted material for advertising or promotional purposes or for creating
new collective works for resale or redistribution, please go to http://www.ieee.org/
publications standards/publications/rights/rights link.html to learn how to obtain a
License from RightsLink. If applicable, University Microfilms and/or ProQuest Library,
or the Archives of Canada may supply single copies of the dissertation.
Contents
Acknowledgments
iii
Abstract
v
Riassunto
vii
1 Introduction
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . .
1.1.1 Domain-Specific Hardware Acceleration . . . .
1.1.2 Data Specialization . . . . . . . . . . . . . . . .
1.1.3 Acceleration through ISA Extensions . . . . . .
1.1.4 Heterogeneous Systems-on-Chip . . . . . . . .
1.2 Outline . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . .
1.4 List of Publications . . . . . . . . . . . . . . . . . . . .
1
1
2
3
4
4
5
7
8
2 Microarchitectural Solutions for Floating-Point Support in Low-Cost MCU Cores
11
2.1 Introduction and Related Work . . . . . . . . . . . . . 12
2.2 Tiny-FPU Architecture . . . . . . . . . . . . . . . . . 14
2.3 Experiment Setup . . . . . . . . . . . . . . . . . . . . 17
2.3.1 Benchmarks . . . . . . . . . . . . . . . . . . . . 18
2.3.2 Performance, Power, and Energy . . . . . . . . 19
2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.1 Area and Code Size . . . . . . . . . . . . . . . 19
2.4.2 Performance . . . . . . . . . . . . . . . . . . . 21
xi
xii
CONTENTS
2.5
2.4.3 Power and Energy . . . . . . . . . . . . . . . .
Conclusions . . . . . . . . . . . . . . . . . . . . . . . .
23
25
3 Architectural Solutions for Domain-Specific Accelerators: To Buffer, or Not to Buffer?
27
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . 30
3.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3.1 PULP Cluster . . . . . . . . . . . . . . . . . . 32
3.3.2 Buffer-less FFT HWPE Architecture . . . . . . 34
3.3.3 Buffer-less FFT HWPE Reordering Scheme . . 37
3.3.4 Buffered FFT HWPE . . . . . . . . . . . . . . 39
3.4 Experimental Results . . . . . . . . . . . . . . . . . . . 41
3.4.1 Area . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.2 Butterfly Unit Utilization . . . . . . . . . . . . 42
3.4.3 Performance . . . . . . . . . . . . . . . . . . . 44
3.4.4 Power . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.5 Bandwidth Considerations . . . . . . . . . . . . 47
3.4.6 SoA Comparison . . . . . . . . . . . . . . . . . 48
3.5 Physical Implementations . . . . . . . . . . . . . . . . 49
3.5.1 Echoes . . . . . . . . . . . . . . . . . . . . . . . 49
3.5.2 Maestro . . . . . . . . . . . . . . . . . . . . . . 54
3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 57
4 Data Type Specialization: Low and Mixed-Precision
RISC-V ISA Extensions
59
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 60
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . 64
4.2.1 Low-Precision Floating-Point Formats . . . . . 64
4.2.2 Floating-Point Architectures and Designs . . . 66
4.2.3 Rounding Modes for Low-Precision NN Training 68
4.3 Mixed-Precision Instructions . . . . . . . . . . . . . . 69
4.4 Architecture . . . . . . . . . . . . . . . . . . . . . . . . 73
4.4.1 Supported FP Formats . . . . . . . . . . . . . . 74
4.4.2 wDotp Unit . . . . . . . . . . . . . . . . . . . . 75
4.4.3 wVsum and Vsum on the wDotp Datapath . . 81
4.4.4 Stochastic Rounding Support . . . . . . . . . . 81
4.4.5 SIMD Wrapper and Integration into FPnew . . 82
CONTENTS
4.5
4.6
4.7
4.8
4.4.6 MiniFloat-NN-Scalar Cluster . . . . . . . . . .
4.4.7 MiniFloat-NN-Vector Cluster . . . . . . . . . .
Experimental Results . . . . . . . . . . . . . . . . . . .
4.5.1 wDotp Unit Results . . . . . . . . . . . . . . .
4.5.2 MiniFloat-NN-Scalar Cluster Results . . . . . .
4.5.3 MiniFloat-NN-Vector Cluster Results . . . . .
From Optimized Kernels to High-Level Applications .
Silicon Implementations . . . . . . . . . . . . . . . . .
4.7.1 Eclipse . . . . . . . . . . . . . . . . . . . . . . .
4.7.2 Occamy . . . . . . . . . . . . . . . . . . . . . .
4.7.3 Carfield . . . . . . . . . . . . . . . . . . . . . .
Conclusion . . . . . . . . . . . . . . . . . . . . . . . .
xiii
84
88
89
89
95
98
103
103
103
105
107
107
5 Mitigating Data Type Specialization Limitations: Extending RISC-V for Efficient Overflow Recovery
111
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 112
5.2 Related work . . . . . . . . . . . . . . . . . . . . . . . 114
5.2.1 Low and Mixed Precision . . . . . . . . . . . . 114
5.2.2 Overflow Exception and Recovery Routines . . 116
5.3 Hardware and Software Baseline . . . . . . . . . . . . 118
5.3.1 MiniFloat Snitch Cluster . . . . . . . . . . . . 118
5.3.2 SSR and FREP-Based MiniFloat Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . 121
5.4 Architecture Extensions . . . . . . . . . . . . . . . . . 122
5.4.1 Low-Cost Overflow Detection . . . . . . . . . . 122
5.4.2 Agile Overflow Recovery . . . . . . . . . . . . . 124
5.5 Overflow Recovery Routine . . . . . . . . . . . . . . . 128
5.6 Experimental Results . . . . . . . . . . . . . . . . . . . 130
5.6.1 Area Results . . . . . . . . . . . . . . . . . . . 130
5.6.2 Performance Results . . . . . . . . . . . . . . . 130
5.6.3 Comparison with State-of-the-Art Solutions . . 134
5.7 Further Notes . . . . . . . . . . . . . . . . . . . . . . . 134
5.7.1 Floating-Point Precise Exceptions . . . . . . . 134
5.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 135
6 Conclusions
137
6.1 Main Results . . . . . . . . . . . . . . . . . . . . . . . 138
6.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . 140
xiv
CONTENTS
A Chip Gallery
143
B Acronyms
151
Bibliography
153
Curriculum Vitae
171
Chapter 1
Introduction
1.1
Motivation
The demand for computational resources is continuously increasing,
mainly driven by the growing complexity of new machine learning
(ML) algorithms and their ever-higher memory and compute requirements. This trend is currently affecting the whole spectrum of hardware systems, from small Internet of Things (IoT) devices to largescale, high-performance computing (HPC) systems.
With the growth of IoT, billions of tiny devices are now gathering
data from sensors. To reduce the amount of data sent to servers,
thereby saving bandwidth and energy, IoT systems process more and
more information directly on the edge [1]. In this context, neural
network (NN) inference has become one of the key workloads for these
devices.
At the same time, HPC systems are used more and more for large
ML training and inference workloads, especially with the advent of
large language models (LLMs). They are, therefore, required to keep
up with the ever-higher demands of modern ML models [2–5]. Recent
LLMs exceed the hundreds of billions of parameters [2, 4], imposing
tough requirements on the underlying hardware to efficiently execute
the related training and inference algorithms.
1
2
CHAPTER 1. INTRODUCTION
Since technology scaling is gradually providing diminishing benefits, new ways to increase the efficiency of hardware architectures are
required [6, 7]. In this scenario, hardware acceleration has become
key, as it boosts the performance and energy efficiency of computing
systems [8] by specializing portions of silicon to specific functions.
1.1.1
Domain-Specific Hardware Acceleration
Hardware accelerators significantly enhance the efficiency of computing systems by dedicating a specific area of silicon to a particular
task [8]. Although highly specialized engines have been around for
decades, they have gathered more and more interest with the diminishing benefits ensured by technology scaling. Many examples
can be found in today’s hardware systems, ranging from fast Fourier
transform (FFT) accelerators [9–11], already proposed decades ago,
up to the low-precision matrix and convolution accelerators [12–14]
that have been dominating the hardware acceleration research field
in the past years, pushed by the NN outburst. Hardware acceleration minimizes the overhead imposed by general-purpose computing,
thereby improving the utilization of the logic gates placed in a specific
portion of the chip. However, this specialization compromises the
flexibility of hardware accelerators, making them attractive or even
suitable only for a few application scenarios [15].
Hardware accelerators can, however, vary in specialization levels.
They can be suitable for a single function or a reduced set of tasks,
up to more versatile systems capable of performing a wide set of
functions but providing a higher efficiency only in a subset of tasks.
More general-purpose accelerators have emerged as an attractive solution for dynamic application scenarios, where the latest findings
could quickly make over-specialized designs obsolete. This pushed,
for example, the outstanding success of graphics processing units
(GPUs) [16, 17], which have proven versatile enough to keep up with
the evolution of NN algorithms. On the embedded side, manycore
general-purpose accelerators built with highly efficient cores, often
enhanced with domain-specialized instruction set architecture (ISA)
extensions, have boosted many application scenarios, from earbuds to
image-processing on the edge to drones [18].
1.1. MOTIVATION
3
Given their custom nature, hardware accelerators present many
degrees of freedom to designers, as well as a tough trade-off between
efficiency and versatility [15]. Depending on the target applications,
different design choices are made. In this thesis, we study several
hardware acceleration solutions, providing specific case studies.
1.1.2
Data Specialization
Data specialization has recently gained traction as a solution to further accelerate and increase the energy efficiency. In this context,
low and mixed-precision arithmetic has been widely employed and
is currently being pushed to narrower and narrower data representations. While scientific computing often requires 64-bit floating-point
(FP) values, the NN algorithms that emerged in the past decade
have proven to be more resilient to noise. Employing lower-precision
formats reduces the memory footprint and increases the compute
resources and energy efficiency [19, 20]. Since low-precision hardware
engines are more compact than higher-precision counterparts, they
allow for more compute modules to be placed into the same unit of
area, thereby increasing the performance and efficiency of the system.
All these benefits are highly desirable with the ever-higher memory
and compute requirements posed by NN algorithms. Low-precision
integer [21–23] and FP formats [24–27] have been widely employed for
error-resilient algorithms, such as NN inference and training. Current
efforts are pushing FP formats to narrower and narrower bitwidths [28],
and new arithmetic formats are currently being considered, such as
logarithmic number systems [29].
The large benefits enabled by low and mixed-precision arithmetic also led to many commercial platforms supporting narrow
data types [30–33]. This had a positive reinforcement, providing
researchers with low-precision hardware support, which is currently
being leveraged to investigate low-precision arithmetic for a wider set
of applications beyond NNs [34, 35].
In this thesis, we study reduced precision solutions on stand-alone
hardware accelerators and their integration into more general-purpose
cores. We show their benefits and provide mitigation strategies for
some of the limitations introduced by low-precision FP formats.
4
1.1.3
CHAPTER 1. INTRODUCTION
Acceleration through ISA Extensions
ISA extensions can be leveraged to boost the performance and efficiency of hardware architectures, specializing a general-purpose system without compromising its basic functionality. This results in
versatile platforms providing higher efficiencies in specific domains
but still being able to address a wide range of scenarios.
In this context, the RISC-V ISA [36] plays an interesting role.
RISC-V has recently gained traction in the industry and academia
thanks to its open, modular, and extensible nature. Its openness
prevents vendor lock-in. Its modularity, with a base integer ISA and a
set of optional ratified extensions, allows tailoring the ISA to a specific
application scenario. This results in having RISC-V central processing
units (CPUs) in embedded systems, such as in the GreenWaves GAP9
system-on-chip (SoC) [18], as well as in the automotive sector, where
SiFive provides a series of RISC-V processors [37], and in data center
platforms, like the Ventana Veiron V1 [38]. Its extensibility, with
part of its encoding space conceived for custom instructions, allows
designers to specialize their cores to specific contexts [39]. Various
custom extensions have been proposed in the literature, from DSP [40]
to NN [14] to sparse computing [41] and security [42] extensions.
In this thesis, we evaluate hardware acceleration solutions leveraging custom ISA extension, as well as the trade-offs of different
microarchitectural implementations employing open-source RISC-V
cores.
1.1.4
Heterogeneous Systems-on-Chip
To mitigate the trade-off between efficiency and versatility, heterogeneous systems-on-chip (HeSoCs) have become widespread. HeSoCs
combine the efficiency of domain-specialized accelerators with the
versatility of general-purpose cores, getting the best of two very distinct worlds [43]. The accelerator may include cores employing a
different ISA than the general-purpose processor. They may also employ different memory systems. The general purpose core, differently
from the accelerator, usually leverages coherent caches and virtual addresses to make the memory hierarchy transparent to the programmer.
1.2. OUTLINE
5
Furthermore, advanced offloading and synchronization techniques are
required to efficiently exploit the heterogeneity of HeSoCs.
For highly heterogeneous systems with many domain-specific
units, power management techniques are also crucial, as the more
specialized the accelerators, the larger the silicon portion unused in a
specific interval of time [44, 45].
This type of SoC is currently dominating the computing hardware
market, finding countless examples, from mobile processors to laptop
chips and automotive systems. Some examples of HeSoCs are the
Qualcomm Snapdragon 8 Gen 3 [46], a mobile processor implemented
in a 4 nm technology, the recently launched Apple M3, implemented in
a 3 nm technology and included both in the latest iMac and MacBook
Pro [47], and the NVIDIA Drive Orin [48] developed for automotive
applications.
In this thesis, we consider HeSoCs focusing on the accelerator side
and consider more general-purpose and more specialized engines.
1.2
Outline
This section provides a brief overview of the thesis outline. Most of the
material covered in the chapters has been previously published in peerreviewed conference and journal papers. Fig. 1.1 shows the structure
of the thesis and the dependencies between chapters. The chapters
are designed to be self-contained, allowing readers to consume them
individually. The chapters are arranged at an increasing complexity
of the hardware system being accelerated, progressively moving from
smaller to larger systems.
Chapter 2 In this chapter, we analyze various hardware acceleration solutions at a microarchitectural level. In particular, we focus on
different approaches to support FP operations on low-cost microcontroller unit (MCU) cores and design a tiny multi-cycle floating-point
unit (FPU) for area-constrained MCUs. We then compare an integeronly core emulating FP operations on the integer datapath with the
same integer core coupled, in one case, with the proposed multi-cycle
FPU and, in the other case, with a high-performance full-fledged
6
CHAPTER 1. INTRODUCTION
Microarchitectural
Solutions
Chapter 2
Introduction to HW
Acceleration
Chapter 1
Architectural
Solutions
Chapter 3
Data Type
Specialization
Chapter 4
Conclusions
Chapter 6
Mitigating Data
Type Specialization
Limitations
Chapter 5
Figure 1.1: Structure of the thesis and dependencies between chapters.
FPU. We evaluate the various solutions in terms of code size, area,
performance, power consumption, and energy efficiency.
Chapter 3 In the next step, we investigate different architectural
solutions for hardware accelerators. Domain-specialized hardware
engines are traditionally designed as stand-alone units with their own
memories, which often dominate the accelerator’s area. They are then
coupled to a general-purpose system with its own memory, thereby
requiring explicit data copies. We design two FFT hardware accelerators, one including large buffers and one storing its partial results
into a scratchpad memory shared with the general-purpose system.
We provide a comprehensive evaluation of the two solutions, showing
the benefits of the latter approach.
Chapter 4 We then concentrate on a different approach, boosting
the performance and energy efficiency of accelerators: data specialization. With the ever-higher memory and compute requirements
posed by the larger and larger NN models, low and mixed-precision
FP arithmetic has become widespread in modern hardware platforms.
We design a low-precision widening dot-product unit supporting two
8-bit and two 16-bit formats and accumulating at higher precision.
1.3. CONTRIBUTIONS
7
We define two RISC-V ISA extensions, one for scalar cores and one
for vector processors, to enhance them with low and mixed-precision
FP support. Finally, we evaluate their benefits in the two scenarios.
Chapter 5 Motivated by the great benefits introduced by low
and mixed-precision arithmetic, low-precision formats are nowadays
supported by many commercial hardware platforms. At the same
time, this support is pushing research efforts to apply low-precision
formats for a broader set of applications, going beyond NN training
and inference. The reduced exponent bitwidth, however, translates to
narrower representable ranges, thereby increasing the probability of
overflow. While NN-related algorithms often saturate overflowing values, this approach would not be viable in safety-critical applications.
In this chapter, we devise an overflow recovery scheme leveraging the
flexibility of the mixed-precision ISA introduced in Chapter 4.
Chapter 6 The final chapter concludes the thesis with a short
summary of the main findings and provides an outlook on future work.
1.3
Contributions
The key contributions of this thesis and the related publications are
summarized below. The associated chapters provide comprehensive
introductions to the topics and in-depth analyses.
1. We design a multi-cycle FPU for cost-constrained MCU cores
(Chapter 2, [49]).
2. We compare different microarchitectural solutions to support
FP operations (Chapter 2, [49]).
3. We design two FFT hardware accelerators, one including internal buffers and a buffer-less design leveraging a scratchpad shared with general-purpose cores to store partial results
(Chapter 3, [50]).
8
CHAPTER 1. INTRODUCTION
4. We evaluate and compare the buffered and buffer-less FFT
accelerators. We integrate the buffer-less design into a singlecore MCU and provide silicon measurements for the resulting
system (Chapter 3, [50, 51]).
5. We design a low and mixed-precision dot-product unit supporting two 8-bit and two 16-bit FP formats and accumulating at
higher precision (Chapter 4, [52, 53]).
6. We define two low and mixed-precision RISC-V ISA extensions,
one for scalar cores and one for vector processors, leveraging the
proposed widening dot product unit. We evaluate the extensions
by integrating them into two open-source clusters of RISC-V
cores (Chapter 4, [52, 53]).
7. We discuss some of the limitations introduced by low-precision
FP arithmetic and devise mitigation strategies (Chapter 5, [54])
1.4
List of Publications
Most of the content covered in this thesis has been published in the
following peer-reviewed conference and journal papers:
[49] L. Bertaccini, M. Perotti, S. Mach, P. D. Schiavone, F. Zaruba, and
L. Benini, “Tiny-FPU: low-cost floating-point support for small RISC-V
MCU cores,” in 2021 IEEE International Symposium on Circuits and
Systems (ISCAS). IEEE, 2021, pp. 1–5
[50] L. Bertaccini, L. Benini, and F. Conti, “To buffer, or not to buffer? a
case study on FFT accelerators for ultra-low-power multicore clusters,” in
2021 IEEE 32nd International Conference on Application-specific Systems,
Architectures and Processors (ASAP). IEEE, 2021, pp. 1–8
[52] L. Bertaccini, G. Paulin, T. Fischer, S. Mach, and L. Benini, “MiniFloat-NN
and ExSdotp: An ISA extension and a modular open hardware unit for
low-precision training on RISC-V cores,” in 2022 IEEE 29th Symposium
on Computer Arithmetic (ARITH). IEEE, 2022, pp. 1–8
[53] L. Bertaccini, G. Paulin, M. Cavalcante, T. Fischer, S. Mach, and L. Benini,
“MiniFloats on RISC-V cores: ISA extensions with mixed-precision short
dot products,” IEEE Transactions on Emerging Topics in Computing, pp.
1–16, 2024
1.4. LIST OF PUBLICATIONS
9
[54] L. Bertaccini, S. Shen, T. Hoefler, and L. Benini, “Extending RISC-V for
efficient overflow recovery in mixed-precision computations,” in 2024 IEEE
42nd International Conference on Computer Design (ICCD). IEEE, 2024,
pp. 268–275
The following publications with contributions by the author have been
partially covered in this thesis:
[51] M. Sinigaglia, L. Bertaccini, L. Valente, A. Garofalo, S. Benatti, L. Benini,
F. Conti, and D. Rossi, “Echoes: a 200 GOPS/W frequency domain
SoC with FFT processor and I2S DSP for flexible data acquisition from
microphone arrays,” in 2023 IEEE International Symposium on Circuits
and Systems (ISCAS), 2023, pp. 1–5
[55] G. Paulin, P. Scheffler, T. Benz, M. Cavalcante, T. Fischer, M. Eggimann,
Y. Zhang, N. Wistoff, L. Bertaccini, L. Colagrande, G. Ottavi, F. K.
Gürkaynak, D. Rossi, and L. Benini, “Occamy: A 432-core 28.1 DPGFLOP/s/W 83% FPU utilization dual-chiplet, dual-HBM2E RISC-Vbased accelerator for stencil and sparse linear algebra computations with
8-to-64-bit floating-point support in 12nm FinFET,” VLSI Symposium,
2024
The author has contributed to the following publications while
working towards his degree, which are not explicitly covered in this
thesis but provide additional related insights:
[56] Y. Tortorella, L. Bertaccini, L. Benini, D. Rossi, and F. Conti, “RedMule:
A mixed-precision matrix–matrix operation engine for flexible and energyefficient on-chip linear algebra and TinyML training acceleration,” Future
Generation Computer Systems, vol. 149, pp. 122–135, 2023
[57] G. Paulin, M. Cavalcante, P. Scheffler, L. Bertaccini, Y. Zhang, F. Gürkaynak, and L. Benini, “Soft tiles: Capturing physical implementation flexibility for tightly-coupled parallel processing clusters,” in 2022 IEEE Computer
Society Annual Symposium on VLSI (ISVLSI). IEEE, 2022, pp. 44–49
[58] T. Benz, L. Bertaccini, F. Zaruba, F. Schuiki, F. K. Gürkaynak, and
L. Benini, “A 10-core SoC with 20 fine-grain power domains for energyproportional data-parallel processing over a wide voltage and temperature
range,” in ESSCIRC 2021-IEEE 47th European Solid State Circuits
Conference (ESSCIRC). IEEE, 2021, pp. 263–266
[59] M. Perotti, G. Tagliavini, S. Mach, L. Bertaccini, and L. Benini, “RVfplib:
A fast and compact open-source floating-point emulation library for tiny
RISC-V processors,” in International Conference on Embedded Computer
Systems. Springer, 2021, pp. 16–32
10
CHAPTER 1. INTRODUCTION
[60] Y. Tortorella, L. Bertaccini, D. Rossi, L. Benini, and F. Conti, “RedMulE: A
compact FP16 matrix-multiplication accelerator for adaptive deep learning
on RISC-V-based ultra-low-power SoCs,” in 2022 Design, Automation &
Test in Europe Conference & Exhibition (DATE). IEEE, 2022, pp. 1099–
1102
[61] V. Potocnik, L. Colagrande, T. Fischer, L. Bertaccini, D. J. Pagliari,
A. Burrello, and L. Benini, “Optimizing foundation model inference on
a many-tiny-core open-source risc-v platform,” IEEE Transactions on
Circuits and Systems for Artificial Intelligence, vol. 1, no. 1, pp. 37–52,
2024
Chapter 2
Microarchitectural
Solutions for
Floating-Point Support
in Low-Cost MCU Cores
In this chapter, we analyze different microarchitectural solutions to
support FP operations on low-cost MCU cores. In particular, we
compare a fully software-based approach emulating the FP operations
on the integer datapath and a full-fledged hardware FPU optimized
for high performance. In addition, we develop a tiny multi-cycle
hardware FPU, providing an intermediate design point for low-cost
area-constraint MCU cores. We provide a detailed evaluation of each
solution and its trade-offs in terms of area, code size, power, performance, and energy efficiency.
The following sections have been published in a slightly different
form in the IEEE International Symposium on Circuits and Systems
(ISCAS) 1 .
1 ©2021 IEEE. Reprinted, with permission, from L. Bertaccini, M. Perotti,
S. Mach, P. D. Schiavone, F. Zaruba, and L. Benini, “Tiny-FPU: low-cost
11
12
2.1
CHAPTER 2. MICROARCHITECTURAL SOLUTIONS
Introduction and Related Work
The era of the IoT is characterized by billions of devices gathering
data from sensors and sending them to powerful servers, where these
data are processed using intelligent algorithms. To save bandwidth
and energy across the whole network infrastructure, IoT devices can
pre-process sensor data with advanced algorithms like filters and compressors [62], giving birth to so-called edge computing devices. Such
algorithms differ in complexity, ranging from requiring only a few
integer or fixed-point operations to performing dense kernels using
FP arithmetic.
While FP arithmetic offers higher precision and dynamic range
than fixed-point, it is usually associated with high implementation
costs in hardware and thus often avoided in area and energyconstrained devices such as MCUs. For this reason, many edge
computing devices support fixed-point arithmetic instead, which
introduces a lower power and area overhead with respect to hardware
FPUs [63]. In this context, several ISA extensions have been
developed to efficiently support fixed-point operations [40, 64] on
embedded systems. However, when applications require a high
dynamic range, system designers face the choice between using slow
software emulation on very low-power and area-efficient integer-only
cores or paying the penalties in area and power for choosing a
processor with fast hardware FPUs.
In the first case, it is desirable to employ small low-power cores
with the minimum set of features to fulfill the requirements, such as
the widely known Arm Cortex-M0 [65] and lowRISC Ibex [66], an
open-source area-optimized RISC-V core supporting the RV32IMC
ISA. These cores need special FP-emulation libraries, often distributed
by toolchain providers together with their compilers. For example,
GNU Compiler Collection (GCC) supports FP emulation via its
low-level runtime library libgcc [67], automatically replacing every
FP operation with a libgcc function call at compile time. However,
software emulation leads to long execution times that can compromise
the overall system’s energy efficiency. Moreover, the emulation
floating-point support for small RISC-V MCU cores,” in 2021 IEEE International
Symposium on Circuits and Systems (ISCAS). IEEE, 2021, pp. 1–5.
2.1. INTRODUCTION AND RELATED WORK
13
functions use several kB of memory, which is a precious resource on
low-cost MCUs.
In the second case, FP operations are compiled to instructions that
are executed using the dedicated datapath of an FPU. This approach
is faster than emulation, taking only a few cycles for each operation,
and it results in a smaller code size as no software library is required.
For example, more performant processors such as the Arm CortexM4 [68] and the RISC-V OpenHW Group CV32E40P [40] natively
support a hardware FPU.
At the same time, the request for extremely efficient devices and
the need to process FP-intensive workloads also led to designs coupling
a tiny integer processor with larger, performant FPUs. This is the case
of Snitch [69], a tiny 32-bit integer RISC-V core extended with an FP
subsystem containing a multi-format FPU optimized for performance
and energy efficiency [70], an FP register file, a set of registers to
break the long combinatorial paths commonly found around FPUs, a
decoder, and a load-store unit. Together, the Snitch integer core and
its FP subsystem form a Snitch core complex (CC).
The extra silicon area required by a full-fledged FPU may,
however, not be affordable for low-cost MCUs. For this reason,
Hockert and Compton [71] and Pimentel et al. [72] proposed hybrid
hardware/software solutions where modular hardware FP-related
sub-routines accelerated the FP computation, introducing low-area
overhead. Each hybrid implementation of an FP operation designed
by Pimentel et al. occupies 77.3 − 97.0% less area than a full
dedicated hardware support for FP fused multiply-add, speeding
up by up to 7.28× FP additions/subtractions and 2.03× for FP
multiplications, when compared to the corresponding software-only
solutions. However, both works only targeted single-precision FP and
required adding new custom instructions to the base ISA.
This chapter compares different microarchitectural solutions
to support FP operations and introduces Tiny-FPU, a RISC-Vcompliant2 FPU, significantly smaller than a full-fledged FPU, and, at
the same time, faster and less memory hungry than software emulation
2 Tiny-FPU supports all the instructions included in the RISC-V ISA double
and single-precision FP extensions, except for division and square-root.
14
CHAPTER 2. MICROARCHITECTURAL SOLUTIONS
(as it executes native FP instructions). In particular, we cover the
following contributions:
1. Tiny-FPU 3 , a novel area-optimized IEEE 754-compliant
RISC-V FPU, designed for cost-constrained MCUs that require
double and/or single-precision FP support.
2. A detailed comparison, in terms of area, code size, performance,
power, and energy efficiency, of our proposed solution with both
a high-performance and energy-efficient FPU and the default
software FP emulation provided by GCC’s low-level runtime
library (libgcc) on Snitch-based systems.
2.2
Tiny-FPU Architecture
To bridge the gap between high-performance hardware FPUs and
inefficient software emulation, we propose Tiny-FPU, a hardware
IEEE 754-compliant RISC-V FPU optimized for area. The unit
exploits iterative, multi-cycle executions to reuse datapath resources
in a time-multiplexed fashion.
Fig. 2.1 shows the proposed FPU architecture highlighting the
main datapath components (and their bitwidths), as well as the finitestate machine that selects the operands and intermediate results cycleby-cycle.
Tiny-FPU is available in two different versions:
1. Tiny64 : A double-precision Tiny-FPU with support for singleprecision instructions, which are computed on the doubleprecision datapath, allowing the single-precision support at the
cost of around 5% area overhead and one extra cycle every time
a multiplication is required (fmul.s, fmadd.s).
2. Tiny32 : a single-precision Tiny-FPU.
Since FP division and square root are rarely used, and their dedicated hardware implementation would remarkably impact the FPU
area, we do not include the support for these two instructions inside
3 https://github.com/pulp-platform/cvfpu/tree/feature/tiny-fpu
Figure 2.1: Tiny-FPU Schematic.
2.2. TINY-FPU ARCHITECTURE
15
16
CHAPTER 2. MICROARCHITECTURAL SOLUTIONS
Tiny-FPU.4 All the other instructions included in the RISC-V double
and single-precision FP ISA extensions (D- and F-extensions) are
supported and can be carried out by Tiny-FPU through multi-cycle
executions.
To develop Tiny-FPU, we take the open-source high-performance
pipelined FPU developed by Mach et al. [70] (Fast-FPU) as a starting
point. Its implementation requires multiple adders, shifters, and a
multiplier, whose bitwidths depend on the number of mantissa bits,
including the hidden one (p) and the number of exponent bits (e).
The largest components for each class of arithmetic hardware blocks
are:
• A p-bit mantissa multiplier;
• A (3p + 4)-bit adder used to sum the product and the addend
during an fmadd;
• A shifter used to normalize a (3p + 4)-bit internal result.
To minimize the area of Tiny-FPU:
• We reduce one of the multiplier inputs from p bits to 2 bits to
compute, leveraging an accumulator, the p-bit multiplication in
p/2 cycles.
• We reduce the adder bitwidth by 1/3. Such an adder allows
accumulating the partial products in one cycle, while it requires
three cycles to compute the addition needed during an fmadd.
• We reduce the shifter input width by 1/3. This component is
used to perform the addend shift, required by an fmadd and an
fadd, in one cycle and the normalization shift in three cycles.
• We minimize the number of arithmetic components. All the
datapath blocks are reused to compute operations requiring the
same or lower bitwidths. Furthermore, the same basic arithmetic components are used to compute all the supported FP
instructions.
4 FP division and square root can be software emulated.
17
2.3. EXPERIMENT SETUP
Table 2.1: FP instruction latencies - Tiny-FPU.
Latency [cycles]
FP32
FP32 (on FP64 datapath)
FP64
fmadd
fadd
fsub
fmul
comparison
cast
21-24
22-25
36-39
10-13
10-13
10-13
10-13
10-13
10-13
18
19
33
2
2
2
9
9
9
• We minimize the number of registers by overwriting them with
new partial results once their content has already been processed
and is no longer relevant.
The FP instruction latencies for both single and double-precision
are listed in Table 2.1. Since the adder bitwidth has been reduced
by 1/3, every time a negative mantissa is produced as a result of
an addition/subtraction, three extra cycles are needed to take the
absolute value of the result and change its sign bit. Whenever an
Inf/Nan input is detected, the result is provided in two cycles.
2.3
Experiment Setup
To evaluate our novel Tiny-FPU and the related area, performance,
and power trade-offs, we couple it to a Snitch core [69] and compare
it with the same core leveraging a full-fledged FPU or software emulation, the two baseline alternative solutions that best fit in an FP
and integer-intensive context, respectively. As a full-fledged FPU, we
take the open-source unit designed by Mach et al. [70]. We will refer
to this unit as Fast-FPU. Fast-FPU is optimized for high performance
and energy efficiency with a tunable number of pipeline stages. For
the evaluation of Fast-FPU, we use a configuration with no pipeline
registers as it could already meet our frequency target.
We compare then five different solutions:
• An integer-only Snitch core (Snitch-int).
• A Snitch-core coupled with a 32-bit Tiny-FPU (Snitch-tiny32).
18
CHAPTER 2. MICROARCHITECTURAL SOLUTIONS
• A Snitch-core coupled with a 64-bit Tiny-FPU (Snitch-tiny64).
• A Snitch-core coupled with a 32-bit Fast-FPU (Snitch-fast32).
• A Snitch-core coupled with a 64-bit Fast-FPU (Snitch-fast64)
In general, we refer to Snitch-tiny for a Snitch core coupled with
Tiny-FPU and Snitch-fast for a Snitch core coupled with Fast-FPU.
Together, an integer Snitch core and its FPU form a Snitch CC (core
complex).
We characterize the power, performance, area (PPA), and energy
efficiency impact of these design points. We evaluate each solution
by running benchmarks with different percentages of FP instructions,
leveraging single-core Snitch-based systems, including a data memory
and an instruction memory. As we target tiny and area-constrained
MCUs, we include a small 2 KiB instruction cache in these systems.
We synthesize the five systems using Synopsys Design Compiler in
Globalfoundries 22FDX, with topographical synthesis, targeting a
clock frequency of 100 MHz under worst-case conditions (SSG, 0.72 V,
−40 ◦ C). We then evaluate the power consumption for the various
solutions under typical conditions (TT, 0.80 V, 25 ◦ C) at the same
clock frequency.
2.3.1
Benchmarks
To cover a wide range of applications, we evaluate the different figures
of merit of the compared solutions as a function of the percentage
of executed FP instructions in the benchmarks. We used a synthetic benchmark (synth), a pattern-recognition algorithm (knn), a
2D convolution (conv2d), a ReLU kernel, an axpy algorithm, and an
excerpt of the FANN library [73] in two versions: integer-only (fannint) and FP-only (fann-fp). The synthetic benchmark has two backto-back matrix multiplication loops, the first operating on integer
numbers and the second on FP numbers, double or single-precision,
depending on the considered system. Tuning the number of iterations
of each loop ensured different percentages of executed FP operations,
producing higher or lower utilizations for the FP subsystem when
present or a higher or lower time spent in emulating FP arithmetic on
the integer-only system. We count the percentage of FP operations
2.4. RESULTS
19
(%FP) in a program as the amount of executed FP computational,
comparison, and conversion instructions [36] divided by the total number of instructions.
Starting from our parametric benchmark, we generate seven C
programs that resulted in different %FP, namely 0.07%, 0.65%, 2.9%,
8.8%, 16%, 32%, 53%.5 We compile the application sources with GCC
10.1.0, targeting RV32IMA[FD] systems. All the emulation functions
come from the libgcc GCC library.
2.3.2
Performance, Power, and Energy
We simulate each benchmark with Mentor Questasim on all our systems at a frequency of 100 MHz. For each simulation, we extract the
related average power with Synopsys PrimeTime and calculate the
associated energy consumption.
2.4
Results
In the following section, we present the results obtained from the synthesis of the Snitch systems leveraging different solutions to support
FP and our experiments on double and single-precision for both the
synthetic and the real benchmarks. We show the throughput, power,
and efficiency results for the double-precision experiments in Fig. 2.4.
Similar trends are obtained with the single-precision counterparts. In
this section, we indicate the results for double-precision with (DP)
and for single-precision with (SP).
2.4.1
Area and Code Size
In Fig. 2.2, we show the area results for the different architectures
considered in the comparison, highlighting the FPU, the remaining
part of the FP subsystem (FP SS), and the integer core (Snitch
CC Int). Tiny-FPU, by itself, occupies around 21 kGE (DP) and
9.5 kGE (SP). It is 54% (DP) and 37% (SP) smaller than Fast-FPU,
while the savings for the overall system are around 20% (DP) and
5 These numbers slightly vary for the same program compiled with float or
double, because of the different compiler target architecture.
20
CHAPTER 2. MICROARCHITECTURAL SOLUTIONS
Area - Snitch64 @100 MHz
Snitch-fast64
Snitch-tiny64
Snitch CC Int
Snitch-int
0
20
40
60
80
FP SS w/o FPU
100
120
FPU
140
Area (kGE)
Area - Snitch32 @100 MHz
Snitch-fast32
Snitch-tiny32
Snitch CC Int
Snitch-int
0
20
40
60
80
FP SS w/o FPU
100
120
FPU
140
Area (kGE)
Figure 2.2: Area comparison of Snitch-fast, Snitch-tiny, and Snitch-int
for double and single-precision. Netlists synthesized at 100 MHz.
8% (SP). Snitch-tiny is 2.8× (DP) and 1.9× (SP) larger than the
integer-only core, also due to the large area overhead associated with
the FP register file, which occupies around 70% of the FP SS area
not occupied by the FPU, and more than Tiny-FPU itself. The
FP register file impact on the area, however, can be mitigated by
leveraging the Zdinx (DP) and Zfinx (SP) RISC-V extensions [74],
which allow for sharing the integer register file for both integer and
FP data, eliminating the need for a separate FP register file and
load/store unit. A Zfinx-compliant Snitch system with the singleprecision Tiny-FPU would be only around 52 kGE, thus only 1.5×
larger than the integer-only core (see Fig. 2.3). As can be seen in
Fig. 2.2, Snitch CC Int’s area is slightly smaller for the Snitch-int
system as the missing FP support reduces the area of some of the
hardware components in the Snitch CC Int, such as the decoder or
the arbitration of the register file write and read ports.
Note that the integer core suffers from the significant code size
overhead introduced by the FP emulation functions. A full software
library occupies up to 14.8 kB (both DP and SP) and 5.5 kB (SP),
whereas supporting only FP division and square root requires 3.1 kB
Area - Snitch64 @100 MHz
Snitch-fast64
Snitch-tiny64
Snitch-int
2.4. RESULTS
0
Snitch CC Int
20
40
60
80
FP SS w/o FPU
100
120
21
FPU
140
Area (kGE)
Area - Snitch32 @100 MHz
Snitch-fast32
Snitch-tiny32-Zfinx
Snitch-int
Snitch CC Int
0
20
40
60
80
FP SS w/o FPU
100
120
FPU
140
Area (kGE)
Figure 2.3: Area comparison of Snitch-fast32, Snitch-tiny32-Zfinx,
and Snitch-int. Netlists synthesized at 100 MHz. Snitch-tiny32-Zfinx
is only around 1.5× larger than the integer-only solution. The FP SS
area is considerably reduced by removing the FP register file and
load/store unit. The integer part of the core is slightly larger due to
an extra register file port added to support FP fused multiply-add
instructions, which require three source registers and one destination
register. Similarly, the overhead can be considerably reduced in the
double-precision support case by leveraging the Zdinx extension.
(both DP and SP) and 1.1 kB (SP). Therefore, Tiny-FPU reduces the
maximum code size needed for FP emulation by 79%.
2.4.2
Performance
Fig. 2.4a traces the reciprocal of the benchmark execution time, normalized on the Snitch-fast results. When the %FP increases, the
integer system’s performance drastically drops, showing the limits of
software emulation. Already for %FP ≈ 4% (DP) and %FP ≈ 8%
(SP), Snitch-int is 5× slower than Snitch-fast. Snitch-tiny always
outperforms Snitch-int, achieving up to 18.5× (DP) and 15.5× (SP)
speedups for %FP ≈ 53%. The large code size of the software emulation functions also contributes to the lower performance of Snitch-int
with respect to Snitch-tiny, as it introduces several instruction cache
misses. For applications requiring few FP instructions, the performance gap between Snitch-fast and Snitch-tiny decreases considerably.
For less than 5% of FP instructions, Snitch-tiny’s execution time is
only up to 1.29× (DP) and 1.16× (SP) longer. On real benchmarks,
Snitch-tiny is 16.9× (DP) and 13.7× (SP) faster than Snitch-int when
22
CHAPTER 2. MICROARCHITECTURAL SOLUTIONS
a) Norm. (Exec. Time)-1 vs. %FP - Snitch CC
Normalized (Exec. Time) -1
Snitch-tiny64
Snitch-fast64
Snitch-int
10 0
10 -1
10 -2
0
10
20
30
40
50
60
%FP computational instructions on total
b) Norm. Power vs. %FP - Snitch CC @100 MHz
Snitch-tiny64
1.2
Snitch-fast64
Snitch-int
Normalized Power
1
0.8
0.6
0.4
0.2
0
0
10
20
30
40
50
60
%FP computational instructions on total
c) Norm. Energy eff. vs. %FP - Snitch CC @100 MHz
Normalized Energy Efficiency
Snitch-tiny64
Snitch-fast64
Snitch-int
10 0
10 -1
10 -2
10 -1
10 0
10 1
10 2
%FP computational instructions on total
Figure 2.4: Performance, average power, and energy efficiency of
Snitch-fast, Snitch-tiny, and Snitch-int, normalized to Snitch-fast.
23
2.4. RESULTS
Table 2.2: Real-Benchmarks Results (fclk = 100 MHz): Execution
Time.
Execution Time [cycles]
Benchmark64
snitch-fast64
snitch-tiny64
snitch-int
axpy (34.4%)
conv2d (20.2%)
fann-fp (18.7%)
ReLU (11.7%)
knn (6.6%)
fann-int (0%)
2.4k (1.0)
7.8k (1.0)
43.8k (1.0)
5.0k (1.0)
11.1k (1.0)
64.0k (1.0)
11.2k (4.6)
22.9k (2.9)
117.7k (2.7)
5.2k (1.1)
13.3k (1.2)
64.0k (1.0)
152.5k
385.9k (49.7)
1009.3k (23.1)
22.6k (4.5)
71.9k (6.5)
69.0k (1.1)
Benchmark32
snitch-fast32
snitch-tiny32
snitch-int
axpy (34.4%)
conv2d (19.1%)
fann-fp (21.0%)
ReLU (14.1%)
knn (7.6%)
fann-int (0%)
2.5k (1.0)
7.8k (1.0)
35.5k (1.0)
4.7k (1.0)
7.8k (1.0)
64.1k (1.0)
7.3k (3.0)
13.9k (1.8)
73.6k (2.1)
5.0k (1.1)
9.2k (1.2)
64.1k (1.0)
109.2k (44.4)
191.7k (24.6)
766.3k (21.6)
15.9k (3.4)
38.0k (4.9)
69.0k (1.1)
running conv2d, and 5.4× (DP) and 4.1× (SP) faster when running
knn. The results for ReLU differ from this trend since the kernel just
contains comparison instructions, which are fast to compute on both
Snitch-tiny and Snitch-int. We summarize the results obtained on the
real benchmarks in Table 2.2.
2.4.3
Power and Energy
Fig. 2.4b and Fig. 2.4c show the power and energy efficiency results for
different %FP, normalized to Snitch-fast. For %FP ≈ 53%, Snitch-tiny
consumes around 50% (DP) and 37% (SP) less power than Snitchfast. The power difference between Snitch-fast and Snitch-tiny is less
than 5% for %FP < 5%. The deflection of Snitch-tiny and Snitch-int
after %FP ≈ 16% is due to a steep increase in Snitch-fast’s power
consumption, generated by a heavier resource utilization, which also
leads to higher performance.
Overall, Snitch-int is the most energy-efficient only when the program has almost no FP instructions. On the other hand, for FPintensive workloads (%FP ≈ 53%), Snitch-tiny reaches 8.5× (DP)
and 9.6× (SP) the energy efficiency of Snitch-int.
24
CHAPTER 2. MICROARCHITECTURAL SOLUTIONS
Table 2.3: Real-Benchmarks Results (fclk = 100 MHz): Snitch CC
Power Consumption.
Power [µW]
Benchmark64
snitch-fast64
snitch-tiny64
snitch-int
axpy (34.4%)
conv2d (20.2%)
fann-fp (18.7%)
ReLU (11.7%)
knn (6.6%)
fann-int (0%)
375.6 (1.0)
341.4 (1.0)
243.5 (1.0)
199.1 (1.0)
220.8 (1.0)
155.6 (1.0)
254.4 (0.7)
272.9 (0.8)
204.7 (0.8)
201.6 (1.0)
220.3 (1.0)
152.1 (1.0)
142.3 (0.4)
124.3 (0.4)
109.4 (0.4)
114.9 (0.6)
116.8 (0.5)
118.7 (0.8)
Benchmark32
snitch-fast32
snitch-tiny32
snitch-int
axpy (34.4%)
conv2d (19.1%)
fann-fp (21.0%)
ReLU (14.1%)
knn (7.6%)
fann-int (0%)
240.4 (1.0)
237.4 (1.0)
227.1 (1.0)
149.2 (1.0)
182.0 (1.0)
141.0 (1.0)
175.4 (0.7)
208.5 (0.9)
184.8 (0.8)
153.2 (1.0)
177.9 (1.0)
140.0 (1.0)
116.4 (0.5)
132.3 (0.6)
111.4 (0.5)
108.1 (0.7)
119.0 (0.7)
118.7 (0.8)
Table 2.4: Real-Benchmarks Results (fclk = 100 MHz): Snitch CC
Energy Consumption.
Energy [nJ]
Benchmark64
snitch-fast64
snitch-tiny64
snitch-int
axpy (34.4%)
conv2d (20.2%)
fann-fp (18.7%)
ReLU (11.7%)
knn (6.6%)
fann-int (0%)
9.2 (1.0)
26.5 (1.0)
106.6 (1.0)
9.9 (1.0)
24.5 (1.0)
99.5 (1.0)
28.4 (3.1)
62.4 (2.4)
240.9 (2.3)
10.6 (1.1)
29.2 (1.2)
97.3 (1.0)
216.9 (23.7)
479.7 (18.1)
1104.5 (10.4)
26.0 (2.6)
84.0 (3.4)
81.9 (0.8)
Benchmark32
snitch-fast32
snitch-tiny32
snitch-int
axpy (34.4%)
conv2d (19.1%)
fann-fp (21.0%)
ReLU (14.1%)
knn (7.6%)
fann-int (0%)
5.9 (1.0)
18.5 (1.0)
80.7 (1.0)
7.1 (1.0)
14.2 (1.0)
90.3 (1.0)
12.9 (2.2)
29.1 (1.6)
136.0 (1.7)
7.7 (1.1)
16.4 (1.2)
89.7 (1.0)
127.1 (21.5)
253.6 (13.7)
853.5 (10.6)
17.2 (2.4)
45.2 (3.2)
81.9 (0.9)
2.5. CONCLUSIONS
25
We summarize the power and energy consumption results obtained
on the real benchmarks in Table 2.3 and Table 2.4.
2.5
Conclusions
In this chapter, we have presented a study on different solutions to
support FP operations: a fully software-based approach, emulating
the FP operation on the integer datapath, a low-cost multi-cycle
FPU, and a high-performance FPU. We have developed Tiny-FPU,
a new area-optimized IEEE 754-compliant RISC-V hardware FPU
that targets area and cost-constrained systems. Tiny-FPU is available
in two versions supporting either double and single-precision or only
single-precision FP. While a full-fledged FPU is desirable whenever
FP operations are often used, not every low-cost MCU core can pay
its area price. Tiny-FPU provides an intermediate design point, occupying 54% (DP) and 37% (SP) less area than a state-of-the-art
FPU optimized for performance and energy efficiency. Further area
reductions can then be achieved by sharing a single register file for
integer and FP data, leveraging the Zdinx and Zfinx RISC-V extensions. Coupling Snitch, a small 32-bit RISC-V core, with Tiny-FPU
allows to save up to 11.7 kB (DP) and 4.4 kB (SP) of code size with
respect to an integer-only version of the same core employing software
emulation functions to compute FP instructions. We showed that
Snitch with Tiny-FPU also outperforms the integer-only core by up to
18.5× (DP) and 15.5× (SP) when executing intensive FP workloads.
While for benchmarks containing less than 5% of FP instructions,
Snitch coupled with Tiny-FPU is up to 5× (DP) and 3× (SP) faster
than the integer-only core employing SW emulation functions while
being only up to 1.29× (DP) and 1.16× (SP) slower than the core
coupled with the high-performance FPU.
Chapter 3
Architectural Solutions
for Domain-Specific
Accelerators: To Buffer,
or Not to Buffer?
In the previous chapter, we have analyzed different microarchitectural solutions to support FP operations. While low-cost implementations are often preferred under stringent area constraints, designs
leveraging greater hardware resources can achieve higher performance
and efficiency. To reach even higher efficiency, even more specialized
components, such as domain-specific hardware accelerators, can be
integrated into an SoC.
In this chapter, we concentrate on different architectural solutions
for domain-specific hardware accelerators. These accelerators often
contain internal memories that end up dominating the area and power
consumption of the engine. We study a special class of hardware
accelerators, programmed by general-purpose processors by writing
into specific memory-mapped registers, and sharing the memory with
the cores. This approach allows for minimizing the internal storage
included in the domain-specific engine, resulting in more area-efficient
27
28
CHAPTER 3. ARCHITECTURAL SOLUTIONS
datapath-dominated units. We focus our study on FFT accelerators
for IoT devices. As one of the fundamental blocks for digital signal
processing (DSP) and a very stable algorithm, the FFT is an ideal
candidate for a dedicated specialized accelerator.
The following sections have been published in a slightly different form in the IEEE International Conference on Applicationspecific Systems, Architectures and Processors (ASAP)1 , except from
Section 3.5, which covers part of the material published in a different
form in the IEEE International Symposium on Circuits and Systems
(ISCAS)2 .
3.1
Introduction
The coming of age of the IoT has increased the demand for small
ultra-low-power edge devices with wireless interfaces. To reduce on-air
time and bandwidth requirements and achieve higher energy efficiencies, more and more processing is performed on the edge, requiring advanced DSP capabilities. Heterogeneous systems augmented
with domain-specific hardware accelerators can enhance IoT SoCs to
achieve higher energy efficiencies and meet tight performance constraints at a lower area cost than purely homogeneous, software-based
ones [8].
In most heterogeneous architectures, hardware accelerators
achieve massive speedup by exploiting dedicated memories for internal
buffering, which, however, often dominates their area and power
consumption [8]. At the same time, many hardware-accelerated
tasks are active only for a fraction of an algorithm’s execution time,
often intermixed with software-based operations. This may lead to
1 ©2021 IEEE. Reprinted, with permission, from L. Bertaccini, L. Benini, and
F. Conti, “To buffer, or not to buffer? a case study on FFT accelerators for ultralow-power multicore clusters,” in 2021 IEEE 32nd International Conference on
Application-specific Systems, Architectures and Processors (ASAP). IEEE, 2021,
pp. 1–8.
2 M. Sinigaglia, L. Bertaccini, L. Valente, A. Garofalo, S. Benatti, L. Benini,
F. Conti, and D. Rossi, “Echoes: a 200 GOPS/W frequency domain SoC with FFT
processor and I2S DSP for flexible data acquisition from microphone arrays,” in
2023 IEEE International Symposium on Circuits and Systems (ISCAS), 2023,
pp. 1–5.
3.1. INTRODUCTION
29
poor utilization of silicon-expensive memory resources, as it requires
marshaling and data transfers every time the execution flips from
hardware-accelerated to software-based.
In this chapter, we investigate a possible avenue for optimization: replacing internal buffers in domain-specific accelerators with
resources already allocated in the system and shared with other components, such as software cores, to improve area and energy efficiency.
We concentrate our study on hardware-accelerated multicore clusters
for ultra-low-power extreme edge nodes, and in particular on FFTs –
a critical and frequently used block in most modulation schemes for
communication, DSP, and audio processing frontend [75]. FFTs are
often supported by hardware accelerator engines, which, depending on
the application scenarios, support different data types. For example,
several sensor applications require high-precision FFTs, often using
16-bit to 32-bit number formats. FFTs are also often employed in preprocessing steps for machine learning algorithms, such as quantized
neural networks working on 8-bit data [76].
FFT acceleration is usually implemented either with accelerators
decoupled from the cores, requiring data copies, or supported by
instruction extensions within the pipeline of processor cores, which do
not require data copies but have a lower potential for acceleration due
to the limited bandwidth of the cores’ load/store interface. Clustercoupled hardware processing engines (HWPEs) have been recently
proposed to get the best of both worlds [77, 78]. In this scenario,
all cores – both general-purpose and HWPEs – cooperate by directly
sharing data through a low-latency multi-banked L1 data memory
called tightly-coupled data memory (TCDM). Contrary to traditional
accelerators, HWPEs can do without an internal buffer and rely entirely on the TCDM. With this design choice, the area efficiency of
the HWPE can be significantly boosted, but questions may arise on
a potential bottleneck at the accelerator interface with the TCDM.
This is a major concern for the specific case of FFT, which is well
known to have complex and critical memory access patterns.
To study this specific point, we focus on a practical FFT use case.
In FFT accelerators, the presence or absence of an internal buffer is
particularly relevant because partial results have to be iteratively shuffled. This fact is well-known and has been extensively studied in the
past. To exploit data locality, various FFT accelerators [79–81] have
30
CHAPTER 3. ARCHITECTURAL SOLUTIONS
been designed with dedicated internal memories. Baas [9] proposed
an intermediate solution, including a smaller cache in the accelerator,
while other architectures, such as Texas Instruments tightly-coupled
accelerator [11], share the memory with cores.
Since the FFT algorithm introduces bit-reversed reordering in the
last stage of a decimation-in-time (DIT) FFT, which inevitably leads
to the presence of banking conflicts in accessing the multi-banked
TCDM, most FFT accelerators have to be designed around these issues. To achieve a conflict-free FFT algorithm, Cohen [82] and Ma [83]
proposed different reordering schemes of the butterfly sequence, which
allow the two butterfly inputs to be always in different memory banks.
However, they target single butterfly engines and do not consider
sub-word FFTs. In this chapter, we present:
1. Two novel designs for cluster-coupled FFT HWPEs: a buffer-less
architecture and a buffered one implemented in a 22 nm technology
at 0.65 V. The former shares a 16-bank TCDM with 8 cores in
the cluster, reducing the hardware footprint. The latter stores the
FFT samples and intermediate results internally, reducing memory
traffic. Both HWPEs can work at different fixed-point precisions
with 8/16/32-bit real and complex parts.
2. A novel reordering scheme for the buffer-less design that boosts
the HWPE performance allowing for conflict-free accesses to the
shared TCDM even when computing sub-word FFTs.
Our experimental results show that the buffer-less accelerator offers
a favorable area, performance, and energy design with respect to the
buffered one, and it outperforms state-of-the-art accelerators [11, 79–
81, 84].
3.2
Related Work
As one of the most widespread algorithms in DSP applications, FFTs
are often computed on dedicated hardware blocks. FFT accelerators
have been extensively investigated both by academia and industry,
exploring numerous fully parallel and iterative implementations. Fully
parallel architectures contain all the butterfly units necessary for an
3.2. RELATED WORK
31
N -point FFT, while iterative accelerators work on smaller datapaths.
The formers provide high performance, like the fully parallel FFT
architecture by Garrido et al. [85], but result in a large area for
high values of N . For this reason, iterative architectures are usually
adopted for embedded applications.
Iterative accelerators are typically memory-based, employing internal buffers to store and shuffle partial results, which dominate the
area and power of such architectures. Chen et al. [80] include two
processing engines in their design, each containing two butterfly units
and 16 KiB of dedicated memory. Wang et al. [79] introduced a reconfigurable 2n 3m 5k FFT accelerator implemented in 16 nm technology
and working on 24-bit precision. The accelerator occupies 0.37 mm2
and has been coupled with a RISC-V core. Their results show how the
dedicated internal memory dominates both the area and the power
consumption of the FFT accelerator. The NXP MAPLE-B [86], a
multi-radix decoupled accelerator contained in the MSC8154 processor, includes an I/O data buffer, and the FFT module in the Infineon
advanced driver assistance subsystems [87], part of the AURIX™ family - TC23xLA, also contains a RAM buffer for storing the calculation
coefficients.
A different approach is followed by Texas Instruments
HWAFFT [11], an FFT accelerator tightly coupled to a DSP core. The
accelerator is located outside the core but shares the core’s memory
bandwidth. Such a design choice avoids the need for an internal buffer
but provides the accelerator with a narrower memory bandwidth,
preventing high degrees of parallelization. TI HWAFFT contains only
a single radix-2 butterfly unit working on 16-bit precision.
The recently proposed cluster-coupled HWPEs [77] represent a
further alternative. HWPEs share the data with the cores through
a low-latency multi-banked data memory but have their dedicated
memory ports. Nevertheless, the absence of an internal buffer
and FFT’s peculiar memory access patterns require a reordering
scheme to avoid systematic conflicts when accessing the TCDM. Many
reordering schemes have been proposed in the past two decades.
Some well-known schemes, such as Cohen [82], Johnson [88], and
Ma [83], only targeted architectures including single butterfly units.
Baek and Choi [89] introduced an address-generation scheme for FFT
architectures containing multiple radix-2 butterflies. Nonetheless,
32
CHAPTER 3. ARCHITECTURAL SOLUTIONS
their proposal, as well as Sorokin and Takala’s [90] and Xia et al.’s [84]
methods, requires the input samples not to be initially stored inside
the memory in natural order. This is not ideal since reordering the
data before the computation can be costly. However, this overhead
can be mitigated when employing an internal buffer since buffered
architectures require explicit copy-in and copy-out phases that can be
exploited to shuffle the samples.
Due to the large variety of applications requiring FFTs, different
levels of precision and range have been targeted. Bo et al. [91] proposed an application-specific instruction-set processor using 8-bit or
16-bit precision, Wang et al. [79] employed 24-bit fixed-point data,
the FFT accelerator contained in the Infineon FFT accelerator [87]
works on 16-bit or 32-bit values, while Chen et al. [80] used floatingpoint data. However, to the best of our knowledge, an accelerator
supporting low, mid, and high precision is still missing.
In this chapter, we present two cluster-coupled FFT accelerators,
one including a large internal buffer and one sharing the memory
with the cores. The buffer-less accelerator employs a newly proposed
reordering scheme to efficiently make use of the external memory.
Both accelerators can work with 8/16/32-bit precision.
3.3
Architecture
The architecture template we build upon is based on the opensource multicore PULP cluster [92], augmented with a cluster-coupled
HWPE3 .
3.3.1
PULP Cluster
The PULP cluster, shown in Fig. 3.1, contains eight RISC-V cores
enhanced with DSP ISA extensions [40], sharing a 4 KiB instruction
cache and 64 KiB of TCDM organized into 16 banks, implementing
word-level interleaving to reduce the probability of contention. The
cluster also includes a direct memory access (DMA), a peripheral interconnect, an Advanced eXtensible Interface (AXI) bus, and provides
3 https://github.com/pulp-platform/hwpe-doc
3.3. ARCHITECTURE
33
Figure 3.1: PULP cluster with FFT HWPE.
support for a software-controlled tightly-coupled hardware accelerator. The cluster cores, the DMA, and the HWPE are connected to
the memory banks through a high-bandwidth logarithmic interconnect
and can access the TCDM with a 1-cycle latency. The HWPE and
the cluster cores communicate through the peripheral interconnect for
control plane interactions.
An open-source example of HWPE4 provides a template to build
hardware accelerators conceived to be integrated into a PULP cluster.
The fixed environment around the accelerator allows various internal
modules to be reused for different HWPEs, speeding up the design
of new accelerators. Such units manage the communication with the
TCDM and the cluster cores and only require to be tuned to each
particular implementation (e.g., depending on the number of memory
ports dedicated to the accelerator). The rest of the accelerator is
intrinsically application-dependent, requiring ad-hoc designs.
4 https://github.com/pulp-platform/hwpe-mac-engine
34
3.3.2
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Buffer-less FFT HWPE Architecture
The proposed FFT accelerator is designed as a pipelined architecture and supports different bitwidths.
It can compute up
to 512/1024/2048-point FFT, respectively for complex data with
8/16/32-bit real and imaginary parts. For conciseness, we define these
formats as C16/C32/C64. We show the structure of the buffer-less
HWPEs in Fig. 3.2. The accelerator implements the radix-2 CooleyTukey FFT algorithm [93] in its DIT flavor and consists of several
submodules.
Butterfly Unit
The butterfly unit is the computational module of the accelerator.
It is fully combinational and contains one C64, one C32, and two
C16 butterfly engines. The HWPE reuses higher-precision engines to
compute lower-precision butterflies. Therefore, the accelerator can
compute either one C64, two C32, or four C16 butterflies each cycle.
Each butterfly engine employs fixed-point arithmetic in the format
Q0.x, requiring the inputs to be smaller than 0.5 and dividing the
outputs by 2 (except in the last FFT stage) to prevent overflow, as
presented, for example, in [94].
Twiddle Factor LUTs
The twiddle factors corresponding to the maximum number of FFT
points are stored in lookup tables (LUTs). An N -point FFT requires
N/2 different twiddle factors. However, when N is a power-of-two,
only (N/8 + 1) twiddle factors are necessary, as the remaining can be
obtained by exploiting trigonometric identities. For this reason, we
only store M/8 twiddle factors in the LUT, where M is the maximum
number of supported FFT points. Since we need up to four twiddle
factors each cycle and the accelerator supports low, mid, and high
precision, we include one C64 LUT containing 65 elements, one C32
LUT including 129 values, and two C16 LUTs with 257 twiddle factors.
Higher-precision LUTs are still used when computing lower-precision
FFTs, rounding their values to reduce the precision.
Figure 3.2: Block diagram of the buffer-less FFT HWPE architecture.
3.3. ARCHITECTURE
35
36
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Streamer
The Streamer is a specialized DMA unit. This module manages
the communication from/to memory. The buffer-less FFT HWPE
contains eight 32-bit memory ports: four used as output ports and
four as input ports. The communication with the TCDM is based on
ready/valid handshakes.
Controller
The Controller is the unit that controls the whole computation. It
manages and synchronizes all the modules in the accelerator. The
Controller contains an FSM that organizes the FFT computation, the
logic to select each cycle the right twiddle factors from the LUT, and
a set of registers where the cores can write to communicate with the
accelerator. In particular, the cores communicate to the HWPE when
to start the FFT computation, the number of FFT points, the base
address of the vector to be transformed, its bitwidth, and whether
or not to implement the final bit-reversed reordering. If the results
are reordered, they are stored in the memory locations following the
input vector, while non-reordered results overwrite the input samples.
Non-reordered FFTs may be employed to speed up FFTs that require
a higher number of points than the supported one. For example,
all the stages of a C32 2048-point FFT, except the last one, can be
computed using two non-reordered C32 1024-point FFT [11].
Butterfly Registers
Two sets of four 64-bit registers are located around the butterfly and
used to shuffle inputs and outputs.
Gather-Scatter
The Scatter unit reorganizes the data coming from the memory in the
set of registers in front of the butterfly unit, while the Gather unit
reads processed data from the butterfly registers, rearranges them,
and forwards them to the Streamer.
3.3. ARCHITECTURE
37
Figure 3.3: Buffer-less FFT HWPE pipeline. The example shows
the beginning of a 64-point C64 FFT first stage. One C64 sample
occupies one 64-bit register. In the case of lower-precision FFTs,
multiple samples are packed inside one 64-bit register.
3.3.3
Buffer-less FFT HWPE Reordering Scheme
To avoid the need for an internal buffer and to shuffle partial results,
we propose a new scheme to reorder the butterfly sequence and the
memory access patterns. Such a scheme relies on having access to
N ≥ 16 memory banks and always loads/stores data at consecutive
memory locations. This prevents data races and conflicts between
loads and stores and allows for accessing two of the required C16 samples with one 32-bit memory port. Furthermore, it can be guaranteed
that the load and store operations are not simultaneously trying to
access the same memory banks by tuning the number of banks and the
accelerator’s pipeline latency. Due to its pipeline depth, the HWPE
takes 6 cycles between a load and a store request addressing the
same memory locations. With 16 TCDM banks available, conflicts
never happen. An example is provided in Fig. 3.3, where the first
simultaneous requests are shown. The load requests the samples of
indices (6, 7), while the store request addresses the samples of indices
(0, 1). Since a C64 example is presented, each sample occupies two
consecutive memory locations; thus, the load requires access to banks
(12, 13, 14, 15), while the store to banks (0, 1, 2, 3).
38
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Since the distance between the left wing of a butterfly and its
right wing is fixed within an FFT stage, the buffer-less FFT HWPE
loads first a set of left wings at consecutive memory addresses, and
then, during the following cycle, the right wings of the same butterflies. After loading new samples, the Scatter unit reorganizes the
data, storing them in two of the four 64-bit registers in front of the
butterfly unit. Once the four registers are full, both the left and right
wings of a certain number of butterflies are stored in the registers,
and the butterfly unit can start fetching and processing data. The
Scatter, in parallel, decides each cycle in which butterfly registers
to store the new inputs, overwriting the samples just processed by
the butterfly engines. An example is provided in Fig. 3.3, where the
samples of indices (2, 3) overwrite (0, 32) after the first butterfly has
been computed. After the butterfly unit, the situation is similar;
partial results are stored in two of the four other 64-bit registers, and
the Gather rearranges the butterfly outputs and passes them to the
Streamer.
In a DIT radix-2 FFT algorithm, the outputs are provided in bitreversed order. Reordering data from natural to bit-reversed orders
requires additional techniques to avoid systematically producing bank
conflicts. The new reordering scheme allows us to write two bitreversed samples every cycle. During the last FFT stage, the inputs
of a butterfly are at consecutive indices. Therefore, the HWPE loads
the samples to compute butterflies whose bit-reversed outputs are
consecutive two-by-two. For example, for a 64-point C32 FFT, it can
load first (0, 1, 2, 3), and then (32, 33, 34, 35). It can then compute the
butterflies (0, 1) and (32, 33), whose bit-reversed outputs are (0, 32)
and (1, 33). However, the accelerator cannot store all the results in
one cycle since it would generate conflicts. It, therefore, exploits only
half of the bandwidth, storing first (0, 1) and then (32, 33), stalling
the pipeline for one cycle to avoid overwriting meaningful data inside
the accelerator registers. For C64 FFTs, since the HWPE stores two
samples per cycle, it can exploit the full output bandwidth, while
for C16 FFTs, only one-fourth of the bandwidth can be used. Bank
conflicts can still arise between the load and store requests, but not
systematically. Other conflicts can happen if the HWPE and the cores
try to access the same banks simultaneously. All the residual conflicts
3.3. ARCHITECTURE
39
are handled by the accelerator, delaying the load request by one cycle
and stalling its pipeline.
3.3.4
Buffered FFT HWPE
The buffered accelerator, in Fig. 3.4, is structured similarly to
the buffer-less HWPE, implementing only a couple of architectural
modifications. The two sets of registers around the butterfly unit
are replaced by a large internal memory. However, a smaller set of
registers is still kept around the butterfly unit to break the system’s
critical path. The internal buffer consists of 4 KiB of standard cell
memory [95], divided into four banks containing C16 words, each
providing two input and two output ports. Larger-precision values
can be stored in the buffer occupying multiple C16 locations. The
buffer is organized in multi-ported banks to avoid conflicts when
reordering and shuffling data. The internal buffer size limits the
maximum number of FFT points supported by the buffered HWPE.
Furthermore, since data are never loaded and stored simultaneously,
only four 32-bit memory ports are needed and used both as input
and output ports. Consequently, the Streamer is considerably simpler
than in the buffer-less case, while some additional logic is necessary
to generate the buffer addresses where to load/store the samples.
The buffered FFT HWPE first loads the samples from the TCDM
to the internal buffer. Then, the computational phase starts, and, each
cycle, data are read from the buffer, processed through the butterfly
unit, and written back to the buffer. Finally, data are read from
the buffer and stored into the TCDM. During the initial loading and
the final storing of the buffer, no computation is performed. In the
buffered accelerator, we implement the bit-reversed reordering when
loading the inputs. The buffer structure allows for writing two bitreversed samples every cycle, similar to the buffer-less case. However,
thanks to the two input ports provided by each buffer bank, reordering
the butterfly sequence is not necessary for such implementation.
CHAPTER 3. ARCHITECTURAL SOLUTIONS
40
Figure 3.4: Block diagram of the buffered FFT HWPE architecture.
3.4. EXPERIMENTAL RESULTS
3.4
41
Experimental Results
To evaluate the different versions of our FFT HWPEs, we integrate
them into a PULP cluster [92] (see Fig. 3.1).
3.4.1
Area
Figure 3.5: Area comparison of the buffered and buffer-less FFT
HWPEs synthesized at 200 MHz.
We synthesize the PULP cluster first with the buffered FFT HWPE
and then with the buffer-less accelerator, using Synopsys Design Compiler in Globalfoundries 22FDX, with topographical synthesis.
We target 200 MHz of clock frequency under worst-case conditions
(SSG, 0.59 V, −40 ◦ C). The buffer-less HWPE occupies 74% less
area than the buffered design and around 2× the area of one of the
RISC-V cores. We show the area comparison in Fig. 3.5 and the
area breakdown of the buffer-less accelerator in Fig. 3.6. The area
is dominated by the butterfly unit, where the C64 butterfly engine
occupies around 70% of the datapath. Furthermore, for the buffered
accelerator, modifying the maximum number of supported FFT points
requires changing both the internal buffer size and the twiddle factor
LUTs, while for the buffer-less HWPE, only the LUTs have to be
adapted. Since the current twiddle factor LUTs occupy only 5% of the
buffer-less accelerator area, the maximum number of supported FFT
points may be increased involving a low area overhead, highlighting
42
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Figure 3.6: Area breakdown of the buffer-less FFT HWPE synthesized
at 200 MHz.
another advantage of implementing buffer-less architectures. In the
buffered case, the internal buffer already counted for more than 75%
of the HWPE area. Increasing the number of FFT points for such an
architecture would then be highly costly.
3.4.2
Butterfly Unit Utilization
The butterfly utilization of the buffer-less HWPE for FFTs, including
and excluding bit-reversed reordering, is shown in Fig. 3.7. Since
the output bandwidth is not fully exploited when reordering C16/C32
results, C16/C32 FFTs without bit-reversed reordering present higher
butterfly utilization.
When doubling the data precision, the number of available butterfly engines halves, thus doubling the number of butterfly cycles.
Therefore, the HWPE reaches the same utilization for an N -point
p-bit FFT and an N/2-point 2p-bit FFT without the bit-reversed
reordering. This is not the case for FFTs including the bit-reversed
reordering, where the final reordering prevents from using the whole
output bandwidth for C16/C32 data, resulting in a lower butterfly utilization. Despite reaching a lower percentage of butterfly utilization,
3.4. EXPERIMENTAL RESULTS
43
Figure 3.7: Butterfly utilization of the buffer-less FFT HWPE for (a)
FFTs excluding the bit-reversed reordering and (b) FFTs including
the bit-reversed reordering. C64 1024/2048-point and C32 2048-point
FFTs are not shown since they are not supported by the proposed
architecture.
44
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Table 3.1: Performance of the buffer-less and buffered HWPEs for all
the supported bitwidths and number of FFT points*.
N
[points]
64
128
256
512
1024
2048
Buffer-less FFT HWPE
Buffered FFT HWPE
C16 FFT C32 FFT C64 FFT
[cycles]
[cycles]
[cycles]
C16 FFT C32 FFT C64 FFT
[cycles]
[cycles]
[cycles]
118
211
415
845
1791
3875
158
305
661
1399
3005
-
249
520
1116
2434
-
115
221
447
929
1955
4133
171
349
735
1569
3363
-
282
605
1311
2549
-
*All the reported values refer to FFTs including bit-reversed reordering.
thanks to the larger number of butterflies computed per cycle, the
HWPE provides higher performance when less precision is required.
For a reordered 512-point C16/C32/C64 FFT, the accelerator takes
835/1399/2434 cycles.
In both the bit-reversed reordered and non-reordered cases, the
utilization increases with the number of FFT points. For high numbers of FFT points, the control overhead and the time spent to fill
and empty the pipeline decreases more and more with respect to the
butterfly cycles. For C16/C32/C64 2048/1024/512 FFTs, the HWPE
reaches over 97% of butterfly cycles when excluding the bit-reversed
reordering and 73%/85%/95% when including the bit-reversed reordering. The 2% utilization drop in the case of a C64 512-point
FFT is due to stalls introduced during the last stage to prevent
conflicts between natural-order loads and bit-reversed stores. The
buffered HWPE reaches a slightly lower utilization due to the time
spent copying in/out the data. For a C16/C32/C64 2048/1024/512
FFT, including bit-reversed reordering, it achieves 68%/76%/90% of
butterfly cycles.
3.4.3
Performance
We report the performance of our FFT accelerators in Table 3.1.
The buffer-less version provides slightly better performance than the
buffered HWPE since it always operates in a compute phase, not
3.4. EXPERIMENTAL RESULTS
45
Figure 3.8: Execution time of the optimized FFT SW kernel running
on 8 cores vs. the buffered and buffer-less FFT HWPEs.
requiring the initial loading and final storing of the buffer when no
butterflies are computed. We compared the performance of our HWPEs with a highly optimized fixed-point FFT software, running on
the 8 cluster cores and exploiting their DSP custom instructions. The
results are shown in Fig. 3.8. Our solutions are around 7× (buffered)
and 8× (buffer-less) faster than the optimized software while adding
22% of area to the PULP cluster in the buffered case and only 6% in
the buffer-less case. The buffer-less architecture is then 4.25× more
area-efficient than the buffered one.
3.4.4
Power
We place and route the PULP cluster including the bufferless/buffered FFT HWPE using Cadence Innovus. We simulate FFT
computations with Mentor Questasim. Then, we extract the related
average power consumption with Synopsys PrimeTime under typical
conditions (TT, 0.65 V, 25 ◦ C), at 350 MHz, the maximum frequency
the designs could reach in this corner. Since the critical block for
timing is the butterfly unit, the buffer-less and buffered accelerator
CHAPTER 3. ARCHITECTURAL SOLUTIONS
46
Technology Voltage Frequency
Area
(1024-point)
Performance
(1024-point)
Average Power
(1024-point)
Energy Efficiency
Table 3.2: Comparison of FFT Hardware Accelerators, stand-alone (top six rows), and in a full-system
context (bottom three rows).
[butterfly/cycle]
Peak Performance
5.5 mW (12.2 mW)a
Design
12.6 mW (13.2 mW)a 121 nJ/FFT (127 nJ/FFT)a
47 nJ/FFT (105 nJ/FFT)a
3363 cycles
126 nJ/FFT
135 nJ/FFTb
3005 cycles
91.3 mW
22.6 mWc
483 nJ/FFT
208 nJ/FFTb
0.093 mm2
26.3 mWb
0.024 mm2
1380 cycles
1905 cyclesb
172.3 mW
350 MHz
2.4 mm2
0.37 mm2
972 cyclesb
350 MHz
1 GHz
320 MHzc
1403 cycles
187 nJ/FFT
159 nJ/FFT
0.65 V
0.9 V
0.57 V
4.6 mm2
1.063 mm2
18.5 mW
0.65 V
16 nm
500 MHz
122.88 MHz
3005 cycles
38.5 mW
19.5 mW
22 nm
45 nm
1V
1.08 V
0.44 mm2
7315 cycles
3363 cycles
22 nm
55 nm
0.51 mm2
4 C16 / 2 C32 / 1 C64
4 FP C64
C48 mixed-radix
65 nm
350 MHz
not available
4 C16 / 2 C32 / 1 C64
Chen et al. [80]
Wang et al. [79]
4 FP C64
C32 mixed-radix
0.65 V
100 MHz
350 MHz
Buffered FFT HWPE
Guo et al. [81]d
Xia et al. [84]
22 nm
1.3 V
0.65 V
Buffer-less FFT HWPE
Cluster w/ Buffer-less HWPE 4 C16 / 2 C32 / 1 C64
22 nm
Results for 972-point FFTs (the authors do not report results for 1024-point FFTs).
The results for Guo et al.’s accelerator [81] are taken from Chen et al.’s comparison table [80].
2820.6 nJ/FFT
not available
b
Values between parenthesis take into account both the HWPE and TCDM contributions.
1 C32
d
Wang et al. report a range of values. The highest result was considered for frequency and power.
a
4 C16 / 2 C32 / 1 C64
TI HWAFFT [11]
Cluster w/ Buffered HWPE
c
3.4. EXPERIMENTAL RESULTS
47
achieve the same maximum frequency. The buffer-less accelerator
consumes on average 4.1/5.5/6.8 mW at 350 MHz, respectively for
C16/C32/C64 2048/1024/512 FFT at full throughput, while the
buffered consumes around 9% more power than the buffer-less one,
even including the TCDM read/write power contribution. This
is because the multi-ported structure of the buffer banks cancels
potential energy improvements related to the use of an internal
buffer. For lower-precision FFTs, the buffer-less HWPE consumes
less power mainly due to the final reordering, which allows using
one-fourth, half, or the whole output bandwidth for C16/C32/C64
data. For higher-precision FFTs, the accelerator reaches a higher
butterfly utilization, increasing its average power consumption.
3.4.5
Bandwidth Considerations
The buffered and buffer-less solutions compared in this chapter leverage different bandwidths towards the memory. In the buffered solution, the bandwidth is halved as the samples are never concurrently
loaded and stored from/to memory, having then four 32-bit memory
ports instead of the eight used by the buffer-less implementation. The
buffered accelerator first loads all the samples into its internal buffer,
then it computes the FFT, and only at the end of the calculation
it stores the results into the scratchpad memory. The buffer-less
implementation instead continuously streams the data in and out, performing concurrent load and store transactions from/to the scratchpad
memory. While this could raise concerns in terms of potential memory
conflicts arising from the higher number of concurrent transactions,
the proposed reordering scheme prevents systematic banking conflicts.
As a result, banking conflicts in the buffer-less execution can appear
only sporadically in the reordering stage or caused by conflicting
requests issued by the RISC-V cores. Furthermore, all the samples are
loaded, processed, and stored back into the memory by the buffer-less
HWPE for every FFT stage. Therefore, for an 2n -point FFT, n times
more scratchpad memory transactions are performed by the buffer-less
accelerator with respect to the buffered one. If the RISC-V cores are
computing while an FFT is being calculated by the HWPE, there is a
higher probability of contention when using the buffer-less accelerator
due to the higher number of concurrent memory requests.
48
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Furthermore, intermediate solutions could be explored, such as
FFT accelerators with smaller internal buffers compared to the buffered
architecture but leveraging eight 32-bit memory ports as in the bufferless accelerator. However, since the scratchpad memory inside the
PULP cluster provides single-cycle access and the proposed reordering
scheme prevents systematic banking conflict, the buffer-less implementation stands out as a favorable solution as it minimizes the area
needed for internal storage in the accelerator without degrading the
performance.
3.4.6
SoA Comparison
We compare our solutions with state-of-the-art accelerators in Table 3.2. Our performance, power, and energy results refer to a C32
application. First, we compared our HWPEs with four stand-alone
FFT accelerators with internal buffers proposed by Wang et al. [79],
Chen et al. [80], Xia et al. [84], and Guo et al. [81]. Chen et al.
and Guo et al. include four C64 floating-point butterfly engines.
Chen et al. reach an energy efficiency comparable to our buffered
solution, while our buffer-less accelerator is 20% more energy-efficient
than their implementation and 4.6× more efficient than Guo et al.’s
implementation. Wang et al. and Xia et al. achieve higher performance than our HWPEs thanks to their mixed-radix butterfly engines.
Since the authors did not report the results for 1024-point FFTs, we
considered their results for 972-point FFTs. Wang et al. target C48
data, while Xia et al. use C32 values. Our buffer-less HWPE is 2×
and 28.5% more energy-efficient than Xia et al. and Wang et al.,
respectively. Note that the energy efficiency gap would widen when
considering the cost of the data copy-in and copy-out phases needed
for the accelerators including internal buffers. The buffer-less HWPE
provides the smallest-area solution, although part of the extra area
needed for the considered state-of-the-art accelerators is dedicated to
the larger floating-point or mixed-radix butterfly engines.
Finally, we compare the results obtained for the PULP clusters
enhanced with the HWPEs with TI HWAFFT [11] since it reports
energy efficiency numbers related to a full system. TI HWAFFT can
only compute C32 FFT and contains one butterfly unit. While our
HWPE also supports higher and lower precision, addressing a wider
3.5. PHYSICAL IMPLEMENTATIONS
49
range of applications. The PULP cluster enhanced with our buffer-less
HWPE is 2.4× faster for a 1024-point C32 FFT than TI HWAFFT and
around 18× more energy-efficient. Since Texas Instruments reports
the accelerator performance for FFT computations without the bitreversed reordering, we were able to identify in the final reordering
the responsible for our 2.4× speed-up when computing a complete
FFT using double the number of C32 butterfly engines. Without
bit-reversed reordering, our HWPE is 2× faster. A 1024-point C32
FFT without reordering takes 5244 cycles on TI HWAFFT, while
2626 cycles on our buffer-less HWPE.
3.5
Physical Implementations
To further evaluate and validate the proposed buffer-less FFT HWPE,
we integrate it into an open-source single-core MCU5 [96] optimized
for fixed-point computations. The resulting system, called Echoes,
was fabricated using a 65 nm technology. A second chip, called Maestro, was then designed. Maestro includes a vector processor and a
floating-point version of the FFT HWPE.
3.5.1
Echoes
We show the block diagram of the Echoes SoC in Fig. 3.9. The
baseline single-core MCU has been extended in Echoes with our bufferless FFT HWPE and a highly configurable I2 S peripheral able to
communicate with up to 16 digital I/O audio devices. The Echoes SoC
has, in this way, been specialized for audio application and frequencydomain processing. In particular, the Echoes SoC contains:
• A CV32E40P RISC-V core [40] enhanced with DSP ISA extensions;
• The proposed buffer-less FFT HWPE;
• An autonomous I/O subsystem including a wide set of peripherals and an I/O DMA engine;
5 https://github.com/pulp-platform/pulpissimo
50
CHAPTER 3. ARCHITECTURAL SOLUTIONS
256KB L2 INTERLEAVED MULTIBANK
PRIVATE 64KB
ROM
16KB
32KB 32KB
8KB
16KB
BANK0
BANK15
LOW LATENCY INTERCONNECT
RX
FLL
FLL
HYPER BUS
TX
I2S
I2C
uDMA
ENGINE
UART
CFG
QSPI
TIMER
APB
SPI
SoC CTRL
FFT
HWPE
I
D
CV32E40P
FPU
DEBUG
EVENT UNIT
INTC
Figure 3.9: Block diagram of the Echoes SoC.
• A 256 KiB of memory organized into 16 banks and shared among
the core, the accelerator, and the peripherals.
The Echoes SoC has been taped out using a 65 nm technology. We
show the Echoes chip in Fig. 3.10, whose die area is 4 mm2 . We report
the Echoes chip specifications in Table 3.3. The chip has been tested
using an Advantest SoC V93000 automatic test equipment.
In Fig. 3.11, we report the maximum operating frequency (in blue)
sweeping the supply voltage from 0.9 V to 1.2 V and the power consumption for two DSP kernels representative of audio processing workloads (in red). The maximum operating frequency in the considered
voltage range spans from 150 MHz to 350 MHz. The power consumption is measured at the highest operating frequency for each voltage.
The power results reported for FFT kernels consider FFT executions
on the buffer-less FFT HWPE. The Echoes chip consumes 133.5 mW
51
3.5. PHYSICAL IMPLEMENTATIONS
Table 3.3: Echoes chip specifications.
Technology
Chip Area
2
SRAM
65 nm
4 mm
256 KiB
Vdd Range
Frequency Range
Power Envelope
0.9–1.2V
150–350MHz
133.5 mW
Figure 3.10: Echoes chip micrograph.
52
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Figure 3.11: Maximum frequency and power consumption measurements on the Echoes chip.
when computing a 512-point C64 FFT on our accelerator at 1.2 V,
350 MHz, while it consumes 121 mW for a 1024-point C32 FFT at the
same operating point.
In Fig. 3.12, we show the performance and energy efficiency of
Echoes when computing FFT kernels, considering both full software
executions on the RISC-V core and computations on our hardware
accelerator. We report the peak performance and efficiency when
leveraging the FFT HWPE, as well as the relative improvements
with respect to software executions, in Table 3.4. By exploiting the
hardware accelerator, the performance can be improved by up to
30.1× for C64 FFTs and 41.1× for C32 FFTs, while the efficiency can
be improved by up to 6.9× for C64 FFTs and 11.2× for C32 FFTs.
Thanks to the multiple data sizes supported by the buffer-less
FFT HWPE, the precision of a specific kernel can be tuned to the
application requirements. Moving from a C64 kernel to a C32 version
improves the performance and energy efficiency by around 2×.
3.5. PHYSICAL IMPLEMENTATIONS
53
Table 3.4: Peak performance and efficiency leveraging the FFT
HWPE. In parenthesis, we report the improvement with respect to a
software FFT kernel.
Kernel
Peak Efficiency
Peak Performance
1024-point C32 FFT
512-point C64 FFT
89.5 GOPS/W (11.2×)
41.8 GOPS/W (6.9×)
5.8 GOPS (41.1×)
3.2 GOPS (30.1×)
Figure 3.12: Performance and energy efficiency of FFT kernels on
Echoes.
54
CHAPTER 3. ARCHITECTURAL SOLUTIONS
3.5.2
Maestro
Another physical implementation of the FFT HWPE is included in
Maestro. In this case, the FFT HWPE has been modified to support FP arithmetic instead of fixed-point data types. We show the
Maestro chip in Fig. 3.13. Maestro has been implemented in a 65 nm
technology, and it features:
• A single-core MCU [96] to communicate with the outside world;
• An accelerator with a similar structure to the PULP cluster already described in this chapter but employing a vector processor
with a SIMD 64-bit FPU [97];
• A matrix-multiplication accelerator [56] connected directly to
the vector register file;
• A FP version of our buffer-less FFT HWPE, supporting FP32
and FP16.
The FP version of the FFT HWPE includes a different LUT and
datapath than the one described in this chapter. The LUT contains,
in this case, FP twiddle factors, while the fixed-point butterfly units
were replaced with FP counterparts.
Floating-Point Butterfly Unit
The FP butterfly units are composed of two parallel dual-output FP
dot-product units, similar to the design proposed by Swartzlander and
Saleh [98]. This dot-product unit has been obtained by extending the
open-source module that will be presented in Chapter 4. In particular,
a butterfly operation requires executing:
(
y[0] = x[0] + w × x[1]
y[1] = x[0] − w × x[1]
(3.1)
where w is the twiddle factor, x is the input, and y is the output of
the butterfly. This means computing:
3.5. PHYSICAL IMPLEMENTATIONS
55
Figure 3.13: Maestro chip.

y[0].re = x[0].re + (w.re × x[1].re − w.im ∗ x[1].im)



y[0].im = x[0].im + (w.re × x[1].im + w.im ∗ x[1].re)

y[1].re = x[0].re − (w.re × x[1].re − w.im ∗ x[1].im)



y[1].im = x[0].im − (w.re × x[1].im + w.im ∗ x[1].re)
(3.2)
The dual-output dot-product unit computes two values:
(
e+a×c+b×d
(out0 , out1 ) = dDotp(a, b, c, d, e) =
e−a×c+b×d
(3.3)
Therefore, two of these modules in parallel can compute a full butterfly
operation:
56
CHAPTER 3. ARCHITECTURAL SOLUTIONS
Figure 3.14: Area comparison: C64 and C32 floating-point vs. fixedpoint butterfly units.
(
(y[0].re, y[1].re) = dDotp(w.re, −w.im, x[1].re, x[1].im, x[0].re)
(y[0].im, y[1].im) = dDotp(w.re, w.im, x[1].im, x[1].re, x[0].im)
(3.4)
FP formats offer various benefits over fixed-point notations. FP
data types provide a dynamic range and can, therefore, represent
both very small and very large values. FP formats are conceived to
have a finer granularity for lower values and coarser granularity for
larger values, with half of the representable numbers included in [-1:1].
Fixed-point data types instead have a fixed accuracy and concentrate
on a narrower range that they represent with evenly distributed values.
The benefits ensured by FP formats, however, come at an area price.
3.6. CONCLUSION
57
We evaluate the cost for an FP butterfly over a fixed-point counterpart. To meet the same timing constraint as the fixed-point accelerator, additional pipeline stages have to be integrated into the
butterfly unit. In terms of area, the FP version of the C64 and C32
butterfly are, respectively, around 1.9× and 2.4× larger than their
fixed-point counterparts. Note that, as we will show in more detail in
Chapter 4, lower-precision FP units suffer more from the additional
control overhead introduced by FP arithmetic. We compare the area
of fixed and floating-point butterfly units in Fig. 3.14.
Maestro has been placed and routed using a 65 nm technology. Its
main features have been validated using an Advantest SoC V93000
automatic test equipment.
3.6
Conclusion
In this chapter, we have compared buffered and buffer-less accelerators, concentrating on FFT HWPEs. For the buffer-less design,
we have introduced a new reordering scheme that allows conflictfree memory accesses also for sub-word FFTs. Our accelerators can
work with complex data with 8/16/32-bit real and imaginary parts,
addressing a wide range of applications. While being around 8×
faster than an optimized FFT software implementation running on
the 8 cluster cores, the buffer-less implementation occupies 74% less
area than the buffered HWPE and adds only 6% of area overhead
to a PULP cluster. By sharing the memory with the cluster cores,
the buffer-less accelerator has its area dominated by the datapath,
achieving then a higher area efficiency. Furthermore, since the internal
buffer size defines the maximum number of FFT points, increasing
the limit on the FFT length would be highly costly for the buffered
implementation. This can instead be achieved at a low area overhead
for the buffer-less accelerator, as in this case, it only requires expanding the twiddle factors LUTs. The buffer-less HWPE consumes
around 4.1/5.5/6.8 mW at 350 MHz when computing 2048/1024/512point FFTs. The buffer-less accelerator also achieves slightly higher
performance than the buffered accelerator, mainly because it avoids
the explicit data copy-in and copy-out phases. We have compared our
58
CHAPTER 3. ARCHITECTURAL SOLUTIONS
buffer-less implementation with state-of-the-art accelerators, showing energy efficiency improvements. Finally, we have fabricated a
single-core chip containing our buffer-less accelerator and validated
our design on silicon.
Chapter 4
Data Type
Specialization: Low and
Mixed-Precision
RISC-V ISA Extensions
In the previous chapter, we have analyzed domain-specific hardware
processing engines by comparing different architectural design choices.
By having internal memories or a custom number of ports to access a
shared memory, large chunks of data can be accessed by these units,
thus providing a high acceleration potential. However, since such units
are highly specialized, they can only be leveraged for very specific
workloads (e.g., the accelerator presented in Chapter 3 computes only
FFTs). While this is attractive for very stable and frequently used
functions, accelerators designed for more dynamic application scenarios might get obsolete very quickly. In this chapter, we concentrate
on a different avenue for increasing energy efficiency: data type specialization and related microarchitectural enhancements. Although
reduced-precision data types can often offer less acceleration potential
than full domain-specific engines, low-precision approaches are not
tied to a specific function, thus providing higher flexibility. At the
59
60
CHAPTER 4. DATA TYPE SPECIALIZATION
same time, they also allow for reducing the data movement and the
memory footprint, which have become increasing concerns due to the
ever-higher NN model sizes.
The following sections have been published in a slightly different
form in the IEEE Transaction on Emerging Topics in Computing
(TETC) 1 , which extended the paper previously published in the
IEEE International Symposium on Computer Arithmetic (ARITH)2 .
Section 4.7.2 briefly introduces the Occamy system published in the
IEEE Symposium on VLSI Technology and Circuits (VLSI)3 .
4.1
Introduction
NN models are currently getting larger and larger, highly increasing
the storage and computing capabilities required to train them. Modern NN models require up to three orders of magnitude more computing than only a couple of years ago [99]. To efficiently train such
models, hardware architectures for NN training need to evolve quickly.
Technology scaling alone is not sufficient, and further architectural,
microarchitectural, and algorithmic innovations are required.
More compact data types have recently driven algorithmic and
microarchitectural breakthroughs [100]. Employing low-precision formats reduces the memory footprint of NN models and can be coupled
with microarchitectural enhancements to highly optimize the hardware computational units. Lower-precision formats use more compact
datapaths to process the information, thus allowing for fitting more
1 ©2024 IEEE. Reprinted, with permission, from L. Bertaccini, G. Paulin,
M. Cavalcante, T. Fischer, S. Mach, and L. Benini, “MiniFloats on RISC-V cores:
ISA extensions with mixed-precision short dot products,” IEEE Transactions on
Emerging Topics in Computing, pp. 1–16, 2024.
2 L. Bertaccini, G. Paulin, T. Fischer, S. Mach, and L. Benini, “MiniFloatNN and ExSdotp: An ISA extension and a modular open hardware unit for lowprecision training on RISC-V cores,” in 2022 IEEE 29th Symposium on Computer
Arithmetic (ARITH). IEEE, 2022, pp. 1–8.
3 G. Paulin, P. Scheffler, T. Benz, M. Cavalcante, T. Fischer, M. Eggimann,
Y. Zhang, N. Wistoff, L. Bertaccini, L. Colagrande, G. Ottavi, F. K. Gürkaynak,
D. Rossi, and L. Benini, “Occamy: A 432-core 28.1 DP-GFLOP/s/W 83% FPU
utilization dual-chiplet, dual-HBM2E RISC-V-based accelerator for stencil and
sparse linear algebra computations with 8-to-64-bit floating-point support in 12nm
FinFET,” VLSI Symposium, 2024.
4.1. INTRODUCTION
61
processing elements at the same area cost and enabling large efficiency
improvements.
NN models have proven to be particularly resilient to the noise
these lower-precision data types introduce. Narrow integer formats
– even as low as ternary and binary formats [22, 23] – have been
extensively investigated for NN inference. NN training, however,
demands the higher dynamic range provided by FP formats and is
usually performed using at least 16-bit FP data types. Only recently,
8-bit formats have been explored in a mixed-precision setup with
16-bit formats [26, 27, 101]. As narrow data types provide reduced
dynamic range and precision, they often cannot be used as drop-in
replacements for larger-precision data types. Instead, they require
ad-hoc modifications to the training algorithms, such as employing
larger-precision accumulators, scaling factors, and stochastic rounding [26, 102].
New functional units and accelerators can be designed to further
specialize the hardware architecture and achieve even higher efficiency
improvements. As a primary operation for both inference and training
of NN models, the sum of products is an ideal candidate for a dedicated hardware block. Merging multiple fused multiply-add (FMA)
operations into a single sum of products – or dot product – unit allows
for a single normalization and rounding step, thereby improving the
accuracy, area, timing, and power consumption of the functional unit.
In the context of low-precision NN training, the reduced precision
of the input formats is often unsuitable for the accumulation phase.
To preserve a higher computational precision, the dot product is
particularly relevant when the accumulation result is returned in a
wider format. In this chapter, we refer to such operations as widening
operations.
Further gains can be achieved by designing specialized matrixmultiplication accelerators [30, 103]. Some of these accelerators are
based on short dot-product modules, others on massively-scaled dotproduct units. However, while a high level of specialization enables
large energy efficiency improvements, it often considerably reduces the
application scenarios in which a hardware architecture represents an
attractive solution. In this context, ISA extensions play an interesting
role. Since they do not compromise the standardized baseline ISA,
62
CHAPTER 4. DATA TYPE SPECIALIZATION
they allow for optimizing the system’s efficiency without narrowing
the targeted application scenarios.
An example of such an approach is provided by Manticore [104], a
chiplet-based hierarchically-scalable architecture built upon the replication of efficient compute clusters. Manticore’s open-source Snitch
cluster [69] contains eight compute cores, plus an additional core controlling a DMA engine, sharing a low-latency L1 scratchpad memory.
Each tiny integer RISC-V core in Snitch is coupled to a large FP
accelerator [70] and enhanced with ISA extensions that maximize the
FPU utilization. Alternatively, a large-scale Manticore-like manycore system can be built using Spatz [97] as the compute cluster’s
processing element. Spatz is a compact vector processor working
on a subset of the RISC-V vector extension [105]. In these families
of compact and efficient clusters, the control overhead is minimized
to dedicate most of the power to the real processing happening in
the FPU. The FPU power consumption can then reach more than
40% of the overall power consumed at the cluster level [69]. The
efficiency of the FPU is, therefore, key in such systems. However,
neither Snitch nor Spatz-based clusters include a fused operation unit
for the fundamental operation of NN workloads, i.e., the widening dot
products on the currently trending low-precision FP formats.
In this chapter, we focus on microarchitectural enhancements for
large general-purpose manycore systems based on scalar and vector
processors to increase their performance and efficiency when computing mixed-precision matrix multiplications, the key operations required during NN training. We extend our previous work [52] and
cover the following contributions:
1. We design an open-source multi-format unit supporting widening
dot-product (wDotp) operations. The unit supports two 8-bit and
two 16-bit formats and accumulates at higher precision, 16 and
32 bits, respectively. We reuse the same datapath to compute
non-widening and widening three-term additions, which we call
vector inner sum (Vsum) and widening vector inner sum (wVsum).
Thanks to the high parameterization of the module, new formats
can be rapidly defined and explored. We integrate the wDotp
4.1. INTRODUCTION
63
unit into an open-source multi-format modular FPU called FPnew.
4
[70].
2. We integrate stochastic rounding support into the wDotp unit. To
the best of our knowledge, this is the first open-source RTL implementation of stochastic rounding in FP architectures. The number
of trailing bits considered for the stochastic rounding decision and
the pseudo-random generator’s internal bitwidth can be tuned at
design time through two dedicated parameters. We analyze the
integration of this new rounding mode into different FP modules.
3. We specify two distinct RISC-V ISA extensions for low-precision
FP training, respectively on scalar and vector manycore architectures exploiting the new computational units: MiniFloat-NNScalar and MiniFloat-NN-Vector. Furthermore, we comment on
the different challenges faced to enable efficient low and mixedprecision computations on a scalar and a vector RISC-V processor.
4. We integrate wDotp support into two compute clusters that can be
hierarchically replicated to build large efficient manycore systems,
one based on scalar cores and one on vector processors. In such
compute clusters, the cores share an L1 instruction cache and a lowlatency scratchpad data memory. We build a MiniFloat-NN-Scalar
cluster, integrating the enhanced FPU into an open-source RISC-V
8-core cluster based on Snitch cores [69] supporting the new scalar
ISA extension and a MiniFloat-NN-Vector cluster containing the
same number of enhanced FPUs but based on Spatz [97], a small
and efficient vector processor.
5. We carry out a detailed evaluation of the standalone wDotp unit
and the enhanced compute clusters, providing area, performance,
and energy efficiency results.
4 Available at: https://github.com/pulp-platform/cvfpu.
64
CHAPTER 4. DATA TYPE SPECIALIZATION
FP32
8
TF32
8
bfloat16
8
23
10
7
DLFloat
6
9
FP16
5
10
FP8
5
2
FP8alt
4
3
Figure 4.1: Relevant floating-point formats for NN training. The
exponent (green) and mantissa (blue) bitwidths are reported for each
data type.
4.2
Related Work
4.2.1
Low-Precision Floating-Point Formats
Employing low-precision FP formats reduces the memory footprint
and the energy costs related to data movement and processing [20,
106]. All these advantages are particularly attractive for NNs, as they
help tackle the challenges introduced by the extremely fast scale-up
of NN models.
While the first training algorithms were based on FP32, the resilience of NN models to noise allowed shifting toward narrower formats. In this scenario, pushed by the high trade-off between dynamic
range and precision in low-precision data types, a large set of new
formats has been investigated (see Fig. 4.1).
The first efforts were based on FP16 datasets. Then, two other
formats, TensorFloat-32 (TF32) [16] and bfloat16 [24], offering a
higher dynamic range than what is provided by the 5 exponent bits
of FP16, were proposed. The former is a 19-bit format with the same
number of mantissa bits as FP16 but keeping the exponent bitwidth,
thus the dynamic range, of FP32. The latter is a 16-bit data type that
4.2. RELATED WORK
65
further prunes the number of mantissa bits to 7 while maintaining the
FP32 dynamic range.
As the mantissa datapath dominates the area and power consumption of FP engines, reducing its width allows placing more computational units at the same area cost and enables larger efficiency
improvements than lowering the number of exponent bits. This resulted in considerable performance and efficiency gains, as testified by
the various generations of GPUs recently released [16, 30]. However,
as narrow mantissae limit the accuracy, formats such as bfloat16
are conceived to be used with higher-precision accumulation data
types [24]. Such a mixed-precision approach combines the best of both
worlds, enabling the benefits of low-precision formats while retaining
a high accuracy [20].
Henry et al. [107] and Osorio et al. [108] investigated alternative
approaches that use bfloat16 without FP32 accumulators. In these
cases, FP32 accumulators are replaced with a compound of bfloat16
elements, and the information, or part of the information, that would
be contained in an FP32 mantissa is split among multiple bfloat16
mantissae. Such an approach provides higher accuracy than using
single bfloat16 elements, achieving accuracy levels comparable to
FP32. Despite not requiring a mix of formats, this method still requires more bits of information used for the accumulation than what
a single bfloat16 element can provide.
To mitigate the need for larger-precision accumulation, IBM introduced DLFloat [25], a 16-bit format composed of a 6-bit exponent and
a 9-bit mantissa, thereby providing an intermediate dynamic range
and precision with respect to FP16 and bfloat16. Finally, as different applications benefit more from different data types, Nannarelli
proposed a variable precision 16-bit format [109] that can be set to
represent FP16, bfloat16, DLFloat, and an additional data type with
a 7-bit exponent and a 9-bit mantissa.
Further gains can be enabled by moving to 8-bit formats. Lately,
two 8-bit FP data types have gathered interest, one consisting of a
5-bit exponent and a 2-bit mantissa and the other of a 4-bit exponent and a 3-bit mantissa. We call the first FP8 and the second
FP8alt. Sun et al. [101] demonstrated considerable improvements for
a large set of NN training tasks by employing FP8alt in the forward
propagation and FP8 in the backward pass. However, these studies
66
CHAPTER 4. DATA TYPE SPECIALIZATION
relied on software emulation and were not performed on hardware
platforms. NVIDIA recently released the H100 GPU [30], where these
8-bit formats are supported and provide a 2× speedup with respect to
16-bit data types and a 4× speedup with respect to TF32. The Tesla
Dojo [110] also supports two 8-bit FP data types with the same number of exponent and mantissa bits but also allows for a configurable
bias. Nonetheless, hardware architectures supporting such formats are
still rather new and not well studied from the application viewpoint.
The vast set of FP data types being investigated motivates the
need for a flexible open-source hardware platform in which new formats could be rapidly explored and easily supported.
4.2.2
Floating-Point Architectures and Designs
As primary operations for many computing kernels, most modern
processors provide support for FMA instructions, usually in the nonwidening variant. However, in the context of NN training, widening
operations are particularly useful, as they allow using low-precision
formats while retaining high accuracy [20]. To address the need
for efficient widening computation, various widening FMA (wFMA)
implementations have been investigated. Brunie [111] proposed a
wFMAs unit multiplying FP16 inputs and accumulating at higher
precision, while Mach et al. [70] designed an FPU for transprecision
computing, including a multi-format FMA module capable of computing wFMAs on a wide set of FP formats. However, as we show
in Section 4.3, wFMAs cause various inefficiencies when used in a
single instruction multiple data (SIMD) or vector processing context.
Furthermore, higher performance and efficiency can be achieved by
developing a widening dot-product unit with accumulation.
A widening dot-product unit with accumulation, which computes
a × b + c × d + e, can be designed in a discrete or fused fashion. The
former consists of two cascaded wFMA modules, as shown in Fig. 4.2.
A fused design involves implementing the non-trivial three-term FP
addition [112] but allows for a single normalization and rounding step,
thereby improving the module’s area, timing, and accuracy.
67
4.2. RELATED WORK
w
w
2w
w w w w 2w
wFMA
w
w
2w
wDotp
wFMA
2w
Cascade2w = aw × bw + (cw × dw + e2w)
2w
wDotp2w = aw × bw + cw × dw + e2w
Figure 4.2: wDotp operation computed by two cascaded wFMA units
vs. one dedicated wDotp unit. Note that the first implementation
computes a × b + (c × d + e), which is not necessarily equal to a × b +
c × d + e when using FP arithmetic.
Sohn and Swartzlander [113] developed a fused FP dot-product
unit computing a × b + c × d. Additionally, they designed a nonwidening three-term FP adder [114], which, merged with their dotproduct unit, can generate a dot-product unit with accumulation.
These two designs were only implemented for double and single precision and were not designed for widening operations.
Intel developed a fused FP many-term dot-product unit [103] computing 32 products and accumulating them at higher precision for the
Nervana NN processor. The unit multiplies bfloat16 inputs two by
two, adds the products using 37 bits of precision, and finally accumulates the result in an FP32 accumulator. The Nervana dot-product
unit increases its internal datapath to reduce, but not fully prevent,
precision losses when cancellation causes a large normalization shift.
Such losses would have a higher impact on low-precision FP formats,
as their final results are more sensitive to small variations. Desrentes
et al. [115] designed another unit for large N-term dot products, which
instead targets exact dot-product operations. Such a module employs
a compressed accumulator to ensure an exact N-term summation.
Other two designs with multiple-precision dot-product capabilities
were proposed by Zhang et al. [116] and Mao et al. [117]. However,
these two units, as well as the exact dot-product unit proposed by
68
CHAPTER 4. DATA TYPE SPECIALIZATION
Desrentes et al. and the Nervana dot-product module, do not support
the recently proposed 8-bit formats. On the contrary, the IBM AI
chip [118] supports FP8 and FP8alt widening dot products but is
only capable of DLFloat FMAs.
As low-precision data types present a high trade-off between accuracy and dynamic range, supporting multiple formats allows for
dynamically adapting to the application requirements. The FPU developed by Nannarelli [109] addresses this by defining a new variableprecision data type that can be set to FP16, bfloat16, DLFloat, and
a fourth 16-bit custom format. However, no support for 8-bit formats
was considered.
Finally, FPnew [70] is a highly parameterized design capable of
working on a wide set of FP data types, from 8-bit to 64-bit, and allows
for a fast definition of new formats. Nonetheless, it does not provide
support for dot-product operations. Due to its highly configurable
environment, we take the open-source FPnew unit as a starting point
for our work. FPnew is also already included in flexible and highly
efficient compute clusters [69,97] that can be hierarchically replicated
to build large manycore systems [104], enabling a fast exploration of
our extensions on real hardware platforms.
In this chapter, we develop an open-source wDotp unit supporting
two 8-bit and two 16-bit formats and accumulating at higher precision,
16 bits and 32 bits, respectively. A quantitative comparison between
our module and state-of-the-art designs is provided in Section 4.5.1.
4.2.3
Rounding Modes for Low-Precision NN
Training
Low-precision NN training kernels accumulating over a 16-bit FP
format may incur stagnation due to the limited accumulation precision. Such a situation arises, for example, when a long series of
small backpropagated increments are lost to rounding. To mitigate
this issue, various studies have proposed stochastic rounding (SR) as
an alternative rounding method [26, 119, 120]. Moreover, Fasi and
Mikaitis [121] showed that an even broader set of algorithms can
benefit from SR.
SR takes the rounding decision based on a probability that depends on the distance to the next representable value. To do that,
4.3. MIXED-PRECISION INSTRUCTIONS
69
Table 4.1: Stochastic Rounding: example considering 3 trailing bits.
Trailing Bits
Stochastic Rounding
000
001
010
011
100
101
110
111
0% round up
12.5% round up
25.0% round up
37.5% round up
50.0% round up
62.5% round up
75.0% round up
87.5% round up
n extra bits are considered from the fixed-point resulting mantissa
before normalization. As an example, we report in Table 4.1 the
round-up probabilities based on the trailing bits when three bits are
considered. For instance, if the trailing bits are 3’b010, the result
will be rounded up with 25% probability and rounded down with 75%
probability. This rounding scheme differs from round to nearest even,
which takes the rounding decision based on the state of the guard,
round, and sticky bits. Furthermore, in the case of SR, the extra
bits not considered for the rounding decision are discarded instead of
OR-ing them in a sticky bit as when rounding to nearest even.
While most state-of-the-art investigations rely on software emulation to provide SR results, the first commercial hardware platforms
providing SR support already appeared on the market, e.g., Graphcore
IPU [33], Intel Loihi [122], AWS Trainium [32], and Tesla Dojo [110].
On the academic side, Mikaitis proposed a hardware SR unit [123],
however, working on fixed-point arithmetic. While a similar approach
can be taken in FP units, as we show in Section 4.4.4, that is not
always straightforward. In this chapter, we present the first opensource hardware FP dot product unit with support for SR.
4.3
Mixed-Precision Instructions
Addressing mixed-precision applications on instruction processors introduces various challenges and trade-offs. In this section, we consider
CHAPTER 4. DATA TYPE SPECIALIZATION
70
a)
FP32
FP32
×
FP16FP32FP16
FP16FP32FP16
ft1
ft0
FMA: aw*bw + cw
×
+
SIMD FMA
+
fa0
FP32
64 bits
FP32
b)
FP16
FP16
×
FP16
FP16
FP16
FP16
SIMD wFMA.u
×
FP16
ft0
wFMA: aw*bw + c2w
ft1
+
fa0
FP16
+
FP32
64 bits
FP32
FP16
wFMA: aw*bw + c2w
ft0
FP16
FP16
FP16
SIMD wFMA.l
FP16
ft1
FP16
FP16
FP16
×
+
fa0
×
+
FP32
64 bits
FP32
c)
FP16
FP16
FP16
×
FP16
FP16
ft1
ft0
wDotp: aw*bw + cw*dw + e2w
FP16
FP16
SIMD wDotp
FP16
×
+
×
+
+
FP32
fa0
×
+
64 bits
FP32
Figure 4.3: Register file utilization: FMA vs. wFMA vs. wDotp. We assume the inputs are stored in
the registers ft0 and ft1, while fa0 contains the accumulator. (a) In the FMA case, the information
contained in the source registers is fully processed; (b) In the wFMA case, only half of the data in two
source registers can be processed each cycle, thus requiring two explicit instructions to cover all possible
source locations; (c) wDotp fully exploits the information contained in the registers, and that can be
passed to the FPU interface while taking advantage of the lower-precision inputs.
4.3. MIXED-PRECISION INSTRUCTIONS
71
a kernel supporting 16-bit FP inputs and requiring 32-bit accumulation. Similar considerations also apply to 8-bit kernels with accumulation in 16 bits.
If no mixed-precision instructions are available, as in the scalar
RISC-V ISA [36], explicit conversions are needed to take advantage
of the lower-precision inputs. After loading the data, the inputs
must be converted to FP32 before computing an FMA. Such explicit
conversions can come at a particularly high cost if the level of input
reuse is low. In this scenario, each FMA would require two cast
instructions, thus highly impacting the system’s performance and
energy efficiency. This overhead pushes towards FP32-only kernels
to avoid losing performance. However, this also implies missing the
advantages enabled by the compressed 16-bit inputs. In a 64-bit
system with support for SIMD operations, this results in computing
two FP32 FMAs per cycle, as shown in Fig. 4.3a.
To efficiently support mixed-precision applications, mixedprecision instructions are needed. As the fundamental instruction
in many kernels, the FMA operation is an interesting candidate to be
extended with mixed-precision capabilities, thus avoiding the need for
explicit conversions. The RISC-V vector extension [105] and Mach et
al. [70], when extending the RISC-V scalar ISA with scalar low and
mixed-precision instructions, have taken such an approach.
In the scalar case, various issues are encountered when adding
low-precision SIMD wFMA instructions. While SIMD operations can
exploit the larger information that can be stored in the register file, as
shown in Fig. 4.3b, SIMD wFMAs do not fully use the FP registers.
wFMAs access only half of the data contained in two source registers,
thus not fully exploiting all the information that can be packed into
those registers while entirely using the register containing the accumulator. Therefore, this approach achieves the same performance achievable with a non-widening FP32-only application, although it enables
the memory footprint and energy efficiency benefits introduced by the
lower-precision inputs. At the same time, to use the entire register
file, multiple instructions to cover all possible source locations would
be needed.
In the vector case, the time-multiplexed nature of vector instructions helps mitigate some of these inefficiencies. As a single vector
instruction typically occupies the datapath for many cycles, there is
72
CHAPTER 4. DATA TYPE SPECIALIZATION
no need to double the number of instructions to cover all possible
source locations. This can be addressed by some minor hardware
modifications. The same data will be read twice from the vector
register file, and only the upper/lower part of the information will
then be forwarded to the FPU. However, as shown in Fig. 4.4, if the
FPU has a single output port and also supports larger and non-mixedprecision operations, the FPU bandwidth is underutilized when using
wFMAs. Since this limits the performance achievable with wFMAs,
an extra read and write port could be added to the register file, and
an extra input and output port could be added to the FPU to enable
the desired 2× speedup when moving to input formats using half of
the bits. This would provide the necessary extra accumulators to fully
process the 64 bits – containing multiple lower-precision values – that
can be passed to each of the other two FPU input ports. However,
adding one read and one output port to the vector register file, which is
a routing-intensive block, can have a non-negligible impact on the area
and timing of the design, potentially limiting the maximum achievable
operating frequency. We evaluated this case in Section 4.5.3, and in
our experiment, such modifications reduced the maximum operating
frequency by around 10%.
In both the scalar and the vector scenarios, a wDotp instruction
would fully utilize the FPU bandwidth and, in the scalar case, also
consume all the accessible bits stored in the FP registers it operates
on, as shown in Fig. 4.3c. More data are provided to the FPU, and
more computation is performed every cycle, thus reaching twice the
performance achievable by exploiting wFMA operations.
In this chapter, we integrate a SIMD wDotp module in two
lightweight open-source RISC-V processors, one scalar and one vector,
called Snitch [69] and Spatz [97], respectively, to build a scalar
and a vector compute clusters for efficient low and mixed-precision
NN training. To enable that, we extend their ISAs with two
custom RISC-V extensions that we named MiniFloat-NN-Scalar and
MiniFloat-NN-Vector. Although ISA extensions for NN training have
already been employed in commercial products, as in the Habana
Gaudi2 [31], they are mostly closed-source and lack open information
on the implementation breakdown costs and design trade-offs. In
this chapter, we provide a detailed description and evaluation of the
proposed hardware unit and the related ISA extensions.
73
4.4. ARCHITECTURE
wFMA
00…0
FP16 FP16
00...0
FP16 FP16
FP32
FP32
FPU
FP32
FP32
FP32
FP32
wDotp
FP16 FP16 FP16 FP16
FP16 FP16 FP16 FP16
FP32
FPU
FP32
Figure 4.4: In a general-purpose FPU that supports high and lowprecision formats, multiple low-precision operations can be computed
in parallel, taking advantage of the larger bandwidth. However,
wFMAs underutilize such bandwidth in both the scalar and vector
processor scenarios. The bandwidth can be fully utilized, providing
more data to the FPU each cycle if support for wDotp operations is
provided.
4.4
Architecture
We discuss the supported FP formats in Section 4.4.1 and present the
architecture of the proposed FP wDotp module supporting wVsum
and Vsum operations on the same datapath in Sections 4.4.2 and 4.4.3.
We cover the integration of wDotp units into a modular open-source
multi-format FPU in Section 4.4.5. Finally, we present the scalar and
vector evaluation compute clusters enhanced with wDotp capabilities
and support for the MiniFloat-NN-Scalar and MiniFloat-NN-Vector
ISA extensions, respectively, in Sections 4.4.6 and 4.4.7.
74
CHAPTER 4. DATA TYPE SPECIALIZATION
Table 4.2: Combinations of source and destination formats supported
by the wDotp unit for each operation.
Destination
Source
FP8
FP8alt
FP32
Vsum
FP16alt wDotp/wVsum
Vsum
Vsum
FP16
wDotp/wVsum
Vsum
Vsum
FP8
wDotp/wVsum wDotp/wVsum Vsum
FP8alt
wDotp/wVsum wDotp/wVsum Vsum
Vsum
Vsum
4.4.1
FP32
FP16alt
FP16
Supported FP Formats
The high parametrization of the open-source FPnew unit allows designers to select a set of supported FP formats and rapidly define
new formats, thereby enabling fast hardware explorations of new data
types. For our study, we add the FP8alt definition and enable the
following FP formats:
• FP64: 11-bit exponent, 52-bit mantissa
• FP32: 8-bit exponent, 23-bit mantissa
• FP16: 5-bit exponent, 10-bit mantissa
• FP16alt: 8-bit exponent, 7-bit mantissa
• FP8: 5-bit exponent, 2-bit mantissa
• FP8alt: 4-bit exponent, 3-bit mantissa
FP16alt has the exponent and mantissa widths of the widely-used
bfloat16 format but follows all IEEE-754 principles, including subnormal number handling.
The extended FPU supports all the RISC-V FP instructions,
except division and square root, for all the enabled formats. Our
wDotp unit focuses on low-precision data types, supporting widening
8-to-16-bit and 16-to-32-bit wDotp and wVsum operations, and nonwidening 8, 16, and 32-bit Vsum computations, as summarized in
Table 4.2.
4.4. ARCHITECTURE
4.4.2
75
wDotp Unit
The wDotp unit computes mixed-precision dot products with accumulation:
wDotp2w = aw × bw + cw × dw + e2w .
(4.1)
Four operands are expressed in a w-bit source format (src fmt), while
the accumulator input and the output are expressed in a 2w-bit destination format (dst fmt).
The wDotp module handles subnormals and has a parametric
number of pipeline stages that can be set at design time. The internal
bitwidths depend on the set of enabled input and output formats.
A source super-format is internally defined to support inputs expressed in different formats sharing the same overall bitwidths. Such
a super-format is characterized by the maximum exponent and mantissa bitwidths among the enabled source formats. Therefore, if FP8,
FP8alt, FP16, and FP16alt are enabled as source formats, the internal
source super-format will have an 8-bit exponent and a 10-bit mantissa.
The wDotp inputs are pre-processed internally to map exponents and
mantissae expressed with fewer bits to the lower and upper part of
the super-format exponent and mantissa, respectively. That allows
the same arithmetic components to be used for all the enabled source
formats.
In this chapter, we consider two wDotp instances: a 16-to-32-bit
unit supporting all the operations reported in Table 4.2 and an 8to-16-bit unit supporting the same set of operations but those that
accumulate in FP32. The data flow in the wDotp unit is shown in
Fig. 4.5. For the sake of simplicity, the diagram does not depict the
exponent datapath, which is responsible for computing the exponent
differences used to sort the two products and the accumulator and
to calculate the shift amounts. The maximum number of mantissa
bits plus one (the hidden one) is called p src and p dst for source and
destination formats, respectively.
Initially, the two mantissa products are computed, producing
(2p src)-wide results. Note that 2p src might differ from p dst. To
match the same precision as the accumulator, the mantissa products
are padded with zeroes. After that step, what is left is a threeterm addition. As FP additions are non-associative, consecutive FP
76
CHAPTER 4. DATA TYPE SPECIALIZATION
a2w
a-vsw
bw
aw
c2w
bw
p_src
c-vsw
p_src
product
00
0
p_dst
2*p_src
00
p_dst
p_dst
1
e2w
p_src
x
p_dst
2*p_src
product
e2w
dw
p_src
x
p_dst
dw
cw
1
is_vsum
0
THREE-ADDEND SORTING
addend_int
addend_min
00..000
00..000
2*p_dst+3
2*p_dst+p_src+3
>>
addend_max
00..000
0..00
addend_int
2*p_dst+3
>>
00..000
00..000
addend_min 0..00
2*p_dst+3
+
2*p_dst+4
sum
00..000
2*p_dst+p_src+4
sum
00..000
2*p_dst+p_src+4
0..00
+
2*p_dst+p_src+5
is_first_sum_exact_zero
sum_final
0
1
NORMALIZATION & ROUNDING
result2w
Figure 4.5: wDotp data flow: examples of how the information is
packed at each stage of the datapath are provided in the blue boxes
(note that we do not include the exponent datapath in this figure,
nor the sticky bits that are generated after the various shifts). The
wDotp operation takes four w-bit inputs and a 2w-bit accumulator,
while the Vsum takes three 2w-bit inputs to produce a 2w-bit output.
The three-addend sorting is based exclusively on the exponent values.
4.4. ARCHITECTURE
77
additions might produce different results if performed in different
orders. In particular, various orderings have been studied to reduce
the error in partial and total sums [124, 125]. When considering two
FP additions, fusing them into a single datapath with only one final
normalization and rounding stage allows for mitigating precision losses
related to the non-associativity of FP additions. Before normalizing
and rounding, the mantissa additions can be performed using a wider
bitwidth, thus ensuring enough extra bits are available to recover from
partial cancellations (see Fig. 4.6 and Fig. 4.7). This, however, does
not prevent potential accuracy losses in the case of total cancellations.
For example, when computing (a + b + c), with a = −c and |a| ≫ |b|,
after aligning the three operands to the maximum exponent, the
mantissa of b could be entirely or partially shifted out of the range
covered by the internal adders. We recover from this situation by
sorting the operands (based on their exponents) in decreasing order
and checking whether the result of the first addition is zero. In that
case, we assign the unshifted minimum addend to the mantissa result
(see Fig. 4.8).
Furthermore, after summing the maximum and intermediate addends, the internal bitwidth is increased by p src bits to maintain
enough extra bits to recover from partial cancellations when the maximum addend is the result of a product between a normal and a
subnormal value. In this case, the leading one can be found up to
after p src bits with respect to the most significant bit of the product.
The minimum addend is accumulated to the padded sum just before
the final normalization and rounding.
Summing the addends with the largest absolute values first and
gradually increasing the bitwidths at each step allows us to mitigate
potential precision losses generated by the non-associativity of two
consecutive FP additions. If the first sum produces a non-zero value,
the increased precision ensures enough precision bits even in case
of cancellation in the second addition. Otherwise, if the first sum
produces an exact zero result, the shifted-out bits of the minimum
addend are recovered by directly assigning the minimum addend to
the result of the second sum.
78
CHAPTER 4. DATA TYPE SPECIALIZATION
NO CANCELLATION
1 1 1 1 1 0 0 1 1 1
+
1 1 0 0 0 0 1 1 0 0
=
1 0 0 0 0 0 1 0 1 1 1 1 1 0 0
+
out of
adder
range
1 1 0 1 1 0 0 0 1 1
=
0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 1 1 0 0 0 1 1 0 0
normalized
trailing bits
1 0 0 0 0 0 1 1 0 0
+
0/1
>
LFSR
Figure 4.6: Three-term addition example focusing on the processing
performed after the mantissa alignment and before rounding in the
absence of cancellation. In blue, we highlight the resulting mantissa
after normalization. Rounding to nearest even, as well as all the
other rounding modes supported by RISC-V, will then be performed
considering the shifted-out bits before the additions and the extra bits
of information not included in the normalized mantissa. Stochastic
rounding is instead implemented by comparing the trailing bits of the
fixed-point internal representation of the resulting mantissa with the
output of an LFSR. If the trailing bits are greater than the LFSR
output, the normalized result will be rounded up. SR is similarly
applied in cases involving cancellations.
79
4.4. ARCHITECTURE
PARTIAL CANCELLATION
1 0 1 1 1 0 0 1 1 1
−
1 0 1 1 1 0 0 1 0 0
=
0 0 0 0 0 0 0 0 0 1 1
+
=
out of
adder
range
1 1 0 1 1 0 0 0 1
0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0
normalized
1 1 0 0 0 0 0 0 1 1
Figure 4.7: Three-term addition example involving partial cancellation: the wider internal bitwidth allows retaining enough information
from the third operand (the minimum addend).
80
CHAPTER 4. DATA TYPE SPECIALIZATION
TOTAL CANCELLATION
1 0 1 1 1 0 0 1 1 1
−
1 0 1 1 1 0 0 1 1 1
=
0 0 0 0 0 0 0 0 0 0 0
+
=
out of
adder
range
1 1 0 1 1 0 0 0 1 1
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0
normalized
1 1 0 1 1 0 0 0 0 0
1 1 0 1 1 0 0 0 1 1
direct assignment
Figure 4.8: Three-term addition example involving total cancellation.
The wider internal bitwidth does not prevent precision losses when the
minimum operand is much smaller than the other two. Sorting the
operands in decreasing order and checking whether the result of the
first addition is zero allow for recovering from this situation by directly
assigning the third operand to the result (in the figure, the information
related to the minimum operand is partially lost; depending on the
exponent difference with respect to the largest operand, the whole
information contained in the minimum operand could be lost).
4.4. ARCHITECTURE
4.4.3
81
wVsum and Vsum on the wDotp Datapath
By setting the inputs bw and dw to one, the wDotp module can
compute a wVsum (4.2) operation.
wVsum2w = aw + cw + e2w .
(4.2)
As discussed in Section 4.4.2, the proposed unit already contains all
the logic necessary to calculate a non-widening three-term addition.
To enable non-widening Vsum (4.3) in the larger dst fmt, we increase
the size of two inputs, operand a and c, to the dst fmt width by
extending them with the a vs and c vs fields.
Vsum2w = a2w + c2w + e2w .
(4.3)
The support for Vsum is added by bypassing the two mantissa multiplications, as shown in Fig. 4.5. Such an operation can be used to
reduce and accumulate the results packed in a register after SIMD
wDotp executions (see Fig. 4.3c).
4.4.4
Stochastic Rounding Support
In the RISC-V ISA, three bits are used to select the rounding mode.
Six out of the eight combinations are used for (i) round to nearest even, (ii) round towards zero, (iii) round down, (iv) round up,
(v) round to nearest ties to maximum magnitude, and (vi) dynamic
rounding mode. We use one of the remaining two combinations to
add support for SR.
To enable SR support in the wDotp unit, we integrate an LFSR
into our rounding module. Whenever SR is selected, the trailing bits of
the fixed-point resulting mantissa are compared against the generated
pseudo-random number. If the value encoded in the trailing bits is
greater than the LFSR output, the FP result will be rounded up;
otherwise, it will be rounded down.
We set the number of trailing bits used to take the SR decision
and the LFSR length through two dedicated parameters tunable at
design time. The number of trailing bits correlates with the precision
considered to take the rounding decision, while the number of LFSR
internal bits sets the periodicity of the pseudo-random number. In
82
CHAPTER 4. DATA TYPE SPECIALIZATION
this chapter, we evaluate an SR implementation using a 32-bit LFSR
and 12 trailing bits.
Note that while SR support can be readily integrated into our
proposed wDotp unit, as well as in conversion units, adding it to FMA
architectures might be problematic. In the case of single-path FMA
units [126], where the product is anchored, the final sum is not always
represented in a true fixed-point notation as required by SR. In such
modules, the product, expressed with 2p bits, has to be added to a
p-bit addend. The product is anchored, and the accumulator is shifted
accordingly to the exponent difference. If the exponent difference
(exp addend − exp prod) is larger than 2p + 2, the product will only be
in the sticky bits. However, since the product is anchored, the addend
will be left-shifted to get in the final sum MSBs. That will be followed
by two zeroes (there, for rounding purposes) and the product. While
this bit string does not represent an exact fixed-point notation, it does
not create issues when rounding to nearest even, as the round bit will
always be zero, and the product will only end up in the sticky bits, thus
not modifying the content of the addend. In this kind of situation,
only two trailing bits could be considered for SR unless the number of
guard bits kept between the extreme positions of the addend and
the product is increased to the number of bits considered for the
stochastic rounding decision. This would, however, increase the size
of the adder and the alignment and normalization shifters, requiring
substantial modifications to the unit and resulting in a higher area
and critical path impact. Otherwise, the FMA should be implemented
with a different architecture, e.g., always shifting the minimum term
between the addend and the product rather than always shifting only
the addend. However, also in this case, the modification would be
substantial and would not allow, for example, to shift the addend in
parallel to the mantissa multiplication. For these reasons, we do not
extend the FMA unit with support for SR, and we keep rounding
FMAs to nearest even when SR is enabled.
4.4.5
SIMD Wrapper and Integration into FPnew
FPnew is natively organized into modules, each one responsible for
one operation group: ADDMUL, DIVSQRT, COMP (comparison),
and CAST (conversion). Each module can be deactivated through a
83
4.4. ARCHITECTURE
64
FPU
64
64
Operands distribution
Operation
group
Operation
Group
Operation
Group
Operation
Group
ADDMUL
COMP
CAST
DOTP
Round-robin output arbitration
64
DOTP
64
64
64
Unpacking
16-to-32
wDotp
16-to-32
wDotp
8-to-16
wDotp
8-to-16
wDotp
Packing
64
Figure 4.9: Block diagram of the extended FPU, with a zoom on the
wDotp SIMD module, containing two 16-to-32-bit and two 8-to-16-bit
wDotp units.
84
CHAPTER 4. DATA TYPE SPECIALIZATION
parameter when instantiating the design. The FPUs included in the
Snitch and Spatz-based compute clusters have the DIVSQRT support
disabled, thus not including the correspondent hardware module. We
add support for wDotp operations through a dedicated operation
group block that we named DOTP.
Since the two targeted compute clusters support double-precision
instructions, their FP register files consist of 64-bit entries. That
means two FP32, four FP16/FP16alt, or eight FP8/FP8alt elements
can be packed in a single register. At the same time, the FPU interface
will accept three 64-bit inputs and produce a 64-bit output. The
native SIMD FMA unit contains one 64-bit, one 32-bit, two 16-bit, and
four 8-bit modules. As larger-precision units are also used for lowerprecision computations to achieve a higher area efficiency, meaning
that a 64-bit unit can perform 64, 32, 16, and 8 operations, the SIMD
FMA unit can perform one 64-bit, two 32-bit, four 16-bit, or eight
8-bit FMAs per cycle. As our wDotp unit focuses on low-precision
formats, we organized our DOTP operation group block in a SIMD
unit, as shown in Fig. 4.9. The SIMD module is composed of two
16-to-32-bit and two 8-to-16-bit wDotp modules, thus achieving two
16-to-32-bit or four 8-to-16-bit wDotp operations per cycle.
4.4.6
MiniFloat-NN-Scalar Cluster
We build a MiniFloat-NN-Scalar cluster by extending the open-source
Snitch cluster with wDotp, wVsum, and Vsum capabilities. The
Snitch cluster includes eight compute cores and a DMA core sharing
a 128 KiB scratchpad memory organized into 32 banks and an L1
instruction cache, as shown in Fig. 4.10. Its compute core, the SnitchCC [69], contains a tiny open-source 32-bit RISC-V processor coupled
with an FPnew [70] instance supporting double and single-precision
FP through a latency-tolerant interface. Snitch uses two custom ISA
extensions (SSR, FREP) to enable around 90% FPU utilization.
The FPU utilization metric is defined as the percentage of cycles
in which an FP arithmetic instruction is offloaded to the FPU with
respect to the overall number of execution cycles. As the FPU datapath is pipelined, each cycle, a new FP operation can be offloaded to
the FPU. Due to additional control instructions, banking conflicts,
or read-after-write dependencies, the FPU utilization moves away
85
4.4. ARCHITECTURE
SB 1
SB 0
...
B0
...
64b
SuperBank 3
B23 ... B31
SB 2
...
B0
B8
...
64b
B16
B0 ...
...
64b
...
64b
512b
64b
512b
Shared L1 Scratchpad Crossbar
64b
Snitch-CC0
...
L0 I$
64b
64b
64b
FPU
64b
...
CC 1
64b
FPU
CC 7
L0 I$
256b
L0 I$
256b
Periph.
64b AXI Crossbar
FPU
256b
Shared L1 I$
DMA
CC 8
L0 I$
256b
512b
512b
512b AXI Crossbar
64b AXI
512b AXI
Figure 4.10: A cluster composed of eight Snitch compute cores, each
one coupled with FPnew extended with wDotp capabilities, plus an
extra DMA core.
86
CHAPTER 4. DATA TYPE SPECIALIZATION
from the theoretical 100% utilization. The FPU utilization, therefore,
directly represents the ratio between the actual achieved FLOPS and
its theoretical limit.
The stream semantic register (SSR) extension [69] maps a regular
load or store access pattern to fixed floating-point registers, effectively eliminating most of the explicit load and store instructions.
The floating-point repetition instruction (FREP) extension [69] allows
buffering and repeating a sequence of FPU instructions to prevent the
loop overhead caused by branching instructions and to relieve pressure
on the instruction cache.
We replace the native FPnew instance in the Snitch-CC with our
enhanced FPU and extend the Snitch decoder to support the new
instructions. We set 3 levels of pipeline registers for the DOTP
operation group, 3 for the ADDMUL, 2 for the CAST, and 1 for
the COMP. To avoid different LFSRs producing the same sequence of
pseudo-random numbers, we set a distinct seed for each LFSR.
To limit the encoding space required by our ISA extension, we do
not replicate the same instruction for different FP formats sharing the
same width. Instead, the alternative formats – FP16alt and FP8alt
– are controlled by two additional bits, src is alt and dst is alt,
in the FP control and status registers (CSRs). An FP16alt kernel
will then differ from an FP16 kernel by a single CSR write. The
MiniFloat-NN-Scalar extension augments the smallFloat5 extension
by adding wDotp, wVsum, and Vsum SIMD instructions:
• wdotp rd, rs1, rs2
• wvsum rd, rs1, rs2
• vsum rd, rs1
For all the three instruction types rd contains the accumulator and
also behaves as rs3/rs2. We summarize the new instructions included
in the MiniFloat-NN-Scalar ISA extension in Table 4.3. Different
instructions are defined for wDotp, wVsum, and Vsum instructions if
the result is added or subtracted to the accumulator. A scalar-SIMD
version of the wDotp instructions, identified with a .R suffix, was also
included in the ISA extension. In this case, the content of the packed
5 https://iis-git.ee.ethz.ch/smach/smallFloat-spec
00111
10111
00111
10111
00111
10111
00111
10111
01011
01011
11101
11101
00111
10111
01011
01011
11101
11101
10
10
10
10
10
10
10
10
10
10
10
10
10
10
10
10
10
10
25
pfltop
29
f2
3130
10111
10111
rs2
rs2
rs2
rs2
10110
10110
rs2
rs2
rs2
rs2
00111
00111
11110
11110
11100
11100
rs2
24 20
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
rs1
1915
0
0
0
1
0
1
0
0
0
1
0
1
0
0
0
0
0
0
R
10
10
10
10
10
10
00
00
00
00
00
00
10
10
10
10
00
00
fmt
14 1312
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
rd
117
0
RVF-type
0110011
0110011
0110011
0110011
0110011
0110011
0110011
0110011
0110011
0110011
0110011
0110011
PFWSUM.H.B
PFNWSUM.H.B
PFWDOTP.H.B
PFWDOTP.H.R.B
PFNWDOTP.H.B
PFNWDOTP.H.R.B
PFWSUM.S.H
PFNWSUM.S.H
PFWDOTP.S.H
PFWDOTP.S.R.H
PFNWDOTP.S.H
PFNWDOTP.S.R.H
0110011 PFSUM.B
0110011 PFNSUM.B
0110011 PFSUM.H
0110011 PFNSUM.H
0110011 PFSUM.S
0110011 PFNSUM.S
opcode
6
rdfp16 [i] += rs1fp8 [2*i+1]+rs2fp8 [2*i]
rdfp16 [i] -= rs1fp8 [2*i+1]+rs2fp8 [2*i]
rdfp16 [i] += rs1fp8 [2*i+1]*rs2fp8 [2*i+1]+rs1fp8 [2*i]*rs2fp8 [2*i]
rdfp16 [i] += rs1fp8 [2*i+1]*rs2fp8 [0]+rs1fp8 [2*i]*rs2fp8 [0]
rdfp16 [i] -= rs1fp8 [2*i+1]*rs2fp8 [2*i+1]+rs1fp8 [2*i]*rs2fp8 [2*i]
rdfp16 [i] -= rs1fp8 [2*i+1]*rs2fp8 [0]+rs1fp8 [2*i]*rs2fp8 [0]
rdfp32 [i] += rs1fp16 [2*i+1]+rs2fp16 [2*i]
rdfp32 [i] -= rs1fp16 [2*i+1]+rs2fp16 [2*i]
rdfp32 [i] += rs1fp16 [2*i+1]*rs2fp16 [2*i+1]+rs1fp16 [2*i]*rs2fp16 [2*i]
rdfp32 [i] += rs1fp16 [2*i+1]*rs2fp16 [0]+rs1fp16 [2*i]*rs2fp16 [0]
rdfp32 [i] -= rs1fp16 [2*i+1]*rs2fp16 [2*i+1]+rs1fp16 [2*i]*rs2fp16 [2*i]
rdfp32 [i] -= rs1fp16 [2*i+1]*rs2fp16 [0]+rs1fp16 [2*i]*rs2fp16 [0]
rdfp8 [i] += rs1fp8 [2*i+1]+rs1fp8 [2*i]
rdfp8 [i] -= rs1fp8 [2*i+1]+rs1fp8 [2*i]
rdfp16 [i] += rs1fp16 [2*i+1]+rs1fp16 [2*i]
rdfp16 [i] -= rs1fp16 [2*i+1]+rs1fp16 [2*i]
rdfp32 += rs1fp32 [1]+rs1fp32 [0]
rdfp32 -= rs1fp32 [1]+rs1fp32 [0]
Table 4.3: Overview of the MiniFloat-NN-Scalar ISA extension encodings (the Minifloat-NN-Vector
ISA extension does not use multiple encodings for the same instruction working on different data types,
as it selects the format based on the RISC-V vector selected element width (VSEW) [105] value). The
alternative formats – FP16alt and FP8alt – are distinguished by configuring a newly introduced 2-bit
FP control and status register called fmode (src is alt, dst is alt).
4.4. ARCHITECTURE
87
88
CHAPTER 4. DATA TYPE SPECIALIZATION
rs1 register is multiplied by a scalar value stored in the rs2 register.
This family of operations is conceived to ease the scaling phase during the gradient update, which follows matrix multiplications in NN
training.
4.4.7
MiniFloat-NN-Vector Cluster
B0
B1
64b
B2
64b
64b
B3
...
B15
...
64b
64b
Shared L1 Scratchpad Crossbar
64b
64b
64b
VRF
VRF
FPU FPU FPU FPU
Spatz-CC-0
L0 I$
FPU FPU FPU FPU
Spatz-CC-1
L0 I$
256b
Periph.
64b AXI Crossbar
64b
256b
Shared L1 I$
512b
512b AXI Crossbar
64b AXI
512b AXI
Figure 4.11: A cluster composed of two Snitch compute cores, each
one coupled with a Spatz vector co-processor containing four FPnew
extended with wDotp.
We integrate our wDotp unit in a Spatz-based compute cluster [97]
by replacing its FPUs with an instance of the enhanced FPU. The
Spatz-based cluster contains two Snitch integer cores coupled with two
Spatz vector units, each one including four FPUs and a 2 KiB vector
register file (32 512-bit registers). We show the Spatz-based cluster
in Fig. 4.11, where two Spatz-CCs, i.e., two Snitch cores enhanced
4.5. EXPERIMENTAL RESULTS
89
with Spatz vector accelerators, share a 128 KiB scratchpad memory
organized into 16 banks and an L1 instruction cache. The RISC-V
vector extension contains wFMA instructions and already supports
vector reductions. Moreover, thanks to the time-multiplexed nature
of vector instructions, selecting the upper or lower half of the inputs
to multiply is not problematic. However, wFMA operations still
underutilize the FPU output bandwidth. The 64-bit wide FPUs
included in the Spatz vector accelerator have three 64-bit inputs.
Two of them will be used only to provide 32 bits each, while the
third containing the accumulators will provide 64 bits, as shown in
Fig. 4.4. To fully process all the data that can be forwarded to the
FPU each cycle while using mixed-precision operations, we extend the
RISC-V vector ISA with a vector-vector and a vector-scalar wDotp
instruction called vfwdotp.vv and vfwdotp.vf, respectively. At
the same time, leveraging the new Vsum capabilities introduced with
the SIMD wDotp unit, vector reductions accumulating at lower than
FP64 precision can be accelerated by 2×. This is because, on the SIMD
wDotp unit, twice the number of FP additions can be executed each
cycle with respect to carrying them out on the SIMD FMA unit. We
then evaluate our vector extension by comparing the baseline Spatzbased cluster with the MiniFloat-NN-Vector Spatz-based cluster.
4.5
Experimental Results
We evaluate the area, performance, and energy efficiency of the proposed wDotp unit and the two enhanced compute clusters and compare them with related state-of-the-art (SoA) architectures.
4.5.1
wDotp Unit Results
Area and Timing
We synthesize the wDotp unit standalone in GlobalFoundries’
12 nm FinFET technology with Synopsys Fusion Compiler
2021.06. We target a relaxed clock frequency of 333 MHz without
pipeline stages in a worst-case corner (SS, 0.72 V, 125 ◦ C). Since two
cascaded wFMA modules can compute a dot product, we compare our
16-to-32-bit and 8-to-16-bit units with a set of two wFMA modules
90
CHAPTER 4. DATA TYPE SPECIALIZATION
Figure 4.12: Area comparison between wDotp units and sets of two
wFMA modules that, arranged in a cascade, can compute a non-fused
wDotp operation.
supporting the same formats. Note that the two cascaded wFMA
units perform a slightly different computation as they round twice and
do not mitigate potential adverse effects due to the non-associativity
of FP additions. Moreover, the cascaded wFMA units lack support for
the Vsum instructions. Since the second wFMA unit in the cascade
requires the output of the first as an input, in an implementation
with no pipeline registers, each wFMA instance is required to work at
667 MHz to ensure that the cascade will run at 333 MHz as the wDotp
unit. We synthesize the FPnew wFMA unit with such a constraint and
provide an area comparison in Fig. 4.12. The wDotp unit is around
30% smaller, and its critical path is around 30% shorter compared to
the cascaded wFMA units.
A 16-to-32-bit and an 8-to-16-bit wDotp unit, while capable of
computing twice the operations per cycle of a single 32-bit and a 16-bit
FMA, respectively, and still using the higher accumulation precision,
only occupy around 10% more area. The newly added rounding modes
4.5. EXPERIMENTAL RESULTS
91
Figure 4.13: Area breakdown of the 8-to-16-bit wDotp unit.
only required an LFSR and a comparator. In the case of 12 trailing
bits considered for the rounding decision and a 32-bit wide LFSR,
the area overhead amounted to around 4% on the wDotp unit and to
around 1% at a full FPU level.
Finally, we show the area breakdown of the 8-to-16-bit wDotp
unit in Fig. 4.13. Fig. 4.13 focuses on the combinatorial logic; depending on the frequency requirements, a number of pipeline levels
can be added by tuning a parameter at design time. At this level
of precision, the area spent for the multipliers is almost negligible,
amounting to only 2% of the total area. In an 8-to-16-bit wDotp
unit, the multiplier only operates on 4-bit inputs. In contrast, the
internal adders and shifters operate on inputs even wider than 30 bits.
This higher precision is needed because of the larger-precision format
used for the accumulation and to prevent precision losses in the two
additions contained in one wDotp operation. The three-term sorting
introduced to mitigate precision losses due to the non-associativity of
FP additions occupies 7% of the 8-to-16-bit wDotp unit. However,
even when removing the three-term sorting, two consecutive additions still require finding the maximum operand to shift the others
92
CHAPTER 4. DATA TYPE SPECIALIZATION
Accumulation of Small Increments on the wDotp Unit
Accumulated Value
10000
FP16 SR
FP32 RNE
FP16 RNE
8000
6000
4000
2000
0
0
2000
4000
6000
Number of Accumulations
8000
10000
Figure 4.14: Accumulation of 10000 FP16 increments uniformly
distributed in [0.1) on the wDotp unit when employing: (i) FP32
as accumulation precision and RNE; (ii) FP16 as accumulation
precision and RNE; (iii) FP16 as accumulation precision and SR. In
the latter case, we report an area delimited by the maximum and
minimum values obtained after running the same accumulation 10
times. Employing SR with an FP16 accumulator allows for overcoming
the saturation obtained when using RNE.
accordingly. Therefore, while the three-term sorting occupies 7% of
the area, the actual overhead introduced by the three-term sorting
is even lower. Handling FP arithmetic requires a complex control,
including input preprocessing, addend alignments, normalization and
rounding, which introduces a high overhead at such a low precision.
This control introduces a much milder overhead on traditional FP
hardware units, as in FP64 and FP32 FMA units, where the multiplier
occupies around 60% and 40% of the overall area.
Stochastic Rounding
To evaluate our stochastic rounding implementation, we accumulate
10 times the same 10000 FP16 increments uniformly distributed in
4.5. EXPERIMENTAL RESULTS
93
[0, 1) using an FP16 accumulator and SR. We compare the obtained
results with the ones generated by employing an FP16 or FP32 accumulator and RNE. We show the results in Fig. 4.14. For the FP16 case
using SR, we report an area delimited by the maximum and minimum
values obtained after running the same accumulation 10 times.
The FP16 execution using RNE saturates after around 4000 accumulations after the accumulator gets to 4096. At this point, the
increments are too small compared to the accumulator value and thus
always rounded out. By leveraging SR, the FP16 accumulator takes
the rounding decision based on a probability that depends on the
distance to the next representable FP value. In this way, when small
increments are added many times, they are not always systematically
rounded out, allowing the accumulator to overcome the saturation
points following the FP32 accumulator trend.
Performance, Energy Efficiency, and SoA Comparison
We summarize the results in terms of performance, area, and energy
efficiency in Table 4.4. The wDotp-enhanced FPU can compute two
16-to-32-bit and four 8-to-16-bit wDotp operations per cycle. Considering 1 FMA as 2 FLOP, as each wDotp computes the equivalent of 2 FMAs, the SIMD wDotp unit reaches 8 FLOP/cycle and
16 FLOP/cycle, respectively. At 1.26 GHz, when computing 8-to-16bit wDotp operations, it achieves 1631 GFLOPS/W at 20.2 GFLOPS.
We compare our enhanced FPU against its baseline counterpart,
FPnew, and two recent SoA architectures with low-precision support
in Table 4.4. The modules proposed by Mao et al. [117] and Zhang et
al. [116] compute SIMD FMAs or dot-product operations at different
levels of precision, 16-bit and higher, while FPnew also supports FP8.
However, none of them can work with FP8alt data. Our extended
FPU, thanks to the proposed wDotp SIMD unit, achieves the highest
energy efficiency among the selected mixed-precision FPUs. It outperforms the module developed by Zhang et al. [116] by 14.4×, and
the multiple-precision unit by Mao et al. [117] by 1.7×. Furthermore,
it provides 30% higher efficiency than FPnew working with FP8 data
and doubles its peak performance when using widening operations.
CHAPTER 4. DATA TYPE SPECIALIZATION
94
Table 4.4: Comparison of FPUs supporting low-precision formats (top four rows) and evaluation of the
MiniFloat-NN-Scalar and MiniFloat-NN-Vector extensions (bottom four rows).
0.049 mm2
0.019 mm2
Area
no
yes
DotP
4/8
8/8
4/8
8/8
8/16
16/16
FP8c
-/-
16/16
FP8altc
14.8 (FP8)
20.2 (exFP8)
[GFLOPS]
1245 (FP8)
1631 (exFP8)
[GFLOPS/W]
Throughputd Efficiencyd
1.26 GHz
Performance [FLOP/cycle]a
0.923 GHz
Mao et al. [117]b
90 nm
28 nm
1.0 V
1.0 V
0.667 GHz
1.43 GHz
0.191 mm2
0.013 mm2
no
yes
yes
yes
-/-
64/64
-/-
-/-
8/8
-/20
-/-
-/-
-/-
-/-
5.3 (FP16)
28.6 (FP16)
113 (FP16)
975 (FP16)
FP16altc FP16c
0.8 V
0.8 V
Technology Voltage Frequency
22 nm
12 nm
Design
FPnew [70]
wDotp FPU
Zhang et al. [116]b
0.66 mm2
0.52 mm2
0.40 mm2
0.44 mm2
yes
yes
32/64
64/64
32/64
64/128
64/64 128/128 128/128
66 (exFP8)f
135 (exFP8)
630 (exFP8)f
860 (exFP8)
-/1.08 GHz
-/1.08 GHz
-/0.8 V
80 (FP64)e
1 GHz
1.26 GHz
0.8 V
575 (exFP8)
0.8 V
0.8 V
12 nm
15 (FP64)
22 nm
12 nm
12 nm
128 (exFP8)
MiniFloat-NN Snitch
64/64 128/128 128/128
Snitch [69]
MiniFloat-NN Spatz
64/128
Spatz [97]
b
Only FMA and wDotp units. No support for other operation groups as cast and comparison.
We report only the performance related to low-precision formats. All the designs in the table support FP32 and FP64 as well.
Peak throughput and energy efficiency reported for the standalone FP modules (top four rows). Throughput and efficiency while computing a matrix
multiplication reported for the clusters (bottom four rows).
a
d
The Snitch cluster in [69] obtained 104 GFLOPS/W for FP32 matrix multiplications. As it was evaluated without SIMD FMA support, the increased
efficiency derives from the reduced activity when performing FP32 FMAs on an FP64 FMA datapath.
Performance reported in the form widening/non-widening, and considering 1 wDotp = 4 FLOP, and 1 FMA = 2 FLOP.
e
Results for 8-to-16-bit wFMA-based kernels are reported for comparison purposes, as current NN training algorithms require at least a 16-bit format
for the accumulation.
c
f
4.5. EXPERIMENTAL RESULTS
95
Figure 4.15: Area breakdown of the extended FPU.
4.5.2
MiniFloat-NN-Scalar Cluster Results
Area and Timing
We use Synopsys Fusion Compiler 2021.06 to synthesize, place
and route the MiniFloat-NN-Scalar Snitch cluster. We constrain the
design for 950 MHz in the worst-case scenario. To achieve the targeted frequency, we insert three pipeline stages for the DOTP and
ADDMUL, two for the CAST, and one for the COMP operation group
to the enhanced FPU. The implemented cluster occupies 4.3 MGE and
achieves 1.26 GHz in a typical corner (0.8 V, 25 ◦ C). The wDotp SIMD
module occupies 44.5 kGE, amounting to 27% of the total area of the
extended FPU, which occupies 165 kGE (Fig. 4.15). The support for
low-precision FP formats and wDotp only adds less than 15% of area
overhead at the cluster level.
Performance
To assess the performance of our MiniFloat-NN-Scalar cluster, we
implemented a set of matrix multiplication kernels using the FMA
96
CHAPTER 4. DATA TYPE SPECIALIZATION
Figure 4.16: MiniFloat-NN-Scalar Snitch cluster performance:
FLOP/cycle per FP format for different matrix multiplication kernels.
and wDotp instructions for different FP formats, as well as the SSR
and FREP custom ISA extensions [69].
We consider only matrix multiplication sizes that fit entirely into
the 128 KiB scratchpad memory. In this way, we provide results independent of the memory hierarchy of the hosting cluster, concentrating
only on the evaluation of our extensions. However, larger matrix
multiplication sizes could be addressed by such clusters by tiling the
matrices and double buffering the data in the scratchpad memory,
moving data in and out of the cluster with a DMA engine while
computing on another tile. This efficiently uses the cluster accelerator
even for larger matrix sizes at the price of some performance penalties
when the DMA engine and the cluster cores try to access the same
memory bank.
The considered kernels are compiled with an extended LLVM-12
compiler using intrinsics for our new instructions. We run the kernels
on the enhanced Snitch cluster and measure the execution cycles in a
cycle-accurate RTL simulation using Mentor Questasim.
4.5. EXPERIMENTAL RESULTS
97
Moving to lower precision allows considerable memory footprint
reductions. When leveraging the proposed wDotp instructions, such
benefits are enabled even when the accumulation is performed at
higher precision. We can fit up to 128×256 matrix multiplications into
the scratchpad memory when leveraging the new 8-to-16-bit wDotp
support, while for FP16-only and FP64, the maximum kernel size that
fits in the memory is 128 × 128 and 64 × 64, respectively.
We report the achieved performance in Fig. 4.16. The MiniFloatNN-Scalar extension enables a peak throughput of 8 FLOP/cycle per
FPU (64 FLOP/cycle per cluster) when using 16-to-32-bit operations
and 16 FLOP/cycle per FPU (128 FLOP/cycle per cluster) when using
8-to-16-bit operations. That doubles the peak performance achievable
with wFMAs, whose SIMD implementation would suffer from a suboptimal register file usage. Moreover, it doubles the peak performance
reachable with non-widening FMAs computing in the dst fmt, and
matches the throughput of the non-widening FMAs computing in the
src fmt while providing higher precision.
The 128 × 256 FP8-to-FP16 matrix multiplication achieves 1.96×
and 7.2× the FLOP/cycle of the 128 × 128 FP16-to-FP32 matrix multiplications and the 64 × 64 FP64 matrix multiplications, respectively.
The deviation from the ideal 8× speedup with respect to the FP64
kernel is due to the initialization of the accumulators to zero, the
additional instructions needed to reduce the final results packed in
the 64-bit registers, and the conversions used to bring the results back
to the input precision upon completing the computation of a matrix
element. In this context, wDotp instructions with four operands –
three source registers and one destination register – would help to
reduce the around 10% performance loss with respect to the FP64
kernel. In this case, the initialization of the accumulators to zero
would be avoided. However, four-operand instructions can be mapped
only to a very narrow encoding space, which in the Snitch case is
already limited by its custom extensions and the replication of the
FP instructions for narrower formats. A similar issue prevented from
adding additional wDotp instructions with three operands, tying the
accumulator input to zero.
98
CHAPTER 4. DATA TYPE SPECIALIZATION
Power and Energy Efficiency
We determine the power consumption of the cluster with Synopsys
PrimePower under typical conditions (0.8 V, 25 ◦ C) based on the
switching activities obtained from a post-layout simulation with Mentor Questasim. We assess the power for a wDotp-based 128 × 256
FP8-to-FP16 matrix multiplication kernel at a frequency of 1.26 GHz.
The MiniFloat-NN-Scalar cluster reaches 128 GFLOPS, consuming
224 mW, thus achieving 575 GFLOPS/W (where 1 wDotp is counted
as 4 FLOP). The extended cluster working on FP8-to-FP16 kernels
reaches 7.2× the efficiency of the native Snitch system computing in
double precision. In the lower half of Table 4.4, we summarize and
compare the system-level results of our MiniFloat-NN-Scalar RISC-V
extension with the native baseline Snitch cluster.
4.5.3
MiniFloat-NN-Vector Cluster Results
Area and Timing
We synthesize, place and route the baseline and the MiniFloatNN-Vector Spatz-based clusters with Synopsys Fusion Compiler
2021.06, targeting 950 MHz in the worst-case scenario. The FPU
instance matches the one used in the Snitch-NN-scalar Snitch cluster.
The MiniFloat-NN-Vector Spatz-based cluster containing 8 FPUs
occupies 3.6 MGE, 10% more than a Spatz-based cluster with no
wDotp capabilities.
We compared our implementation to the potential alternative solution introduced in Section 4.3. To enable the desired 2× speedup
when moving from FP32 to FP16 kernels with accumulation in FP32,
instead of integrating the proposed wDotp SIMD unit, the following
modifications could be implemented: (i) adding an extra read and an
extra write port to the register file, (ii) adding an extra input and
an extra output to the FPU, and (iii) adding two extra 16-to-32-bit
and two extra 8-to-16-bit wFMA units. This would allow reaching the
same number of operations per cycle that can be achieved with the
proposed wDotp unit. However, it also stresses the vector register file,
leading to long paths, which reduce the maximum operating frequency
by around 10%. At the same time, while the extra functional units
introduce a lower area overhead than the wDotp SIMD module, the
4.5. EXPERIMENTAL RESULTS
99
Figure 4.17: MiniFloat-NN-Vector Spatz cluster performance:
FLOP/cycle per FP format for different matrix multiplication sizes.
extra area associated with the extra register file ports and the related
higher timing pressure lead to comparable area results with respect
to the proposed wDotp-enhanced Spatz cluster.
Performance
To evaluate our extensions to the Spatz-based cluster, we run a set
of matrix multiplication kernels using RISC-V vector instructions and
targeting different levels of precision, as well as wFMA and wDotpbased mixed-precision setups. We report the performance results in
Fig. 4.17.
By exploiting the wDotp units, the MiniFloat-NN-Vector Spatzbased cluster can double the computation per cycle with respect to
using wFMA operations. At the same time, fewer vector register
file reads are necessary, as the information that can be read in one
cycle is fully processed every cycle. When using wFMAs, only half of
that data can be processed, requiring reading the same inputs twice.
Furthermore, to avoid the area overhead that ad-hoc modules would
introduce, wFMA operations are computed on FMA units. Therefore, the smallest FMA unit that can be used to compute a wFMA
100
CHAPTER 4. DATA TYPE SPECIALIZATION
operation is the one sized to the destination precision, i.e., a 32-bit
FMA module in the case of 16-to-32-bit wFMA. All this results in a
linear speedup when reducing the precision and using FMA or wDotp
instructions, while wFMA-based kernels match the performance of
the FMA-based applications working at the accumulation precision
of the wFMA-based workload. An 8-to-16-bit wDotp-based matrix
multiplication achieves 3.97× and 7.95× the performance of an FP32
and FP64 kernel, respectively, while an 8-to-16-bit wFMA kernel would
only improve the FMA counterpart by 1.94× and 3.91×.
The Spatz-based cluster achieves up to 98% FPU utilization (the
FPU utilization metric is defined in Section 4.4.6), thus reaching
higher utilization than the Snitch cluster, which peaks at around
90%. Moreover, the Snitch kernels suffer more from the overhead
introduced by moving to lower-precision SIMD. That leads to around
80% of the FLOP/cycle theoretically achievable in an 8-to-16-bit
matrix multiplication while keeping the FPU busy with computational
and non-computational instructions for 88% of the execution time. On
the other hand, thanks to its scalar ISA, the Snitch cluster maintains
higher flexibility, which can be exploited in workloads less regular than
matrix multiplications.
Power and Energy Efficiency
We report the power consumption and energy efficiency results in
Fig. 4.18 and Fig. 4.19. In such an efficient and compact cluster, little
overhead is added to the fundamental computation. The FPU power
consumption can, therefore, amount to more than 50% of the power
consumed at the cluster level. The efficiency of such a module is thus
crucial.
Since lower-precision vector instructions reuse larger-precision
units for a higher area efficiency, the power consumed by the Spatzbased cluster does not decrease when moving to lower-precision
workloads based on the same fundamental operation. For example, a
vector FP16 FMA on the 64-bit FPU computes four FP16 FMAs each
cycle. Two of these will be computed on FP16 FMA units, one on the
FP32 FMA module and one on the FP64 datapath. While computing
FP16 operations on the FP64 datapath consumes less power than
computing FP64 operations on the FP64 datapath, a vectorial FP16
4.5. EXPERIMENTAL RESULTS
101
Figure 4.18: MiniFloat-NN-Vector Spatz results at 1.08 GHz: Power
consumption in mW for different matrix multiplication sizes.
execution does not consume less power than the FP64 one because of
the extra contribution from the FP16 and FP32 units, it does not result
in a power decrease. Lower power results can be achieved by inserting
more low-precision units to avoid reusing larger-precision datapaths
for the vectorial low-precision operations. However, this would add a
30% area overhead to the FPU in the case of units supporting multiple
formats only if these are expressed with the same number of bits, such
as FP16 and FP16ALT, and even up to 80% overhead in the case only
one format is supported.
When moving to kernels built on different fundamental operations, such as wFMA and wDotp, larger power differences can be
observed. An FP16-to-FP32 wFMA-based kernel consumes around
40% less power than the FP16 FMA-based application but at half the
performance and 36% less power than the FP32 FMA-based kernel
at the same performance. When moving to an FP16-to-FP32 wDotpbased kernel, the Spatz-based cluster consumes around 30% higher
power than the FP16-to-FP32 wFMA-based workload, reaching twice
the performance, 15% lower power than the FP32 FMA-based kernel,
at twice the performance, and 20% lower power than the FP16 FMAbased application achieving the same performance. The latter result
102
CHAPTER 4. DATA TYPE SPECIALIZATION
Figure 4.19: MiniFloat-NN-Vector Spatz results at 1.08 GHz: Energy
efficiency in GFLOPS/W per FP format for different matrix multiplication sizes.
is due to the FP16-to-FP32 wDotp having dedicated units and not
needing to compute part of the vector operations on higher precision
datapaths as an FP16 FMA on an FP64 datapath. This suggests that
higher superlinear energy efficiency improvement can be reached when
designing units fully specialized on a single format or combination of
formats, as also shown by Mach et al. [70].
On a full matrix multiplication kernel, when comparing 16-to32-bit wFMA-based and wDotp-based kernels, the latter introduces
a 1.96× speedup while increasing the energy efficiency by around
50%. When considering 8-to-16-bit wFMA-based and wDotp-based
kernels, the efficiency improvement is 36% at twice the performance. At 1.08 GHz, the MiniFloat-NN-Vector Spatz-based cluster
consumes 156 mW when computing the 8-to-16-bit workload. The
8-to-16-bit wDotp-based kernel achieves the highest efficiency point,
860 GFLOPS/W, improving the FP32 performance by 3.97× and
efficiency by 4.5×, and the FP64 workload performance by 7.95× and
efficiency by 9×.
4.6. FROM OPTIMIZED KERNELS TO HIGH-LEVEL APPLICATIONS103
4.6
From Optimized Kernels to HighLevel Applications
In this chapter, we have concentrated on hardware and low-level
kernels designed to achieve higher efficiency when low and mixedprecision FP formats can be used. We have limited our evaluation of
the proposed extended acceleration clusters to matrix-multiplication
kernels whose sizes can entirely fit in the available scratchpad memory.
The reason for this choice is that such a low-level kernel is the key
basic block for many high-level applications. Therefore, optimizing it
directly correlates to higher-level benefits. As covered in Section 4.5.2,
to address wider kernel sizes, a DMA engine can be used to move
data in and out of the acceleration, double buffering the data while
the cores are computing on another matrix tile. To enhance the
system’s programmability, the acceleration cluster can be coupled to a
Linux-capable host, offloading matrix multiplication to one or multiple
clusters where the computation is accelerated [104]. Furthermore,
parallel programming frameworks, such as the Data-Centric Parallel
Programming (DaCe) tool proposed by Ben-Nun et al. [127], can help
map high-level Python/NumPy applications to the available hardware
resources.
4.7
Silicon Implementations
4.7.1
Eclipse
To validate our MiniFloat-NN-Scalar extension and the newly proposed wDotp unit, we have fabricated a silicon prototype in a 65 nm
technology. In this case, the extended FPU was scaled down to a
32-bit instance and integrated into a smaller single-core MCU [96].
We show the resulting chip, named Eclipse, in Fig. 4.20. The overall
system, similarly to the one presented in Section 3.5.1, includes:
• A CV32E40P core [40] enhanced with a 32-bit version of
MiniFloat-NN-Scalar extension and with an accelerator-agnostic
interface (core-v-xif-v0.1.0) 6 ;
6 https://github.com/openhwgroup/core-v-xif
104
CHAPTER 4. DATA TYPE SPECIALIZATION
Figure 4.20: Eclipse chip.
• The FPU presented in this chapter, attached to the acceleratoragnostic interface and reduced to a 32-bit version with wDotp
and SR support (as a consequence, the number of wDotp units
is halved in the DOTP operation group block);
• An autonomous I/O subsystem including a wide set of peripherals and an I/O DMA engine;
• a 128 KiB of memory organized in 4 banks and shared among
the core, the accelerator, and the peripherals.
The Eclipse chip has been tested using an Advantest SoC V93000
automatic test equipment. Eclipse was used to validate on silicon the
extended FPU and the newly designed wDotp unit.
4.7. SILICON IMPLEMENTATIONS
105
Figure 4.21: Occamy chiplet.
4.7.2
Occamy
The MiniFloat extension has also been included in Occamy, a much
larger project with more than 20 contributors involved, aiming at
exploring a scaled-out version of the Snitch cluster extended with the
MiniFloat-NN-Scalar extension and an extension accelerating sparse
and stencil workloads [41].
Occamy is a 2.5D integrated dual-chiplet system. Each chiplet has
been implemented in a 12 nm technology and is 10.5 mm × 6.95 mm
wide. As shown in Fig. 4.21, it contains:
• A high-bandwidth memory (HBM2e) controller;
• A double-data-rate die-to-die link to communicate with the
other chiplet;
• A Linux-capable host (CVA6);
• 24 Snitch clusters including 8 compute cores enhanced with ISA
extensions for MiniFloat and sparse computations and a ninth
106
CHAPTER 4. DATA TYPE SPECIALIZATION
Figure 4.22: A passive silicon interposer carries two Occamy chiplets,
each with a private HBM2e.
Figure 4.23: Dual-chiplet Occamy system.
4.8. CONCLUSION
107
core used as a DMA controller. The Snitch clusters are organized
into six groups of four clusters.
The full Occamy system includes two Occamy chiplet, each one
with a private 16 GiB HBM2e, as shown in Fig. 4.22 and Fig. 4.23.
The resulting system has been tested using an Advantest SoC
V93000 automatic test equipment. Occamy reaches at 1 GHz a peak
performance of 0.768 TFLOPS for FP64, 1.536 TFLOPS for FP32,
3.072 TFLOPS for 16-bit FP formats, and 6.144 TFLOPS for 8-bit
FP data types.
4.7.3
Carfield
The Minifloat-NN-Vector extension has been included in another
ambitious project with more than 20 contributors involved, named
Carfield. We show the Carfield chip in Fig. 4.24, which includes:
• A dual-core Linux-capable host enhanced with hypervisor support and an ISA extension protecting from timing channels;
• A security island based on the OpenTitan project [128], which
serves as a hardware root-of-trust;
• A safety island with three cores operating in lockstep and able
to identify and recover single faults;
• A multi-core accelerator based on the PULP cluster and specialized for integer computations;
• The Spatz cluster enhanced with wDotp and MiniFloat support.
Carfield has been implemented in a 16 nm technology.
A reduced version of the Spatz cluster including a single Spatz
vector processor with wDotp and MiniFloat support has also been
included in the Maestro chip presented in Section 3.5.2.
4.8
Conclusion
In this chapter, we have investigated the benefits offered by data
specialization and related microarchitectural enhancements. We have
108
CHAPTER 4. DATA TYPE SPECIALIZATION
Figure 4.24: Carfield chip.
presented two RISC-V ISA extensions, one for scalar and one for
vector cores. The extensions are based on mixed-precision dot product
operations carried out on a dedicated hardware unit supporting two 8bit and two 16-bit input formats and accumulating at higher precision.
We have integrated our unit into an open-source FPU and evaluated
the extensions on a cluster of vector cores and a cluster of scalar
cores. We have shown that mixed-precision dot-product operations
offer many benefits over mixed-precision FMAs, as they optimize the
utilization of the register file and the utilization of a general-purpose
FPU bandwidth. We have implemented a cluster of scalar cores and a
cluster of vector cores, both containing eight of the extended FPUs in
a 12 nm technology. The first reaches 128 GFLOPS of throughput
and 575 GFLOPS/W of energy efficiency when computing FP8-toFP16 matrix multiplications at 1.26 GHz, 0.8 V. The second achieves
4.8. CONCLUSION
109
135 GFLOPS of throughput and 860 GFLOPS/W of energy efficiency
when calculating on the same data types at 1.08 GHz, 0.8 V. That
results in 1.93× higher efficiency than computing on FP16-to-FP32
and 3.97× the performance and 4.5× the energy efficiency of the same
cluster computing on FP32. The solutions proposed in this chapter
have been implemented in various silicon test chips.
Chapter 5
Mitigating Data Type
Specialization
Limitations: Extending
RISC-V for Efficient
Overflow Recovery
In the previous chapter, we have investigated low and mixed-precision
solutions to improve the energy efficiency and reduce the memory
footprint. While these narrow-bitwidth formats offer great benefits,
they also introduce various challenges. One example, stagnation,
has been provided in Chapter 4 and mitigated through stochastic
rounding. In this chapter, we focus on a further limitation. The
narrow exponent bitwidths of low-precision formats result in smaller
representable ranges, thereby increasing the probability of overflow.
Although overflowing values are often saturated in NN applications,
a large set of applications is currently being revised for low precision,
and saturation would not be a feasible solution in every context, such
111
112
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
as in safety-critical scenarios. We, therefore, investigate RISC-V extensions for low-cost overflow detection and efficient overflow recovery
leveraging a mixed-precision ISA.
The following sections have been published in a slightly different form in the IEEE International Conference on Computer Design
(ICCD) 1 .
5.1
Introduction
Low and mixed-precision arithmetic is becoming increasingly
widespread, as it reduces energy and memory footprint for many
different classes of applications [20, 53, 129]. In particular, mixed
precision is key in facing the staggering memory and compute
requirements introduced by the ever-larger NN models [5, 29]. This
motivated increasing investments in hardware architectures efficiently
supporting these new data types and operations, leading to many
commercial platforms targeting now low and mixed-precision FP
computing [17,31,110]. Hardware support has a positive feedback and
is currently promoting further research efforts aiming at exploiting
low-precision formats for applications beyond NN algorithms [34, 35].
Low-precision formats, however, imply narrow exponent bitwidths,
thus small dynamic ranges. For this reason, such data types are
generally less robust to overflow [130]. While in NN algorithms,
overflowing values are often saturated to the maximum representable
number [27], this would not be acceptable in every application field.
Considering the higher probability of overflowing when employing
narrow data types, the destructive consequences of overflowing, and
the large spectrum of applications being revised for low and mixed
precision, several applications would benefit from an agile and low-cost
overflow recovery mechanism.
Various works have investigated how to adapt the precision at
compilation time to meet the application requirements and minimize
the energy and memory footprint [131, 132]. However, such methods
1 ©2024 IEEE. Reprinted, with permission, from L. Bertaccini, S. Shen,
T. Hoefler, and L. Benini, “Extending RISC-V for efficient overflow recovery in
mixed-precision computations,” in 2024 IEEE 42nd International Conference on
Computer Design (ICCD). IEEE, 2024, pp. 268–275.
5.1. INTRODUCTION
113
exploit static information, lacking the capability to react to unforeseen
events, which would be necessary, for example, in safety-critical applications. In addition, these solutions would need to be conservative to
cover cases that only rarely happen. An online recovery mechanism
could instead leverage low precision for the common scenarios and
recompute at higher precision only if deemed necessary after having
computed in low precision.
Whenever an overflow occurs, as per IEEE-754 floating-point standard, an exception is signaled through a status flag. Several reactions
to overflow have been studied [133–135]: (i) producing an INF, (ii)
terminating the execution, and (iii) trapping and handling the exception. The first two options compromise a successful computation,
while the latter can recover from such a destructive event. This
can be implemented, for example, by recomputing with an extended
range [135], i.e., by using a wider format whenever supported by the
hardware or emulating it in software. In this study, we concentrate
on this approach and show the benefits enabled in this context by
mixed-precision ISAs.
Not all ISAs, however, natively trap FP exceptions. In the case
of RISC-V [36], FP exceptions are only stored in an floating-point
control and status register (FCSR). Therefore, to detect an overflow,
this FCSR should be explicitly polled in software, and the overflow
flag bit appropriately masked. This process, which is needed to detect
overflow to then react to it and consequently protect the current
application, can add a significant overhead.
In this chapter, we focus on increasing the resilience of clusters
of RISC-V cores with low and mixed-precision FP support and their
target applications to overflow. In particular, we devise an agile and
low-cost online recovery scheme and provide the following contributions:
1. We extend an open-source cluster of RISC-V cores2 supporting low
and mixed-precision FP [53] to minimize the overflow detection
overhead and optimize the overflow recovery routine. The former
is achieved by adding new branch instructions depending on the
FP overflow flag. The latter exploits new instructions computing
on 8-bit inputs and accumulating in FP32.
2 https://github.com/pulp-platform/snitch cluster
114
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
2. We implement an online recovery scheme leveraging the supported
mixed-precision ISA and evaluate it at an increasing probability of
overflowing. Our solution adds less than 1% of hardware overhead
to the RISC-V core and enables less than 2% of performance overhead in the absence of overflow in a 128×128 matrix multiplication.
Supporting overflow detection and recovery allows for running the
majority of the kernel in low precision and tuning the accumulation
precision at run-time only if necessary. In the case of a single overflow, this only adds an overall 3% performance penalty with respect
to a fragile baseline mixed-precision computation while maintaining
its storage and performance benefits with respect to a full-precision
baseline.
5.2
Related work
5.2.1
Low and Mixed Precision
The energy and memory footprint benefits introduced by low and
mixed-precision arithmetic [27, 29, 109, 136] made larger and larger
NN models feasible. For this reason, many commercial hardware
products nowadays support a large set of FP data types, down to
8-bit FP formats [17, 31, 110]. The NVIDIA H100 GPU [17], for
example, provides support for FP64, FP32, TF32, FP16, BF16, and
two FP8 formats. This allows choosing the data type that best fits
the targeted application, minimizing the energy consumption and the
memory footprint.
Very-low-precision formats, however, necessarily face a tough
tradeoff between the number of exponent and mantissa bits, thus
between the representable range and the available precision. For this
reason, 8-bit formats [137], which only have 2 or 3 bits dedicated to
the mantissa field, are not a good fit for long accumulations. In this
case, they are instead used in mixed-precision operations where the
accumulation is performed using a higher-precision data type, such as
FP16 or FP32.
115
5.2. RELATED WORK
Table 5.1: Different FP formats and their maximum values.
Format
FP64
FP32
BF16
FP16
FP8p3*
FP8p4
*
(Exponent, Mantissa)
Maximum Value
Bitwidth
(11, 52)
( 8, 23)
( 8, 7)
( 5, 10)
( 5, 2)
( 4, 3)
1.8 × 10308
3.4 × 1038
3.4 × 1038
65504
49152
224
In this chapter, we call FP8 the FP8p3 format.
At the same time, reduced exponent bitwidths, such as the one
provided by FP16, translate to narrow representable ranges (see Table 5.1), increasing the overflow probability. In NN algorithms, overflowing values are often saturated to the maximum representable number [27]. However, low-precision FP data types are more and more
often applied to different application scenarios, such as audio processing [34] and climate modeling [35]. This could soon apply even to
safety-critical applications, where, however, neither saturating overflowing values nor producing an INF can be acceptable.
In this chapter, we concentrate on a highly efficient generalpurpose open-source cluster of RISC-V cores supporting MiniFloat
formats [53] and propose hardware/software extensions for agile and
low-cost overflow recovery. In particular, we focus on an architecture
that only provides FP16 accumulation for mixed-precision operations
on FP8 inputs. While some commercial architectures, such as the
Tesla Dojo [110] and Intel Gaudi 3 [31], provide FP32 accumulation
support when operating on FP8 inputs, other implementations, like
the IBM AI chip [118], only provide FP16 accumulation to avoid the
expensive FP32 accumulation [118]. The architecture considered in
this study follows a similar approach, supporting FP16 accumulation
for 8-bit inputs and FP32 accumulation for 16-bit inputs.
Another solution is provided by the NVIDIA Hopper transformer
engine, which supports both FP16 and FP32 accumulation precision
116
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
when operating on FP8 inputs but achieves half the performance when
accumulating in FP32 [138]. Similarly to the NVIDIA solution, we
add FP32 accumulation to the open-source cluster of RISC-V cores
considered for this study at half the performance. This has negligible
impact on the architecture, as it leverages pre-existing hardware for
the computation. We then exploit these new instructions to recompute overflowing portions of low-level kernels when accumulating in
FP16. The same recovery scheme could be applied to an architecture
providing support for FP8 operations with FP32 accumulation achieving the same performance as when accumulating in FP16 to maximize
its energy efficiency. The accumulation precision has a high impact on
the FP dot-product engines when processing FP8 inputs [139]. In this
case, the input mantissae only use 3 or 4 bits (including the hidden
one), thereby requiring minimal area for the related multipliers [53].
A large portion of the FP8 engine is then dedicated to the adders and
shifters [53, 139], whose bitwidths are affected by the accumulation
precision.
5.2.2
Overflow Exception and Recovery Routines
The IEEE-754 standard defines five exceptions that should be signaled through a status flag when they arise. One of these flags is
dedicated to overflow. Upon detecting an overflow, the system can
take different actions. For example, it can respond by terminating
the execution or producing an INF, which often turns into a NaN in
following calculations. Both these solutions compromise a successful
computation, which might not be tolerable in safety-critical applications. Alternatively, the exception can be trapped and processed
by a trap handler. Several approaches to handle overflow traps have
been proposed [133–135]: (i) exponent wrapping, (ii) scaling, and
(iii) recomputing with extended range.
Exponent wrapping [133, 134] produces a result upon overflow,
whose exponent is wrapped around. This is equivalent to scaling the
5.2. RELATED WORK
117
result by a factor of two. By counting how many times this situation occurred, the correct result can be obtained from the exponentwrapped one. This method is particularly useful in extended products
and quotients [133] like:
Q=
(a0 + b0 )(a1 + b1 )(a2 + b2 )...(an + bn )
(c0 + d0 )(c1 + d1 )(c2 + d2 )...(cn + dn )
(5.1)
where intermediate results overflow while the final result would be in
the representable range. It would, however, be problematic in long
accumulations as, during the computation of dot products, a nonexponent-wrapped value could be summed to an exponent-wrapped
one.
Scaling [135] requires examining the inputs of the computation
to then pre-adjust them so that all intermediate results will be in
the representable range. The final result can then be obtained by
adjusting the scaled result. However, using this method as a reaction
to an unexpected overflow would produce a large overhead since it
requires deriving some statistics on the inputs.
Recomputing with extended range [135] uses a wider format whenever supported by the hardware or emulates the extended range in
software. Considering the wide set of FP data types supported by
today’s hardware architectures, this approach represents an attractive
solution for overflow recovery. It is simpler than scaling, as it does
not need any information on the inputs and only requires rolling back
and recomputing at higher precision.
However, not all ISAs natively trap FP overflow exceptions.
RISC-V, for example, only signals overflows by setting the overflow bit
flag into an FCSR. Detecting overflow then requires explicitly polling
this register in software, potentially adding a non-negligible overhead.
Not trapping FP exceptions is also not unusual in systems specialized
for intense linear algebra computations, such as the open-source
MiniFloat Snitch cluster [53,69] we consider in this study and NVIDIA
GPUs. Li et al. [140] conceived an FP detection scheme for NVIDIA
GPUs, checking, at runtime, the resulting mantissa and exponent after
each instruction. However, this added a large performance penalty,
producing, in some cases, even more than a 5× slowdown.
In this chapter, we propose hardware extensions to minimize the
overhead required for overflow detection on RISC-V cores and to
118
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
optimize the overflow recovery routine by leveraging mixed-precision
instructions when recomputing with an extended range. Similarly to
the scheme proposed by Menon et al. [141], we roll back to a predefined
checkpoint to recover from an exception. We, however, specialize and
optimize the recovery routine for FP overflow exceptions. We concentrate on low and mixed-precision linear algebra kernels as critical basic
blocks for data processing. High-level applications could then be built
leveraging these overflow-protected low-level kernels. In particular, we
focus on applications including series of accumulations and provide a
detailed discussion and evaluation for matrix multiplications.
5.3
Hardware and Software Baseline
5.3.1
MiniFloat Snitch Cluster
We build our study upon the open-source MiniFloat Snitch cluster [53, 69]. In such a cluster, 8 RISC-V compute cores and one
DMA core share an L1 instruction cache and 128 KiB of scratchpad
memory organized in 32 banks. The Snitch core is composed of a
tiny 32-bit integer core coupled with an open-source 64-bit multiformat FPU3 [53, 70]. We show the block diagram of the Snitch FPU
in Fig. 5.1 and the architecture of the MiniFloat Snitch cluster in
Fig. 5.2.
The ISA of the compute cores has been extended to support a wide
set of FP formats (FP64, FP32, FP16, BF16, and two FP8 data types),
SIMD three-term FP additions and short wDotp instructions [53]
working on 8 or 16-bit FP inputs and accumulating with 16 or 32 bits,
respectively. The FPU is organized into hardware blocks dedicated
to specific operation group blocks. Each of them includes multiple
instances of the same modules tailored to different precision levels and
is therefore capable of carrying out SIMD operations on several FP formats (see Fig. 5.1). To limit the hardware overhead, higher-precision
units are reused for lower-precision computations when calculating
SIMD operations.
Furthermore, two extra custom extensions [69], SSR and FREP,
have been added to minimize the control overhead and keep the FPU
3 https://github.com/pulp-platform/cvfpu
16 16 16
64
16-bit
FMA
Packing
64
32 32 32
32-bit
FMA
64
Unpacking
64
8
8-bit FMA
8 8
ADDMUL
Operation
group
FPU
64
64
64
CAST
Operation
Group
64
Round-robin output arbitration
COMP
Operation
Group
Operands distribution
DOTP
Operation
Group
64
64
64
8 8
64
8 16
8-to-16
wDotp
Packing
a16 × b16 +
c16 × d16 + e32
16-to-32
wDotp
8
Unpacking
16 16 16 16 32
DOTP
Figure 5.1: Block diagram of the 64-bit SIMD FPU coupled to the Snitch compute cores, with a zoom
on the SIMD dot-product unit containing two 8-to-16-bit and two 16-to-32-bit wDotp units. The latter
are also used for lower-precision 8-to-16-bit computations. Therefore, up to two 16-to-32-bit wDotp
operations or four 8-to-16-bit wDotp operations can be computed per cycle. A similar approach is taken
on the ADDMUL operation group block, where non-mixed-precision SIMD FMAs are carried out.
64-bit
FMA
64 64 64
ADDMUL
5.3. HARDWARE AND SOFTWARE BASELINE
119
120
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
SB 1
SB 0
B0
64b
...
...
B8
64b
SuperBank 3
B23 ... B31
SB 2
...
...
B16
B0 ...
...
64b
...
64b
64b
512b
Scratchpad Interconnect
Periph.
...
64b AXI Crossbar
64b
SSR
FPU
Core 0
L0 I$
64b
SSR
FPU
Core 1
L0 I$
64b
...
64b
SSR
FPU
Core 7
L0 I$
DMA
Core 8
L0 I$
Shared L1 I$
512b AXI Crossbar
Figure 5.2: MiniFloat Snitch cluster architecture: eight compute cores
and a DMA core share 32 banks of scratchpad memory. Each 32bit RISC-V core is coupled with a 64-bit SIMD multi-format FPU
supporting mixed-precision operations.
5.3. HARDWARE AND SOFTWARE BASELINE
121
busy for most of the time. The SSR [142] extension implicitly maps
regular memory accesses to fixed FP registers. This allows Snitch
cores to directly stream data from memory, avoiding explicit loads
and instructions to update the addresses from which to load new
data. The FREP extension allows buffering and repeating a sequence
of FPU instructions, thereby reducing the pressure on the shared L1
instruction cache and preventing loop handling overhead. Neither
FREP nor SSRs support exceptions and interrupts during their operation. Having very limited exception support is, however, not unusual
for cores highly specialized in number crunching. This is also the case,
for example, for GPUs [140, 143].
This cluster represents an attractive starting point for our study
since it is open-source and provides multi-format and mixed-precision
support.
5.3.2
SSR and FREP-Based MiniFloat Matrix
Multiplication
In this study, we focus on low and mixed-precision linear algebra
kernels as these are one of the main critical blocks for data processing. In particular, we consider a matrix multiplication with input
and output matrices expressed in FP8 and the intermediate results,
i.e., the partial accumulations, performed in FP16. This application
leverages the MiniFloat extension [53], as well as the native SSR and
FREP extensions introduced in Section 5.3.1. Similar considerations
would apply to other basic linear algebra kernels, including series of
accumulations, such as dot products and convolutions.
We show a pseudo-code implementation of this kernel in Fig. 5.3.
Three nested loops carry out the computation, where the innermost
is implicit, as handled directly in hardware by the FREP. The FREP
buffers and repeats the FP instructions that form the innermost loop.
Such a loop only contains wDotp instructions, as the inputs of the
wDotp operations are provided by the SSRs. In this specific case,
the innermost loop is unrolled eight times to avoid read-after-write
dependencies (due to the FPU being pipelined) and to produce eight
results, which occupying 8 bits each will be packed in a 64-bit value
and stored in the memory every iteration of the middle loop. The
middle loop also includes pre-processing and post-processing, such as
122
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
<program SSR>
for (int m = 0; m < M; m++) {
for (int n = 0; n < N/8; n++) {
<initialize accumulators to zero>
<program FREP>
wDotp.h.b fa0, ft0, ft1
x8
Innermost loop
…
<reduce partial accumulators packed into a 64b reg>
<cast back to fp8 and store results>
}
}
Figure 5.3: Baseline low and mixed-precision (M ×K)·(K ×N ) matrix
multiplication leveraging SSR and FREP pseudo-code. The innermost
loop is unrolled 8 times and implicitly handled as an FREP loop.
initializing the accumulators to zero, converting the results back to
the input precision, and storing the final results.
5.4
Architecture Extensions
5.4.1
Low-Cost Overflow Detection
Since the MiniFloat Snitch cluster is based on the RISC-V ISA [36],
FP exceptions are signaled by setting the designated flag bit into
an FCSR. Such exceptions are not trapped by default, and adding
the related support would require substantial hardware modifications
(also due to the Snitch custom extensions not supporting exceptions).
To detect an overflow, an application should then read the FCSR,
mask the overflow flag, and check its value in software. Since the
overflow detection is based on an explicit check, its granularity directly
correlates to how promptly the system realizes an overflow occurred
and can react to it, and how large the performance overhead to pay
for detecting overflows is. To the extreme, if this is executed for every
low-precision FP compute instruction, the application can achieve
worse performance than the same kernel executed at higher precision
without the overflow check.
123
5.4. ARCHITECTURE EXTENSIONS
Overflow
Check
EXTENDED ISA
% FCSR check overhead out
% of loop iteration
mv
a0, zero
mv
a1, 128
start: <loop_body>
...
addi a0, 1
bneov a0, a1, start
csrr a5, fcsr
andi a5, a5, 4
bneqz a5, recovery
...
Loop
BASELINE ISA
% FCSR check overhead per
% loop iteration
mv
a0, zero
mv
a1, 128
start: <loop_body>
...
csrr a5, fcsr
andi a5, a5, 4
bneqz a5, recovery
addi a0, 1
bne
a0, a1, start
...
Overflow
Check
Loop
OVERFLOW DETECTION
Figure 5.4: Overflow detection: baseline ISA vs. extended ISA. Leveraging the custom overflow-conditioned branch (bneov) instruction, the
overflow check can be maintained at the same granularity but paying
the performance overhead outside the loop. The loop is exited after
the last iteration or upon an overflow. To check whether the loop
naturally came to an end or an overflow occurred, the related FCSR
is then polled outside the loop.
124
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
Focusing on the low and mixed-precision matrix multiplication kernel introduced in Section 5.3.2, as overflowing is still a rather unlikely
scenario, we target an overflow detection at a middle-loop granularity,
meaning that the overflow check is performed after every iteration of
the loop over n in Fig. 5.3.
To minimize the overhead introduced by overflow detection, we
extend the RISC-V ISA with overflow-conditioned branches (bneov,
beqov). Such branch instructions will jump to a specific instruction
address whenever the related condition on the two source registers is
satisfied and the overflow flag bit is not set. This allows keeping the
overflow check at a middle-loop granularity but paying its overhead
at the outermost loop level (see Fig. 5.4). The middle loop will be
exited only when the loop counter reaches its limit or if an overflow
arises. Therefore, for every outermost-loop iteration, the kernel polls
the FCSR to check whether the middle loop was exited on one or the
other condition to detect if an overflow occurred.
5.4.2
Agile Overflow Recovery
Once an overflow is detected, we apply the last approach covered in
Section 5.2.2, i.e., recomputing with extended range. In particular, in
the MiniFloat Snitch cluster, the wide set of FP formats supported in
hardware can be leveraged to recompute the portion of execution that
produced an overflow at higher precision. Recovering from overflow
then requires:
(a) Stopping the current computation;
(b) Rolling back to the previous checkpoint and re-evaluating the
related portion of the computation with extended range. To
ensure that the internal state does not get irrecoverably compromised between checkpoints, we place checkpoints at the boundaries of portions of the kernel that do not overwrite their inputs
in the memory. For the matrix multiplication case studied in this
chapter, a checkpoint is defined for each middle-loop iteration
(each middle loop contains the whole calculation of an element
of the resulting matrix);
(c) Resuming the normal operation.
FP16
×
FP16
×
FP32
fa0
FP32
64 bits
FP32
FP32
FP32
fa0
+
+
+
×
+
+
×
+
×
+
×
64 bits
×
+
+
×
FP32
×
+
×
FP8 FP8 FP8 FP8 FP8 FP8 FP8 FP8
ft1
FP8 FP8 FP8 FP8 FP8 FP8 FP8 FP8
ft1
fa0
ft1
ft0
FP8 FP8 FP8 FP8 FP8 FP8 FP8 FP8
ft0
FP8 FP8 FP8 FP8 FP8 FP8 FP8 FP8
ft0
+
×
FP16
FP16
+
×
FP16
FP16
64 bits
FP16
FP16
SIMD wDotp.l8-to-32: acc32 += a8 × b8 + c8 × d8
b)
SIMD wDotp.u8-to-32: acc32 += a8 × b8 + c8 × d8
SIMD wDotp16-to-32: acc32 += a16 × b16 + c16 × d16
c)
×
+
+
×
+
+
×
×
+
+
×
FP16FP32FP16
×
64 bits
FP16FP32FP16
FP32
+
+
×
fa0
ft1
×
ft0
acc16 += a8 × b8 + c8 × d8
FP8 FP8 FP8 FP8 FP8 FP8 FP8 FP8
SIMD wDotp8-to-16:
FP8 FP8 FP8 FP8 FP8 FP8 FP8 FP8
Figure 5.5: wDotp SIMD instructions: (a) FP16-to-FP32 wDotp, (b) FP8-to-FP32 wDotp, requiring two
instructions, one operating on the upper and one on the lower parts of the source registers, and (c)
FP8-to-FP16 wDotp. In this chapter, we extended the MiniFloat Snitch ISA with the FP8-to-FP32 wDotp.
To minimize the area overhead, the FP8-to-FP32 wDotp is computed on the 16-to-32-bit unit. The SIMD
parallelism is also limited by the accumulators that can be packed in the destination register. Therefore,
the SIMD FP8-to-FP32 wDotp reaches the same performance as FP16-to-FP32 wDotp without requiring
an explicit conversion of the inputs to FP16, which means half the maximum performance of FP8-to-FP16
wDotp.
a)
5.4. ARCHITECTURE EXTENSIONS
125
126
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
To cover (a), we extended the SSR architecture to support
flushing and re-programming with a specific offset. In this way,
the current SSR execution can be blocked, and the SSRs can
be soft-reset. They can then be re-programmed, specifying an
initial offset. Considering a three-dimensional access pattern, with
i = (0 : N − 1), j = (0 : M − 1), and k = (0 : K − 1), the SSR can
now be re-programmed to follow the same pattern but starting, for
example, from (i, j, k) = (0, 2, 0), thus skipping some middle-loop
iterations only in the first occurrence of the outermost loop.
To address (b), the inputs should be explicitly converted to a
higher precision format, and then the middle-loop iteration should
be re-evaluated.
Since Snitch has a 64-bit SIMD FPU, it can compute four 8-to-16bit wDotp per cycle (see Fig. 5.5c). When moving to higher-precision,
two 16-to-32-bit wDotp operations per cycle can be carried out (see
Fig. 5.5a). In addition to taking twice as many cycles for the wDotp
computations, the inputs should also be converted to FP16. As four
16-bit values can be packed into a 64-bit register, two SIMD cast
instructions are necessary to convert the eight FP8 data saved into
each of the wDotp input registers, one converting the upper and one
the lower four. We show an example looping over a wDotp instruction
in Fig. 5.6. This requires six instructions to re-evaluate the single
8-to-16-bit wDotp instruction. Recovering from an overflow can then
considerably degrade the performance. Note that if the level of input
reuse is high, the conversion can be moved out of the loop, drastically
reducing the overhead. However, this requires using extra space in the
memory, which, depending on the resources available and the footprint
of the kernel, might not be available.
For a more agile recovery routine, we fuse these operations into a
wDotp operation working on 8-bit inputs and accumulating in FP32:
acc32 += a8 × b8 + c8 × d8
(5.2)
Snitch FPU already includes two 16-to-32-bit wDotp units that can
be reused for lower-precision computations (the high flexibility of such
modules [53] already allows for FP8 inputs and FP32 outputs). To add
the support for such operations in the MiniFloat Snitch cluster, we
only adapt the decoder and slightly modify the wDotp SIMD wrapper
127
5.4. ARCHITECTURE EXTENSIONS
to correctly redistribute the inputs and output to the 16-to-32-bit
wDotp units in case of 8-to-32-bit wDotp SIMD operations.
Loop
OVERFLOW RECOVERY
% 8-to-16b wDotp
wDotp.h.b fa0, ft0, ft1
Loop
BASELINE ISA
% 8-to-16-to-32b wDotp
fcvtl.h.b fa2, ft0
fcvtu.h.b fa3, ft0
fcvtl.h.b fa4, ft1
fcvtu.h.b fa5, ft1
wDotp.s.h fa0, fa2, fa4
wDotp.s.h fa1, fa3, fa5
Loop
Recovery
EXTENDED ISA
% 8-to-32b wDotp
wDotpl.s.b fa0, ft0, ft1
wDotpu.s.b fa1, ft0, ft1
Figure 5.6: Overflow recovery for 8-to-16-bit wDotp: 8-to-16-to-32-bit
wDotp (baseline) vs. 8-to-32-bit wDotp. Each instruction in the figure
is reported in the notation instruction.dst format.src format.
The l or u suffix identifies whether an operation that does not fully
use input and destination registers works on the lower or upper part
of the partially consumed registers. In the example, the content of
ft0 and ft1 is directly streamed from memory and updated at every
iteration of the loop, while fa0 and fa1 are accumulators.
Two 8-to-32-bit wDotp operations can be offloaded in parallel to
FPU with a single SIMD instruction. An 8-to-32-bit wDotp SIMD
instruction fully processes the information contained in the accumulator, but only half of the data packed into the 64-bit source
registers, as shown in Fig. 5.5b. Therefore, two of these instructions
are necessary to fully consume the information in the source registers,
one selecting the lower and one the upper 32 bits. Despite running
at half the performance with respect to 8-to-16-bit wDotp SIMD
operations, by leveraging these new instructions, the recovery routine
128
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
does not need explicit conversions (see Fig. 5.6). Furthermore, this
solution leverages the already available datapath, thereby optimizing
the recovery routine of a rare event, like overflowing, at negligible area
cost.
5.5
Overflow Recovery Routine
For mixed-precision kernels like the one introduced in Section 5.3.2,
where partial results are accumulated at higher precision and the final
results are returned in a low-precision format, we distinguish between
intermediate and final overflow. We refer to an intermediate overflow
whenever a partial result overflows, whereas if the accumulations were
evaluated with a wider range, the final result would be in the lowerprecision format representable range. This can happen, for example,
if after producing a positive value out of the available range, following
subtractions move back the accumulated value to a representable
number. We refer to a final overflow whenever the final accumulation
result is out of the low-precision representable range. In this case,
accumulating with an extended range would only move the overflow
to the output conversion.
During the execution, an application can only check the overflow
flag bit in the related FCSR. This would be set to one both in the
case of intermediate and final overflows. To differentiate between the
two cases, upon an overflow detection, the overflow online recovery
routine is called, the overflow flag is cleared, the middle loop iteration
is recomputed with a wider accumulation range, and after converting
the result to the targeted output format, the FCSR is checked again.
If another overflow is detected, the overflow is final, and the related
results are recovered by converting the final 32-bit accumulators to a
wider-range format, such as BF16. Otherwise, the normal execution
is restored. We summarize our online recovery routine in Fig. 5.7.
Leveraging the custom mixed-precision ISA extension, the accumulation precision can be readily augmented. Furthermore, since
the baseline kernel stores partial results only in the FP register file,
changing the accumulation format does not affect the content of the
memories. However, as multiple cores compute the resulting matrix
in parallel, a core storing BF16 data instead of FP8 results might
129
5.5. OVERFLOW RECOVERY ROUTINE
Matmul Start
Overflow recovery
Program SSR or
re-program SSR
post recovery
starting from the
next iteration
Flush SSR
Re-program SSR
re-starting from
the latest iteration
wDotp
FP8 → FP16
wDotp
FP8 → FP32
innermost
loop
Result Cast
FP16 → FP8
Result Cast
FP32 → FP8
middle
loop
NO
last
iteration
Matmul
End
innermost
loop
middle
loop
Overflow?
YES
NO
Overflow?
YES
Result Cast
FP32 → BF16
Figure 5.7: Online overflow detection and recovery: upon detecting an
overflow, the normal execution is blocked, and the overflow recovery
routine is called. This clears the overflow flag bit in the related FCSR
and recomputes one iteration of the matmul middle loop accumulating
at higher precision. After that, the overflow flag bit is re-checked; if
the overflow is resolved, the normal execution is restored starting from
the next middle loop iteration. Otherwise, that means a final overflow
occurred. Therefore, the results are then converted to BF16, a format
offering a wider range than the previous output format FP8.
130
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
overwrite other values stored by another processor. The BF16 results
are then written in a different memory portion, keeping track of the
resulting matrix elements they correspond to. Following processing
blocks could then be conceived to work with a mixture of many
low-precision data and few higher-precision elements. Alternatively,
the whole resulting matrix can be produced in BF16 by converting
to BF16 the results already produced in FP8. This conversion would
add around 1% of overhead. As a wide combination of input, accumulation, and output data types are supported by the Snitch cluster,
thanks to its MiniFloat extension [53] and the extra 8-bit instructions
accumulating in 32 bits described in this chapter, following calculations can select the correct input format by checking the output format
of the matrix multiplication (a global variable is set if a final overflow
occurred).
5.6
Experimental Results
5.6.1
Area Results
To assess the area overhead introduced by our modifications, we synthesize, place and route the baseline MiniFloat Snitch cluster and
our extended version with support for low-cost overflow detection and
agile overflow recovery using Synopsys Fusion Compiler. We use
GlobalFoundries 12 nm technology and target 950 MHz of clock
frequency in a worst-case corner (0.72 V, 125 ◦ C).
The cluster occupies around 5 MGE, and our extensions to the
integer and FP part of the processor added less than 1% of area
overhead to the MiniFloat Snitch core. At a cluster level, the area
overhead is within 0.5%, and the maximum operating frequency has
not been affected.
5.6.2
Performance Results
To evaluate our solution, we assess the performance penalty introduced by overflow detection. Furthermore, we compare the matrix
131
5.6. EXPERIMENTAL RESULTS
128x128 Matmul + Overflow (OF) Recovery
Overhead
Baseline 8-to-32b
N
ch o O
eck F
OF
16
1O
F
2O
F
3O
F
80k
70k
60k
50k
40k
30k
20k
10k
0
N
ch o O
eck F
0O
F
Execution Time [cycles]
Baseline 8-to-16b
Figure 5.8: Performance results: 8-to-16-bit matrix multiplication
baseline vs. baseline enhanced with low-cost overflow detection in
the absence of overflow vs. baseline enhanced with low-cost overflow
detection and recovery at an increasing number of matmul blocks
overflowing vs. 8-to-32-bit matrix multiplication baseline. For a
128×128 matrix multiplication, we divide the kernel into 256 portions,
inserting an overflow checkpoint in each of them.
multiplication performance, considering an increasing number of overflow recoveries, with a counterpart version of the kernel always accumulating at higher precision. We report the results in Fig. 5.8.
Thanks to our optimizations that allow for checking whether an
overflow occurred every middle-loop iteration, but paying the performance price at every outer-loop iteration, overflow detection adds
less than 2% of performance overhead for a 128×128 matrix multiplication kernel. Considering that the innermost loop of the matrix
multiplication used in our study is unrolled 8 times and that the
kernel is parallelized over 8 cores, the overall computation contains
132
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
256 middle-loop iterations per core. Recovering from one or more
overflows occurring in the same middle loop iteration adds around 3%
of performance overhead, while recovering from overflows occurring
in two middle loop iterations adds around 4% of overhead. The
overhead for 16 middle-loop iterations overflowing is instead around
22%. The solution studied in this chapter allows recovering up to
around 52 middle loop iterations overflowing out of the total 256
without introducing a performance penalty with respect to always
accumulating at higher precision, e.g., by running the 8-to-32-bit
matrix multiplication. Note that the performance overhead in this
case, as well as in the case of different matrix sizes, can be tuned by
having the overflow check at a coarser or finer granularity.
Without our modifications, the vanilla Snitch core cannot block
the SSR and FREP execution after detecting an overflow. Having limited support for exceptions is not unusual for units highly specialized
in number crunching [140,143]. The vanilla Snitch core can, therefore,
react only after computing the full workload. For this reason, it would
poll the related FCSR only at the end of the computation. To recover
from overflow, the vanilla core would then (i) complete the 8-to-16-bit
128×128 matrix multiplication, (ii) detect the overflow, (iii) convert
all the inputs to a 16-bit format if there is enough free memory, and
(iv) recompute the matrix multiplication this time at higher precision,
i.e., moving to a 16-to-32-bit implementation. As reported in Fig. 5.8
(and in the third column of Fig. 5.9), recovering a single overflow adds
around 3% of performance overhead when leveraging our extensions,
while it would be 2.7× slower on the vanilla implementation (see the
second column of Fig. 5.9). If the available memory is not sufficient
to store the matrices converted to a 16-bit format, the conversion
would appear in the innermost loop. This would add extra SIMD cast
instructions, as shown in Fig. 5.6, thus compromising the performance
even more. In the mixed-precision matrix multiplication considered
in this study, the innermost loop is unrolled, and one of the wDotp
operands does not change within an innermost-loop iteration. Therefore, only one of the two inputs needs to be converted for each of
the wDotp executions. In this case, the kernel execution would be
4.4× slower with respect to our optimized solution (see first column
of Fig. 5.9). Alternatively, the kernel could forfeit protection, always
leveraging a higher-precision accumulation. However, this would make
133
5.6. EXPERIMENTAL RESULTS
Baseline Overflow Recovery vs. Online Overflow Recovery
(128x128 Matmul)
Execution Time [cycles]
8-to-16b
Matrix Cast
16-to-32b
1 OF Online Recovery
8-to-16-to-32b
(Inner-Loop Cast)
200k
150k
100k
50k
0
Baseline Recovery
(Full Memory)
Baseline Recovery
Online Recovery
Figure 5.9: Baseline overflow recovery vs. online overflow recovery.
Without the proposed ISA extensions, SSRs could not be blocked and
re-programmed. Therefore, the overflow check would happen at the
end of the whole matrix multiplication execution. If an overflow is
detected, the full kernel would then be re-evaluated with an extended
range leveraging the 16-to-32-bit wDotp SIMD instructions. This
requires converting the input matrices to FP16. If there is not
enough free memory available, the inputs should be converted on
the fly, thereby adding a large overhead (first column of the plot).
Alternatively, the matrices can be first converted, and then the matrix
multiplication can operate on the already-converted inputs (second
column of the plot). By leveraging our ISA extension and the proposed
recovery scheme, the overhead to recover can be kept minimal (third
column of the plot).
134
CHAPTER 5. MITIGATING MINIFLOAT LIMITATIONS
it around 1.7× slower than the lower-precision counterpart in the
absence of overflow.
5.6.3
Comparison with State-of-the-Art Solutions
We compare our solution with the scheme proposed by Li et al. [140],
which detects FP exceptions in NVIDIA GPUs by injecting additional
code after each instruction execution. The additional code checks the
mantissa and exponent in the destination register to detect exceptions,
as NVIDIA GPUs do not provide exception status registers. This
covers a wider set of FP exceptions at a finer granularity than the
approach proposed in this chapter. At the same time, it adds large
performance penalties, in some cases, introducing even more than
a 5× slowdown. We keep the overhead minimal by specializing in
the detection of overflow, extending the ISA, and performing the
check with a coarser granularity. Our detection scheme can be tuned
to add less than 2% of performance penalty for a 128×128 matrix
multiplication.
We compare our recovery scheme with the low-cost exception recovery support for GPUs proposed by Menon et al. [141]. Similarly to
our approach, they define a series of checkpoints in the code where the
GPU can roll back and restart the computation with less than 4% of
performance overhead. However, the cost for FP overflow detection is
not included in the reported overhead, as different kinds of exceptions
are considered in their study. Furthermore, their recovery scheme is
based only on the recomputation of a specific portion of the code,
which helps in the case of faults but not in the case of overflows. To
the best of our knowledge, this is the first work assessing the cost and
benefits of an online overflow recovery scheme for hardware platforms
specialized for intense FP computing.
5.7
Further Notes
5.7.1
Floating-Point Precise Exceptions
A similar approach to the one described in this study can be implemented for cores trapping FP precise exceptions. In that case,
5.8. CONCLUSION
135
the extensions to minimize the overflow detection overhead are not
necessary since the overflow would be detected in hardware, thereby
adding no performance penalty. However, adding support for precise
exceptions to a core has a significant architectural cost and can considerably impact its area and power, especially in the case of SIMD
pipelined units [143].
Even when supporting precise exceptions, the overflow recovery
can still be optimized by leveraging mixed-precision operations as
covered in Section 5.4.2 and Section 5.5. By exploiting such operations, explicit cast instructions can be avoided when re-evaluating the
computation at higher precision.
5.8
Conclusion
We presented a low-cost recovery scheme for floating-point overflow
leveraging a mixed-precision RISC-V ISA extension. The proposed
scheme is based on a series of checkpoints that check for overflows. If
one or multiple overflows are detected, the recovery mechanism rolls
back to the previous checkpoint and recomputes only the portion of
the program that generated the overflow at higher precision. We extended a cluster of RISC-V cores supporting low and mixed-precision
floating-point instructions to minimize the performance overhead to
detect overflows and optimize the overflow recovery routine. Such
modifications added less than 1% of hardware overhead. Supporting
overflow detection and recovering from a single overflow can be tuned
to only add a 3% performance penalty in a 128×128 matrix multiplication with respect to a fragile baseline mixed-precision computation,
while always computing with a higher accumulation precision would
make the execution 1.7× slower.
Chapter 6
Conclusions
As technology scaling offers diminishing returns and applications
demand more and more memory and compute resources, the whole
spectrum of hardware architectures, from small IoT devices to large
HPC systems, has to rapidly evolve. In this context, domainspecific hardware acceleration has emerged as a ubiquitous solution
to increase the performance and energy efficiency of computing
systems. These architectures minimize the overhead necessary to
provide general-purpose capabilities, thereby increasing the silicon
efficiency. Mainly motivated by the higher and higher memory and
compute requirements of new ML algorithms, data specialization
has recently been extensively researched, and its attractive benefits
are now pushing further research efforts, aiming at leveraging the
lower-precision benefits in different application scenarios.
This thesis has investigated various solutions and trade-offs around
hardware acceleration. Initially, we have concentrated on different
microarchitectural approaches to support FP operations on generalpurpose cores. Then, we have focused on architectural aspects,
showing the benefits of hardware engines tightly coupled to a cluster of
general-purpose cores. This class of architectures allows for datapathdominated accelerators instead of memory-dominated units by sharing
the memory with the general-purpose core. Finally, we have explored
data specialization by designing new low and mixed-precision FP units
and defining related RISC-V ISA extensions. We have highlighted
137
138
CHAPTER 6. CONCLUSIONS
their benefits and discussed mitigation strategies for the challenges
low-precision formats introduce, such as higher overflow probabilities.
The remainder of this chapter provides an overview of the main
results of this thesis and discusses possible research directions for
future work.
6.1
Main Results
Comparison among Microarchitectural Solutions to Support
Floating-Point Operations
The first part of this thesis has investigated different microarchitectural solutions to support FP operations on low-cost MCU cores
and compared them in terms of area, code size, power, performance,
and energy efficiency: (i) a fully software-based solution emulating
FP instructions on the integer datapath, (ii) a compact multi-cycle
resource-constrained hardware FPU designed to minimize area at the
cost of performance, and (iii) a full-fledged high-performance FPU.
Hardware datapaths introduce an area overhead but can significantly reduce the code size since full FP64 and FP32 FP emulation
libraries occupy up to 11.7 kB and 4.4 kB, respectively. The fullfledged hardware support is the most attractive solution due to
its higher performance and efficiency whenever FP operations are
frequently used and the area overhead introduced by the FPU is
affordable. In very cost-constrained scenarios, the multi-cycle tiny
FPU offers a viable alternative, occupying 54% and 37% less area
than the full-fledged FPU in the double and single-precision case,
respectively.
Buffer-less FFT Accelerator Sharing the Memory with
General-Purpose Cores
In the second part of this thesis, we have focused on different
architectural solutions. Domain-specific hardware accelerators are
often dominated by memories, which take up the majority of their
area and power consumption. However, it is desirable to have their
datapath dominating the accelerator architecture to spend most of
the area and power for the actual computation. To address this
6.1. MAIN RESULTS
139
challenge, we have designed an FFT hardware accelerator that uses
a memory shared with general-purpose cores as a buffer. This also
allows for a more efficient cooperation between the general-purpose
and the domain-specific part of the system, avoiding explicit data
copies between the accelerator buffer and the main memory. We have
then compared this solution with the same accelerator including a
large internal buffer to store its partial results. The buffer-less solution
occupies 74% less area than the buffered one while being slightly faster
as no explicit data copy is required. The accelerator is around 8×
faster than an FFT software implementation running on 8 RISC-V
enhanced with DSP extensions while occupying only 2× the area of
one of these cores.
RISC-V ISA Extension for Low and Mixed-Precision
Floating-Point
We have then concentrated on data specialization. We have designed
a widening FP dot-product unit and defined a scalar and a vector
RISC-V ISA extension for low and mixed-precision FP. Leveraging
low-precision formats, the memory footprint can be readily scaled
down and the efficiency increased. By exploiting our widening dotproduct unit, the performance of widening operations matches the
non-widening ones, ensuring the desired performance increase while
retaining a higher accumulation precision.
We have evaluated the ISA extensions in two clusters of RISC-V
cores, one employing scalar cores enhanced with a SIMD FPU and
streaming the data from memory, and one including vector processors.
By employing 8-bit data types instead of 64-bit FP values, the
memory footprint is reduced by 8× while increasing the efficiency. By
leveraging our extension on a compact RISC-V vector processor, when
reducing the bitwidth by 8×, the performance improves by 7.95× and
the efficiency by 9× even when accumulating with 16 bits.
RISC-V ISA Extension for Efficient Overflow Recovery in
Mixed-Precision Computations
While low-precision solutions remarkably improve the performance
and energy efficiency and reduce the memory footprint, they also
140
CHAPTER 6. CONCLUSIONS
introduce some challenges. For example, reduced exponent bidwidths result in smaller representable ranges, thereby increasing the
probability of overflow. We have devised a RISC-V ISA extension
for low-cost overflow detection and efficient overflow recovery. Our
recovery scheme is based on a set of checkpoints. If an overflow is
detected, the program rolls back and re-computes the related portion
of code between two consecutive checkpoints at higher precision.
Our modifications added less than 1% of hardware overhead, and
their performance penalty to detect overflow and performance gain to
recover an overflow can be tuned by making the checkpoints more or
less fine-grained. We have provided an example for a 128 × 128 matrix
multiplication requiring only 2% of performance penalty to detect
overflow and only 3% to detect and recover from a single overflow.
Computing the full kernel at higher precision would instead make the
execution 1.7× slower.
6.2
Outlook
This thesis has explored several hardware acceleration solutions. The
findings discussed in the previous chapters inspire further research directions to boost hardware acceleration. This section outlines possible
avenues for future investigations in this field.
Novel Data Formats
Data type specialization is an established paradigm that allows for
reducing the memory footprint while enhancing the performance and
energy efficiency. In this thesis, we have investigated the benefits
introduced by low-precision data types. Thanks to the advantages
of these narrow-bitwidth formats, new data types are currently being
researched, such as block FP [144–146], logarithmic formats [29], and
lower-than-8-bit FP [28]. As most of these investigations concentrated on highly specialized architectures, their integration into more
general-purpose accelerators, as the cluster of cores extended with
custom instructions presented in Chapter 4, can be studied.
6.2. OUTLOOK
141
Novel Operations and Datapaths
To mitigate potential accuracy losses due to low-precision accumulations, instructions accumulating at higher precision can be investigated. In this thesis, we have investigated instructions working
on 8-bit inputs and accumulating using 16 or 32 bits. For those
instructions, the accumulation precision is linked to the FP register file
width, as the accumulator or multiple accumulators are packed into
a register file entry. In this context, employing internal accumulators
inside the FPU can be explored to decrease the energy necessary to
store the accumulator and allow for a higher degree of freedom when
setting the accumulator width. Furthermore, several solutions for
LLM inference use FP scaling factors coupled with integer data [147].
To efficiently compute these workloads, new mixed FP-INT datapaths
can be researched (e.g., FP16 × INT4).
Scaled-Out vs. Scaled-Up Accelerators
Hardware units integrated directly within the processor’s pipeline
and programmed through custom instructions have their bandwidth
constrained by the core’s load/store unit. This limits the possibility of
massively scaling these accelerators up. On the other hand, domainspecific hardware processing engines, such as the ones presented in
Chapter 3, can leverage dedicated ports to a shared memory. By
increasing the number of these ports, more data can be provided to
the accelerator every cycle, allowing for its datapath to be scaled up.
However, this scaling cannot continue indefinitely due to architectural
constraints, such as the scalability of the memory interconnect, which
gets more and more complex to route when increasing its number
of ports. This bound can be researched in various configurations.
At the same time, an orthogonal approach can be taken, scaling out
the accelerators. Such a method involves replicating the accelerator
and requires investigating efficient offloading and synchronization
mechanisms to ensure an efficient use of the replicated units.
Compiler Extensions
This thesis has explored ISA extensions, which involved extending the
RISC-V compiler with custom instructions. This has been primarily
142
CHAPTER 6. CONCLUSIONS
achieved through intrinsics. However, to facilitate the adoption of
these ISA extensions, further modifications to the compiler can be
investigated. These extensions would allow for more automated ways
to handle low and mixed-precision operations.
Appendix A
Chip Gallery
This appendix lists all fabricated chips the author has contributed
to in the form of technical work or supervision. A complete, upto-date list can be found online at: http://asic.ethz.ch/authors/
Luca Bertaccini.html.
143
144
Name
APPENDIX A. CHIP GALLERY
Echoes
Designers
Mattia Sinigaglia, Luca Valente,
Luca Bertaccini, Angelo Garofalo,
Francesco Conti, Davide Rossi
Application / Publication PULP / Research Project
Technology / Package
TSMC65 / QFN56
Dimensions
2000 µm × 2000 µm
Voltage
1.2 V
Clock
250 MHz
Echoes is a PULPissimo-based [96] chip for IoT applications. It
contains a single RISC-V core enhanced with ISA extensions that
specialize the processor for digital signal processing. Furthermore, it
contains the FFT hardware accelerator described in Chapter 3 and an
I2S peripheral supporting multi-audio sources.
145
Name
Eclipse
Designers
Jiayong Li, Fabio Cappelini, Enci Zhang,
Luca Bertaccini, Gianna Paulin, Tim Fischer
Application / Publication PULP / Research Project
Technology / Package
TSMC65 / QFN56
Dimensions
2000 µm × 2000 µm
Voltage
1.2 V
Clock
250 MHz
Eclipse is a single-core chip based on the PULPissimo architecture [96]. The CV32E40P core has been extended to offload operations to its FPU through an accelerator-agnostic interface (core-v-xifv0.1.0). The FPU has then been enhanced with the MiniFloat ISA
extension described in Chapter 4. Unlike the case study presented in
Chapter 4, this FPU is 32-bit wide. Therefore, its SIMD units have
been accordingly scaled down, reducing the parallelism by a factor of
two. However, it still supports the two 16 and two 8-bit FP formats,
as well as stochastic rounding and widening dot-product operations.
Finally, the open-source division and square-root unit contained in
the 32-bit T-Head E906 has been integrated into the Eclipse FPU.
146
Name
APPENDIX A. CHIP GALLERY
Occamy
Designers
Gianna Paulin, Florian Zaruba,
Alfio Di Mauro, Andreas Kurth, Gianmarco
Ottavi, Luca Bertaccini, Luca Colagrande,
Manuel Eggimann, Matheus Cavalcante, Nils
Wistoff, Noah Huetter, Paul Scheffler,
Samuel Riedel, Stefan Mach, Thomas Benz,
Tim Fischer, Yichao Zhang,
Zerun Jiang, Beat Muheim, Frank K.
Gürkaynak, Davide Rossi, Luca Benini
Application / Publication HPC / Research Project
Technology / Package
GF12 / Custom
Dimensions
10.5 mm × 6.95 mm
Voltage
0.8 V
Clock
1 GHz
Occamy is a prototype exploring the scalability of the cluster of
RISC-V cores enhanced for high FPU utilizations and extended for
low and mixed-precision FP computing, presented in Chapter 4, in
a 2.5D integrated chiplet systems. The final system is composed
147
of two 73 mm2 Occamy chiplets, each of which contains a Linuxcapable host and 216 RISC-V cores. Each chiplet has a 16 GiB highbandwidth memory (HBM2e) and can communicate with the other
chiplet through a double-data-rate die-to-die link.
The full Occamy system reaches a peak performance of
0.768 TFLOPS for FP64, 1.536 TFLOPS for FP32, 3.072 TFLOPS for
16-bit FP formats, and 6.144 TFLOPS for 8-bit FP data types.
148
Name
Designers
APPENDIX A. CHIP GALLERY
Carfield
Angelo Garofalo, Alessandro Ottaviano,
Matteo Perotti, Robert Balas, Yvan
Tortorella, Michael Rogenmoser, Chi Zhang,
Luca Bertaccini, Nils Wistoff, Maicol Ciani,
Thomas Benz, Cyril Koenig, Luca Valente,
Mattia Sinigaglia, Paul Scheffler, Manuel
Eggimann, Matheus Cavalcante, Beat
Muheim, Zerun Jiang, Davide Rossi,
Frank K. Gürkaynak, Luca Benini
Application / Publication Automotive / Research Project
Technology / Package
Intel16 / BGA1733
Dimensions
4.085 mm × 3.925 mm
Voltage
0.8 V
Clock
400 MHz − 600 MHz
149
Carfield is a highly heterogeneous, safe, resilient, and timepredictable system-on-chip. It includes a dual-core Linux-capable host
enhanced with hypervisor support and an ISA extension protecting
from timing channels. The host is coupled with several specialized
domains: (i) the safety island, targeting safety-critical applications,
is composed of three cores operating in lockstep, thus being able to
detect and recover single faults, (ii) the security island, based on the
OpenTitan project [128], serves as a hardware root of trust, (iii) a
multi-core accelerator specialized for quantized NN inference, and (iv)
a vector accelerator specialized for multi-format and mixed-precision
FP workloads. The latter has been presented in Chapter 4.
150
Name
APPENDIX A. CHIP GALLERY
Maestro
Designers
Mattia Sinigaglia, Yvan Tortorella, Luca
Bertaccini, Marco Bertuletti, Riccardo Tedeschi,
Aurora Di Giampietro, Luigi Ghionda,
Francesco Conti, Davide Rossi
Technology / Package TSMC65 / QFN56
Dimensions
4000 µm × 3000 µm
Voltage
1.2 V
Clock
200 MHz
Maestro contains a single-core host enhanced with a large accelerator including a vector processor and two domain-specialized hardware
accelerators. Maestro targets radar and machine learning workloads.
The vector processor is a scaled-down version of the system enhanced
with MiniFloat capabilities presented in Chapter 4. One of the hardware accelerators is an FP version of the FFT processing engine
presented in Chapter 3 and is coupled to the scratchpad memory.
The other is a matrix multiplication accelerator connected directly to
the vector register file.
Appendix B
Acronyms
AXI
Advanced eXtensible Interface
CC
CPU
CSR
core complex
central processing unit
control and status register
DIT
DMA
DSP
decimation-in-time
direct memory access
digital signal processing
FCSR
FFT
FMA
FP
FP SS
FPU
FREP
floating-point control and status register
fast Fourier transform
fused multiply-add
floating-point
FP subsystem
floating-point unit
floating-point repetition instruction
GCC
GPU
GNU Compiler Collection
graphics processing unit
HeSoC heterogeneous system-on-chip
151
152
APPENDIX B. ACRONYMS
HPC
high-performance computing
HWPE hardware processing engine
IoT
ISA
Internet of Things
instruction set architecture
LFSR
LLM
LUT
linear feedback shift register
large language model
lookup table
MCU
ML
microcontroller unit
machine learning
NN
neural network
PPA
power, performance, area
RNE
round to nearest, ties to even
SIMD
SoA
SoC
SR
SSR
single instruction multiple data
state-of-the-art
system-on-chip
stochastic rounding
stream semantic register
TCDM tightly-coupled data memory
TF32
TensorFloat-32
Vsum
vector inner sum
wDotp widening dot-product
wFMA widening FMA
wVsum widening vector inner sum
Bibliography
[1] J. Chen and X. Ran, “Deep learning with edge computing: A
review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674,
2019.
[2] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari,
J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti
et al., “Using deepspeed and megatron to train megatron-turing
NLG 530B, a large-scale generative language model,” arXiv
preprint arXiv:2201.11990, 2022.
[3] A.
Bechtolsheim,
“The
AI
Datacenter,”
https://
www.opencompute.org/events/past-events/2023-ocp-globalsummit, Accessed: 2024-03-04.
[4] A. Chowdhery et al., “Palm: Scaling language modeling with
pathways,” Journal of Machine Learning Research, vol. 24, no.
240, pp. 1–113, 2023.
[5] S. Lie, “Cerebras architecture deep dive: First look inside the
hardware/software co-design for deep learning,” IEEE Micro,
vol. 43, no. 3, pp. 18–30, 2023.
[6] J. Shalf, “The future of computing beyond moore’s law,”
Philosophical Transactions of the Royal Society A, vol. 378, no.
2166, p. 20190061, 2020.
[7] C. E. Leiserson, N. C. Thompson, J. S. Emer, B. C. Kuszmaul,
B. W. Lampson, D. Sanchez, and T. B. Schardl, “There’s plenty
153
154
BIBLIOGRAPHY
of room at the top: What will drive computer performance after
moore’s law?” Science, vol. 368, no. 6495, 2020.
[8] W. J. Dally, Y. Turakhia, and S. Han, “Domain-specific hardware accelerators,” Communications of the ACM, vol. 63, no. 7,
pp. 48–57, 2020.
[9] B. M. Baas, “A low-power, high-performance, 1024-point FFT
processor,” IEEE Journal of Solid-State Circuits, vol. 34, no. 3,
pp. 380–387, 1999.
[10] A. Wang and A. Chandrakasan, “A 180-mv subthreshold fft
processor using a minimum energy design methodology,” IEEE
Journal of solid-state circuits, vol. 40, no. 1, pp. 310–319, 2005.
[11] M. McKeown, “FFT implementation on the TMS320VC5505,
TMS320C5505, and TMS320C5515 DSPs,” Texas Instruments
Incorporated, White Paper SPRABB6B, 2010.
[12] Q. Zhang, M. Zhang, T. Chen, Z. Sun, Y. Ma, and B. Yu,
“Recent advances in convolutional neural network acceleration,”
Neurocomputing, vol. 323, pp. 37–51, 2019.
[13] Y. Chen, Y. Xie, L. Song, F. Chen, and T. Tang, “A survey of
accelerator architectures for deep neural networks,” Engineering, vol. 6, no. 3, pp. 264–274, 2020.
[14] C. Silvano, D. Ielmini, F. Ferrandi, L. Fiorin, S. Curzel,
L. Benini, F. Conti, A. Garofalo, C. Zambelli, E. Calore et al.,
“A survey on deep learning hardware accelerators for heterogeneous hpc platforms,” arXiv preprint arXiv:2306.15552, 2023.
[15] A. Fuchs and D. Wentzlaff, “The accelerator wall: Limits of chip
specialization,” in 2019 IEEE International Symposium on High
Performance Computer Architecture (HPCA). IEEE, 2019, pp.
1–14.
[16] J. Choquette, W. Gandhi, O. Giroux, N. Stam, and R. Krashinsky, “Nvidia a100 tensor core gpu: Performance and innovation,” IEEE Micro, vol. 41, no. 2, pp. 29–35, 2021.
BIBLIOGRAPHY
155
[17] J. Choquette, “Nvidia Hopper H100 GPU: Scaling performance,” IEEE Micro, 2023.
[18] G. Technologies, “GAP9 next generation processor for hearables
and smart sensors,” https://greenwaves-technologies.com/wpcontent/uploads/2022/06/Product-Brief-GAP9-SensorsGeneral-V1 14.pdf, Accessed: 2024-02-28.
[19] A. Rodriguez et al., “Lower numerical precision deep learning
inference and training,” Intel White Paper, vol. 3, pp. 1–19,
2018.
[20] N. J. Higham and T. Mary, “Mixed precision algorithms in
numerical linear algebra,” Acta Numerica, vol. 31, pp. 347–414,
2022.
[21] Y. Choukroun, E. Kravchik, F. Yang, and P. Kisilev, “Lowbit quantization of neural networks for efficient inference,” in
2019 IEEE/CVF International Conference on Computer Vision
Workshop (ICCVW). IEEE, 2019, pp. 3009–3018.
[22] H. Qin, R. Gong, X. Liu, X. Bai, J. Song, and N. Sebe, “Binary
neural networks: A survey,” Pattern Recognition, vol. 105, p.
107281, 2020.
[23] H. Alemdar, V. Leroy, A. Prost-Boucle, and F. Pétrot, “Ternary
neural networks for resource-efficient ai applications,” in 2017
international joint conference on neural networks (IJCNN).
IEEE, 2017, pp. 2547–2554.
[24] “Bfloat16
hardware
numerics
definition,”
https:
//www.intel.com/content/dam/develop/external/us/
en/documents/bf16-hardware-numerics-definition-whitepaper.pdf, Accessed: 2023-13-07.
[25] A. Agrawal, S. M. Mueller, B. M. Fleischer, X. Sun, N. Wang,
J. Choi, and K. Gopalakrishnan, “DLFloat: A 16-b floating
point format designed for deep learning training and inference,” in 2019 IEEE 26th Symposium on Computer Arithmetic
(ARITH). IEEE, 2019, pp. 92–95.
156
BIBLIOGRAPHY
[26] N. Wang, J. Choi, D. Brand, C.-Y. Chen, and K. Gopalakrishnan, “Training deep neural networks with 8-bit floating point
numbers,” Advances in neural information processing systems,
vol. 31, 2018.
[27] P. Micikevicius et al., “FP8 formats for deep learning,” arXiv
preprint arXiv:2209.05433, 2022.
[28] X. Sun, N. Wang, C.-Y. Chen, J. Ni, A. Agrawal, X. Cui,
S. Venkataramani, K. El Maghraoui, V. V. Srinivasan, and
K. Gopalakrishnan, “Ultra-low precision 4-bit training of deep
neural networks,” Advances in Neural Information Processing
Systems, vol. 33, pp. 1796–1807, 2020.
[29] B. Dally, “Hardware for deep learning,” in 2023 IEEE Hot Chips
35 Symposium (HCS). IEEE Computer Society, 2023, pp. 1–58.
[30] “Nvidia
Hopper
architecture
in-depth,”
https://
developer.nvidia.com/blog/nvidia-hopper-architecture-indepth/, Accessed: 2023-31-03.
[31] “Habana Gaudi2 white paper,” https://habana.ai/wp-content/
uploads/pdf/2021/gaudi2-whitepaper.pdf, Accessed: 2023-3103.
[32] “AWS Trainium,” https://aws.amazon.com/machine-learning/
trainium/, Accessed: 2023-17-02.
[33] S. Knowles, “Graphcore,” in 2021 IEEE Hot Chips 33 Symposium (HCS). IEEE, 2021, pp. 1–25.
[34] G. Cardarilli, L. Di Nunzio, R. Fazzolari, R. La Cesa,
A. Nannarelli, and M. Re, “Tunable floating point for high
quality audio systems: The sound of numbers,” in 57th Annual
Asilomar Conference on Signals, Systems, and Computers.
IEEE, 2023.
[35] E. A. Paxton, M. Chantry, M. Klöwer, L. Saffin, and T. Palmer,
“Climate modeling in low precision: Effects of both deterministic and stochastic rounding,” Journal of Climate, vol. 35, no. 4,
pp. 1215–1229, 2022.
BIBLIOGRAPHY
157
[36] A. Waterman and K. Asanović, “The RISC-V instruction set
manual, volume i: User-level ISA, document version 20191213,”
2019.
[37] SiFive, “SiFive Automotive,” https://www.sifive.com/cores/
automotive, Accessed: 2024-02-28.
[38] Ventana, “Ventana Introduces Veyron, World’s First
Data Center Class RISC-V CPU Product Family,”
https://www.ventanamicro.com/ventana-introduces-veyronworlds-first-data-center-class-risc-v-cpu-product-family/,
Accessed: 2024-02-28.
[39] E. Cui, T. Li, and Q. Wei, “RISC-V instruction set architecture
extensions: A survey,” IEEE Access, vol. 11, pp. 24 696–24 711,
2023.
[40] M. Gautschi, P. D. Schiavone, A. Traber, I. Loi, A. Pullini,
D. Rossi, E. Flamand, F. K. Gürkaynak, and L. Benini, “Nearthreshold RISC-V core with DSP extensions for scalable iot
endpoint devices,” IEEE Transactions on Very Large Scale
Integration (VLSI) Systems, vol. 25, no. 10, pp. 2700–2713,
2017.
[41] P. Scheffler, F. Zaruba, F. Schuiki, T. Hoefler, and L. Benini,
“Indirection stream semantic register architecture for efficient
sparse-dense linear algebra,” in 2021 Design, Automation & Test
in Europe Conference & Exhibition (DATE). IEEE, 2021, pp.
1787–1792.
[42] N. Wistoff, M. Schneider, F. K. Gürkaynak, G. Heiser, and
L. Benini, “Systematic prevention of on-core timing channels by
full temporal partitioning,” IEEE Transactions on Computers,
vol. 72, no. 5, pp. 1420–1430, 2022.
[43] A. Kurth, P. Vogel, A. Capotondi, A. Marongiu, and L. Benini,
“HERO: Heterogeneous embedded research platform for exploring RISC-V manycore accelerators on FPGA,” arXiv preprint
arXiv:1712.06497, 2017.
158
BIBLIOGRAPHY
[44] M. B. Taylor, “Is dark silicon useful? harnessing the four
horsemen of the coming dark silicon apocalypse,” in DAC
Design Automation Conference 2012. IEEE, 2012, pp. 1131–
1136.
[45] A. M. Rahmani, P. Liljeberg, A. Hemani, A. Jantsch, and
H. Tenhunen, The Dark Side of Silicon. Springer, 2016.
[46] Snapdragon, “Snapdragon 8 Gen 3 Mobile Platform,” https://
docs.qualcomm.com/bundle/publicresource/87-71408-1 REV
C Snapdragon 8 gen 3 Mobile Platform Product Brief.pdf,
Accessed: 2024-02-28.
[47] Apple, “Apple unveils M3, M3 Pro, and M3 Max,
the most advanced chips for a personal computer,”
https://www.apple.com/newsroom/2023/10/apple-unveilsm3-m3-pro-and-m3-max-the-most-advanced-chips-for-apersonal-computer/, Accessed: 2024-02-28.
[48] M. Ditty, “Nvidia orin system-on-chip,” in 2022 IEEE Hot Chips
34 Symposium (HCS). IEEE Computer Society, 2022, pp. 1–17.
[49] L. Bertaccini, M. Perotti, S. Mach, P. D. Schiavone, F. Zaruba,
and L. Benini, “Tiny-FPU: low-cost floating-point support
for small RISC-V MCU cores,” in 2021 IEEE International
Symposium on Circuits and Systems (ISCAS). IEEE, 2021,
pp. 1–5.
[50] L. Bertaccini, L. Benini, and F. Conti, “To buffer, or not
to buffer? a case study on FFT accelerators for ultra-lowpower multicore clusters,” in 2021 IEEE 32nd International
Conference on Application-specific Systems, Architectures and
Processors (ASAP). IEEE, 2021, pp. 1–8.
[51] M. Sinigaglia, L. Bertaccini, L. Valente, A. Garofalo, S. Benatti,
L. Benini, F. Conti, and D. Rossi, “Echoes: a 200 GOPS/W
frequency domain SoC with FFT processor and I2S DSP for
flexible data acquisition from microphone arrays,” in 2023 IEEE
International Symposium on Circuits and Systems (ISCAS),
2023, pp. 1–5.
BIBLIOGRAPHY
159
[52] L. Bertaccini, G. Paulin, T. Fischer, S. Mach, and L. Benini,
“MiniFloat-NN and ExSdotp: An ISA extension and a modular open hardware unit for low-precision training on RISC-V
cores,” in 2022 IEEE 29th Symposium on Computer Arithmetic
(ARITH). IEEE, 2022, pp. 1–8.
[53] L. Bertaccini, G. Paulin, M. Cavalcante, T. Fischer, S. Mach,
and L. Benini, “MiniFloats on RISC-V cores: ISA extensions
with mixed-precision short dot products,” IEEE Transactions
on Emerging Topics in Computing, pp. 1–16, 2024.
[54] L. Bertaccini, S. Shen, T. Hoefler, and L. Benini, “Extending
RISC-V for efficient overflow recovery in mixed-precision computations,” in 2024 IEEE 42nd International Conference on
Computer Design (ICCD). IEEE, 2024, pp. 268–275.
[55] G. Paulin, P. Scheffler, T. Benz, M. Cavalcante, T. Fischer,
M. Eggimann, Y. Zhang, N. Wistoff, L. Bertaccini, L. Colagrande, G. Ottavi, F. K. Gürkaynak, D. Rossi, and L. Benini,
“Occamy: A 432-core 28.1 DP-GFLOP/s/W 83% FPU utilization dual-chiplet, dual-HBM2E RISC-V-based accelerator for
stencil and sparse linear algebra computations with 8-to-64-bit
floating-point support in 12nm FinFET,” VLSI Symposium,
2024.
[56] Y. Tortorella, L. Bertaccini, L. Benini, D. Rossi, and F. Conti,
“RedMule: A mixed-precision matrix–matrix operation engine
for flexible and energy-efficient on-chip linear algebra and
TinyML training acceleration,” Future Generation Computer
Systems, vol. 149, pp. 122–135, 2023.
[57] G. Paulin, M. Cavalcante, P. Scheffler, L. Bertaccini, Y. Zhang,
F. Gürkaynak, and L. Benini, “Soft tiles: Capturing physical
implementation flexibility for tightly-coupled parallel processing
clusters,” in 2022 IEEE Computer Society Annual Symposium
on VLSI (ISVLSI). IEEE, 2022, pp. 44–49.
[58] T. Benz, L. Bertaccini, F. Zaruba, F. Schuiki, F. K. Gürkaynak,
and L. Benini, “A 10-core SoC with 20 fine-grain power domains
for energy-proportional data-parallel processing over a wide
160
BIBLIOGRAPHY
voltage and temperature range,” in ESSCIRC 2021-IEEE 47th
European Solid State Circuits Conference (ESSCIRC). IEEE,
2021, pp. 263–266.
[59] M. Perotti, G. Tagliavini, S. Mach, L. Bertaccini, and L. Benini,
“RVfplib: A fast and compact open-source floating-point emulation library for tiny RISC-V processors,” in International
Conference on Embedded Computer Systems. Springer, 2021,
pp. 16–32.
[60] Y. Tortorella, L. Bertaccini, D. Rossi, L. Benini, and F. Conti,
“RedMulE: A compact FP16 matrix-multiplication accelerator
for adaptive deep learning on RISC-V-based ultra-low-power
SoCs,” in 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2022, pp. 1099–1102.
[61] V. Potocnik, L. Colagrande, T. Fischer, L. Bertaccini, D. J.
Pagliari, A. Burrello, and L. Benini, “Optimizing foundation
model inference on a many-tiny-core open-source risc-v platform,” IEEE Transactions on Circuits and Systems for Artificial
Intelligence, vol. 1, no. 1, pp. 37–52, 2024.
[62] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing
and its role in the internet of things,” in Proceedings of the
first edition of the MCC workshop on Mobile cloud computing.
ACM, 2012, pp. 13–16.
[63] B. Barrois and O. Sentieys, “Customizing fixed-point and
floating-point arithmetic—a case study in k-means clustering,”
in 2017 IEEE International Workshop on Signal Processing
Systems (SiPS). IEEE, 2017, pp. 1–6.
[64] ARM, “ARMv7-M Architecture Reference Manual,” https://
developer.arm.com/documentation/ddi0403/latest/.
[65] ARM, “Cortex-M0 Technical Reference Manual,” https://
developer.arm.com/documentation/ddi0432/latest/.
[66] P. D. Schiavone, F. Conti, D. Rossi, M. Gautschi, A. Pullini,
E. Flamand, and L. Benini, “Slow and steady wins the race?
BIBLIOGRAPHY
161
a comparison of ultra-low-power RISC-V cores for internetof-things applications,” in 2017 27th International Symposium
on Power and Timing Modeling, Optimization and Simulation
(PATMOS), Sep. 2017, pp. 1–8.
[67] The GCC low-level runtime library. https://gcc.gnu.org/
onlinedocs/gccint/Libgcc.html#Libgcc.
[68] ARM. Cortex-M4 Technical Reference Manual. https://
developer.arm.com/documentation/ddi0439/b/.
[69] F. Zaruba, F. Schuiki, T. Hoefler, and L. Benini, “Snitch: A tiny
pseudo dual-issue processor for area and energy efficient execution of floating-point intensive workloads,” IEEE Transactions
on Computers, pp. 1–1, 2020.
[70] S. Mach, F. Schuiki, F. Zaruba, and L. Benini, “Fpnew: An
open-source multiformat floating-point unit architecture for
energy-proportional transprecision computing,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2020.
[71] N. Hockert and K. Compton, “Ffpu: Fractured floating point
unit for fpga soft processors,” in 2009 International Conference
on Field-Programmable Technology, 2009, pp. 143–150.
[72] J. J. Pimentel, B. Bohnenstiehl, and B. M. Baas, “Hybrid
hardware/software floating-point implementations for optimized
area and throughput tradeoffs,” IEEE Transactions on Very
Large Scale Integration (VLSI) Systems, vol. 25, no. 1, pp.
100–113, 2016.
[73] X. Wang, M. Magno, L. Cavigelli, and L. Benini, “Fann-onmcu: An open-source toolkit for energy-efficient neural network
inference at the edge of the internet of things,” IEEE Internet
of Things Journal, vol. 7, no. 5, pp. 4403–4417, 2020.
[74] Zfinx RISC-V Extension. https://github.com/riscvarchive/
riscv-zfinx/blob/main/zfinx-1.0.0.pdf.
[75] D. N. Rockmore, “The FFT: an algorithm the whole family can
use,” Computing in Science & Engineering, vol. 2, no. 1, pp.
60–64, 2000.
162
BIBLIOGRAPHY
[76] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with
low precision weights and activations,” The Journal of Machine
Learning Research, vol. 18, no. 1, pp. 6869–6898, 2017.
[77] F. Conti, C. Pilkington, A. Marongiu, and L. Benini, “He-p2012:
Architectural heterogeneity exploration on a scalable manycore platform,” in 2014 IEEE 25th International Conference
on Application-Specific Systems, Architectures and Processors.
IEEE, 2014, pp. 114–120.
[78] F. Conti, P. D. Schiavone, and L. Benini, “XNOR neural engine:
A hardware accelerator IP for 21.6-fJ/op binary neural network
inference,” IEEE Transactions on Computer-Aided Design of
Integrated Circuits and Systems, vol. 37, no. 11, pp. 2940–2951,
2018.
[79] A. Wang, B. Richards, P. Dabbelt, H. Mao, S. Bailey, J. Han,
E. Chang, J. Dunn, E. Alon, and B. Nikolić, “A 0.37mm2
LTE/Wi-Fi compatible, memory-based, runtime-reconfigurable
2n3m5k FFT accelerator integrated with a RISC-V core in 16nm
FinFET,” in 2017 IEEE Asian Solid-State Circuits Conference
(A-SSCC). IEEE, 2017, pp. 305–308.
[80] X. Chen, Y. Lei, Z. Lu, and S. Chen, “A variable-size FFT
hardware accelerator based on matrix transposition,” IEEE
Transactions on Very Large Scale Integration (VLSI) Systems,
vol. 26, no. 10, pp. 1953–1966, 2018.
[81] L. Guo, Y. Tang, Y. Lei, Y. Dou, and J. Zhou, “Transposefree variable-size FFT accelerator based on-chip SRAM,” IEICE
Electronics Express, pp. 11–20 140 171, 2014.
[82] D. Cohen, “Simplified control of FFT hardware,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 24,
no. 6, pp. 577–579, 1976.
[83] Y. Ma and L. Wanhammar, “A hardware efficient control
of memory addressing for high-performance FFT processors,”
IEEE transactions on signal processing, vol. 48, no. 3, pp.
917–921, 2000.
BIBLIOGRAPHY
163
[84] K.-F. Xia, B. Wu, T. Xiong, and T.-C. Ye, “A memory-based
FFT processor design with generalized efficient conflict-free
address schemes,” IEEE Transactions on Very Large Scale
Integration (VLSI) Systems, vol. 25, no. 6, pp. 1919–1929, 2017.
[85] M. Garrido, K. Möller, and M. Kumm, “World’s fastest FFT
architectures: Breaking the barrier of 100 GS/s,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 66, no. 4,
pp. 1507–1516, 2018.
[86] “MSC8154 Reference Manual,” https://www.nxp.com/docs/
en/reference-manual/MSC8154RM.pdf, NXP Semiconductors
N.V., Jun. 2011.
[87] “Advanced
driver
assistance
subsystem
(ADAS),”
https://www.infineon.com/dgdl/Infineon-AURIX Advanced
Driver Assistance Subsystem-Training-v01 00-EN.pdf?fileId=
5546d4626e651a41016e6522e0600003, Infineon Technologies
AG, Jun. 2019.
[88] L. Johnson, “Conflict free memory addressing for dedicated
FFT hardware,” IEEE Transactions on Circuits and Systems II:
Analog and Digital Signal Processing, vol. 39, no. 5, pp. 312–316,
1992.
[89] J. Baek and K. Choi, “New address generation scheme for
memory-based FFT processor using multiple radix-2 butterflies,” in 2008 International SoC Design Conference, vol. 1.
IEEE, 2008, pp. I–273.
[90] H. Sorokin and J. Takala, “Conflict-free parallel access scheme
for mixed-radix FFT supporting I/O permutations,” in 2011
IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP). IEEE, 2011, pp. 1709–1712.
[91] Y. Bo, J. Han, Y. Zou, and X. Zeng, “A low power ASIP
for precision configurable FFT processing,” in Proceedings
of The 2012 Asia Pacific Signal and Information Processing
Association Annual Summit and Conference. IEEE, 2012, pp.
1–4.
164
BIBLIOGRAPHY
[92] F. Conti, D. Rossi, A. Pullini, I. Loi, and L. Benini, “PULP:
A ultra-low power parallel accelerator for energy-efficient and
flexible embedded vision,” Journal of Signal Processing Systems,
vol. 84, no. 3, pp. 339–354, 2016.
[93] J. W. Cooley and J. W. Tukey, “An algorithm for the machine
calculation of complex fourier series,” Mathematics of computation, vol. 19, no. 90, pp. 297–301, 1965.
[94] P. Welch, “A fixed-point fast fourier transform error analysis,”
IEEE Transactions on Audio and Electroacoustics, vol. 17, no. 2,
pp. 151–157, 1969.
[95] A. Teman, D. Rossi, P. Meinerzhagen, L. Benini, and A. Burg,
“Power, area, and performance optimization of standard cell
memory arrays through controlled placement,” ACM Transactions on Design Automation of Electronic Systems (TODAES),
vol. 21, no. 4, pp. 1–25, 2016.
[96] P. D. Schiavone, D. Rossi, A. Pullini, A. Di Mauro, F. Conti,
and L. Benini, “Quentin: an ultra-low-power PULPissimo SoC
in 22nm FDX,” in 2018 IEEE SOI-3D-Subthreshold Microelectronics Technology Unified Conference (S3S), 2018, pp. 1–3.
[97] M. Cavalcante, D. Wüthrich, M. Perotti, S. Riedel, and
L. Benini, “Spatz: A compact vector processing unit for
high-performance and energy-efficient shared-L1 clusters,” in
Proceedings of the 41st IEEE/ACM International Conference
on Computer-Aided Design, 2022, pp. 1–9.
[98] E. E. Swartzlander and H. H. Saleh, “FFT implementation
with fused floating-point operations,” IEEE Transactions on
Computers, vol. 61, no. 2, pp. 284–288, 2010.
[99] S. Lie, “Cerebras architecture deep dive: First look inside the
HW/SW co-design for deep learning: Cerebras systems,” in
2022 IEEE Hot Chips 34 Symposium (HCS). IEEE Computer
Society, 2022, pp. 1–34.
[100] G. Armeniakos, G. Zervakis, D. Soudris, and J. Henkel,
“Hardware approximate techniques for deep neural network
BIBLIOGRAPHY
165
accelerators: A survey,” ACM Computing Surveys, vol. 55, no. 4,
pp. 1–36, 2022.
[101] X. Sun et al., “Hybrid 8-bit floating point (HFP8) training
and inference for deep neural networks,” Advances in neural
information processing systems, vol. 32, 2019.
[102] “Using FP8 with transformer engine,” https://docs.nvidia.com/
deeplearning/transformer-engine/user-guide/examples/
fp8 primer.html, Accessed: 2023-24-03.
[103] B. Hickmann, J. Chen, M. Rotzin, A. Yang, M. Urbanski, and
S. Avancha, “Intel Nervana neural network processor-t (nnp-t)
fused floating point many-term dot product,” in 2020 IEEE 27th
Symposium on Computer Arithmetic (ARITH). IEEE, 2020,
pp. 133–136.
[104] F. Zaruba, F. Schuiki, and L. Benini, “Manticore: A 4096core RISC-V chiplet architecture for ultraefficient floating-point
computing,” IEEE Micro, vol. 41, no. 2, pp. 36–42, 2020.
[105] “RISC-V ‘V’ vector extension,” https://github.com/riscv/riscvv-spec, Accessed: 2023-21-03.
[106] M. Brand, F. Hannig, O. Keszocze, and J. Teich, “Precision-and
accuracy-reconfigurable processor architectures–an overview,”
IEEE Transactions on Circuits and Systems II: Express Briefs,
2022.
[107] G. Henry, P. T. P. Tang, and A. Heinecke, “Leveraging
the bfloat16 artificial intelligence datatype for higher-precision
computations,” in 2019 IEEE 26th Symposium on Computer
Arithmetic (ARITH). IEEE, 2019, pp. 69–76.
[108] J. Osorio, A. Armejach, E. Petit, G. Henry, and M. Casas, “A
BF16 FMA is all you need for dnn training,” IEEE Transactions
on Emerging Topics in Computing, vol. 10, no. 3, pp. 1302–1314,
2022.
[109] A. Nannarelli, “Variable precision 16-bit floating-point vector
unit for embedded processors,” in 2020 IEEE 27th Symposium
on Computer Arithmetic (ARITH). IEEE, 2020, pp. 96–102.
166
BIBLIOGRAPHY
[110] E. Talpes et al., “The microarchitecture of dojo, tesla’s exa-scale
computer,” IEEE Micro, 2023.
[111] N. Brunie, “Modified fused multiply and add for exact low
precision product accumulation,” in 2017 IEEE 24th Symposium
on Computer Arithmetic (ARITH). IEEE, 2017, pp. 106–113.
[112] Y. Tao, G. Deyuan, F. Xiaoya, and R. Xianglong, “Threeoperand floating-point adder,” in 2012 IEEE 12th International
Conference on Computer and Information Technology. IEEE,
2012, pp. 192–196.
[113] J. Sohn and E. E. Swartzlander, “Improved architectures for
a floating-point fused dot product unit,” in 2013 IEEE 21st
Symposium on Computer Arithmetic (ARITH). IEEE, 2013,
pp. 41–48.
[114] J. Sohn and E. E. Swartzlander, “A fused floating-point threeterm adder,” IEEE Transactions on Circuits and Systems I:
Regular Papers, vol. 61, no. 10, pp. 2842–2850, 2014.
[115] O. Desrentes, B. D. de Dinechin, and F. de Dinechin, “Exact
fused dot product add operators,” in 2023 ARITH-30th IEEE
International Symposium on Computer Arithmetic, 2023.
[116] H. Zhang, D. Chen, and S.-B. Ko, “Efficient multiple-precision
floating-point fused multiply-add with mixed-precision support,” IEEE Transactions on Computers, vol. 68, no. 7, pp.
1035–1048, 2019.
[117] W. Mao et al., “A configurable floating-point multiple-precision
processing element for HPC and AI converged computing,”
IEEE Transactions on Very Large Scale Integration (VLSI)
Systems, vol. 30, no. 2, pp. 213–226, 2021.
[118] S. K. Lee et al., “A 7-nm four-core mixed-precision AI chip
with 26.2-TFLOPS hybrid-FP8 training, 104.9-TOPS INT4
inference, and workload-aware throttling,” IEEE Journal of
Solid-State Circuits, vol. 57, no. 1, pp. 182–197, 2021.
BIBLIOGRAPHY
167
[119] M. Croci, M. Fasi, N. J. Higham, T. Mary, and M. Mikaitis,
“Stochastic rounding: implementation, error analysis and applications,” Royal Society Open Science, vol. 9, no. 3, p. 211631,
2022.
[120] M. P. Connolly, N. J. Higham, and T. Mary, “Stochastic
rounding and its probabilistic backward error analysis,” SIAM
Journal on Scientific Computing, vol. 43, no. 1, pp. A566–A585,
2021.
[121] M. Fasi and M. Mikaitis, “Algorithms for stochastically rounded
elementary arithmetic operations in IEEE 754 floating-point
arithmetic,” IEEE Transactions on Emerging Topics in Computing, vol. 9, no. 3, pp. 1451–1466, 2021.
[122] M. Davies et al., “Loihi: A neuromorphic manycore processor
with on-chip learning,” IEEE Micro, vol. 38, no. 1, pp. 82–99,
2018.
[123] M. Mikaitis, “Stochastic rounding: algorithms and hardware
accelerator,” in 2021 International Joint Conference on Neural
Networks (IJCNN). IEEE, 2021, pp. 1–6.
[124] N. J. Higham, Accuracy and stability of numerical algorithms,
2nd ed. Philadelphia, PA, USA: SIAM, 2002.
[125] J. Demmel and Y. Hida, “Accurate and efficient floating point
summation,” IAM Journal on Scientific Computing, vol. 25,
no. 4, pp. 1214–1248, 2004.
[126] J.-M. Muller et al., Handbook of floating-point arithmetic,
2nd ed. Boston, MA, USA: Birkhäuser, 2018.
[127] T. Ben-Nun, J. de Fine Licht, A. N. Ziogas, T. Schneider,
and T. Hoefler, “Stateful dataflow multigraphs: A data-centric
model for performance portability on heterogeneous architectures,” in Proceedings of the International Conference for High
Performance Computing, Networking, Storage and Analysis,
2019.
[128] “Opentitan,” https://opentitan.org/, Accessed: 2024-15-02.
168
BIBLIOGRAPHY
[129] G. Tagliavini, S. Mach, D. Rossi, A. Marongiu, and L. Benini,
“A transprecision floating-point platform for ultra-low power
computing,” in 2018 Design, Automation & Test in Europe
Conference & Exhibition (DATE). IEEE, 2018, pp. 1051–1056.
[130] A. Haidar, H. Bayraktar, S. Tomov, J. Dongarra, and N. J.
Higham, “Mixed-precision iterative refinement using tensor
cores on GPUs to accelerate solution of linear systems,” Proceedings of the Royal Society A, vol. 476, no. 2243, 2020.
[131] C. Rubio-González et al., “Precimonious: Tuning assistant for
floating-point precision,” in Proceedings of the international
conference on high performance computing, networking, storage
and analysis, 2013, pp. 1–12.
[132] W.-F. Chiang, M. Baranowski, I. Briggs, A. Solovyev,
G. Gopalakrishnan, and Z. Rakamarić, “Rigorous floating-point
mixed-precision tuning,” ACM SIGPLAN Notices, vol. 52, no. 1,
pp. 300–315, 2017.
[133] W. Kahan, “Lecture notes on the status of IEEE
standard 754 for binary floating-point arithmetic,”
https://people.eecs.berkeley.edu/∼wkahan/ieee754status/
IEEE754.PDF, 1996, Accessed: 2024-17-01.
[134] Sun Microsystems, Inc., “Numerical computation guide,”
https://docs.oracle.com/cd/E19059-01/fortec6u2/806-7996/
806-7996.pdf, Accessed: 2024-16-01.
[135] J. R. Hauser, “Handling floating-point exceptions in numeric
programs,” ACM Transactions on Programming Languages and
Systems (TOPLAS), vol. 18, no. 2, pp. 139–174, 1996.
[136] J. Zhang, H. Tan, and L. Huang, “SFDoP: A scalable fused
bfloat16 dot-product architecture for DNN,” in 2023 IEEE 41st
International Conference on Computer Design (ICCD). IEEE,
2023, pp. 62–65.
[137] “Floating point formats for machine learning: IEEE working
group P3109 interim report on 8-bit binary floating-point
formats,”
https://github.com/P3109/Public/tree/main/
BIBLIOGRAPHY
169
Shared%20Reports/P3109%20WG%20Interim%20Report.pdf,
Accessed: 2024-16-01.
[138] “NVIDIA ADA GPU Architecture Whitepaper v2.02,”
https://images.nvidia.com/aem-dam/Solutions/DataCenter/l4/nvidia-ada-gpu-architecture-whitepaper-V2.02.pdf,
Accessed: 2024-07-05.
[139] M. van Baalen et al., “FP8 versus INT8 for efficient deep
learning inference,” arXiv preprint arXiv:2303.17951, 2023.
[140] X. Li, I. Laguna, B. Fang, K. Swirydowicz, A. Li, and
G. Gopalakrishnan, “Design and evaluation of GPU-FPX:
A low-overhead tool for floating-point exception detection in
NVIDIA GPUs,” in Proceedings of the 32nd International
Symposium on High-Performance Parallel and Distributed Computing, 2023, pp. 59–71.
[141] J. Menon, M. De Kruijf, and K. Sankaralingam, “iGPU:
exception support and speculative execution on GPUs,” ACM
SIGARCH Computer Architecture News, vol. 40, no. 3, pp.
72–83, 2012.
[142] F. Schuiki, F. Zaruba, T. Hoefler, and L. Benini, “Stream semantic registers: A lightweight RISC-V ISA extension achieving
full compute utilization in single-issue cores,” IEEE Transactions on Computers, vol. 70, no. 2, pp. 212–227, 2020.
[143] I. Tanasic, I. Gelado, M. Jorda, E. Ayguade, and N. Navarro,
“Efficient exception handling support for GPUs,” in Proceedings
of the 50th Annual IEEE/ACM International Symposium on
Microarchitecture, 2017, pp. 109–122.
[144] M. Drumond, T. Lin, M. Jaggi, and B. Falsafi, “Training
dnns with hybrid block floating point,” Advances in Neural
Information Processing Systems, vol. 31, 2018.
[145] B. D. Rouhani et al., “Microscaling data formats for deep
learning,” arXiv preprint arXiv:2310.10537, 2023.
170
BIBLIOGRAPHY
[146] B. D. Rouhani et al., “With shared microexponents, a little
shifting goes a long way,” in Proceedings of the 50th Annual
International Symposium on Computer Architecture, 2023, pp.
1–13.
[147] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq:
Accurate post-training quantization for generative pre-trained
transformers,” arXiv preprint arXiv:2210.17323, 2022.
Curriculum Vitae
Luca Bertaccini was born in Bologna, Italy, on
September 25, 1995. He received his BSc and
MSc degrees from the University of Bologna
in 2017 and 2020, respectively. In 2020,
he joined the Integrated Systems Laboratory
of ETH Zurich as a PhD candidate under
the supervision of Prof. Dr. Luca Benini.
His research interests include heterogeneous
systems-on-chip, energy-efficient hardware accelerators, and mixed-precision computing.
171