High Performance Kernel Smoothing Library For Biomedical Imaging A Thesis Presented by Haofu Liao to The Department of Electrical and Computer Engineering in partial fulfillment of the requirements for the degree of Master of Science in Electrical and Computer Engineering Northeastern University Boston, Massachusetts May 2015 NORTHEASTERN UNIVERSITY Graduate School of Engineering Thesis Signature Page Thesis Title: High Performance Kernel Smoothing Library For Biomedical Imaging Author: Haofu Liao Department: Electrical and Computer Engineering NUID: 001988944 Approved for Thesis Requirements of the Master of Science Degree Thesis Advisor Dr. Deniz Erdogmus Signature Date Signature Date Signature Date Signature Date Signature Date Signature Date Thesis Committee Member or Reader Dr. David R. Kaeli Thesis Committee Member or Reader Dr. Gunar Schirner Thesis Committee Member or Reader Dr. Rafael Ubal Department Chair Dr. Sheila S. Hemami Associate Dean of Graduate School: Dr. Sara Wadia-Fascetti Contents List of Figures iv List of Tables vi Abstract of the Thesis vii 1 2 3 Introduction 1.1 Background . . . . . 1.2 Related Work . . . . 1.3 Contributions . . . . 1.4 Outline of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kernel Smoothing 2.1 Univariate Kernel Density Estimation . 2.2 Multivariate Kernel Density Estimation 2.3 Kernel Functions . . . . . . . . . . . . 2.3.1 Univariate Kernels . . . . . . . 2.3.2 Separable Multivariate Kernels . 2.4 Bandwidth . . . . . . . . . . . . . . . . 2.4.1 Types of Bandwidth . . . . . . 2.4.2 Variable Bandwidth . . . . . . . 2.5 Kernel Density Derivative Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1 2 3 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 6 7 8 9 10 11 11 13 15 Vesselness Measure 3.1 Gradients and Hessian Matrices of Images . . . . . 3.1.1 Gradient . . . . . . . . . . . . . . . . . . . 3.1.2 Hessian . . . . . . . . . . . . . . . . . . . 3.2 Finding 1st and 2nd Order Derivatives From Images 3.2.1 Gradient Operator . . . . . . . . . . . . . 3.2.2 Gaussian Smoothing . . . . . . . . . . . . 3.2.3 Kernel Density Derivative Estimation . . . 3.3 Frangi Filtering . . . . . . . . . . . . . . . . . . . 3.4 Ridgeness Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 18 18 20 22 23 25 27 28 29 ii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 5 6 7 GPU Architecture and Programming Model 4.1 GPU Architecture . . . . . . . . . . . . . 4.2 Programming Model . . . . . . . . . . . 4.3 Thread Execution Model . . . . . . . . . 4.4 Memory Accesses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 32 34 36 38 Algorithms and Implementations 5.1 Efficient Computation of Separable Multivariate Kernel Derivative . . . . . 5.1.1 Definitions and Facts . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.3 Complexity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 High Performance Kernel Density and Kernel Density Derivative Estimators 5.2.1 Multi-core CPU Implementation . . . . . . . . . . . . . . . . . . . 5.2.2 GPU Implementation in CUDA . . . . . . . . . . . . . . . . . . . 5.3 Efficient k-Nearest Neighbors Bandwidth Selection For Images . . . . . . . 5.3.1 k-Nearest Neighbors Covariance Matrix of Images . . . . . . . . . 5.3.2 r-Neighborhood Covariance Matrix of Images . . . . . . . . . . . 5.3.3 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.4 GPU Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 40 41 44 45 46 47 48 59 59 61 64 65 Experiments and Results 6.1 Environment . . . . . . . . . . . . . . . . . 6.2 Performance Evaluation . . . . . . . . . . . 6.2.1 Efficient SMKD . . . . . . . . . . 6.2.2 High Performance KDE and KDDE 6.2.3 Efficient k-NN Bandwidth Selector 6.3 Vesselness Measure . . . . . . . . . . . . . 6.3.1 Frangi Filtering . . . . . . . . . . . 6.3.2 Ridgeness Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 68 69 69 71 75 77 77 78 Conclusion and Future Work 7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 82 83 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 iii List of Figures 2.1 2.2 2.3 2.4 2.5 2.6 2.7 The relation between under-five mortality rate and life expectancy at birth Univariate kernel density estimate . . . . . . . . . . . . . . . . . . . . . Multivariate kernel density estimate . . . . . . . . . . . . . . . . . . . . Truncated Gaussian kernel function . . . . . . . . . . . . . . . . . . . . Univariate kernel density estimates of different bandwidths . . . . . . . . Comparison of three bandwidth matrix parametrization classes . . . . . . Univariate sample point kernel density estimate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 7 8 10 12 13 14 3.1 3.2 3.3 3.4 3.5 Gradient of the standard Gaussian function . Image gradient . . . . . . . . . . . . . . . Visualized Eigenvalues with ellipsoid . . . Derivatives of Gaussian filters . . . . . . . Vesselness measure using Frangi filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 20 22 26 30 4.1 4.2 4.3 4.4 4.5 4.6 4.7 GPU block diagram . . . . . . . . . . . GPU hardware memory hierarchy . . . Programming model . . . . . . . . . . GPU software memory hierarchy . . . . Warp scheduler . . . . . . . . . . . . . Aligned and consecutive memory access Misaligned memory access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 34 35 36 37 39 39 5.1 5.2 5.3 5.4 5.5 5.6 5.7 Relation between nodes in graph G . . . . . . . . . . . . . . Graph based efficient multivariate kernel derivative algorithm Memory access patterns of matrices and cubes . . . . . . . . Memory access pattern without using shared memory . . . . Memory access pattern using shared memory . . . . . . . . The covariance and disk operators of r = 4 . . . . . . . . . Searching circles of different radii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 45 51 58 58 63 64 6.1 Multiplication number comparison between the naive method and the proposed efficient method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Execution time comparison between the naive method and the proposed efficient method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 iv . . . . . . . . . . . . . . 70 72 6.3 6.4 6.5 6.6 6.7 6.8 6.9 The comparison of speed-ups between different optimization methods on synthetic 2D data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The comparison of speed-ups between different optimization methods on synthetic 3D data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performance of the k-NN bandwidth selector on 2D images using naive algorithm and CPU efficient algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performance of the k-NN bandwidth selector on 3D images using naive algorithm, CPU efficient algorithm and GPU efficient algorithm . . . . . . . . . . . . . . . . Vesselness measure results using Frangi filter . . . . . . . . . . . . . . . . . . . . Algorithm pipeline of the ridgeness filtering based vessel segmentation . . . . . . . Vesselness measure results using ridgeness filter . . . . . . . . . . . . . . . . . . . v 73 73 75 76 79 80 81 List of Tables 3.1 Possible orientation patterns in 2D and 3D images . . . . . . . . . . . . . . . . . . 23 4.1 Compute capability of Fermi and Kepler GPUs . . . . . . . . . . . . . . . . . . . 38 6.1 6.2 Experiment environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Global memory transactions between different optimization methods . . . . . . . . 69 74 vi Abstract of the Thesis High Performance Kernel Smoothing Library For Biomedical Imaging by Haofu Liao Master of Science in Electrical and Computer Engineering Northeastern University, May 2015 Dr. Deniz Erdogmus, Adviser The estimation of probability density and probability density derivatives has full potential for applications. In biomedical imaging, the estimation of the first and second derivatives of the density is crucial to extract tubular structures, such as blood vessels and neuron traces. Probability density and probability density derivatives are often estimated using nonparametric data-driven methods. One of the most popular nonparametric methods is the Kernel Density Estimation (KDE) and Kernel Density Derivative Estimation (KDDE). However, a very serious drawback of using KDE and KDDE is the intensive computational requirements, especially for large data sets. In this thesis, we develop a high performance kernel smoothing library to accelerate KDE and KDDE methods. A series of hardware optimizations are used to deliver a high performance code. On the host side, multi-core platforms and parallel programming frameworks are used to accelerate the execution of the library. For 2 or 3-dimensional data points, the Graphic Processing Unit (GPU) platform is used to provide high levels of performance to the kernel density estimators, kernel gradient estimators as well as the kernel curvature estimators. Several Compute Unified Device Architecture (CUDA) based techniques are used to optimize their performances. What’s more, a graph-based algorithm is designed to calculate the derivatives efficiently and a fast k-nearest neighbor bandwidth selector is designed to speed up the variable bandwidth selection for image data on GPU. vii Chapter 1 Introduction 1.1 Background Density estimation constructs an estimate of underlying probability density function using an observed data set. In density estimation, there are three types of approaches, parametric, semi-parametric and nonparametric. Both paramedic and semi-parametric techniques require a prior knowledge of the underlying distribution of the sample data. In parametric approaches, the data should be from a known family. In semi-parametric approaches, the knowledge of the mixture distribution is assumed to be known. On the contrary, nonparametric methods, which attempt to flexibly estimate an unknown distribution, require less structure information about the underlying distribution. This advantage makes them a good choice for robust and more accurate analysis. Kernel density estimation (KDE) is the most widely studied and used nonparametric technique. It is first introduced by Rosenblatt [1], and then discussed in detail by Paren [2]. Typically, a kernel density estimate is constructed by a sum of kernel functions centered at observed data points and a smoothing parameter called bandwidth is used to control the smoothness of the estimated densities. KDE has a broad range of applications such as image processing, medical monitoring and market analysis. On the other hand, the estimation of density derivative, though have only received relatively scant attention, also has a full potential for applications. Indeed, nonparametric estimation of higher order derivatives of the density functions can provide lots of important information about a multivariate data set, such as local extrema, valleys, ridges or saddle points. In the gradient estimation case, the well known mean-shif t algorithm can be used for clustering and data filtering. It is very popular in the areas of low-level vision problems, discontinuity preserving smoothing and image segmentation. 1 CHAPTER 1. INTRODUCTION Another use of gradient estimation is to find filaments in point clouds, which has applications in medical imaging, remote sensing, seismology and cosmology. In the Hessian estimation case, the eigenvalues of Hessian matrix are crucial to manifolds extraction and curvilinear structure analysis. Moreover, the prevalent Frangi filter [3] and its variants also require the calculation of Hessian matrix. Smoothing parameter or bandwidth plays a very important role in KDE and kernel density derivative estimation. It determines the performance of the estimator in practice. However, in most of the cases only constrained bandwidth is used. In adaptive kernel density estimation case, the bandwidth is a symmetric positive definite matrix; it allows the kernel estimator to smooth in any direction whether coordinate or not. In even simpler case, a bandwidth matrix is only a positive scalar multiple of the identity matrix. There are three reasons for the widely use of simpler parameterization than the unconstrained counterpart. First, in practice they need less smoothing parameters to be tuned. Second, due to the difficulties encountered in the mathematical analysis of estimators with unconstrained bandwidth. Third, unconstrained bandwidth is not suitable for most of the existed fast KDE algorithms. 1.2 Related Work Around 1980s, KDE becomes the de facto nonparametric method to represent a continuous distribution from a discrete point set. However, a very serious drawback of KDE methods is the expensive computational complexity for the calculation of probability at each target data vector. A typical KDE method is usually of computational order O(n2 k), where n is the number of observations and k is the number of variables. In many cases, such as database management and wildlife ecology, the size of n can be as large as hundreds of millions. What’s more, data-driven methods of bandwidth selection can also add additional order of computational burden to KDEs. Currently, there are two different approaches to satisfy the computational demands of KDEs. The first one is to use approximate techniques to reduce the computational burden of kernel estimation. In 1982, Silverman [4] proposed a fast density estimation method based on Fast Fourier Transformation (FFT). However, this method requires source points to be distributed on an evenly spaced grid and it can only compute univariate kernels. In 1994, Wond [5] extended Silverman’s method to multivariate case and proposed a well-known binned estimation method. But it still requires a binned data set. Another approach is proposed by Elgammal [6]. He designed a Fast Gauss Transform (FGT) method, where the data are not necessarily on grids. But the problem is the complexity of 2 CHAPTER 1. INTRODUCTION computation and storage of the FGT grows exponentially with dimension. Therefore, Changjiang Yang et al. [7] proposed an Improved Fast Gauss Transform (IFGT) which can efficiently evaluate the sum of Gaussian in higher dimension. But both algorithms are limited to work with only Gaussian kernel. Moreover, Sinha and Gupta [8] proposed a new fast KDE algorithm through PDDP which they claimed that their algorithm is more accurate and efficient than IFGT. Recently, an ε-sample algorithms is proposed by Phillips [9]. His algorithm studied the worst case error of kernel density estimates via subset approximation which can be helpful for sampling large dataset and hence can result a fast kernel density estimate. The second approach is to use parallel computing. Some of the most important parallel computing technologies are clustering computing, multicore computing and general-purpose computing on graphics processing units (GPGPU). For clustering computing, Zheng et al [10] implemented the kernel density estimation on Hadoop cluster machines using MapReduce as the distributed and parallel programming framework. Łukasik [11] and Racine [12] presented parallel methods based on Message Passing Interface standard in multicomputer environment. For multicore computing, Michailidis and Margaritis [13] parallelized kernel estimation methods on multi-core platform using different programming frameworks such as Pthreads, OpenMP, Intel Cilk++, Intel TBB, SWARM and FastFlow. The same authors also presented some preliminary work of kernel density estimation using GPU approach [14]. Recently, Andrzejewski et al [15] proposed a GPU based algorithm to accelerate the bandwidth selection methods of kernel density estimators. However, all the authors ignore the more complicated unconstrained bandwidth case for multivariate kernel density estimation. The kernel density derivative estimation is not considered as well. 1.3 Contributions We developed a highly efficient and flexible kernel smoothing library. This library supports both univariate and multivariate kernels. Unlike other existing kernel smoothing libraries [16, 17, 18, 19], it supports not only the constrained(restricted) bandwidth, but also the more general unconstrained bandwidth. The bandwidth, both constrained and unconstrained, is not limited to be fixed. What’s more, a sample-point based variable bandwidth is supported as well. The input data has no dimensional limitation. Basically, as long as the hardware is permitted, the library can support data of any dimension. To improve the computational efficiency, kernel functions with finite support can be used and only the data points within the kernel function’s support will be calculated. Besides the kernel density estimators, the kernel density derivative estimators are implemented as 3 CHAPTER 1. INTRODUCTION well. For separable kernel functions, this library is able to calculate the derivatives of any order. A graph based algorithm is designed to calculate the derivatives efficiently. A series of hardware optimizations are used to deliver a high performance code. On the host side, multi-core platforms and parallel programming frameworks are used to accelerate the execution of the library. For 2 or 3-dimensional data points, the GPU platform is used for speeding up the kernel density estimators, kernel gradient estimators as well as the kernel curvature estimators. Several CUDA based algorithms are designed to optimize their performance. Finally, an efficient k-nearest neighbor based variable bandwidth selector is designed for image data and a high-performance CUDA algorithm is implemented for this selector. 1.4 Outline of the Thesis This thesis is organized as follows: In Chapter 2, we discuss the detailed knowledge background of the kernel smoothing theory. We introduce both the univariate and multivariate KDE methods, provide the direct calculation of the separable multivariate kernel and kernel derivatives, present a variety of bandwidth types, and give the strict definition of KDDE methods. In Chapter 3, we first introduce the gradients and Hessian matrices of images. Then, discuss three ways of finding 1st and 2nd order derivatives from images. Finally, we present two vesselness measure algorithms that use gradients and Hessian matrices of images. Chapter 4 gives a detailed introduction of the GPU architecture and the CUDA programming framework. We present three major contributions of this thesis in Chapter 5 and discuss their performance in Chapter 6. Finally, conclusions and future works are given in Chapter 7. 4 Chapter 2 Kernel Smoothing Given data X 1 , X 2 , . . . , X n are drawn from density f , How do we estimate the probability density function f from these observations? 80 Life expectancy at birth 75 70 65 60 55 50 45 40 0 50 100 150 200 250 Under-five mortality (per 1000 live births) F IGURE 2.1: The relation between under-five mortality rate and life expectancy at birth in different countries and regions. The original data is from the department of Economic and Social Affairs, United Nation. 5 CHAPTER 2. KERNEL SMOOTHING 2.1 Univariate Kernel Density Estimation Given a set of n independent and identically distributed (i.i.d.) random samples X1 , X2 , . . . , Xn from a common density f , the univariate kernel density estimator is fˆ(x; h) = n−1 n X i=1 Here K is a kernel function which satisfies R h−1 K(h−1 (x − Xi )). (2.1) K(x)dx = 1, and h > 0 is a smoothing parameter called the bandwidth. By introducing a rescaling notation Kh (u) = h−1 K(h−1 x), the above formula can be written in a more compact way fˆ(x; h) = n−1 n X i=1 Kh (x − Xi ). (2.2) As we can see from Equation (2.2), the kernel density estimate is a summation of scaled kernel functions with each of a probability mass n−1 . In a intuitive view, we can look this as a sum of ‘bumps’ placed at the observation points X1 , X2 , . . . , Xn . The kernel function K determines the shape of the bumps while the bandwidth h determines their width. An illustration is given in Figure 2.2, where observations Xi are marked in dots on x-axis and their corresponding scaled kernel ‘bumps’ n−1 Kh (x − Xi ) are shown in the dotted lines. Here, the kernel K is chosen to be the standard normal pdf N (0, 1). The resulting univariate kernel density estimate fˆ is given in the solid line. We can find that the estimate is bimodal, which is a reflection of the distribution of observations. Usually, it is not appropriate to construct a density estimate from such a small number of samples, but a sample size of 5 has been chosen here for the sake of clarity. As illustrated in Figure 2.2, the value of the kernel estimate at point x is simply the average of the n kernel ordinates at that point. The estimate combines contributions from each data. Hence, in regions where there are many observations, the estimate will have a relative large value. It is consistent with the truth that a densely distributed region will have a high probability density, and vice versa. Notice that, in this case the scaled kernel Kh is simply the N (0, h2 ) density. In this case, the bandwidth parameter h can be seen as a scaling factor which determines the spread of the kernel. In common, the bandwidth controls the amount of smoothness of kernel density estimators. It is the most important factor in KDE and KDDE. We will cover more details of bandwidths in Section 2.4. 6 CHAPTER 2. KERNEL SMOOTHING 0.6 0.5 f^(x) 0.4 0.3 0.2 0.1 0 -3 -2 -1 0 1 2 3 x F IGURE 2.2: Univariate kernel density estimate: dots on x-axis - sample (training) points, solid line - kernel density estimate, dashed lines - scaled kernels at different sample points. 2.2 Multivariate Kernel Density Estimation The d-dimensional multivariate kernel density estimator, for a set of n i.i.d. sample X 1 , X 2 , . . . , X n from a common density f , is fˆ(x; H) = n−1 n X i=1 KH (x − X i ), (2.3) where • x = (x1 , x2 , . . . , xd )T , X i = (Xi1 , Xi2 , . . . , Xid )T , i = 1, 2, . . . , n; • K is the unscaled kernel, which is usually a spherically symmetric probability density function; • KH is the scaled kernel. It is related with the unscaled kernel by KH (x) = |H|−1/2 K(H −1/2 x); • H is the d × d bandwidth matrix, which is non-random, symmetric, and positive defined. The same as the univariate case, the multivariate kernel density estimate is calculated by placing a scaled kernel of mass n−1 at each data point and then aggregating to form the density estimate. Figure 2.3 illustrates a multivariate kernel density estimate in 2-dimension. The left hand figure shows observations (marked in dots) from density f (denoted by the isolines). On the right is the estimate fˆ. Since the ground truth f is actually a linear combination of five bivariate normal density functions, as we can see from the right that the fˆ gives a good estimate of this function. 7 8 8 7 7 6 6 5 5 4 4 3 3 f^(x) f (x) CHAPTER 2. KERNEL SMOOTHING 2 1 2 1 0 0 -1 -1 -2 -2 -3 -5 0 -3 5 -5 0 5 x x F IGURE 2.3: Multivariate kernel density estimate. Left: the contour denotes the density function f , and the dots are the sample/training points that draw from f . Right: the estimate fˆ calculated from the dots in the left figure. Define S = H −1/2 and evaluate fˆ at some points of interest x1 , x2 , . . . , xm , Equation (2.3) can be rewritten as fˆ(xi ; S) = n−1 n X j=1 KS (xi − X j ), i = 1, 2, . . . , m, j = 1, 2, . . . , n, (2.4) where xi = (xi1 , xi2 , . . . , xid )T , X j = (Xj1 , Xj2 , . . . , Xjd )T , i = 1, 2, . . . , m, j = 1, 2, . . . , n and K S (x) = |S|K(Sx). Here xi is called test point, X i is called training point, and S is called scale. The scale and bandwidth is related by H −1 = S T S. Equation (2.4) provides a more direct form when considering its implementation and complexity. Instead of a continuous function fˆ(x), the discrete form fˆ(xi ) is more intuitive for its software implementation and the scale S reduces the complexity by avoiding the calculation of inverse square root of bandwidth H. In the subsequent discussions, we will mostly use this form for the formulas and equations. Since there are m test points and for each test point there are n scaled kernel function evaluations at the d-dimensional training point, then the complexity of Equation (2.4) is O(mnd). 2.3 Kernel Functions 8 CHAPTER 2. KERNEL SMOOTHING 2.3.1 Univariate Kernels A univariate kernel is a one dimensional, non-negative, real-valued, integrable function k which satisfies • R +∞ −∞ k(u)du = 1; • k(−u) = k(u) for all values of u. The first requirement ensures that the result of kernel density estimator is a probability density function. The second requirement makes sure that the kernel function has zero mean and the kernel function placed at certain training point has an average value the same as the corresponding training point. To help reduce the computational complexity, a univariate bounding box can be used to the kernels for finite support at some costs of less accuracy. The truncated kernel is given as Z b Z a ktrunc (x; a, b) = [ k(u)du − k(u)du]−1 k(x)b(x; a, b), −∞ (2.5) −∞ where b(x; a, b) is the bounding box expanding from the lower bound a to the upper bound b, and 1, if a ≤ x ≤ b, , (2.6) b(x; a, b) = 0, otherwise. Rb Ra The the normalization factor [ −∞ k(u)du − −∞ k(u)du]−1 is introduced to ensure the truncated kernel function satisfies the requirement that Z +∞ ktrunc (u)du = 1. (2.7) −∞ If the accuracy outweighs the computational complexity, then the bounding box can be ignored by setting the lower bound and upper bound to −∞ and +∞ respectively. In this case, we get ktrunc (x) = k(x). There are a range of univariate kernels commonly used, such as uniform, triangular, biweight, triweight, Epanechnikov, etc. However, the choice of the univariate kernel function k is not crucial to the accuracy of kernel density estimators [20]. Due to the convenient of its mathematical properties and the smooth density estimates it results, the normal kernel is often used k(x) = φ(x), where φ is the standard normal density function and it is defined as 1 2 1 φ(x) = √ e− 2 x . 2π 9 (2.8) CHAPTER 2. KERNEL SMOOTHING 1.4 kernel bounding box truncated kernel 1.2 1 y 0.8 0.6 0.4 0.2 0 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 x F IGURE 2.4: Truncated Gaussian kernel function. The solid line - the truncated Gaussian kernel, the square dash line - the bounding box, the dot line - the untruncated Gaussian kernel. 2.3.2 Separable Multivariate Kernels The multivariate kernel functions, based on their separability, can be divided into two categories: separable kernel function and nonseparable kernel function. Due to the computational simplicity, we mainly focus on the separable multivariate kernel functions in this section. A separable multivariate kernel K(x) : Rd → R can be written as [21] K(x) = d Y k(xl ), (2.9) l=1 where xl ∈ R represents the l-th component of x = (x1 , x2 , . . . , xd )T . Notice that the kernels can have either finite or infinite support. In the finite case, we omit the truncation subscript and bounding box for simplicity. According to Equation (2.5) we know that the separable multivariate kernel K is only valid for x ∈ support{K(·)} and values are zero outside the support. Similarly, the first order partial derivatives of K can be written as d Y ∂K (x) = k 0 (xc ) k(xl ), ∂xc l=1 (2.10) l6=c where x, and ∂K ∂xc (x) is the first order partial derivative of the K with respect xc , the c-th component of k 0 (xc ) is the first order derivative of the univariate kernel function k(xc ). The second order 10 CHAPTER 2. KERNEL SMOOTHING partial derivatives of K is d Q k 00 (xc ) k(xl ), l=1 l6=c ∂K (x) = d Q ∂xr ∂xc 0 (x )k 0 (x ) k k(xl ), r c l=1 l6=c r = c, , (2.11) r 6= c. l6=r where ∂K ∂xr ∂xc (x) is the seconder order partial derivative of the K with respect to xr and xc , and k 00 (xc ) is the second order derivative of the univariate kernel function k(xc ). The above definition of the first and second order partial derivatives of kernel K can be extended to higher order case. Given a mutiset N = {n1 , . . . , nr | ni ∈ 1, . . . , d, i ∈ 1, . . . , r}, then the r-th order partial derivative of K with respect to xn1 , . . . , xnr can be given as d Y ∂rK (x) = k (N (i)) (xi ), ∂xn1 , . . . , ∂xnr (2.12) i=1 where N (i) denotes the number of element of value i in set N , k (N (i)) is the N (i)-th order derivative of the univariate kernel function k. 2.4 Bandwidth In common with all smoothing problems, the most important factor is to determine the amount of smoothing. For kernel density estimators, the single most important factor is the bandwidth since it controls the amount and orientation of the smoothing. 2.4.1 Types of Bandwidth For the univariate case, the bandwidth h is a scalar. If the standard normal density function is used to approximate univariate data, and the underlying density being estimated is Gaussian then it can be shown that the optimal choice for h is [22] h=( 4σ̂ 5 1/5 ) 3n (2.13) For the multivariate case, the bandwidth H is a matrix. The type of orientation of the kernel function is controlled by the parameterization of the bandwidth matrix. There are respectively three main classes of parameterization [23]: 11 CHAPTER 2. KERNEL SMOOTHING 0.45 0.05 0.1 0.35 reference 0.4 0.35 f^(x) 0.3 0.25 0.2 0.15 0.1 0.05 0 -3 -2 -1 0 1 2 3 x F IGURE 2.5: Univariate kernel density estimates of different bandwidths. • the class of all symmetric, positive definite matrices: h21 h12 . . . h1n h12 h2 . . . h2n 2 H= . .. .. .. .. . . . 2 h1n h2n . . . hn (2.14) • the class of all diagonal, positive definite matrices: h21 0 0 h2 2 dgH = . .. .. . 0 0 (2.15) ... 0 0 .. . 2 . . . hn ... .. . • the class of all positive constants times the identity matrix: h2 0 . . . 0 0 h2 . . . 0 h2 I = . . . . .. .. . . .. 0 0 . . . h2 (2.16) The first class defines a full bandwidth matrix, which is the most general bandwidth type. It allows the kernel estimator to smooth in any direction whether coordinate or not. The second class defines 12 CHAPTER 2. KERNEL SMOOTHING the diagonal matrix parameterization, which is the most commonly used one. A diagonal matrix bandwidth allows for different degrees of smoothing along each of the coordinate axis. The third class h2 I allows the same smoothing to be applied to every coordinate direction, which is too restrictive for general use. The visualization of the scaled kernel functions using these different classes of bandwidths is given in Figure 2.6. It is worth mentioning that, for a bivariate bandwidth matrix, the full bandwidth matrix of the first class can also be parameterized as 2 h1 ρ12 h1 h2 H= ρ12 h1 h2 h22 (2.17) 3 3 2 2 2 1 1 1 0 0 0 y 3 y y where ρ12 is the correlation coefficient, which can be used as a measure of orientation. -1 -1 -1 -2 -2 -2 -3 -3 -2 -1 0 1 2 -3 -3 3 x -2 -1 0 1 2 x 3 -3 -3 -2 -1 0 1 2 3 x F IGURE 2.6: Comparison of three bandwidth matrix parametrization classes. Left: positive scalar times the identity matrix. Center: all diagonal, positive definite matrices. Right: all symmetric, positive definite matrices. 2.4.2 Variable Bandwidth So far, the bandwidths we’ve used in kernel density estimators are f ixed, which means an unified bandwidth is used for every testing point xi , i = 1, . . . , m and training point X j , j = 1, . . . , n. In this section, we will generalize these fixed bandwidth estimators to variable bandwidth estimators. There are two main classes of variable bandwidth estimators fˆ(xi ; H) = n−1 n X j=1 and fˆ(xi ; Ω) = n−1 n X j=1 KH(xi ) (xi − X j ), i = 1, . . . , m, j = 1, . . . , n, (2.18) KΩ(X j ) (xi − X j ), i = 1, . . . , m, j = 1, . . . , n. (2.19) 13 CHAPTER 2. KERNEL SMOOTHING where functions H(·) and Ω(·) are bandwidth functions. They are considered to be non-random functions, in the same way as we consider a single bandwidth to be a non-random number or matrix. The first kernel density estimator is called the balloon kernel density estimator. Its bandwidths are different at each testing point xi , i = 1, . . . , m. The second kernel density estimator is called the sample point kernel density estimator. Its bandwidths are different at each training point X j , j = 1, . . . , n. In this thesis, we only covers the sample point kernel density estimators. We won’t cover balloon kernel density estimators for two reasons. First, balloon estimators typically do not integrate to 1 so they are not true density functions, a result from focusing on estimating locally rather than globally [24]. Second, balloon estimators are generally less accurate than sample point estimators [25, 26]. 0.8 0.7 0.6 f^(x) 0.5 0.4 0.3 0.2 0.1 0 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 x F IGURE 2.7: Univariate sample point kernel density estimate: solid line kernel density estimate, dashed lines individual kernels. For the sample point kernel density estimators, there are usually two choices for the bandwidth function Ω. One commonly used form is Ω(X j ) = h2 f (X j )−1 I, j = 1, . . . , n, (2.20) where h is a constant. Using the reciprocal of f leads to an O(h4 ) bias rather than the usual O(h2 ) bias for fixed bandwidth estimators [26]. This form of the bandwidth appeals intuitively since it states that the smaller bandwidths should be used in those parts of the data set with high density of points, which is controlled by the value of f , and larger bandwidths in parts with lower density. This combination of small bandwidths near the modes and large bandwidths in the tails should be 14 CHAPTER 2. KERNEL SMOOTHING able to detect fine features near the former and prevent spurious features in the latter. One possible solution for finding the estimate of the bandwidth function Ω is to use a pilot estimate fˆ to give Ω̂(X j ) = hfˆ(X j )−1/2 . The other choice of Ω is to use the k-nearest neighbor function of X j [27]. The k-nearest neighbor function is defined as a symmetric positive definite second order covariation matrix associated with the neighborhood of X j . It can be written as C(X j ) = n−1 k nk X (X j − X jk )(X j − X jk )T , j = 1, . . . , n, (2.21) k=1 where nk = dn−γ e is the number of neighbors, and X jk denotes k-th nearest neighbor of X j . Here, nk is chosen to be significantly smaller than the number of samples, but large enough to reflect the variations. The parameter γ depends on the dimension of the space and sparsity of the data points. Thus, the bandwidth function Ω can be given as Ω(X j ) = σ 2 C(X j ), j = 1, . . . , n, (2.22) where σ is the scalar kernel width. According to the notations from Equation (2.4), and allowing the scaled kernels to have different weights at each training point, the variable bandwidth kernel density estimator can be written as fˆ(xi ; S j , ωj ) = n X j=1 ωj KS j (xi − X j ), i = 1, . . . , m, j = 1, . . . , n, (2.23) where wj , j = 1, . . . , n is the weight of the scaled kernel at training point X j and S j is the scale. For simplicity, we write Ωj and C j instead of Ω(X j ) and C(X j ). The variable bandwidth and the T scale is related by Ω−1 j = S j S j . We can extract S i by utilizing the eigendecomposition of C j Ωj = σ 2 Qj Λj QTj , j = 1, . . . , n, (2.24) where the columns elements of Qj and diagonal elements of Λj are the eigenvectors and eigenvalues of C j . Therefore, the scale matrix S j can be written as −1/2 S j = σ −1 Λj 2.5 QTj , j = 1, . . . , n. (2.25) Kernel Density Derivative Estimation Before considering the r-th derivative of a multivariate density, we first introduce the notation of r-th derivatives of a function [28, 29]. From a multivariate point view, the r-th derivative of a function is 15 CHAPTER 2. KERNEL SMOOTHING understood as the set of all its partial derivatives of order r, rather than just one of them. All these r-th partial derivatives can be neatly organized into a single vector as follow: if f is a real d-variate r density function and x = (x1 , . . . , xd ), then we denote D⊗r f (x) ∈ Rd the vector containing all the partial derivatives of order r of f at x, arranged so that we can formally write D⊗r f = ∂f , (∂x)⊗r (2.26) where D⊗r is the r-th Kronecker power [30] of the operator D. Thus we write the r-th derivative of f as a vector of length dr . Notice that, using this notation, we have D(D⊗r f ) = D⊗(r+1) f . Also, the gradient of f is just D⊗1 f and the Hessian ∇2 f = ∂2f ∂x∂xT is such that vec ∇2 f = D⊗2 f , where vec denotes the vector operator [31]. According to the previous notation we can then write the r-th kernel density derivative estimator D⊗r f as D ⊗r fˆ(xi ; S j , ωj ) = = n X j=1 n X j=1 ωj D⊗r KS j (xi − X j ) (2.27) ⊗r ωj |S j |S ⊗r j D K(S j (xi − X j )), i = 1, . . . , m, j = 1, . . . , n, Here, we follow the definition in Equation (2.4) and Equation (2.23), where xi is the i-th data of the test set, X j is the j-th data of the training set, S j is the variable scale, KS j is the scaled kernel and ωj is the weight of the r-th derivative of the scaled kernel. In this thesis, we will mainly focus on the first and second derivatives of the kernel density function because they have full potential for applications and are crucial to identify significant features of the distribution [32]. The first order derivatives can be given from the kernel gradient estimator ∇fˆ(xi ; S j , ωj ) = n X j=1 ωj ∇KS j (xi − X j ), i = 1, . . . , m, j = 1, . . . , n, (2.28) where ∇ is the column vector of the d first-order partial derivatives and ∇KS (x) = |S|S T ∇K(Sx). (2.29) Similarly, the second order derivatives can be given from the kernel curvature estimator ∇2 fˆ(xi ; S j , ωj ) = n X j=1 ωj ∇2 KS j (xi − X j ), i = 1, . . . , m, j = 1, . . . , n, (2.30) where ∇2 denotes the matrix of all second-order partial derivatives and ∇2 KS (x) = |S|S T ∇2 K(Sx)S. 16 (2.31) CHAPTER 2. KERNEL SMOOTHING It should be pointed out that the kernel curvature estimator is actually the estimator for the Hessian matrix of the density function. 17 Chapter 3 Vesselness Measure The vesselness measure intuitively describe the likelihood of a point being part of a vessel. It is not reliable to judge whether a point belongs to a vessel or not only based on the point’s intensity. Because the analysis of vesselness relies on some structural information such as local extrema, valleys, ridges or saddle points, which can only be given from the derivatives of the intensity function. In this chapter, we will first introduce the basic knowledges of gradients and Hessian matrices and their relation with structural features, then we give three different methods to find the gradients and Hessian matrices from images, finally we provide two popular algorithms for vesselness measure. 3.1 Gradients and Hessian Matrices of Images Gradients and Hessian matrices are crucial in finding structural information from images. In this section we will start from the definition of the gradient and Hessian matrix, then introduce their mathematical properties and finally discuss how to extract structural information from image gradients and Hessian matrices. 3.1.1 Gradient Given a differentiable, scalar-valued function f (x), x = (x1 , . . . , xn )T of standard Cartesian coordinates in Euclidean space, its gradient is the vector whose components are the n partial derivatives of 18 CHAPTER 3. VESSELNESS MEASURE f . It can be written as ∇f (x) = ∂ f (x) = ∂x ∂f ∂x1 .. . ∂f ∂xn . (3.1) In mathematics, the gradient points in the direction of the greatest rate of increase of the function and its magnitude is the slope of the graph in that direction. An example is illustrated in Figure 3.1. On the left, we constructed a bivariate Gaussian function f (x, y) = 1 −0.5(x2 +y 2 ) . 2π e Its gradients as well as its contours are given in the right. Here, each blue arrow represents a gradient vector of f at its current location. The direction of the arrow denotes the direction of the gradient vector and the length of the arrow denotes its magnitude. We can see that all the blue arrows point to the center where the function f reaches its peak value. 2 1.5 0.2 0.5 0.1 y f (x; y) 1 0.15 0.05 0 -0.5 0 2 -1 1 2 y -1.5 1 0 0 -1 -1 -2 -2 -2 -2 x -1 0 1 2 x F IGURE 3.1: Gradient of the standard Gaussian function. Left: Standard bivariate Gaussian function. Right: Gradients (blue arrows) of the Standard 2D Gaussian function. For image processing and computer vision, the gradient of an image is defined the same way as a mathematical gradient, except that the f is now an image intensity function I. Since an image is usually either 2D or 3D, then an image gradient can be written as g(x, y) = ∂I ∂x ∂ ∂y or g(x, y, z) = ∂I ∂x ∂I ∂y ∂I ∂z . (3.2) where I(x, y) and I(x, y, z) are the image intensity function for 2D and 3D respectively. For a 2D 19 CHAPTER 3. VESSELNESS MEASURE image, the magnitude and direction of the gradient vector at point (x0 , y0 ) is s ∂ ∂ |g(x0 , y0 )| = ( I(x0 , y0 ))2 + ( I(x0 , y0 ))2 ∂x ∂y (3.3) and θ = atan( ∂ ∂ I(x0 , y0 ), I(x0 , y0 )). ∂y ∂x (3.4) F IGURE 3.2: Image gradient. On the left, an intensity image of a cameraman. In the center, a gradient image in the x direction measuring horizontal change in intensity. On the right, a gradient image in the y direction measuring vertical change in intensity. Usually, the intensity function I(x, y) or I(x, y, z) of digital image is not given directly. It is only known at discrete points. Therefore, to get its derivatives we assume that there is an underlying continuous intensity function which has been sampled at the image points. With some additional assumptions, the derivative of the continuous intensity function can be approximated as a function on the sampled intensity function, i.e., the digital image. Approximations of these derivative functions can be defined at varying degrees of accuracy. We will discuss them in details in Section 3.2. 3.1.2 Hessian Suppose f : Rn → R is a function taking a vector x = (x1 , ..., xn )T ∈ Rn and outputting a scalar f (x) ∈ R. If all second order partial derivatives of f exist, then the Hessian matrix of f is an n × n square matrix, which is defined as follows: ∇2 f (x) = ∂2f 2 ∂x2 1 ∂ f ∂x2 ∂x1 ∂ f (x) = ∂xT x .. . ∂2f ∂xn ∂x1 20 ∂2f ∂x1 ∂x2 ∂2f ∂x22 ... .. . ... .. . ∂2f ∂xn ∂x2 ... ∂2f ∂x1 ∂xn ∂2f ∂x2 ∂xn .. . ∂2f ∂x2n , (3.5) CHAPTER 3. VESSELNESS MEASURE where ∂2f ∂xi ∂xj is the second order partial derivative of f with respect to the variable xi and xj . Specifically, if f has continuous second partial derivatives at any given point in Rn , then ∀i, j ∈ {1, 2, . . . , n}, ∂2f ∂xi ∂xj = ∂2f ∂xj ∂xi . Thus, the Hessian matrix of f is a symmetric matrix. This is true in most “real-life” circumstances. For a digital image, the Hessian matrix is defined in the same way, except that function f is now the image intensity function I, which is usually a 2D or 3D discrete function. By convention, we use H to denote Hessian matrices of an image, and it can be written as 2 ∂ I 2 2 ∂x2 ∂ I ∂ I 2 2 ∂x∂y ∂ I H(x, y) = ∂x2 and H(x, y, z) = ∂y∂x ∂ I ∂2I ∂y∂x ∂y 2 ∂2I ∂z∂x ∂2I ∂x∂y ∂2I ∂y 2 ∂2I ∂z∂y ∂2I ∂x∂z ∂2I ∂y∂z ∂2I ∂z 2 (3.6) A symmetric n × n Hessian matrix can be decomposed into the following form using eigenvalue decomposition, H = QΛQT , (3.7) where Q is the square matrix whose i-th column is the eigenvector q i of H and Λ is the diagonal Q and Λ can be written as ... 0 ... 0 (3.8) .. .. . . 0 . . . λn matrix whose diagonal elements are the corresponding eigenvalues. λ1 0 0 λ2 h i Q = q 1 , q 2 , . . . , q n and Λ = . .. .. . 0 In 2D case, eigenvalues of H can be visualized by constructing an ellipsoid v T Hv = 1 (3.9) where v = (x, y)T . By performing eigenvalue decomposition to H, so that Λ is a diagonal matrix and Q is a rotation (orthogonal) matrix h i λ1 0 and Q = q , q . Λ= 1 2 0 λ2 (3.10) v T QΛQT v = (QT v)T Λ(QT v) = 1 (3.11) We have, Let v 0 = QT v = (x0 , y 0 )T , we get v 0T Λv 0 = λ1 x02 + λ2 y 02 = x02 ( √1λ )2 1 21 + y 02 ( √1λ )2 2 =1 (3.12) CHAPTER 3. VESSELNESS MEASURE We can see that Equation 3.12 is a standard ellipsoid equation in coordinates (x0 , y 0 ). Its semiprincipal axes are √1 λ1 and √1 . λ2 Since Q is a rotation matrix, thus Equation 3.9 is actually a rotated ellipsoid in coordinates (x, y). y 1 p ⇥ 1 p x y ⇤ H x y =1 2 1 x ⇥ x y ⇤ ⇤ x y =1 F IGURE 3.3: Visualized Eigenvalues with ellipsoid. Intuitively, for 2D images, if a pixel is close to the centerline of a vessel, it should satisfy the following properties • one of the eigenvalues λ1 should be very close to zero; • the absolute value of the other eigenvalue should be a lot greater than zero, λ2 0. For 3D images, let λk be the eigenvalue with the k-th smallest magnitude, i.e. |λ1 | ≤ |λ2 | ≤ |λ3 |. Then, an ideal tubular structure in 3D image satisfy • λ1 should be very close to zero; • λ2 and λ3 should be of large magnitude and equal sign (the sign is an indicator of brightness/darkness). The respective eigenvectors point out singular directions: q 1 indicates the direction along the vessel (minimum intensity variation). q 2 and q 3 form a base for the orthogonal plane. 3.2 Finding 1st and 2nd Order Derivatives From Images Typically, the intensity function of a digital image is only known at evenly distributed discrete places. Thus, instead of the continuous intensity function I(x, y), I(n1 , n2 ) is usually used to refer 22 CHAPTER 3. VESSELNESS MEASURE 2D 3D orientation pattern λ1 λ2 λ1 λ2 λ3 N N N N N noisy, no preferred direction L L H- plate-like structure (bright) L L H+ plate-like structure (dark) L H- L H- H- tubular structure (bright) L H+ L H+ H+ tubular structure (dark) H- H- H- H- H- blob-like structure (bright) H+ H+ H+ H+ H+ blob-like structure (dark) TABLE 3.1: Possible orientation patterns in 2D and 3D images, depending on the value of the eigenvalues λk (H=high, L=low, N=noisy, usually small, +/- indicate the sign of the eigenvalue). The eigenvalues are ordered: |λ1 | ≤ |λ2 | ≤ |λ3 | [3]. an image at discrete points. Here, (n1 , n2 ) is the indices of a pixel on the image. It is related to (x, y) by (x, y) = (n1 ∆n1 , n2 ∆n2 ), where ∆n1 and ∆n2 is the distance between two adjacent pixels in horizontal and vertical direction. To compute the derivatives of an image, we need to the use pixels from I(n1 , n2 ) to approximate the underlying I(x, y) and its derivatives. In this section, three approximations are introduced: Gradient Operator, Gaussian Smoothing, and Kernel Density Derivative Estimation. 3.2.1 Gradient Operator For a 2D image, the partial derivative of its continuous intensity function I(x, y) in the x direction is defined as Ix (x, y) = I(x + h2 , y) − I(x − h2 , y) ∂ I(x, y) = lim . h→0 ∂x h (3.13) Thus, for a constant h, Equation (3.13) can be approximated by following I(x + h2 , y) − I(x − h2 , y) Iˆx (x, y) = , h and the error of the approximation is O(h2 ) [33]. In the discrete case, if we let (3.14) h 2 = ∆n, then the approximation of I(n1 , n2 ) can be written as I(n1 + 1, n2 ) − I(n1 − 1, n2 ) Iˆx (n1 , n2 ) = . 2∆n 23 (3.15) CHAPTER 3. VESSELNESS MEASURE Usually, the sampling factor 1 2∆n is ignored, since it is constant throughout the image. Therefore, the approximation of Ix (n1 , n2 ) can be simplified by writing Iˆx (n1 , n2 ) = I(n1 + 1, n2 ) − I(n1 − 1, n2 ). (3.16) Similarly, the approximation of Iy (n1 , n2 ) can be written as Iˆy (n1 , n2 ) = I(n1 , n2 + 1) − I(n1 , n2 − 1). (3.17) Let h1 (n1 , n2 ) = δ(n1 + 1, n2 ) − δ(n1 − 1, n2 ) and h2 (n1 , n2 ) = δ(n1 , n2 + 1) − δ(n1 , n2 − 1), where δ(n1 , n2 ) is defined as δ(n1 , n2 ) = 1, 0, n 1 = n2 = 0 , (3.18) otherwise then Equation (3.16) and (3.17) can be written as Iˆx (n1 , n2 ) = I(n1 , n2 ) ∗ h1 (n1 , n2 ) (3.19) Iˆy (n1 , n2 ) = I(n1 , n2 ) ∗ h2 (n1 , n2 ), (3.20) Here h1 and h2 is called gradient operators. Usually, the gradient operators is written in the form of matrices −1 h i h1 (n1 , n2 ) = ; h (n , n ) = −1, 0, 1 ; 0 1 1 2 1 (3.21) The approximated second order derivatives can be derived from the first order derivatives directly Iˆxx (n1 , n2 ) = Ix (n1 , n2 ) ∗ h1 (n1 , n2 ) ≈ (I(n1 , n2 ) ∗ h1 (n1 , n2 )) ∗ h1 (n1 , n2 ) (3.22) = I(n1 , n2 ) ∗ h1 (n1 , n2 ) ∗ h1 (n1 , n2 ) Similarly, Iˆxy (n1 , n2 ) = I(n1 , n2 ) ∗ h1 (n1 , n2 ) ∗ h2 (n1 , n2 ) (3.23) Iˆyy (n1 , n2 ) = I(n1 , n2 ) ∗ h2 (n1 , n2 ) ∗ h2 (n1 , n2 ) (3.24) 24 CHAPTER 3. VESSELNESS MEASURE The discussion above can be easily applied to higher dimensional case. Let Ixi denotes the first order derivative of I with respect to xi , then its approximation is given as Iˆxi (n1 , . . . , nd ) = I(n1 , . . . , nd ) ∗ hi (n1 , . . . , nd ) (3.25) where hi (n1 , . . . , nd ) = δ(n1 , . . . , ni + 1, . . . , nd ) − δ(n1 , . . . , ni − 1, . . . , nd ) and d is the dimension of the image. Similarly, the second order derivative Ixi xj can be written as Iˆxi xj (n1 , . . . , nd ) = I(n1 , . . . , nd ) ∗ hi (n1 , . . . , nd ) ∗ hj (n1 , . . . , nd ) (3.26) This method is computationally efficient due to the simple structure of the gradient operators. But since the estimation of the derivatives only involves adjacent pixels, which contains limited information of the neighborhood, this method is not accurate. Especially, this method can’t provide the accurate derivative information for images of large scale. 3.2.2 Gaussian Smoothing For a 2D image, from the sampling theorem[34] we know the continuous intensity function I(x, y) can be ideally reconstructed from the discrete image function I(n1 , n2 ) by I(x, y) = n X n X k1 =−n k2 =−n I(k1 , k2 )K(x − k1 ∆n1 , y − k2 ∆n2 ), (3.27) where K is a sinc like function K(x, y) = sin(πx/∆n1 ) sin(πy/∆n2 ) . πx/∆n1 πx/∆n2 (3.28) Here, ∆n1 and ∆n2 are the sampling intervals. However, K decays proportionally to 1/x and 1/y, which is a rather slow rate of decay. Consequently, only values that are far away from the origin can be ignored in the computation. In other words, the summation limit n must be large, which is a computationally undesirable state of affairs. In addition, if there is aliasing, the sinc function will amplify its effects, since it combines a large number of unrelated pixel values. Instead, a Gaussian function, which passes only frequencies below a certain value and has a small support in the spatial domain, can be a good replacement. Thus, we have ˆ y) = I(x, n X n X k1 =−n k2 =−n I(k1 , k2 )G(x − k1 ∆n1 , y − k2 ∆n2 ), 25 (3.29) CHAPTER 3. VESSELNESS MEASURE where G is a Gaussian function at scale h 1 − x2 +y2 2 e 2h . 2πh2 G(x, y) = (3.30) Thus, the approximated first order derivatives and second order derivatives of I(x, y) with respect to x can be given as n X ∂ ˆ Iˆx (x, y) = I(x, y) = ∂x ∂2 ˆ Iˆxx (x, y) = I(x, y) = ∂x2 n X k1 =−n k2 =−n n n X X I(k1 , k2 )Gx (x − k1 ∆n1 , y − k2 ∆n2 ), (3.31) I(k1 , k2 )Gxx (x − k1 ∆n1 , y − k2 ∆n2 ), (3.32) k1 =−n k2 =−n where Gx (x, y) and Gxx (x, y) are first and second order derivatives of Gaussian in direction x x − x2 +y2 2 ∂ G(x, y) = − e 2h , ∂x 2πh4 2 2 ∂2 1 x2 − x +y 2h2 . Gxx (x, y) = G(x, y) = ( − 1)e ∂x2 2πh4 h2 Gx (x, y) = (3.34) 0.1 @2 @x2 G(x; y) @ @x G(x; y) 0.1 (3.33) 0.05 0 0.05 0 -0.05 -0.1 -0.05 -0.15 -0.1 4 -0.2 4 2 2 4 y 4 2 0 0 -2 -2 -4 -4 2 0 y x 0 -2 -2 -4 -4 x F IGURE 3.4: Derivatives of Gaussian filters. Left: the first order derivative of the bivariate Gaussian function with respect to x. Right: the second order derivative of the bivariate Gaussian function with respect to x. Sample Iˆx (x, y) and Iˆx (x, y) the same way as I(n1 , n2 ), we have Iˆx (n1 , n2 ) = n X n X k1 =−n k2 =−n I(k1 , k2 )Gx (n1 − k2 , n2 − k2 ) = I(n1 , n2 ) ∗ Gx (n1 , n2 ), 26 (3.35) CHAPTER 3. VESSELNESS MEASURE Iˆxx (n1 , n2 ) = n X n X k1 =−n k2 =−n I(k1 , k2 )Gxx (n1 − k2 , n2 − k2 ) (3.36) = I(n1 , n2 ) ∗ Gxx (n1 , n2 ), where Gx (n1 , n2 ) and Gxx (n1 , n2 ) are sampled Gaussian derivative functions. Therefore, the first and second order derivatives of an image can be easily got by convolving it with the corresponding Gaussian derivative filters. We call this the Gaussian Smoothing. In general, for a d dimensional image I(n1 , . . . , nd ), its first and second order derivatives can be given as Iˆxi (n1 , . . . , nd ) = I(n1 , . . . , nd ) ∗ Gxi (n1 , . . . , nd ) (3.37) Iˆxi xj (n1 , . . . , nd ) = I(n1 , . . . , nd ) ∗ Gxi xj (n1 , . . . , nd ), (3.38) where Gxi is the first order Gaussian derivative filter with respect to xi and Gxi xj is the second order Gaussian derivative filter with respect to xi and xj . Since the computation uses convolution only and the size of the Gaussian derivative filter is usually small, the Gaussian smoothing method is also computationally efficient. Unlike the Gradient operator which calculate the derivatives at the finest scale, the smoothing of this method is controlled by the scale parameter h, which can decide the degree of information that should be used to calculate the derivatives. The problem of this method is that the smoothing is only performed along the coordinates and the choice of a proper scale h is hard. 3.2.3 Kernel Density Derivative Estimation We’ve discussed kernel density derivative estimation in Section 2.5. We know that the approximation of the derivatives of a function f : Rd → R can be given from a set of sample data Xi , i = 1, . . . , m by Equation (2.28) and Equation (2.30). To get the derivatives of a 2D image I(x, y), the same idea can be applied. Let the weight ωj = I(k1 , k2 ) and rearrange the indices of the summation operator, we can rewrite Equation (2.28) and (2.30) as XX ˆ y) = ∇I(x, I(k1 , k2 )∇KS k1 k2 (x − Xk1 , y − Yk2 ) k1 2ˆ ∇ I(x, y) = k2 XX k1 (3.39) k2 I(k1 , k2 )∇2 KS k1 k2 (x − Xk1 , y − Yk2 ) (3.40) ˆ y) is the estimated gradient of where I(k1 , k2 ) is the intensity of the image at pixel (k1 , k2 ), ∇I(x, ˆ y) is the estimated Hessian of the image, S k k is the scale matrix, K(.) is the the image, ∇2 I(x, 1 2 kernel function and (Xk1 , Yk2 ) = (k1 ∆n1 , k2 ∆n2 ) is the location of the pixel (k1 , k2 ) on the image. 27 CHAPTER 3. VESSELNESS MEASURE ˆ y) and ∇2 I(x, ˆ y) the same way as I(n1 , n2 ), we have Sample ∇I(x, XX ˆ 1 , n2 ) = I(k1 , k2 )∇KS k1 k2 (n1 − k1 , n2 − k2 ), ∇I(n k1 ˆ 1 , n2 ) = ∇2 I(n XX k1 (3.41) k2 k2 I(k1 , k2 )∇2 KS k1 k2 (n1 − k1 , n2 − k2 ). (3.42) Here, ∇KS k1 k2 (n1 , n2 ) is the sampled kernel gradient and ∇2 KS k1 k2 (n1 , n2 ) is the sampled kernel Hessian. Similarly, the above equations can be easily extended to higher dimensional case X ˆ ∇I(n) = I(k)∇KS k (n − k), (3.43) k ˆ ∇2 I(n) = X k I(k)∇2 KS k (n − k). (3.44) where n = (n1 , . . . , nd ) and k = (k1 , . . . , kd ). The kernel density derivative estimators can give the most accurate estimation of the derivatives of the image. Because, it is able to smooth the image locally, which means it can decide the smoothing direction and scale level for each pixel accordingly. But the problem of this method is that it is too computationally insensitive. To solve this problem, we proposed a high performance solution, which based on the GPU CUDA framework, in Chapter 5. 3.3 Frangi Filtering Frangi filter[3], developed by Frangi et al. in 1998, is a popular method in highlighting tubular structures in images. It uses the eigenvalues of the Hessian matrices obtained from an image to analysis the likelihood of a pixel being on the tubular structure. For a 3D image, assume we’ve obtained a Hessian H matrix at voxel (n1 , n2 , n3 ) using any of the methods discussed in Section 3.2, then by performing eigenvalue decomposition to H and sorting the resulting eigenvalues, we have |λ1 | ≤ |λ2 | ≤ |λ3 |. (3.45) From the discussion in Section 3.1.2, we know that if a voxel is on the tubular-like structure, the eigenvalues satisfy |λ1 | ≈ 0, |λ1 | |λ2 |, and λ2 ≈ λ3 . Combining these constrains with the relations in Table 3.1, we can define the following three dissimilarity measures: • To distinguish between blob-like and nonblob-like structures, we define |λ1 | RB = p . |λ2 λ3 | 28 (3.46) CHAPTER 3. VESSELNESS MEASURE This ratio attains its maximum for a blob-like structure, which satisfies |λ1 | ≈ |λ2 | ≈ |λ3 |, and is close to zero whenever λ1 ≈ 0 or λ1 and λ2 tend to vanish. • To distinguish between plate- and line-like structures, we define RA = |λ2 | . |λ3 | (3.47) RA → 0 implies a plane-like structure and RA → 1 implies a line-like structure. • To distinguish between background (noise) and foreground, we define q S = λ21 + λ22 + λ23 . (3.48) This measure will be low in the background where no structure is present and the eigen- values are small for the lack of contrast. Combining the measures above, we can define a vesselness function as following 0, if λ2 > 0 or λ3 > 0, V(n1 , n2 , n3 ) = 2 RB 2 RA 2 S (1 − e− 2α2 )e− 2β2 (1 − e− 2c 2 ), otherwise. (3.49) where α, β and c are the thresholds which control the sensitivity of the measures RA , RB and S. Similarly, for 2D images the vesselness function can be given as 0, if λ2 > 0, V(n1 , n2 ) = 2 RB 2 S e− 2β2 (1 − e− 2c 2 ), otherwise. Here, RB = |λ1 | |λ2 | is the blobness measure in 2D and S = (3.50) p λ21 + λ22 is the backgroundness measure. Note that Equation (3.49) and (3.50) are given for bright tubular-like structures . For dark objects the conditions should be reversed. 3.4 Ridgeness Filtering Another approach to extract tubular structure from images is to use ridgeness filter [35, 36]. In ridgeness filtering, a tubular structure can be viewed as a ridge or principal curve of the continuous intensity function I(x) : Rd → R. A rigde is defined as a set of curves whose points are local maxima of the function in at least one direction. This more mathematically rigorous definition of a tubular structure provides us more mathematical tools to analyze the likelihood of a point being on 29 CHAPTER 3. VESSELNESS MEASURE F IGURE 3.5: Lef t: Original X-Ray vessel image. Right: enhanced vessel image using Frangi filter. the ridge. What’s more, unlike a Frangi filter which only uses the information from a local Hessian matrix H(x) of images, the ridgness filter combines both local gradient and Hessian to measure the ridgeness. Let q i and λi be the i-th eigenvector and eigenvalue pair of the Hessian matrix H(x) of I(x) such that |λ1 | ≤ . . . ≤ |λd |. In general, a point is on the k-dimensional ideal ridge structure iff it satisfies the following conditions • the gradient g(x) is collinear with the first k eigenvectors, i.e. g(x) k q i (x), i = 1, . . . , k, and is orthogonal to d − k eigenvectors, i.e. g(x)T q i (x) = 0, i = k + 1, . . . , d; • λk+1 , . . . , λd have the same sign. • |λk | ≈ 0; Thus, this point on the ridge is the local maximum of the function in the subspace spanned by the d−k, i.e. S⊥ = span(q k+1 , . . . , q d ) and the tangential space is spanned by the remaining k eigenvectors, Sk = span(q 1 , . . . , q k ). If we consider the inner product between g(x) and H(x)g(x), we can get T g(x) H(x)g(x) = = d X i=1 k X λi g(x)T qi (x)(g(x)T qi (x))T λi (g(x)T qi (x))2 + i=1 d X i=k+1 30 (3.51) λi (g(x)T qi (x))2 . CHAPTER 3. VESSELNESS MEASURE Note that since the eigenvalues are sorted in a descend order based on their magnitude, then the third condition is true for all the first k eigenvalues, i.e. |λi | ≈ 0, i = 1, . . . , k. Hence, we can find that the inner product between g(x) and H(x)g(x) is close to zero for a point x on the ridge. Therefore, a measure for being on the ridge can be formulated in terms of the inner product between g(x) and H(x)g(x), ζ(x) = abs( g(x)T H(x)g(x) ), kH(x)g(x)kkg(x)k (3.52) where abs is the absolute operator. This function is bounded between 0 and 1, due to the normalization factor in the denominator. 31 Chapter 4 GPU Architecture and Programming Model 4.1 GPU Architecture A GPU is connected to a host through PCI-E bus. It has its own device memory, which usually can be up to several gigabytes in current GPU architecture. A GPU manages its device memory independently and it can’t work on host memory directly. Typically, a data in host memory need to be transferred to GPU device memory through programmed DMA, so that it can be read and written by GPU. The device memory on GPU supports very high data bandwidth with relatively high latency. Since most data access on GPU begins in device memory, it is very important for programmers to leverage the high bandwidth of device memory to achieve peak throughput of GPU. NVIDA GPUs consist of several streaming multiprocessors (SMs), each of which works independently from each other. Each multiprocessor contains a group of CUDA cores (processors), load/store units or special function units (SFUs). Each core is capable of performing integer and floating point operations. Multiprocessors create, manage, schedule, and execute threads in groups of 32 parallel threads called warps. A warp is a minimum unit of execution on GPU. When a multiprocessor is issued with a block of threads, it will first partition them into warps and then schedule those warps by a warp scheduler for execution. All the threads in a warp, if there is no divergence, execute one common instruction at a time. Such architecture is called SIMT (Single Instruction, Multiple Threads). Each SM also contains several warp schedulers and instruction dispatch units, which select warps and instructions that will be executed on the SM. 32 CHAPTER 4. GPU ARCHITECTURE AND PROGRAMMING MODEL Multicore CPU Execution Queue Control SMX N Warp Scheduler … Dispatch Warp Scheduler Dispatch Dispatch Warp Scheduler Dispatch Dispatch Warp Scheduler Dispatch Dispatch Dispatch core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST SFU core Warp DPScheduler Unit LD/ST SFU Dispatch Dispatch Dispatch Dispatch Dispatch Dispatch Dispatch core Warp core core Dispatch DP Unit core core core DP Unit LD/ST Warp Scheduler Scheduler Warp Scheduler Warp Scheduler SFU SMX 1 Warp Scheduler core Scheduler coreWarpcore DP Unit Warp coreScheduler core SMX 0 core core core DP Unit Dispatch Dispatch Dispatch core core DP Unit core Dispatch core … … … … … … … … … core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST SFU SFU SFU SFU … coreDispatch core Dispatch core DPDispatch Unit LD/ST core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST SFU Dispatch core SFU core core core DP Unit core core Hardware/Software core DP Unit LD/ST SFU core core core DP Unit User core Selectable core core DP Unit LD/STCache SFU core core core DP Unit core core core DP Unit LD/ST SFU core core core DP Unit core core core DP Unit LD/ST SFU … SFU … DP Unit … core SFU LD/ST … core LD/ST DP Unit LD/ST SFU … core DP Unit … core core … core core … DP Unit … … core core core … … core DP Unit … … core DP Unit … core core … … core core … … core core … … Host Memory User Selectable Hardware/Software Cache L2 Cache DMA Device Memory F IGURE 4.1: GPU block diagram. GPU’s memory hierarchy can be divided into two categories: the off-chip memory and the on-chip memory, where the chip is referred to the multiprocessor. A off-chip memory of GPU is usually slower than on-chip memory because they are relatively far from GPU. There two types of off-chip memory: L2 cache and global memory. A L2 cache is a part of the GPU’s cache memory hierarchy. It is typically smaller than CPU’s L2 or L3 cache, but has higher bandwidth available, which makes it more suitable for throughput computing. The L2 cache is shared by all multiprocessors on GPU and it is invisible to programmers. The global memory here is referred to GPU’s device memory. But in a more precise definition, the global memory is actually only a part of device memory. A global memory is a programming concept and we will discuss it in detail later in the Programming Model sections. The global memory is also shared by all multiprocessors. There are five types of on-chip memory: register, L1 cache, shared memory, constant cache and texture cache (read-only data cache). Unlike the registers on CPU, the register file on GPU is very large. And it is the fastest on-chip memory. A L1 cache is also a part of the GPU’s cache memory hierarchy. It is has larger cache line size and lower latency than the L2 cache. As a on-chip memory, the L1 cache is only 33 CHAPTER 4. GPU ARCHITECTURE AND PROGRAMMING MODEL Device SM 1 Register File ... CUDA SM 1 Core CUDA Core … CUDA Core Register File SM 0 Shared Memory/L1 Cache Register File CUDA Core … Cache Texture Memory CUDA Core CUDA Core CUDA Core CUDA CUDA … Constant Memory Cache Core Core Shared Memory/L1 Cache ... Shared Texture Memory/L1 MemoryCache Cache Texture Constant Memory Memory Cache Cache Constant Memory Cache L2 Cache Device Memory F IGURE 4.2: GPU hardware memory hierarchy. accessible by the multiprocessor which it belongs to. And the same as the L2 cache, the L1 cache is also invisible to programmers. The shared memory is a programmable cache. It actually shares the same physical cache component with L1 cache, which make shared memory extremely fast. Typically, the shared memory/L1 cache is 100 faster than the global memory. The shared memory is fully visible to programmers. The constant cache and texture cache are used for caching the constant memory and texture memory, which we will discuss in detail in the following section. 4.2 Programming Model In this section, we are going to talk about NVIDA’s CUDA programming model. Since this is a programming model, all the concepts talked in this section will be visible by programmers. Please note that some concepts may have their physical counterpart and some concepts won’t have. 34 CHAPTER 4. GPU ARCHITECTURE AND PROGRAMMING MODEL Host Block (2, 0) Device Grid 0 Block (0, 0) Block (1, 0) Block (2, 0) Block (0, 1) Block (1, 1) Block (2, 1) Kernel 0 Grid 1 Kernel 1 Block (0, 0) Block (1, 0) Block (0, 1) Block (1, 1) Block (0, 2) Block (1, 2) Thread (0, 0) Thread (1, 0) Thread (2, 0) Thread (3, 0) Thread (0, 1) Thread (1, 1) Thread (2, 1) Thread (3, 1) Thread (0, 2) Thread (1, 2) Thread (2, 2) Thread (3, 2) Thread (0, 3) Thread (1, 3) Thread (2, 3) Thread (3, 3) Thread (0, 4) Thread (1, 4) Thread (2, 4) Thread (3, 4) F IGURE 4.3: Programming Model. In CUDA programming model, programmers write GPU through a C like functions called kernels. To distinguish it from kernels used in kernel-based density and density derivatives estimates, we will call these functions gpu − kernels. A gpu-kernel is nothing but a task that the programmer want to assign to the GPU. Usually, the task or gpu-kernel is large and can’t be executed by GPU at a time. Therefore, the gpu-kernel is divided into several equally sized task chunks called blocks. Each block will be executed on one multiprocessor. And a multiprocessor can execute several blocks at a time. A block consists a number of threads, which is the minimum task unit on GPU (recall that a warp is the minimum execution unit on GPU). All the threads in a gpu-kernel must be able to be executed independently. All the blocks in a gpu-kernel form the so-called computation grid. When a programmer want to write a gpu-kernel function, he must define the grid size and block size in advance, so that the GPU can know how assign the task to multiprocessors. From a programming model, we can view the memory hierarchy in a different perspective. In this new memory hierarchy, all the memory are visible to programmers. Even though they are called memory, they are actually the memory resources that the GPU assigns to the application or the gpu-kernel. These memory doesn’t exist physically. Instead, they are created and managed at runtime. When a gpu-kernel begins its execution on GPU, each thread will be assigned a number of dedicated registers and, if needed, a private local memory space. But a programmer should always avoid using 35 CHAPTER 4. GPU ARCHITECTURE AND PROGRAMMING MODEL too much local memory. Because the local memory is allocated in the off-chip device memory, which is much slower than the on-chip registers. Each block has shared memory visible to all threads of the block and with the same lifetime as the block. Note that the shared memory we mention here is not a physical component. It is a part of shared memory resource that assigned to this block. All threads have access to the same global memory. There are also two additional read-only memory spaces accessible by all threads: the constant and texture memory spaces. The global, constant, and texture memory spaces are optimized for different memory usages. The global memory, constant memory and texture memory are allocated in the off-chip device memory as well. They are persistent across kernel launches by the same application. Grid Block (1, 0) Block (0, 0) Shared Memory Registers Registers Thread (0, 0) Local Memory Shared Memory Registers Thread (0, 0) Thread (1, 0) Local Memory Local Memory Registers Thread (1, 0) Local Memory Global Memory Constant Memory Texture Memory F IGURE 4.4: GPU software memory hierarchy. 4.3 Thread Execution Model When a kernel is invoked, the CUDA runtime will distribute the blocks across the multiprocessors on the device and when a block is assigned to a multiprocessor, it is further divided into groups of 32 threads, which is a warp. A warp scheduler then selects available warps and issues one instruction from each warp to a group of sixteen cores, sixteen load/store units, or four special function units. CUDA’s warp scheduling mechanism will help hide instruction latency. Each instruction of a kernel may require more than a few clock cycles to execute (for example, an instruction to read from global 36 CHAPTER 4. GPU ARCHITECTURE AND PROGRAMMING MODEL memory will require multiple clock cycles). The latency of long-running instructions can be hidden by executing instructions from other warps while waiting for the result of the previous warp. Instruction Dispatch Unit Instruction Dispatch Unit Warp 8 instruction 11 Warp 8 instruction 12 Warp 8 instruction 42 Warp 8 instruction 43 Warp 8 instruction 42 Warp 8 instruction 42 ... ... time Warp Scheduler Warp 8 instruction 11 Warp 8 instruction 12 Warp 8 instruction 42 Warp 8 instruction 43 Warp 8 instruction 42 Warp 8 instruction 42 F IGURE 4.5: Warp scheduler. It is critical for a GPU to achieve high occupancy in its execution. But unlike the CPU, it is usually very hard to keep GPU busy all the time. Because there are several factors that will affect GPU’s occupancy. Those factors are maximum register number per thread, maximum thread number in a block, and shared memory size per block. Think of a multiprocessor as a container, which has limited resources such as registers, shared memory size, cores and other ALU resources. As we’ve already talked about previously, a gpu-kernel is executed block by block on multiprocessors. How many blocks can be executed on a multiprocessor is decided by the block size, i.e., the number of resources a block need to use, and the number of resources that is available on the multiprocessor. In order to achieve a high occupancy on GPU, our goal is to select a number of proper sized block to execute on multiprocessors so that most of the resources on multiprocessors are occupied. For example, the Kepler architecture supports maximum 63 registers per thread, 1024 threads per block, and 48KB shared memory per multiprocessor. 37 CHAPTER 4. GPU ARCHITECTURE AND PROGRAMMING MODEL FERMI FERMI KEPLER KEPLER GF100 GF104 GF104 GF110 Max Warps / SMX 48 48 64 64 Max Threads / SMX 1536 1536 2048 2048 Max Thread Blocks / SMX 8 8 16 16 32bit Registers / SMX 32768 32768 65536 65536 Max Registers / Thread 63 63 63 255 Max Threads / Thread Block 1024 1024 1024 1024 16KB 16KB 32KB 32KB 48KB 48KB Shared Memory 16KB 16KB Size Configurations (bytes) 48KB 48KB TABLE 4.1: Compute capability of Fermi and Kepler GPUs. 4.4 Memory Accesses Current GPU’s device memory can only be accessed via 32-byte, 64-byte or 128-byte transaction. All the memory transactions are naturally aligned. They take place at 32-byte, 64-byte or 128-byte memory segments, i.e., the address of the first byte of the memory segment must be a multiple of the transaction size. If the memory addresses are misaligned and they distribute across two memory segments rather than one, then it will take one more memory transaction to read or write the data. To make fully use of each memory transaction, memory accesses are usually coalesced by warp. When a warp executes an instruction that need to access the device/global memory, it looks at the distribution of memory addresses across the threads within it. Instead of generate a memory transaction for each thread, it coalesces the memory accesses that read/write data from the same memory segment into just one memory transaction. Typically, the more transactions are necessary, the more unused words are transferred in addition to the words accessed by the threads, reducing the instruction throughput accordingly. For example, if a 32-byte memory transaction is generated for each thread’s 4-byte access, throughput is divided by 8. Besides using memory coalescing to increase the global memory throughput, programmers can also speed up their application by reducing unnecessary global memory traffic. One way to achieve this is to use shared memory. As we’ve already known that a shared memory is nothing but a programmable L1 cache. A traditional cache is invisible to programmers and they can’t decide which data can be cached. Shared memory, on the other hand, is fully controlled by programmers. When the programmer identifies that some data is accessed repeatedly by threads in the same block, he can 38 CHAPTER 4. GPU ARCHITECTURE AND PROGRAMMING MODEL 256 128 Address … … Thread ID 0 31 F IGURE 4.6: Aligned and consecutive memory access. 256 257 128 Address … … Thread ID 0 31 F IGURE 4.7: Misaligned memory access. load those data from global/device memory into the shared memory first and then the later accesses of those data can be done through shared memory instead of global memory, which can reduce a great amount of global memories. Another advantage of using shared memory is that it can be accessed simultaneously. Shared memory is divided into equally-sized memory modules, called banks. Any memory read or write request made of n addresses that fall in n distinct memory banks can therefore be serviced simultaneously, yielding an overall bandwidth that is n times as high as the bandwidth of a single module. However, if two addresses of a memory request fall in the same memory bank, there is a bank conflict and the access has to be serialized. The hardware splits a memory request with bank conflicts into as many separate conflict-free requests as necessary, decreasing throughput by a factor equal to the number of separate memory requests. If the number of separate memory requests is n, the initial memory request is said to cause n-way bank conflicts. One exception of bank conflict is that if all threads in a warp access the same shared memory address at the same time, only one memory request will be generated and the data will be broadcasted to all the threads. We call this mechanism broadcasting. 39 Chapter 5 Algorithms and Implementations In this chapter, we present three main contributions of this thesis. In Section 5.1, we propose an algorithm which can calculate the separable multivariate kernel derivatives (SMKD) efficiently. In Section 5.2, we introduce some core functions in our kernel smoothing library. Several optimization algorithms for these functions are proposed. Finally, we design a fast k-nearest neighbors bandwidth selector in Section 5.3. 5.1 Efficient Computation of Separable Multivariate Kernel Derivative As we mentioned in Section 2.5, the implementation of Equation (2.27) requires the calculation of D⊗r K(x), which is a vector containing all the partial derivatives of order r of the kernel function K at point x. For a separable kernel, these partial derivatives are given by Equation (2.12). A brute force implementation of calculating D⊗r K(x) is to calculate these partial derivatives respectively. But this results in calculating the same set of kernel density, kernel derivative and multiplication operations repeatedly, which is clearly not computationally efficient. Let’s consider a motivating example. Assume a separable 4-variable kernel K(x) = k(x1 )k(x2 )k(x3 )k(x4 ). Its first order partial derivative with respect to x4 would be are ∂2 K(x) ∂x24 ∂ ∂x4 K(x) = k(x1 )k(x2 )k(x3 )k 0 (x4 ), similarly two second order derivatives = k(x1 )k(x2 )k(x3 )k 00 (x4 ) and ∂2 ∂x3 x4 K(x) = k(x1 )k(x2 )k 0 (x3 )k 0 (x4 ). One can observe that carrying out calculation of these derivatives respectively would yield the computation of k(x1 )k(x2 )k(x3 ) and k(x1 )k(x2 ) redundant with three and two repetitions, respectively. This redundancy grows as the number of dimensions and the derivative order increases, which leads to 40 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS significant room for optimization. Therefore, to avoid these redundant calculations, we proposed a graph-based efficient algorithm in this section. 5.1.1 Definitions and Facts Our algorithm is based on a directed acyclic graph where each node denotes a set of multivariate kernel partial derivatives and each edge denotes a univariate kernel derivative. In order to propose a well-defined and mathematically strict description of our algorithm, some required notations, definitions and facts are introduced in this section. (j) Definition 1. ki is the value of the j-th order derivative of the univariate kernel k at xi , (j) ki = k (j) (xi ). (5.1) (r) Definition 2. Nd denotes the set whose members are the unique partial derivatives of order r of Q the kernel function K(x1 , ..., xd ) = di=1 k(xi ), (r) Nd (n1 ) (n2 ) (n ) k2 . . . kd d = {k1 | d X i=1 ni = r, ni ∈ N0 }, d ∈ N+ , r ∈ N0 . (r) (5.2) (r) Definition 3. Sd is the number of the elements in set Nd , (r) (r) Sd = |Nd |. (5.3) Definition 4. The product of a scalar ω and a set A = {a1 , a2 , . . . , an } is defined using the operator ×, such that A × ω = {a1 ω, a2 ω, . . . , an ω}. (5.4) Definition 5. Define a directed acyclic graph G(V, E). Each node in V stands for a set and each edge in E has a weight. The relation between nodes and edges in G is given by Figure 5.1. Here, graph (a) contains two nodes, which stands for two sets, A and B. The edge (A, B) has a weight ω. Then, according to the graph, the relation between set A and B is B = A × ω, where the operator × is defined in Definition 4. Similarly, we can find from graph (b) that the node C is pointed by two edges from node A and B respectively. In this case, the relation between these three sets is C = (A × ωA ) ∪ (B × ωB ). (i) Fact 5.1. Set N1 contains only the i-th derivative of the kernel k(x1 ), (i) (i) N1 = {k1 }. 41 (5.5) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS ! A B B =A⇥! (a) A !A C !B C = (A ⇥ !A ) [ (B ⇥ !B ) B (b) F IGURE 5.1: Relation between nodes in graph G. (j) Fact 5.2. Set Ni (l) can be derived from sets Ni−1 , l ∈ 0, . . . , j, (j) Ni = j [ (l) (j−l) (Ni−1 × ki ). (5.6) l=0 Proof. According to Definition 2, we have (j) Ni = (n ) (n ) (n ) {k1 1 k2 2 . . . ki i | (n1 ) (n2 ) (n ) k2 . . . ki i | = {k1 i X l=1 i−1 X l=1 nl = j, nl ∈ N0 } nl = j − ni , nl ∈ N0 , ni ∈ 0, . . . , j}. (j) Since ni can be any value from 0 to j, we can split set Ni into j + 1 mutually disjoint subsets such that, (j) Ni (n1 ) (n2 ) (0) k2 . . . ki ={k1 (n1 ) (n2 ) (1) k2 . . . ki {k1 (n1 ) (n2 ) (j) k2 . . . ki {k1 | | | i−1 X l=1 i−1 X l=1 i−1 X l=1 42 nl = j − 0, nl ∈ N0 }∪ nl = j − 1, nl ∈ N0 } ∪ . . . ∪ nl = j − j = 0, nl ∈ N0 }. CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Then, according to Definition 4, we can find that for any p ∈ 0, . . . j we have (n ) (n ) (p) {k1 1 k2 2 . . . ki | i−1 X (j−p) nl = j − p, nl ∈ N0 } = Ni−1 l=1 (p) × ki . Therefore, (j) Ni = (j−0) (Ni−1 × (0) ki ) ∪ (j−1) (Ni−1 × (1) ki ) (j) × (j) ki ) = j [ (l) (j−l) (Ni−1 × ki ). l=0 (l) equals to the sum of the elements in sets Ni−1 , l ∈ Fact 5.3. The number of elements in set Ni 0, . . . , j, (j) Si ∪ ... ∪ (j−j) (Ni−1 = j X (l) Si−1 . (5.7) l=0 Proof. According to Fact 5.2, we have (j) |Ni | =| j [ (l) (j−l) (Ni−1 × ki )|. l=0 Thus, (j) Si = j X l=0 According to Definition 2 and 4, we know (l) (j−l) |Ni−1 × ki (l) (j−l) |Ni−1 × ki (n ) (j−l) (n1 ) (n2 ) k2 . . . ki−1i−1 ki | = |{k1 (n ) (n1 ) (n2 ) k2 . . . ki−1i−1 = |{k1 = | (l) Si−1 . i−1 X p=1 |. | i−1 X p=1 np = l, np ∈ N0 }| np = l, np ∈ N0 }| Therefore, (j) Si = j X (l) Si−1 . l=0 (r) , Fact 5.4. The number of elements in set Nd is d+r−1 r d+r−1 (r) Sd = . r 43 (5.8) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Proof. This statement can be easily proved by induction. (r) Basic: Show that the statement holds for d = 1. According to Fact 5.1, we know that N1 (r) (r) Thus, S1 = 1. Since 1+r−1 = rr = 1, we can get S1 = 1+r−1 . r r (r) = {k1 }. Basic: Show that the statement holds for r = 0. According to Definition 2, we know that (0) (0) (0) d−1 d+0−1 Nd = {k1 k2 . . . kd }. Thus, Sd = 1. Since d+0−1 = = 1, we can get S = . d 0 0 0 (j) (j−1) Inductive step: Show that if Si−1 and Si know that (j) Si = j X l=0 Reapply Fact 5.3 to (j) Si Pj−1 l=0 (j−1) = Si (j) hold, then Si (l) Si−1 = j−1 X (l) also holds. According to Fact 5.3, we (j) Si−1 + Si−1 . l=0 (l) Si−1 , we get (j) + Si−1 = i+j−1−1 i−1+j−1 i+j−1 + = . j−1 j j Since both the basis and the inductive step have been performed, by mathematical induction, the (r) statement Sd holds for all d ∈ N+ and r ∈ N0 . Q.E.D. 5.1.2 Algorithm Our algorithm is illustrated in Figure 5.2. Consistent with Definition 5, each node in Figure 5.2 (j) stands for a set. As is defined in Definition 2, a set Ni contains all the partial derivatives of order j of the i-variable function K(x1 , . . . , xi ). Each edge in the graph defines a product operation, which is defined in Definition 4, between its head node and its weight. The weight of an edge is a univariate kernel derivative which is given by Definition 1. The relationship between an edge’s head, weight and tail is demonstrated in Figure 5.1. (r) Ignoring the output node Nd , we have a matrix of nodes in the graph. Each column contains the sets whose elements are the partial derivatives of the same kernel function. Each row contains the sets whose elements are the partial derivatives of the same order. Our algorithm start from the left side of the graph, where we initialize all the nodes in the first column by assigning its corresponding univariate kernel derivative as denoted by Fact 5.1. Then, according to Fact 5.2, we compute the sets in each column from the sets in previous column. Repeat this step until we reach the (d − 1)-th column. Finally, once we’ve got the output of (d − 1)-th column, we can reapply Fact 44 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS (0) 1 k1 … (0) N1 (0) ki (0) 1 Ni (0) Ni … (0) 1 Nd (1) ki (1) 1 k1 … (1) N1 (r) (0) ki (1) Ni 1 (1) Ni (r) ki … kd (1) 1 Nd (r 1) kd … … … … (r 1) ki (r) 1 k1 … (r) N1 (0) ki (r) 1 (r) Ni Ni … (0) (r) 1 kd Nd (r) Nd F IGURE 5.2: Graph based efficient multivariate kernel derivative algorithm. (r) 5.2 by Nd = Pr (i) i=0 (Nd−1 (r−i) × kd ) and output the result. The outline of this algorithm is shown in Algorithm 1. 5.1.3 Complexity Analysis Instead of computing the multivariate partial derivatives from the products of univariate derivatives directly, in this algorithm, we reuse the results from previous columns as much as possible, which (j) can reduce a great number of operations. Since all the univariate kernel derivatives ki , i ∈ 1, . . . , n, j ∈ 0, . . . , r can be calculated efficiently in advance, then the only operations for calculating the multivariate kernel derivative are multiplications. Thus, in the rest of this section, we only focus on counting the number of multiplications in the proposed efficient algorithm and comparing it with the naive method. The naive algorithm calculates the product of univariate derivatives for each multivariate partial derivative respectively. Assume we want to calculate the r-th order partial derivatives of the d-variable kernel function K(x1 , . . . , xd ). Then, according to Fact 5.4, the number of the r-th order partial derivatives is d+r−1 . Since each partial derivative is a product of d univariate kernel derivatives, r which results d − 1 multiplications, then the total number of multiplications in the naive algorithm is d+r−1 Mn = (d − 1) . (5.9) r From Algorithm 1, we know that the multiplication of the proposed efficient algorithm happens (l) (j−l) at the calculation of Ni−1 × ki (i) (r−i) and Nd−1 × kd 45 . According to Definition 4, we know that the CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Algorithm 1 Efficient Multivariate Kernel Derivative 1: procedure M ULTIVARIATE D ERIVATIVE(n, r ) 2: for i ← 0, r do (i) (i) 3: N1 ← k1 4: end for 5: for i ← 2, d − 1 do 6: for j ← 0, r do (j) 7: Ni ← Ø 8: for l ← 0, j do (j) (j) (l) (j−l) 9: Ni ← Ni ∪ (Ni−1 × ki ) 10: end for 11: end for 12: end for 13: for i ← 0, r do (r) (r) (i) (r−i) 14: Nd ← Nd ∪ (Nd−1 × kd ) 15: end for (r) 16: return Nd 17: end procedure number of multiplications in performing × operation is equal to the size of the set. Thus, the number (l) (j−l) of multiplication in computing Ni−1 × ki (i) (r−i) and Nd−1 × kd (l) (i) is Si−1 and Sd−1 . Therefore, by applying this to all the for loops in Algorithm 1, we can get Me = j d−1 X r X X (l) Si−1 + i=2 j=0 l=0 r X (i) Sd−1 . (5.10) i=0 According to Fact 5.3, the above equation can be simplified as Me = d−1 X i=2 (r) Si+1 + (r) Sd = d−1 X r+i i=2 r d+r−1 + . r (5.11) Comparing Equation (5.10) with (5.9), we can find that the number of multiplications in the efficient algorithm is significantly smaller than the naive algorithm. Hence, our algorithm can achieve a considerable speed up in theory. The detailed experimental results are given in Chapter 6. 5.2 High Performance Kernel Density and Kernel Density Derivative Estimators Kernel density and kernel density derivative estimation methods usually have very high computational requirements. From the discussions in Section 2.5 and Section 5.1.1, we know that the direct 46 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS computation of the KDE and KDDE requires mn d+r−1 r kernel evaluations, where m is the number of test points, n is the number of training points, d is the dimension of the data, and r is the order of estimators. In many cases, the size of the data sets are becoming larger and larger in recent years. In our case, the test data and training data are usually of size 106 to 107 . Fortunately, the evaluations of KDE and KDDE are independent for different test points, which makes it a perfect fit for parallel computing. In this section, we proposed a multi-core CPU and GPU platform based solution to accelerate the computation of KDE and KDDE. Several optimization techniques are used to achieve significant performance gains. 5.2.1 Multi-core CPU Implementation The goal of the multi-core CPU implementation is to deliver a set of kernel smoothing functions to achieve high flexibility as well as a good performance. For flexibility, this implementation supports input data of any dimension, can compute kernel density derivatives of any order and has a flexible choice of kernel and bandwidth types. To achieve a good performance, this implementation uses the POSIX Threads (PThreads) programming interface to utilize parallelism on multi-core CPU platform. In this section, we only focus on the most general case (unconstrained variable bandwidth, any dimension, any order, and Gaussian kernel) due to its high computational and mathematical complexity. According to Equation (2.27), we know that KDDE is usually calculated at n different test points xi . And for each KDDE at test point xi , we need to calculate the weighted sum of the scaled r-th order kernel derivative D⊗r KS j at m different shifted locations xi − X j . Thus, there will be m × n scaled r-th order kernel derivative calculations involved. Note that the scaled r-th order kernel derivative and unscaled r-th order kernel derivative at shifted location xi − X j is related ⊗r by D⊗r KS j (xi − X j ) = |S j |S ⊗r j D K(S j (xi − X j )). Hence, the calculation of the scaled r-th order kernel derivative D⊗r KS j at xi − X j can be divided into four steps: • calculate the scaled and shifted data y = S j (xi − X j ), where y is a d dimensional vector y = [y1 , y2 , . . . , yd ]T ; • for each variable yl , l = 1, . . . , d in y calculate its univariate kernel and kernel derivatives k (0) (yl ), k (1) (yl ), . . ., k (r) (yl ) and store the results into a d × r matrix F , where F (u, v) = k (v) (yu ); 47 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS • calculate the multivariate r-th order kernel derivative D⊗r K from the univariate kernel derivatives in F using the efficient algorithm introduced in Section 5.1. Note that D⊗r K requires dr r-th order partial derivatives which carries some redundancies. However, the efficient algorithm only gives d+r−1 unique partial derivatives. Hence, we need to repeat some results r we got from the efficient algorithm to fill the redundant locations in D⊗r K; ⊗r ⊗r ⊗r • calculate |S j | and S ⊗r j , and update D KS j by |S j |S j D K. Since the calculations of KDDE at different test points are independent, we can parallelize our algorithm at the test point level. Therefore, we propose our general KDDE algorithm on multi-core platform in Algorithm 2. Algorithm 2 Parallel CPU Kernel Density Derivative Estimation 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: procedure KDDE(x, X, S, ω, r) d, m ← S IZE(x) n ← S IZE(X, 2) D⊗r f ← Z EROS(n, dr ) parfor i ← 0, m − 1 do for j ← 0, n − 1 do y ← S(j)(x(i) − X(j)) F ← U NIVARIATE D ERIVATIVES(y, r) D⊗r K ← M ULTIVARIATE D ERIVATIVES(F, d, r) D⊗r f (i) ← D⊗r f (i) + ω(j)|Sj|S ⊗r (j)D⊗r K end for end parfor return D⊗r f end procedure 5.2.2 GPU Implementation in CUDA From Chapter 3, we know that in the context of image processing and pattern recognition, the estimation of the first and second derivatives of the density is crucial to locate significant feature regions on images. Therefore, in this section we’ll focus only on the kernel gradient and curvature estimator, as denoted by Equation (3.43) and (3.44), for 2D and 3D images. What’s more, since the choice of kernel function is not crucial to the accuracy of KDE and KDDE, we choose the standard Gaussian function as the kernel. Based on these assumptions and because of the multivariate estimators are far more computationally and mathematically involved, we present several optimized GPU KDE and KDDE implementations in this section. 48 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Naive Implementation The same as the multi-core CPU implementation, the naive GPU implementation also parallelizes at the test point level. we create four gpu-kernel functions (ShiftAndScale, UnivarDeri, MultivarDeri, and Update) corresponding to the four steps in calculating the scaled kernel derivatives. Each gpu-kernel is designed to complete its job for all the test points concurrently. To achieve this, based on the CUDA programming model in Section 4.2, we divide the gpu-kernel into dm/te equally-sized blocks and each block contains t threads. Thus, there are roughly m threads in the gpu-kernels. Each of these m threads is responsible for calculating the linear combination of the shifted D⊗r KS j functions at a test point. The naive implementation is shown in Algorithm 3. To illustrate the all the problems and optimization techniques clearly, we give the most complex KDDE function, which can compute the kernel density, kernel gradient and kernel curvature at the same time, from our kernel smoothing library. Since we are only interested in the first and second order derivatives, we implement the estimators according to Equation (2.28) and (2.30). Optimization I – Kernel Merging, Loop Unrolling, and Memory Layout Optimization If we take a close look at Algorithm 3, we can find that there exists several problems: • Too many small gpu-kernels. The four gpu-kernels generate lots of redundant global memory transactions. Every gpu-kernel has to save its results back to the global memory so that they can be used by the following gpu-kernels. As mentioned in Section 4.1, all the global memory transactions are off-chip, which makes them much slower than the on-chip memories. Thus, the redundant off-chip global memory transactions will introduce lots of warp stalls and eventually slow down the execution of the GPU. • Bad memory layouts. The memory layout of matrices and cubes are column-major (first dimension first) in the naive implementation. As illustrated in Figure 5.3 (a) and (c), such memory layout will result strided memory accesses. As mentioned in Section 4.4, memory accesses are coalesced by warps. It means scattered or strided memory access will require more memory transactions since they can’t be efficiently grouped in one memory transaction. • Unnecessary loops. The calculation of multivariate kernel derivatives involves lots of matrix operations. To complete these operations, we need to write many for loop statements in the gpu-kernels. Usually, the sizes of those for loops are determined by data dimension. However, 49 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Algorithm 3 Naive GPU Kernel Density Derivative Estimation 1: procedure KDDE(x, X, S, w) 2: d, m ← S IZE(x) 3: n ← S IZE(X, 2) 4: f ← Z EROS(1, m) 5: g ← Z EROS(d, m) 6: H ← Z EROS(d ∗ d, m) 7: for i ← 0, n − 1 do 8: y i ← S HIFTA ND S CALE(x, X, S, i) 0 00 9: ki , ki , ki ← U NIVAR D ERI(y i ) 0 00 10: f i , g i , H i ← M ULTIVAR D ERI(ki , ki , ki ) 11: f , g, H ← U PDATE(f i , g i , H i , S, w, i) 12: end for 13: return f , g, H 14: end procedure 49: end for 50: g i [j ∗ d + p] ← gp 51: H i [j ∗ d ∗ d + p ∗ d + p] ← Hpp 52: end for 53: f i [j] ← f 54: for p ← 1, d − 1 do 55: for q ← 0, p − 1 do 0 56: Hpq ← ki [j ∗ d + q] ∗ g i [j ∗ d + p]/ki [j ∗ d + q] 57: H i [j ∗ d ∗ d + p ∗ d + q] ← Hpq 58: H i [j ∗ d ∗ d + q ∗ d + p] ← Hpq 59: end for 60: end for 61: return f i , g i , H i 62: end procedure 15: procedure S HIFTA ND S CALE(x, X, S, i) 16: j ← blockDim.x ∗ blockIdx.x + threadIdx.x 17: d, n ← S IZE(x) 18: m ← S IZE(X, 2) 19: for k ← 0, d − 1 do 20: t[j ∗ d + k] ← x[j ∗ d + k] − X[i ∗ d + k] 21: end for 22: for p ← 0, d − 1 do 23: for q ← 0, d − 1 do 24: y i [j ∗ d + p] ← y i [j ∗ d + p] + t[j ∗ d + q] ∗ 63: procedure U PDATE(f i , g i , H i , S, w, i) 64: j ← blockDim.x + blockIdx.x + threadIdx.x 65: f [j] ← f [j] + f i [j] ∗ w[i] 66: for p ← 0, d − 1 do 67: t←0 68: for q ← 0, d − 1 do 69: t ← t + g i [j ∗ d + q] ∗ S[j ∗ d ∗ d + d ∗ q + p] 70: end for 71: g[j ∗ d + p] ← g[j ∗ d + p] + w[j] ∗ t 72: end for 73: for p ← 0, d − 1 do 74: for q ← 0, d − 1 do 75: t←0 76: for r ← 0, d − 1 do 77: t ← t + H i [j ∗ d ∗ d + p ∗ d + r] ∗ S[j ∗ d ∗ S[d ∗ d ∗ i + d ∗ q + p] 25: end for 26: end for 27: return y i 28: end procedure 29: procedure U NIVAR D ERI(y i ) 30: j ← blockDim.x + blockIdx.x + threadIdx.x 31: yij ← y i [j] 32: k ← 1/SQRT(2 ∗ π) ∗ EXP(−0.5 ∗ yij ∗ yij ) 33: ki [j] ← k 0 34: ki [j] ← −yij ∗ k 00 35: ki [j] ← (yij ∗ yij − 1) ∗ k 00 0 36: return ki , ki , ki 37: end procedure 0 00 38: procedure M ULTIVAR D ERI(ki , ki , ki ) 39: j ← blockDim.x + blockIdx.x + threadIdx.x 40: d, n ← S IZE(ki ) 41: f ←1 42: for p ← 0, d − 1 do 43: f ← f ∗ ki [j ∗ d + p] 0 44: gp ← ki [j ∗ d + p] 00 45: Hpp ← ki [j ∗ d + p] 46: for q ← p + 1, d + p − 1 do 47: gp ← gp ∗ ki [j ∗ d + MOD(q, d)] 48: Hpp ← Hpp ∗ ki [j ∗ d + MOD(q, d)] d + r ∗ d + q] end for H tmp [p ∗ d + q] ← t end for end for for p ← 0, d − 1 do for q ← 0, d − 1 do t←0 for r ← 0, d − 1 do t ← t+S[j∗d∗d+r∗d+p]∗H tmp [r∗d+q] end for H[j ∗ d ∗ d + p ∗ d + q] ← H[j ∗ d ∗ d + p ∗ d + q] + w[j] ∗ t 89: if p 6= q then 90: H[j ∗ d ∗ d + q ∗ d + p] ← H[j ∗ d ∗ d + q ∗ d + p] + w[j] ∗ t 91: end if 92: end for 93: end for 94: return f , g, H 95: end procedure 78: 79: 80: 81: 82: 83: 84: 85: 86: 87: 88: 50 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS since we are only interested in 2D and 3D data in this section, the for loop size is actually only 2 or 3, which is definitely inefficient and should be unrolled to remove the loop overhead. Global Memory x(0,0) Thread IDs x(1,0) x(2,0) x(0,1) 0 x(1,1) x(2,1) … X(0,N-1) … 1 X(1,N-1) X(2,N-1) x(2,1) … N-1 (a) x(0,0) x(0,1) … X(0,N-1) 0 1 … N-1 x(1,0) … x(1,1) X(1,N-1) x(2,0) X(2,N-1) (b) x(0,0,0) x(1,0,0) x(0,1,0) x(1,1,0) x(0,0,1) x(1,0,1) x(0,1,1) x(1,1,1) 0 … x(0,0,N-1) … 1 x(1,0,N-1) x(0,1,N-1) x(1,1,N-1) N-1 (c) x(0,0,0) x(0,0,1) 0 1 … x(0,0,N-1) … N-1 x(1,0,0) x(1,0,1) … x(1,0,N-1) x(0,1,0) x(0,1,1) … x(0,1,N-1) x(1,1,0) x(1,1,1) … x(1,1,N-1) (d) F IGURE 5.3: Memory access patterns of matrices and cubes. (a) Memory access pattern of column-major matrix. (b) Memory access pattern of row-major matrix. (c) Memory access pattern of column-major cube (3D matrix). (d) Memory access pattern of column-major cube (3D matrix). Therefore, to solve these problems, we propose our optimized implementation in Algorithm 4. Here, we merged the four small gpu-kernels into two big gpu-kernels to avoid redundant global memory transactions. However, big gpu-kernels usually consume more registers because they use lots of variables. According to Section 4.3, we know that GPU has limited register resources. If 51 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS a gpu-kernel uses too much registers, it can not achieve a high occupancy, which results a bad performance. Hence, in the optimized implementation I, we reduce the usage of register by reusing previous results or local variables as much as possible. We also changed our column-major memory layout of matrices and cubes into row-major (second dimension first) and slice-major (third dimension first) respectively. Then, from Figure 5.3 (b) and (d), we can find that the memory access is now continuous in either case. What’s more, we unrolled the loops in the optimized implementation. Since the order of the statements is now not restricted to be what it was in the loop, the loop unrolling not only avoided executing loop control instructions, but also provided us more flexible controls over statements once inside the for loop. Optimization II – Simplified Math Expressions In Optimization I, we improved the implementation from the perspective of the GPU code. However, the implementation can also improved at the algorithm level. If we take a look at the kernel function itself, we can find that a better representation can be used to simplify the KDDE, and thus, speed up the calculation. As we know from Equation (2.9) that a separable multivariate Gaussian kernel can be written as K(x) = d Y 1 2 √ e−xl /2 . 2π l=1 (5.12) If we evaluate this equation directly, it will result in computing the exponential function d times. However, we can use the property of exponential function and simplify the above equation as 1 T 1 K(x) = ( √ )d e− 2 x x . 2π (5.13) We can find that the simplified kernel function only requires one exponential function. Similarly, instead of calculating Equation (2.10) and (2.11), the gradient and Hessian of the separable multivariate Gaussian kernel can be given as 1 T 1 ∇K(x) = ( √ )d e− 2 x x x 2π 1 T 1 ∇2 K(x) = ( √ )d e− 2 x x (xxT − I) 2π 52 (5.14) (5.15) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Algorithm 4 Optimized GPU Kernel Density Derivative Estimation I 1: procedure KDDE(x, X, S, w) 2: m ← S IZE(x, 1) 3: n ← S IZE(X, 1) 4: f ← Z EROS(n, 1) 5: g ← Z EROS(n, 3) 6: H ← Z EROS(n, 9) 7: for i ← 0, n − 1 do 8: yi ← S HIFTA ND S CALE(x, X, S, i) 9: f, g, H ← K ERNEL C ORE(x, yi , S, w[i], i) 10: end for 11: return f, g, H 12: end procedure 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47: 48: 49: 50: 51: 52: 53: 54: fij ← wi ∗ c fij ← fij ∗ E XP(−0.5 ∗ xj0 ∗ xj0 ) fij ← fij ∗ E XP(−0.5 ∗ xj1 ∗ xj1 ) fij ← fij ∗ E XP(−0.5 ∗ xj2 ∗ xj2 ) t0 ← xj0 ∗ xj1 ∗ fij t1 ← xj0 ∗ xj2 ∗ fij t2 ← xj1 ∗ xj2 ∗ fij t3 ← s0 ∗ (xj0 ∗ xj0 − 1) ∗ fij + s1 ∗ t0 + s2 ∗ t1 t4 ← s0 ∗ t0 + s1 ∗ (xj1 ∗ xj1 − 1) ∗ fij + s2 ∗ t2 t5 ← s0 ∗ t1 + s1 ∗ t2 + s2 ∗ (xj2 ∗ xj2 − 1) ∗ fij f[j] ← f[j] + fij g[j] ← g[j] − fij ∗ (s0 ∗ xj0 + s1 ∗ xj1 + s2 ∗ xj2 ) H[j] ← H[j] + s0 ∗ t3 + s1 ∗ t4 + s2 ∗ t5 13: procedure S HIFTA ND S CALE(x, X, S, i) s0 ← S[i ∗ 9 + 3] 14: j ← blockDim.x ∗ blockIdx.x + threadIdx.x s1 ← S[i ∗ 9 + 4] 15: n ← S IZE(x, 1) s2 ← S[i ∗ 9 + 5] 16: m ← S IZE(X, 1) t6 ← s0 ∗ t3 + s1 ∗ t4 + s2 ∗ t5 17: yi0 ← x[j + n ∗ 0] − X[i + m ∗ 0] g[j + n ∗ 1] ← g[j + n ∗ 1] − fij ∗ (s0 ∗ xj0 + s1 ∗ 18: yi1 ← x[j + n ∗ 1] − X[i + m ∗ 1] xj1 + s2 ∗ xj2 ) 19: yi2 ← x[j + n ∗ 2] − X[i + m ∗ 2] 55: H[j + n ∗ 1] ← H[j + n ∗ 1] + t6 20: yi [j + n ∗ 0] ← yi0 ∗ S[i ∗ 9 + 0] + yi1 ∗ S[i ∗ 9 + 3] + 56: H[j + n ∗ 3] ← H[j + n ∗ 3] + t6 yi2 ∗ S[i ∗ 9 + 6] 57: t6 ← s3 ∗ t3 + s4 ∗ t4 + s5 ∗ t5 21: yi [j + n ∗ 1] ← yi0 ∗ S[i ∗ 9 + 1] + yi1 ∗ S[i ∗ 9 + 4] + 58: H[j + n ∗ 2] ← H[j + n ∗ 2] + t6 yi2 ∗ S[i ∗ 9 + 7] 59: H[j + n ∗ 6] ← H[j + n ∗ 6] + t6 22: yi [j + n ∗ 2] ← yi0 ∗ S[i ∗ 9 + 2] + yi1 ∗ S[i ∗ 9 + 5] + 60: t3 ← s0 ∗ (xj0 ∗ xj0 − 1) ∗ fij + s1 ∗ t0 + s2 ∗ t1 yi2 ∗ S[i ∗ 9 + 8] 61: t4 ← s0 ∗ t0 + s1 ∗ (xj1 ∗ xj1 − 1) ∗ fij + s2 ∗ t2 23: return yi 62: t5 ← s0 ∗ t1 + s1 ∗ t2 + s2 ∗ (xj2 ∗ xj2 − 1) ∗ fij 24: end procedure 63: t6 ← s3 ∗ t3 + s4 ∗ t4 + s5 ∗ t5 64: g[j + n ∗ 2] ← g[j + n ∗ 2] − fij ∗ (s3 ∗ xj0 + s4 ∗ 25: procedure K ERNEL C ORE(x, yi , S, wi , i) xj1 + s5 ∗ xj2 ) 26: j ← blockDim.x ∗ blockIdx.x + threadIdx.x 65: H[j + n ∗ 4] ← H[j + n ∗ 4] + s0 ∗ t3 + s1 ∗ t4 + s2 ∗ t5 27: n ← S IZE(yi , 1) 66: H[j + n ∗ 5] ← H[j + n ∗ 5] + t6 28: xj0 ← x[j + n ∗ 0] 67: H[j + n ∗ 7] ← H[j + n ∗ 7] + t6 29: xj1 ← x[j + n ∗ 1] 68: t3 ← s3 ∗ (xj0 ∗ xj0 − 1) ∗ fij + s4 ∗ t0 + s5 ∗ t1 30: xj2 ← x[j + n ∗ 2] 69: t4 ← s3 ∗ t0 + s4 ∗ (xj1 ∗ xj1 − 1) ∗ fij + s5 ∗ t2 31: s0 ← S[i ∗ 9 + 0] 70: t5 ← s3 ∗ t1 + s4 ∗ t2 + s5 ∗ (xj2 ∗ xj2 − 1) ∗ fij 32: s1 ← S[i ∗ 9 + 1] 71: H[j + n ∗ 8] ← H[j + n ∗ 8] + s3 ∗ t3 + s4 ∗ t4 + s5 ∗ t5 33: s2 ← S[i ∗ 9 + 2] 72: return f, g, H 34: s3 ← S[i ∗ 9 + 6] 73: end procedure 35: s4 ← S[i ∗ 9 + 7] 36: s5 ← S[i ∗ 9 + 8] 53 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Therefore, the simplified kernel density, kernel gradient, kernel curvature estimators can be written as n X 1 T T 1 fˆ(xi ; S j , ωj ) = ( √ )d ωj |S j |e− 2 (xi −X j ) S j S j (xi −X j ) 2π j=1 (5.16) n 1 T T 1 dX ˆ ∇f (xi ; S j , ωj ) = ( √ ) ωj |S j |e− 2 (xi −X j ) S j S j (xi −X j ) S Tj S j (xi − X j ) 2π j=1 (5.17) n X 1 T T 1 ∇2 fˆ(xi ; S j , ωj ) =( √ )d ωj |S j |e− 2 (xi −X j ) S j S j (xi −X j ) 2π j=1 (5.18) (S Tj S j (xi − X j )(xi − X j )T S j S Tj − S Tj S j ) The advantage of these simplified forms is that it not only simplifies the computation but also de1 T S T S (x −X ) j i j j creases the usage of variables. First, they contain a common expression ωj |S j |e− 2 (xi −X j ) Hence, we only need to compute this expression once and save the result for reuse. Second, expression S Tj S j (xi − X j ) appears repeatedly. Thus, its evaluation result can also be used in multiple places. Third, if S Tj S j is computed in advance, then there will be no square matrix multiplications. Since square matrix multiplication involves lots of addition and multiplication operations and need to use lots of variables to store temporary results, the simplified forms can greatly reduce the usage of registers. What’s more, in Optimization I, we need to call the gpu-kernels for each training point. Since the number of training points is usually very large, there will be a great number of gpu-kernel calls, which results a significant kernel call overhead. One solution of this problem is to design a big kernel to enclose the outer for loop. Then, there will be only one gpu-kernel call. Previously, this solution is not practical because such a large gpu-kernel will consume too much GPU resources, which lead to a low GPU occupancy. But due to the low variable usage of the simplified forms, it is now possible to merge all the gpu-kernel calls into just one single call. Based on the analysis above, we propose our optimized implementation in Algorithm 5. We can find that the main function KDDE now contains only two gpu-kernels SquareAndDet and LinCombKernels. SquareAndDet is responsible for calculating the squared scale S Tj S j and the scale determinant |S j | for each training point. The outer for loop is now moved into the LinCombKernels, which basically computes Equation (5.16), (5.17), and (5.18). Optimization III – Exploiting Temporal Locality Using Shared Memory As we mentioned in Section 4.1, shared memory is a programmable cache on GPU. It is way more faster than the off-chip global memory. However, to utilize the shared memory efficiently, there has 54 . CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Algorithm 5 Optimized GPU Kernel Density Derivative Estimation II 1: procedure KDDE(x, X, S, w) 31: t2 ← xi2 − X[j + m ∗ 2] 2: SS, c ← S QUARE A ND D ET(S, w) 32: t3 ← t1 ∗S[j∗9+0]+t2 ∗S[j∗9+3]+t3 ∗S[j∗9+6] 3: f, g, H ← L IN C OMB K ERNELS(x, X, S, SS, c) 33: t4 ← t1 ∗S[j∗9+1]+t2 ∗S[j∗9+4]+t3 ∗S[j∗9+7] 4: return f, g, H 34: t5 ← t1 ∗S[j∗9+2]+t2 ∗S[j∗9+5]+t3 ∗S[j∗9+8] 5: end procedure 35: fij ← c[j] ∗ E XP(t3 ∗ t3 + t4 ∗ t4 + t5 ∗ t5 ) 36: fi ← fi + fij 6: procedure S QUARE A ND D ET(S, w) 37: ss11 ← SS[j ∗ 6 + 0] 7: i ← blockDim.x ∗ blockIdx.x + threadIdx.x 38: ss12 ← SS[j ∗ 6 + 1] 8: s11 ← S[i∗9+0], s12 ← S[i∗9+3], s13 ← S[i∗9+6] 39: ss13 ← SS[j ∗ 6 + 2] 9: s21 ← S[i∗9+1], s22 ← S[i∗9+4], s23 ← S[i∗9+7] 40: ss22 ← SS[j ∗ 6 + 3] 10: s31 ← S[i∗9+2], s32 ← S[i∗9+5], s33 ← S[i∗9+8] 41: ss23 ← SS[j ∗ 6 + 4] 11: SS[i ∗ 6 + 0] ← s11 ∗ s11 + s21 ∗ s21 + s31 ∗ s31 42: ss33 ← SS[j ∗ 6 + 5] 12: SS[i ∗ 6 + 1] ← s11 ∗ s12 + s21 ∗ s22 + s31 ∗ s32 43: t3 ← ss11 ∗ t0 + ss12 ∗ t1 + ss13 ∗ t2 13: SS[i ∗ 6 + 2] ← s11 ∗ s13 + s21 ∗ s23 + s31 ∗ s33 44: t4 ← ss12 ∗ t0 + ss22 ∗ t1 + ss23 ∗ t2 14: SS[i ∗ 6 + 3] ← s12 ∗ s12 + s22 ∗ s22 + s32 ∗ s32 45: t5 ← ss13 ∗ t0 + ss23 ∗ t1 + ss33 ∗ t2 15: SS[i ∗ 6 + 4] ← s12 ∗ s13 + s22 ∗ s23 + s32 ∗ s33 46: g[i + n ∗ 0] ← g[i + n ∗ 0] − fij ∗ t3 16: SS[i ∗ 6 + 5] ← s13 ∗ s13 + s23 ∗ s23 + s33 ∗ s33 47: g[i + n ∗ 1] ← g[i + n ∗ 1] − fij ∗ t4 17: t0 ← s11 ∗ s22 ∗ s33 + s12 ∗ s23 ∗ s31 + s13 ∗ s21 ∗ s32 48: g[i + n ∗ 2] ← g[i + n ∗ 2] − fij ∗ t5 18: t1 ← s13 ∗ s22 ∗ s31 + s23 ∗ s32 ∗ s11 + s12 ∗ s21 ∗ s33 49: H[i + n ∗ 0] ← H[i + n ∗ 0] + fij ∗ (t3 ∗ t3 − ss11 ) 19: c[i] ← w[i] ∗ c ∗ A BS(t0 − t1 ) 50: H[i + n ∗ 1] ← H[i + n ∗ 1] + fij ∗ (t3 ∗ t4 − ss12 ) 20: return SS, c 21: end procedure 51: H[i + n ∗ 2] ← H[i + n ∗ 2] + fij ∗ (t3 ∗ t5 − ss13 ) 52: H[i + n ∗ 4] ← H[i + n ∗ 4] + fij ∗ (t4 ∗ t4 − ss22 ) 53: H[i + n ∗ 5] ← H[i + n ∗ 5] + fij ∗ (t4 ∗ t5 − ss23 ) 22: procedure L IN C OMB K ERNELS(x, X, S, SS, c) 54: 23: i ← blockDim.x ∗ blockIdx.x + threadIdx.x 55: 24: n ← S IZE(x, 1) 56: f[i] ← fi 25: m ← S IZE(X, 1) 57: H[i + n ∗ 3] ← H[i + n ∗ 1] 26: fi ← 0 58: H[i + n ∗ 6] ← H[i + n ∗ 2] 27: xi0 ← x[i+n∗0], xi1 ← x[i+n∗1], xi2 ← x[i+n∗2] 59: H[i + n ∗ 7] ← H[i + n ∗ 5] 28: for j ← 0, m − 1 do 60: return f, g, H 29: t0 ← xi0 − X[j + m ∗ 0] 30: t1 ← xi1 − X[j + m ∗ 1] H[i + n ∗ 8] ← H[i + n ∗ 8] + fij ∗ (t5 ∗ t5 − ss33 ) end for 61: end procedure 55 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS to be enough temporal locality (reuse of data) in the gpu-kernel. Before introducing our optimized implementation using shared memory, let’s take a look at the memory access pattern in Optimization II. Assume m is the number of training points, n is the number of threads (equal to test point number), and r is the number of variables that access data from global memory, then the memory access pattern in Optimization II can be illustrated as Figure 5.4. Here, colored block groups V1 , V2 , . . . , Vr are arrays with m elements each. They all exist in global memory. Gray circles stand for threads. Each step is an iteration of the for loop in Optimization II. As we already knew that each thread will evaluate the kernel density, kernel gradient and kernel curvature for a single test point. Since such evaluation are similar for every thread, they usually read the same data from the global memory in each step. In the figure, we can find that, in the first step, all the threads read the first element in V1 , V2 , . . . , Vr . Since there are n threads, it means the same data is read n times, which is obviously inefficient. To analysis this problem quantitatively, we calculate the total number of global memory access of this case, Mg = m × n × r. (5.19) We can solve this problem by introducing shared memory. The new memory access pattern is shown in Figure 5.5. Here, the colored blocks stand for the data in global memory. Each color corresponds to an array in Figure 5.4. The number in the block stands for the index of an element in the array. The reason we choose this form of memory layout instead of the layout in Figure 5.4 is that in this arrangement of memory, the data read by threads are consecutive, which can reduce global memory transactions due to memory coalescing. The threads are divided into colored groups. Each group stands for a thread block mentioned in Section 4.2. Since data in shared memory can only be accessed by threads from the same block, here we draw shared memory separately for each thread block. The idea of using shared memory is that if certain data is used several times, we can store it into shared memory first, then read it from the shared memory directly in later usage. We can see from Figure 5.5 that the threads in the same block read the same data from the global memory only once (threads in different block still have to read the same data repeatedly, because data in shared memory can not be accessed between thread blocks). Then, this data can be accessed by other threads in the same block directly from the shared memory. The total number of global memory accesses is n 1 Mg = m × r × d e × , b c 56 (5.20) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Algorithm 6 Optimized GPU Kernel Density Derivative Estimation III 1: procedure KDDE(x, X, S, w) 2: m, d ← S IZE(X) 3: r ← 1 + d + d ∗ d + (1 + d) ∗ d/2 4: for i ← 0, m − 1 do 5: for j ← 0, d − 1 do 6: y[r ∗ i + j] ← X[m ∗ j + i] 7: for k ← 0, d − 1 do 8: y[r∗i+(j +1)∗d+k] ← S[i∗d∗d+j ∗d+k] 9: end for 10: end for 11: end for 12: y ← S QUARE A ND D ET(y, w) 13: f, g, H ← L IN C OMB G AUSSIAN K ERNELS(x, y) 14: return f, g, H 15: end procedure 16: procedure S QUARE A ND D ET(y, w) 17: i ← (blockDim.x ∗ blockIdx.x + threadIdx.x) ∗ 19 18: s11 ← y[i + 3], s12 ← y[i + 6], s13 ← y[i + 9] 19: s21 ← y[i + 4], s22 ← y[i + 7], s23 ← y[i + 10] 20: s31 ← y[i + 5], s32 ← y[i + 8], s33 ← y[i + 11] 21: t0 ← s11 ∗ s22 ∗ s33 + s12 ∗ s23 ∗ s31 + s13 ∗ s21 ∗ s32 22: t1 ← s13 ∗ s22 ∗ s31 + s23 ∗ s32 ∗ s11 + s12 ∗ s21 ∗ s33 23: y[i + 12] ← w[i/19] ∗ c ∗ A BS(t0 − t1 ) 24: y[i + 13] ← s11 ∗ s11 + s21 ∗ s21 + s31 ∗ s31 25: y[i + 14] ← s11 ∗ s12 + s21 ∗ s22 + s31 ∗ s32 26: y[i + 15] ← s11 ∗ s13 + s21 ∗ s23 + s31 ∗ s33 27: y[i + 16] ← s12 ∗ s12 + s22 ∗ s22 + s32 ∗ s32 28: y[i + 17] ← s12 ∗ s13 + s22 ∗ s23 + s32 ∗ s33 29: y[i + 18] ← s13 ∗ s13 + s23 ∗ s23 + s33 ∗ s33 30: return y 31: end procedure 32: procedure L IN C OMB G AUSSIAN K ERNELS(x, y) 33: i ← blockDim.x ∗ blockIdx.x + threadIdx.x shared t[6144] 34: 35: n ← S IZE(x, 1), m ← S IZE(X, 1) 36: fi ← 0 37: gi0 ← 0, gi1 ← 0, gi2 ← 0 38: H11i ← 0 39: H12i ← 0, H22i ← 0 40: H13i ← 0, H23i ← 0, H33i ← 0 41: xi0 ← x[i+n∗0], xi1 ← x[i+n∗1], xi2 ← x[i+n∗2] 42: for j ← 0, m − 1 do 43: 44: 45: 46: 47: 48: 49: 50: 51: 52: 53: 54: 55: i ← M OD(j, 323) if i == 0 then S YNCTHREADS() t[thread + 0] ← y[j ∗ 19 + thread + 0] t[thread + 1024] ← y[j ∗ 19 + thread + 1024] t[thread + 2048] ← y[j ∗ 19 + thread + 2048] t[thread + 3072] ← y[j ∗ 19 + thread + 3072] t[thread + 4096] ← y[j ∗ 19 + thread + 4096] t[thread + 5120] ← y[j ∗ 19 + thread + 5120] S YNCTHREADS() end if i ← i ∗ 19 t0 ← xi0 − t[i + 0], t1 ← xi1 − t[i + 1], t2 ← xi2 − t[i + 2] 56: 57: 58: 59: 60: 61: 62: 63: 64: t3 ← t0 ∗ t[i + 3] + t1 ∗ t[i + 6] + t2 ∗ t[i + 9] t4 ← t0 ∗ t[i + 4] + t1 ∗ t[i + 7] + t2 ∗ t[i + 10] t5 ← t0 ∗ t[i + 5] + t1 ∗ t[i + 8] + t2 ∗ t[i + 11] fij ← t[i + 12] ∗ E XP(t3 ∗ t3 + t4 ∗ t4 + t5 ∗ t5 ) t3 ← t[i + 13] ∗ t0 + t[i + 14] ∗ t1 + t[i + 15] ∗ t2 t4 ← t[i + 14] ∗ t0 + t[i + 16] ∗ t1 + t[i + 17] ∗ t2 t5 ← t[i + 15] ∗ t0 + t[i + 17] ∗ t1 + t[i + 18] ∗ t2 fi ← fi + fij gi0 ← gi0 − fij ∗ t3 , gi1 ← gi1 − fij ∗ t4 , gi2 ← gi2 − fij ∗ t5 65: 66: 67: 68: 69: 70: 71: 72: 73: 74: 75: H11i ← H11i + fij ∗ (t3 ∗ t3 − t[i + 13]) H12i ← H12i + fij ∗ (t3 ∗ t4 − t[i + 14]) H13i ← H13i + fij ∗ (t3 ∗ t5 − t[i + 15]) H22i ← H22i + fij ∗ (t4 ∗ t4 − t[i + 16]) H23i ← H23i + fij ∗ (t4 ∗ t5 − t[i + 17]) H33i ← H33i + fij ∗ (t5 ∗ t5 − t[i + 18]) end for i ← blockDim.x ∗ blockIdx.x + threadIdx.x f[i] ← fi g[i+n∗0] ← gi0 , g[i+n∗1] ← gi1 , g[i+n∗2] ← gi2 H[i+n∗0] ← H11i , H[i+n∗3] ← H12i , H[i+n∗6] ← H13i 76: H[i+n∗1] ← H12i , H[i+n∗4] ← H22i , H[i+n∗7] ← H23i 77: H[i+n∗2] ← H13i , H[i+n∗5] ← H23i , H[i+n∗8] ← H33i 78: return f, g, H 79: end procedure 57 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Global Memory V1 Step 1 0 Thread IDs 1 … m-1 0 1 0 … 0 Thread IDs … m-1 1 0 1 m-1 0 1 0 m-1 n-1 Vr V2 … … … 1 V1 Step 2 Vr V2 … … m-1 0 1 … m-1 … 1 n-1 … V1 Step m 0 Thread IDs 1 V2 … m-1 0 1 0 Vr … … m-1 0 1 … m-1 … 1 n-1 F IGURE 5.4: Memory access pattern without using shared memory. Global Memory 0 Thread IDs 0 Shared Memory 0 Thread IDs 0 0 … 1 1 … … 1 0 1 b-1 b 0 b-1 … 1 b-1 b … … b+1 1 … 1 … b+1 2b-1 … … b-1 … m-1 2b-1 m-1 … k 0 … k+1 1 k F IGURE 5.5: Memory access pattern using shared memory. 58 m-1 k+1 … k+b-1 b-1 … k+b-1 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS where b is the block size, c is the memory coalescing factor. We can see that the number of global memory accesses using shared memory is reduced b × c times. The total number of shared memory accesses is Ms = m × n × r × where w is the half warp size. We need the factor 1 w 1 , w (5.21) is because all the threads always read the data at the same location each time. Thus, the shared memory access will be broadcasted to the threads in the same warp. The outline of this optimized implementation is shown in Algorithm 5. 5.3 Efficient k-Nearest Neighbors Bandwidth Selection For Images In Section 2.4.2, we introduced a k-nearest neighbors based bandwidth selection method. The key point of this method is to calculate the covariance matrix of the k-nearest neighbors for each training point X j , j = 1, . . . , n. To compute the covariance matrix at X j , a naive implementation is to find the k-nearest neighbors of X j first and then calculate the covariance matrix using Equation (2.21). However, to find the k-nearest neighbors, one need to calculate the distances between this point and all other training points, and find the k-nearest neighbors points of X j by sorting the resulting distances. It can be easily proven that the time complexity of such k-nearest neighbors search algorithm has O(n2 ) time complexity, where n is the size of training set. Since the size of training set is usually very large, this is clearly computationally intensive. Therefore, to avoid the directly k-nearest neighbors search, we propose a covariance filtering based algorithm in this section. 5.3.1 k-Nearest Neighbors Covariance Matrix of Images Given a set of d-dimensional training points S = {x1 , x2 , . . . , xn } and an image intensity function 1, x ∈ S, I(x) = , (5.22) 0, otherwise. then the k-nearest neighbors covariance matrix at xi can be written as C(xi ) = k 1X (xi − xp(i,j) )(xi − xp(i,j) )T , i = 1, . . . , n, k j=1 59 (5.23) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Algorithm 7 Naive k-Nearest Neighbors Bandwidth Selection 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: procedure I MAGE BANDWIDTH S ELECTION(I, k, σ ) for each point x in image such that I(x) 6= 0 do i←0 for each point y i in the image such that I(x) 6= 0 and x 6= y i do di ← calculate the distance between x and y i i←i+1 end for p(1), p(2), . . . , p(k) ← find the indices of the k smallest distances in D = {d0 , d1 , . . . , di−1 } the covariance matrix C(x) ← 0 for i ← 1, k do C(x) ← C(x) + k1 (x − y p(i) )(x − y p(i) )T end for Q(x), Λ(x) ← E IGENDECOMPOSITION(C(x)) S(x) ← σ −1 Λ(x)−1/2 Q(x)T end for return S end procedure where function p(i, j) returns the index of j-th nearest neighbors of xi . For a 2D image, the training point xi = [xi , yi ]T , xi , yi ∈ Z, i = 1, . . . , n, then k 1 X xi xp(i,j) xi xp(i,j) T C(xi ) = ( − )( − ) k yi yp(i,j) yi yp(i,j) j=1 1 Pk 1 Pk 2 (xi − xp(i,j) ) j=1 (xi − xp(i,j) )(yi − yp(i,j) ) k . = Pk k j=1 1 1 Pk 2 j=1 (xi − xp(i,j) )(yi − yp(i,j) ) j=1 (yi − yp(i,j) ) k k (5.24) Let C11 (xi ) = k 1X (xi − xp(i,j) )2 , k j=1 C12 (xi ) = k 1X (xi − xp(i,j) )(yi − yp(i,j) ), k (5.25) j=1 C22 (xi ) = k 1X (yi − yp(i,j) )2 , k j=1 then Equation (5.24) can be written as C11 (xi ) C12 (xi ) . C(xi ) = C12 (xi ) C22 (xi ) 60 (5.26) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Similarly, for a 3D image, the training point xi = [xi , yi , zi ]T , xi , yi , zi ∈ Z, i = 1, . . . , n and we have where C11 (xi ) C12 (xi ) C13 (xi ) C(xi ) = C12 (xi ) C22 (xi ) C23 (xi ) , C13 (xi ) C23 (xi ) C33 (xi ) C13 (xi ) = (5.27) k 1X (xi − xp(i,j) )(zi − zp(i,j) ), k j=1 C23 (xi ) = k 1X (yi − yp(i,j) )(zi − zp(i,j) ), k (5.28) j=1 C33 (xi ) = k 1X (zi − zp(i,j) )2 . k j=1 5.3.2 r-Neighborhood Covariance Matrix of Images Our algorithm is based on the fact that the locations of the pixels are evenly distributed on the image. Thus, there is a potential that the neighbor searching can be done by filtering. In this section, we give a simple problem where neighbor searching can be completed by filtering easily. Consider a training point xi ∈ S, and a set of training points whose distance from xi is smaller than r, i.e. Nr (xi ) = {x | x ∈ S, x 6= xi , kx − xi k < r}, we want to calculate the covariance matrix of Nr (xi ) at xi . Here, Nr (xi ) is called the r-neighborhood of xi . For a 2D image, according to Equation (5.26) the covariance matrix can be written as D11 (xi ) D12 (xi ) D(xi ) = D12 (xi ) D22 (xi ) (5.29) where D11 (xi ) = |Nr (xi )|−1 D12 (xi ) = |Nr (xi )|−1 D22 (xi ) = |Nr (xi )|−1 X (xi − x)2 , x∈Nr (xi ) X (xi − x)(yi − y), x∈Nr (xi ) X (yi − y)2 , x∈Nr (xi ) 61 (5.30) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS and |Nr (xi )| denotes the number of elements in Nr (xi ). Define the covariance operators h11 , h12 , and h22 and the disk operator d as follows, h11 (x) = h12 (x) = h22 (x) = x2 , kxk < r, 0, otherwise. xy, kxk < r, 0, otherwise. y 2 , kxk < r, , , (5.31) , 0, otherwise. 1, kxk < r, d(x) = . 0, otherwise. Here, both the covariance and disk operators can be expressed by (2r + 1) × (2r + 1) matrices as illustrated in Figure 5.6. It can be easily proved that I(xi ) ∗ h11 (xi ) = I(xi ) ∗ h12 (xi ) = I(xi ) ∗ h22 (xi ) = X (xi − x)2 , x∈Nr (xi ) X (xi − x)(yi − y), x∈Nr (xi ) X (5.32) (yi − y)2 , x∈Nr (xi ) I(xi ) ∗ d(xi ) = |Nr (xi )|. Thus, the covariance matrix of neighborhood Nr (xi ) can be calculated by D11 (xi ) D12 (xi ) D(xi ) = D12 (xi ) D22 (xi ) I(xi ) ∗ h11 (xi ) I(xi ) ∗ h12 (xi ) 1 = I(xi ) ∗ d(xi ) I(xi ) ∗ h12 (xi ) I(xi ) ∗ h22 (xi ) 62 (5.33) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS -4 -3 -2 -1 y 0 1 2 3 4 0 0 0 0 1 0 0 0 0 0 0 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 0 -4 -3 -2 -1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 0 0 0 1 1 1 1 1 0 0 0 0 0 R=4 0 y 1 0 0 0 0 0 1 2 3 4 x -4 -3 -2 -1 0 1 2 3 4 0 0 0 0 0 0 0 0 0 0 9 9 9 9 9 0 0 0 4 4 4 4 4 4 4 0 0 1 1 1 1 1 1 1 0 -4 -3 -2 -1 16 (a) -4 -3 -2 -1 y 0 1 2 3 4 0 0 0 0 0 0 0 0 0 -4 0 0 0 0 6 3 6 4 2 3 2 1 0 0 0 -3 -2 -1 -6 -4 -2 0 -6 -3 0 0 0 -3 -2 -1 0 0 0 0 0 0 0 0 0 0 x 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 4 4 4 4 4 4 4 0 0 0 9 9 9 9 9 0 0 16 0 1 2 3 4 x 0 0 0 R=4 0 0 0 0 0 (b) 0 0 0 -3 -6 0 -2 -4 -6 -1 -2 -3 0 0 0 1 2 3 2 4 6 3 6 0 0 0 0 1 2 3 0 0 0 R=4 0 y 0 0 0 0 0 4 (c) -4 -3 -2 -1 0 1 2 3 4 0 0 0 0 0 0 0 0 0 0 0 4 1 0 1 4 0 0 0 9 4 1 0 1 4 9 0 0 9 4 1 0 1 4 9 0 -4 -3 -2 -1 16 0 0 0 0 9 4 1 0 1 4 9 9 4 1 0 1 4 9 16 0 0 x 1 9 4 1 0 1 4 9 0 0 4 1 0 1 4 0 0 0 0 R=4 0 0 0 0 0 0 2 3 4 (d) F IGURE 5.6: The covariance and disk operators of r = 4. (a): disk operator. (b) h11 covariance operator. (c) h12 covariance operator. (d) h22 covariance operator. Similarly, for a 3D image the covariance matrix of neighborhood Nr (xi ) is given by D11 (xi ) D12 (xi ) D13 (xi ) D(xi ) = D12 (xi ) D22 (xi ) D23 (xi ) D13 (xi ) D23 (xi ) D33 (xi ) I(xi ) ∗ h11 (xi ) I(xi ) ∗ h12 (xi ) I(xi ) ∗ h13 (xi ) 1 I(xi ) ∗ h12 (xi ) I(xi ) ∗ h22 (xi ) I(xi ) ∗ h23 (xi ) = I(xi ) ∗ d(xi ) I(xi ) ∗ h13 (xi ) I(xi ) ∗ h23 (xi ) I(xi ) ∗ h33 (xi ) 63 (5.34) CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS where h13 (x) = h23 (x) = h33 (x) = 5.3.3 Algorithm xz, kxk < r, 0, otherwise. yz, kxk < r, 0, otherwise. z 2 , kxk < r, 0, , , (5.35) . otherwise. R=3 R=4 F IGURE 5.7: Searching circles of different radii. Assume k = 6, we can find that there are only 2 neighbor training points inside the green searching circle of radius 3. Thus, we increase the searching radius by one and find that the orange searching circle of radius 4 contains 6 neighbor training points. Therefor, if we choose a searching radius r = 4, we have C(x) = D(x). Based on the discussion in Section 5.3.1 and 5.3.2, we propose our efficient k-nearest neighbors bandwidth selection algorithm in this section. From the definition of Nr (xi ), we know that ∀x ∈ S and x ∈ / Nr (xi ) : kx − xi k ≥ r. Thus, all the training points inside Nr (xi ) are closer to xi than those training points outside Nr (xi ). Then, the r-neighborhood Nr (xi ) can also be viewed as the |Nr (xi )|-nearest neighbors of xi . Therefore, the covariance matrix C(xi ) = D(xi ) iff 64 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS k = |Nr (xi )|. Since D(xi ) can be easily calculate by filtering, then, as long as we can find a proper r such that |Nr (xi )| = k, we can calculate C(xi ) from D(xi ) efficiently. Therefore, we need to find the correct searching radius r for xi . Assume that all points within the searching circle (or sphere) are training points, which means πr2 = k (or 43 πr3 = k), then we can set 3k −1/3 the initial value of r to d( πk )−1/2 e(or d( 4π ) e). According to Equation (5.32), we can calculate |Nr (xi )| using the disk operator. Compare the value of |Nr (xi )| with k. If |Nr (xi )| < k, then increase r until |Nr (xi )| ≥ k for all xi in S. The increasing step of r determines the performance of this algorithm. A small r will result a very accurate approximation of C(xi ) but the speed will be relatively slow. The minimal choice of the increasing step is 1, since the training points are the indices of pixels on the image. It should be pointed out that different training point xi has different searching radius. Thus, we need to update C(xi ) according to their searching radius r respectively. After we got all the covariance matrices, we can them apply eigendecomposition to these matrices and calculate the bandwidth accordingly. The outline of this algorithm is shown in Algorithm 8. 5.3.4 GPU Implementation One advantage of our algorithm is that it can be easily accelerated by GPUs. First, the calculation of r-neighborhood covariance matrix involves lots of image convolutions which can be easily done by GPUs. One way to achieve this is to calculate the convolution directly via Matlab’s Parallel Computing Toolbox (PCT). The PCT provides the built-in GPU accelerated functions conv2 and convn for 2D and 3D convolution respectively. The other way is to accomplish convolution through FFT base on the convolution theorem [37], F{f ∗ g} = F{f } · F {g}. (5.36) The time complexity of FFT is O(nlog2 n) [38], which is much faster than convolution’s O(n2 ) when n is large. Lots of GPU packages are available for high performance FFT implementation, such as PCT, Jacket, CUFFT, OPenCL FFT. Second, our algorithm need to perform the eigendecomposition to the covariance matrix for each training point in the image. Since there are a large number of training points and their eigendecompositions are independent, it is a good idea to put these computations on GPU. Several GPU libraries (CULA, MAGMA, etc.) are available for computing QR decomposition, but they are only efficient and competitive with large matrices, at least over 1000 × 1000 [39]. Since the covariance matrix in our case is only 2 × 2 or 3 × 3, we implemented our own GPU based function 65 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS Algorithm 8 Efficient k-Nearest Neighbors Bandwidth Selection 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: procedure I MAGE BANDWIDTH S ELECTION(I, k, σ ) if the image I is 2D then Initialize the filtering radius r ← d( πk )−1/2 e else if the image I is 3D then 3k −1/3 Initialize the filtering radius r ← d( 4π ) e end if for each point x in the image do the number of neighbors N (x) ← 0 the covariance matrix C(x) ← 0 end for while exists point x in the image such that N (x) < k and I(x) 6= 0 do d ← create a disk operator with radius r Nr ← filter the image I with the disk operator d if the image I is 2D then h11 , h12 , h22 ← create the covariance operators C11 , C12 , C22 ← filter the image I with covariance operators h11 , h12 , h22 else if the image I is 3D then h11 , h12 , h13 , h22 , h23 , h33 ← create the covariance operators C11 , C12 , C13 ← filter the image I with covariance operators h11 , h12 , h13 C22 , C23 , C33 ← filter the image I with covariance operators h22 , h23 , h33 end if for each point x in the image such that Nr (x) ≥ k and C(x) = 0 do if the image I is 2D then C (x) C12 (x) C(x) ← Nr (x)−1 11 C12 (x) C22 (x) else if the image I is 3D then C11 (x) C12 (x) C13 (x) C(x) ← Nr (x)−1 C12 (x) C22 (x) C23 (x) C13 (x) C23 (x) C33 (x) end if end for r ←r+1 end while for each point x such that I(x) 6= 0 do Q(x), Λ(x) ← E IGENDECOMPOSITION(C(x)) S(x) ← σ −1 Λ(x)−1/2 Q(x)T end for return S end procedure 66 CHAPTER 5. ALGORITHMS AND IMPLEMENTATIONS to perform millions of small matrix eigendecompositions simultaneously. For more details about the performance of our algorithm, please see Chapter 6. 67 Chapter 6 Experiments and Results In this chapter, we present the experimental results of the efficient methods, optimization techniques, and vesselness measures that we introduced in Chapter 3 and 5. We first introduce the hardware environment of the experiments in Section 6.1. Then, in Section 6.2, we investigate the speed performance of the efficient methods and optimization techniques that we used in our kernel smoothing library. Finally, in Section 6.3, we test the overall performance of the kernel smoothing library when applied to two medical imaging techniques. 6.1 Environment The experiments were performed on two platforms. One platform is a GPU node on Northeastern University’s discovery cluster. This node has an NVIDIA Tesla K20m GPU, dual Intel Xeon E5-2670 CPU and 256GB RAM. The NVIDIA Tesla K20m GPU has 2496 cuda cores (13 SMs, 192 cores each), 0.7GHz clock rate, 5GB GDDR RAM, and 3.0 compute capability and we conduct all the GPU experiments using CUDA 6.5 Toolkit. The Intel Xeon E5-2670 CPU has 2.6GHz clock rate and 8 physical cores. Since each physical core has 2 logical cores and there are 2 Intel Xeon E5-2670 CPUs, this platform has 32 logical cores in total. The other platform is a computer with a Intel Core i7-3615QM CPU, an NVIDIA GeForce GT 650M GPU, and 8GB RAM. The NVIDIA GeForce GT 650M GPU has 0.9GHz clock rate, 384 cuda cores (2 SMs, 192 cores each), and 1GB GDDR RAM. The Intel Core i7-3615QM CPU has 2.3GHz clock rate and 8 logical cores (4 physical cores, 2 logical core each). A summary of the specification of these two platform is given in Table 6.1. 68 CHAPTER 6. EXPERIMENTS AND RESULTS GPU CPU NVIDIA NVIDIA GeForce Intel Xeon Intel Core Tesla K20m GT 650M E5-2670 i7-3615QM Clock Rate 0.7 GHz 0.9 GHz 2.60 GHz 2.30 GHz GPU/CPU Cores 2496 384 8 4 Device/Host Memory 5120 MB 1024 MB 256GB 8GB Name TABLE 6.1: Experiment environment. 6.2 Performance Evaluation In this section, we present the experimental results of algorithms introduced in Chapter 5. We first evaluate the performance of the efficient separable multivariate kernel derivative (SMKD) algorithm, in Section 6.2.1, by providing the visualized complexity analysis as well as the detailed running time between the naive and the efficient algorithms. In Section 6.2.2, we compare the speed-ups between different versions of high-speed KDE and KDDE methods. An memory performance analysis is also provided to illustrate the effectiveness of different memory optimization techniques used in these methods. Finally, the performance comparison results of the efficient k-NN bandwidth selection method on CPU and GPU platforms are given in Section 6.2.3. 6.2.1 Efficient SMKD We perform all the experiments for efficient SMKD algorithm on the Intel Xeon E5-2670 CPU platform. For the first set of experiments, we compare the theoretical number of multiplications between the naive and efficient algorithms with different dimensions and orders at a single sample point. This comparison is based on the complexity analysis Equation (5.9) and (5.10). The experimental results are given in Figure 6.1. As can be seen from the figures, the efficient algorithm outperforms the naive algorithm significantly as the dimension and derivative order increase. For example, the top left figure shows that when the order and data dimension is low, there is only a slightly difference in multiplication number between naive algorithm and efficient algorithm. However, when the order and data dimension increases, as shown in the bottom right figure, the multiplication number of the naive algorithm can be several times higher than the efficient algorithm. 69 number of multiplications number of multiplications number of multiplications order = 1 2000 naive algorithm efficient algorithm 1500 1000 500 0 5 0 #10 10 20 30 40 dimension order = 3 5 naive algorithm efficient algorithm 4 3 2 1 0 5 0 10 20 30 40 dimension order = 5 #10 7 naive algorithm efficient algorithm 4 3 2 1 0 2.5 0 10 20 30 40 dimension order = 7 #10 9 number of multiplications number of multiplications number of multiplications number of multiplications number of multiplications CHAPTER 6. EXPERIMENTS AND RESULTS naive algorithm efficient algorithm 2 1.5 1 0.5 0 0 10 20 30 40 dimension 4 #10 4 order = 2 naive algorithm efficient algorithm 3 2 1 0 5 0 #10 10 20 30 40 30 40 30 40 30 40 dimension order = 4 6 naive algorithm efficient algorithm 4 3 2 1 0 4 0 10 20 dimension order = 6 #10 8 naive algorithm efficient algorithm 3 2 1 0 15 0 10 20 dimension order = 8 #10 9 naive algorithm efficient algorithm 10 5 0 0 10 20 dimension F IGURE 6.1: The comparison of the number of multiplications in computing different orders of derivatives of separable multivariate kernel function using the naive method and the proposed efficient method with dimensions from 1 to 40. Next, we test the performance of the naive and efficient algorithms on the synthetic data. The synthetic data is generated based on the univariate Gaussian kernel N (0, 1) and its derivatives. For both algorithms, some basic memory optimization techniques are used to minimize the latency introduced by memory operations. Thus, our experiments focus mostly on the computational 70 CHAPTER 6. EXPERIMENTS AND RESULTS differences between these two algorithms. The experimental results is given in Figure 6.2. We investigate the performance of these two algorithms at different orders and dimensions. Here, we choose the orders from 1 to 6 and dimensions from 2 to 20. As we can see from the top left figure that when the order is 1 and dimension is smaller than 20, the proposed algorithm is slightly slower than the naive algorithm. It is because that there is a constant computational overhead when deciding (j) the number of elements in set Ni . And as the order and dimension increase, the proposed efficient algorithm outperforms the naive algorithm significantly. The execution time increasing rate in Figure 6.2 is consistent with the multiplication number increasing number in Figure 6.1. It proves that our complexity analysis in Section 5.1.3 is correct. 6.2.2 High Performance KDE and KDDE We perform our CPU experiments on the Intel Core i7-3615QM platform and perform our GPU experiments on the NVIDIA Tesla K20m platform. Because the speed performances of KDE and KDDE algorithms are insensitive to the data type. Our experiments are based on the synthetic data. We generate the synthetic training points and test points from random number generators directly. The experiments are divided into three groups. For the first group, we investigate the speed-ups of our different optimization methods on synthetic 2D data. For the second group, we discover the speed performance of different optimization methods on synthetic 3D data. And for the final group, we try to find the GPU device memory performance of these methods. In all these experiments, we use the functions, which can perform kernel density estimation, kernel gradient estimation and kernel curvature estimation at the same time, from our kernel smoothing library. For the first group, we perform two set of experiments. First, as can be seen in the left bar graph of Figure 6.3, we present the speed-up comparison between CPU serial, CPU parallel, and GPU naive methods. Four experiments have been performed with data size ranging from 107 to 1010 . The data size is denoted by m × n, where m is the number of test points and n is the number of training points. Here, we can see that the CPU parallel method is almost three to four times faster than the CPU serial methods. It is reasonable, since the Intel Core i7-3615QM contains four CPU cores. We also can find that the naive GPU method is way more fast than the CPU methods. Especially, when the data size is 1010 , the naive GPU method can be more than 100 times faster than the serial GPU method. However, the naive GPU method doesn’t achieve its best performance when the data size is small. The reason is that the naive GPU method can’t achieve a good occupancy when the workload is low. 71 CHAPTER 6. EXPERIMENTS AND RESULTS Order = 1 naive algorithm efficient algorithm 0.01 0.008 0.006 0.004 0.002 0 0 5 10 Order = 2 0.1 Execution Time Execution Time 0.012 15 naive algorithm efficient algorithm 0.08 0.06 0.04 0.02 0 20 0 5 Dimension Order = 3 naive algorithm efficient algorithm 0.6 0.4 0.2 0 0 5 10 15 Execution Time Execution Time 20 15 10 5 5 10 15 20 1 0 5 10 Order = 6 120 naive algorithm efficient algorithm 0 20 2 Dimension 25 0 15 3 0 20 Order = 5 30 20 naive algorithm efficient algorithm 4 Dimension 35 15 Order = 4 5 Execution Time Execution Time 0.8 10 Dimension 15 80 60 40 20 0 20 Dimension naive algorithm efficient algorithm 100 0 5 10 Dimension F IGURE 6.2: The execution time of naive method and the proposed efficient method in computing the different orders of derivatives of the multivariate kernel function on synthetic data. Here the number of samples is 10000, and the data dimension ranges from 2 to 20. Second, the right bar graph shows the performance differences between GPU optimization methods. Since the GPU methods are much faster than CPU methods, we use bigger data sizes in this set of experiments. We can see that the GPU Optimization I method is almost two to three times faster than the naive method, which means the kernel merging, loop unrolling and memory layout optimization techniques can bring about two to three times speed up to naive method. Then, we can 72 CHAPTER 6. EXPERIMENTS AND RESULTS 120 100 40 CPU Serial CPU Parallel GPU Naive 35 GPU Naive GPU Optimization I GPU Optimization II GPU Optimization III 30 Speed Up Speed Up 80 60 25 20 15 40 10 20 5 0 1e+07 1e+08 1e+09 0 1e+10 1e+09 Data Size 1e+10 1e+11 1e+12 Data Size F IGURE 6.3: The comparison of speed-ups between different optimization methods on synthetic 2D data. find that if the data size is large the Optimization II method can bring about 7 times speed up to the Optimization I method. It looks like the simplification of Gaussian kernel and the removal of outer loop bring lots of benefits to Optimization II method. The right bar graph also shows that the Optimization III method speeds up the Optimization II method by two times, when the data size is large. However, we can find that when the data size is small, the Optimization III algorithm has a bad performance. It is even slower than the Optimization II method when the data size is 109 . This is because that we need to rearrange the memory layout for Optimization III method. The cost of this rearrangement is constant and is ignored when the total running time is long. But when the data size is small, this rearrangement can’t be ignored and thus will affect the overall performance of Optimization III method. 80 70 60 CPU Serial CPU Parallel GPU Naive 50 GPU Naive GPU Optimization I GPU Optimization II GPU Optimization III 60 Speed Up Speed Up 40 50 40 30 30 20 20 10 10 0 1e+07 1e+08 1e+09 0 1e+10 Data Size 1e+09 1e+10 1e+11 1e+12 Data Size F IGURE 6.4: The comparison of speed-ups between different optimization methods on synthetic 3D data. 73 CHAPTER 6. EXPERIMENTS AND RESULTS For the second group, we do the similar experiments as the first group, except that this time we test their performances on 3D data. The test results are given in Figure 6.4. We can see that, for 3D data, the performance results in the second group are close to the results in the first group. One thing we should notice is that, in the 3D case, the Optimization III method is four to five times faster than the Optimization II method, which is much better than the 2D case. It is because that the data structure in the 3D case is more complex than the 2D case. Thus, more global memory operations are involved when computing the 3D data. Hence, once the shared memory is introduced, the 3D KDE and KDDE algorithm can have more benefits. GPU Implementation Global Memory Transactions Naive 16.4M Optimization I 1.96M Optimization II 1.56M Optimization III 11.7K TABLE 6.2: Global memory transactions between different optimization methods. For the GPU optimization methods, we perform the third group of experiments. We analysis the number of global memory transactions for each optimization method using the CUDA GPU profiler gprof. The experimental results are shown in Table 6.2. Here, the size of data is 106 . We can see that, from the naive method to the Optimization I method, the number of global memory transactions is decreased by 8 times. Such a big improvement is because both kernel merging and memory layout optimization techniques aim at reducing global memory transactions. And from Optimization I to Optimization II, there is only a slight global memory transaction decrease. Since Gaussian kernel simplification only focuses on reducing computation and the outer loop removal only focuses on decreasing gpu-kernel call overhead, this result is reasonable. We can also see that the use of shared memory reduces global memory transactions significantly. From Optimization II to Optimization III the number of global memory transactions has been decreased more than 100 times. However, according to the quantitative global memory analysis given by Equation (5.19) and (5.19), 74 CHAPTER 6. EXPERIMENTS AND RESULTS we know that the theoretical global memory transaction differences should be b × c times. Since, in the experiment, the block size we use is 1024 and the memory coalescing factor c is 32, then the theoretical global memory transaction decreasing should be 32768 times, which is much bigger than our experimental result. The reason of this problem is because, in the Optimization II method, the GPU L1 cache will be used to help reduce global memory transactions. 6.2.3 Efficient k-NN Bandwidth Selector In this section, we perform our CPU experiments on the Intel Core i7-3615QM platform and perform our GPU experiments on the NVIDIA Tesla K20m platform. We investigate the performance of naive and efficient algorithms. For the efficient algorithm, we implemented it on both CPU and GPU, and we call them the GPU efficient algorithm and the CPU efficient algorithm. We divide the experiments into two groups. In the first group, we test the execution time of naive and efficient k-NN bandwidth selector on 2D images. And in the second group, we conduct our experiments subject to naive, CPU efficient and GPU efficient algorithms. We test their performances on 3D images. 20 18 Naive Efficient CPU 16 Execution Time 14 12 10 8 6 4 2 0 64 # 64 64 # 128 128 # 128 128 # 256 256 # 256 Image Size F IGURE 6.5: Performance of the k-NN bandwidth selector on 2D images using naive algorithm and CPU efficient algorithm. For the first group, the experimental results are shown in Figure 6.5. Here, we perform our 75 CHAPTER 6. EXPERIMENTS AND RESULTS experiments with five different image sizes. When the image size is small, we can find that the performances of the naive algorithm and efficient algorithm are similar. However, the execution time of the naive algorithm increases much faster than the efficient algorithm as the image size increases. When the image size increases to 256 × 256, the efficient algorithm is 6 times faster than the naive algorithm. 100 90 Naive Efficient CPU Efficient GPU 80 Execution Time 70 60 50 40 30 20 10 0 32 # 32 # 32 32 # 32 # 64 64 # 64 # 64 64 # 64 # 128 128 # 128 # 128 Image Size F IGURE 6.6: Performance of the k-NN bandwidth selector on 3D images using naive algorithm, CPU efficient algorithm and GPU efficient algorithm. For the second group, five different 3D image sizes are used in the experiments. For image size of 32 × 32 × 32 and 32 × 32 × 64, we conduct experiments subject to naive, CPU efficient and GPU efficient algorithms. For larger 3D image sizes, the naive algorithm will face an out of memory problem, which is because this algorithm need to store a huge distant map between points. Hence, we only conduct our experiments subject to GPU and CPU efficient algorithm on large 3D images. The experimental results are shown in Figure 6.6. We can see that the GPU efficient algorithm is way more faster than the CPU efficient algorithm and the naive algorithm. 76 CHAPTER 6. EXPERIMENTS AND RESULTS 6.3 Vesselness Measure In this section, we apply our kernel smoothing library to the two vesselness measure algorithms introduced in Section 3.3 and Section 3.4. We compare both the filtering results and speed performances of these algorithms when using and not using our kernel smoothing library. All the experiments are conducted on the discovery cluster platform with NVIDIA Tesla K20m GPU and Intel Xeon E5-2670 CPU. 6.3.1 Frangi Filtering As we discussed in Section 3.3, a Frangi filtering based vesselness measure uses the eigenvalues of the Hessian matrices obtained from an image to analysis the likelihood of a pixel being on the tubular structure. To calculate the Hessian matrices, there are three different ways to achieve this: gradient operator, Gaussian smoothing, and KDDE (See Section 3.2). In the original Frangi paper [3], the Hessian matrix is computed through Gaussian smoothing, which is actually identical with the binned estimation method of kernel density estimation theory. However, this method only uses constrained bandwidth meaning that the same smoothing is applied to every coordinate direction. Therefore, to get a more accurate Hessian matrix, one can choose the variable and unconstrained bandwidth for their kernel density derivative estimator. In this way, the estimator is allowed to smooth in any direction whether coordinate or not. In this section, we implement the Frangi filter in three different ways. For the first way, we implement the filter exactly as the original paper. We use the Gaussian smoothing method to calculate the Hessian matrices and perform all the calculation on CPU only. For the second way, we calculate the Hessian matrices using variable unconstrained bandwidth KDDE. This is still performed on CPU. The third way is similar with the second way, except that we now use our GPU accelerated kernel smoothing library to calculate KDDE. We test the performance of these implementations respectively. The vesselness measure results are given in Figure 6.7. We can see that the Frangi filtering result using KDDE, in the middle, gives more details about the retina image than the result using Gaussian smoothing. However, to get such a good vesselness measure result is extremely expensive. The Gaussian smoothing based Frangi filtering only takes 0.38 seconds to get the result. However, if we calculate using KDDE with CPU only, the execution time would be 3433 seconds! Fortunately, this can be accelerated using our GPU accelerated kernel smoothing library, and the execution time is 14.9 seconds. In this case, we only spend a little bit more time to get a much better vesselness measure result. 77 CHAPTER 6. EXPERIMENTS AND RESULTS 6.3.2 Ridgeness Filtering We implement a ridgeness filtering based vessel segmentation algorithm according to the pipeline in Figure 6.8. For a given image, we first perform preprocessing algorithms, such as anisotropic diffusion, unsharp masking, adaptive histogram equalization, etc, to suppress background noise and highlight tubular structures. Then, for the preprocessed image, we use k-N N bandwidth selector to calculate the bandwidth of each training points (nonzero points). Based on the bandwidths and the preprocessed image, we calculate kernel density estimates f , kernel gradient estimates g, and kernel curvature estimates H. For each kernel curvature estimate H, we use lamabda selector to calculate its largest absolute eigenvalue |λ|max . We compute the ridgeness scores s using ridgeness f ilter from kernel gradient estimates and kernel curvature estimates. Finally, based on the values of |λ|max , s, and f , the classifier makes a combined decision on the vesselness of each pixel. Before outputing the segmented image, a postprocessing procedure is used to refine the results from the classifier. The vessel segmentation results are shown in Figure 6.7. We compare the results of this algorithm when using and not using our kernel smoothing library. As can be seen from the figure that the results are exactly the same. It means our kernel smoothing library won’t bring any inaccuracies when accelerating the algorithm. The total execution time of the GPU accelerated implementation is 12.56 seconds. And the total execution time of the non-GPU implementation is 938.7293 seconds. It shows that we achieved 75 times speed-up when using our kernel smoothing library for the ridgeness filtering based vessel segmentation. 78 CHAPTER 6. EXPERIMENTS AND RESULTS F IGURE 6.7: Vesselness measure results using Frangi filter. Top: Original image. Middle: Frangi filtering result using KDDE. Bottom: Frangi filtering result using Gaussian smoothing. 79 CHAPTER 6. EXPERIMENTS AND RESULTS Image Preprocessing k-NN Bandwidth Selector KDDE Lambda Selector Ridgeness Filtering Classifier Postprocessing Output F IGURE 6.8: Algorithm pipeline of the ridgeness filtering based vessel segmentation. 80 CHAPTER 6. EXPERIMENTS AND RESULTS F IGURE 6.9: Vesselness measure results using ridgeness filter. Top: Original image. Middle: Ridgeness filtering result with GPU. Bottom: Ridgeness filtering result without GPU. 81 Chapter 7 Conclusion and Future Work 7.1 Conclusion We started our discussion with the background introduction of kernel smoothing theory in Chapter 1. Then, in Chapter 2, 3 and 4, we provided the essential background knowledge for the discussion in Chapter 5 and 6. In Chapter 2, Section 2.2 and 2.5 gave a detailed knowledge about the kernel density and kernel density derivative estimation that we need for implementing the high performance functions in Section 5.2. Section 2.3 introduced the separable multivariate kernels, which provided the foundation for our discussion in Section 5.1. The discussion of k-nearest neighbors bandwidth selection, in Section 2.4, helped the understanding of the efficient method in Section 5.3. In Chapter 3, we introduced two vesselness measure algorithms in Section 3.3 and 3.4. We used these two algorithms to demonstrate the full potential of our kernel smoothing library in applications. Section 3.1 and 3.2 provided the knowledge background of these two algorithms. Section 6.3 provided the performance of these two algorithms when using the kernel smoothing library. In Chapter 4, we gave a detailed introduction of the GPU architecture and the CUDA programming framework. It helped to understand the optimization techniques we used in Section 5.2. Based on the background knowledge introduced in the previous chapters, we presented three major contributions of our kernel smoothing library in Chapter 5. First, we proposed an efficient method to calculate the separable multivariate kernel derivative. Second, we implemented the kernel density and kernel density derivative estimators using several optimization techniques on multi-core CPU and GPU platforms. Third, we also designed an efficient k-nearest neighbors bandwidth selection algorithm for image processing. We provided a GPU implementation for this algorithm 82 CHAPTER 7. CONCLUSION AND FUTURE WORK as well. In Chapter 6, we designed a series of experiments to evaluate the performance of the algorithms and implementations we presented in Chapter 5. It shows that the presented algorithms and implementations achieved significant speed-ups than their direct or naive counterparts. The performance evaluation of our kernel smoothing library on two vesselness measure algorithms was provided as well. 7.2 Future Work There are several places we can improve in the future. First, in the current version, our kernel smoothing library only implemented the GPU accelerated KDE and KDDE functions for 2D and 3D data. In the future, we can add a GPU implementation for higher dimensional data. Second, the bandwidth selection methods are crucial in kernel smoothing. But we only implemented one bandwidth selection method in our library. We should add more implementations for bandwidth selection methods. Since some bandwidth selection methods are also computational intensive, there is a potential to implement them on GPU. Finally, object-oriented programming can be used in our library. 83 Bibliography [1] M. Rosenblatt et al., “Remarks on some nonparametric estimates of a density function,” The Annals of Mathematical Statistics, vol. 27, no. 3, pp. 832–837, 1956. [2] E. Parzen, “On estimation of a probability density function and mode,” The annals of mathematical statistics, pp. 1065–1076, 1962. [3] A. F. Frangi, W. J. Niessen, K. L. Vincken, and M. A. Viergever, “Multiscale vessel enhancement filtering,” in Medical Image Computing and Computer-Assisted Interventation—MICCAI’98. Springer, 1998, pp. 130–137. [4] B. Silverman, “Algorithm as 176: Kernel density estimation using the fast fourier transform,” Applied Statistics, pp. 93–99, 1982. [5] M. Wand, “Fast computation of multivariate kernel estimators,” Journal of Computational and Graphical Statistics, vol. 3, no. 4, pp. 433–445, 1994. [6] A. Elgammal, R. Duraiswami, and L. S. Davis, “Efficient kernel density estimation using the fast gauss transform with applications to color modeling and tracking,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 25, no. 11, pp. 1499–1504, 2003. [7] C. Yang, R. Duraiswami, N. A. Gumerov, and L. Davis, “Improved fast gauss transform and efficient kernel density estimation,” in Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on. IEEE, 2003, pp. 664–671. [8] A. Sinha and S. Gupta, “Fast estimation of nonparametric kernel density through pddp, and its application in texture synthesis.” in BCS Int. Acad. Conf., 2008, pp. 225–236. [9] J. M. Phillips, “ε-samples for kernels,” in Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms. SIAM, 2013, pp. 1622–1632. 84 BIBLIOGRAPHY [10] Y. Zheng, J. Jestes, J. M. Phillips, and F. Li, “Quality and efficiency for kernel density estimates in large data,” in Proceedings of the 2013 ACM SIGMOD International Conference on ACM, 2013, pp. 433–444. Management of Data. [11] S. Łukasik, “Parallel computing of kernel density estimates with mpi,” in Computational Science–ICCS 2007. Springer, 2007, pp. 726–733. [12] J. Racine, “Parallel distributed kernel estimation,” Computational Statistics & Data Analysis, vol. 40, no. 2, pp. 293–302, 2002. [13] P. D. Michailidis and K. G. Margaritis, “Parallel computing of kernel density estimation with different multi-core programming models,” in Parallel, Distributed and Network-Based Processing (PDP), 2013 21st Euromicro International Conference on. IEEE, 2013, pp. 77–85. [14] ——, “Accelerating kernel density estimation on the gpu using the cuda framework,” Applied Mathematical Sciences, vol. 7, no. 30, pp. 1447–1476, 2013. [15] W. Andrzejewski, A. Gramacki, and J. Gramacki, “Graphics processing units in acceleration of bandwidth selection for kernel density estimation,” International Journal of Applied Mathematics and Computer Science, vol. 23, no. 4, pp. 869–885, 2013. [16] T. Duong et al., “ks: Kernel density estimation and kernel discriminant analysis for multivariate data in r,” Journal of Statistical Software, vol. 21, no. 7, pp. 1–16, 2007. [17] M. Wand and B. Ripley, “Kernsmooth: Functions for kernel smoothing for wand & jones (1995),” R package version, vol. 2, pp. 22–19, 2006. [18] T. Hayfield and J. S. Racine, “Nonparametric econometrics: The np package,” Journal of statistical software, vol. 27, no. 5, pp. 1–32, 2008. [19] A. W. Bowman and A. Azzalini, Applied Smoothing Techniques for Data Analysis: The Kernel Approach with S-Plus Illustrations: The Kernel Approach with S-Plus Illustrations. Oxford University Press, 1997. [20] V. A. Epanechnikov, “Non-parametric estimation of a multivariate probability density,” Theory of Probability & Its Applications, vol. 14, no. 1, pp. 153–158, 1969. [21] M. Shaker, J. N. Myhre, and D. Erdogmus, “Computationally efficient exact calculation of kernel density derivatives,” Journal of Signal Processing Systems, pp. 1–12, 2014. 85 BIBLIOGRAPHY [22] B. W. Silverman, Density estimation for statistics and data analysis. CRC press, 1986, vol. 26. [23] T. Duong, Bandwidth selectors for multivariate kernel density estimation. University of Western Australia, 2004. [24] G. R. Terrell and D. W. Scott, “Variable kernel density estimation,” The Annals of Statistics, pp. 1236–1265, 1992. [25] M. Jones, “Variable kernel density estimates and variable kernel density estimates,” Australian Journal of Statistics, vol. 32, no. 3, pp. 361–371, 1990. [26] I. S. Abramson, “On bandwidth variation in kernel estimates-a square root law,” The Annals of Statistics, pp. 1217–1223, 1982. [27] L. Breiman, W. Meisel, and E. Purcell, “Variable kernel estimates of multivariate densities,” Technometrics, vol. 19, no. 2, pp. 135–144, 1977. [28] J. E. Chacón, T. Duon, and M. Wand, “Asymptotics for general multivariate kernel density derivative estimators,” 2009. [29] J. E. Chacón, T. Duong et al., “Data-driven density derivative estimation, with applications to nonparametric clustering and bump hunting,” Electronic Journal of Statistics, vol. 7, pp. 499–532, 2013. [30] J. R. Magnus and H. Neudecker, “Matrix differential calculus with applications in statistics and econometrics,” 1995. [31] H. V. Henderson and S. Searle, “Vec and vech operators for matrices, with some uses in jacobians and multivariate statistics,” Canadian Journal of Statistics, vol. 7, no. 1, pp. 65–81, 1979. [32] T. Duong, A. Cowling, I. Koch, and M. Wand, “Feature significance for multivariate kernel density estimation,” Computational Statistics & Data Analysis, vol. 52, no. 9, pp. 4225–4242, 2008. [33] T. M. Apostol, “Mathematical analysis,” 1974. [34] C. E. Shannon, “Communication in the presence of noise,” Proceedings of the IRE, vol. 37, no. 1, pp. 10–21, 1949. 86 BIBLIOGRAPHY [35] U. Ozertem and D. Erdogmus, “Locally defined principal curves and surfaces,” The Journal of Machine Learning Research, vol. 12, pp. 1249–1286, 2011. [36] E. Bas and D. Erdogmus, “Principal curves as skeletons of tubular objects,” Neuroinformatics, vol. 9, no. 2-3, pp. 181–191, 2011. [37] Y. Katznelson, An introduction to harmonic analysis. Cambridge University Press, 2004. [38] V. Y. Pan, “The trade-off between the additive complexity and the asynchronicity of linear and bilinear algorithms,” Information processing letters, vol. 22, no. 1, pp. 11–14, 1986. [39] R. Solcà, T. C. Schulthess, A. Haidar, S. Tomov, I. Yamazaki, and J. Dongarra, “A hybrid hermitian general eigenvalue solver,” arXiv preprint arXiv:1207.1773, 2012. 87