INF-VERK 3830/4830 project assignment: Image denoising using simple diffusion operators Image denoising refers to the removal of noises from a noise-contaminated image, such that the restored image becomes closer to the original noise-free image. Since noisy images are present in many real-life situations, image denoising has become an important task in modern use of computers. An example of image denoising is illustrated in Figure 1 below. Figure 1: Left: a noisy image; Light: a denoised image after applying a simple diffusion operator. The purpose of this project assignment is to let the students get familiarized with the following important aspects of scientific programming: 1. Translation of simple mathematical formulas to a working code. 2. Input and output of data files in both ascii and binary format. 1 3. Simple program structure design via object orientation. 4. Compilation and use of external software libaries. 1 Introduction An image can be thought as a mathematical function u(x, y), which only has discrete values on a rectangle of pixels. For an image with m × n pixels, this means that u has the following values u(m, 1) u(m, 2) · · · u(m, n) .. .. .. .. . . . . u(2, 1) u(2, 2) · · · u(2, n) u(1, 1) u(1, 2) · · · u(1, n) For simplicity, the discrete values of u(x, y) can also be denoted by u1,1 , u1,2 , . . . , um,n . By a noisy image we mean that u(x, y) = s(x, y) + f (x, y), (1) where s(x, y) is the original image without noise, whereas f (x, y) is the noisy addition having a random characteristic. The purpose of image denoising is to restore s(x, y) from u(x, y), on the basis of some (limited) knowledge of the random characteristic of f (x, y). Often, the noisy addition f (x, y) is a fastoscillating random function, so the contaminated image u(x, y) displays a noisy feature. Image denoising is a non-trivial task, because we at the best only have some partial description of f (x, y). (For example, when f (x, y) is a Gaussian noise we may be lucky to know its mean value and the standard deviation.) 2 Denoising by Diffusion There exist many image denoising methods, with varying degrees of complexity and effectiveness. We will in this project only concentrate on a simple class of denoising methods that make use of diffusion operators. The simplest diffusion-based denoising method is defined as follows: ū(x, y) = u(x, y) + ∆t∇2 u(x, y), (2) where ū(x, y) denotes the resulting denoised image, ∆t is a prescribed constant, and ∇2 u(x, y) denotes the Laplacian of the noisy image u(x, y). More specifically, ∇2 u = ∇ · (∇u) = 2 ∂2u ∂2u + 2. ∂x2 ∂y (3) Note that the result of the³Laplacian ´ can be equally achieved by first computing ∂u ∂u the gradient vector ∇u = ∂x , ∂y , and then taking the divergence (∇ · ) of the gradient vector. The hope is that ū, arising from formula (2), becomes a better representation of the original image s. This is because the above denoising method is able to “smooth out” the noises to a certain extent. In real-life applications, the operation of formula (2) needs to be repeated several times to achieve a desirable denoising effect. That is, For i = 1, 2, . . . , until a given number of iterations: Apply one iteration of denoising: ū(x, y) = u(x, y) + ∆t∇2 u(x, y) Copy ū(x, y) into u(x, y): u(x, y) = ū(x, y) Go back and repeat The above image denoising strategy can be called isotropic diffusion because the amount of diffusion happens isotropically, independent of the spatial location (x, y). A replacement of formula (2) thus arises from using anisotropic diffusion as follows: ū(x, y) = u(x, y) + ∆t∇ · 3 ∇u ¡ ∂u ¢2 ³ ∂u ´2 . 1 + ∂x + ∂y (4) Numerical Differentiation To implement the two image denoising strategies described above, we have to be able to compute the partial derivatives of u(x, y) numerically, because the u function only has values on the discrete pixels. Recall that we have the m×n discrete values of u(x, y) as u1,1 , u1,2 , . . . , um,n . The following finite difference formulas should be used to compute the needed numerical partial derivatives: ∇2 u(xi , yj ) ∂u (x1 , yj ) ∂x ∂u (xi , yj ) ∂x ∂u (xm , yj ) ∂x ∂u (xi , y1 ) ∂y ∂u (xi , yj ), ∂y ∂u (xi , yn ) ∂y ≈ ui,j−1 + ui−1,j − 4ui,j + ui+1,j + ui,j+1 , 0, ≈ u2,j − u1,j , 1≤j≤n ≈ ui+1,j − ui−1,j , 2 ≈ um,j − um−1,j , ≈ ui,2 − ui,1 , ≈ ui,j+1 − ui,j−1 , 2 ≈ ui,n − ui,n−1 , 2 ≤ i ≤ m − 1, 2 ≤ j ≤ n − 1 (5) otherwise (one-sided difference) 2 ≤ i ≤ m − 1, 1 ≤ j ≤ n 1≤j≤n 1≤i≤m (centered difference) (one-sided difference) (one-sided difference) 1 ≤ i ≤ m, 2 ≤ j ≤ n − 1 1≤i≤m 3 (centered difference) (one-sided difference) (6) (7) (8) (9) (10) (11) It is clear that the numerical Laplacian needed by formula (2) is given straightforwardly by formula (5). However, the computation needed in formula (4) should be carried out by first computing an intermediate discrete vector function (gx , gy ) as gx (x, y) = 1+ gy (x, y) ∂u ∂x ¡ ∂u ¢2 + ∂x ∂u ∂y = 1+ ¡ ∂u ¢2 ∂x + ³ ³ ∂u ∂y ∂u ∂y ´2 , ´2 , and then completing the compuation of formula (4) as µ ¶ ∂gy ∂gx ū(x, y) = u(x, y) + ∆t + . ∂x ∂y 4 The Project Assignment The above two diffusion-based image denoising strategies should be implemented in the project assignment, together with supporting functionality. The implementation should be done in both C and C++, where the C++ version is basically an object-oriented redesign of the C version. 4.1 The C Implementation The following data structure should be adopted for representing an image with m × n pixels: struct image { float** image_data; int m; int n; }; /* a 2D array of float */ /* # pixels in x-direction */ /* # pixels in y-direction */ Based on struct image, the following three functions should be implemented for memory allocation (of the image data array), memory deallocation, and copy from an image to another, respectively: void allocate_image (image *u, int m, int n); void deallocate_image (image *u); void copy_image (const image *a, image *b); /* copy from a to b */ Moreover, the following functions should be implemented for reading and writing an image from/to data file: void void void void read_ascii_file (const char* filename, image* a); write_ascii_file (const char* filename, const image* a); read_binary_file (const char* filename, image* a); write_binary_file (const char* filename, const image* a); 4 where the format of an ascii image file is assumed to be as follows: m=xxx n=xxx u(1,1)=xxx u(1,2)=xxx u(1,3)=xxx ... The format of a binary image file should be determined by the student herself/himself. Finally, the following two functions should implement the isotropic and anisotropic diffusion-based image denoising strategies, respectively: void iso_duffsion_denoising (image* u, image* u_bar, float dt, int num_iters); void ani_duffsion_denoising (image* u, image* u_bar, float dt, int num_iters); All the above C fucntions are considered by the project as the C version of a simple user-defined library for image-denoising. 4.2 The C++ Implementation A C++ class with name Image should be implemented to replace the C data structure struct image. This new C++ class should also incorporate similar member functions as the above C functions (allocate image, deallocate image, copy image, read ascii file, write ascii file, read binary file, write binary file). Moreover, another C++ class with name DiffusionDenoising should be implemented with the following structure: class DiffusionDenoising { protected: int num_iters; float dt; public: DiffusionDenoising (); ~DiffusionDenoising (); setNumIters (int iters); setDt (float dt); virtual void denoising (Image& u, Image& u_bar) =0; }; Note that class DiffusionDenoising is to act as a pure virtual base class, whose member function denoising has to be explicitly implemented in its subclasses. As two concrete subclasses, class IsotropicDiffusionDenoising and class AnisotropicDiffusionDenoising should be develped, each having a concrete implementation of the member function denoising. The above four C++ classes thus constitute the C++ version of a simple user-defined library for image-denoising. 5 4.3 Main Program in C A main program should be implemented in C to test all the functions belonging to the C version of the user-defined library. The following ascii file contains a 100 × 100 pixel image s(x, y) without noise: http://heim.ifi.uio.no/~xingca/inf-verk3830/noise_free_image.txt To generate a noisy image u(x, y) = s(x, y) + f (x, y), the standard C function rand should be used to add a random noise of range −20 ≤ f (x, y) ≤ 20 on top of s(x, y). Note that each call of rand returns a random number between 0 and RAND MAX, so some kind of scaling and shifting must be applied to obtain random values between -20 and 20 (see more info by e.g. google rand function). When u(x, y) is generated, five iterations of both the isotropic and anisotropic denoising strategies should be carried out, with ∆t = 0.5. The five denoising iterations should be carried out one by one, so that the deviation from ū(x, y) to the original noise-free image s(x, y) can be checked (by computing the Euclidean norm of [ū1,1 − s1,1 , ū1,2 − s1,2 , . . . ūm,n − sm,n ]). The main program in C should also write one of the denoised images to a binary data file (using function write binary file). This binary data file should then be read in (using function read binary file) to make sure that the two functions and the format of the binary data file works as intended. 4.4 Main Program in C++ Using an External Library Another C++ main program, making use of the C++ version of the user-defined library, should also be developed. All the actions implemented in the C main should be repeated. In addition, a real-life noisy image (the left picture of Figure 1) should be tested. However, this is a JPEG formatted image, http://heim.ifi.uio.no/~xingca/inf-verk3830/noisy-paprika.jpg which can not be read directly by either read ascii file or read binary file. To enable import and export of JPEG formatted images, an extenal C library package is to be used. More specifically, the following gzipped tar file http://heim.ifi.uio.no/~xingca/inf-verk3830/simple-jpeg.tar.gz should be downloaded and packed out under a separate directory (e.g. with directory name external code). It will be seen that the external C library package contains a set of header files and C files. A makefile should be written so that all the C files are compiled as an external library file (e.g. with name libsimplejpeg.a). Only two functions within the external library are meant for use in the project: void import_JPEG_file (const char* filename, unsigned char** image_chars, int* image_height, int* image_width, int* num_components); void export_JPEG_file (const char* filename, const unsigned char* image_chars, int image_height, int image_width, int num_components, int quality); 6 These two functions can be used, respectively, to read and write a data file of the JPEG format. We remark that each pixel in a grey-scale JPEG image uses one byte, and a one-dimensional array of unsigned char (of total length mn) is used to contain all the pixel data of a grey-scale JPEG image. (In the case of a color JPEG image, a 1D unsigned char* array of rgbrgbrgb... values is read in.) This means that a conversion must be carried out to translate from a 1D unsigned char* array to a 2D float** array, required by our denoising functions. We remark also that a pixel in a JPEG image can only have value between 0 and 255. Moreover, the integer variable num components will contain value 1 after the import JPEG file function finishes reading a grey-scale JPEG image. The value 1 should also be given to num components when invoking export JPEG file to export a grey-scale JPEG image. We remark that the last argument quality of the export JPEG file function is an integer indicating the compression level of the resulting JPEG image. A value of 75 is the typical choice of quality. Once the external library libsimplejpeg.a is ready compiled, it can be used in the C++ main program as follows: #include <Image.h> #include <IsotropicDiffusionDenoising.h> extern "C" { void import_JPEG_file(const char *filename, unsigned char** image_c, int* m, int* n, int *k); void export_JPEG_file(const char * filename, unsigned char* image_c, int m, int n, int k, int quality); } int main (int nargs, const chars* args) { // repeat the tasks in the C main program int m,n,k,i,j; unsigned char* image_chars; // 1 byte per pixel in a grey JPEG image // read from file a JPEG formatted image import_JPEG_file ("noisy-paprika.jpg", &image_chars, &m, &n, &k); // // // // create u and u_bar objects with dimension [m,n] convert data inside ’image_chars’ into the ’image_data’ array of u create an object of IsotropicDiffusionDenoising and call ’denoising’ convert the ’image_data’ array of u_bar back to ’image_chars’ // write the denoised image to a new JPEG formatted image file export_JPEG_file ("denoised-paprika.jpg", image_chars, m, n, k, 75); } 4.5 return 0; Submission Each student should independently work with the project assignment. Before the deadline, Friday May 25, 2007 (12:00 o’clock), each student should submit a gzipped tar-file (using commands tar and gzip) with the following content: README src/C src/C++ src/external_code 7 test/C main.c make.sh makefile test/C++ main.cpp make.sh makefile clean.sh The README file should contain a simple overview of all the files submitted. The source code of the C and C++ user-defined libraries goes into the src/C and src/C++ directories, respectively. The src/external code directory contains the external C library package for handling JPEG formatted image files. The two test directories test/C and test/C++ contain the main programs and have either a make.sh or a makefile. The latter is a true makefile to be used with the make command, while the former is a simple shell script that builds the user-defined library, and compiles and links the main program. The clean.sh file is a shell script for traversing the entire directory tree and removing all binary files (which can easily be regenerated). The relevant Unix command for clean.sh is find . \( -name ’*.o’ -o -name ’*.a’ -o -name ’*.so’ \) \ -print -exec rm -rf {} \; Remember to run clean.sh before you pack the tar file and submit it to the teacher! (Otherwise big object and library files make the tar file much larger than necessary.) 8