To incorporate the anatomical detail needed for better diagnosis or guidance in radiotherapy, image fusion of whole body functional F-18 FDG PET or SPECT with anatomical CT, or MRI can be performed. Linear and non-linear image registration techniques can be applied. Linear techniques are suitable if the patient body configuration and breathing conditions are similar between the separate scans. If PET images are acquired with different breathing patterns and in a different body configuration than CT, or MRI, then rigid-body image registration may not be sufficient. We have developed both linear and non-linear techniques for PET/CT/MRI image registration based on mutual information techniques. We have designed a practical, fast and fully automated algorithm for elastic 3D image registration of whole-body PET and CT thoracic images, which compensates for the non-linear deformation due to the breathhold imaging technique and variable body configuration. We employ 3D computer graphics techniques for fast deformation of image volumes. Similar registration approaches can be used for the follow-up assessment and comparison of serial scans during the course of therapy. Image registration and fusion enhances the use of sensitive functional modalities such as PET as a guide for the radiation therapy planning.