Abstract.
In this paper, we present our experience in building a mobile imaging system that incorporates multi-modal sensors for road surface mapping and inspection applications. Our proposed system leverages 3D laser-range sensors, video cameras, global positioning systems (GPS) and inertial measurement units (IMU) towards the generation of photo-realistic, geometrically accurate, geo-referenced 3D models of road surfaces. The models that our mobile system delivers serve as input to machine vision algorithms for surface defect and crack detection. Based on our summary of the state-of-the-art systems, we address the need and hence several challenges in the real-time deployment, integration and visualization of data from multiple sensors. We demonstrate the final integrated
3D data providing accurate depth information for characterization of cracks as a significant improvement over contemporary video-based vision systems.
Traditionally, road surface evaluation and maintenance is based on the inspection staff’s judgment and experience [1]. The inspection personnel walks or drives slowly through asphalt and concrete pavements observing surface defects and degradation to make recommendations for immediate and long term maintenance. The manual inspection procedure is not only cumbersome, time consuming and expensive but is also susceptible to human error and inefficiency. With safety of the personnel and passengers in mind, this functional and important process of inspection can be significantly improved using a formalized imaging system that will ease the effort required to inventory road surfaces like highways and runways through periodic evaluations and subsequent surface distress management.
The key to successful road surface evaluation lies in identifying different types of distress and linking them to the cause. Recognizing the defects and also understanding their cause based on their appearance helps rate pavement conditions and select cost effective repair measures. As a first step towards automation, high speed digital imaging sensors deployed on mobile vehicles [2] have been successfully demonstrated in combination with image processing algorithms [3] for crack detection. Such video based vision systems have two major drawbacks. They do not provide sufficient depth information and also have ambient illumination requirements. The depth information is of particular significance in surface evaluation because the rating scheme for the road surfaces [1] is not just dependent on the length and width of the cracks alone as is the case with pavement distress applications but also on the depth. Crack depths in the order of a few millimeters require high precision distance measurements. Hence, the design requirements for a comprehensive visual data collection system should address

accuracy and precision in three dimensions of measurement, speed of acquisition, time required for post processing, ease of visualization and evaluation. To that end, we present a prototype mobile 3D data acquisition system in this paper. Our system integrates visual range and color data with position and orientation information through hardware measurements and provides better accuracy for fast digitization of large scale road surfaces at almost equal acquisition and processing time. We are able to generate accurate geo-referenced 3D models that are compensated for sensor motion caused by the changing physical environment. The multi-sensor integrated 3D models improve automatic crack detection and classification. Furthermore, accurate archives of such 3D models of road surfaces over time can be used for statistical wear and tear analysis.
We have organized this paper to address the design challenges in the construction of the multi-modal integrated imaging system that is capable of real-time data collection without restriction on the planarity of road surfaces. We have begun by listing the specific requirements as a problem statement. In Section 2, we summarize contemporary commercial systems targeting road surface inspection. The literature survey emphasizes on the design methods implemented thus far and also serves as a reference study to understand the difficulty in building systems for real-time deployment. We introduce our prototype system and explain the idea behind using multi-modal sensors in Section 3. After the data acquisition, we deal with the integration of multi-modal data in Section 4. The integration involves the representation of range and visual data into a spatially meaningful form using the information from position and motion sensors. We show the 3D models generated using our system driving a van along a test area containing different types of cracks in Section 5 and conclude with recommendations for possible improvements and reproducibility of our system in Section 6.
Related work towards road surface distress survey, especially on airport runways and army maintained highways dates back to early 1980’s. The pavement management system (PMS) idea was proposed by the U.S Army [4] and has since then undergone metamorphosis keeping pace with improving imaging technology. However, transportation departments met with limited real-time success using digital imaging techniques towards automatic crack detection and filling [5], until the late nineties. Non-visual sensors and several improvements on image-based methods were proposed during this period. We summarize these methods in Figure 1 and discuss the advantages and disadvantages of the different types of sensing methodologies. Analog films have been completely replaced by digital cameras. Among digital systems, video cameras are preferred to line scan methods for the ease of use without special illumination requirements, though line scan methods offer very high resolution data. Range sensors that directly give depth measurements have limited field of view while profilometers and acoustic sensors though inexpensive can only provide low resolution and a low dynamic range.

Vision-based methods
Analog films
3D range sensing
Digital imaging
Road surface inspection technologies
Non-visual methods
Acoustic sensing
Profilometers
Line scan Video
Fig. 1.
Summary of technologies demonstrated for road surface inspection.
In 1987, Mendelsohn [6] listed several of these methods including acoustic sensors and profilometers and suggested that the imaging modality was a promising approach.
At that time, the processing and image acquisition speeds challenged the feasibility of a fast and efficient inspection system. Several surveys were conducted to make an assessment of the feasibility of incorporating image acquisition and processing methods for both development and implementation of automated road surface inspection
[7, 8]. The conclusions of the survey encouraged by improving hardware and processing equipment have led to most of the commercial video-based systems available today that basically consist of an array of high speed imaging sensors supported with illumination equipment. The video data from such systems though promises to be sufficient for distress detection [9], requires spatial information for crack filling after detection and maintenance. A potential solution AMPIS [10] was proposed that combined GPS information with video to create GIS-like databases of road surfaces.
AMPIS claims improved road network identification, pavement inspection for better maintenance and data management over the base framework of PMS.
Taking a robotic approach, Hass et al. [11] proposed a system to overcome the shortcomings of the video-based system to make depth measurements by incorporating a laser range sensor. Hass et al. concluded that combining laser range data and video image data can provide overall better accuracy and speed of crack detection although due to the time consuming aspect of laser range sensing in 1992, they demonstrated range imaging for crack verification after the detection using the video based system. Several 3D approaches have been demonstrated since then. Laurent et al. propose a synchronized laser scanning mechanism to capture high precision 3D range and texture profiles. Bursanescu and Blais [12] reiterate a 3D optical sensor as the answer to high resolution and high accuracy acquisition and redesign a Biris sensor to meet the specific requirements of the pavement inspection application. They demonstrate six such sensors mounted on a mobile platform acquiring data at normal highway speeds.
In summary of the related work, we tabulate different other systems in the literature that have combined different methodologies in Table 1. We list the systems, their accuracies, processing speeds and the modalities included to better understand how

each one of these modalities contributes towards better and efficient pavement inspection.
Table 1.
Summary of the state-of-the-art systems.
System/
Group
Komatsu
WiseCrax
GPSVan
National
Optics
NRC
(Canada)
AMPIS
Modalities used
Video, Line scan
Dual video cameras
Stereo and analog camera, GPS , INS
3D range
3D range
Video, GPS
Resolution of imagery
4 mega pixel image.
Detects 3mm wide cracks
Built for large scale imaging
< 0.5 mm in width and depth
3mm wide,4mm deep cracks
0.3 mega pixels
Special notes
Collects data in the night with argon lights at 10 km/hr.
Can collect data at 80 km/hr.
Acquires geo-spatial data for urban planning.
Novel synchronized laser scanning approach proposed.
An array of Biris 3D sensor used.
Limited field of view.
RoadCrack Array of CCD
1mm crack width.
Can collect 400 km of data in one day at highway speeds.
The main goal of pavement inspection is being able to identify crack patterns, rut depths and the roughness of the cracks. Current data collection methods still necessitate integration of several heterogeneous technologies.
We further identify the scope for improvements in system design targeting the time of acquisition and processing and list the important characteristics of a real-time deployable system.
An ideal road data collection system must operate in real time gathering and post processing speeds.
The duration required for data analysis should not overwhelm the time required for acquisition. A single pass data collection should be sufficient for cost-effective distress identification and localization, the critical aspect being the accuracy and robustness of the system and its extendibility to arbitrary terrain. With all these system requirements in mind we now present our prototype system in the following section.
We are looking to digitize road surfaces as accurately and quickly as possible with available technology and processing equipment. Our focus is more on the image acquisition system than image processing as we describe the data collection system in this section. We begin with the list of hardware in Table 2 that we have used and share our design ideas. We show the schematic diagram of our system in Figure 2.
Table 2.
Design specifications of components in our prototype.
Component System Design Specification
3D sensor (IVP SC- Depth accuracy ~ 1 mm configured for a baseline of 70cm and

386)
Video camera (Sony) Placed on a rod for increased field-of-view orthogonal to the road surface and also pre-registered with the range profile.
GPS (Leica) stand off 70cm and a triangulation angle of 45 degrees gives
1m width of view at 2000 profiles/sec.
IMU (Xsens)
Base station placed in an area with no obstructions with one
GPS radio receiver and antenna on the mobile platform (in line with the 3D sensor) to communicate with the base station.
Is placed as close as possible to the 3D range sensor camera and measures orientation angles accurate up to a tenth of a degree .
Fig. 2.
Schematic of our data collection system that also shows the different reference frames of each sensor.
Our primary goal being able to measure depth of cracks along the road, a 3D range sensor on a mobile platform directly solves this problem. We tried to use three different 3D acquisition methods (triangulation-based, time-of-flight and structured lighting) for the data collection. We concluded that the triangulation based system matched our requirements for high speed and high accuracy. The sheet-of-light triangulationbased sensor that we used (IVP RANGER SC386) is capable of acquiring 2000 profiles/second that corresponds to 6 mm separation between successive profiles driving at 30 miles/hour. In terms of accuracy, our system that was placed 70 cm above the road surface and configured for a baseline of 70 cm and a triangulation angle of 45 degrees gives 1mm accurate depth information. The price that we however pay in using such a system is the field-of-view. We are able to scan 0.6 m wide section of the road using a single sensor. We recommend using an array of such sensors as a possible solution to large scale data collection.
Though the 3D information alone is sufficient for crack detection and surface rating, we need spatial information for crack localization and repair. We collect physical

location information by setting up a GPS base station and placing a receiver on the mobile platform. The GPS data is accurate up to 4 cm in the motion direction and gives us 10 samples of position information in one second. The GPS can be thought of as sampling the 3D motion of the mobile platform that houses the sensors. In the schematic shown in Figure 2, we have shown a video camera mounted on a rod, whose image axis is orthogonal to the road surface. We prefer the orthogonal field-of-view because it makes the registration of range and intensity profiles trivial and considerably improves integration time without having to consider CCD calibration and rectification of images. The video data in addition to providing visual cues for crack detection also helps in estimating the motion. The pose from video algorithm [13] is particularly useful when the GPS satellite signals are intermittently not available from the satellites during certain time intervals of the day.
The system components that we have described thus far are sufficient for scanning planar road surfaces. For roads with sufficient distress, varying terrains and embanked pavements, the effect of driving on such terrain and roads with bumps needs to be compensated. The oscillations on the mobile platform caused by the suspension system also have to be considered. We have hence used the IMU for measuring the orientation Euler angles (roll, pitch and yaw) of the sensor mount during data collection.
We have used a high performance computer with a Pentium 4 processor that supports hyper threading with 1GB of RAM and with special high speed serial interface cards as the processing equipment. Our acquisition programs are multi-threaded, multidocument graphical user interfaces written in C++ capable of real-time buffer memory management and storage.
Now that we have collected data from multiple sensors, we describe the procedure for integrating the data into one complete single visualizable multi-modal dataset. This task of spatial alignment is not trivial because, each of the measurement systems has their own reference co-ordinate system differently oriented in free space. We illustrated this issue in Figure 2. Hence as a first step towards integration and fusion of the data, we need to choose a global reference co-ordinate system to represent our data.
We have used the GPS co-ordinate frame as our reference frame and transformed the range and intensity profiles to that frame without losing geographic location information of the road surface.
We need to deal with another important issue before transforming the data to the real-world co-ordinates. We attribute this issue to different acquisition rates from different sensors. The GPS supplies data at a frequency of 10 Hz, video camera at 30
Hz, the IMU at 100 Hz, while the range profiles are acquired at nearly 2000 Hz. We have two choices in resolving this issue. Either to discard the range data and use the profiles that are time synchronized with the GPS data or use all the points of the range data and align the profiles based on interpolated GPS path for various time intervals that we have the range data for. We observed that we lose more information in discarding acquired data by choosing the former solution. We hence suggest cubic spline

interpolation of the GPS path as a 3D curve at time stamps recorded by the range sensor. The IMU orientation data also needs to be interpolated. But before that, having characterized our IMU sensor, we applied simple smoothing techniques to reduce the noise in its measurements before interpolation.
Towards the spatial alignment of data, if we can denote the Euler angles of roll, pitch and yaw from the IMU by (

,

,

)and 3D range measurements by x t r
, y t r,
z t r, at a particular time t (note that we have already interpolated all the sensor data to synchronize in time) and let the GPS measurements after considering the distance along each dimension of the range sensor from the GPS receiver be
( x g
, y g
, z g
)
, the mapping to the real-world co-ordinate system W t neous time t can be made using Equation 1. of the profile acquired for that instanta-
1
R t
D t






 cos
 cos sin
 cos sin



[ x t r
, y t r
, z t r
]
T
 sin sin
 sin
 sin
R t

D t cos



P t cos

W t
 sin
, where
  cos

 sin
 sin

  cos cos

 cos
 cos
 sin sin

 sin cos cos




 sin cos
 sin

 sin cos





 is the measurement from the 3D range sensor.
P t

[ x t g
, y t g
, z t g
]
T is the position of the range sensor through GPS measurements.
The transformation and alignment based on multi-sensor data (collected over a time period) gives us an unorganized point cloud of data that, for visualization purposes, we triangulate using the method described by Hoppe in [14]. We represent the triangulated mesh as a VRML model that can be rendered in most computers that support
OpenGL or DirectX. The dense point cloud that we have converted into a mesh can then be textured using the color images from the video. By the design of our setup and initial hardware registration step, we can actually map the color pixels in the CCD to the range profile as a quick method for multi-modal visualization. We summarize the entire integration process as a simple block diagram in Figure 3.
Multi-sensor data
Noise reduction and sample interpolation
Spatial alignment
(Range, IMU and
GPS.)
Fig. 3.
Block diagram of the integration pipeline.
3D meshing and video texture mapping
3D
Visualization
In Figure 4a, we show our system mounted on a van along with various other components that we have used. We show an area of interest that we have scanned using our data collection system (Figure 4b) along with the GPS path on a satellite map (Original image from www.maps.google.com
) and the multi-modal integrated data in Figure
5. The discontinuity in the GPS path is because we could not get back precisely to the

starting point. To draw attention to the resolution at which we have imaged we show some zoomed in images of cracks and rough asphalt surfaces in the same figure. We have color-coded the depth to emphasize the cracks in blue color. The small cracks on the right are about 2cm wide and 2cm deep while the longitudinal crack in the top-left is 3cm wide and 4 cm deep. We have not shown the entire path (75m) at that high resolution considering the size of the data and memory resources required to render the model. However, to highlight the quality of the actual data we show 3D multisensor integrated models with the geographic location information as textured datasets in Figure 6 after registration of intensity and range profiles. Figure 6a shows texture details and underlying geometry of an asphalt surface. We underscore the need to integrate both visual and range data with the help of Figure 6b where the geometry leads to a different conclusion about the cracks while texture clearly indicates the presence of grass. The textured 3D data contains the visual information for both 2D and 3D crack detection and classification algorithms.
We have laid out the specifications required for a multi-modal integrated 3D imaging system for scanning road surfaces as a significant improvement over the state-of-theart. With the 3D models that we have generated crack detection has become easy with simple threshold-based algorithms giving us fast and accurate results. We have overcome the illumination requirements of the contemporary systems and are able to scan driving at 30 miles/hour at 1 mm depth accuracy on the cracks and 6mm inter-profile distance. With data samples from four sensors supplying data at different rates, we have integrated photo-realistic and accurate 3D models for surface condition archival and convenient visualization.
The GPS signal availability is a major issue and though we have attempted a solution by trying to estimate motion from video, such algorithms are based on feature matching. We had difficulties in identifying features on smooth planar road surfaces.
Through the system design, we have reduced the computations required for the final integration. The post-processing time visualizing the data was almost equal to the time duration of scanning. Though it is a considerable improvement over the available 3D systems, we attribute it to large amounts of data that need to be processed simultaneously. For a 1m wide dataset that is 45m long driving at 30mph, we are thinking about a few million points at least 60 megabytes of data in its compressed form. Hence, our future efforts will target real-time rendering, visualization and archival of such large datasets.

(a) (b)
Fig. 4.
Real-time data collection. (a) Our data collection system on the van. (b) GPS path (path length of 75 m) marked on a satellite image.
Fig. 5.
Multi-modal integrated 3D data of an area of interest with three small zoomed in sections of areas with different roughness and depth of cracks. The zoomed in sectional views show the color and the color-coded range data side-by-side.
(a) (b)
Fig. 6.
Multi-sensor integrated textured 3D data. (a) A flat terrain asphalt road surface 1m in width and 5m of displacement in motion. (b) Road surface with patches of green grass.
1. Walker, D.: Asphalt Airfield Pavements, Pavement Surface Evaluation and Rating. University of Wisconsin, Madison, Transportation Information Center, (2004).

2. Cheng, H.D., Miyojim, M.: Novel System for Automatic Pavement Distress Detection System. J. Computing in Civil Engg. Vol. 12. (1998) 145-152.
3. Li, L., Chan, P., Rao, A., Lytton, R.L.: Flexible pavement distress evaluation using image analysis. In Proc. 2nd Int. Conf. on Applications of Advanced Technologies in Transportation Engineering, ASCE, New York, (1991) 473–477.
4. TM 5-623, Department of Army, Technical Maintenance Management, Technical Manual,
(1982).
5. McGhee, K.H.: NCHRP Report 334: Automated Pavement Distress Collection Techniques.
Transportation Research Board, National Research Council. Washington D.C. (2004).
6. Mendelsohn, D. H.: Automated pavement crack detection: An assessment of leading technologies. In Proc. 2nd North Am. Conf. on Managing Pavements. Vol. 3. (1987) 3.297-
2.314.
7. Howe, R., and Clemena, G. G.: An assessment of the feasibility of developing and implementing an automated pavement distress survey system incorporating digital image processing. Rep. No. VTRC 98-R1. Virginia Transportation Research Council (1998).
8. Wang, K.C.P.: Design and Implementation of Automated Systems for Pavement Surface
Distress Survey. ASCE Journal of Infrastructure Systems. Vol. 6. 1 (2000) 24-32.
9. Meignen, D., Bernadet, M., Briand, H.: One Application of Neural Networks for Defects
Using Video Data Bases: Identification of Road Distresses. DEXA Workshop (1997) 459-
464.
10. Chung, H. C., Girardello, R., Soeller, T., Shinozuka, M.: Automated management for pavement inspection system, Proc. of the SPIE Smart Structures and Materials Symposium.
Vol. 5057 (2003) 634-644.
11. Haas, C.: Investigation of a Pavement Crack-Filling Robot, Report to the Strategic Highway Research Program, Pittsburgh, PA, (1992).
12. Bursanescu, L., Blais, F.: Automated Pavement Distress Data Collection and Analysis: A 3-
D Approach. Proc. of Intl. Conf. on Recent Advances in 3-D Digital Imaging and Modeling
(1997) 311-317.
13. Grinstead, B., Koschan, A., Abidi, M. A.: A Comparison of Pose Estimation Techniques:
Hardware vs. Video. In Proc. of SPIE Unmanned Vehicle Technology VII. Vol. 5804.
(2005) 166-173.
14. Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., and Stuetzle, W.: Surface Reconstruction from Unorganized Points. Computer Graphics. Vol. 26. 2 (1994) 71-78.