Linear Referencing Workshop
by Ethan Welty, 08.14.2009
Software: ESRI ArcGIS 9.3, Microsoft Excel
(An ArcGIS Desktop license is sufficient for this protocol)
Introduction
Objective
A powerfully flexible and holistic method for mapping surveyed units to a stream network of
arbitrary complexity. While best suited for surveys complicated by poor or missing GPS data,
where mapped landmarks, knowledge of the landscape and surveyed length may prove more
influential in the final result, this protocol can also be followed with best results when depending
entirely on waypoint positions.
not covered
▪ building the hydrology "backbone"
▪ making attractive figures of the results
Linear Referencing
route ~ line along which events can be placed
events ~ entries of a table with measures that define a unique position along a route
measures ~ a system of measurement that defines the distance along a route (distance, time...)
Point event positions are uniquely defined by one measure; line events require two (from & to).
This is highly dependent on the current route geometry, which is why it is so important that we
start with the best possible hydro layer.
Workshop Files
river.shp ~ shapefile of a simplified watershed
possible origins:
▪ deduced from DEM (Digital Elevation Model)
▪ traced from aerial photographs
▪ clipped from the NHD (National Hydrography Dataset)
▪ pulled together from GPS tracks and waypoints
minimum requirements:
▪ reflects as accurately as possible the state of the river at the time the survey was collected
▪ erroneous gaps have been filled and stray vertices removed
data.xls ~ table of simple survey results
recommended:
▪ contains all unit-based data (ie. fish and habitat) merged together
▪ surveyed unit lengths
minimum requirements:
▪ units bounded by GPS waypoints or known landmarks
waypoints.shp ~ waypoints collected by GPS
minimum requirements:
▪ field by which to uniquely identify each point
A. Build Routes
Open ArcMap and add river.shp to a new map.
(Tools → Add Data)
Add Editor Toolbar to ArcMap.
(Tools → Customize → Toolbars → check 'Editor')
Start an editing session on river.shp.
(Editor → Start Editing)
Add an RID (Route ID) short integer field.
(Attribute Table → Options → Add Field)
Populate the RID field such that RID = 1 for mainstem and RID = 2+ for each subsequent
tributary or secondary channel. Complex selections can be made using the Select Features cursor,
and Select Features by Attribute and Select Features by Location dialogues – then right click on
the RID field in the river.shp Attribute Table, choose Field Calculator and type in the wanted
RID value, making sure to check “Calculate selected records only.”
Linear Referencing Tools → Create Routes
"Create routes from existing lines. The input line features that share a common identifier are
merged to create a single route."
Makes sure to specify the correct Coordinate Priority, the corner from which route measures will
increase from zero – this should be the downstream end of the stream network.
river_routes.shp ~ output route shapefile
properties:
▪ features are Polyline M, indicating that they are capable of storing route M-easures
▪ each branch of the network starts at M = 0 on the downstream end
▪ line segments have been merged by matching RID into continuous route features
Route Editing
In an editing session, double click a route with the Edit Tool cursor. Route vertices should appear
in green. Sketch Properties will bring up a table of X, Y and M values at each vertex.
flip segment ~ right click the route and select Flip
edit measures ~ right click the route: Route Measure Editing lists a number of useful tools
If a route geometry has been changed, use Set As Distance to update the route measures to reflect
the change. Apply Factor can be used to convert measures to a different distance unit, while
Offset is particularly useful when a start value other than M = 0 is desired.
B. Prepare Data
Open Microsoft Excel and add data.xls to a new map. Add an RID field and populate it according
to the RID values in river_routes.shp. Add FROM_M and TO_M fields, calculated from
LENGTH and the information provided in the NOTES. The survey units are now line events,
each with a location fully described by an RID and FROM_M & TO_M combination. Save your
changes.
ArcGIS Geodatabase
There is a danger of losing data when transferring between software systems and file formats.
Null cells become replaced with zeros if saved in a dBase (dbf) table, and the same is true when
saving to a shapefile (shp) within ArcGIS. Text tables (csv, txt...) are not true databases – lacking
an index field – and thus have limited functionality in ArcGIS. Excel spreadsheets (xls) are the
most reliable third-party format, and ArcGIS can access each worksheet as a separate table,
making them particularly convenient for storing multiple related tables. The native format in
ArcGIS is the geodatabase (mdb), a specialized Microsoft Access database capable of storing GIS
features. This is by far the most functional way of storing and manipulating data within ArcGIS –
null cells are supported, tables are indexed, and many advanced features, like Geometric
Networks, can only be accessed within a geodatabase.
Open ArcCatalog. Create a new personal geodatabase (right click destination folder → New →
Personal Geodatabase). Import data.xls$Sheet1 into the database as data_units (right click
database → Import → Table). In the Content tab, verify that all fields have been imported
properly.
Back in ArcMap, add the data_units as route events (Tools → Add Route Events), making sure to
specify the correct Route Reference (river_route.shp), Route Identifier Field (RID), Event Table
(data_units), and since these are Line Events, the From-Measure (FROM_M) and To-Measure
(TO_M). The results are added to the map as data_units Events.
Change the layer symbology so that individual units can be distinguished (double click
layer → Layer Properties → Symbology). I recommend using a Categories → Unique Values
scheme on the UID (Unit ID) field. Click Add All Values, then right click inside symbol table →
Properties for All Symbols → Width = 2. The default Color Ramp should be sufficient.
Label the units for faster identification (double click layer → Layer Properties → Labels). Check
“Label features in this layer.” Any field can be specified as the Label Field. To build more
complex labels, open the Label Expression dialogue (→ Expression...).
expression: [LENGTH] & "m - " & [UNIT_TYPE]
labels:
250m - RA
30m - PL
200m - GLP
C. Prepare Waypoints
Back in Microsoft Excel (data.xls), copy the fields RID, DSWP, USWP, FROM_M and TO_M to
a new worksheet (Sheet2). The objective is to build a point route events table of our waypoints
using the length-derived measures of our survey units. String together DSWP and USWP into a
single WP field, FROM_M and TO_M into a single MEAS field and string together two copies of
RID to match the length of the other columns.
Remove duplicates by using a data filter (Data → Filter → Advanced Filter → check Unique
Records Only). Make sure to specify a location for the results of the filter and then copy them
over the original table (otherwise, duplicate rows are hidden from view but not actually removed).
Waypoints bounding multiple units on different routes (at stream junctions) will exist as multiple
entries, one for each RID – these are not duplicates, and must be preserved for later.
Add waypoints.shp to ArcMap (File → Add Data) and add x and y coordinate fields to the
shapefile (Data Management Tools → Features → Add XY Coordinates).
Add the waypoint event table (data.xls$Sheet2) to the map (File → Add Data). Since it is a table,
it will not be shown in the default Display tab, only in the Source tab. Join the waypoint shapefile
to the event table (right click table → Joins and Relates → Join) based on their shared waypoint
ID fields (WP ↔ Id).
Export the result as a dBase file (right click table → Data → Export Data), then add the dBase
table back into ArcMap (Tools → Add XY Data) using the POINT_X and POINT_Y fields added
earlier. Export the result as shapefile waypoints_final.shp (right click layer → Data → Export
Data). Choose to add the result to the current map and delete unnecessary and duplicate fields
resulting from the join (right click shapefile → Open Attribute Table → right click field header →
Delete Field).
waypoints_final.shp ~ full-featured waypoint file
necessary features:
▪ combines x & y coordinates, waypoint ID, route ID and route measures for all waypoints
▪ waypoints bounding units on different routes exist as multiple entries, one for each RID
D. Build Anchors
Export a copy (anchors.shp) for further editing (right click layer → Data → Export Data), and
have the new shapefile added to the map. Label the anchor points for faster identification (double
click layer → Layer Properties → Labels), using WP as the Label Field.
The Trick (part 1)
The anchor points we have created have one large discrepancy – they are carriers of route
measures (MEAS) derived from the surveyed length of the units, NOT the route measures that
would correspond to their actual, geographic location along the river routes.
Why? During the final calibration step, the anchor points will need to act as pivots between two
competing realities – one of survey units bounded by the absolute geographic position of the
waypoints, one of survey units strung out along the routes according to their surveyed length. The
former is on display with anchors.shp, the latter is on display with data_units Events. Clearly the
two do not agree perfectly, and the best-guess is likely a compromise between the two. The
anchors, moved, deleted or unchanged, will absolutely determine the final outcome: the data units
will be snapped to the anchors corresponding to their waypoint endpoints, wherever we may have
moved them, and where an anchor was deleted, the survey lengths will be used, scaled as
necessary to fit within the still-present anchor points.
Start an editing session on anchors.shp (Editor → Start Editing). To make it easier to select and
move the anchor points, uncheck all other layers in the Selection tab and set the snapping
environment to river routes (Editor → Snapping) – edge, end and/or vertex – depending on the
move required. Remember to save edits regularly.
NOTE: The anchor points do not have to be snapped to the river routes; it is their perpendicular
position along the river route that matters.
Moved
#1 ~ mouth of river
#5 ~ downstream RID 1&2 junction
#7 ~ Goblin Creek tributary junction
#6 ~ downstream edge of logjam (as determined from imaginary aerials)
#3 ~ powerline crossing (as determined from imaginary USGS DRGs)
#2 ~ 30m downstream of #3, in order to protect the small pool unit during the calibration process
Removed
#9 / 10 ~ very poor GPS signal
#4 ~ on a straight reach with no landmarks, it may be better to rely on surveyed length when the
GPS data is not fully reliable
Unchanged
#8 ~ provides a best guess that more-or-less agrees with the surveyed length, on a curved reach
that was likely difficult to accurately measure with a laser rangefinder
E. Calibrate Routes
Linear Referencing Tools → Calibrate Routes
“Recalculates route measures using points.”
Specify river_routes.shp as the Input Route Features, RID as the Route Identifier Field,
anchors.shp as the Input Point Features, RID as the Point Identifier Field, and MEAS as the
Measure Field. Specify an appropriate Search Radius large enough to include any anchor points
that were not snapped to the river routes. The output (river_routes_calibrate.shp) will look
identical to river_routes.shp – the geometry is unchanged – but the M values have changed.
The Trick (part 2)
Recall that the original routes (river_routes.shp) used map distance for measures (as they should).
However, like the ticks of a ruler, route measures are essentially arbitrary; we are free to change
them however suits us. In calibrating the route measures of river_routes.shp according to the
MEAS field carried by anchors.shp, we are enforcing the hypothetical route measures determined
from our surveyed length at the locations where we have placed the anchor points. With the ticks
of the ruler displaced in this way, the data units, whose positions are determined only by the
unchanged FROM_M and TO_M, will fall into place between the anchors.
Again, add the data_units to the map as route events (Tools → Add Route Events), this time with
river_routes_calibrate.shp as the Route Reference. Change the symbology as previously
described – the units should now fall exactly between the anchors. However, the route measures
must be updated as they no longer reflect reality.
Linear Referencing Tools → Transform Route Events
“Transforms the measures of events from one route reference to another and writes them to a new
event table.”
In other words, this tool will convert the route measures of our data units such that the geometry
will be unchanged from the current view when mapped onto the original river routes. The Input
Event Table is data_units, the Source Route Features are river_routes_calibrate.shp, and the
Target Route Features are river_routes.shp. Direct the tool to the route identifier and measure
fields for each table and specify the output name (data_units_final), location and field names.
Add data_units_final to the map as route events (Tools → Add Route Events), this time
with river_routes.shp as the Route Reference, changing the symbology as needed to
inspect the results.
As a final quality control step, it is recommended to compare the surveyed LENGTH field to the
mapped length of the resulting data_units_final Events. Add a MAP_LENGTH double field
(Attribute Table → Options → Add Field) and populate it with the shape lengths (right click field
header → Calculate Geometry → Length). For easier comparison, add an additional field and
populate it with the difference between the two lengths (right click field header → Field
Calculator) using the expression:
[MAP_LENGTH] – [LENGTH]
Large discrepancies should be revisited for possible oversight and error, but hopefully, the results
are reasonable. And it may be said that the digitized survey units reflect, as closely as possible,
their real-world counterparts! To the best of our knowledge.