GIS Data: Processing
ArcToolbox, Geoprocessing and
ModelBuilder
Spring 2008
1
GISC 6382 Applied GIS UT-Dallas Ron Briggs
ArcToolbox, Geoprocessing, ModelBuilder : Overview
ArcToolbox provides access to geoprocessing
• ArcToolbox is a dockable window in ArcCatalog
or ArcMap
• Comprises a set of Tools (750+)
• Tools support all data types (coverages,
shapefiles, gdb, raster)
• Multi- step workflow available via Modelbuilder
and/or scripting
• Command line interface also available for
traditionalists and/or power users
2
GISC 6382 Applied GIS UT-Dallas Ron Briggs
What is geoprocessing?
• the modification and analysis of spatial (geographic) data.
CAD
Projections
GDB
Conversion
Database
Spatial
analysis
Data
management
3
GISC 6382 Applied GIS UT-Dallas Ron Briggs
What is Geoprocessing used for?
–Data processing/ data automation: Creation,
Preparation, Conversion, Management
–Data analysis/spatial modeling: Suitability,
Sensitivity, Capacity, Risk
4
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Accessing geoprocessing tools
• Geoprocessing is carried out using tools stored in ArcToolbox
– A dockable window available in ArcMap, ArcCatalog, ArcGlobe, or ArcScene
• Tools run processes
Opens ArcToolbox
5
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Inside of ArcToolbox
• Toolbox: Container
for toolsets and tools
Toolbox
– Note: cannot have a
toolbox within a toolbox
• Toolset: Logical
container of tools
and other toolsets
(i.e. folder)
• Tool: Single
geoprocessing
operation (includes
dialogs, models,
and scripts)
Toolset
Tools
GISC 6382 Applied GIS UT-Dallas Ron Briggs
6
Supported data types
• The tools work with many data types
Coverage
Geodatabase
Tables
Shapefile
Layer file
CAD
Behavior
Raster
Geoprocessing
7
GISC 6382 Applied GIS UT-Dallas Ron Briggs
The tools work with many data types
…Caution
• Often there are different tools for different data types. Often
there is one:
– For shapefiles and geodatabse features classes
– Another for coverages
– Another for raster datasets
• And they have the same name!
– For example, there is a separate Clip tool for each of these input
data types
– Similarly, there is a separate Define tool for each of these input
data types
• Occasionally, there is a capability only available for
coverages
– e.g. conversion of SDTS (Spatial Data Transfer Format) or VPF (a
vector format used by military)
– Convert first to coverage, then convert coverage to shapefile
8
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Tools with the same name
• Some tools have the same name
– Must distinguish between these tools in scripts
– Be sure to get the correct tool for the data type you are processing
Clips a
coverage
Clips a
feature
class
Clips a
raster
9
GISC 6382 Applied GIS UT-Dallas Ron Briggs
A special note on the
Coverages Tools toolbox
• Only available if you have ArcInfo
Workstation installed
• The toolset titles (Analysis,
Conversion, Data Management)
duplicate many of the toolboxes in
ArcToolbox as a whole
– You need to use these tools to process
coverages
– In some cases, there are capabilities
here not available for other data types
• e.g. conversion of SDTS (Spatial Data
Transfer Format) or VPF (a vector format
used by military)
• Convert first to coverage, then convert
coverage to shapefile
10
GISC 6382 Applied GIS UT-Dallas Ron Briggs
System (Esri provided) toolboxes
• All tools are organized
into toolboxes
• System toolboxes
are provided by ESRI.
Number depends on
–License level (ArcVIEW,
ArcEditor, ArcInfo)
–Extensions licensed
• and you can create
your own toolboxes
(and tools)
GISC 6382 Applied GIS UT-Dallas Ron Briggs
11
Tool Counts—nearly 750 now (9.2)
• Counts vary depending on license/extensions
– ArcView: 102 tools
– ArcEditor: 104 tools
– ArcInfo: 251 tools
– Spatial Analyst extension: 158 tools
– 3D Analyst extension: 45 tools
– Geostatistical Analyst extension: 1 tool
• A total of 455 tools came with 9.0
– Feature class tools – 120
– Spatial Analyst tools – 185
– Geocoding tools – 5
– 3D Analyst tools– 20
– Linear Referencing–10
– Coverage tools – 57
Counts as of 9.0.
9.1 has an additional 171 tools
9.2 has additional 121 tools
GISC 6382 Applied GIS UT-Dallas Ron Briggs
12
The geoprocessing framework:
2
four ways to run tools
Command line
1
3
Models
Dialog
4
Scripts
4
GISC 6382 Applied GIS UT-Dallas Ron Briggs
ArcObjects
13
Selecting the way to run tools:
Choosing between the four ways
Selection depends on a user’s needs and abilities
1. As a dialog inside of any ArcGIS application
•
•
Simple, one at a time processing (GIS for dummies?)
Good for ad hoc processing
2. As a command from a command line
•
•
Very efficient for command line afficianados
Supports intelligent auto-completion of commands
3. As a process in a model (Modelbuilder)
•
•
An “executable flow chart” for GIS processing
Powerful multi-step processing, yet simple to create and use
4. As a function call in a script or program
•
•
Similar to AML in ArcInfo 7, but avoids learning proprietary
language
provides decision making and looping to support batch processing
In this course we will use #1 and #3
14
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Parameters
• all tools have one or more parameters
– pieces of information that the tool needs in order
to run
• some parameters are required,
– such as the name of the input data set
• some parameters are optional
• most parameters have a default set, which
– you should always review, and
– change if it’s inappropriate for your application
15
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Environment settings
• “Customizable defaults”
• Common parameters that are applied to all
tools within a geoprocessing session
• Settings work (“persist”) in all
environments (dialogs, command line,
models, scripts,)
General settings
Current workspace
Output coordinate system
Geodatabase settings
Raster settings
XY Domain
Cell size
M and Z Domains
Mask
Geodatabase raster
Coverage environment
Comparison between prj files
Statistics
Compression
Precision for new coverages
GISC 6382 Applied GIS UT-Dallas Ron Briggs
16
Four levels of environment settings
• User Profile
– The settings will persist throughout
all applications loaded by the user
• Application Level
– The settings will apply to all tools
executed from that application
• For a specific model or script
– Settings apply to all tools within this
specific model or script.
• For an individual tool
– Settings apply to a specific tool in a
model or script
Lower order (e.g. individual tool) settings
override higher order (e.g. user profile) settings
17
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Using Modelbuilder
ArcGIS 9.2 has some major
enhancements to Modelbuilder:
Whats_New_in_Arctoolbox.pdf
18
GISC 6382 Applied GIS UT-Dallas Ron Briggs
•
Why Use Modelbuilder?
document the processing steps (workflow) that
you carried out
–
–
•
•
•
•
You will forget
You may need to do it again
re-run the analysis to experiment with different
parameters to see how results are affected
rerun the analysis if you find a mistake
carry out repetitive, multi-step tasks
provide other users (e.g. operations staff) with
an exact workflow that they can replicate
19
GISC 6382 Applied GIS UT-Dallas Ron Briggs
The Elements of ModelBuilder
• Model diagram window
– Input variable
– Tool
– Derived data variable
• Toolbar
–
–
–
–
–
Add Data or Tools
Layout
Zoom and Pan
Add connection
Run
20
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Building a model: constructing
• Create a new model
• Drag tool into model
• Drag data into
model or onto a tool
• Link data and tool
21
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Building a model: running and documenting
• Save and rename
the model
• Run model
• Document the
model
22
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Setting model parameters
Or double click on the tool’s process
box to open its parameter dialog
23
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Element States
• Not ready to run
(parameters not set)
• Ready to run
• Has been run
(note the grey
shadow)
24
GISC 6382 Applied GIS UT-Dallas Ron Briggs
An example,
which we will
now build
25
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Alternatives to Modelbuilder:
Options for Developing Additional Capabilities
• Modelbuilder
– Multi-step, sequential processing of ArcTools, using a visual
development environment which comes as part of ArcGIS
• Python, Jscript and VBScript
– standardized and relatively simple scripting languages for repetitive
processing, including loops and decision trees, using ArcTools
– Python scripts can be generated from Modelbuilder
• Visual Basic for Applications
– Permits writing of VB macros for sophisticated customization and
development within standard ArcGIS (ArcMap/ArcCatalog) interface
(and thus requires license for these)
– may incorporate ArcObjects, the COM compliant software objects
out of which ArcGIS is constructed.
• ArcGIS Engine
– Set of embeddable GIS components (ArcObjects software objects)
for use in building custom applications, independent of ArcGIS
interface
– Runs under Windows, Unix and Linux, with support for Java, C++,
26
COM and .NET
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Appendix
ESRI GIS Software
27
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Environmental Systems Research Institute, Inc.
(ESRI)
Redlands, CA
•
•
•
•
•
•
Privately held by Jack & Laura Dangermond
One of the 50 largest software companies in the world
Leader in GIS with at least 1/3rd of market
1 million users (2002) and 2,600 employees
Originator of commercial GIS in 1981 with release of ArcInfo
Released their first GUI (graphics user interface) product, ArcView, in
1991 using proprietary Avenue programming language (for MS
Windows, UNIX, Macintosh)
• These two product lines (ArcView 3 and ArcInfo 7) combined together
in ArcGIS v.8 released in 2000
– complete rewrite based on Microsoft COM/Active X software objects
• Only run on MS Windows
– completely different interface from earlier ArcView and ArcInfo
– old, Avenue based, ArcView still available as ArcView 3.3,
• But Old Avenue scripts will not run in ArcGIS 8
– old command line ArcInfo 7 still available as ArcGIS Workstation
• Old AML scripts will run in ArcGIS 9
• ArcGIS 9 released in 2004
• Current 2008 release is ArcGIS 9.2
GISC 6382 Applied GIS UT-Dallas Ron Briggs
28
ESRI Product Line-up: ArcGIS client products
ArcReader (“adobe acrobat” for maps) & ArcExplorer (spatial data viewer)
–
Free viewers for geographic data.
ArcGIS 9.x Desktop: two primary modules (MS NT/2000/XP only)
1.
ArcMap: for data display, map production, spatial analysis, data editing
2.
ArcCatalog: for data management and preview
ArcToolbox, for specialized data conversions and analyses, available as a window in both
Available capabilities within these modules are “tiered”
•
•
•
ArcView: viewing, map production, spatial analysis, basic editing
ArcEditor: ArcView, plus specialized editing
ArcInfo: ArcView & ArcEditor plus special analyses and conversions
Extensions: for special apps.: Spatial Analyst, 3D Analyst, Geostatistics, Business Analyst,
Network Analyst.
ArcObjects: build specialized capabilities within ArcMap or ArcCatalog using VB for Applications
ArcGIS Workstation (for UNIX and MS NT/2000/XP)
–
the old command line ArcInfo 7.1
ArcGIS Engine (MS NT/2000/XP)
–
–
Set of embeddable GIS components (ArcObjects software objects) for use in building custom
applications
Runs under Windows, Unix and Linux, with support for Java, C++, COM and .NET
–
Replaces MapObjects which were based upon a previous generation of GIS objects
Notes:
ArcGIS 8 released 2000 to integrate two previous standalone products: ArcView and ArcInfo
ArcGIS 9 released 2004 providing the full capability that should have been in ArcGIS 8!!!
--full support for all data types (coverages, shapefiles, geodatabases)
--full support for all previous geoprocessing analyses
--Modelbuilder for scripting and repetitive processing
--ArcEngine for building custom applications
ArcView 3.3 (the predecessor to ArcGIS 8.x) the only GUI option for UNIX.
ESRI Product Line-up: ArcGIS server products (Spring 2007)
ArcGIS Server
–
–
Permits the creation of server-based specialized GIS applications
Provides full range of GIS capabilities to a user without a desktop GIS
ArcIMS (Internet Map Server)
–
–
Software to develop Internet server-based mapping and basic analysis
Provides maps and simple query to the user without a desktop GIS
SDE (Spatial Database Engine)
–
–
middleware to support spatial data storage in standard DBMS
Supports all major industry databases:
•
Oracle, SQL-Server, IBM DB2, Ingres
Note: SDE and ArcIMS now distributed as part of ArcGIS Server package as of 9.2
ArcGIS Services
–
–
–
1/13/2006
Server based applications built and operated by ESRI or its partners
and made available on the Internet for subscription
Normally charged on a “per transaction” basis, but can be flat fee
presumably built using ArcGIS Server
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Other ESRI Products:
• ArcPAD
– Mapping on PDAs (“handhelds”) with Windows CE operating system
• ArcLogistics Route
– Specialized business application for delivery routing
• ArcFM
– water and telecom: industry specific facilities management
• ArcGIS Extensions
–
–
–
–
Spatial Analyst: raster data analysis
3D Analyst: 3-dimensional data display
Geostatistics: surface analysis
Business Analyst: marketing and site selection
– Survey Analyst: update of ArcInfo COGO (coordinate geometry) module
– Network Analyst: network routing algorithims; shortest path, etc.
– Maplex: automated, high quality labeling for maps (now part of ArcGIS 9)
– Publisher: creates .MXP maps for reading with ArcReader
Extensions work irrespective of ArcView/ArcEditor/ArcInfo tier
• BusinessMap:
– $99 standalone business mapping (originally Richardson-based MapLynx)
31
GISC 6382 Applied GIS UT-Dallas Ron Briggs
ArcGIS System
c:\ ArcGIS Workstation
Clients
ArcInfo
ArcEditor
ArcView
ArcMap
ArcCatalog
ArcToolbox
ArcMap
ArcCatalog
ArcToolbox
ArcMap
ArcCatalog
ArcToolbox
ArcEngine/
ArcObjects
Application
Development &
Customization
$
Consistent interface
Increasing capability
ArcExplorer
ArcServer Services
ArcIMS Services
ArcSDE Services
Internet
ArcPad
Files
Handheld/Wireless
Browser
(Personal Geodatabase,
Shapefiles, Coverages,
Grids, tins, etc)
Databases
Multi-user Geodatabases
(in Oracle, SQL Server,
IBM DBII, etc)
Source: ESRI with mods.
Some additional detail…
33
GISC 6382 Applied GIS UT-Dallas Ron Briggs
Discontinued Products
• ArcCAD
– CAD product from ESRI
• PC ArcInfo
–
–
–
–
1st effort at PC based GIS
DOS based, command-line driven
Data not compatible with ArcInfo UNIX
Replaced by ArcInfo 8 and ArcView 3.2
• DAK (Data Automation Kit)
– Subset of PC ArcInfo for data preparation for ArcView
3.2
• Atlas/GIS
– once a leader in PC-based mapping
– Bought by ESRI in 1996 & discontinued in 2001
GISC 6382 Applied GIS UT-Dallas Ron Briggs
34
ArcGIS Version 8/9
With Version 8 & 9, now have two flavors:
Desktop:
– Largest Microsoft COM/ActiveX application to date
– Full GUI interface
– Customization via Visual Basic for Applications
– New data base concepts: Geodatabase
– Runs on XP/2000/NT only
– no UNIX version available
Workstation:
– classic, command-line ArcInfo with AMLs (Arc Macro Language) for
customization
– same as version 7 and earlier, with minor enhancements
– the only option for UNIX, but also available on MS XP
– With release of ArcGIS 9, little reason to use unless under UNIX
• Now, all capabilities have been moved to ArcGIS
35
GISC 6382 Applied GIS UT-Dallas Ron Briggs
ArcGIS Desktop Primary Characteristics
• GUI-based tools
– ArcCatalog, ArcMap, ArcToolbox
• Geodatabase spatial data model supported in
standard database environments
– MS Access (.msb) for personal applications
– File-based geodatabase with 9.2
– Any industry db via SDE for multi-user applications
• Modeling of real world as intelligent objects
– Houses, poles, not points, lines, polygons
• COM/ActiveX components (ArcEngine) for
embedding geography in other applications
ArcInfo7: simple data
complex applications
ArcInfo8/9: intelligent data
simpler applications
36
GISC 6382 Applied GIS UT-Dallas Ron Briggs
ArcGIS 9 Desktop Modules
ArcCatalog (schema editor, with VISIO generation)
• The base application for ArcInfo Desktop
• Windows Explorer-like interface
• for organizing access to data and metadata
• For launching other Desktop apps: MAP and TOOLBOX
ArcMap (object editor)
• GUI for map creation and spatial data editing
• ArcPlot/ArcEdit (from ArcInfo v. 7) & ArcView 3.2
View/Layout combined
• Map projections on the fly (not via conversion as in AV)
ArcToolbox (geoprocessor)
• An interface to geoprocessing tools
• In ArcGIS 8 it was a separate module
– In ArcGIS Release 9 it’s an integrated window in ArcCatalog or
ArcMap
GISC 6382 Applied GIS UT-Dallas Ron Briggs
37
ArcGIS Desktop Capability Tiers:
Each tier has the same interfaces (ArcCatalog, ArcMap,
and ArcToolbox), but an increasing set of capabilities
are available within them (and $ price rises
accordingly!)
ArcView:
– viewing, map production, spatial analysis, basic editing
ArcEditor:
– ArcView, plus topologic editing of geodatabases and
editing of coverages
ArcInfo:
– ArcView and ArcEditor, plus more geoprocessing
analysis, conversions, and full support for coverages.
– Old, command line ArcInfo including AML support
38
GISC 6382 Applied GIS UT-Dallas Ron Briggs
ArcGIS 9.0 versus ArcGIS 8.3
• Main differences from 8.x are in ArcToolbox
– ArcToolbox built into ArcCatalog and ArcMap rather
than a separate module
– All ArcToolbox tools support all data types
(geodatabase, shapefiles, coverages)
• 8.3 primarily support coverages
– ModelBuilder diagrammatic modeling tool
• Invaluable for tracking and replicating geoprocessing steps
– New scripting capability for repetitive actions
• Python, JScript and VBScript--simpler to use than VB for
Applications, the only alternative in 8.x
• Old aml (arc macro language from ArcInfo 7) also supported
What ArcGIS 8 should have been when it was first released!
Incorporates just about everything from ArcInfo 7/ArcView 3.
GISC 6382 Applied GIS UT-Dallas Ron Briggs
39
ArcGIS 9.1 & 9.2 versus ArcGIS 9.0
For Spring 2007 we will use ArcGIS 9.2
Ormsby text contains copy of 9.0
Gorr and Kurland contains copy of 9.1
Main differences in 9.1 from 9.0
• General performance improvement and user enhancements (e.g. pause
drawing)
• Additional geoprocessing tools (63 in ArcView, 86 in ArcEditor, 22 in ArcInfo)
• Improved labeling of features on maps (Big change here was in 9.0)
• Support for linear referencing
• Network Analyst extension for transportation networks (shortest path, etc..)
Main differences in 9.2 from 9.1
• Geodatabase and map documents (.mxd files) incompatible!
• File based geodatabase
– Faster, larger capacity (x 1,000) than MS Access-based personal geodatabase
– Higher precision of coordinate storage
•
•
Identity/info. and measurement tools rewritten and greatly enhanced
Graphing re-written
– Supports brushing (linking) between graph & map (as in Anselin’s GeoDA) software)
•
Supports cartographic representation
– Can re-position features on map without changing coordinate values in geodatabase
Computing
Evolution
And it will all keep changing!
Pervasive
Computing
Internet
Desktop
Workstation
Mini
Source: ESRI, Inc.
Mainframe
• Small Hardware
(Nano)
• Wireless Internet
• Interoperable
• Embedded
41
GISC 6382 Applied GIS UT-Dallas Ron Briggs