MapServer and SDE
Howard Butler
Center for Survey Statistics & Methodology
Iowa State University
Beginning
Middle
End
National Resources Inventory
• National longitudinal survey of multiple
natural resources
• Estimates of status and change over time
on nonfederal lands
–
–
–
–
Beginning
Middle
End
Land cover/use changes
Soil erosion trends
Urbanization and loss of prime farmland
Wetland dynamics
• http://www.nrcs.usda.gov/technical/NRI/
National Resources Inventory
• > 300,000 area segments
– 160 acres each
– 70,000 samples / year
– Low altitude photography
Beginning
Middle
End
National Resources Inventory
• 1/3 digital -- 2/3 analog
• > 18 gb of support data
– Federal land
– Water
– Sample location layers
– Support imagery
Beginning
Middle
End
NRI GIS Data History
• Recently migrated to ESRI SDE
• Oracle database for tabular data
• Changes/edits tracked with ArcGIS
editing tools
Beginning
Middle
End
Use of MapServer in NRI
• The need for speed
– ArcIMS doesn’t even compare on the
same hardware
– We can get by with cheaper hardware
because of improved performance
– Lower software licensing costs
Beginning
Middle
End
Use of MapServer in NRI
• Still need SDE features, though
– Versioned queries
• Edits to data are tracked through a
versioned editing workflow. Web mapping
applications must be able to see these
various versions
– Joins
Beginning
Middle
End
• Spatial and tabular joins
NRI Versioned Data Tree
1997
2000
2001
2002 version of the 2000
control data
2002
2003
2004
N
Survey year
Survey edits
Revision edits
Frozen edit state
Use of MapServer in NRI
• MapServer brings features, too
– Fluent WMS allows us to leverage other data
sources that we don’t have to manage
– Bulletproof projection on-the-fly support
– MapScript is simpler, more robust, more
flexible (supports many languages), and faster
than many of the ArcIMS connector
technologies such as ActiveX and JSP
Beginning
Middle
End
Recent improvements in MapServer
SDE Support
• Support for Versioned Queries
– Can specify the version on the CONNECTION
string to query from
• Auto-sensing of SDE table structure
– SDE maintains a unique ID for each table. The
user can change the name of this column at
will, and many applications within the ESRI
family are not consistent.
Beginning
Middle
End
Recent improvements in MapServer
SDE Support
• New internal query syntax
– ArcSDE 8.x changed the query syntax in its C
API. The new methods allow the user to prejoin feature and business tables, making the
queries much faster.
– This change also will allow us to support userspecified joins similar to XBASE joins in the
future.
Beginning
Middle
End
Compiling in SDE Support
• Linux
– Easy. Just turn on the configure line
parameters related to SDE (--with-sde, -with-sde-version)
• Solaris
Beginning
Middle
End
– Same as Linux, except that make sure
everything is compiled with the same
compiler
Compiling in SDE Support
• Windows
– Harder. Configure all of your settings in
nmake.opt
– Ensure that the .def table for all of the C
API functions is available to map the
function names to the Windows calling
convention
Beginning
Middle
End
How to Connect to SDE
• Define a Layer in your MapFile
LAYER
NAME
states
TYPE
POLYGON
CONNECTION "sdemachine.iastate.edu,port:5151,sde,username,password,SDE.DEFAULT"
CONNECTIONTYPE SDE
DATA "HOBU.STATES_LAYER,SHAPE"
TEMPLATE '/where/the/template/file/is/located'
CLASS
STYLE
SYMBOL 'circle'
SIZE 3
COLOR -1 -1 -1
OUTLINECOLOR 0 0 0
END
END
END
Beginning
Middle
End
More detail at:
http://mapserver.gis.umn.edu/cgi-bin/wiki.pl?MapServerSDE
How to Connect to SDE
CONNECTIONTYPE SDE
• MapServer will report an error of SDE
Support not available if SDE has not
been properly compiled-in
Beginning
Middle
End
How to Connect to SDE
CONNECTION
"sdemachine.iastate.edu,
port:5151,
sde,
username,
password,
SDE.DEFAULT"
Beginning
Middle
End
-Machine Name
-Can be an IP Address
-Often needs to be the
FQDN
How to Connect to SDE
CONNECTION
"sdemachine.iastate.edu,
port:5151,
sde,
username,
password,
SDE.DEFAULT"
Beginning
Middle
End
-SDE Service location
-port: is important
-esri_sde for systems
specified in
/etc/services
How to Connect to SDE
CONNECTION
"sdemachine.iastate.edu,
port:5151,
sde,
username,
password,
SDE.DEFAULT"
Beginning
Middle
End
-The database name
in which SDE is stored
-This is often “SDE”
-Not required by all
DBMS, but some text
is required by
MapServer
How to Connect to SDE
CONNECTION
"sdemachine.iastate.edu,
port:5151,
sde,
username,
password,
SDE.DEFAULT"
Beginning
Middle
End
-User must be granted
rights to select the
table in the DATA
directive using either
ArcCatalog or the SDE
command-line utilities
How to Connect to SDE
CONNECTION
"sdemachine.iastate.edu,
port:5151,
sde,
username,
password,
SDE.DEFAULT"
Beginning
Middle
End
-The user’s password
-CASE SENSITIVE
How to Connect to SDE
CONNECTION
"sdemachine.iastate.edu,
port:5151,
sde,
username,
password,
SDE.DEFAULT"
-The version to
connect to
-SDE.DEFUALT is
used if none is
specified
Beginning
Middle
End
-CASE SENSITIVE
How to Connect to SDE
DATA "HOBU.STATES_LAYER,SHAPE"
Schema
Table Name
Spatial
Column
• No spaces!
• Oracle and MSSQL usually require the
schema name – use the full name shown
Beginning
Middle
End
in ArcCatalog if not sure
How to Connect to SDE
TEMPLATE '/where/the/template/file/is/located'
• Must be specified to query attributes
• Unnecessary performance hog if you are
only drawing layers – especially if the layer
has a lot of attributes
Beginning
Middle
End
An Example Application
• NRI DataView
• ID-Based
• Python MapScript
• TerraServer WMS
• pyTerra for image dates
Beginning
Middle
End
• PIL for post-processing image
manipulation
An Example Application
• 300,000 PSUs
– QueryByAttribute would require a sequential scan of the
entire table
– The ID has natural hierarchy
• 19001_010101R
– First two digits are the state FIPS code
– Second three digits are the county FIPS code
• Use QueryByAttribute on the USSTATES table to limit the
extent of the query of the USCOUNTIES table
• Use QueryByAttribute on the USCOUNTIES table to limit
the extent of the query on the PSU table
Beginning
Middle
End
An Example Application
Beginning
Middle
End
An Example Application
Beginning
Middle
End
Upcoming MapServer SDE
Improvements
• QueryByShape
– Currently only extents are supported
– SDE has support for advanced
operations
• Buffer select, touching, common edge
Beginning
Middle
End
Upcoming MapServer SDE
Improvements
• Joins
– Will be implemented similar to XBASE
and MySQL
– As fast or as slow as your database
– We get around this limitation with
database views right now
Beginning
Middle
End
Upcoming MapServer SDE
Improvements
• SDE Raster Support
– Been talked about and requested for a
long time
– No one has the time or desperate need
for it to fund development
– Could be implemented through GDAL
Beginning
Middle
End
The End
• Questions?
Beginning
Middle
End