KDD-98: A Comparison of Leading Data Mining Tools A Comparison of Leading Data Mining Tools John F. Elder IV & Dean W. Abbott Elder Research Fourth International Conference on Knowledge Discovery & Data Mining Friday, August 28, 1998 New York, New York KDD-98: A Comparison of Leading Data Mining Tools Copyright © 1998, John F. Elder IV and Dean W. Abbott All rights reserved Manufactured in the United States of America © 1998 Elder Research T8-2 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Contacting Elder Research http://www.datamininglab.com Dr. John F. Elder IV 1006 Wildmere Place Charlottesville, VA 22901 Dean W. Abbott 3443 Villanova Avenue San Diego, CA 92122-2310 elder@datamininglab.com 804-973-7673 Fax: 804-995-0064 dean@datamininglab.com 619-450-0313 © 1998 Elder Research T8-3 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Tutorial Goals • Compare and Summarize Data Mining Tools which: – – – – – – Offer multiple modeling and classification algorithms Support project stages surrounding model construction Stand alone Are general-purpose Cost a lot We could get our hands on • Include some (focused) Desktop Tools Other Reports: Two Crows, Aberdeen Group, Elder Research (forthcoming ), Data Mining Journal © 1998 Elder Research T8-4 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Topics • • • • • Products covered Review of algorithms Comparative tables of properties Screen shots exemplifying qualities Summary of distinctives © 1998 Elder Research T8-5 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Caveats • We don’t know every tool well (and are sure to have missed some!) – Level of exposure noted for each tool • Our background (biasing our perspective) – Very technical, “early adopters” – Emphasize solving real-world applications – More classification than estimation • Field of tools is quite dynamic – New versions appear regularly © 1998 Elder Research T8-6 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Data Mining Products PRW Model 1 © 1998 Elder Research T8-7 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Product Our Experience http://www.isl.co.uk/clem.html http://www.think.com/html/products/products.htm http://www.datamindcorp.com http://www.sas.com/software/components/miner.html http://www.urbanscience.com/main/gainpage.htm http://www.software.ibm.com/data/iminer/ http://www.sgi.com/Products/software/MineSet/ http://www.unica-usa.com/model1.htm http://www.abtech.com http://www.unica-usa.com/prodinfo.htm 4 3.0.1 2.1.1 Beta 4.0.3 2 2.5 3.1 1 2.1 Moderate Moderate High Moderate Low Low Low Moderate Moderate High http://www.salford-systems.com http://www.wardsystems.com/neuroshe.htm mailto://olpars@partech.com http://www.cognos.com/busintell/products/index.html http://www.rulequest.com/see5-info.html http://www.mathsoft.com/splus/ http://www.wizsoft.com/why.html 3.5 3 8.1 2 1.07 4 1.1 Moderate Moderate High Moderate Moderate High Moderate Company Integral Solutions, Ltd. Integral Solutions, Ltd. Clementine Thinking ThinkingMachines, Machines, Corp. Corp. Darwin DataMind DataMind DataCruncher SASInstitute Institute Enterprise Miner SAS UrbanScience Science Urban GainSmarts IBM Intelligent Miner IBM SiliconGraphics, Graphics, Inc. Silicon Inc. MineSet Group1 1/Unica Technologies Group Model 1 AbTechCorp. Corp. AbTech ModelQuest Unica Technologies, Inc. Unica PRW CART NeuroShell OLPARS Scenario See5 S-Plus WizWhy URL Version Tested Tools Evaluated Salford Systems Ward Systems Group, Inc. PAR Government Systems Cognos RuleQuest Research MathSoft WizSoft © 1998 Elder Research T8-8 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Categories for Comparisons • Platforms Supported • Algorithms Included – Decision Trees – Neural Networks – Other • • • • Data Input and Model Output Options Usability Ratings Visualization Capabilities Modeling Automation Methods © 1998 Elder Research T8-9 updated October 19, 1998 √+ CART Scenario NeuroShell OLPARS See5 S-Plus WizWhy √ √ √ √ √ √ √ © 1998 Elder Research √ √ √ √ √ √+ √ √ √ √ √ √ √ √ √ √ Database Connectivity √ NT Server / PC Client Unix Standalone Clementine Darwin DataCruncher Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest PRW Platforms Unix Server / PC Client PC Standalone (95/NT) KDD-98: A Comparison of Leading Data Mining Tools √ √ √ √ √ √ √ √ √ √ Key blank no capability √– some capability √ good capability √+ excellent capability √+ √ √ √+ √− T8-10 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Tool Groupings Desktop High End • PC (standalone) • One or Two Algorithms • Multiple Platforms, ClientServer • Flat Files or Direct Database Access • Multiple Algorithm Types • Data Fits into RAM • Large Databases • Flat Files © 1998 Elder Research T8-11 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools End User Perspectives Business Technical • Intuitive Interface • Algorithm Options – Clear steps in data mining process – Non-technical terminology – Familiar environment – Knobs to enhance model performance • Model Automation – Simplify model design cycle – Documentation of steps used in generating models (repeatability) • Descriptive Reporting – Domain terminology – Graphical representations © 1998 Elder Research T8-12 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools OLPARS: Interface © 1998 Elder Research T8-13 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools MineSet: Interface © 1998 Elder Research T8-14 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools PRW: Experiment Manager © 1998 Elder Research T8-15 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Intelligent Miner: “Explorer” Interface © 1998 Elder Research T8-16 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Enterprise Miner: Visual Interface © 1998 Elder Research T8-17 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Clementine: Visual Interface © 1998 Elder Research T8-18 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools DataCruncher: Process Flow Diagram © 1998 Elder Research T8-19 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Data Input & Model Output Clementine Darwin DataCruncher Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest PRW CART Scenario NeuroShell OLPARS See5 S-Plus WizWhy © 1998 Elder Research Automatic Header Save Data Format √ √ √− √ √ √ √ √ √ Native ODBC Database Drivers √ √ √ √ √ √ √ √ √ √ √ √ √ Summary Reports Output Source Code √ √ √ √ √ √− √ √ √ √ √− √ √ √ √ √ √ √ √ √ √ √ √ √− √ √ √ T8-20 √ √ √ updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools PRW: Row Splitting © 1998 Elder Research T8-21 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Model1: Variable Selection © 1998 Elder Research T8-22 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Model1: Data Definitions © 1998 Elder Research T8-23 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Scenario : Special Data Types © 1998 Elder Research T8-24 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Decision Trees a>4 n y b>3.5 n y 2 a> 1 n 1 © 1998 Elder Research b>2 n 2 y 3 y 0 T8-25 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Polynomial Networks Z17 = 3.1 + 0.4a - .15b2 + 0.9bc - 0.62abc + 0.5c3 Layer 0 (Normalizers) Layer 1 Layer 2 a N1 z1 z16 k N9 Unitizers z6 f Layer 3 Double 16 z9 Single 14 z14 N6 d N4 h N8 z17 z4 MultiLinear 15 z21 z15 Triple 21 U2 Y1 U7 Y2 Triple 17 z8 z19 Double 19 Double 20 z20 e N5 z5 © 1998 Elder Research T8-26 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools “Consensus” Models Parametrically Summarize Data Points Regression orders, terms Polynomial Networks (e.g. GMDH, ASPN) Decision Trees (e.g., CART, CHAID, C5) ∑ ∑ ∑ ∑ Logistic or Sigmoidal Networks (ANNs) Hinging Hyperplanes, MARS © 1998 Elder Research T8-27 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools “Consensus” Models (continued) orientation, bin width function Radial Basis Function family, order © 1998 Elder Research Histogram Wavelets T8-28 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools “Contributory” Models retain data points; each potentially affects estimate at new point shape, spread k, distance metric Goal, iterations Kernels k-Nearest Neighbor Delaunay Planes Spread, index Projection Pursuit Regression © 1998 Elder Research T8-29 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Properties of Algorithms Algorithm Accurate Scalable Interpretable Useable Robust Versatile Classical (LR, LDA) – C C – Neural Networks C D C– D D Visualization C Decision Trees D DD C C C C– – CC C – D Polynomial Networks C D C– –D K-Nearest Neighbors D – DD Kernels C DD C– D – –D –D D © 1998 Elder Research C T8-30 D C Fast Hot Key C D C good DD C – neutral DDD C– C– C– – D –D C – D D C D D bad updated October 19, 1998 CART Cognos NeuroShell OLPARS See5 SPlus WizWhy © 1998 Elder Research Kohonen Sequential Discovery Time Series Generalized Linear Models Polynomial Networks Rule Induction Bayes Radial Basis Functions √ Association Rules √ √ K Means √ Nearest Neighbor √ √ Multi-layer Perceptrons Clementine Darwin Datamind Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest PRW Linear/Statistical Algorithms Decision Trees KDD-98: A Comparison of Leading Data Mining Tools √ √ √ √ √ √+ √ √ √ √ √ √ √ √ √ √+ √ √− √ √+ √ √ √ √+ √ √ √ √− √ √ √ √ √ √ √ √ √ √ √ √ √ √ √− √ √ √ √ √ √ √ √+ √ √− √ √ √ √ √+ √ √ √ √ T8-31 updated October 19, 1998 √ √ © 1998 Elder Research √ √ √ √ Automatic Model Selection Other Cost functions Advanced Learning Alg. Normalize Inputs Cross-Validation √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ Multiple Stop Criteria Multiple Activation Functions √ Parameter Summary NeuroShell OLPARS √ Network Visual √ √ √ √ √ √ Momentum Clementine Darwin Enterprise Miner Intelligent Miner Model 1 PRW Learning Rate Decay Multi-Layer Perceptrons Learning Rate KDD-98: A Comparison of Leading Data Mining Tools √ √ √ √ √ T8-32 √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Enterprise Miner: Neural Network Training © 1998 Elder Research T8-33 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools PRW: Neural Network Training © 1998 Elder Research T8-34 updated October 19, 1998 CART Scenario S-Plus See5 © 1998 Elder Research √− √ √ √ √ √ √ √ √ √ √− √+ √ √ √ √ √+ T8-35 √ √ √ √ √ Visual Trees √ √ √ √ √ √ √ √ √ √− √ Pruning Severity √ √+ √ √ √ √ Missing Data Classification Costs Other CHAID Priors Clementine Darwin Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest C5 or C4.5 Decision Trees "CART" KDD-98: A Comparison of Leading Data Mining Tools √ √− √ √ √ √ √ √ √ √ √ √ √ updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools CART: Tree Browsing © 1998 Elder Research T8-36 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Scenario: Tree Browsing © 1998 Elder Research T8-37 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Enterprise Miner: Tree Browsing © 1998 Elder Research T8-38 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Enterprise Miner: Tree Results © 1998 Elder Research T8-39 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools MineSet: Tree Browser © 1998 Elder Research T8-40 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Regression / Stats Linear Clementine Y √ √+ √+ √− √ √ √ √ √ Clementine Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest Enterprise PRW S-Plus S-Plus Scenario © 1998 Elder Research √+ Logistic Complexity CrossPenalty Validation √+ √+ √ √ √ √ √ √+ √ √ √ √+ T8-41 √ Input Selection Factor Analysis √ √ √ √ √ √ √ √ √+ √ √+ √ √ √ √ √ √ updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools MineSet: Bayes Distributions © 1998 Elder Research T8-42 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Enterprise Miner: Clustering Results © 1998 Elder Research T8-43 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Intelligent Miner: Clustering Results © 1998 Elder Research T8-44 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Intelligent Miner: Cluster Explode © 1998 Elder Research T8-45 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Data Loading and Manipulation Model Building Model Understanding Technical Support Overall Clementine Darwin DataCruncher Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest Enterprise PRW √+ √ √+ √ √+ √ √ √+ √ √+ √+ √ √+ √ √ √ √+ √+ √+ √+ √+ √+ √ √ √ √ √+ √+ √+ √+ √+ √ √ √ √ √ √ √+ √+ √+ √+ √ √ √ √ √ √+ √+ √+ √+ CART Scenario NeuroShell OLPARS See5 S-Plus WizWhy √− √ √ √− √ √ √ √ √+ √ √ √ √ √ √ √+ √ √ √ √+ √+ √ √ √ √ √ √ √ √ √+ √ √ √ √ √ Usability © 1998 Elder Research T8-46 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Intelligent Miner: Statistics Report © 1998 Elder Research T8-47 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Scenario: Result Reporting © 1998 Elder Research T8-48 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools WizWhy: Reporting © 1998 Elder Research T8-49 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools DataCruncher: Output Sensitivities © 1998 Elder Research T8-50 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Darwin: Predictions © 1998 Elder Research T8-51 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Darwin: Lift Chart (Excel) © 1998 Elder Research T8-52 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools CART: Gains Chart © 1998 Elder Research T8-53 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Visualization Pie Histograms Charts Clementine Darwin DataCruncher Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest Enterprise PRW CART Scenario NeuroShell OLPARS See5 S-Plus WizWhy © 1998 Elder Research √ √− √ √ √− √ √ √ √ √ √− √ √ √ √ Scatter/ Classification Rotating Conditional Line Decision Scatter Plots Plots Regions √ √− √ √ √− √ √ √ √ √ √ √− √ √ √− √ √ Correlation Plots √ √ √ √ √ √ √ √ √ √ √ √ √ √ √− √ √ T8-54 √ √ updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Clementine: Visualization User-created sub-region © 1998 Elder Research T8-55 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools OLPARS: Visualization © 1998 Elder Research T8-56 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools OLPARS: Decision Space © 1998 Elder Research T8-57 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Model 1: Target Sensitivities © 1998 Elder Research T8-58 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools MineSet: Geographical Visualization © 1998 Elder Research T8-59 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Automation Clementine Darwin DataCruncher Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest PRW CART Scenario NeuroShell OLPARS See5 S-Plus WizWhy © 1998 Elder Research Free Text Annotation of Steps Method of Automation Visual Programming, Programming Language Programming Language (Task manager) Visual Programming, Programming Language Macro Language, Wizards (Wizards) Data History, Log Model Wizard Batch Agenda Experiement Manager; Macros √ √ √ √− √ Built-in Basic Scripting Scripting (S); C/C++ T8-60 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools PRW: Experiment Manager © 1998 Elder Research T8-61 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Model 1: Model Summary © 1998 Elder Research T8-62 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools A Recent Breakthrough: Bundling 1) Construct varied models, and 2) Combine their estimates Generate component models by varying: • Case Weights • Data Values • Guiding Parameters • Variable Subsets Combine estimates using: • Estimator Weights • Voting • Advisor Perceptrons • Partitions of Design Space © 1998 Elder Research T8-63 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Example Bundling Techniques • Bayes: sum estimates of possible models, weighted by priors • GMDH (Ivakhenko 68) -- multiple layers of quadratic polynomials, using two inputs each, fit by LR • Stacking (Wolpert 92) -- train a 2nd-level (LR) model using leave-1-out estimates of 1st-level (neural net) models • Bagging (Breiman 96) (bootstrap aggregating) -- bootstrap data (to build trees mostly); take majority vote or average • Bumping (Tibshirani 97) -- bootstrap, select single best • Boosting (Freund & Shapire 96) -- weight error cases by βτ = (1-e(t))/e(t), iteratively re-model; weight model t by ln(βτ) • Crumpling (Anderson & Elder 98) -- average cross-validations • Born-Again (Breiman 98) -- invent new X data... © 1998 Elder Research T8-64 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Distinctives Strengths Weaknesses Clementine Darwin DataCruncher Enterprise Miner GainSmarts Intelligent Miner MineSet Model 1 ModelQuest PRW vis ual inte rfa c e ; algorithm bre a d th e ffic ient c lie nt-s e rve r; intuitive inte rfa c e o p tions e a s e o f us e d e p th of algorithm s ; visual inte rfa c e data trans fo rmations , built on SAS ; a lgorithm option depth algorithm bre a d th; graphical tre e /clus te r output data visualization e a s e o f us e ; automate d m o d e l dis c o ve ry bre a d th of alg o rithms e xte n s ive a lgorithms; automate d m o d e l s e le c tion s c a lability no uns upe rvis e d ; limite d vis ualization s ingle a lgorithm harde r to u s e ; ne w product is s ue s no uns upe rvis e d ; limite d vis ualization fe w a lg o rithm options ; no automation fe w a lg o rithms; no model e xport re a lly a ve rtical to o l s o m e non-intuitive inte rfa c e o p tions limite d visualization CART Scenario NeuroShell OLPARS See5 S-Plus WizWhy d e p th of tre e o p tions e a s e o f us e multiple neural ne twork archite c ture s multiple s tatis tical algorithms; cla s s -bas e d vis ualization d e p th of tre e o p tions d e p th of algorithm s ; visualization; programable /e xte ndable e a s e o f us e ; e a s e o f mode l unde rs tanding difficult file I/O; limite d visualization narrow analysis path unorthodox inte rfa c e ; only neural ne tworks date d inte rfa c e ; difficult file I/O limite d visualization; fe w data options limite d inductive m e tho d s ; s te e p le a rning curve limite d visualization © 1998 Elder Research T8-65 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Closing Observations • Data Mining Tools Can: – Enhance inference process – Speed up design cycle • Data Mining Tools Can Not: – Substitute for statistical and domain expertise • Users are advised to: – Get training on tools – Be alert for product upgrades © 1998 Elder Research T8-66 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Index to Tools (Page numbers in italics refer to screen captures) Clementine Darwin DataCruncher Enterprise Miner Gainsmarts Intelligent Miner MineSet Model1 ModelQuest PRW 7, 8, 10, 18 , 20, 31, 32, 35, 41, 46, 54, 55 , 60, 65 7, 8, 10, 20, 31, 32, 35, 46, 51 , 52 , 54, 60, 65 7, 8, 10, 19 , 20, 31, 46, 50 , 54, 60, 65 7, 8, 10, 17 , 20, 31, 32, 33 , 35, 38 , 39 , 41, 43 , 46, 54, 60, 65 7, 8, 10, 20, 31, 35, 41, 46, 54, 60, 65 7, 8, 10, 16 , 20, 31, 32, 35, 41, 44 , 45 , 46, 47 , 54, 60, 65 7, 8, 10, 14 , 20, 31, 35, 40 , 41, 42 , 46, 54, 59 , 60, 65 7, 8, 10, 20, 22 , 23 , 31, 32, 35, 41, 46, 54, 58 , 60, 62 , 65 7, 8, 10, 20, 31, 35, 41, 46, 54, 60, 65 7, 8, 10, 15 , 20, 21 , 31, 32, 34 , 41, 46, 54, 60, 61 , 65 CART Neuroshell pcOLPARS Scenario See5 S-Plus WizWhy 7, 8, 10, 20, 31, 35, 36 , 46, 53 , 54, 60, 65 7, 8, 10, 20, 31, 32, 46, 54, 60, 65 7, 8, 10, 13 , 20, 31, 32, 46, 54, 56 , 57 , 60, 65 7, 8, 10, 20, 24 , 31, 35, 37 , 41, 46, 48 , 54, 60, 65 7, 8, 10, 20, 31, 35, 46, 54, 60, 65 7, 8, 10, 20, 31, 35, 41, 46, 54, 60, 65 7, 8, 10, 20, 31, 46, 49 , 54, 60, 65 © 1998 Elder Research T8-67 updated October 19, 1998 KDD-98: A Comparison of Leading Data Mining Tools Forthcoming Report • Report provides detailed comparison of high-end data mining tools, including capabilities, ease of use, and practical tips. • Available for $695 from Elder Research (http://www.datamininglab.com), Q4 1998. • Purchasers receive brief free consulting session to explore report findings in more detail, if desired. Note: The analyses and reviews were performed completely independently, and were made possible by the cooperation of the vendors, for which Elder Research is very grateful. The companies, however, provided no financial support, and had no influence on its editorial content. © 1998 Elder Research T8-68 updated October 19, 1998