AGRESEARCH SHOW AND TELL
Information Services Software Development
And Specialists Team
Development, Web and Data Services
Greg Peyroux, Information Solutions Manager
greg.peyroux@agresearch.co.nz
Alan McCulloch, Software Engineer
alan.mcculloch@agresearch.co.nz
OVERVIEW
Demonstrations of applications, services and data
of interest, discussion, expressions of interest
•Group Structure
•Team
•Who and What We Do
•New Direction and Business Focus
•Development and Management Tools
•Open source and cross platform tools and
frameworks
•Examples
INFORMATION SERVICES GROUP
Information Technology and Knowledge Services
• Knowledge
•
•
•
Library
Records management a
Knowledge advisory
• IT
•
•
•
Technical
User Support
Development, Web, Data and Bioinformatics
THE TEAM – REBUILT AND REPOSITIONED
Recent History of AgResearch:
•Series of Reviews
•‘Fit for Purpose’ Initiatives
•Significant Team Turnover
Opportunity to innovate and improve
THE TEAM – REBUILD AND REPOSITION
How do you know that you add value?
•Restructured team: Aligned to Outcomes
•Emphasis on “All AgResearch” + the Bigger Picture
•Focus on strategic goals
•Greater Management and Business Awareness
•Transparency: Work request portal on intranet
A NEW DIRECTION FOR DEVELOPMENT
Development Services
•Priority on solutions for science and farms
•Request Analysis
– Determine optimum solution
»Alternative approach e.g. open source
»Reuse/repurpose existing system
»Purchase shrink wrapped S/W or consultancy
»Develop
•Close-off, Feedback and Executive Reporting
THE TEAM – REBUILDING
Desperately Seeking:
•Fresh Ideas and Concepts
•Interns
•Effective Communication Skills
• Increased Science Capability:
•PHD, MScs, BScs
•Statistical, Research + Technician Experience
•Critical Mass
Is B “Best”?
THE EXTENDED TEAM
Software Development And Specialists Team (Information Solutions)
Development, Web and Data Services
• Developers
– Web, SharePoint
– Windows, Mobile Devices
– Lab (LIMS) and Farm Systems
• Bioinformatics
• Specialists
–
–
–
–
–
GIS
ERP analyst
Reporting Specialist
Database administrator
Records
TEAM IMPACT
• Broad Span
–
–
Internal – All AgResearch
–
Science
–
Support
–
Farms
External
–
Farms
–
Meat processors
–
Commercial Labs
–
Accountants and consultancy firms
Group and Team Structures Create Consistent , Aligned Service
DEVELOPMENT TOOLS
Microsoft Stack
•
•
•
•
•
NET framework
Migrating to TFS 2010
SQL Server
BizTalk
SharePoint 2010 (CMS, team, documents, My Sites etc)
Keen to collaborate on:
• Public Records Act
• Research Data Management
• Mapping Systems
Open Source and Cross Plaform Tools (Alan)
Is B “Best”?
KEY PROJECTS
Organisation-Wide Data Integration
• Support
• Farm-Systems
• Science data
• Location
• Field Data recording
Is B “Best”?
INTEGRATION
Integration of Information Systems
• Resolution Farm Management and Planning
INTEGRATION - LOCATIONZ
• Resolution Farm Management and Planning
INTEGRATION - LOCATION
Global Location Integration and Management
• Spatial Location
•
•
Silver light application – draw, annotate, search and view
Import
• GPS points
• SHP
• KML etc
• Known
•
Desktop Application
• Primarily for sample tracking
• Global Location ID
•
Web service to guarantee retrieval
• Resolution Farm Management and Planning
OVERVIEW OF FARMING/RESEARCH DATA
Farm and Related Research Data Components
Environmental
Primary data sets
Location/GIS-related
Paddock/Transect
Animal
Primary data sets
Animal
locations and
movements
Milk yield and vat
Herd
Tests and results etc
Points, Lines Polygons
Integrate into data warehouse, mine for traits
e.g. Best calvers in the wet
SMART DATA RECORDING AND STORAGE
Dynamic task creation and recording
• Mobile device
– User creates a custom task on their PC
– Records task with a range of checks and rich validation
– User uploads task data into a database
Excel Import and Metadata annotating
FARMING/RESEARCH
DATA
TaskOVERVIEW
Recorder –OF
Integrated
Task Creation Framework
TOOLS FOR MANAGING PEOPLE, PRODUCTS,
PROJECTS AND CHANGE
Ugh! Look at his new
recording system, that
won’t last!
MANAGEMENT TOOLS AND FRAMEWORKS
Value of Tools
•
•
•
•
•
•
Porter’s 5 Forces
PEST
SWOT
Fatal Flaw
Stakeholder mapping
Tipping Point
• Use of levers
• Incubate advocates
• Plan for resistors
MANAGEMENT TOOLS
Managing Change: Kotter’s 8 Steps to Transformation
Kotter, J. (2007). Leading change: Why
transformational efforts fail?
Harvard Business Review , 85(1), 96-103.
OPEN SOURCE AND CROSS-PLATFORM
ENVIRONMENTS AND FRAMEWORKS
Allan McCulloch
WINDOWS AND UNIX HPC RESOURCES
CROSS-PLATFORM SOFTWARE DEVELOPMENT STATUS
EXTENDED DATA MODELLING
DESIGN PATTERNS FOR DATA MODELLING
COLLABORATION SUPPORT - WIKIS AND DRUPAL
WINDOWS AND UNIX HPC RESOURCES
We use the Condor scheduler - creates and manages both Windows
and Unix compute pools. Mature, scalable, cross-platform
•
•
Windows = desktop machines , jobs run in background. Users
approached and volunteer machines to the pool (+ “captive”
machines in training rooms and IS and departmental staff !) as
and when required . Max size so far, simulation run on about
40 machines. Saved $$$ on leased high performance windows
boxes.
Unix = Centos Redhat / Centos based , HP blades (mix of 32 and
64 bit , 2 and 4 core, 2Gb RAM ). B-word data pre-processing
mainly
“LARGE MEMORY” SMP UNIX MACHINES
•
1 x Redhat Linux database server (postgres , mysql, Oracle)
•
•
•
•
Data warehousing
B-word
GIS data (postgres)
1 x Redhat Linux compute server
•
•
•
•
Data Mining
Stats (R , ASREML)
B-word miscellany
Virtual machine provisioning (early days)
CROSS-PLATFORM (LINUX-WINDOWS) IN-HOUSE
SOFTWARE DEVELOPMENT
• Java based middleware supporting various B-word web
interfaces (not being further developed)
• Small scale python, perl and R scripting for specific problems or
pipelines (/ bucket brigades). Unusual for us to develop anything
significant from scratch – huge communities, many packages and
solutions already out there.
• Python based Object-Relational system supporting mainly B-word
databases
• Mathematical modellers developing C++ based models
EXTENDED DATA MODELLING
What is it ?
• ER : Entity-Relationship model : Chen (1976), formalized in Ng
(1981). Standard Terminology.
• Extended Entity Relationship (EER) : little standard terminology ,
many contributors
• EER only one of a number of semantic models
• Developed to meet complex requirements beyond dataprocessing in business and industry – CAD/CAM db, image and
graphics db, cartographic and geological db.
• Ref ElMasri, R.A. , Navathe , S.B. : Fundamentals of Database
Systems
BRDF FRAMEWORK – SOME EER ASPECTS
• Inheritance used to provide a core data provenance and record
identification and classification layer for all tables
•
•
•
•
•
Automatic date stamps and user ids for updates on any table
Keywords field for search engines
Record status for versioning
Class typeid for object-relational mapping support
Universal unique record identifier across all tables – really useful !
• Heterogenous lists can include member records from many tables –
e.g. search engine results; projects manifests (subjects, samples, lab
resources etc)
• Normalised free text comments attached to any record
• Provenance of all data imports - records are linked to the data source
they came from .
• Entity types and relationship types treated uniformly
BRDF FRAMEWORK
• Postgres based (….but portable to dbs like Oracle, SQLServer)
• Easy to install - single setup script
• Around 8 instances mostly b-word related but also hosting
genetics and epidemiological data
• Python based object-relational mapping and web interface
• About to begin work on a C# /.net based OR mapping and new
front-end
• (Custom object relational mappings are cost-effective and work
well for data processing , but objects do not usually plug into
standard GUI componentry , and so therefore are not cost
effective for UI development )
• (and vice-versa !!!  )
DESIGN PATTERNS FOR DATA MODELLING
• “Design Patterns” (Gamma, Helm etc al.) – reusable pieces of
design in Object Oriented software engineering (e.g. observer,
visitor, factory method, façade, flyweight,….)
• We need Design Patterns for databases to control complexity
and cost
• Design patterns easier to define for EER based systems
IMPORT (IN) AND DISPLAY (OUT) DESIGN PATTERNS
STUDY-OBSERVATION DESIGN PATTERN IN
BRDF FRAMEWORK
STUDY-OBSERVATION DESIGN PATTERN SUGGESTED
DISCRETIZING CONTINUOUS MS OBSERVATIONS INTO
BINS
DRUPAL BASED COLLABORATION SUPPORT : MULTITENANTED DESIGN
APPLICATIONS
Some examples of applications
FARM APPLICATIONS
• Resolution Farm
Management and
Planning
• ETS Tools - Forestry
• Resolution Farm Management and Planning
FARM APPLICATIONS - ETS
• MAF lookup
tables and rules
• Google Map
interface
• Extended GPS
support
SUMMARY
•Open source and cross platform
•Windows and Unix HPC resources
•Cross-platform Software development – status
•Extended Data Modelling
•Design patterns for data modelling
•Collaboration support - wikis and Drupal
•Development and Management Tools
•Data integration
•Change management
•About Us
•Team
•Who and What We Do
•Direction and Business Focus
AGRESEARCH SHOW AND TELL
Thank you for your attention
Questions
Please feel free to contact us
Greg Peyroux, Information Solutions Manager
greg.peyroux@agresearch.co.nz
Alan McCulloch, Software Engineer
alan.mcculloch@agresearch.co.nz