SciDB
An Open Source Data Base Project
by
Michael Stonebraker
(and others)
Outline
Why
science folks are unhappy with RDBMS
How
we plan to fix that
The
details
Why SciDB?
“Big
science” very unhappy with RDBMS

Astronomy

HEP

Fusion

Bio

Remote sensing
Why?
Experience
Tried
to use Postgres for science databases
Failed

of Sequoia 2000 (mid 1990s)
badly……
Main science data type is an array – horribly
inefficient to simulate arrays on top of tables

Required features absent (provenance, uncertainty,
version control)

SQL operations wrong (regrid – not join)
Why SciDB?
Net

result
Mentality of “roll your own from the ground up” for
every new science project

Realization by the science community that this is
long-term suicide
Community
wants to get behind something
better

Great commonality of needs among domains
A Little Context
XLDB-1
 Genesis
Asilomar
 Small
of the need
conference (March 2008)
conference to generate requirements
A Little Context
March
2008 – September 2008

Initial design completed

Fund raising

Recruiting of initial team

Detailed use cases specified
Our Partnership
Science
and high-end commercial folks

Who will put up some resources

And review design
DBMS

brain trust
Who will design the system, oversee its
construction, and perform needed research
Non-profit
company

Which will manage the open source project

And support the resulting system

May need long term funding help
Partners – Science
(We are recruiting more….)
LSST

astronomy project
DBMS work co-ordinated by SLAC
Pacific

Northwest National Laboratory (PNNL)
Various bio projects
Lawrence

Livermore National Laboratory
Fusion projects
UCSB

Remote sensing
Partners -- DBMS
Mike
Stonebraker (MIT)
Dave
DeWitt (Wisconsin -> Microsoft)
Jignesh
Patel (Wisconsin)
Jennifer
Widom (Stanford)
Dave
Maier (Portland State)
Stan
Zdonik (Brown)
Sam
Madden (MIT)
Ugur
Cetintemal (Brown)
Magda
Mike
Balazinska (Washington)
Carey (UCI)
Partners -- Other
E-Bay
Vertica
Microsoft
LSST
SLAC
Will
hit up NSF and DOE
The SciDB Data Model
Nothing
(e.g. Hadoop, Pig, Hive, …)?
 Most
Hadoop
of you have schemas
is not a good starting point
 Slow
 No
HA
The SciDB Data Model
Tables?
 Makes
 Used
a few of you happy
by Sloan Sky Survey
But
 PanStarrs
scalability
(Alex Szalay) wants arrays and
The SciDB Data Model
Arrays?
 Superset
of tables (tables with a primary
key are a 1-D array)
 Makes
HEP, remote sensing, astronomy,
oceanography folks happy
But
 Not
biology and chemistry (who wants
networks and sequences)
The SciDB Data Model
Multidimensional
 Superset
 Makes
grids
of arrays (non-uniform cells)
solid modeling folks happy
But
 Complex
and slow
SciDB Data Model
Nested
Array
multidimensional arrays
values are a tuple of values and arrays
Sightings (sid, details) [x, y, z, t]
Objects (type, [sid]) [id]
Basic Arrays
Positive
integer dimensions, no gaps
Bounded
or unbounded
Enhanced Arrays
“Shape”
function
 Supports
irregular boundary
Enhanced Arrays
Co-ordinate
systems
 User
defined functions that map integers to
something else
 E.g.

mercator
Use dimension notation to access, e.g.
 A[17,36]
or
 A{468.2,
917.6}
SciDB Query Language
“Parse-tree”
With
representation of array operations
a “binding” to:

MatLab

C++

Python

IDL

There may be more….
User
extendable operations (Postgres-style)
Operations
Standard
relational ones (filter, join)
Plus
whatever you want (regrid, interpolate,
fourier transform, eigenvalues, …)
Plus
We
add your own (Postgres-style)
need science input here!!!
Environment and Storage
Extendable
With
And
grid (cloud) of Linux machines
built-in high availability and failover
built in disaster recovery
In Situ Processing
Operate
on data with loading it
Supported
by a SciDB self-describing file
format
 And
some number of adaptors, e.g. HDF-5,
NetCDF
 Or
write your own
Storage Model
Arrays
are “chunked” in storage
 Chunk
Chunks
Go
size can vary
are partitioned across the grid
for scalability to petabytes
Other Features
Which Science Guys Want
(These could be in RDBMS, but Aren’t)
Uncertainty

Data has error bars

Which must be carried along in the computation
(interval arithmetic)

Will look at more sophisticated error models later
Other Features
Provenance
(lineage)

What calibration generated the data

What was the “cooking” algorithm

In general – repeatability of data derivation
Supported
by a command log

with query facilities (interesting research problem)

And redo
Other Features
Time
travel

Don’t fix errors by overwrite

I.e. keep all of the data

Supported by an extra array dimension (history)
Spatial
support
Named
versions

Recalibration usually handled this way

Supported by allocating an array for the new
version and “diffing” against its parent
Other Features
(Optionally)
integration of the real time data
capture system

“cooking” inside DBMS

Makes provenance capture easier

Sometimes important
Time Line
Q4/08
 start
Late
2009

Late
company, begin research activities
Demoware available
2010

V1 ships
Project Organization
(Build-it for real)
CEO
(Andy Palmer -- Vertica)
Project
CTO
management (Bobbi Heath -- Vertica)
(Stonebraker)
Project Organization
(Design and Research)
Overall
co-ordination (Stonebraker, DeWitt)
Storage
Query
and execution (Madden, Cetintemal)
layer and semantics (Zdonik, Maier)
Provenance
(Widom, Patel)
Resource
management (Balazinska)
Language
bindings (Carey)
SciDB Has a Good Chance at Success
Community
realizes shared infrastructure is
good
“Lighthouse”
Strong
customers
team
Computation
 Easier
 And
goes inside the DBMS
to share
reuse
How Can You Help?
Get
involved!!!!