HMI Status
•
HMI is virtually done.
– Virtually  similar to but not in fact
– Front window issue resolved, flight window now at LMSAL will be
installed next week
•
•
– Light leaks to be fixed next week.
– Some electronics boards need conformal coating
– Some software issues remain
Vibration and EMI/EMC testing done
Thermal Balance and Thermal Vacuum testing starting week after next.
•
•
•
•
•
In Air and Vacuum Calibration next few weeks.
Pre-Ship Review set for 25 Sept.
Ship date estimated 5 October.
Launch date slip to probably 15 Jan 2009 pending review at HQ.
Some serious implications from delay
JSOC Overview August 2007
Page 1
JSOC-SDP Status
•
Capture system complete, waiting testing with DDS at White Sands
•
DRMS and SUMS virtually done
•
Level-0 work progressing, to be done by delivery
•
Level-1 work to start after delivery
•
Getting serious about work on basic pipeline modules
JSOC Overview August 2007
Page 2
HMI & AIA JSOC Architecture
GSFC
White Sands
MOC
DDS
Stanford
Housekeeping
Database
Quicklook
Viewing
Primary
Archive
30-Day
Archive
JSOC Overview August 2007
HMI & AIA
Operations
HMI JSOC Pipeline
Processing System
Redundant
Data
Capture
System
Offsite
Archiv
e
LMSAL
Catalog
Offline
Archiv
e
Data
Export
& Web
Service
AIA
Analysis
System
Local
Archive
High-Level
Data Import
World
Science Team
Forecast Centers
EPO
Public
Page 3
JSOC Dataflow Rates
LMSAL secure host
0.04
Hk
Joint
Ops
Dataflow (GB/day)
Quick Look
1610
1230
Data Capture
1230
2 processors each
HMI &
AIA
Science
1210
Level 0
(HMI & AIA)
Level 1
(HMI)
HMI High Level
Processing
2 processors
16 processors
c. 200 processors
1210
75
1610
1200
30d cache
40TB each
Online Data
325TB+50TB/yr
rarely
needed
240
1820
Redundant data
capture system
Data Exports
1230
JSOC Overview August 2007
LMSAL Link
(AIA Level 0, HMI
Magnetograms)
Science
Archive
440TB/yr
(Offiste)
HMI Science
Analysis
Archive
650TB/yr
2 processors
SDO Scientist &
User Interface
Page 4
JSOC-SDP Major Components
JSOC Overview August 2007
Page 5
JSOC Pipeline Processing System Components
Pipeline
Operato
r
Pipeline
processing
plan
JSOC Science
Libraries
Processing
script, “mapfile”
PUI
Pipeline User
Interface
Pipeline Program, “module”
List of pipeline
modules with
needed datasets for
input, output
Utility Libraries
SUMS Disks
DRMS Library
Record
Manage
ment
Keyword
Access
Link
Manage
ment
Record Cache
Data
Access
SUMS
Storage Unit
Management System
DRMS
Processing
History Log
Data Record
Management System
SUMS
Tape
Farm
Database Server
JSOC Overview August 2007
Page 6
Pipeline client-server architecture
Pipeline client process
Analysis code
C/Fortran/IDL/Matlab
OpenRecords
CloseRecords
Generic file I/O
GetKeyword, SetKeyword OpenDataSegment
GetLink, SetLink
CloseDataSegment
JSOC Library
Data Segment I/O
JSOC Disks
JSOC Disks
JSOC Disks
JSOC Disks
Record Cache (Keywords+Links+Data paths)
DRMS socket protocol
DataRecord
Record
Data
Data
Record
ManagementService
Service
Management
Management
Service
(DRMS)
(DRMS)
(DRMS)
Storage unit transfer
AllocUnit
GetUnit
PutUnit
Storage Unit
Management Service
(SUMS)
Storage unit transfer
SQL queries
PostgreSQL Database SQL queries
Server
Series
Tables
JSOC Overview August 2007
SQL queries
Record
Record
Catalogs
Record
Catalogs
Tables
Tape Archive
Service
Storage Unit
Tables
Page 7
Pipeline batch processing
•
A pipeline batch is encapsulated in a single database transaction, “DRMS session”:
–
If no module fails all data records are commited and become visible to other clients of the JSOC catalog
at the end of the session
–
If failure occurs all data records are deleted and the database rolled back
– It is possible to commit data produced up to intermediate checkpoints during sessions
Pipeline batch = atomic transaction
Module 2.1
Commit Data
Register
Module N
&
Module 1
session
…
Deregister
DRMS API
DRMS API
DRMS API
DRMS API
DRMS API
Module 2.2
DRMS API
Input data Output data
records
records
DRMS Service = Session Master
Record & Series
Database
JSOC Overview August 2007
SUMS
Page 8
HMI module status and MDI heritage
Intermediate and high level data products
Primary
observables
Heliographic
Doppler velocity
maps
Mode frequencies
And splitting
Ring diagrams
Local wave
frequency shifts
Doppler
Velocity
Tracked Tiles
Of Dopplergrams
Internal rotation
Spherical
Harmonic
Time series
Time-distance
Cross-covariance
function
Wave travel times
Egression and
Ingression maps
Wave phase
shift maps
Internal sound speed
Full-disk velocity,
sound speed,
Maps (0-30Mm)
Carrington synoptic v and
cs maps (0-30Mm)
High-resolution v and cs
maps (0-30Mm)
Far-side activity index
Line-of-sight
Magnetograms
Stokes
I,Q,U,V
Full-disk 10-min
Averaged maps
Vector Magnetograms
Fast algorithm
Tracked Tiles
Vector Magnetograms
Inversion algorithm
Coronal magnetic
Field Extrapolations
Solar limb parameters
Coronal and
Solar wind models
Brightness feature
maps
Brightness Images
JSOC Overview August 2007
Tracked full-disk
1-hour averaged
Continuum maps
Standalone
production codes
in use at Stanford
Research codes in use
by team
Deep-focus v and cs
maps (0-200Mm)
Stokes
I,V
Continuum
Brightness
MDI pipeline
modules exist
Line-of-Sight
Magnetic Field Maps
Vector Magnetic
Field Maps
Codes being
developed in the
community
Codes to be
developed at HAO
Codes to be
developed at
Stanford
Page 9
AIA Level-2
JSOC Overview August 2007
Page 10
JSOC Data Volumes from Proposal
this version modified to show the links to the hardware plan
Data Path Assumptions
img size
Processe Volume Combin Online
d at
(GB/day) ed
disk
(GB/day cache
channel cadenc compre
)
days
s
e
ss
In from
DDS
HMI: 55,000,000 bps **
SU
553
AIA: 67,000,000 bps **
SU
674
Level-0
HMI: 4k*4k*2 bytes/2-seconds*(pi/4)
0.39 SU
530
3.4E+07
2
4
AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds
3.4E+07
8
10
0.50 SU
1,080
HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4)
3.4E+07
5.5
45
0.39 SU
130
AIA: Level 1.0 same as level-0
3.4E+07
8
10
0.50 tbd
1,080
Higher
level
HMI: See below
7.5E+10
1 86400
1.00 SU
70
AIA (lev1a): movies & extracted regions. @ 20%
6.7E+06
8
10
0.50 LM
216
LMSAL
Link
HMI: Magnetograms (M, B)
3.4E+07
5
90
0.39 na
59
AIA: Full Level-0 data+lev1_extract
3.5E+07
8
10
0.50 na
1,134
Export
HMI: 2 * Higher Level products + 5*10 min B
SU
149
AIA: 3* higher Level products (TRACE < 1)
SU
648
HMI: tlm
SU
553
AIA: tlm
SU
674
Local tape HMI: Lev0, Lev-1, All Higher
SU
730
AIA: Lev0, Lev1a
SU
1,296
Level-1
Offsite
tape
Totals
1,227
1,610
1,210
286
1,193
797
Fixed
Disk
cache
(TB)
Perm
Tape
Tape per
disk per Archive year
year
Fraction (TB)
(TB)
Nearline
retain
days
Nearline
Cache
(TB)
30
16
200%
395
90
49
30
20
200%
482
90
59
100
52
100%
189
180
93
30
32
100%
386
1,900
2,004
0
0
90
95
10%
39
0
0
25
100%
25
0
0
77
100%
77
0
0
46
0
0
0
0
0
100
6
0
0
100
111
0
0
60
1
0
0
60
6
0
0
100%
198
24
100%
241
30
1,227
2,026
412
93
743
2,004
HMI Totals
68
71
610
118
AIA Totals
146
77
984
2,034
Combined (TB)
214
148
1,594
2,151
Tape shelf size (TB)
Tape shelf number of tapes - mixed density
JSOC Overview August 2007
7,968
11,257
Page 11
Stanford JSOC effort plan
HMI-SU Effort Distribution
On-Demand Support
1000
Irradiance
Forecast - farside…
900
Coronal Inferences
Mag Field - Vector Field
800
Mag. Field - Line-of-Sight
Level 2 - Local HS Holography
Level 2 - Local HS Ring Diagram
700
Level 2 - Local HS Time Distance
Level 2 - Global HS
Percent FTE
600
Level 2 - Quick Look
HK & FDS
500
Level-0
JSOC Verification & Test
400
Data Quality and Proc. Metadata
General Env.: cvs,os,oracle,compile
300
Archive h/w arch.
Processing Hardware Arch.
User Tools: API, Data Export, & Browsing
200
pui
Data Capture
100
DRMS
SUMS
0
Support for AIA
FY
FY
FY
FY
FY
FY
FY
FY
FY
FY
FY
FY
FY
2006 2006 2006 2006 2007 2007 2007 2007 2008 2008 2008 2008 2009
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
Q1
JSOC Overview August 2007
Page 12