Introducing NOAA’s Microwave Integrated Retrieval System (MIRS)
advertisement
Introducing NOAA’s Microwave Integrated Retrieval System (MIRS) S-A. Boukabara, F. Weng, R. Ferraro, L. Zhao, Q. Liu, B. Yan, A. Li, W. Chen, N. Sun, H. Meng, T. Kleespies, C. Kongoli, Y. Han, P. Van Delst, J. Zhao and C. Dean NOAA/NESDIS Camp Springs, Maryland, USA 15th International TOVS Study Conference (ITSC-15), Maratea, Italy, October 9th 2006 Contents 1 Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 2 Overview Stated Goals of MIRS Algorithm for sounding, imaging, or combination thereof Applicable to all Microwave Sensors Extend over non-oceanic surfaces & in all-weather conditions Operate independently from NWP model forecasts Benefits Reduction of Time/Cost to Adapt to New Sensors Reduction of Time/Cost to Transition to Operations Improvements in Severe Weather Forecasts Better Climate Data Records 3 MIRS Concept Variational Retrieval (1DVAR) CRTM as forward operator, validity-> clear, cloudy and precip conditions Emissivity spectrum is part of the retrieved state vector Algorithm valid in all-weather conditions, over all-surface types Cloud & Precip profiles retrieval (no cloud top, thickness, etc) EOF decomposition Sensor-independent Highly Modular Design Flexibility and Robustness Selection of Channels to use, parameters to retrieve Modeling & Instrumental Errors are input to algorithm 4 Contents 1 Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 5 Cost Function Minimization Cost Function to Minimize: ( ) ( ) T Radiance ⎤ ⎤ ⎡ 1Jacobians ⎡1 T −1 m & J(X)= ⎢ (X − X0 ) × B × (X − X0 )⎥ + ⎢ Y − Y(X) × E−1 × Ym Simulation − Y(X)⎥ ⎦ ⎦ ⎣ 2 from Forward Operator: CRTM ⎣2 ∂J(X) = J'(X) = 0 To find the optimal solution, solve for: ∂X ⎡ ⎤ y(x) = y(x ) + K x − x ⎢ ⎥ Assuming Linearity ⎢⎣ 0 0 ⎥⎦ This leads to iterative solution: ⎫ −1 = B−1 + KnTE−1Kn KnTE−1⎪⎪⎬⎡⎢⎛⎜⎝ Ym − Y(Xn ) ⎞⎟⎠ + KnΔXn ⎤⎥ ΔX ⎦ n+1 ⎪⎣ ⎧ ⎪⎪⎛⎜ ⎨⎜ ⎪⎝ ⎪⎩ ΔX More efficient (1 inversion) ⎧ ⎪⎪ ⎨ ⎪ ⎩⎪ ⎞ ⎟ ⎟ ⎠ = BKT K n BKT n n +E n+1 ⎛ ⎜ ⎜ ⎝ ⎞ ⎟ ⎟ ⎠ ⎪⎭ −1⎫⎪⎪⎡ ⎛ ⎬⎢⎜ ⎪⎣⎝ ⎭⎪ Ym − Y(Xn ) ⎞⎟⎠ + K nΔXn ⎤⎥ ⎦ Preferred when nChan << nParams (MW) 6 Quality Control of MIRS Outputs Convergence Metric: χ2 Uncertainty matrix S: −1 S = B − B× KT K × B× KT + E × K × B ⎛ ⎜ ⎜ ⎝ Contribution Functions D: happening for each parameter. ⎞ ⎟ ⎟ ⎠ indicate amount of noise amplification D = B×KT K ×B×KT + E ⎛ ⎜ ⎜ ⎝ Average kernel A: ⎞ ⎟ ⎟ ⎠ −1 × ⎛⎜⎜ Y(X) − K × X ⎝ 0 ⎞ ⎟⎟ ⎠ A = D× K If close to zero, retrieval coming essentially from background If close to unity, retrieval coming from radiances: No artifacts from background 7 System Design & Architecture Raw Measurements Ready-To-Invert Radiances Level 1B Tbs Heritage Algorithms Radiance Processing 1st Guess HeritageNOAA-18/METOP Algorithms Radiometric Bias NEDT Matrx E MSPPS Legend Inputs To Inversion Other MW sensors Advanced Vertical DMSP SSMI/S Monitored Integration & Retrieval WINDSAT Used In Geoph. Monit. Post-processing (1DVAR) FNMOC Operational Algs Combination of : (Grody 1991, Ferraro et al. 1994, Weng and Grody 1994, selection Alishouse et al. 1990) + + Locally Developed Developed InversionLocally Process MIRS Algorithms for: Algorithms for: Temperature Profile Water Vapor Profile Emissivity Spectrum External Data & Tools Products Temperature Profile Water Vapor Profile Emissivity Spectrum - Existing Published Ready-To-Invert Algorithms Radiances Complemented by RTM Uncert. Matrx - Locally Developed F Statistical Algorithms NWP Ext. Data EDRs 8 System Design & Architecture Raw Measurements Ready-To-Invert Radiances Level 1B Tbs Radiance Processing External Data & Tools Legend Inputs To Inversion Advanced Algorithm Outputs Advanced 1st Guess Vertical Monitored Heritage Vertical Integration and Post-Processing Integration & Retrieval Algorithms Used In Geoph. Monit. Advanced Algorithm (1DVAR) Measured Radiances Post-processing TPW Temp. Profile RWP Humidity Profile Vertical IWP Radiometric Bias Yes Comparison: Fit Integration CLW Solution Simulated Radiances selection Liq. Amount Prof Ready-To-Invert Within Noise Level ? NEDT Matrx E Ice. Amount Prof Initial State Vector Rain Amount Prof Emissivity Spectrum Skin Temperature Core Products CRTM Reached Radiances No Inversion Process MIRS Update Post Products State Vector -Snow Pack Properties -Land Moisture/Wetness RTM Uncert. Matrx -Rain Rate F -Snow Fall Rate Processing -Wind Speed/Vector (Algorithms) -Cloud Top -Cloud Thickness NWP Ext. Data -Cloud phase New State Vector -Etc. EDRs 9 Contents 1 Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 10 Performance Evaluation 4 Hurricane Conditions 3 Over Land (TPW) Validation 2 Profiling (T,Q) 1 Imaging (TPW) 11 QC of the Validation Set Error increases with increasing time separation 12 Improvement Assessment (wrt Heritage) MSPPS: NOAA’s operational system responsible for deriving microwave products MSPPS (bias) MIRS (Bias) MSPPS (Std) MIRS (Std) Improvement (%) N15 1.87 0.49 4.57 3.85 16% N16 1.31 -1.10 4.22 3.85 9% N17 2.51 -0.2 4.26 3.30 23% Average TPW Standard Deviation Improvement is 16% over ocean Better scan angle handling Independence from NWP forecast outputs Capability extended over land 13 Performance Evaluation 4 Temperature & Humidity profiles using N15,16,17,18 Hurricane Conditions with Comparison radiosondes (statistical) 3 Over Land (TPW) Validation 2 Profiling (T,Q) 1 Imaging (TPW) 14 Temperature & Humidity Profiles Raob Profiles with at least 30 levels used. Ocean cases only. Retrievals up to 0.05 mbars. Assessment only up to 300 mbars. These are real data performances (stratified by sensor) Results shown here are cloudy (up to 0.15 mm from MIRS retrieval) Independent from NWP forecast information, including surface pressure Improvements in progress (scan-dependent covariance Matrix, air-mass preclassification, etc) 300 mb 300 mb NOAA-15 NOAA-16 Pressure (mb) Pressure (mb) NPOESS IORD-II Requirements Temperature Std Deviation Error NOAA-17 Humidity Std Deviation Error 15 Importance of the Evaluation Set 300 Polar Profiles 300 Tropical Profiles Temperature Std Deviation Error Temperature Std Deviation Error 300 Focus of current efforts: All Profiles & All sensors together - Air-mass preclassification of the background - Improvement expected from first guess Temperature Std Deviation Error Caution must be exercised when comparing performances of algorithms on different sets. Types of sets are critical (tropical, polar, clear, cloudy, etc and their relative percentage in the set). 16 Performance Evaluation 4 TPW retrieval extended over land Hurricane Comparison with GDAS analyses Conditions Comparison with Radiosondes over land Sanity Check of MIRS Emissivity Over Land Validation 3 Over Land (TPW) 2 Profiling (T,Q) 1 Imaging (TPW) 17 Microwave TPW Extended over Land snow-covered surfaces need better handling GDAS Analysis Retrieval over sea-ice and most land areas capturing same features as GDAS MIRS Retrieval 18 Validation of TPW Retrieval over Land MIRS-based TPW Performances over Land Bias: -1.13 mm Std Dev: 4.09 mm Corr. Factor: 0.86 #Points: 4293 ~4000 NCDC IGRA points collocated with NOAA-18 radiances Only convergent points over land used Only points within 0.5 degrees and within 1hour Cloudy points included, up to 0.15 mm 19 Extension of MIRS Validity Over Land Same MIRS code used for retrieval over land and ocean. Approach based on a switch between surface emissivity constraints matrices River/Coast signature(based on surface classifier/topography) 20 Global Temperature Profiling No Scan-Dependence in retrieval Smooth Transition Land/Ocean QC-failure is based on convergence: Focus of on-going work Similar Features Captured 21 Performance Evaluation 4 Hurricane Conditions Temperature profiling in active regions (using NOAA-18 sounders) Validation 3 Over Land (TPW) 2 Profiling (T,Q) 1 Imaging (TPW) 22 Challenges of Profiling in Active Areas • Case of July 8th 2005 700 mb DeltaT=3K MHS footprint size at nadir is 15 Kms. But at this angles range (around 28o), the were MHS 4 Dropsondes footprint is around 30 Kms All these dropped within 45 minutes and are located within 10 kms from each other 700 mb Zoom in space (over the Hurricane Eye) and Time (within 2 hours) Temperature [K] DeltaQ=4g/Kg Water Vapor [g/Kg] 23 N-18 Profiling In Active Areas 700 mb 700 mb 0.2 Hrs 2.6 Kms 0 15 700 mb 0.30 Hrs 11.1 Kms 30 Deg C Retrieval GDAS DropSonde 0 0.7 Hrs 4.2 Kms 15 30 0 15 30 [Deg. C] [Kms] Profile of DS Distance Departure 24 Contents 1 Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 25 MIRS Applications MIRS is applied to a number of microwave sensors, Each time gaining robustness and improving validation for Future New Sensors √ √ POES N18 DMSP SSMIS √ √ CORIOLIS WINDSAT Cumulative Validation & Consolidation of MIRS NPP/NPOESS ATMS, MIS √: Applied Daily √: Applied occasionally √: Tested in Simulation 26 Online Access Online Scrolling Menus 27 Products Performance Monitoring – Functionalities (cont’d) 28 Questions? BACKUP SLIDES 30 Core Retrieval Mathematical Basis Theorem (of Joint probabilities) In Bayes plain words: Main Goal in=ANY to find a vector X P(X, Y) P(Y |Retrieval X) × P(X) =System P(X | Y)is × P(Y) with a maximum probability of being the source responsible for the measurements vector Ym Mathematically: m P(Y | X)× P(X) P(X | Ym ) = m ) is to find a vector X: Main Goal in ANY RetrievalP(Y System P(X|Ym) is Max =1 31 Core Retrieval Mathematical Basis Maximizing In plain words: P(X | Y m ) = ⎧ ⎡ ⎤ ⎡ ⎪exp⎢ 1 ⎛⎜ X X ⎞⎟ T B −1 ⎛⎜ X X ⎞⎟⎥ exp⎢ 1 ⎛ Ym ×⎝ − 0 ⎠⎥ × − ⎨ ⎢ −2 ⎝ − 0 ⎠ × ⎢ − 2 ⎜⎝ ⎪ ⎢ ⎢ ⎥ ⎣ ⎣ ⎦ ⎩ ⎤⎫ ⎞⎟ T ×E −1 ×⎛⎜ Ym − Y(X)⎞⎟⎥ ⎪⎬ Y(X) ⎠ ⎝ to find a vector X ⎠⎥ Main Goal in ANY Retrieval System is with a maximum probability of being the source responsible for the measurements vector Ym ⎥⎦ ⎪⎭ Is Equivalent to Minimizing m )⎞ ⎛ − ln P(X | Y ⎜ ⎟ Mathematically: Probability PDF Assumed Gaussian Mathematically: ⎝ ⎠ PDF Assumed Gaussian Probability around Background Y(X) with a around Background X0 with a Covariance E MainWhich Goalamounts in ANY Retrieval System is to findFUNCTION a vector X: Covariance B to Minimizing J(X) –also called COST – ⎡ m⎞)⎤⎥ is Max ⎞ ⎛ − P(X|Y ⎢− ⎛⎜ Same − cost × × ⎜ used − in 1DVAR Data ⎡ Assimilation System ⎤ ⎟ Function ⎟⎥ ⎢ ⎣⎢ ⎝ ⎠ ⎝ ⎠ ⎦⎥ ( ⎢− ⎢ ⎢⎣ ⎛ ⎜ ⎝ ⎞ ⎟ ⎠ − ) × − × ⎛⎜ − ⎝ ( ⎞⎥ ⎟⎥ ⎠ ⎥⎦ ) T ⎤ ⎤ ⎡1 m ⎡1 T −1 J(X) X0 ) × = ⎢ (Xto−how × (X − X0 )⎥ + ⎢ Y − Y(X) × E−1 × Ym − Y(X)⎥ Problem reduces toBmaximize: ⎦ ⎦ ⎣2 ⎣ 2T ⎡ ⎤ ⎡ ⎧ ⎪ exp ⎢ 1 ⎛ X − ⎨ ⎢ 2 ⎜⎝ ⎪ ⎢⎣ ⎩ − X ⎞⎟ 0⎠ ×B −1 ×⎛⎜ X − X ⎝ ⎞⎥ 0 ⎟⎠ ⎥ ⎥⎦ ⎤⎫ T 1 m 1 m − × exp ⎢⎢ − ⎛⎜ Y − Y(X)⎞⎟ ×E ×⎛⎜ Y − Y(X)⎞⎟ ⎥⎥ ⎪⎬ 2⎝ ⎠ ⎝ ⎠ ⎪ ⎢ ⎥ ⎣ × ⎦⎭ 32 System Design & Architecture Raw Measurements Ready-To-Invert Radiances Level 1B Tbs External Data & Tools Radiance Processing Heritage Algorithms 1st Guess Advanced Retrieval (1DVAR) Legend Inputs To Inversion Vertical Integration & Post-processing Monitored Used In Geoph. Monit. Radiometric Bias selection NEDT Matrx E Inversion Process MIRS Products Ready-To-Invert Radiances RTM Uncert. Matrx F NWP Ext. Data EDRs 33 System Design & Architecture External Data & Tools Raw Measurements Level 1B Tbs Radiance Processing & Radiometric Bias Monitoring Radiometric Bias Ready-To-Invert Radiances NEDT Matrx E Inversion Process RTM Uncert. Matrx F Geophysical Bias NWP Ext. Data EDRs Comparison In-Situ Data 34 Nominal approach: Simultaneous Retrieval Necessary Condition (but not sufficient) F(X) Does not Fit Ym within Noise X is not the solution F(X) Fits Ym within Noise levels X is a solution X is the solution All parameters are retrieved simultaneously to fit all radiances together 35 Assumptions Made in Solution Derivation The PDF of X is assumed Gaussian Operator Y able to simulate measurements-like radiances Errors of the model and the instrumental noise combined are assumed (1) non-biased and (2) Normally distributed. Forward model assumed locally linear at each iteration. 36 Retrieval in Reduced Space (EOF Decomposition) All retrieval is done in EOF space, which allows: Retrieval of profiles (T,Q, RR, etc): using a limited number of EOFs More stable inversion: smaller matrix but also quasi-diagonal Time saving: smaller matrix to invert Mathematical Basis: EOF decomposition (or Eigenvalue Decomposition) • By projecting back and forth Cov Matrx, Jacobians and X Θ = LT × B× L Diagonal Matrix Transf. Matrx Covariance matrix (used in reduced space retrieval) (computed offline) (geophysical space) 37 & (2) No risk of having unphysical negative values Advantages: (1) Distributions made more Gaussian Retrieval in Logarithm Space P=416 mb P=506 mb P=606 mb P=718 mb P=810 mb P=915 mb Applied to WV, Cloud and precip P=416 mb P=606 mb P=810 mb P=506 mb P=718 mb P=915 mb ∂R ∂x = ∂R × J = = J× x l ∂ Log(x) ∂ x ∂ Log(x) 38 Validation Approach Use of Multiple Microwave Sensors: AMSU A/B (or MHS) onboard NOAA-15-16-17-18 WINDSAT onboard CORIOLIS SSMI/S onboard DMSP F-16 Two Types of Validation, depending on parameter Quantitative Validation • • • • NWP Data (GDAS) Heritage Algorithms (MSPPS) Conventional Radiosondes (from NCEP and from NCDC) GPS-DropSondes Qualitative Validation • Science Constraints in Retrieval System • Capture of known meteorological phenomena Metrics: Standard statistical metrics Bias/RMS/StdV/Correlation Case By Case Evaluation (especially for active areas) 39 TPW Comparison MIRS vs MSPPS MSPPS TPW used as reference MSPPS TPW [mm] MIRS retrieves the humidity profile. The TPW is integrated in postprocessing stage. MSPPS relies on NWP forecast for both SST and Wind (emissivity). MIRS TPW [mm] MIRS is independent of NWP data (even from surface pressure). 40 Cloud Retrieval Using SSMI/S FNMOC Operational Algorithm MIRS is more sensitive to small values (due to use of higher frequency channels) MIRS Algorithm Retrieval using DMSP F16 SSMI/S 41 In-Situ Global Distribution Source Period Coverage # of Points Ref. POES NOAA15 NCEP 2002-2004 Ocean 1255 Liu & Weng 2004 POES NOAA16 NCEP 2002-2004 Ocean 1655 Liu & Weng 2004 POES NOAA17 NCEP 2002-2004 Ocean 1522 Liu & Weng 2004 POES NOAA18 NCDC-IGRA 2005-2006 Land ~8,000 Durre et al. 2006 Dataset used for both TPW and Profiling Validation 42 Retrieval Bias vs. Viewing Angles 6 Match-up TPW from radiosondes Bias variation to viewing angles. Bias = radiosonde – AMSU TPW bias (m m ) and AMSU retrieval in 2002. NOAA-15, bias = radiosonde - retrieval 5 4 3 2 1 dvar 1 MSPPS 0 -1 -2 -3 -60 4 -40 NOAA-16, bias=radiosonde-retrieval -20 0 20 40 60 viewing angle (degree) 7 2 1 0 -1 -2 -60 -40 -20 0 20 viewing angle 40 60 1 dvar 6 MSPPS 5 TP W bias (m m ) TPW bias (m m ) 3 NOAA-17, bias-radiosonde-retrieval 4 3 1 dvar 2 MSPPS 1 0 -1 -2 -60 -40 -20 0 20 40 60 viewing angle (degree) 43 Emissivity Qualitative Validation Emissivity Difference @ 31 GHz Em31GHz MIRS Retrieval EM31GHz Analytical Computation Analytic Extraction: Using MIRS and AMSU/MHS: Diagonal emissivity variances boosted (over land) Off-diagonal zeroed out (channels independent) Use of GDAS (T,Q, Tskin) as 1st guess & Backg but parameters free to move = ⋅ ⋅ + ↑ + ↓ ⋅ ( − )⋅ ⎞ ⎞ T − T↑ ⎟⎟ − Γ.T↓ ⎟⎟ Two conditions: ⎟ B ⎟ ⎠ ⎠ ε= Ts ≠ T↓ ⎞ ⎛ Γ.⎜⎜ Ts − T↓ ⎟⎟ ⎟ ⎜ ⎠ ⎝ Γ≠0 ⎛⎛ ⎜⎜ ⎜⎜ ⎜⎜ ⎝⎝ 44 Global Humidity Profiling No Scan-dependence noticed: Angle dependence properly accounted for 45 Performance Evaluation 5 Hurricane Conditions Effect of using scattering RTM on 1 convergence Advanced Validation vs Skin temperature retrieval using WINDSAT Heritage 3 in eye of hurricane Temperature profiling in active regions Profiling 2 (using NOAA-18 sounders) 4 Over Land Advanced Products 46 WINDSAT Retrieval (Chi Square) Rain Model OFF Rain Model ON Rain Model OFF Retrieval using Windsat data (sdr68) Spatial resolution of 6.8 GHz (50 kms) But with a lot of oversampling Non-convergence along coast lines and at cloud/rainy cell boundaries Beam filling (cloud and coastal) Not handled properly 47 SST Retrieval In Hurricane Conditions During Hurricane Dennis on July 6th 2005, WINDSAT Data captured Skin Temperature Cooling inside Eye of Hurricane Dennis Hurricane Track, Courtesy of National Hurricane Center MIRS SST Retrieval Using WINDSAT (Frq 6 to 37 GHz) GDAS SST Analysis 48
