> > > > > > > > > > > > > partner > > > > > > > > > > > Fans 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000 0 Fans > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > partner RiskMetrics RiskBurst™ RiskMetrics Group Offers industry-leading products and services in the disciplines of risk management, corporate governance and financial research & analysis Scaling on-premise computation to the Cloud Integration of RiskMetrics extensive on-premise capability with Windows Azure We are running on 2,000 instances on Windows Azure We have plans to use 10,000+ instances in 2010 What are RiskMetrics doing with so much computing power? Calculation of financial risk Simulate scenarios for the movement of market factors over time & price financial assets in those scenarios Notoriously complex – can involve Monte Carlo2 for complex asset classes of the kind that the triggered the 'credit crunch‘ Results in very high computational loads for RiskMetrics Daily risk analysis load equivalent to calculating risk on 4 trillion US Stocks Computational loads are characterised by high demand peaks Strong growth trend in calculation complexity www.riskmetrics.com 20 Peak Load Characteristics www.riskmetrics.com 21 Growth trend in calculation complexity Relative Equity Equivalent Units (Log Scale) Maximum Complexity of Risk Analysis Processing Request 10 9 8 Risk problem complexity has doubled every 6 months 7 6 5 4 Processor power doubles every 2 years 3 2 1 0 1994 www.riskmetrics.com Moore’s Law 1996 1998 2000 2002 2004 2006 2008 22 Analytics Architecture: Large-Scale Data Dependent Processing vs. Distributable Work Packets Market and Pricing Data Pricer Pricer Pricer Pricer Pricer Pricer RiskServer RiskServer Scenario Pricing: Load Balancer RiskServer Work Packets are self-contained RiskServer Scenario Generation and Aggregation: These Services dependent on high speed access to large scale data stores and caches www.riskmetrics.com RiskServer Velocity Scenario Cache 23 Work Packet Example: Pricing request for a Mortgage Backed Security Compute Time: 150ms - 30s www.riskmetrics.com 24 Analytics Architecture: Integration of Cloud Resources? Market and Pricing Data Pricer Pricer Pricer Pricer Pricer Pricer Pricer Pricer Pricer Pricer Pricer Pricer RiskServer RiskServer Scenario Pricing: Load Balancer RiskServer Work Packets are self-contained RiskServer Scenario Generation and Aggregation: These Services dependent on high speed access to large scale data stores and caches www.riskmetrics.com RiskServer Velocity Scenario Cache 25 RiskBurst™ Project Timeline Q1 2010 November December September October July - August March June •Project Conception •Choice of Platform www.riskmetrics.com •Initial MSFT Meetings •RiskMetrics joins TAP •TAP team actively involved in architecture decisions •Engineering work on scaling proof of concept •Deep-dive sessions •Large-scale testing with test load (2002000 nodes) •‘Industrialisation’ of architectural pattern •Run parallel with in-house solution •Production •Large-scale UAT using load application •Complete work on operational integration 26 RiskBurst™ An architectural pattern for large scale computational applications www.riskmetrics.com 27 Architectural Pattern Building large scale computation requires careful design Problem: Need to avoid the Von Neumann Bottleneck Keywords: Reason and Instrument No changes to the application Run on-premise on HPC Server or in cloud on Azure Pattern has end-to-end decoupling Horizontal scaling of decoupled components Workload Workload Workload Generation Workload Generation Generation Generation www.riskmetrics.com Messaging & Messaging & Storage Messaging Storage & Messaging Storage & Messaging Storage & Messaging Storage & Messaging Storage & Storage Computational Computational Resources & Computational Resources & Computational Application Resources & Application Computational Resources & Application Computational Resources Application & Resources & Application Application 28 RiskBurst™ Workflow: Windows Azure & HPC Server RiskBurst™ Server Windows Azure WCF Request WCF Request Workload Receiver Batching and Sending Input Message Queue(s) WCF Request Scenario WCF Response Generator WCF Response Windows Azure Worker Output Monitoring www.riskmetrics.com Output Message Output Queue(s) WCF Response WCF Error Response Input Outstanding Request Timeout Sweeper 29 Windows Azure Storage Component Usage Input Queues (To Do Jobs) Azure Queue Azure Queue Azure Queue Azure Queue Azure Queue RiskBurst Server Input Blob Storage Worker Role Worker Role Worker Role Instance Worker Role Instance Instance Instance Support files in Blob Storage Azure Queue Azure Queue Azure Queue Azure Queue Azure Queue Local storage Data Output Queues (Job done) Output Blob Storage www.riskmetrics.com 30 Mapping to the Azure Environment Visual Studio 2008 Azure development SDK mimics cloud Mix code running in dev locally, with cloud resources such as Blob storage or queues Good for features, does not assist with scale Existing 32-bit .NET C++/CLI application with 3 third-party libraries Initial idea - run directly in web-role – but 32-bit(!) Run within worker role Preserve WCF interface – no changes whatsoever to analytics app Only changes to existing code base are: Retrieve Cash-flow library support files from Blob storage on demand Some diagnostic information added www.riskmetrics.com 31 Getting to Cloud Resources: Bandwidth & Latency Problem: Bandwidth to Azure gateway limited by Internet Solution: pass by reference & blobs Replace pass-by-value calls with pass-by-reference Create key for scenario Large, repeated objects (scenarios) pushed to blob storage WCF call contains only key Each of 1000 scenarios, used for all assets Problem: Communications Latency Within data centre, 20ms latency on WCF call through HPC SOA platform Queues and Blob storage are off-device; engineering must respect this! Work packet : 200ms computation Solution: batch requests within input queues But, more simultaneous work requests (threads outstanding on input) www.riskmetrics.com 32 Utilizing Cloud Resources: Generating Load www.riskmetrics.com 33 Utilizing Cloud Resources: Generating Load Problem: Generating Load for Cloud Resources Threading architecture Workload originally generated by synchronous calls in client Number of outstanding pricing requests = nodes x batch size Implies large number of threads in wait states in scenario generators Work request made asynchronous RiskBurst™ Server Logic Creates a balanced workload – uses a work item’s average run time Made calls to RiskBurst™ Server asynchronous Incoming calls create batch entry synchronously with request Map created from message id to wait handlers When batch full, sent on to Azure input queue Sweeper thread gathers up output messages and uses map to associate with wait handlers Scales well to over 1000 simultaneous requests per RiskBurst™ Server Horizontal scale of RiskBurst™ Servers – each creates own input queue www.riskmetrics.com 34 Horizontal Scaling within the Cloud Problem: Saturation behaviour of queues Can create situation where queues are saturated, made worse by retry logic Complexity due to varied processing time Controller will move busy queues to independent hardware Use exponential back-off algorithm Batch work items for each queue read or write (using 10 work packets per queue item) Amortizing the cost of IO against CPU time is key Batch compute sizes need to be big enough both to occupy the CPU for long enough and not cause the swamping of the queues Also, more items contained in queue item -> fewer queue hits But, larger batches imply more simultaneous outstanding connections on client side Variable run-time of assets – from 150ms – 30 seconds Carry out processing concurrently with queue access Pushing IO onto background threads is critical (the writes and the deletes are independent background tasks) On-node caching within worker role to avoid queue reads www.riskmetrics.com 35 Exception Management in Distributed Applications Keep it simple Large distributed system implies need to engineer robustness to failure Distinguish between events that are random and unpredictable and poison-message kind of failures Do not over-engineer efficient handling of occasional exceptions Return exceptions to client application Client can track number of attempts to process a work item Distinguish poison messages and give up Parallel handling on HPC Server SOA platform Complexity from varying message processing times Time-outs can be caused by several long-running pricings in same job Re-try time-outs by sending all pricings in batch independently www.riskmetrics.com 36 Diagnostics and Run-time Monitoring A challenge for large scale applications, even more so for Cloud Logging and monitoring must be switchable so as to reduce overhead Variable level of diagnostics and logging Requirement to filter information through decoupled architecture (on node; centralized in Azure; returned to client) Key data for architectural pattern Request and result queue; successful/unsuccessful read, write and delete; time taken for all operations Empty request queue gets Count of successful/unsuccessful work packets % Processor Time performance counter Cache misses We utilized custom built solution during TAP Nodes broadcast over service bus Clients subscribe to trace messages New diagnostic & monitoring package provides platform support www.riskmetrics.com 37 Final Comments Integrating on-premise and cloud applications www.riskmetrics.com 38 Production Services across On-Premise and Cloud Operational Integration Fully integrate Windows Azure capabilities with RiskMetrics Operational Infrastructure Provisioning plus diagnostic & monitoring packages “Outside-In” Services Control and visibility of the services on the cloud consistent with on-premise services. Resource View Nodes Queues Blob Stores Process View Throughput & Performance Traceability Problem identification Process linkage (intra- & inter-cloud) Binding SLA Commitments Operational Support Escalation www.riskmetrics.com 39 RiskBurst™ on Windows Azure Effective architectural pattern delivers key business benefits Elastic scaling Enhanced services Empowered innovation High reliability Improved agility Windows Azure was an obvious choice of cloud platform Minimize impedance mismatch between on-premise and off-premise .NET/WCF/HPC SOA in data center extended to cloud Configure to run in either environment Familiar development environment Massive scalability View of Azure as extension of OS into Cloud Undertake work with HPC Server Team in 2010 Ability to target either Azure-hosting WCF services or HPC Server hosted WCF services in a seamless manner Synchronization of on-premise Velocity instance with Azure instance www.riskmetrics.com 40 Acknowledgements Prototype Development: Supporting Cast: Stuart Hartley (University of York, UK) Alistair Beagley (DPE / Azure) Simon Davies (TAP programme) Patrick Butler Monterde (TAP Programme) Production Development Team: Azure Product Group (Hoi Vo, Brad Calder, Rich Bower (Team Lead) Tom Fahrig, Joe Chau) Kelly Crawford (RiskBurst Server/Client) Hunter Cadzow & Analytics Development at RiskMetrics Simon Davies (TAP Programme) Jonathan Blair (Microsoft Consulting) Tom Stockdale (RiskMetrics CTO) www.riskmetrics.com www.riskmetrics.com 41 > > > > > > > channel9.msdn.com/learn Built by Developers for Developers…. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.