Dynamic River Network Simulation at Large Scale ∗ Frank Liu IBM Research Austin Austin, TX Ben R. Hodges Center for Research for Water Resources University of Texas at Austin May 17, 2012 Abstract Fully dynamic modeling of large scale river networks is still a challenge. In this paper we describe SPRINT, an inter-disciplinary collaborative effort between computer engineering and hydroscience to address the computational aspect of this challenge. Although algorithmic details differ, SPRINT draws many design considerations from SPICE, one of the most fundamental EDA tools. Experimental results demonstrate that SPRINT is capable of simulating large river basins at over 100× faster than real time. 1 Introduction EDA tools are essential for designing today’s VLSI semiconductor products. One of the earliest and most fundamental EDA tools is SPICE[1][2]. By working in conjunction with compact device models[3][4], SPICE provides a virtual environment so that designers can rapidly assess the function correctness, performance as well as power consumption. This “virtual prototyping” environment not only ensures the correctness of the complex design before they are committed to the lengthy and expensive manufacturing processes, but also provides the designers directions to optimize the design in order to meet the specifications. Mathematically the function of SPICE can be described as a DAE (differential algebraic equation) solver for electronic circuits. The circuit consists of nonlinear components (e.g., BJT or MOS transistors) and linear components (e.g., resistors, capacitors or inductors). The circuit behavior is described by governing laws: KCL (Kirchhoff’s Current Law), KVL (Kirchhoff’s Voltage Law) and BCR (Branch Constituent Relations). The two Kirchhoff’s circuit laws (KCL and KVL) are derived from Maxwell’s equations. KCL states that the algebraic sum of currents flowing into any ∗ Author’s preprint of invited paper to be presented in Special Session 32 (4 p.m. Wednesday, June 6, 2012) at the Design Automation Conference 2012. CITE CONFERENCE PAPER AS: Liu, F. and B.R. Hodges (2012), “Dynamic River Network Simulation at Large Scale,” in Proceedings of the 49th ACM/EDAC/IEEE Design Automation Conference, June 3-7, 2012, San Francisco, USA, 6 pgs. 1 node within a circuit is zero. KVL states that the algebraic sum of voltage drops along any branch loop is zero. BCR describes the behavior of a circuit element with respect to its branch voltages and currents. For example, the BCR of a resistor is basically the Ohm’s Law. The BCR of multi-terminal MOSFET devices is described by a set of complex equations within the compact model. From a given circuit specification, by applying KCL, KVL and BCR, SPICE constructs a system of nonlinear differential-algebraic equations by using either sparse tableau[5], or modified nodal analysis[6]. The voltages of the circuit nodes as well as branch currents of certain devices (e.g., of inductors) are the canonical unknowns which need to be solved. For linear circuits (with only passive devices), a linear matrix solver is all required to solve the circuits. When nonlinear devices (e.g., MOS transistors) are present in the circuit, modified Newton-Raphson’s method and numerical integration methods (e.g., Gear’s methods[7]) are applied to solve the associated nonlinear DAE. Three types of analyses are available in the original version of SPICE: steady-state (DC), small-signal AC, and transient analysis (TR) with time-varying input excitations. The first version of SPICE was released to public in 1971, well before Open Source Software became a big movement. Over the years, many revisions were made, including SPICE3, which is the first version written in C, instead of FORTRAN as in earlier versions[8]. Today SPICE is still part of the core curriculum on circuit simulation in universities across the globe[9][10][11]. Even over forty years after its inception, modern versions of SPICE are still an important part of the product portfolio of many EDA software companies and are generating multi-million dollar annual revenues[12][13][14]. The success of SPICE has inspired many other simulation projects; some of them are well beyond the traditional boundary of electrical engineering. For example, SUGAR[15] is an open source simulation tool for micro-electromechanical systems (MEMS). Another example is Bio-SPICE, which is an open source framework and software toolset for Systems Biology[16]. In this paper, we describe a simulation software package which is intended to perform dynamic simulation of large scale river networks. Water is essential for the life forms as we know today. Only 3% percent of earth’s water is in the form of fresh water, mainly locked in polar icecaps and continental glaciers. River networks are the fastest distribution conduits through the landscape for fresh water, and hence play vital roles in municipal water supply, low-cost bulk transport of grain, agricultural irrigation, wildlife habitat, and recreation. River networks also play a key role in flooding, either carrying away potential flood waters or causing inundation when the carrying capacity is exceeded. Floods cause devastating loss of lives and economic harm throughout the world. For example the Thailand floods in 2011 not only caused hundreds of casualties and billion dollars of direct property damage to the local communities, but also severely disrupted the electronics supply chain with implications across the globe. Perhaps the most commonly used software for river modeling is HEC-RAS, developed by US Army Corps of Engineers[17]. It primarily serves as a desktop tool for hydraulic engineers conducting flow analysis and flood-plain assessments over main stem rivers or local catchments. What has not been previously available is an efficient scalable river network simulator, which capable of simulating not only the main river stems, but also thousands of kilometers of upstream tributaries. This paper presents the initial development of just such a simulator. The present work is the result of close inter-disciplinary collaboration between computer engineering and civil engineering hydroscience. Although the target applications and detailed algorithms differ, the design philosophy and many design considerations in our simulator are strongly influenced by SPICE. The result is an efficient river network simulator which is capable of simulating large river basins at over 100× of real time. With the recognition to the influence of SPICE, we name 2 our software SPRINT (Simulation Program for RIver NeTworks). For the remaining portion of this paper, we present the first-principle physics model of river networks in Section 2; the simulation and implementation of SPRINT in Section 3, followed by some experimental results in Section 4. 2 Modeling River Networks To fully describe the intricacies of water movement within a river channel requires the 3D NavierStokes and continuity equations. Much as in electrical circuit analyses where Maxell’s equations (the fundamental physical laws with high computational costs) are replaced with simpler models with lower computational costs, in river networks a simpler model is commonly used in place of the Navier-Stokes equations. This model is named after a French mathematician, Adhémar Jean Claude Barré de Saint-Venant, and is usually called the Saint-Venant equations (although is sometimes seen as the St. Venant equations). Several approximations are made to derive and apply a model based on the Saint-Venant equations. Some of these are: • River flow is approximately 1-dimensional; • River bed slope is relatively small; • Effects of river bed friction and turbulence can be modeled as resistance terms. The 1D approximation allows us to integrate 3D Navier-Stokes and continuity equations over a river cross-section without knowledge of the exact velocity distribution. The second approximation is part of a “hydrostatic approximation” that removes smaller-scale dynamic pressure effects. The third approximation allows simplified empirical equations to model resistance, which we will discuss later. Although these approximations are generally valid for large-scale rivers analyses, there are places (such as steep mountain streams) where departures from these idealized flows may be significant. 2.1 Continuity Equation The continuity equation describes the mass conservation in river flow. To illustrate this, consider the cross section of a river segment in Fig. 1. For a control volume highlighted in the diagram (between A2 A1 x2 x1 Q Figure 1: Illustration of a river channel cross section. x1 and x2 ), the mass of water flowing into the control volume between time instances t1 and t2 can be written as: Z t2 [(ρvA)x1 − (ρvA)x2 ] dt (1) t1 3 where ρ is the water density, v is the flow velocity. x1 and x2 defines the control volume along the channel, and the cross-sectional area A is a variable that is a function of the water surface elevation and riverbed geometry. In the same time window, the change of storage within the control volume can be written as: Z x2 [(ρA)t2 − (ρA)t1 ] dx (2) x1 Mass conservation dictates that the algebraic sum of the water flowing into the control volume equals the accumulation of the water in the control volume: Z t2 Z x2 [(Q)x1 − (Q)x2 ] dt = 0 (3) [(A)t2 − (A)t1 ] dx + t1 x1 where Q = vA is the flow rate. For fresh water in a river, variations of density ρ with temperature have negligible effects on flow dynamics, hence we can drop it from both sides of the equation. To derive the differential form of the mass equation, we follow the Taylor expansion: (A)t1 = (A)t2 + ∂A ∂ 2 A ∆t2 ∆t + 2 + ··· ∂t ∂t 2 (4) (Q)t1 = (Q)t2 + ∂Q ∂ 2 Q ∆t2 ∆t + + ··· ∂t ∂t2 2 (5) and where ∆t = t1 − t2 . Retaining only the first two terms in the approximations in Eqn. (4) and (5), then in the limit as ∆t → 0, we have: Z x2 Z x2 Z t2 ∂A [(A)t2 − (A)t1 ] dx = lim dt dx (6) t2 →t1 x x1 t1 ∂t 1 and similarly: Z t2 x2 →x1 Z t2 Z x2 [(Q)x2 − (Q)x1 ] dt = lim t1 t1 x1 ∂Q dx dt ∂x Hence the mass conservation in Eqn. (3) becomes: Z x 2 Z t2 ∂A ∂Q + dx dt = 0 ∂t ∂x x1 t1 (7) (8) or the commonly known continuity equation in differential form: ∂A ∂Q + =0 ∂t ∂x (9) When there is lateral inflow to the channel, the continuity equation becomes: ∂A ∂Q + = ql ∂t ∂x where ql is the lateral inflow per unit length along the channel. 4 (10) 2.2 Dynamic Equation Although continuity can be used with empirical models for simplified representation of river networks, the correct dynamical solution of the Saint-Venant equations requires momentum conservation. The flows into/out of a control volume carry momentum, which is also affected by four forces on the control volume: the water pressure upstream and downstream of the control volume, the pressure from the river banks, gravity, and frictional force developed from the river bed and internal fluid shear. The relationship between forces and momentum is governed by Newton’s second law of motion, which reduces to the dynamic Saint-Venant equation for 1D flow. It can be presented as: ∂ Q2 ∂h ∂Q + + gA = gA(S0 − Sf ) (11) ∂t ∂x A ∂x where g is gravity, h is the depth. S0 is the slope of the river bottom. Sf is the friction slope which we will discuss in the next subsection. Note that unlike the continuity equation in Eqn. (10), the dynamic equation has multiple nonlinearities, with variables Q, A Sf and h. The derivation for Eqn. (11) can be found in many hydraulics books, e.g. [18]. 2.3 Friction Term The frictional forces that extract momentum and energy from the flow are represented by empirical resistance models. A commonly-used model (adopted herein) is the Chézy-Manning formula, which was introduced by an Irish engineer Robert Manning[19]: v= Q 1 2/3 1/2 = Rh Sf A n (12) where Rh is the hydraulic radius, which is the ratio of the wetted cross section area (A) and the “wetted perimeter” (i.e. the distance measured along the river bottom across the river). n is known as “Manning’s n,” serving as an empirical friction coefficient that may have a wide range of uncertainty, so serves as a calibration parameter. 2.4 Saint-Venant Equation To summarize the behavior of a river segment is modeled by nonlinear Saint-Venant equations: ( ∂Q ∂A = ql ∂t + ∂x (13) 2 ∂Q ∂ Q ∂h = gA(S0 − Sf ) ∂t + ∂x ( A ) + gA ∂x where the nonlinear friction slope is described by the Chézy-Manning equation: Sf = n2 Q2 1 A2 R4/3 (14) h There are two independent variables in Saint-Venant equations, the flow rate Q and the wetted area A. Once these two quantities are known, the other quantities, e.g., water depth h, flow velocity v, are dependent functions of Q, A and river cross-section geometry (which may have substantial variability throughout a network). 5 A key difference between electric circuits and river networks is in the upstream propagation of information in the latter. Although mass flows downstream in a river, information of downstream events (e.g. a collapsed bridge that obstructs the flow) propagates upstream as a rising/falling water depth (changing A). Thus, the directionality of information propagation is both upstream and downstream through the network. Thus, the Saint-Venant equations must be solved as a distributed model and cannot be replaced with an equivalent lumped model as used in electronic circuits. 3 Simulation and Implementation The Saint-Venant equations in Eqn. (13) are a coupled set of nonlinear, time-varying, partial differential equations that are solvable by either explicit or implicit numerical methods. Over the years, there have been plethora of explicit methods proposed. For example, the leap-frog method in [20], the popular Lax-Wendroff method[21] in [22], and an modified Lax-Friedrichs method in [23]. The biggest advantage of explicit methods is that there is no need to construct and solve the nonlinear matrix otherwise required for implicit solution of the Saint Venant equations. However, this benefit comes at the cost of limited step size. Explicit methods have to meet the Courant-Friedrichs-Lewy (CFL) condition[24] in every element of a network; a constraint that can result in time steps on the order of seconds in a river simulation. On the other hand, implicit methods are not stability limited by the CFL condition. A moderate step size (CFL ¡ 10) is still preferred for accuracy, but localized high CFL conditions are tolerable. Many practical hydrology software packages use implicit methods[18] due to their robustness. The most commonly used implicit methods to solve Saint Venant equations are the method on staggered grid in [25], and the method on collocated grid in [26]. We focus on the four-points scheme from [26]. 3.1 Four-Point Scheme The four point scheme approximates spatial and temporal differences by using the average of four neighboring points in the x − t plane[26]. It is generally over-damped and works very well for slow moving flow conditions. In a simplified version of the four-point scheme, a time difference is approximated by: 1 ∂ n f (x, t) ' (f n+1 − fj+1 + fjn+1 − fjn ) ∂t 2∆t j+1 (15) where the subscript j represents spatial indices and the superscript n represents temporal indices. The spatial difference is approximated by: 1 ∂ f (x, t) ' (f n+1 − fjn+1 ) ∂x ∆x j+1 (16) and we take the average of two adjacent points to approximate the function value itself. f (x, t) ' 1 n+1 (f + fjn+1 ) 2 j+1 (17) Applying these discretization formulae, we can discretize Saint-Venant equations in Eqn. (13). At each time point n + 1, the discretized continuity and dynamic equation specify a nonlinear relationship between unknowns (Q and A) at time point n + 1, as well as at time point n. 6 3.2 Nonlinear Equation Solver One of the most popular method to solve nonlinear equations is the Newton-Raphson’s method. To compute the solution to a nonlinear problem F(x) = 0, the method iteratively improve solution x from a starting point x0 by using the local gradients: xk+1 = xk − α · ( ∂F −1 ) ·F ∂x (18) where α is a damping factor. Although Newton-Raphson’s method is easy to implement, some cautions need to be exercised. One issue is the convergence. Since Newton-Raphson’s method is a local search algorithm, it may fail to converge. In transient simulations this is rarely an issue because the solution at time n + 1 is close to the solution at time n, which can be used as the starting point. However, for DC solve, convergence strongly depends on the selection of initial starting points; hence convergence could fail. In SPICE, this problem is partially solved by using a bounding algorithm to limit the operating range change of nonlinear devices. For difficult circuits, more advanced homotopy methods are still required circuits[27]. For river flow simulation, again this issue mainly arises for a steady-state solution with limited initialization information. We address this issue by first solving the Saint-Venant equations approximately, so that we can place a reasonable starting point for iteration. Secondly we implement a bounding algorithm based on fluid dynamics principles to limit the solution range. The combination of the two approaches provides good results. Another issue with Newton-Raphson’s method is that the functions have to be C 1 (i.e., the functions should be continuous themselves and also have continuous first-order derivatives with respect to unknowns). Depending on the complexity of the river cross-section shape, this is not always the case. We address this issue by approximating the river channel data into a C 1 function template. The approach not only solves the function continuity issues, but also help us to achieve considerable speed-up in the simulation. As shown in Eqn. (18), Newton-Raphson’s method requires the factorization of the Jacobian matrix ∂F ∂x . The size of the matrix is 2m where m is the number of the computational nodes, which can be quite large for a continental-scale river network. However, as we have observed in SPICE, the runtime complexity of factorizing a sparse matrix is usually O(N 1.2 ). In other words, it is only slightly super linear, versus the cubic complexity of factorizing dense matrix. It can be shown that the stencil of the Jacobian matrix in Saint-Venant equations using the four-point discretization scheme is only 4. Given the continuing rapid improvement of computer hardware and the efficiency of today’s sparse linear matrix packages such as [28][29], the capacity required and the runtime for the factorization of the Jacobian matrix is not a concern. 3.3 General Simulation Flow To illustrate the simulation flow, we use the symbolic three-branch river network in Fig. 2. The river network is first partitioned into computational nodes indicated by red dots. At each computational node, the river cross section description is used as input, along with the friction coefficients (Manning’s n) and the bottom slope S0 . Next the computational nodes are connected from upstream to downstream following discritized Saint-Venant equations. At the junction point of the three branches, two linear relationships are specified. The first equation is mass conversation: the sum of the flow rate of two tributaries should equal the flow rate 7 Qs ql Qs ql ql h Figure 2: Illustration of computational nodes on a river network the downstream stem. Another linear relationship is used to specify the relative contributions of two upstream branches to the downstream branch. Boundary condition flow rates are the forcing input Qs at two upstream points, as well as the possible lateral inflows along the channels. A downstream boundary condition in the form of the depth at the most downstream node is applied. This value is typically the depth where a river enters a lake or ocean. These procedures are used to build an entire river network with any number of tributary branchings. Once the network is constructed, we first perform steady-state solution with fixed forcing terms. We then apply the time-varying forcing terms to compute unsteady solutions. The solution provides the depth and flow velocity at every computational node for every time point in the simulation. 3.4 SPINT Implementation SPRINT is implemented in C++ in a modular design approach. Besides the nonlinear and matrix solution methods described in the previous subsections, it also has many functionalities to enhance runtime performance and robustness, as well as other facilities such as model pre-processing and topological checking of the river networks. To facility the deployment in a cloud-based simulation environment, SPRINT also has an http front-end which enables remote simulation. 4 Experimental Results In this section, we present some experimental results of SPRINT. 8 4.1 Comparison with Analytical Solutions In this experiment, a river segment with rectangular cross section is used. Due to its simplicity, the steady-state solutions can be calculated analytically, which are compared with the steady-state solutions computed by SPRINT. The depth at each node is plotted in Fig. 3. The difference is negligible at the resolution of the graph. 20 Analytical Solution Simulation 19 Flow Depth (m) 18 17 16 15 14 13 0 50 100 150 200 250 300 Distance from pour port (km) 350 400 450 500 Figure 3: Comparison between SPRINT output and analytical solutions. Y-axis represents depth. 4.2 A Small Creek in Central Texas This experiment is a creek in central Texas. It has three branches with the total length of approximately 7.2 miles. The simulation takes merely a few seconds on a common desktop computer for a 5-day event. Fig. 4 shows the simulated results at a computational node where observed gauge data are available. Note that we haven’t gone through detailed calibration process so the simulated results will not match the observed depth data exactly. However, all the key waveform characteristics of the observed data are captured by the simulation results. 4 observed model output 3.5 3 2.5 2 1.5 1 0.5 0 5 10 15 20 25 30 35 40 Figure 4: Comparison between SPRINT output and observed depth data. Note no calibration was performed. 9 4.3 A River Basin in Central Texas This experiment is a relatively large river basin in central Texas. It consists of over 3, 500 river branches with the total length of over 9, 000 miles. The whole river network is modeled by over 110K nodes and the simulation is performed on a regular desktop computer. It takes about an hour to simulate a 12-day event, which translates into about 100× speedup of real-time. Three snapshots of the simulated results are presented in Fig. 5. From the top figure, one can see the approximated diagram of the river networks. Note that many small tributaries are not included in the figure to avoid over crowding the graph. The width of each segment approximates the flow rate, while the color represents relative depths. Figure 5: A river basin in central Texas. Three snapshots show the simulation output at different time points. The width represents the relative flow rate in the river branches. The color represents the relative depth. 10 5 Final Remarks SPRINT represents an inter-discipline collaboration between computer engineering and civil engineering hydroscience. Although its algorithms cannot be directly applied, SPICE has been influential on the design of SPRINT, particularly in terms of design principle and design philosophy. Experimental results show that our tool can achieve considerable capacity and performance. We hope our effort will be beneficial to hydrology and hydraulic modeling community, just as SPICE has long lasting impact on the semiconductor industry. References [1] L. W. Nagel and R. A. Rohrer, “Computer analysis of nonlinear circuits, excluding radiation,” IEEE Journal of Solid-State Circuits, vol. 6, pp. 166–182, Aug. 1971. [2] L. W. Nagel and D. O. Pederson, SPICE: Simulation Program with Integrated Circuit Emphasis. University of California, Berkeley, 1973. [3] H. K. Gummel and H. C. Poon, “An integral charge control model of bipolar transistors,” Bell System Technology Journal, pp. 827–852, 1970. [4] B. J. Sheu, D. L. Scharfetter, P.-K. Ko, and M.-C. Jeng, “BSIM: Berkeley short-channel IGFET model for MOS transistors,” IEEE Journal of Solid-State Circuits, vol. 22, pp. 558–566, Aug. 1987. [5] G. D. Hachtel, R. K. Brayton, and F. G. Gustavson, “The sparse tableau approach to network analysis and design,” IEEE Transactions on Circuit Theory, vol. 18, no. 1, pp. 101–113, 1971. [6] C.-W. Ho, A. E. Ruehli, and P. A. Brennan, “The modified nodal approach to network analysis,” IEEE Transactions on Circuits and Systems, vol. 22, no. 6, pp. 504–509, 1975. [7] C. W. Gear, Numerical initial value problems in ordinary differential equations. Englewood Cliffs, NJ: Prentice-Hall, 1971. [8] L. W. Nagel. Personal Communication, 2011. [9] L. T. Pillage, R. A. Rohrer, and C. Visweswariah, Electronic circuit and system simulation methods. McGraw-Hill, 1995. [10] J. Ogrodzki, Circuit simulation methods and algorithms. CRC Press, 1994. [11] F. N. Najm, Circuit simulation. Wiley, 2010. [12] Cadence Corporation, “Virtuoso spectre circuit simulator.” http://www.cadence.com, 2012. [13] Mentor Graphics Corporation, “Eldo classic.” http://www.mentor.com, 2012. [14] Synopsys Corporation, “HSPICE.” http://www.synopsys.com, 2012. [15] J. V. Clark, N. Zhou, and K. S. J. Pister, “MEMS simulation using SUGAR v0.5,” in Proc. Solid-State Sensors and Actuators Workshop, pp. 191–196, 1998. 11 [16] “Bio-SPICE.” http://www.biospice.org, 2012. [17] US Army Corps of Engineers. http://www.hec.usace.army.mil/software/hec-ras/, 2011. [18] J. A. Cunge, F. M. Holly, and A. Verwey, Practical aspects of computational river hydraulics. Boston MA: Pitman Publishing Ltd, 1980. [19] R. Manning, “On the flow of water in open channels and pipes,” Transaction of the Institute of Civil Engineers, Ireland, vol. 20, pp. 161–209, 1891. [20] V. I. Koren and L. S. Kuchment, “Numerical integration of de Saint Venant equations with explicit schemes when computing the unsteady flow in rivers,” Trudy G.N.I., no. 8, 1967. [21] P. D. Lax and B. Wendroff, “Systems of conservation laws,” Comm. on Pure and Applied Math., vol. 13, pp. 217–237, 1960. [22] D. D. Houghton and A. Kashahara, “Nonlinear shallow fluid flow over an isolated ridge,” Comm. on Pure and Applied Math., vol. 21, 1968. [23] J. Burguete and P. Garcı́a-Navarro, “Improving simple explicit methods for unsteady open channel and river flow,” International journal for numerical methods in fluids, vol. 45, no. 2, pp. 125–156, 2004. [24] R. Courant, K. Friedrichs, and H. Lewy, “Über die partiellen Differenzengleichungen der mathematischen Physik,” Mathematische Annalen, vol. 1, no. 100, pp. 32–74, 1928. [25] M. B. Abbott and F. Ionescu, “On the numerical computation of near horizontal flows,” Journal of Hydrolic Research, vol. 5, no. 2, pp. 97–117, 1967. [26] A. Preissmann, “Propagation des intumescences dans les cannaux et rivièrrs,” in Congrès de l’Assoc Française de Calcul, pp. 433–442, 1961. [27] J. Roychowdhury and R. Melville, “Delivering global DC convergence for large mixed-signal circuits via homotopy/continuation methods,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 25, no. 1, pp. 66–78, 2006. [28] I. S. Duff, “MA28: a set of Fortran subroutines for sparse unsymmetric linear equations,” Harwell Report AERE-R 8730, Atomic Energy Research Establishment Report, 1980. [29] T. A. Davids, “Algorithm 832: UMFPACK v4.3 – an unsymmetric-pattern multifrontal method,” ACM Trans Mathematical Software, vol. 30, pp. 196–199, June 2004. 12