Focus Area 4: Simulation of Multi-scale, Multi-physics,
Heterogeneous Systems
Research Background and Significance
CFSES scientists and engineers pool their knowledge and resources by forming
collaborative research partnerships. Supporting these partnerships, and in many ways
making them possible, are mathematical modeling and computer simulation. In fact,
predictive computational simulation may be the only practical means available for the
human mind to grasp the complexities of the subsurface environment.
Focus Area 4 develops cutting-edge techniques for simulating complex processes in
underground reservoirs and aquifers that govern the disposal of CO2, radionuclide wastes,
and other byproducts of energy production. An overarching principle in the research is to
bridge the scales in both space and time from fine molecular scales to the basin scale for
the multitude of physical processes that affect CO2 and nuclear waste movement,
reaction, and storage.
The basin scale is the coarsest, and it is the scale on which the simulation must be made.
A typical reservoir or aquifer is extremely large in space. It may cover an area the size of
several counties or even an entire state. In time, migration of CO2 may last hundreds of
years, and nuclear contaminants millions of years.
The physical, chemical, and even biological processes within the subsurface operate over
sub-millimeter space and sub-second time scales. These fine-scale processes have
profound effects on the large and long-time basin-scale behavior, so they must be
modeled so that they can be accurately simulated on the coarse basin-scale. The world’s
fastest supercomputers, even well into the foreseeable future, are inadequate to resolve
directly these fine scales over the basin-scale. Thus Focus Area 4 uses insight gained
from the other Focus Areas operating on finer scales and develops upscaled
computational technologies suitable for coarse, basin-scale simulation that bridge the
multiple scales inherent in the problem.
Illustration 1: Water phase saturations (top figures) and CO2 concentration (bottom
figures) profiles at 50 and 100 days. The CO2 moves in complicated and unexpected
ways.
2
Illustration 2: Movie simulation of a CO2 leak through an abandoned well. Deep saline
aquifers are considered to be one of the most important formations because of their
worldwide availability and generally low economic value. It is attractive to use the
existing infrastructure of oil and gas wells to inject the CO2 into a formation. However,
formations that have been explored for oil and gas production are typically perforated by
a large number of wells, a fact that increases the risk of leakage through such wells. A
simple leakage scenario involving one CO2 injection well, one leaky well, two aquifers
and an aquitard are shown. The leaky well connects the two aquifers. As CO2 is injected
underground in to the lower reservoir, it rises due to buoyancy. When it finds the old
abandoned wellbore, it rises through it, contaminating the upper reservoir. Simulation
helps scientists and engineers visualize and study such undesirable scenarios, so that they
can devise prevention or response strategies.
3
Illustration 3: Movie simulation showing fluid-induced dynamic fracturing around a
borehole. Such fracturing affects the flow of fluids and well-bore stability. Fracturing
of the cap rock could create pathways for CO2 leakage.
4
Research Questions
1. How can we conduct simulations that capture the essential features of basin-scale
behavior that emerge from fine-scale phenomena, such as flow, transport,
reaction, and deformation in the Earth’s subsurface, without resolving all finescale features? In many ways, this is our first fundamental research question.
2. Can we devise accurate models and robust numerical algorithms for simulating
events that are driven by far-from-equilibrium conditions? This is our second
fundamental research question.
3. What are appropriate basin-scale reaction dynamic parameters, and how do they
change in time as reservoir conditions change? Reactions take place on the
molecular scale, but we cannot resolve individual molecules in a basin-scale
simulation. In fact, computational limitations dictate that we must average all the
molecules within perhaps a 10-meter3 volume, and yet still account for the fact that
most of them are not in contact with each other to react. A particular challenge is to
understand reactive mineral surface-area factors and how they change as diagenesis
proceeds along a reaction front.
4. How do coarser-scale interactions such as changing temperature, or nutrient
introduction via fracture flow, influence the growth of microbes? How does
microbial growth feed back to change basin-scale properties, such as
permeability, surface chemistry, reactive surface area, and microbe-mineral
interactions? Coarse scale phenomena affect fine-scale processes, which in turn
feed back to modify coarse-scale phenomena. These must be modeled carefully, or
erroneous conclusions may be drawn.
5. How can the averaging that underlies a continuum basin-scale model be
improved to account for underlying spatially correlated properties? The
subsurface environment is highly heterogeneous, meaning that the physical
properties of rocks change greatly from one point to another over fine-scales.
Again, these fine scales cannot be resolved by the fastest supercomputers, and some
upscaling is required.
6. How do we reliably model changes in mechanical properties of rocks due to
chemical interactions between the injected fluid and rock mass, and how do we
incorporate such transient rock property models in accurate geo-mechanical
models for fracture initiation and propagation? Compaction of the reservoir rock
and the overlying cap rock seal integrity are key issues in the sequestration of CO2.
7. Can we design more efficient computational algorithms for solving the model
equations, so as to take full advantage of modern supercomputer technology?
Modern computers achieve high rates of computational power by working in
parallel. New algorithms are needed to coordinate the computations of thousands of
small processor nodes to keep each productively employed at all times during the
simulation.
8. How do we assimilate seismic and other measured data into high-resolution
models for simulating CO2 storage? During a CO2 injection process, data will be
continually collected. It must be compared to simulation predictions to refine and
improve the underlying model so it is both consistent with measured data and a
more reliable predictive tool.
5
9. How do we represent and quantify uncertainty as to the fate of injected fluids in
the subsurface due to approximate modeling, data measurement errors, and
incomplete geologic characterization? The answer to this question is needed to
assess risks to human health and the environment associated with human interaction
with geo-systems.
Research Innovations and Goals
The main goal of research Focus Area 4 is to develop cutting-edge techniques for
accurate and reliable simulation of CO2 sequestration and other energy waste storage
techniques in the subsurface at reservoir and basin scales. The work involves:
1. Developing new and improved models of tightly interconnected physical
processes, including multi-phase compositional flow, reactive transport, the
behavior of multiple phases, hysteretic relative permeability and capillary
pressure relations, and geo-mechanics;
2. Coupling multi-physics processes across spatial and temporal scales, including
innovative multi-scale techniques, multi-physics couplings of physical processes,
and multi-scale time-stepping and linear solver techniques;
3. Uncertainty quantification and validation, including more efficient data
assimilation and uncertainty assessment tools, and characterization and validation
against data from actual field sites.
Expected Outcome
CFSES will develop an underlying scientific knowledge base and prototypical simulator
codes. These will facilitate the science and engineering community in developing the
technology needed to reliably predict, control, and manage human interaction with geosystems. Through computational simulation, it is possible to assess the efficacy of storage
designs and operation protocols in a virtual world. Failure in this context causes no
damage to the environment. By simulating various scenarios, uncertainty and risk to
human health can also be quantified and minimized. Once proper designs and protocols
have been developed, they can be implemented to achieve the societal objective: the safe
and long-term storage of CO2 or nuclear waste. Spin-off of the technology might be used
to protect underground water resources from compaction and contamination, to enhance
fossil fuel production, and to foster groundwater sustainability.
6
7
Highlights: Recent Results
Meshing and Discrete Approximations
At the University of Texas at Austin, research groups in the Bureau of Economic
Geology (BEG) conduct laboratory and field experiments at CO2 injection demonstration
sites. Collaboration with these groups provides data for calibration, verification, and
validation of models. Figure 1 shows a hexahedral mesh from the BEG Frio Pilot Site.
This mesh has elements with non-planar faces, some of which are highly distorted, that
represent geological structures such as layers, faults, and pinch outs.
Figure 1: An illustration of the Frio Pilot CO2 injection test site. The grid on the right is
of very poor quality for numerical simulation.
Geological data describing reservoirs and aquifers is often poorly suited for numerical
simulations. The goal of the gridding effort is to transform the data to improve the quality
of the numerical simulations. The focus has been in describing the given geometry by a
grid or mesh of points in a more accurate way and with fewer mesh points. At the same
time, each individual mesh cell or element is kept as nearly as possible to a cube, as
8
illustrated in Figure 2. All this translates into shorter simulation times and more accurate
results.
Figure 2: On the left is the original data from the Frio Pilot Test field site. It has 100,000
elements. In the middle is a straightforward but poor quality computational mesh with
26x26x21 elements. On the right is an improved mesh with 18x24x3 elements. The mesh
on the right is created with the GridPro mesh generator.
Standard methods, such as mixed finite element methods, allow us to approximate the
pressure field within the reservoir. The gradient of the pressure field is proportional to
the fluid velocity, which determines where CO2 migrates. On general hexahedral meshes,
standard methods may not converge to the true pressure distribution as the mesh spacing
is refined. In a recently developed cell-centered numerical method, CFSES researchers
have no convergence issues for general meshes. This new computer model gives accurate
prediction of CO2 flow in deep saline aquifers, which is crucial in preventing the leakage
of CO2.
9
A Multi-scale Preconditioner
The simulation of complex subsurface phenomena such as CO2 sequestration requires
highly specialized solution techniques, leveraging high performance computation on
massively parallel supercomputers. In order to minimize computation time and maximize
solution accuracy, it is advantageous to allow portions of the domain to contain different
types of physics, scales, and numerical methods. For example, a compositional flow
model in a salt dome could be coupled with a basin scale geo-mechanical model to assess
the feasibility of long-term carbon storage. To meet these goals, a computational
framework has been developed that uses non-overlapping domain decomposition to allow
multi-scale and multi-physics coupling using mortar finite elements. Our approach is
referred to as the multi-scale mortar mixed finite element method.
This approach is both efficient and flexible: A reduced number of unknowns are
consolidated on the interfaces and the resulting algebraic system is solved with an
iterative method. In this way, the interface operator need not be formed since only its
action is needed; each interface iteration requires the solution of sub-domain problems
that can be performed in parallel. The solution can be resolved to a very fine level of
detail around critical areas of interest, as well as adaptively resolved using mesh
refinement approaches. Moreover, the physical models can also be placed in different
areas throughout the domain where they are deemed necessary.
A new type of preconditioner has recently been developed to improve the efficiency of
the multi-scale mortar method, known as the frozen Jacobian multi-scale preconditioner.
Multiphase flow simulations are highly nonlinear, transient systems, which are typically
solved using a Newton-Krylov algorithm. The basic idea of the preconditioner is to precompute the solutions to several sub-domain problems with various boundary conditions
in parallel for a fixed state of the system, meaning for a fixed Jacobian and time step.
These form what is known as a multi-scale basis, which can be used to approximate the
action of the interface operator for several time steps. As the dynamics of the simulation
change, the effectiveness of the preconditioner degrades in time. Our results demonstrate
that the multi-scale preconditioner can be recomputed sparingly throughout the
simulation to balance the overhead of its construction with its potential reduction in
interface iterations.
10
Figure 1: Oil concentration in a two-phase multi-scale water flood simulation, using two
sub-domains connected with high order mortar elements at their interface. It is a 400
day simulation where a heterogeneous reservoir is initially saturated with an oil phase,
water is injected in the upper left corner, and oil is produced in the lower right corner
from an aerial perspective.
Figure 2: The effectiveness of the multi-scale preconditioner at reducing the amount of
work necessary to solve this problem. The graph represents the number of interface
iterations per time step, where the red curve is unpreconditioned, the blue curve is with a
preconditioner computed at the initial time, and the black curve is where the
preconditioner is recomputed periodically throughout the simulation. In this example,
the use of the multi-scale preconditioner directly translated to a reduction in runtime by
over 40% over the original approach.
11
Dynamically Coupled Flow and Mechanics for Geo-systems
Flow through porous materials is coupled to solid mechanics by virtue of Terzaghi’s
principle of effective stress, wherein stresses in porous materials are borne by both pore
pressure and the stress field in the solid skeleton. Coupled flow and mechanics in
heterogeneous media is of particular importance to simulation of subsurface activities
related to US energy security, including, for example, studies of geologic sequestration of
anthropogenic CO2 and geologic disposal of heat-generating high-level nuclear waste.
We are developing simulation tools to enable modeling of multiphase, multi-component
flow in heterogeneous media (see Figure 1) and of nonlinear structural response of geomaterials. We have recently developed a new computational capability for coupling of
multiphase, multi-component porous flow with nonlinear geo-mechanics. The geomechanics and porous flow models exist as separate, stand-alone codes, and are coupled
via a solution control approach. Geo-mechanics is coupled to flow via the variation in the
fluid pore pressures, whereas the flow problem is coupled to mechanics by the
concomitant material strains which alter the pore volume (porosity field) and hence the
permeability field. To facilitate coupling with disparate flow and mechanics time scales,
the coupling strategy allows for different time steps in the flow solve compared to the
mechanics solve. If time steps are synchronized, the controller allows intra-time-step
iterations. The coupling is dynamically controlled by monitoring a norm measuring the
degree of variation in the deformed porosity, thereby controlling the frequency of
mechanics solves compared to flow solves. The implementation was verified by solving a
subsidence problem discussed by Dean et al., (SPE-79709, 2003), as depicted in Figure 2
below.
Leakage Curves
Figure 1: On the right is an illustration of CO2 injection into one realization of a
heterogeneous aquifer, which contains an abandoned well. On the left is the distribution
of leakage as a percentage of the injection rate for several realizations.
12
Figure 2. Comparison of the time history of subsidence due to production in a porous
layer, under two levels of geo-mechanics/flow coupling, with the solution of Dean et al.
(SPE-79709, 2003). Also shown is the final porosity distribution and (exaggerated)
subsidence on a cross section through the layer.
13
Discrete fracture generation
A new computational method for modeling fluid-induced discrete-fracture generation and
growth has been developed. The method is based on modeling the solid continuum with
polyhedral finite elements obtained from a randomly close-packed Voronoi tessellation
(see Figure 1). Each cell of the Voronoi mesh is formulated as a conforming
displacement-based finite element.
New cohesive fracture surfaces are allowed to
nucleate only at the inter-element faces of the Voronoi cells. New fracture surfaces are
inserted by creating new nodes during the simulation. The RCP Voronoi mesh possesses
a number of advantageous properties including large included angles within the cells
(finite-element), and a space of statistically isotropic crack paths. The a priori crack paths
of the RCP Voronoi mesh are viewed as instances of realizable random crack paths
within a random field representation of the continuum material properties. Fluid flow
within the fractures is modeled using a Reynold’s lubrication theory. An example
simulation showing fluid-induced fracturing around a borehole is shown in Figure 2.
Figure 1: Randomly close-packed Voronoi tessellation (finite element mesh).
Figure 2: A finite element simulation showing fluid-induced fracturing around a
borehole. The maximum principal stress field is shown.
14
Parameter Estimation
The subsurface geological formations are subject to high uncertainty because of limited
knowledge of rock properties. Therefore, consideration of the uncertainty calls for a
stochastic description of the underground formations that necessitate using multiple
realizations for uncertainty quantification. The prior uncertain geological models (model
parameters) are obtained by integration of the static data from different sources such as
well logs, core sample, and 3D seismic data. This prior uncertainty in model parameters
is reduced by assimilation of dynamic (observed) data.
Our main focus has been on developing a general parallel framework for sequential data
assimilation with implementation of ensemble Kalman filter (EnKF) algorithm. The
effects of incorporation of dynamic data on calibration of model parameters (logpermeability) and saturation distribution for a synthetic case are studied (Figure). The
assimilation of the observed data results in better model parameters and improves
saturation distribution compared to the reference case. We will investigate these effects
for more realistic cases such as Frio field.
Figure: Horizontal log-permeability fields (left column) and CO2 saturation distribution
after 1000 days (middle column) and after 2000 days (right column). The first row is the
reference (true) case, the second row is from the prior model (our guess), and the third
row shows the final results after assimilation of all dynamic data.
15
Convergence of Monte-Carlo Uncertainty Estimation
A statistical method has been developed for verifying the grid or mesh convergence of a
sequence of statistical distributions generated by direct Monte Carlo sampling of
stochastic partial differential equations. Example systems include those from fluid or
solid mechanics, particularly those with instabilities and sensitive dependence on initial
conditions or system parameters. The convergence assessment is based on demonstrating
empirically that a sequence of cumulative distribution functions converges in the Lnorm. The effect of finite sample sizes is quantified using confidence levels from the
Kolmogorov-Smirnov statistic. The statistical method is independent of the underlying
distributions. The figure shows various statistical quantities used in the method:
continuous cumulative distribution functions, their samples and resulting distances. This
statistical method will help validate computational models of complex systems subjected
to uncertain system parameters.
Figure: Two cumulative distribution functions (black), the cumulative distribution
functions of their samples (red), and their confidence bounds (blue) along with their L 
distances (arrows).
16
Phase Behavior and Physical Properties
The Peng-Robinson cubic equation of state (EOS) is used for phase behavior of binary
system of CO2 and water as a function of pressure and temperature followed by the flash
algorithm to determine the mole fractions of CO2 and water in two equilibrium phases.
The CO2 module of IPARSv-3 is non-isothermal compositional EOS and is coupled with
biogeochemical reactions. Recent developments to improve the phase behavior and
petrophysical properties are outlined as:
 An additional species in the aqueous phase is added to model the brine salinity
expressed as total dissolved solids. The EOS variables of binary interaction
coefficients and volume shift parameters are then modified according to the salt
concentration and temperature using published correlations. These correlations
proved to give more accurate CO2 solubility in brine and brine density.
 Published correlation for interfacial tension (IFT) between water and supercritical
CO2 is implemented that accounts for effects of pressure, temperature, and brine
salinity on IFT.
 We have incorporated a scaling group (trapping number) that combines the forces
of gravity, viscous, and capillarity that controls the flow or trapping of both water
and CO2 phases.
 Relative permeabilities and capillary pressures are adjusted as a function of
trapping number because of the shift in trapped phase saturations.
1
Bennion and Bachu, 2006
0.9
water- 1378 kPa
0.8
Gas-1378 kPa
Relative Permeability
water- 6890 kPa
0.7
Gas-6890 kPa
water- 20000 kPa
0.6
Gas-20000 kPa
Brine
0.5
0.4
CO2
0.3
0.2
0.1
0
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Gas Saturation
Figure: Measured water/CO2 relative permeability curves at different pressures that
cause the shift in interfacial tension (data from Bennion and Bachu, 2006)
17
Solvers
The modeling of CO2 sequestration requires the coupling of compositional flow, thermal
effects, geo-mechanics, and transport of multiple reactive chemical species at multiple
temporal and spatial scales. Moreover, unstructured gridding is generally needed for
treating geological heterogeneities. Thus the development of accurate, efficient and
robust solvers coupled with adaptive time stepping for solving the large nonlinear
dynamical systems that arise from finite element discretizations represents a formidable
challenge.
For many years, researchers from Focus Area 4 have been investigating algorithms for
linear and nonlinear solvers to address this "bottleneck" of subsurface flow computations.
Some of these efforts include fast banded-solvers, Krylov subspace methods with a
recycling strategy, two-stage preconditioning, algebraic multigrid, and specialized
domain decomposition methods.
Collaboration with the SciDAC Towards Optimal Petascale Simulations Center at Sandia
National Laboratories has started recently. The goal is to implement an interface between
the Trilinos library developed at Sandia and the CO2 simulator used in Focus Area 4
allowing to explore new software tools aimed at maximizing the solver performance and
parallel scalability on emerging architectures.
18
High Performance Computing
Modeling of the long-term movement of CO2 will require assimilation of huge datasets
into simulators that incorporate complex physical and chemical processes such as
coupled multiphase flow with chemical transport and geo-mechanics. Therefore,
performing a large-scale simulation run will take significant amount of time. Also, we
require multiple simulations to quantify the uncertainty in the model parameters and
prediction performances. Thus, use of high performance computing is advantageous in
reduction of time cycle for reservoir characterization.
In our modeling experiments, the main simulation tool is the Integrated Parallel Accurate
Reservoir Simulator (IPARS). IPARS provides multi-scale and multi-physics capabilities
to model multiphase and multi-component (compositional) flow in porous media coupled
with geo-mechanics and reactive transport. It is easy to implement IPARS in a highperformance, parallel computational framework. We have developed a two-level parallel
EnKF framework in which multiple realizations are spawned off in parallel on several
partitions of a parallel machine (cluster) each of which are further sub-divided among
different nodes (processors) and communication performed at data assimilation time,
between the partitions before proceeding again to next assimilation step. Our results,
depicted in the Figure, show that the high-performance computing process (parallel
EnKF) significantly reduces the computation time.
Figure: Parallel speedup as a function of number of processors with 150 realization
19
The Senior Research Team
The University of Texas at Austin
1. Todd Arbogast, PhD, Professor, Department of Mathematics (Focus Area 4 coleader)
2. Mojdeh Delshad, PhD, Research Professor, Department of Petroleum &
Geosystems Engineering
3. Susan Hovorka, PhD, Senior Research Scientist, Bureau of Economic Geology
4. Gergina Pencheva, PhD, Research Associate, Institute for Computational
Engineering and Sciences
5. Mary F. Wheeler, PhD, Virginia Cockrell Chair in Engineering Professor,
Department of Aerospace Engineering & Engineering Mechanics and Department
of Petroleum & Geosystems Engineering
Sandia National Laboratory
1. Joseph E. Bishop, PhD, Principal Member of the Technical Staff, Engineering
Sciences Center (Focus Area 4 co-leader)
2. Mario J. Martinez, PhD, Principal Member of the Technical Staff, Engineering
Sciences Center
3. C. Michael Stone, PhD (Retired)
Post-doctoral Associates
1.
2.
3.
4.
Ben Ganis, PhD.
Mika Juntunen, PhD.
Reza Tavakoli, PhD.
Guangri (Gary) Xue, PhD.
Graduate Students
1. Nick Alger
2. Mohammad Reza Beygi
3. Horacio Florez
4. Omar al Hinai
5. Zak Kassas
6. Xianhui Kong
7. Zhen (Jane) Tao
8. Bin Wang
9. Hailong Xiao
10. Changli Yuan
Events


Drs. Daniil Svyatskiy
, Mary F. Wheeler
, Guangri Xue
, and Ivan Yotov
organized the mini-symposium “Multiscale Methods for Porous Media
Applications” for the 2011 SIAM Conference on Mathematical & Computational
Issues in the Geosciences, March 20-24, 2011.
Drs. Anozie Ebigbo
, Rainer Helmig
, Mary F. Wheeler
, and Guangri
Xue organized the mini-symposium “Large Scale Simulations and Porous Media
Applications” for the 2011 SIAM Conference on Mathematical & Computational
20


Issues in the Geosciences, March 20-24, 2011.
Drs. Mary F. Wheeler, Todd Arbogast, and Mojdeh Delshad hosted the Annual
Center for Subsurface Modeling (CSM) Industrial Affiliates Meeting, October 2627, University of Texas at Austin.
Drs. Mary F. Wheeler, Todd Arbogast, Mojdeh Delshad, and Ian Duncan hosted
the workshop: “The Role of Computation in Protecting the Environment: A
Workshop on Carbon Sequestration Simulation for High School Mathematics and
Science Teachers,” June 15-16, 2010, University of Texas at Austin.
News


Professor Mary F. Wheeler’s research was featured on the University of Texas at
Austin Cockerel School of Engineering web site in the article “Engineer Reduces
Big Problems To Manageable Numbers.”
Professor Mary F. Wheeler was elected a fellow of the American Academy of
Arts and Sciences in 2010.
21