eScience Simulation of Clinic Electrophysiology in 3D
Human Atrium
Sanjay Kharche1, Gunnar Seemann2, Lee Margetts3, Joanna Leng3, Arun V Holden4, and
Henggui Zhang1
1
School of Physics and Astronomy, University of Manchester, Manchester, M60 1QD, UK
2
Institute of Biomedical Engineering, University of Karlsruhe (TH), Karlsruhe, Germany
3
Directorate of Information Systems, University of Manchester, Manchester, M13 9PL, UK
4
Institute of Cell Membrane and Systems Biology, University of Leeds, Leeds, LS1 9JT, UK
Abstract
Atrial fibrillation (AF) is a common cardiac disease with high rates of morbidity, leading to major
personal and NHS costs. Computer modeling of AF using a detailed cellular model with realistic
3D anatomical geometry allows investigation of the underlying ionic mechanisms in far more detail
than in a physiology laboratory. We have developed a 3D virtual human atrium that combines
detailed cellular electrophysiology, including ion channel kinetics and homeostasis of ionic
concentrations, with anatomical detail. The segmented anatomical structure and multi-variable
nature of the system makes the 3D simulations of AF large and computationally intensive. The
computational demands are such that a full problem solving environment requires access to
resources of High Performance Computing (HPC), High Performance Visualization (HPV), remote
data repositories and a backend infrastructure. This is a classic example of eScience and Gridenabled computing. Initial work has been carried out using multiple processor machines with shared
memory architectures. As spatial resolution of anatomical models increases, requirement of HPC
resources is predicted to increase many-fold (~ 1 – 10 teraflops). Distributed computing is
essential, both through massively parallel systems (a single supercomputer) and multiple parallel
systems made accessible through the Grid.
1. Introduction
AF is a condition in which the upper chambers
of the heart contract at a very high rate and in a
disorganized manner. As a result, the pulse of
AF patients is highly irregular. AF affects about
4% of the population over the age of 60 years.
Mortality rate associated with AF has increased
over the past two decades. In the US, the age
standardized death rate (per 100,000 US
population) increased from 28 to 70 in 1998 [1].
In the UK alone, around 500,000 patients are
affected by AF and the mortality rate has
doubled since 1993. AF also increases the risk
of stroke and heart failure among other
complications.
The electrophysiology of the atrium, as in
other cardiac tissue, is extremely complex. The
membrane potential is determined by the
integrated actions of ionic channels, Na/Ca
exchanger and Na/K pump currents. In addition,
regulative
mechanisms
governing
the
homeostasis
of
intracellular
calcium
concentration also affects the membrane
potential. Modeling such a system requires use
of high order, stiff ordinary differential
Figure 1. 3D anatomical model of human
female atrium showing PM (blue), CT (red), RA
(purple), LA (green). SAN and BB are
embedded close to the CT and cannot be seen
equations. For example, the Courtemanche et al.
[2] (CRN) model for human atrial cells consists
of 24 state-variables of ODE. Such a description
is necessary for the simulation of the effects of
drugs, age and genetic disorders on ionic
channel kinetics and thus single cell and tissue
electrical activities. Simple models, e.g.
FitzHugh-Nagumo (FHN) excitation model,
A
2. Heterogenities in human atrial AP
and anatomy
20
mV
0
atrium
pm
CT
appendage
AV ring
-20
-40
-60
-80
200
300
400
500
600
700
800
t (ms)
B
1200
800
[Ca2+]i
atrium
pm
ct
appendage
AV ring
400
0
200
220
240
260
t (ms)
Figure 2. (A) AP heterogeneity in human
atrium. LA, RA, PM and BB have an APD90 of
325 ms (solid blue), PM (dash dot-doted black),
CT is 330 ms (dashed green). In the atrial
appendage the APD90 is known to be 310 ms
(dotted red), in the AV ring is 300 ms (dash
dotted cyan). (B) Intracellular calcium
concentrations associated with APs in A. Color
coding and dash styles are as in A.
allow fast prototyping and (relatively)
computationally inexpensive alternative for
simulating electrical propagation in 3D
geometries. Such models can be used as a
starting point for the insertion of biophysically
detailed cell models into anatomically detailed
tissue models.
The human atrium consists of left atrium
(LA), right atrium (RA), pectinate muscles
(PM), cristae terminalis (CT), the Bachmann’s
bundles (BB), and sino-atrial node (SAN).
Figure 1 shows a composite of all these
components. The SAN is the pacemaker of the
heart that initiates electrical action potential
(AP). PM, CT and BB assist in conduction of
electrical excitation waves. The human atrium is
a heterogeneous tissue with cells in different
regions having different action potential
characteristics,
preferential
conduction
pathways and varied fibre orientations.
The CRN model is able to simulate AP in
generic human atrial myocyte. However, the
electrical activity in atrium is heterogeneous.
The human atrium consists of several
distinctively different regions with cells in each
of the regions having different AP
characteristics. The LA, RA, PM and BB have
similar APs with an action potential duration
(APD90) of approximately 325 ms. However, in
the CT the APD90 is 330 ms, in the atrial
appendage the APD90 is known to be 310 ms, in
the atrioventricular (AV) ring is 300 ms. The
CRN model can be adapted by adjusting
maximal conductances of the transient outward
current (Ito), the L-type calcium current (ICaL),
and the rapidly activated potassium current (IKr)
to simulate these heterogeneous APs [3] based
on experimental data [4]. The results are shown
in Figure 2, along with the corresponding
heterogeneous calcium transients. Although
there is heterogeneity in AP and calcium
transients, the resting potential remains almost
the same (-81.5 mV), as do the upstroke
velocity (220 mV/ms) and the peak potentials
(25 mV).
The SAN is a pacemaker, and is autorhythmic, with a period of approximately 70 per
minute pacing the atrium at the base of the PM
close to the CT. We have modified the CRN
model by including a hyperpolarising-activated
current (If) as given in Zhang et al. [5] to
simulate the auto-rhythmic AP in SAN as
shown in Figure 3. Details of modifications to
the CRN model to simulate SAN AP are given
in [3].
The 3D anatomical model for an adult
female human atrium [6] was obtained from the
National Library of Medicine of the National
Institute of Health in Bethesda, Maryland, USA
[7, 8]. This spatial resolution of the data set is
much higher than that of clinical MRI. The
voxel size is 0.33 mm x 0.33 mm x 0.33 mm.
The data were segmented as described in [6].
In the atrium, fibre orientation is not as
structured as in the ventricle. The fibre
orientation of the atrial working myocardium
has a complex and nearly random fashion [9].
Only the conduction pathways, i.e. CT, BB and
PM and the tissue near the mitral and tricuspidal
ostia, near the superior and inferior vena cavae
and near the pulmonary veins have a consistent
orientation. The macroscopic anisotropy of the
electrical excitation conduction pattern, as well
as mechanical contraction, is strongly
influenced by the spatial layout of the cardio-
20
mV
0
-20
-40
-60
-80
0
500
1000
1500
2000
2500
t (ms)
Figure 3. Pacemaking AP in the SAN simulated
using modifications to the CRN model as given
in [3].
myocytes, i.e., the orientation of the muscle
fibres and layers inside the myocardium. A
nearly random fibre orientation is reported
inside the human SAN [10]. These fibre
orientations have been incorporated in our
model.
Normal electrical excitation in the atrium
has been reconstructed previously [5, 11].
Normal conduction starts at the SAN, which is
the primary pacemaker of the heart with autorhythmic activity. Then the fast right atrial
conduction pathways i.e. CT and PM transmit
the excitation. During that phase, the right atrial
working myocardium gets depolarised. At the
same time, BBs transmit the activation towards
the LA and the whole atrium is activated. All
conduction pathways are characterized by a
large electrical conductivity along the fibre
direction. The excitation reaches the AV node,
which is the only electrical path between atria
and ventricles in the physiologically normal
case. After a short delay, the excitation is then
conducted into the ventricles. The simulated
conduction pattern of atrial excitation is shown
in Figure 4, using simple FHN cell model.
3. Computational resouces
The simulations were performed using a number
of different computing facilities. These included
systems hosted by the Computational Biology
Laboratories (CBL) at Leeds, the University of
Manchester and the National HPC Service,
CSAR [12]. At CBL Leeds a 24 processor SunFire-880 with UltraSPARC chip and a memory
of 24 GB is available. The principal author’s
laboratory has a 4 processor Sun-Fire-880 with
UltraSPARC chip with 16 GB of memory. A
local
university
wide
(University
of
Manchester) SGI machine with 32 SGI R14k
processors and a memory of 16 GB is also
available [13]. All these are shared memory
systems (SMP). A distributed memory system
with 16 dual processor Sun-Blade-1000 nodes is
also available at CBL, Leeds.
CSAR provides, amongst other facilities, a
512 processor SGI Origin 3800 machine with
512 GB memory. Access to this system was
granted through a Class 3 project, available for
new users to gain access to the system for
evaluation purposes.
The Sun-Fire machines have suitable up to
date C/C++ compilers for parallelisation using
OpenMP.
MPI codes are compiled with
MPICH (freely downloadable from the web) on
the Sun machines. CSAR has MIPSPro 7 series
and Intel v series of compilers.
4. Computing aspects of the 3D model
– necessisity of HPC
The 3D anatomical atrial model, as described in
the previous section, consists of 325 x 269 x
298 nodes. This amounts to more than 26 x 106
nodes. The following description of computing
resource is based on implementation of the
CRN 24 variable cell model, within human
virtual atria. During a 3D simulation, the
following arrays are required to be held in
memory. A tissue information array (~108
integer values), the state array (109 consisting of
double-precision floating-point data type), the
diffusion matrix (109 double-precision floatingpoint data type), spatial derivatives are required
to compute the heterogeneous conduction (108
double-precision floating-point data type). We
can see that a minimum of 17.2 GB of memory
is necessary. Instead of the CRN model, if a
simple FHN model is used, the memory
requirement is reduced to 10 GB due to a large
reduction in number of independent variables
associated with each node. In either case, large
amounts of contiguous memory is required.
The serial code for 3D models with FHN as
cell model was ran to estimate scalability.
Simulating 1000 time units of activity took 5
hours of computer time on a standard Sun-Fire
880 workstation. Upon parallelization, the same
run for FHN case was reduced to elapsed time
of 1.5 hours using 4 processors, showing good
scalability.
Further
optimization
was
implemented by use of binary output. This
further improved performance by 7 % for the
1000 time unit simulation. Results of this
simulation are shown in Figure 4.
Any pathology motivated case study
requires many repeated simulations with
different stimulus protocols or parameters in the
model. This increases enormously the compute
demand. AF due to re-entrant atrial excitation in
A
B
C
D
E
F
Figure 4. Simulation of normal conduction pattern in human female atrium. Translucent blue denoted
the atrial geometry and solid yellow represents the excitation. Propagation is from the SAN to the
LA. Model is stimulated at SAN (A, t = 0) and conduction spreads in the RA and along the BB and
PM into the LA (B, C, D, E, F). Simulation of 1000 time units of activity (frame F) takes 95 minutes
on a standard 4 processor Sun Sparc.
3D atrium is primarily propagating scroll
waves. Scroll waves can be initiated in the
atrium using several protocols. Simple S1-S2 or
cut wave, or the following possible protocols to
initiate re-entrant scroll waves are [14]:
1. S1 would be a stimulus at SAN. Then
an ectopic excitation S2 located within
the RA near the superior vena cavae,
15 mm away from the SAN.
2. Another protocol is the S1-S2-S3
protocol. The S1-S2 is as the same as
described above, but after a time delay,
a S3 stimulus is applied to the same
location of the S2.
3. This protocol rapidly paces the SAN at
high frequency. This results in re-entry
on the RA surface after sufficient
number of stimuli.
Initiation of re-entry at the required location in
the 3D geometry using the S1-S2, S1-cut wave,
or protocols 1 and 2 has to be done by trial and
error, involving several trial simulations. Other
possible stimulation protocols to induce normal
and scroll waves have been described in [15,
16].
Upon using the cut wave protocol, we
initiated re-entry and the results are shown in
Figure 5. Such a simulation of 1 s takes about
44 hours on 16 processors using an
improvement as described below.
Full geometrical model demands very large
amounts of contiguous memory. We have,
however, exploited problem specific features in
our simulations and reduced these overheads
considerably. Atrial tissue geometry occupies
about 8% geometry of the total data set, due to
atrium being thin walled, large holes of atrial
chambers and vena caves. We re-structure (or
renumber) the arrays mentioned in section 4
such that the real atrial nodes (leaving out the
empty nodes) of the data set, i.e. only 8% of the
total 26 million nodes and related information
are stored. This improved efficacy of memory
usage. By re-numbering the real atrial nodes we
are not storing any data points that are not
atrium. This reduced the memory required by
FHN model to less than 3 GB. The memory
required by CRN is reduced to less than 10 GB.
As a first step toward biophysically detailed
modeling of human atrium, we have simulated
electrical propagation using shared memory
systems. A shared memory system is not
necessary and the same results can be obtained
using distributed memory systems. Distributed
memory parallelism may give better scalability.
5. 2D Atrial and other small sized
simulations
Often, 2D tissue simulations offer useful
insights into the mechanisms underlying the
A
B
D
E
C
F
Fig 5. Initiation and propagation of scroll wave in human female atrium considering the healthy, or
control case. Translucent blue shows atrial geometry, and solid yellow denotes propagating electrical
activity. The SAN was stimulated to induce a solitary propagation. After an appropriate duration of
time, the upper half of the geometry was reset to resting conditions
simulating
the cutexamples
wave protocol
(A,
Following
are two
where
t = 210 ms). Propagation of scroll wave can be seen in distributed
B (t = 230 ms),
C
(t
=
250
ms),
D
(t
=
270
ms)
memory computing can assist in
and E (t = 290 ms). In control case, re-entrant scroll waves
self terminate
(F,and
t = 1D
335tissue
ms). behaviors.
investigation
of cell
genesis of AF, without dealing with complex
3D simulations. 2D tissue simulations were
performed to investigate the link between AF
genesis and gain-of-function in IK1 due to Kir2.1
gene mutation [17]. In this simulation, tissue
size was taken to be 37.5 mm x 37.5 mm, in
accordance with normal size of atrial
appendage. Spiral re-entry was initiated with a
standard cross-field stimulation protocol [18].
To characterize the stability of reentry, a 10 s
long run of simulations were performed for 3
cases, consisting of control, for heterozygous
and homozygous Kir2.1 gene mutation. Surface
potentials was taken, to allow reconstruction of
animation of activity from t = 0 to t = 10 s. In
addition, pseudo-ECG and spiral wave tip
positions were computed at run time. Sample
frames from the 2D simulation results are
shown in Figure 6.
Parallelization helps to reduce computing
time and allows for a more intensive
investigation. The program was parallelized
using OpenMP and was ran on a dual processor
Sun workstation. The time taken was 29 hours.
We then ran it on 4 processor Sun-Fire and the
elapsed time reduced to 16 hours. Our 2D code
shows good scalability with near ideal speedup.
The small fraction of code that has to be
necessarily serial is while doing essential file
output.
Cell models need to be investigated for various
behaviors [19]. In cell models, a generic
investigation is that of pacing based behavior of
AP. Detailed of biophysical cell models in
themselves are not overly demanding on
memory, requiring storage of several tens or
few hundreds of variables (e.g. models
incorporating Markov chain formulations of
cellular processes). Pacing based investigations
are however, long by the nature of the problem.
A ventricular cell model developed by PriebeBeckmann (PB) [20] for non-failing and failing
heart was pacing at various pacing rates from
basic cycle length (BCL) = 100 ms, to BCL =
1200 ms in increments of 5 ms with trains of
100 stimuli. AP behavior for the final 10 stimuli
were noted and plotted against BCL as a
bifurcation diagram, to investigate the existence
of AP alternans or 2:2 responses. Figure 7
shows the results from such a simulation.
Simulation carried out on single processor as a
serial run involved running this model for 60
hours. Inter-process communication in such a
simulation is minimal, with the only
synchronization required being in accumulation
of the results. The pacing at a given BCL is
independent of pacing result at another BCL.
This being a cell model, the simulation is not
demanding for memory of each node in the
distributed memory system. Shared memory
systems with large amount of memory are not
required. If the code is parallelized for a
I
II
III
Figure 6. Representative frames from 2D simulations while investigating the effects of gain-offunction in IK1 on AF. Top panel shows frames for control, middle panel for heterozygous, and
bottom panel for homozygous mutant type. 2D simulations at t = 400 ms in column I, t = 1300 ms in
column II, and t = 2650 ms in column III. A total of 10 s of activity was simulated.
6.Conclusions and Discussions
In this study, we have developed a 3D computer
model of human atrium with detailed
anatomical
structures
and
cellular
electrophysiology. This model provides an
alternative to experimental methods to
investigate the cellular ionic and molecular
mechanisms underlying the genesis and control
of AF in humans. Due to large size of geometry
and multi-variable nature of the system, the
model demands extensive computing resources
600
500
APD90 (ms)
distributed memory system, then run time can
be reduced drastically, depending on the
number of nodes.
Another example where low cost distributed
memory systems can be utilized is shown in
Figure 8. Here we have a 1D transmural strand
of human ventricle of length 15 mm with a
spatial resolution of 0.1 mm with 150 cells, of
which 40 are endo-, 50 are mid-, and 60 are
epi-. Computing vulnerability window (VW) at
a ventricular cell location requires running the
1D serial program on 1 processor for 1.3 hours.
Again, as in the cell model case, computation of
VW at a cell location is independent of
computation of VW at another cell location. The
memory requirement by the 1D model is
moderate and can be well managed by nodes of
any reasonable distributed memory system.
Since VW computations at each of the cell
locations are independent of each other,
synchronization is not required. This problem
also lends itself distributed memory parallelism.
heart failure
Control
400
300
200
2:2
100
200
400
600
800
1000
1200
BCL (ms)
Figure 7. APD90 alternans (2:2 response) in PB
model. Blue denotes control, red denotes heart
failure. PB model is paced at BCLs from 100
ms to 1200 ms in increments of 5 ms.
Dynamic response of last 10 APs from a train of
100 was noted. Alternans occur at low BCL.
and is an ideal test bed for HPC algorithms.
Simulations with low memory demands, but
require long computations due to nature of
problem are ideal for distributed computing.
500
VW (ms)
heart failure
475
450
ENDO-
MID-
EPI-
425
Control
400
0
2
4
6 8 10 12 14
x (mm)
Fig 8. Computation of VW at each cell location
of a 15 mm human virtual ventricular 1D strand
with endo-, mid-, and epi- regions. A serial run
of the program takes 200 hours. A single
determination of VW takes 1.33 hours.
Acknowledgements
This work was supported by British Heart
Foundation
(PG/03/140)
and
BBSRC
(BBS/B/1678X) UK.
References
[1] Wattingney WA, Mensah GA, Croft JB.
Increased Atrial Fibrillation mortality: United
States, 1980 – 1998. Am J Epidemiol. 2002;
155: 819 – 826.
[2] Courtemanche M, Ramirez RJ, Nattel S.
Ionic mechanisms underlying human atrial
action potential properties: insights from a
mathematical model. Am J Physiol. 1998 Jul;
275 (1 Pt 2): H301-21.
[3] Seemann G. Modeling of Electrophysiology
and Tension Development in the Human Heart.
2005; Ph.D. Thesis. University of Karlsruhe.
[4] Feng J, Yue L, Wang Z, and Nattel S. Ionic
mechanisms of regional action potential
heerogeniety in the canine right atrium. Circ.
Res. 1998; 83: 541-551.
[5] Zhang H, Holden AV, Kodama I, Honjo H,
Lei M, Vagues T, and Boyett MR.
Mathematical models of action potentials in the
periphery and centre of the rabbit sinoatrial
node. J. Am. Physiol. 2000; 279 (1): H397H421.
[6] Seemann G, Hoper C, Sachse FB, Dossel O,
Holden AV, Zhang H. 3D anatomical and
electrophysiological model of human sinoatrial
node and atria. Phil. Trans. Roy. Soc., 2006. In
press.
[7] Ackerman MJ. “Viewpoint: The Visible
Human Project”. J. Biocommunications. 18 (2):
14 1991.
[8] “Visible Human Project, National Library of
Medicine, Bethsda, USA”.
[9] Boineau JP, Canavan TE, Schuessler RB,
Cain ME , Corr PB, Cox JL. Demonstration of a
widely distributed atrial pacemaker complex in
the human heart. Circ. 1988; 77: 1221-1237.
[10] Anderson RH, Ho SY. The architecture of
the sinus node, the atrioventricular conduction
axis, and the internodal atrial myocardium. J.
Cardiovasc. Electrophysiol. 1998; 9: 12331248.
[11] Harrild DM, Henriquez CS. A computer
model of normal conduction in the human atria.
2000; Circ. Res. 87 (7): e25-e36.
[12] http://www.csar.cfs.ac.uk/
[13] http://www.mc.manchester.ac.uk/services
[14] Virag N, Jacquemet V, Henriquez CS,
Zozor S, Blanc O, Vesin J-M, Pruvot E,
Kappenberger L. Study of atrial arrhythmias in
a computer model based on magnetic resonance
images of human atria. 2000; Chaos 12 (3): 754
– 763.
[15] Vigmond EJ, Ruckdeschel R, Trayanova
N. Re-entry in a morphologically realistic atrial
model. 2001; J. Cardiovas. Electrophysiol. 12
(9): 1046 – 1054
[16] Haissaguerre M, Jais P, Shah DC,
Takahashi A, Hocini M, Quiniou G, Garrigue S,
Le Mouroux A, Le Metayer P, Clementy J.
Spontaneous Initiation of Atrial Fibrillation by
Ectopic Beats Originating in the Pulmonary
Veins. N Engl J Med. 1998; 339 (10): 659-666
[17] Kharche S, Moore H, Garratt CJ, Hancox
JC, Zhang H. Gain-of-Function in Kir2.1 and its
Effects on Atrial Fibrillation in Homogeneous
Virtual Human Atrial Tissue: A Computer
Simulation Study. Poster at the ISCE 31st
Meeting, Niagara-on-the-Lake, Canada. April
2006.
[18] Biktasheva IV, Biktashev VN, and Holden
AV. Wave-breaks and self-termination of spiral
waves in a model of human atrial tissue. LNCS
2005; 3504: 293-303.
[19] Kharche S, Zhang H, Holden AV.
Vulnerability in a One-Dimensional Transmural
Model of Human Ventricular Tissue in Heart
Failure. Computers in Cardiology Conference,
Lyon,
2005;
32:
563-566
[20] Priebe L and Beuckelmann DJ. Simulation
Study of Cellular Electric Properties in Heart
Failure, Circ Res. 1998; 82 (11): 1206-1223.