Basic Concepts of
FEM Framework &
API
Gunavardhan Kakulapati
(kakulapa@cs.uiuc.edu)
1
Introduction
 FEM Framework overview
 Using the framework API
 Installation
 Conversion Example

2
Motivation

Finite Element Method
Used extensively
 Computationally intensive


Do it in parallel !!
3
FEM Programs
FEM programs manipulate elements and
nodes
 Element is a portion of problem domain,
surrounded by nodes
 Element computation: based on fields
of surrounding nodes
 Elements contribute to fields of
surrounding nodes

4
FEM Mesh
Element
Surrounding
Nodes
E1
N1
N3
N4
E2
N1
N2
N4
E3
N2
N4
N5
5
FEM Program Structure
read mesh, connectivity, boundary conditions
time loop
element loop- Element deformation applies
forces to surrounding nodes
node loop- Forces and boundary conditions
change node positions
end time loop
write out mesh data for postprocessing
6
Parallelization
Partition the FEM Mesh into multiple
chunks
 Distribute elements, replicate shared
nodes
 Shared nodes = Communication !!
 Partition so that communication is
minimized

7
Partitioning
Element
Surrounding Nodes
E1
N1
N3
N4
E2
N1
N2
N3
Element
Surrounding Nodes
E1
N1
N2
N3
Shared Nodes
A
B
N2
N1
N4
N3
8
Parallel FEM Program
read/get chunk mesh data, connectivity, shared nodes
chunk time loop
element loop- Element deformation applies
forces to surrounding nodes
<update forces on shared nodes>
node loop- Forces and boundary conditions
change node positions
end time loop
write chunk mesh data for postprocessing
>>
9
FEM Framework: Goals
Separate parallel implementation from
numerical algorithms
 Parallel version to closely resemble the
serial program
 Allow features like load balancing,
visualization.

10
FEM Framework: Responsibilities
FEM Application
(Initialize, Registration of Nodal Attributes, Loops Over Elements, Finalize)
FEM Framework
(Update of Nodal properties, Reductions over nodes or partitions)
Partitioner
METIS
Combiner
Charm++
(Dynamic Load Balancing, Communication)
I/O
11
FEM Framework Program

May contain user-written, library-called
subroutines:



init
driver
mesh_updated (more on this later…)
init and mesh_updated are called on
processor 0
 driver is called on every chunk

12
Structure of an FEM
Application
init()
driver
driver
Shared Nodes
Update
driver
Shared Nodes
Update
Update
13
init
subroutine init
read the serial mesh and
configuration data
inform the framework about
the mesh
end subroutine
14
driver
subroutine driver
get local mesh chunk
time loop
FEM computations
update shared node fields
more FEM computations
end time loop
end subroutine
15
Data output
Parallel output at the end of driver
 Possible to visualize the data rather than
printing it out (netfem).
 Framework calls like FEM_Mesh_updated.
(more on this later …)

>>
16
Framework Calls

FEM_Set_*



FEM_Create_field


Registers a node data field with the framework,
supports user data types
FEM_Update_field



Called from initialization to set the serial mesh
Framework partitions mesh into chunks
Updates node data field across all processors
Handles all parallel communication
Other parallel calls (Reductions, etc.)
17
Framework Calls: Mesh

FEM_Set_*
From init, these routines describe the mesh
to the FEM Framework
 Framework calls the partitioner


FEM_Get_*

From each chunk, the local portion of the
mesh is obtained by the driver
18
FEM_*_elem
FEM_Set_elem(int elType,int nEl,
int doublePerEl,int nodePerEl);
subroutine FEM_Set_elem(elType,nEl,doublePerEl,nodePerEl)
integer, intent(in) :: elType,nEl,doublePerEl,nodePerEl
FEM_Get_elem(int elType,int* nEl,
int* doublePerEl,int* nodePerEl);
subroutine FEM_Get_elem(elType,nEl,doublePerEl,nodePerEl)
integer, intent(in) :: elType
integer, intent(out) :: nEl,doublePerEl,nodePerEl
19
FEM_*_node
subroutine FEM_Set_node(nNode,doublePerNode)
integer, intent(in) :: nNode,doublePerNode
subroutine FEM_Get_node(nNode,doublePerNode)
integer, intent(out) :: nNode,doublePerNode
20
Element Connectivity
subroutine FEM_Set_Elem_Conn_r(elType,conn)
integer, intent(in) :: elType
integer, intent(in), dimension(nodePerEl,nEl) :: conn
subroutine FEM_Get_Elem_Conn_r(elType,conn)
integer, intent(in) :: elType
integer, intent(out), dimension(nodePerEl,nEl) :: conn
subroutine FEM_Set_Elem_Conn_c(elType,conn)
integer, intent(in) :: elType
integer, intent(in), dimension(nEl,nodePerEl) :: conn
subroutine FEM_Get_Elem_Conn_c(elType,conn)
integer, intent(in) :: elType
integer, intent(out), dimension(nEl,nodePerEl) :: conn
21
Additional Data for Nodes and
Elements
subroutine FEM_Set_node_data_r(data)
REAL*8, intent(in),
dimension(doublePerNode,nNode) :: data
subroutine FEM_Get_node_data_r(data)
REAL*8, intent(out),
dimension(doublePerNode,nNode) :: data
subroutine FEM_Set_elem_data_r(data)
REAL*8, intent(in),
dimension(doublePerElem,nElem) :: data
subroutine FEM_Get_elem_data_r(data)
REAL*8, intent(out),
dimension(doublePerElem,nElem) :: data
22
Node Fields
Framework handles combining data for
shared nodes and keeps them in sync
 Framework does not understand meaning of
node fields, only their location and types
 Framework needs to be informed of locations
and types of fields
 Create_field once, Update_field every
timestep

23
FEM_Create_field
To handle the updating of shared node
values.
 Tell the framework where the shared
data items of each node are.
 Creates a “field” and pass the field ID
for updating shared nodal values.

24
FEM_Create_simple_field
function integer :: FEM_Create_simple_field(
base_type,
vec_len)
integer, intent(in) :: base_type, vec_len
Base_type
•FEM_BYTE- INTEGER*1, or CHARACTER*1
•FEM_INT- INTEGER*4
•FEM_REAL- REAL*4
•FEM_DOUBLE- DOUBLE PRECISION, or REAL*8
25
Create_simple_field Example
! 3D Force for each node
! stored as 3*n real*8 array
REAL*8 ALLOCATABLE, DIMENSION(:) :: nodeForce
INTEGER :: fid
... allocate nodeForce as 3*n_nodes...
fid = FEM_Create_simple_field(FEM_DOUBLE,3)
26
FEM_Create_field
function integer :: FEM_Create_Field(base_type, vec_len,
offset, dist)
integer, intent(in) ::
base_type, vec_len, offset, dist
27
Node Fields
28
Create_field Example
! 3D force is contained as fXYZ variable
! in a user-defined type node_type
TYPE(node_type), ALLOCATABLE, DIMENSION(:) :: nodes
INTEGER :: fid
...allocate nodes array as n_nodes...
fid = FEM_Create_Field(FEM_DOUBLE,3,
offsetof(nodes(1), nodes(1)%fXYZ),
offsetof(nodes(1), nodes(2)) )
29
Update and Reduce Field
subroutine FEM_Update_Field(fid,nodes)
integer, intent(in) :: fid
varies, intent(inout) :: nodes
subroutine FEM_Reduce_Field(fid,nodes,outVal,op)
integer, intent(in) :: fid,op
varies, intent(in) :: nodes
varies, intent(out) :: outVal
op is
•FEM_SUM
•FEM_MIN
•FEM_MAX
30
Utility
function integer :: FEM_Num_Partitions()
function integer :: FEM_My_Partition()
function double precision :: FEM_Timer()
subroutine FEM_Print_Partition()
subroutine FEM_Print(str)
character*, intent(in) :: str
31
Advanced FEM calls

FEM_Update_mesh


Reassembles chunks of the mesh
FEM_Add_node

Adds a new node into the mesh
32
FEM_Update_mesh
Reassembles all the chunks
 Can be called only from driver
 Must be called from all chunks
 Useful scenarios

Giving out data as the simulation runs
 Repartition the mesh
 Serial output when simulation finishes

33
FEM_Update_mesh
C call:
void FEM_Update_mesh(int callMeshUpdated, int doWhat)
Fortran call:
subroutine FEM_Update_mesh(callMeshUpdated,doWhat)
integer, intent(in) :: callMeshUpdated, doWhat
34
FEM_Update_mesh
parameters
callMeshUpdated is non-zero => call
mesh_updated (callMeshUpdated)
 doWhat:

doWhat
0
Repartition
No
FEM_Update_mesh
Non-blocking mesh_updated
1
Yes
Blocks for repartitioning
2
No
Blocking mesh_updated
35
FEM_Update_mesh examples

FEM_Update_mesh(k,0)


FEM_Update_mesh(k,1)


Call mesh_updated(k) on assembled mesh,
while the driver continues
Repartition after mesh_updated
FEM_Update_mesh(k,2)

Block driver routines till mesh_updated(k)
36
Adding Nodes

One can add new nodes, and update
connectivity in driver
Use FEM_Set_* subroutines
 New nodes are considered private


Framework can repartition the mesh

Optionally calls user’s mesh_updated
subroutine
37
FEM_Add_node
C call:
void FEM_Add_node(int localIdx,
int nBetween,
int * betweenNodes)
Fortran call:
subroutine
FEM_Add_node(localIdx,nBetween,betweenNodes)
integer, intent(in) :: localIdx,nBetween
integer, intent(in) :: betweenNodes(nBetween)
>>
38
Installing FEM
Framework
39
Where to Get It ?
FEM Framework is included in Charm++ distribution,
available under CVS
CSH:
setenv CVSROOT ":pserver:checkout@charm.cs.uiuc.edu:/cvsroot"
Or BASH:
export CVSROOT=":pserver:checkout@charm.cs.uiuc.edu:/cvsroot"
You should now be able to do a
> cvs login
(no password needed, just type [Enter] at prompt)
and then
> cvs co -P charm
to get the entire Charm++ source.
40
How to Build It ?
> cd charm
and do
> ./build FEM net-linux -O
This will make a net-linux directory,
with bin, include, lib etc subdirectories.
Platforms: net-sol, mpi-origin, mpi-linux etc.
41
How to Write Programs ?

Write from scratch
Concepts and API discussed earlier
 Read the FEM-Framework Manual

http://charm.cs.uiuc.edu/manuals/fem

Convert existing program

To be covered in next section
42
How to Compile & Link ?

Use “charmc”: available under bin




Linking



a multi-lingual compiler driver, understands f90
Knows where modules and libraries are
Portable across machines and compilers
use “-language femf” : for F90
Use “–language fem” : for C/C++
See example Makefiles

pgms/charm++/fem/…
43
How to Run ?

Just run it! (net- versions only)


Serial, but nice for debugging/testing
Use Charmrun
A portable parallel job execution script
 Specify number of processors: +pN
 Special “nodelist” file for net-* versions
 Multiple chunks per processor: use +vpM

44
Charmrun Example
Nodelist File: $(HOME)/.nodelist
group main
host tur0001.cs.uiuc.edu
host tur0002.cs.uiuc.edu
host tur0003.cs.uiuc.edu
etc…
./charmrun pgm +p4 +vp8
>>
45
FEM Framework
Conversion
example
46
A Serial Program
Processing
Input
Output
47
A Parallel Framework Program
Parallel
Processing
Output
Input
Parallel
Infrastructure
48
Real Names of Pieces
Driver
Finalize or
Mesh_updated
Init
Charm++ FEM
Framework
49
Serial Example Program

F90 example
Reads input mesh in “Triangle” format
 Does simple explicit mechanics
computation (CST triangles)
 Writes Tecplot output


Not a toy example (by Philippe
Geubelle)
Reads a parameter file
 Applies boundary conditions
 Uses real mechanics

50
FEM Framework Version
Uses all the same code as serial version
 Main program becomes init subroutine
 Split up init and driver

Copy entire mesh into framework (FEM_Set)
 Copy portion of mesh out (FEM_Get)

Changes to time loop are minimal
 Output poses an interesting problem

51
Splitting the program
Declare variables
call readTriGlobals
Read mesh conn. Data
Init
Dynamics loop
Calc forces
Calc accl,vel
Apply bound. conditions
Update displacements
Output current data
Driver
52
Init – Basic differences
Serial version
Parallel version
Declare variables
. . .
call readTriGlobals
Declare variables
. . .
call readTriGlobals
. . .
Read mesh conn. data
. . .
Read mesh conn. data
. . .
Call the set methods
53
Driver
Serial version
Parallel version
Dynamics loop
Call get methods
Create field
Dynamics loop
Calc forces
Calc accl,vel
Apply bound. Cond
Update disp
Output current data
Calc forces
Update field
Calc accl,vel
Apply bound. Cond
Update disp
Visualize current data
54

More information:

Read the manual and FEM framework
details at http://charm.cs.uiuc.edu
55