EPICS Internals
Nick Rees
Outline
•
•
•
•
•
Introduction
Libraries and modules
Data Structures
Process types
Examples of processing
INTRODUCTION
Background
• When I accepted this challenge I thought there
was an EPICS internals talk I could copy.
– It turns out there wasn’t.
• I decided the aim of the talk was to provide EPICS
developers with a vague idea of what to expect to
see when they look at a running EPICS system.
– It turns out this wasn’t that easy
– It also was very difficult to do this in 40 minutes.
– By focussing on this I missed out more than I covered.
• The result is something which even I find a bit
difficult to understand (and I wrote the talk), so I
don’t know about the rest of you.
– For the non-programmers out there, good luck.
What is EPICS core?
What is EPICS core?
• 109 directories
• 1567 files, 239241 lines of all types
–
–
–
–
281 files, 93451 lines of C
203 files, 49807 lines of C++
329 files, 34985 lines of header files
28 files, 14964 lines of html
• Over 20 years of development by a competent
world-wide team.
• Not an enormous body of code.
• A pretty reliable body of code that does what it is
designed to do.
• A reasonably useful, flexible and portable tool
What isn’t EPICS core?
•
•
•
•
•
•
Consistently designed
Designed
Buzzword compliant, or even object-oriented
Well understood by lots of people.
Stylistically consistent.
Pretty, elegant or pure
Another view
• The Application Developers guide states:
– “EPICS consists of a set of core software and a set of
optional components. The core software, i.e. the
components of EPICS without which EPICS would not
function, are:
•
•
•
•
•
•
Channel Access - Client and Server software
IOC Database
Scanners
Monitors
Database Definition Tools
Source/Release”
• This talk focuses on the first 4 points, with most
emphasis on the IOC Database.
LIBRARIES AND MODULES
Helper libraries
• If you write EPICS base software you must know,
understand and use the basic helper libraries it
provides. For example:
– ellLib – doubly linked list library identical to VxWorks
listLib
– gpHash – efficient hashes
– freeList – malloc replacement giving efficient handling
of fixed size blocks.
– epicsRingBytes – ring buffers.
– epicsThread – thead creation and control
– epicsMutex – OS independent mutexes
– etc...
Essential EPICS code
• dbStaticLib.c
– Static database access routines
• Used during EPICS development or before iocInit.
• dbAccess.c
– Run-time database access routines.
– dbNameToAddr, dbPutField, dbPut, dbGetField, dbGet,
dbProcess etc.
• dbConvert.c
– Type conversions
• camessage.c
– Channel Access server function table implementations.
• dbEvent.c
– Implements database event handling
• etc.....
What EPICS libraries are there?
• Common
–
–
–
–
–
Cap5
ca
cas
Com
gdd
• Host and IOC:
–
–
–
–
asHost
asIoc
dbStaticHost
dbStaticIoc
• IOC only
–
–
–
–
–
–
–
–
dbIoc
dbtoolsIoc
miscIoc
recIoc
registryIoc
rsrvIoc
softDevIoc
testDevIoc
EPICS library dependencies
Conclusion
• If you want to know more, consult the source.
DATA STRUCTURES
dbBase.h
• Basic database data structures are in dbBase.h
• These define the structures that hold the
database.
• The are defined in the dbd and db files, and
created by dbStaticLib.
• At run-time, they are accessed by dbAccess
routines.
dbBase.h record tree
• dbBase
└ recordTypeList
└ dbRecordType
└ recordList
└ dbRecordNode
└ name
└ precord
└ dset
└ lset etc.
└ ...
└ deviceList
└ ...
└ dbFldDes[]
└ rset
└ ...
dbBase.h tree
dbBase definition
typedef struct dbBase {
ELLLIST
menuList;
ELLLIST
recordTypeList;
ELLLIST
drvList;
ELLLIST
registrarList;
ELLLIST
functionList;
ELLLIST
variableList;
ELLLIST
bptList;
void
*pathPvt;
struct dbPvd *ppvd;
struct gphPvt *pgpHash;
short
ignoreMissingMenus;
short
loadCdefs;
}dbBase;
dbRecordType definition
typedef struct dbRecordType {
ELLNODE
node;
ELLLIST
attributeList;
/*LIST of attributes*/
ELLLIST
recList;
/*LIST of sorted dbRecordNodes*/
ELLLIST
devList;
/*LIST of device support*/
ELLLIST
cdefList;
/*LIST of Cdef text items*/
char
*name;
short
no_fields;
/* number of fields defined
*/
short
no_prompt;
/* number of fields to configure*/
short
no_links;
/* number of links
*/
short
no_aliases;
/* number of aliases in recList */
short
*link_ind;
/* addr of array of ind in papFldDes*/
char
**papsortFldName;/* ptr to array of ptr to fld names*/
short
*sortFldInd;
/* addr of array of ind in papFldDes*/
dbFldDes
*pvalFldDes;
/*pointer dbFldDes for VAL field*/
short
indvalFlddes;
/*ind in papFldDes*/
dbFldDes
**papFldDes;
/* ptr to array of ptr to fldDes*/
/*The following are only available on run time system*/
struct
rset
*prset;
int
rec_size;
/*record size in bytes
*/
}dbRecordType;
dbRecordNode definition
typedef struct dbRecordNode {
ELLNODE node;
void
*precord;
char
*recordname;
ELLLIST infoList; /*LIST of info nodes*/
int
flags;
}dbRecordNode;
Record (dbCommon) definition
typedef struct dbCommon {
char
name[61]; /* Record Name */
char
desc[41]; /* Descriptor */
char
asg[29]; /* Access Security
Group */
epicsEnum16 scan; /* Scan Mechanism */
epicsEnum16 pini; /* Process at iocInit */
epicsInt16
phas;/* Scan Phase */
epicsInt16
evnt; /* Event Number */
epicsInt16
tse; /* Time Stamp Event */
DBLINK
tsel; /* Time Stamp Link */
epicsEnum16 dtyp; /* Device Type */
epicsInt16
disv; /* Disable Value */
epicsInt16
disa; /* Disable */
DBLINK
sdis; /* Scanning Disable */
epicsMutexId mlok; /* Monitor lock */
ELLLIST
mlis; /* Monitor List */
epicsUInt8
disp; /* Disable putField */
epicsUInt8
proc; /* Force Processing */
epicsEnum16 stat; /* Alarm Status */
epicsEnum16 sevr; /* Alarm Severity */
epicsEnum16 nsta; /* New Alarm Status */
epicsEnum16 nsev; /* New Alarm Severity */
epicsEnum16 acks; /* Alarm Ack Severity */
epicsEnum16 ackt; /* Alarm Ack Transient */
epicsEnum16 diss; /* Disable Alarm Sevrty */
epicsUInt8
lcnt; /* Lock Count */
epicsUInt8
pact; /* Record active */
epicsUInt8
putf; /* dbPutField process */
epicsUInt8
rpro; /* Reprocess */
struct asgMember *asp; /* Access Security Pvt */
struct putNotify *ppn; /* addr of PUTNOTIFY */
struct putNotifyRecord *ppnr;
struct scan_element *spvt; /* Scan Private */
struct rset
*rset; /* Address of RSET */
struct dset
*dset;/* DSET address */
void
*dpvt; /* Device Private */
struct dbRecordType *rdes; /* dbRecordType */
struct lockRecord *lset;
/* Lock Set */
epicsEnum16 prio; /* Scheduling Priority */
epicsUInt8
tpro; /* Trace Processing */
char
bkpt; /* Break Point */
epicsUInt8
udf; /* Undefined */
epicsTimeStamp
time;
/* Time */
DBLINK
flnk; /* Forward Process Link */
} dbCommon;
dbAddr definition
• Virtually all runtime access to the database is via a dbAddr
handle.
typedef struct dbAddr {
struct dbCommon *precord; /* address of record
void
*pfield;
/* address of field
struct dbFldDes *pfldDes; /* address of fldDes
long
no_elements;
/* number of elements
short
field_type;
/* type of field
short
field_size;
/* size of the field
short
special;
/* special processing
short
dbr_field_type;
/* request type
/* DBR_STRING,...,DBR_ENUM,DBR_NOACCESS
} dbAddr;
*/
*/
*/
*/
*/
*/
*/
*/
*/
PROCESS TYPES
EPICS and threads
• A typical C/C++ programmer’s first reaction is
that EPICS has lots of threads
• However, they all have there uses, and once you
understand them they are not so confusing.
• The need arises from the need to have nonblocking processing and strict priorities.
• It is actually simpler and more efficient than a lot
of select() calls.
EPICS Threads
Name
cbHigh
timerQueue
scanOnce
scan0.1
scan0.2
cbMedium
scan0.5
scan1
scan2
scan5
scan10
cbLow
CAC-event
dbCaLink
CAS-client
CAS-event
CAS-TCP
CAS-beacon
CAS-UDP
errlog
taskwd
Priority
71
70
70
66
65
64
64
63
62
61
60
59
51
50
20
19
18
17
16
10
10
• IOC management:
– timerQueue
– taskwd
– errlog
• Database processing:
– cb*
– scan*
• Channel Access:
– CAS-*
– CAC-event
– dbCaLink
• Plus various driver threads...
IOC management threads
• timerQueue
– Implements all EPICS base delays
• taskwd
– Monitors tasks for any suspensions
• errlog
– Handles asynchronous processing of log messages.
– Forwards log messages to the console or to files at low
priority.
Database processing threads
• cb*
– callback tasks – used for event and I/O Interrupt
scanned records.
– Calls any specified function asynchronously
– 3 priorities (low, medium high)
• scan*
– periodic scan tasks
– Calls record process() routines at regular intervals.
– Higher scan rates have higher priorities.
• + all driver threads
– Every asyn port has its own thread
Channel Access threads
• CAS-*
– Channel access server tasks
•
•
•
•
CAS-UDP listens for channel lookups
CAS-TCP listens for TCP connections
CAS-beacon looks for CA beacons
One CAS-client and CAS-event task per client
• dbCaLink
– Channel Access client that processes CA requests on
behalf of the database.
• CAC-event
– Handles the CA client callback events (I think)
EXAMPLES OF PROCESSING
Example sequence diagram
Task name
C filename
C function
Client
IOC tasks
IOC function calls
CA server call processing
• Every CA client message has a function code defined
in caProto.h of the form CA_PROTO_XXXX
– E.g.: CA_PROTO_WRITE
• This will convert into a function in one of the CA jump
tables (tcpJumpTable or udpJumpTable). Functions
are off the form xxxx_action(mp,pPayload,client)
– Parameters are pointers to
• CA header block
• Data
• client structure.
– Performs:
• Net to host byte swapping
• Access security checks
– E.g.: write_action(mp,pPayload,client)
CA server call processing
• xxxx_action often calls a “CA db_access” routine
with underscores, which handles an ugly “old to
new” type enumeration conversion
– E.g.: db_put_field(pAddr, oldDataType,pPayload,count)
• This calls a dbAccess routine which is part of the
database code.
– E.g.: dbPutField(pAddr,newDataType,pDate,count)
• This locks the database and implements the
appropriate action.
caget from a record
Channel life cycle
caput to an ai record
dbPutField
long epicsShareAPI dbPutField(DBADDR *paddr, short dbrType,
{
long
status = 0;
dbFldDes
*pfldDes = paddr->pfldDes;
dbCommon
*precord = paddr->precord;
short
dbfType = paddr->field_type;
const void *pbuffer, long nRequest)
/* Various sanity tests deleted here... */
dbScanLock(precord);
status = dbPut(paddr, dbrType, pbuffer, nRequest);
if (status == 0) {
if (paddr->pfield == (void *)&precord->proc ||
(pfldDes->process_passive && precord->scan == 0 && dbrType < DBR_PUT_ACKT)) {
if (precord->pact) {
precord->rpro = TRUE;
} else {
/* indicate that dbPutField called dbProcess */
precord->putf = TRUE;
status = dbProcess(precord);
}
}
}
dbScanUnlock(precord);
return status;
}
dbPut
dbPut(DBADDR *paddr, short dbrType, const void *pbuffer, long nRequest)
{
long special
= paddr->special;
short field_type = paddr->field_type;
long no_elements = paddr->no_elements;
if (special) dbPutSpecial(paddr, 0);
if ( IS AN ARRAY ) prset->get_array_info(paddr, &dummy, &offset);
dbPutConvertRoutine[dbrType][field_type](paddr,
pbuffer,
nRequest,
no_elements,
offset);
if ( IS AN ARRAY ) prset->put_array_info(paddr, nRequest);
if (special) dbPutSpecial(paddr,1);
if ( not ( VALUE FIELD and PP ))
db_post_events(precord, paddr->pfield, DBE_VALUE | DBE_LOG);
return;
}
camonitor from an ai record
caput callback from an ai record
What’s the stack trace like?
Call
Location
clone()
libc.so.6
start_thread()
CAS-client thread
start_routine(arg)
libpthread.so.0
osdThread.c:320
Socket recv call (caput starts
here)
camsgtask.c:123
camessage(client) Process CA messages camessage.c:2442
write_action(mp,pPayload,client) caput CA action
camessage.c:810
camsgtask(pParm)
db_put_field(paddr,src_type,psrc,no_elements)
db_access.c:1216
dbPutField(paddr,dbrType,pbuffer,nRequest)
dbAccess.c:1257
dbProcess(precord)
dbAccess.c:630
process(pao)
librecIoc.so
writeValue(pao)
aoRecord.c:537
write_ao(pao)
devAoSoft.c:72
caput to an ai record
Call
Location
clone()
libc.so.6
start_thread()
CAS-client thread
start_routine(arg)
libpthread.so.0
osdThread.c:320
Socket recv call (caput starts
here)
camsgtask.c:123
camessage(client) Process CA messages camessage.c:2442
write_action(mp,pPayload,client) caput CA action
camessage.c:810
camsgtask(pParm)
db_put_field(paddr,src_type,psrc,no_elements)
db_access.c:1216
dbPutField(paddr,dbrType,pbuffer,nRequest)
dbAccess.c:1257
dbProcess(precord)
dbAccess.c:630
process(pao)
librecIoc.so
writeValue(pao)
aoRecord.c:537
write_ao(pao)
devAoSoft.c:72
caput callback to an ai record
Call
Location
clone()
libc.so.6
start_thread()
CAS-client task
libpthread.so.0
start_routine(arg)
osdThread.c:282
camsgtask(pParm)
camsgtask.c:123
camessage(client)
camessage.c:2508
write_notify_action(mp,pPayload,client)
camessage.c:1823
dbPutNotify(ppn)
dbNotify.c:342
putNotifyCommon(ppn,precord)
dbNotify.c:236
dbProcess(precord)
dbAccess.c:643
process(precord)
aiRecord.c:177
recGblFwdLink(precord)
recGbl.c:267
dbNotifyCompletion(precord)
dbNotify.c:422
callbackRequest(pcallback)
callback.c:154
camonitor from an ai record
Call
Location
clone()
start_thread()
scan1 task
start_routine(arg)
periodicTask(arg)
scanList(psl)
dbProcess(precord)
process(prec)
recGblFwdLink(precord)
dbScanFwdLink(plink)
dbScanPassive(pfrom,pto)
dbProcess(precord)
process(precord)
monitor(prec)
db_post_events(pRecord,pField,caEventMask)
db_post_single_event_private(event)
libc.so.6
libpthread.so.0
osdThread.c:282
dbScan.c:553
dbScan.c:658
dbAccess.c:643
calcRecord.c:126
recGbl.c:265
dbAccess.c:519
dbAccess.c:490
dbAccess.c:643
aiRecord.c:175
aiRecord.c:403
dbEvent.c:767
dbEvent.c:715
Camonitor event
Call
Location
clone()
start_thread()
CAS-event task
start_routine(arg)
event_task(pParm)
event_read(ev_que)
read_reply(pArg,paddr,eventsRemaining,pfl)
cas_send_bs_msg(pclient,lock_needed)
send()
libc.so.6
libpthread.so.0
osdThread.c:282
dbEvent.c:941
dbEvent.c:865
camessage.c:627
caserverio.c:63
libc.so.6
caget from an ai record
Call
Location
clone()
libc.so.6
start_thread()
libpthread.so.0
CAS-client task
start_routine(arg)
osdThread.c:282
camsgtask(pParm)
camsgtask.c:123
camessage(client)
camessage.c:2508
read_notify_action(mp,pPayload,client)
camessage.c:757
read_reply(pArg,paddr,eventsRemaining,pfl)
camessage.c:557
db_get_field(paddr,buffer_type,pbuffer,no_elements,pfl) db_access.c:543
dbGetField(paddr,dbrType,pbuffer,options,nRequest,pflin) dbAccess.c:989
dbGet(paddr,dbrType,pbuffer,options,nRequest,pflin)
dbAccess.c:1009
getOptions(paddr,poriginal,options,pflin)
dbAccess.c:393
dbFastGetConvertRoutine[field_type][dbrType]
dbAccess.c:1069
(localAddr.pfield, pbuffer, &localAddr)
The end
• What did I miss?
–
–
–
–
–
–
–
–
–
–
–
–
IOC initialisation
Database definition
Static database access
dbStatic
Test facilities
Access security
IOC shell
Registry
OSI
libCom
Build rules
Record, device and driver support
• Basically everything.....
Resources
• EPICS Application Developers Guide
• Channel Access Protocol Specification
– http://epics.cosylab.com/cosyjava/JCACommon/Documentation/CAproto.html
• Training slides on EPICS web site
• Eclipse is your friend...
• “module load ddd” gives you the ddd graphical
debugger.