“Resource Management on Keystone
Devices" by Justin Sobota
1/16/13
4/8/2015
TI Confidential – NDA Restrictions
1
Agenda
• Why is Resource Management Needed on
Keystone Devices?
• Keystone I Resource Manager (RM) LLD
• Keystone I RM Lessons Learned
• Keystone II RM
• Q&A
• Demo
4/8/2015
TI Confidential – NDA Restrictions
2
Why is Resource Management
Needed on Keystone Devices?
•
Resource sharing problems introduced with release of multicore devices based on
Keystone architecture
–
Resource contention was managed by LLDs on DSP-only multicore solutions
•
–
NySh
Resource contention not managed on (ARM + DSP) solutions
•
•
ARM (Linux kernel) doesn’t use LLDs
Resource contention between ARM & DSPs – running Linux, BIOS, and bare-metal
–
•
Combination of SYS/BIOS and Keystone IP initialization uncovered sharing problems at
application/task level
–
–
RTSC allows for plugging of pre-main startup routines
Arbitration of IP hardware startup when integrating an existing post-main application with app
that utilizes SYS/BIOS pre-main capabilities
•
•
No easy way to sync resource usage between ARM and DSP cores
OpenMP & QMSS
Keystone II architecture introduces devices with multiple ARMs, as well as
DSPs
–
4/8/2015
Resource sharing problems will only get worse if not solved in a robust manner
TI Confidential – NDA Restrictions
3
Keystone I Resource Manager (RM) LLD
• Prevents DSP applications from stepping on resources taken by ARM
Linux
• Initialization and usage permissions for select LLD resources defined
in user-defined resource table
– QMSS, CPPI, and PA LLDs
• Operates under hood of LLDs
– Resource init and usage requests made between LLDs and RM directly
• LLD operates normally if resource request “allowed”
• LLD returns error if resource request “denied”
– Application must take steps to handle
• Permission tables located in shared memory
– Restricts RM to DSP cores only
• LLD resource storage still in hands of LLD
– Full backwards compatibility with applications not utilizing RM
Quick, “dirty” resource management solution for Keystone I
devices
4/8/2015
TI Confidential – NDA Restrictions
4
Keystone I RM Lessons Learned
• Very tedious to add new resources since RM has static
definitions in both code and data of resources it manages
• Privileges for DSP cores only skims surface
• Resource table with privileges defined at compile time
– Privileges cannot be manipulated at runtime
• System integrator must align Linux DTB resources with
DSP RM resource table
– Tedious and error-prone
• Shared memory architecture provides no communication
path to ARM
– Not easily portable to new devices
4/8/2015
TI Confidential – NDA Restrictions
5
Keystone II RM – Major Requirements
• Remove static definitions of managed resources from RM
– Easy addition of new resources
• Enable management of resources at all levels within
system software architecture
– Core, task, application component (LLD)
– Pre/post-main execution
• Runtime modification of resource permissions
• Automate reservation of resources taken by Linux kernel
• Generic, processor-independent transport interface that
allows RM instances to communicate regardless of device
hardware architecture
– Transport glue logic provided by application
– Easy to port RM to new devices
4/8/2015
TI Confidential – NDA Restrictions
6
Keystone II RM - Overview
– Instance-based Client/Server Architecture:
•
•
•
Three instance hierarchy:
– RM Server – Global management of resources and permission policies
– RM Client Delegate (CD)
» Offloads management of resource subsets from Server.
» Provides singular data path to Server
– RM Client – Provide resource services to system software elements
Resource services provided via instance service API
No restrictions on RM instance topology within system
– RM Instances Communication Over Generic Transport Interface
•
•
Application must setup data paths between RM instances
Allows RM to run on any device architecture without modification to RM source
– Resources Tracked by RM Defined in Global Resource List (GRL)
•
•
•
GRL captures all resources that will be tracked for a given device
Based on device spec and CSL
Facilitates automatic extraction of resources used by ARM Linux from Linux DTB
– Policies Specify RM Instance Resource Privileges
•
•
Resource initialization, usage, and exclusive right privileges assigned to RM instances
– Resource assignment to RM instances allows resource management at all software system levels
Runtime modification of policy privileges
– APIs and Linux CLI (Planned)
– Resources Stored within Balanced Search Tree Allocators
•
•
4/8/2015
Reduce memory usage and resource lookup times
Allocators facilitated by NameServer
TI Confidential – NDA Restrictions
7
Keystone II RM - Overview
ARM/DSP n
ARM/DSP n+1
User Mode (ARM)
Global
Resource
List
(GRL)
Resource
Policies
Linux
DTB
Memory
Allocator
Available
resources are
inverse of
Linux DTB
QMSS
CPPI
RM Server Instance
RM CD Instance
Allocation
policies
QMSS
Resource
Allocators
CPPI
PA
Service
CD Service
Transaction
Handler
Resources
Allocated
from Server
CD Service
Transaction Handler
Transport API
ARM  DSP Transport
PA
Service
Etc
Port
DSP  DSP Transport
Port
Etc
Transport-Specific Data Path
Transport API
ARM  DSP Transport
DSP  DSP Transport
QMSS
DSP  DSP Transport
QMSS
Transport API
CPPI
Client Service
Transaction Handler
PA
Mem Alloc
Service
Port
RM Client Instance
Service
Port
RM Client Instance
Etc
ARM/DSP n+2
4/8/2015
Client Service
Transaction Handler
PA
Mem Alloc
Etc
TI Confidential – NDA Restrictions
Transport API
CPPI
ARM/DSP n+3
8
Keystone II RM - Instances
Server
•
•
•
•
•
•
•
•
•
Assigned resource
permissions in policy based
on instance name given at
instance instantiation
Can satisfy standard service
requests via service API
Can satisfy pre-main service
requests
Manages all system
resources
Multiple Servers within
system must manage
mutually exclusive sets of
resources
Manages all system policies
Maintains NameServer
Manages resource recovery
in case of fault in CD or Client
instance (Planned)
Can register with any number
of CDs and Clients
4/8/2015
TI Confidential – NDA Restrictions
Client Delegate (CD)
•
•
•
•
•
•
•
Assigned resource
permissions in policy based
on instance name given at
instance instantiation
Can satisfy standard service
requests via service API
Can satisfy pre-main service
requests
Manages subset of system
resources provided by Server
(Planned)
Provided a sub-policy that is
sync’d with Server level policy
(Planned)
Can register with any number
of Clients
Can register with at most one
Server
Client
•
•
•
Assigned resource
permissions in policy based
on instance name given at
instance instantiation
Can satisfy standard service
requests via service API
Can register with at most one
CD or Server
9
Keystone II RM – Instance Topology
Example 1
DSP Multicore Application
IPC
BIOS
LLD
Etc
RM Server
LLD
Etc
LLD
RM Client Delegate
RM Client
DSP 1
DSP 2
Etc
Linux User-Space
ARM
4/8/2015
TI Confidential – NDA Restrictions
10
Keystone II RM – Instance Topology
Example 2
DSP Multicore Application
LLD
Etc
IPC
BIOS
RM Server
RM Client
Delegate
LLD
LLD
Linux User-Space
RM Client
RM Client
ARM
4/8/2015
TI Confidential – NDA Restrictions
DSP
11
Keystone II RM - Services
•
Services available from any RM instance
–
•
Instance name used as the basis for permission checks
Current services
–
Allocate (initialization, usage)
•
•
•
–
Free
•
•
–
–
–
•
•
Allocate static resource
Allocate resource of unspecified base and alignment, length must be specified
Allocate resource based on existing NameServer name
Free static resource
Free resource based on existing NameServer name
Map resource(s) to NameServer name
Get resource(s) tied to existing NameServer name
Unmap resource(s) from existing NameServer name
Non-blocking service requests directly return result
Blocking service requests return ID to system
–
–
4/8/2015
Result of service returned at a later time via callback function provided with original service
request
Service result ID will match ID returned to system at time of request
TI Confidential – NDA Restrictions
12
Keystone II RM – Transport Interface
• RM instances communicate via generic transport API
– Application provides transport path between RM instances
– Transport paths registered with RM instances
• Destination RM instance type specified
• Transport path send/receive function pointers provided
• Management of transports between RM instances handled
by application
• Facilitates ability of RM to run on any device topology
• RM source code porting not required in order to run on
new devices and architectures
4/8/2015
TI Confidential – NDA Restrictions
13
Keystone II
RM Transport
Example
ARM
DSP 2
RM Server
RM Client 1
transHandle
dstType
transHandle
dstType
0x0001
Client
Delegate
0x0001
Client
Delegate
RM Transport
APIs
transHandle
Transport
0x0001
ARMDSP
RM Transport
APIs
App Transport
Routing Map
App Transport
Routing Map
transHandle
Transport
0x0001
DSPDSP
ARMDSP
Transport
DSPDSP
Transport
rmHandle
transHandle
Transport
0x0001
0x0001
DSPARM
0x0001
0x0002
DSPDSP
0x0001
0x0003
Internal
0x0002
0x0001
Internal
App
Transport
Routing Map
RM Transport
APIs
transHandle
dstType
0x0001
Server
0x0002
Client 1
0x0003
Client 2
RM Transport
APIs
transHandle
dstType
0x0001
Client
Delegate
RM Client 2
RM Client Delegate
DSP 1
4/8/2015
TI Confidential – NDA Restrictions
14
Keystone II RM - Global Resource List
(GRL)
• Specified in Device Tree Source (DTS) format
– Open source, dual GPL/BSD-licensed LIBFDT used for parsing GRL
• Input to Server on initialization
• Server instantiates allocator for each resource specified in GRL
• A GRL specification for a resource includes:
–
–
–
–
Resource name
Resource range (base + length)
Linux DTB alias path (if applicable)
Resource NameServer assignments (if applicable)
• Permissions not specified in GRL
4/8/2015
TI Confidential – NDA Restrictions
15
Keystone II RM - GRL Example
/dts-v1/;
/ {
device-name = "TCI6614";
/* Device Resource Definitions based on HW spec and CSL */
timers {
resource-range = <0 16>;
ns-assignment = “Core_0_Watchdog”, <0 1>;
};
qmss {
pdsps {
resource-range = <0 2>;
};
memory-regions {
resource-range = <0 20>;
linux-dtb-alias = "hwqueue@2a00000 regions region-12 id", <1 0>;
};
accumulator-ch {
resource-range = <0 48>;
};
gp-queue {
resource-range = <896 7296>;
};
};
};
4/8/2015
TI Confidential – NDA Restrictions
16
Keystone II RM - Policies
• Specified in Device Tree Source (DTS) format
– Open source, dual GPL/BSD-licensed LIBFDT used for parsing GRL
• Assign resource permissions to RM instances
• Available resource permissions:
– Initialization (i) – Allocate resource for initialization
– Usage (u) – Allocate resource for use
– Exclusive (x) – Resource not shared, can be allocated to no more than
one RM instance
– Linux Shared (s) – Resource reserved by Linux kernel but can be shared
with specified RM instances
• Policies can be modified, created, and deleted runtime (Planned)
– Policy APIs available to application
– Linux CLI
4/8/2015
TI Confidential – NDA Restrictions
17
Keystone II RM - Policy Example
/dts-v1/;
/* RM Server global policy */
/ {
/* All RM instances expected to be resource assignees within the policy. RM will fail at init if
* it finds a resource assignee that is not in the user-instances list */
valid-instances = "RM_Server",
"RM_Client_Delegate",
"RM_Client";
/* RM will deny any resource requests for resources not defined in the policy. */
/* Format for assigning resources to specific RM instances */
qmss {
gp-queue {
assignments = <2000 1000>, "iu=(RM_Server RM_Client_Delegate RM_Client)",
<3000 1>, "iux=(RM_Server) & iu=(RM_Client_Delegate RM_Client)";
<4000 1>, “s=(RM_Client_Delegate RM_Client)”;
};
};
cppi {
pass-rx-flow-id {
assignments = <0 20>, "iux=(*)";
};
};
};
4/8/2015
TI Confidential – NDA Restrictions
18
Keystone II RM Name
Synchronization
Global Resource List (GRL)
/ {
…
myFirstResource {
resource-range = <0 20>
/* 20 myFirstResource’s in system */
};
mySecondResource {
resource-range = <0 50>;
/* Alias path must be path to value in Linux DTB */
linux-dtb-alias = “hw 2nd-res values”, <2 0 1>;
};
};
RM Instance Policy
/ {
};
Resource
Manager
- myFirstResource
- mySecondResource
Resource Names
must be
synchronized!
Services API
valid-instances = “myRmInst”,
“myOtherRmInst”;
…
myFirstResource {
/* All myFirstResource can be
* used by anyone in system */
assignments = <0 20>, “iu=(*);
};
mySecondResource {
/* All mySecondResource is split
* between the two valid RM
* instances in system */
assignments = <0 25>, “iu=(myRmInst)”,
<25 25>, “iu=(myOtherRmInst)”;
};
RM Service Port from RM Instance
“myRmInst”
char resName[] = “myFirstResource”;
serviceInfo.resourceName =
&resName[0];
serviceInfo.resourceIndex= 4;
(*rmService)
(&serviceInfo);
hw {
2nd-res {
/* Kernel takes mySecondResource
* in range 20 - 30 */
values = <20 10>;
};
3rd-res {
…
};
};
Linux DTB
4/8/2015
TI Confidential – NDA Restrictions
19
BACKUP
4/8/2015
TI Confidential – NDA Restrictions
20
Keystone I Resource Manager (RM) LLD (cont’d)
Core 0
Core 1
App
App
Queue/cppiChOpen/ …
Rm_init
QMSS LLD
CPPI LLD
Queue/cppiChOpen/ …
PA LLD
Rm_start
QMSS LLD
CPPI LLD
resource_table.h
rmResourceTable{
{RES_ID, START, END},
{ …, …, … },
…
}
Init/usePermissionCheck
Init/usePermissionCheck
RM
Permission
validation
Permission
validation
Rm_init maps
rmResourceTable to
internal Permission
Tables
…
Shared Permissions
21
TI Confidential – NDA Restrictions
RM
PA LLD
Keystone II RM - Architecture
Task
Application Code
IPC
LLDs
BIOS
Linux Kernel/XDC
Runtime/BIOS
BIOS and LLD Internal
RM Callout APIs
Service API
OSAL
Other OS
Resource Manager (RM)
Transport API
DSP<->DSP
Application
Transport
4/8/2015
TI Confidential – NDA Restrictions
Allocation Policy CLI
(ARM Linux Only)
OS Abstraction
Layer OSAL APIs
ARM<->DSP
Application
Transport
ARM<->ARM
Application
Transport
22
Keystone II RM - Architecture
• Key RM Interfaces
– Service API
• Service ports opened from the RM service API
• Service ports provide resource services to system entities
• Services will be labeled as originated from the RM instance from
which the service port originated
– Transport API
• Generic transport interface over which RM communicates between
instances
• Application registers instances with one another over transport API
– Application provide transport glue function API to RM instances at
registration
• RM calls registered transport glue functions when sending/receiving
RM packets between RM instances
4/8/2015
TI Confidential – NDA Restrictions
23
RM Server
• One RM server for a given exclusive set of system resources
• Maintains global view of allocator, policy, and NameServer states
• Can satisfy pre-main resource service requests
– OpenMP, BIOS, IPC, etc.
• Maintains the global policy that defines the resource permissions for
all RM instances
– Policy synchronization with policies stored on CDs not yet implemented
• Maintains all resource allocators
– Who has which resources
• Maintains RM NameServer
4/8/2015
TI Confidential – NDA Restrictions
24
RM Client Delegate (CD)
•
•
•
Acts as a proxy for RM Server to RM Clients
Quick response mechanism to Clients for certain services (Planned)
Can be used as single path to Server when transport path to server is longer
than transport path to Clients
– Instead of all Clients connecting to Server over a high cycle transport path they can
connect to CD
– CD can allocate resources that have been provided it by the Server (Planned)
• Will request new block of resources from Server when it runs out of resources to allocate
to Client service ports and service ports hanging off itself (Planned)
• Can satisfy pre-main allocation requests
– Pre-main policy will be relayed to Server after initialization and transport
registration (Planned)
– Currently, the pre-main policy must be reflected in the global resource
policy provided to the Server at initialization
• Due to policy synchronization feature not being implemented
4/8/2015
TI Confidential – NDA Restrictions
25
RM Client
• “Dumb” RM instance
• Sole purpose is to provide gateway to RM
services for system components
• Minimum permissible policy unit
• All service requests offloaded to the registered
RM CD or RM Server
• No restriction on where Clients can exist
4/8/2015
TI Confidential – NDA Restrictions
26
Keystone II RM – Instance Topologies
• Minimal requirements for RM instances within a
system
– If there are multiple RM Servers exist within a system
the resources they track must be mutually exclusive
– RM CDs must be connected to at most one RM Server
– RM Clients must be connected to at most one RM
Server or RM CD
• No restrictions on where RM instances can be
created
• Service ports can be opened from any RM
instance type
4/8/2015
TI Confidential – NDA Restrictions
27
GRL and Policy Compilation
Device-specific
Global Resource List
(GRL)
and any Application
Policies
DTC Executed on Host
(Linux)
Human
Readable
Device
Tree
Source
(DTS)
File
Device Tree
Compiler
(DTC)
Converted
Binary GRL
Converted Binary
Application Policy
Compiled
GRL Device
Tree Blob
(DTB)
Compiled
Policy
Device Tree
Blob (DTB)
Application
Code
RM Library
(compiled
and linked
with libfdt)
Compile Full Application
Rm_init(… *dtbPointer …)
Global
Resource
DTB
Data from
GRL DTB
used to
initialize
resource
allocators
RM Server Init
libfdt
APIs
Policy data
is validated
at init.
Extracted
later to
check
permission
s
4/8/2015
TI Confidential – NDA Restrictions
Full
Application
Binary
Policy DTB
28
Porting RM to New Devices
• Required to port RM to new device
– Update GRL to exclude/include device-specific
resources that are to be tracked by RM
– Update Policy resources permissions to match new
GRL resources
– Update GRL with any new resource path alias to
reserved resources in Linux DTB
• No need to modify RM source
• Application source will need to change
4/8/2015
TI Confidential – NDA Restrictions
29
To Do
• Fault management
• Policy editing Linux CLI
• Remove RM NameServer
– Add NameServer interface so that an existing
NameServer can be registered with RM
• Resource and policy front-loading on RM CD
4/8/2015
TI Confidential – NDA Restrictions
30
Linux DTB Resource Extraction
• Cannot rely on Linux DTB to match GRL resource
specification exactly and vice versa
• Alias path is clean way of extracted Linux reserved
resources from Linux DTB
• Alias path specifies node path down to node containing
value that should be reserved by RM for Linux kernel
• Resources specified as reserved by the Linux kernel are
not able to be used by any RM instance unless otherwise
specified in the Policy
4/8/2015
TI Confidential – NDA Restrictions
31
Keystone II RM - Name
Synchronization
• Key to abstraction of static resource definitions away from RM
• Global Resource List (GRL) defines each resource that RM will track
and how many of that resource exist
• Policies define which RM instances are allowed to use the resources
defined in the GRL and how the instances can use them
• Utilize Linux Device Tree framework specification, ePAPR, to define
easy to read, modify, and parse GRL and policy specifications
– Facilitates extraction of resources used by Linux kernel from Linux DTB
file
– LIBFDT library included by RM to read, parse, and modify the GRL and
Policy DTBs
• Dual GPL/BSD licensed
4/8/2015
TI Confidential – NDA Restrictions
32
Allocators
• RM Allocators implemented using a open-source binary
balance search tree algorithm
– BSD-licensed
• Resource allocator nodes specify a resource base and
length as well as a list of RM instances the resource is
allocated to
• Wrapper written around the tree algorithm that expands
and contracts the allocator tree nodes as resources are
allocated and freed
• Saves time when searching for resources to allocate
• Saves space when RM can be potentially managing
thousands to millions of resources
4/8/2015
TI Confidential – NDA Restrictions
33
Allocators
- Example
• Not exact to
tree
algorthm
Assumptions:
-System has 1024 GP
Queues
1
-Allocate queues 335 – 345
-Allocate queue 767
-Policy specifies blocks of
256
qmss-gp-queues
1024
qmss-gp-queues
1024
0-255
Free
2
256-511 512-767 768-1023
Free
Free
Free
0-255
Free
256-511 512-767 768-1023
Parent
Parent
Free
256-334 335-345 346-511 512-766
767
Free
Reserved
Free
Free
Reserved
-Free queues 335 – 345
3
-Allocate queues 384-400
qmss-gp-queues
1024
0-255
Free
256-511 512-767 768-1023
Parent
Parent
Free
256-345 346-511 512-766
767
Free
Parent
Free
Reserved
4/8/2015
TI Confidential – NDA Restrictions
346-383 384-400 401-511
Free
Reserved
Free
34
RM NameServer
• Internal, simple NameServer
• Not intended for long term use
• Allows association of resources with a user-specified
name
• System entities can use service ports to map and unmap
resources to names, and allocate/free resources by name
– Instance permissions will not be checked against the name given
the resource but the resource itself
• Planning for future revisions of RM to have external NameServer
where RM would not have knowledge of names
• Aware of growing of NameServers in SDK offering
– Plan to remove RM NameServer and replace with API allowing for
hookup to an external, existing NameServer
4/8/2015
TI Confidential – NDA Restrictions
35
Pre-Main Resource Allocation
• Available on Servers and CDs
• Startup policy checked on pre-main allocations
• After initialization startup policy will be synced with
Server and along with any allocations
– Startup policy must be in sync with global policy given
to server at init time
4/8/2015
TI Confidential – NDA Restrictions
36