Open Source | Open Possibilities
Building the Linux
Kernel and User Space
for the Hexagon™ DSP
with LLVM
Anshu Dasgupta and Pavel Potoplyak
Linux Plumbers Conference
September 19, 2013
Credits: Thomas Brezinski and Anand
Kodnani
PAGE 1
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Qualcomm Innovation Center, Inc.
5775 Morehouse Drive
San Diego, CA 92121
U.S.A.
© 2013 Qualcomm Innovation Center, Inc.
Not to be used, copied, reproduced, or modified in whole or in part, nor its contents revealed in any manner to others without the express
written permission of Qualcomm Innovation Center, Inc.
Qualcomm is a trademark of QUALCOMM Incorporated, registered in the United States and other countries. All QUALCOMM
Incorporated trademarks are used with permission. Other product and brand names may be trademarks or registered trademarks of their
respective owners.
This technical data may be subject to U.S. and international export, re-export or transfer (“export”) laws. Diversion contrary to U.S. and
international law is strictly prohibited.
Presentation Title
80-BAxxx-x Rev. x
PAGE 2
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Introduction
Who are we?
 Anshu Dasgupta – manage Qualcomm’s Hexagon™ DSP compiler team
 Pavel Potoplyak – engineer working with the Hexagon tools team
PAGE 3
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Introduction
What are we doing?
Bringing up a LLVM compiler for the Hexagon™ DSP
 GCC is a mature toolset with thousands of hours of testing behind it
 How do we achieve that level of robustness and maturity?
One solution: Build Linux kernel and user space with LLVM for the Hexagon™ DSP
PAGE 4
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Introduction
Why are we building the Linux kernel and user space with LLVM?
Compiler testing
•
Need extensive testing to deploy commercially
•
VLIW code paths and optimizations a challenge for correctness, performance
•
Compiler bugs are very difficult to track down in embedded applications
Hexagon codebases transitioning from GCC to LLVM
•
PAGE 5
Programmers code to compiler behavior not to C standard
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Hexagon™ DSP
Hexagon™ DSP: Qualcomm’s multithreaded VLIW DSP
•
Part of the Snapdragon platform
C, C++ compiler
•
Processor designed to be programmed in C and C++
•
LLVM-based compiler (transitioned from GCC compiler)
VLIW architecture provides several opportunities and challenges for the compiler
PAGE 6
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Compiler Correctness
Primary goal of compiler: Generate correct code
•
Relax correctness and any performance goal can be met :-)
•
We have an elaborate set of internal tests for correctness that runs
every night
Compiling Hexagon DSP Linux kernel and user space with LLVM

Stress test for compiler correctness
•
Several bugs uncovered during Hexagon Linux bringup
Example packet:
{
r7 += mpyi(r21, r20)
r8 = add(r8, #16)
r12 = add(r12, #8)
if (cmp.gt(r2, r12.new)) jump:t .LBB0_4
}
PAGE 7
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Example of Unsupported Feature: Global Register Variables
#ifndef __llvm__
register struct thread_info *__current_thread_info asm(QUOTED_THREADINFO_REG);
#define current_thread_info()
__current_thread_info
#else
inline struct thread_info *current_thread_info()
{
struct thread_info *x;
asm ("%0 = " QUOTED_THREADINFO_REG : "=r"(x));
return x;
}
#endif
Several other GCC extensions not implemented in LLVM

http://clang.llvm.org/docs/UsersManual.html#gcc-extensions-notimplemented-yet
PAGE 8
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Example of Compiler Exploiting C99 Undefined Behavior
char *killer = NULL;
#ifndef __llvm__
*killer = 1;
#else
__asm__ __volatile__(
"r0 = #0\n\t"
"r1 = #1\n\t"
"memb(r0+#0) = r1\n\t"
::: "r0", "r1", "memory");
#endif
PAGE 9
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Esoteric Difference #1: File Scope Inline ASM
void sleep1()
{
sleep(1);
}
/* notice nop placed between two functions */
asm("nop");
void sleep2()
{
sleep(2);
}
PAGE 10
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Esoteric Difference #1:
GCC code generation with –fno-toplevel-reorder
.file
"example1.c"
.text
.type
sleep1, @function
sleep1:
allocframe(#0)
r0 = #1
call sleep
deallocframe
jumpr r31
.size
sleep1, .-sleep1
//APP
nop
# NOTICE NOP PLACEMENT
//NO_APP
.p2align 2
.globl sleep2
.type
sleep2, @function
sleep2:
allocframe(#0)
...
PAGE 11
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Esoteric Difference #1: LLVM code generation
.file
"example1.c"
nop
# NOTICE NOP PLACEMENT
.text
.type
sleep1,@function
sleep1:
allocframe(#8)
r0 = #1
call sleep
memw(r29+#4) = r0
dealloc_return
.Ltmp0:
.type
sleep1,@function
allocframe(#8)
...
PAGE 12
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Esoteric Difference #2: Controlling Names in Assembler Code
void f1()
{
f2();
}
extern int f2() asm ("f3");
PAGE 13
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Esoteric Difference #2: GCC code generation
.file
"example2.c"
.text
.p2align 2
.globl f1
.type
f1, @function
f1:
// saved LR + FP regs size (bytes) = 8
allocframe(#0)
call f3
# NOTICE CALL TO f3
deallocframe
jumpr r31
.size
PAGE 14
f1, .-f1
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Esoteric Difference #2: LLVM code generation
.file
"example2.c"
.text
.globl
f1
.falign
.type
f1,@function
f1:
allocframe(#8)
call f2
# NOTICE CALL TO f2
memw(r29+#4) = r0
dealloc_return
.Ltmp0:
.size
PAGE 15
f1, .Ltmp0-f1
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Esoteric Difference #3: Preprocessing
$ echo '#include "foo.h"' | \
hexagon-clang -E -dM -MD -MP -MF foo.out -xc - -o
out.out -MT 'a b'
$ cat foo.out
a b: <stdin> foo.h
foo.h:
$ echo '#include "foo.h"' | \
hexagon-gcc -E -dM -MD -MP -MF foo.out -xc - -o
out.out -MT a b'
$ cat foo.out
a b:
foo.h
foo.h:
PAGE 16
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Conclusion
Our conclusion
 Built more than 55 user space packages with LLVM for Hexagon DSP
 Uncovered several latent compiler bugs
 Buiding and running large codebases through toolset is extremely
beneficial
 Significantly improves the quality of the compiler shipped to our
customers
 Would like to automate the build without any source code changes
PAGE 17
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION
Open Questions
How do you quantify the quality of the toolset, kernel and user space?
•
Components change. For instance: new compiler, new optimizations
•
What are the different dimensions of measurable quality
•
•
Correctness
•
Build time
•
Code size
•
Execution time
How do we capture, archive, and analyze these metrics on a macro scale?
Which GCC-isms should LLVM support? Which GCC-isms should be
eliminated from Linux user space?
PAGE 18
Open Source | Open Possibilities
MAY CONTAIN U.S. AND INTERNATIONAL EXPORT CONTROLLED INFORMATION