introduction to CTS
Sep 2, 2010
Jingtao
agenda

back ground

CTS history

CTS work flow

download, install and run CTS

check CTS report

typical fail cases
background

Google define an “Android compatibility” device as one that can run any application written by
third-party developers using the Android SDK and NDK.

Device that are properly compatible can seek approval to use Android trademark and Android
Market client software.
5
how to verify compatibility
 the Android Compatibility Definition Document(CDD) spells out the specific device
configuration that will be considered compatible.
 CTS(an automatically compatibility test tool) test
 Third party application test
6
the process to get certification
 Google release CTS according to each Android platform publicly.
 manufacturer run CTS on their device.
 manufacture fix any bug found by CTS.
 if manufacture think any CTS case is wrong, he should upload the
CTS report and detail description to Google's database.
 if Google accept manufacture's description, he will grant wavier to
that device.
 Google will think the device is Android compatible if it pass all the
CTS case or all the fail cases have waiver.
7
requirement in CDD
• software API/UI compatibility
• reference app test
• application packaging compatibility
• multimedia compatibility
• developer tool compatibility
• hardware compatibility
• performance compatibility
• security model compatibility
• system update
hardware compatibility
•
display
•
keyboard
•
Non-touch navigation
•
screen orientation
•
touch-screen input: must have a touch screen
•
USB: must support adb and mass storage mode
•
navigation keys: function of home, menu and back must be available to user at any time.
•
wireless data networking: must support for at least one wireless data standard capable of 200kbit/sec or greater.
•
camera: must include a rear-facing camera, at least 2 megapixels
•
accelerometer: must include a 3-axis accelerometer and must be able to deliver events at 50 Hz or greater.
•
compass: must include a 3-axis compass and must be able to deliver events 10 Hz or greater.
•
GPS: must include a GPS receiver
•
telephony
•
memory and storage: /data partition must be at least 150MB and must have at least 92MB memory.
•
application shared storage: must be at least 2GB in size.
•
Bluetooth: must include a Bluetooth transceiver and enable the RFCOMM-based Bluetooth API.
What is CTS

Compatibility Test System

open source

commercial-grade(20000+ cases)

Automation compatibility test tool
what does it test
• Signature
• Platform API
• Dalvik VM
• Platform Data Model
• Intents
• Permissions
• Resources
CTS History
• Android1.0 - 16784 cases
• Cupcake - 21383
• Donut - 22756
• Eclair r3 - 23138
• Froyo r2 - 23830
CTS work flow
download CTS

http://source.android.com/compatibility/downloads.html
1, git ls-remote --heads git://192.168.1.250/android-mirror/platform/manifest.git
...
refs/heads/android-cts-2.1_r3
refs/heads/android-cts-2.1_r4
refs/heads/android-cts-2.2_r1
refs/heads/android-cts-2.2_r2
...
2, repo init -u git://192.168.1.250/android-mirror/platform/manifest.git -b android-cts-2.2_r2
3, repo sync cts
build CTS
• svn://192.168.1.250/oms/branches/horse
make BUILD_CTS=true cts
install CTS

install Android SDK

unzip android-cts-2.1_r3-x86.zip
android-cts
|-- docs
|-- repository
`-- tools
|-- cts.jar
`-- startcts

edit android-cts/tools/startcts, set

SDK_ROOT=<absolute fold of Android SDK>
run CTS
connect phone to PC
android-cts/tools/startcts to enter CTS shell
3 mode to start test:

plan mode

package mode

case mode
Sample of CTS commands
• start --plan CTS
• start --plan Android -p
android.app.cts.ActivityManagerTest
• start --plan Android -t
android.app.cts.ActivityManagerTest#testGe
tRunningServices
what does CTS do
install test case apk
(adb -s deviceID install -r ...apk)
execute test
(am instrument -w -e [para] <component>...)
uninstall test case apk
(adb -s deviceID uninstall ...)
run CTS under Windows
1. unzip CTS package and android SDK package to your windows PC.
2. Modified the SDK_ROOT in cts\android-cts\tools\startcts :
SDK_ROOT=D:/CTS/sdk/android-sdk_r05-windows/android-sdk-windows/tools
3. use below command to enter CTS shell:
java -Xmx512M -cp D:\CTS\cts\android-cts\tools\cts.jar; D:\CTS\cts\androidcts\tools\hosttestlib.jar;D:\\lib\ddCTS\cts\android-cts\tools\junit.jar;D:\CTS\sdk\android-sdk_r05windows\android-sdk-windows\tools\lib\ddmlib.jar com.android.cts.TestHost D:\CTS\cts\androidcts\repository\host_config.xml
Please change red marked to you current file path.
known issue:
The file (cts_result.css, cts_result.xsl, logo.gif, newrule-green.png ) under CTS result can not
automatically copy in new session folder.
Attention:
If you need to clear the data in addon emulator, Please use: emulator @addon -wipe-data
config phone before run CTS
empty SD card

never Screen Timeout

English locale

install and set input method to LatinIME

open WIFI and GPRS

correct date/time

/opl/etc/mountd.conf, system.prop

CtsDelegatingAccessibilityService.apk
test report

firefox repository/results/20xx.../testResult.xml
Fail Cases
Case Study: CTS on FBW1.4


Waivered

Internal resource ID: 38 cases

Locale related: 11 cases

Others: 4 cases

Fixed certification time: 3 cases

Fixed camera parameter: 1 case

UI related( For example, check box layout): 6 cases

Mock Locations: 22 cases (just test in eng build)
Waiting for waiver

Google Search dependant: 2 cases

no popup window when application force close
Public Resource ID
• http://developer.android.com/reference/andr
oid/R.html
• Frameworks/base/core/res/res/values/public
.xml
public static final int
accelerate_decelerate_interpolator
Constant Value: 17432580 (0x010a0004)
Internal Resource ID
out/target/common/R/com/android/internal/R.java
com.android.internal.R.id.alertTitle
cupcake: 0x0102014e
borqs:
0x01020167
168
String expected = dateFormat.format(mCalendar.getTime());
169
TextView tv = (TextView)
d.getWindow().findViewById(com.android.internal.R.id.alertTitle);
170
assertEquals(expected, tv.getText());
resource ID
• R.id.contentPanel
• com.android.internal.R.id.topPanel
• R.id.alertTitle
• R.id.custom
Mock Location
• 84
// test that mock locations are allowed so a more descriptive
error message can be logged
• 85
• 86
if (Settings.Secure.getInt(mContext.getContentResolver(),
Settings.Secure.ALLOW_MOCK_LOCATION, 0) == 0) {
• 87
fail("Mock locations are currently disabled in Settings - this
test requires "
• 88
+ "mock locations");
• http://androidappdocs.appspot.com/guide/topics/location/index.html
• during initial development in the emulator, you may not have access to
real data from a real location provider (Network or GPS). In that case,
it may be necessary to spoof some data for your application using a
mock location provider.
Settings-> Applications-> Development
check box
camera
• android.hardware.cts.CameraTest#testAccessParameters
junit.framework.AssertionFailedError: expected:<2048> but was:<2560> at
android.hardware.cts.CameraTest.assertParameters(CameraTest.java:392)
381
private void assertParameters(Parameters parameters) {
...
387
final int ORIGINALPICWIDTH = 2048;
388
final int ORIGINALPICHEIGHT = 1536;
389
390
// Before Set Parameters
391
assertEquals(PixelFormat.JPEG,
parameters.getPictureFormat());
392
assertEquals(ORIGINALPICWIDTH,
parameters.getPictureSize().width);
Case Study: CTS on OPhone 2.0
Case number: 23107 cases
Nexus One
Fail cases:
3
Execut time: 6 h
Oscar-r42369
S1-r49137
34
5
6h
4h
fail case summary on Oscar
• internal resource ID: 17
• UI: 8. (check box, grid layout, list padding, pixel
buffer)
• phone number utils: 4
• network type: 1
• SD card write permission: 1
• CTS case issue: 3
phone number util cases(92585)
• android phone do not have sys.min.match.digits but can pass
• set sys.min.match.digits to 7 can pass the case but phone will not
match incoming call correctly.
• e.g. create user A(tel: 01150539) and user B(mobile:13701150539) in
sequence in Contacts, when mobile 13701150539 incoming call,
phone will always show user A incorrectly on screen.
Test Package
android.telephony.cts.PhoneNumberUtilsTest
Test Cases
testCallMethods
android.telephony.cts.PhoneNumberUtilsTest
testCompareLoosely
android.telephony.cts.PhoneNumberUtilsTest
testCompareStrictly
android.telephony.cts.PhoneNumberUtilsTest
testJudgeMethods
comments
According to CTA, we need
to set sys.min.match.digits
to 11. These 4 cases will
pass if set the value to 7.
Sdcard write permission case
• android.permission.cts.NoSdCardWritePermissionTest
• android.permission.WRITE_EXTERNAL_STORAGE
uid=1000: sysetm user
gid=1015: Sdcard write group
SDcard: android
075
oms(before)oms(fixed)
777
775
Ipv6 related fail
• tests.api.java.net.SocketTest#test_getLocalAddre
ss
junit.framework.AssertionFailedError:
ANY address not returned correctly (getLocalAddress) with
preferIPv6Addresses=true,
preferIPv4Stack=false
0.0.0.0/0.0.0.0:49380 at
tests.api.java.net.SocketTest.test_getLocalAddress(Socket
Test.java:645)
known failure actually passed
• android.provider.cts.ContactsTest#testGroupMembershipTable
bug 2258907, needs investigation
will pass after install gms. the gms package link:
smb://192.168.0.146/share/gms
user: b184
pass: zhanghui
rule to pass CTS

Do not modify framework.

review with SDK team before modify

run related CTS cases after modify
summary
• CTS can block SA
• manufacture will submit ticket relate to CTS
• we can help you setup CTS env
41