Cosc 5/4730
Mobile Media API (MMAPI) v1.1 JSR
135 and Android media
Part 2: Pictures and Video
Emulators
• The emulators all have problems, much of the
following code has been tested on the actual phones
• Video playback code works, but the video may not
always display in the emulators
• android
– Video/picture capture does not work as documented.
• Blackberry
– Camera, you can choose a picture with the emulator to use
or some emulators will use your background image
– Video capture does not work as documented.
Blackberry MMAPI
PLAY MEDIA
Video formats.
• MP4
– H.264, MPEG-4 with simple profile level 3, and
H.263
• AVI (MPEG4 profile level 3)
• WMV (simple profile, WMV3)
• at most the video resolution can be the size
screen of the device.
– 30 frames per second
Playing video
• Similar to audio
– use the Player, realize, play, etc…
– But you need a “View Player field”
• same idea of the “View finder” when we get to taking a
picture
– We get the viewer from the VideoControl object.
– We can use the MediaKeyListener for “media” key
events (assuming the phone has them), but there is no
easy way to display controls, like play, pause, etc..
The Basics
• Just like audio, get the player and realize()
player = javax.microedition.media.Manager.createPlayer(
"file:///SDCard/BlackBerry/videos/the-empire.3gp");
player.realize();
• Get the VideoControl pieces and get a “View Player field”
VideoControl videoControl = (VideoControl) player.getControl( "VideoControl");
Field videoField = (Field) videoControl.initDisplayMode(
VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field" );
• add the field to the screen
add(videoField);
• I found it didn’t work unless I went fullscreen. audio worked, but
no video
videoControl.setDisplayFullScreen(true);
• start playing.
player.start();
MediaKeyListener PlayListener
• Use the PlayListener so we figure out when
the media has finished and be able to close
down stuff.
– Plus deal with other issues as necessary
• use the MediaKeyListener to deal with
“standard” blackberry media buttons as well.
For simplicity…
• Since I had to run in full screen
– I added menus to play, pause, stop.
– volume up/down are handled by the
MediaKeysListener, plus the rest media keys too.
• Except they don’t exist on the phones I have access too.
• The mute button on
many BB is a play/pause
toggle button.
Including bold and storm.
Volume Control
• Handled via a VolumeControl object.
volume = (VolumeControl)
player.getControl("VolumeControl");
• int getlevel() and setLevel(int)
• setMute(boolean)
– true mute, false unmute;
seeking
• player.getMediaTime() returns a long that
represents the spot in the media
– 1,000,000 is about 1 sec
• For a 102 second video, the duration is 102,898,000
– since I can’t see smaller then a second, assuming the video is
102 seconds + almost 1 more second.
• player.setMediaTime(long) allows you to
change the position it is playing.
Buffering.
• Get content off the web
– like I do with many examples… except buffering
doesn’t work.
– The advanced Blackberry Development book has
an example of how to create a streaming video
buffer from none streaming media, such mp4 and
3gp files.
Code example
• To run video player example
– Note, it assumes you have an SDCARD and that
file:///SDCard/BlackBerry/videos/the-empire.3gp
this file exists.
– It’s in the /res/ directory.
Using native player
• Blackberry has a native media player.
• You can launch it to play audio/video
– Remember you lose control and you app is now in the
background.
• First get the registry with you full class name.
Registry reg = Registry.getRegistry( "edu.cs4755.VideoCapture");
– Then create a invocation of the file you want to play and invoke
it.
Invocation invocation = new Invocation(
"file:///SDCard/Blackberry/videos/testrecording.3gp");
reg.invoke(invocation);
• If this all reminds you of a Android, it’s very close an intent.
Android android.media
PLAY MEDIA
Supported formats
• In general Android’s support is consistent with other
mobile phones.
• It supports the 3GP (.3gp) and MPEG-4 (.mp4) file
formats.
– 3GP is a video standard derived from MPEG-4 specifically
for use by mobile devices.
• As far as codecs go, Android supports H.263, a codec
designed for low-latency and low-bitrate
videoconferencing applications. H.263 video is
supported in either MPEG-4 (.mp4) or 3GP (.3gp) files.
Android also supports MPEG-4 Simple Profile in 3GP
files (.3gp) as well as H.264.
Android
• First method
– For greater flexibility you will need to use the mediaPlayer
and a surfaceView
• MediaPlayer like the audio and use a surfaceView to display the
video.
• There are examples in the API demo, plus several of the books.
• The second method uses a VideoView to display.
– The VideoView uses a MediaController to start and stop
and provide functions to control the video play back.
– With a MediaPlayer like the audio. prepare(), then start()
– This is one I’ll cover in this lecture.
VideoView
• VideoView is a View that has video playback
capabilities and can be used directly in a
layout.
• We can then add controls (play, pause,
forward, back, etc) with the MediaController.
ViewView example
• Get the ViewView out of the layout
vv = (VideoView) this.findViewById(
R.id.VideoView);
• Setup where the file to play is
Uri videoUri =
Uri.parse(Environment.getExternalStorageDirectory
().getPath() + "/example.mp4");
vv.setVideoURI(videoUri);
• play the video
vv.start();
Adding media controllers.
• Very simple
vv = (VideoView) this.findViewById(R.id.VideoView);
vv.setMediaController(new MediaController(this));
• Now media controls will show up on the
screen.
Using native media player
• Call for an Intent and send it.
Uri data = Uri.parse(VideoFile);
intent.setDataAndType(data, "video/mp4");
startActivity(intent);
• Remember, your app is now in the
background.
Blackberry and Android
DISPLAYING A PICTURE
Displaying Pictures
• For both android and blackberry
– See code already covered to how display pictures.
• bitmapfield, ImageView, etc…
Blackberry MMAPI
TAKING A PICTURE
Taking a picture.
• Taking a picture requires that you get a player
and then VideoControl, finally create what
amounts a view finder.
– Then you can take the picture, which comes back
as bytes (but in the format you spec’d, such as
jpeg)
– After that you do what you want with it.
Example
• First get the player part.
try {
Player player = Manager.createPlayer(
"capture://video?encoding=jpeg" );
player.realize();
player.start();
• Now get the Video Control
VideoControl videoControl = (VideoControl)player.getControl(
"javax.microedition.media.control.VideoControl");
•
Create a “View Finder”
Field CameraView = (Field)videoControl.initDisplayMode(
VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field");
add(CameraView); //display the viewfinder, but you don’t have
too.
} catch (Exception e) {
System.out.println("Error"+e.getMessage());
e.printStackTrace();
}
Capture://
•
•
•
•
•
•
•
•
•
•
•
•
My code used the default, but you can specify
"encoding=jpeg&width=1600&height=1200&quality=superfine"
"encoding=jpeg&width=1600&height=1200&quality=fine"
"encoding=jpeg&width=1600&height=1200&quality=normal"
"encoding=jpeg&width=1024&height=768&quality=superfine"
"encoding=jpeg&width=1024&height=768&quality=fine"
"encoding=jpeg&width=1024&height=768&quality=normal"
"encoding=jpeg&width=640&height=480&quality=superfine"
"encoding=jpeg&width=640&height=480&quality=fine"
"encoding=jpeg&width=640&height=480&quality=normal"
The supported imageType(s) can also be queried by invoking
System.getProperty("video.snapshot.encodings")
“View Finder”
• This line requires a bit of explaining
– Field CameraView = (Field) videoControl.initDisplayMode(
VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field");
• InitDisplayMode has two options:
– USE_GUI_PRIMITIVE
• In this mode, arg can be the fully qualified class name of primitive GUI
components, for example javax.microedition.lcdui.Item or
net.rim.device.api.ui.Field. Depending on the value of arg, initDisplay returns
an Item or a Field.
– USE_DIRECT_VIDEO
• When this mode is used, arg must be the fully qualified class name of Canvas,
for example javax.microedition.lcdui.Canvas or a subclass of it.
• In this case, initDisplay will return an instance of Canvas and the video will be
directly rendered on this Canvas.
• Then we cast it as field to display it as specified in the arguments.
Get the picture
• Finally we can now take the picture.
– Use a method called Snapshot
– byte picData[] = videoControl.getSnapshot(null);
• Null says take the default (or whatever we already
specified) and use the default camera settings.
• You could again, specify the image quality in the
getSnapshot parameters.
• To get the image into a bitmap object
– image = Bitmap.createBitmapFromBytes(picData,
0, picData.length, 1);
Android
TAKING A PICTURE
What to use
• Android packages
• import android.hardware.CameraDevice;
• import android.hardware.CameraDevice.CaptureParams;
– Permissions and features
• <uses-permission
android:name="android.permission.CAMERA" />
• This too, if you change the how the camera is functioning.
• <uses-feature android:name="android.hardware.camera" />
• <uses-feature
android:name="android.hardware.camera.autofocus" />
Taking a picture
• In brief
– CameraDevice cameraDevice =
CameraDevice.open()
– To preview you need a surfaceHolder then use
setPreviewDisplay(surfaceHolder) and
cameraDevice.startPreview()
– Finally use the takePicture(…) to get the picture
– release() and close() the CameraDevice to release
it.
“View Finder”
• The view finder is implemented via a SurfaceHolder and SurfaceView
– In the layout, the a surfaceView is used.
• Example:
public class PicCapture extends Activity implements OnClickListener,
SurfaceHolder.Callback, Camera.PictureCallback {
…
cameraView = (SurfaceView) this.findViewById(R.id.CameraView);
surfaceHolder = cameraView.getHolder();
surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
surfaceHolder.addCallback(this);
• To finally take the picture we need all this too.
cameraView.setFocusable(true);
cameraView.setFocusableInTouchMode(true);
cameraView.setClickable(true);
cameraView.setOnClickListener(this);
“View Finder” (2)
• The code to implement the surfaceHolder can be very simple.
• When created open the camera and set the display
surfaceCreated() {
camera = Camera.open();
try { camera.setPreviewDisplay(holder);
catch (IOException exception) {
camera.release(); }
}
• Once it’s ready, start the preview
surfaceChanged() {
camera.startPreview();
}
“View Finder” (3)
• When we are done
surfaceDestroyed() {
camera.stopPreview();
camera.release();
}
Get the Picture
• Using the Camera.PictureCallBack we implement,
we can get the data for the picture and decide
what to do it with it.
• In its simplest form, which doesn’t nothing with
the picture
public void onPictureTaken(byte[] data, Camera
camera) {
• byte[] data is the picture that was taken
• this just restarts the preview.
camera.startPreview();
}
Get the Picture (2)
• To take the picture we use the
• camera.takePicture method in the onClick method for the SurfaceView
public void onClick(View v) {
camera.takePicture(null, null, null, this);
}
• takePicture (Camera.ShutterCallback shutter, Camera.PictureCallback raw,
Camera.PictureCallback postview , Camera.PictureCallback jpeg)
• We only need the last to get the picture and it show on the previous slide.
– shutter
• the callback for image capture moment, or null
– raw
•
the callback for raw (uncompressed) image data, or null
– postview
• the callback with postview image data, may be null
– jpeg
• the callback for JPEG image data, or null
Android Example Code
• A note for the example code
– The example code will put any pictures taken into
the “Camera roll” and can be seen with the gallery
app.
– Remember it uses touch to take the picture.
Android android.media
RECORDING VIDEO
First…
• Most examples and code on the web and from
the android books, DO NOT work.
– lots of subtle errors
• debugging is made more difficult, because
– the camera throws errors
• CameraInput Recording is not ready … frame dropped.
– the AudioFlinger shows constant buffer overflow
warnings.
– And this is when the app is working correctly.
Uses a surfaceView
• Like a taking a picture, we need a view finder
which uses a surfaceView.
• Call for the MediaRecorder
• and setup the encoding.
– both audio and video.
Example SurfaceView
• in onCreate
recorder = new MediaRecorder();
//setup recorder settings Next Slide
SurfaceView cameraView = (SurfaceView) findViewById(R.id.CameraView);
holder = cameraView.getHolder();
holder.addCallback(this);
holder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
• implemented methods.
public void surfaceCreated(SurfaceHolder holder) {
recorder.setPreviewDisplay(holder.getSurface());
recorder.prepare();
}
Example SurfaceView (2)
public void surfaceChanged(SurfaceHolder holder, int format,
int width, int height) {
}
public void surfaceDestroyed(SurfaceHolder holder) {
if (recording) {
recorder.stop();
recording = false;
}
recorder.release();
finish();
}
Recorder Settings
• We need to set the sources for audio and video
recorder.setAudioSource( MediaRecorder.AudioSource.MIC);
recorder.setVideoSource(
MediaRecorder.VideoSource.DEFAULT);
– MediaRecorder.VideoSource.CAMERA should work as well.
• Now we need to setup encoders.
– In android 2.2 we can use profiles instead of setting everything
manually.
– CamcorderProfile.QUALITY_HIGH or QUALITY_LOW
CamcorderProfile cpHigh =
CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);
recorder.setProfile(cpHigh);
• And set the output location
recorder.setOutputFile("/sdcard/videocapture_example.mp4");
Recorder Settings (2)
• Manual settings could look like this:
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);
recorder.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP);
• QUALITY_HIGH settings are MP4 file
• QUALITY_LOW settings are 3gp file
• The manual list is very long, see the android doc’s
or Apress - Pro Android Media Developing
Graphics, Music, Video, and Rich Media Apps for
Smartphones and Tablets, Chapter 11 for a full
list of settings.
Recording
• To record
recorder.start();
• To stop
recorder.stop();
– The file should be there at this point.
• Remember when you are done
recorder.release();
Example code
• The example code
– need an Sdcard to test code and the file will be located
/sdcard/videocapture_example.mp4
• The code uses an extended surfaceView call
captureSurface, instead of just a surfaceView
– The code is all there, but rearranged from the slides.
• Honesty, the code just didn’t work without an extended
SurfaceView
• The code starts up with the viewfinder, touch the
screen to start recording, again to stop recording. It
will then launch the native media player to replay the
video.
Blackberry MMAPI
RECORDING VIDEO
Video Recording.
• Use all the same things from pic capturing
– need a “view finder”
– We the capture line
• And RecordControl
– use startRecord() to start it (needs be a thread)
– use stopRecord() to pause recording
– use commit() to finish it.
• Player
– choose to setRecordLocation or stream.
Example code
• To get the “view finder” going
player = Manager.createPlayer(
"capture://video?encoding=video/3gpp&mode=standard" );
player.realize();
player.start();
videoControl = (VideoControl)player.getControl( "VideoControl");
recorder = (RecordControl)player.getControl("RecordControl");
CameraView = (Field)videoControl.initDisplayMode(
VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field");
add(CameraView);
Capture line
• capture://video?encoding=video/3gpp&mode=standar
d
– standard video based on specifications of the phone
• capture://video?encoding=video/3gpp&mode=mms
– creates a video based on MMS standards
• intended for sending the video via mms
• You can ask the phone which encoding it can use with:
– String encodingsString = System.getProperty(
"video.encodings" );
– And parse the string.
Encodings from the Bold 9000
•
•
•
•
•
•
•
•
•
•
•
•
•
•
encoding=video/3gpp&mode=standard
encoding=video/3gpp&mode=mms
encoding=video/3gpp&width=480&height=320&video_codec=MPEG-4&audio_codec=AMR
encoding=video/3gpp&width=176&height=144&video_codec=MPEG-4&audio_codec=AMR
encoding=video/3gpp&width=480&height=320&video_codec=MPEG-4&audio_codec=PCM
encoding=video/3gpp&width=176&height=144&video_codec=MPEG-4&audio_codec=PCM
encoding=video/3gpp&width=480&height=320&video_codec=H263&audio_codec=AMR
encoding=video/3gpp&width=176&height=144&video_codec=H263&audio_codec=AMR
encoding=video/3gpp&width=480&height=320&video_codec=H263&audio_codec=PCM
encoding=video/3gpp&width=176&height=144&video_codec=H263&audio_codec=PCM
encoding=video/3gpp&width=480&height=320&video_codec=H264&audio_codec=AMR
encoding=video/3gpp&width=176&height=144&video_codec=H264&audio_codec=AMR
encoding=video/3gpp&width=480&height=320&video_codec=H264&audio_codec=PCM
encoding=video/3gpp&width=176&height=144&video_codec=H264&audio_codec=PCM
Start Recording.
• Once the view finder is up and running
• Note the view finder fails in a popup, I had to use a
MainScreen to get it to work.
• Set the location
recorder.setRecordLocation("file:///SDCard/Blac
kberry/videos/testrecording.3gp");
• and start recording.
recorder.startRecord();
finish recording.
• stopRecord()
– pauses the recording. You can run startRecord()
again to continue recording
• commit()
– finishes and write outs the find file (assuming
location used)
• Then you can close the player.
Recording to a stream.
• Use the connector.open to provide a stream
• get the outputsrearm and use
setRecordStream
fc = (FileConnection)Connector.open(
STREAM_VIDEO_FILE );
rc.setRecordStream(fc.openOutputStream());
• After you issue commit()
• close the outputstream
Example code.
• Two examples provided
– VideoCapture.zip
• very basic app to record video
• Launches the native player to play the video after
recording.
– VideoRecordingApplication.zip
• more complex (writing by blackberry)
– Note the videoplay code doesn’t work.
• Both require an Sdcard, since they store the
video on the card.
References
• Blackberry
– http://docs.blackberry.com/en/developers/deliverables/11942/Create_BB_ap
p_disp_image_from_camera_viewFinder_739696_11.jsp
– http://docs.blackberry.com/en/developers/deliverables/17968/Take_a_pictur
e_in_a_BB_device_app_1228201_11.jsp
– http://www.leterry.com/how-to-take-a-snapshot-using-the-built-in-camera-ofa-blackberry-smartphone.html
– Advanced Blackberry Development, Chapter 2 catpure, chapter 3 playback.
• Android
– http://developer.android.com/intl/zh-CN/guide/topics/media/index.html
– http://www.brighthub.com/mobile/google-android/articles/43414.aspx (a
difficult example to follow and it’s for 1.6)
– Apress - Pro Android Media Developing Graphics, Music, Video, and Rich
Media Apps for Smartphones and Tablets
• Chapter 2 for taking pictures, chapter 9 for video playback.
– API examples, com.example.android.apis.media
Q&A