Cosc 5/4730
Android media Part 2:
Pictures and Video
Emulators and Samsung
• The emulators all have problems, much of the
following code has been tested on the actual
phones
• Video playback code works, but the video may
not always display in the emulators
• android
– Video/picture capture does not work as documented.
• Samsung hardware!
– Some Samsung devices have many problems.
Something about their hardware… search for the
video and samsung for some possible answers.
Android android.media
PLAY MEDIA
Supported formats
• In general Android’s support is consistent with other
mobile phones.
• It supports the 3GP (.3gp) and MPEG-4 (.mp4) file
formats.
– 3GP is a video standard derived from MPEG-4 specifically
for use by mobile devices.
• As far as codecs go, Android supports H.263, a codec
designed for low-latency and low-bitrate
videoconferencing applications. H.263 video is
supported in either MPEG-4 (.mp4) or 3GP (.3gp) files.
Android also supports MPEG-4 Simple Profile in 3GP
files (.3gp) as well as H.264.
Android
• First method
– For greater flexibility you will need to use the mediaPlayer
and a surfaceView (or TextureView API 14+)
• MediaPlayer like the audio and use a surfaceView to display the
video.
• There are examples in the API demo, plus several of the books.
• The second method uses a VideoView to display.
– The VideoView uses a MediaController to start and stop
and provide functions to control the video play back.
– With a MediaPlayer like the audio. prepare(), then start()
– This is one I’ll cover in this lecture.
VideoView
• VideoView is a View that has video playback
capabilities and can be used directly in a
layout.
• We can then add controls (play, pause,
forward, back, etc) with the MediaController.
ViewView example
• Get the ViewView out of the layout
vv = (VideoView) this.findViewById(
R.id.VideoView);
• Setup where the file to play is
Uri videoUri =
Uri.parse(Environment.getExternalStorageDirectory
().getPath() + "/example.mp4");
vv.setVideoURI(videoUri);
• play the video
vv.start();
Adding media controllers.
• Very simple
vv = (VideoView) this.findViewById(R.id.VideoView);
vv.setMediaController(new MediaController(this));
• Now media controls will show up on the
screen.
Using native media player
• Call for an Intent and send it.
Uri data = Uri.parse(VideoFile);
intent.setDataAndType(data, "video/mp4");
startActivity(intent);
• Remember, your app is now in the
background.
Example code
• The VideoPlay example code
– This will play a video from the internet.
– If you can uncomment the code to have it play the
video from the sdcard, but you will need to copy
the file to the sdcard.
DISPLAYING A PICTURE
Displaying Pictures
– See code already covered to how display pictures.
• ImageView for example…
Android
TAKING A PICTURE
Camera vs Camera2 API
• Starting in API 21 (Lollipop) there is a new set
of APIs.
– There is no backward compatibly either.
• So first we look at Camera v1 APIs
• And then Camera2 APIs
– These are more flexible and should allow for
“filters” to be added on the fly to image.
• Finally, you can just use an intent have the
native camera app take the picture.
Taking A Picture
CAMERA V1 APIS
What to use
• Android packages
• import android.hardware.CameraDevice;
• import android.hardware.CameraDevice.CaptureParams;
– Permissions and features
<uses-permission
android:name="android.permission.CAMERA" />
– Note, This one will requirement permission checking in API 23+
• This too, if you change the how the camera is functioning.
<uses-feature android:name="android.hardware.camera" />
<uses-feature
android:name="android.hardware.camera.autofocus" />
Taking a picture
• In brief
– CameraDevice cameraDevice =
CameraDevice.open()
– To preview you need a surfaceHolder then use
setPreviewDisplay(surfaceHolder) and
cameraDevice.startPreview()
• Or TextureView if it’s API 14+
– Finally use the takePicture(…) to get the picture
– release() and close() the CameraDevice to release
it.
“View Finder”
• The view finder is implemented via a SurfaceHolder and SurfaceView
– In the layout, the a surfaceView is used.
• Example:
public class PicCapture extends Activity implements OnClickListener,
SurfaceHolder.Callback, Camera.PictureCallback {
…
cameraView = (SurfaceView) this.findViewById(R.id.CameraView);
surfaceHolder = cameraView.getHolder();
surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
surfaceHolder.addCallback(this);
• To finally take the picture we need all this too.
cameraView.setFocusable(true);
cameraView.setFocusableInTouchMode(true);
cameraView.setClickable(true);
cameraView.setOnClickListener(this);
“View Finder” (2)
• The code to implement the surfaceHolder can be very simple.
• When created open the camera and set the display
surfaceCreated() {
camera = Camera.open();
try { camera.setPreviewDisplay(holder);
catch (IOException exception) {
camera.release(); }
}
• Once it’s ready, start the preview
surfaceChanged() {
camera.startPreview();
}
“View Finder” (3)
• When we are done
surfaceDestroyed() {
camera.stopPreview();
camera.release();
}
Get the Picture
• Using the Camera.PictureCallBack we implement,
we can get the data for the picture and decide
what to do it with it.
• In its simplest form, which doesn’t nothing with
the picture
public void onPictureTaken(byte[] data, Camera
camera) {
• byte[] data is the picture that was taken
• this just restarts the preview.
camera.startPreview();
}
Get the Picture (2)
• To take the picture we use the
• camera.takePicture method in the onClick method for the SurfaceView
public void onClick(View v) {
camera.takePicture(null, null, null, this);
}
• takePicture (Camera.ShutterCallback shutter, Camera.PictureCallback raw,
Camera.PictureCallback postview , Camera.PictureCallback jpeg)
• We only need the last to get the picture and it show on the previous slide.
– shutter
• the callback for image capture moment, or null
– raw
•
the callback for raw (uncompressed) image data, or null
– postview
• the callback with postview image data, may be null
– jpeg
• the callback for JPEG image data, or null
Taking A Picture
CAMERA V2 APIS
Camera2 APIs
• Like v1, this breaks up into 2 major pieces
– Viewfinder
• Connect it to a SurfaceView or TextureView.
– Taking the picture
Camera2 ViewFinder
• First find the “id” for the camera to use
– CameraManager manager = (CameraManager)
getActivity().getSystemService(Context.CAMERA_SERVICE);
– String cameraId = manager.getCameraIdList()[0];
• In this case just use the first one (tends to be the back camera)
• Then it gets complex, because we open the camera via
this
– manager.openCamera (String cameraId,
CameraDevice.StateCallback callback, Handler handler)
• StateCallback is a listener “first” and the handler is this case is
likely null.
– The handler is the thread to use, null for main. Another thread is likely
where you would apply a filter to the image, before it is displayed.
» Note, someone want to figure this out?
Camera2 ViewFinder (2)
• The call back is pretty simple, but the complex code is here.
CameraDevice.StateCallback mStateCallback = new CameraDevice.StateCallback() {
@Override
public void onOpened(CameraDevice camera) {
mCameraDevice = camera;
//We now have the camera, so setup the capture of the current surface.
startPreview(); //in a method since it will be called several times.
}
@Override
public void onDisconnected(CameraDevice camera) {
Log.e(TAG, "onDisconnected");
}
@Override
public void onError(CameraDevice camera, int error) {
Log.e(TAG, "onError");
}
};
startPreview
• First we need a surface (from the SurfaceView or Textview).
We are using a surfaceView
Surface surface = mHolder.getSurface();
– mHolder is our surfaceview
• mPreviewBuilder = mCameraDevice.createCaptureRequest(
CameraDevice.TEMPLATE_PREVIEW);
mPreviewBuilder.addTarget(surface);
– Note, if wanted to display it on more then 1 surfaceview, we
then add the rest here. (I think).
• mCameraDevice.createCaptureSession(Arrays.asList(surfac
e), CameraCaptureStateCallback);
– The CameraCapture is listed on the next slide.
startPreview(2)
•
CameraCaptureSession.StateCallback CameraCaptureStateCallback = new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(CameraCaptureSession session) {
mPreviewSession = session;
//now setup the update preview.
mPreviewBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);
HandlerThread thread = new HandlerThread("CameraPreview"); //this time we need a thread.
thread.start();
Handler backgroundHandler = new Handler(thread.getLooper());
mPreviewSession.setRepeatingRequest(mPreviewBuilder.build(), null, backgroundHandler);
// Request endlessly repeating capture of images by this session and place it in the surfaceview
}
@Override
public void onConfigureFailed(CameraCaptureSession session) {
Toast.makeText(context, "onConfigureFailed", Toast.LENGTH_LONG).show();
}
}, null); //null again is for a handler we don’t need.
Now we have preview
• To take a picture, it has a very similar method
but we start with the createCaptureSession
method.
– We need to setup an ImageReader, so we can save
it as JPEG (or other format), another handler,
Surfaces, plus another captureBuilder
Taking the picture
• First setup some variables we need
reader = ImageReader.newInstance(width, height, ImageFormat.JPEG, 1);
–
Width and height handled in the code, but not shown here.
outputSurfaces = new ArrayList<Surface>(2);
outputSurfaces.add(reader.getSurface()); //here the surface is a reader, which will save to a file later on.
–
Again add more surfaces here as needed.
• Setup our handler threads here.
HandlerThread thread = new HandlerThread("CameraPicture");
thread.start();
backgroudHandler = new Handler(thread.getLooper());
• Readlistener is shown later.
reader.setOnImageAvailableListener(readerListener, backgroudHandler); //this is where the save is.
• configure the catureBuilder, which you will build, in the listener code, later on.
captureBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
captureBuilder.addTarget(reader.getSurface());
captureBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);
• Fix Orientation, getJpegOreientation is shown in the code, but not here. The build for capture is in a
listener.
int deviceorientation = context.getResources().getConfiguration().orientation;
captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, getJpegOrientation(characteristics,
deviceorientation));
readerListener
•
A new image is available, ie the capture, so
save file.
ImageReader.OnImageAvailableListener
readerListener = new
ImageReader.OnImageAvailableListener() {
@Override
public void onImageAvailable(ImageReader
reader) {
Image image = null;
image = reader.acquireLatestImage();
ByteBuffer buffer =
image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.capacity()];
buffer.get(bytes);
save(bytes);
}
•
This is just a helper method to save the file
which at this point is just an array of bytes.
private void save(byte[] bytes) throws
IOException {
OutputStream output = null;
try {
output = new FileOutputStream(file);
output.write(bytes);
} finally {
if (null != output) {
output.close();
}
}
}
};
Taking the picture (2)
• This is the line you call to actually take the
picture
mCameraDevice.createCaptureSession(
outputSurfaces, mCaptureStateCallback,
backgroudHandler);
– Like in the preview, we need the stateCallback
• The outputSurfaces, and handler were declared 2 slides
back.
Taking the picture (3)
•
So this CameraCaptureSession.StateCallback doesn’t deal with the preview, instead is readies to save a file.
CameraCaptureSession.StateCallback mCaptureStateCallback = new CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(CameraCaptureSession session) {
session.capture(captureBuilder.build(), captureListener, backgroudHandler);
}
@Override
public void onConfigureFailed(CameraCaptureSession session) {
}
};
•
And the last call back
CameraCaptureSession.CaptureCallback captureListener = new CameraCaptureSession.CaptureCallback() {
@Override
public void onCaptureCompleted(CameraCaptureSession session,
CaptureRequest request, TotalCaptureResult result) {
super.onCaptureCompleted(session, request, result);
startPreview(); //start the preview again.
}
};
Taking A Picture
WITH AN INTENT
Via an Intent.
• You can also have the default “camera app”
take the picture and return it to your app.
• Also doesn’t require any permissions.
Intent intent = new Intent(
android.provider.MediaStore.ACTION_IMAGE_C
APTURE);
startActivityForResult(intent, myID);
• Picture is returned in onActivityResult
Via an Intent (2)
protected void onActivityResult(int requestCode, int
resultCode, Intent data) {
super.onActivityResult(requestCode, resultCode,
data);
//get the picture out of the intent and display in an
image view.
Bitmap bp = (Bitmap)
data.getExtras().get("data");
iv.setImageBitmap(bp);
}
Android Example Code
• A note for the example code PicCapture
– Remember it uses touch to take the picture.
– Both a V1 and v2 APIs are used in separate fragments.
• PicCapture2 uses a button and separates the
code a little better.
– Both a V1 and v2 APIs are used in separate fragments.
• PicCapture3 uses an intent to take the picture.
• CameraPreview is only camera2 APIs and it
separated up into multiple classes to make the
code easier to read.
– Similar to how it shown in the slides.
Android android.media
RECORDING VIDEO
Recording VIDEO
CAMERA V1 APIS
First…
• Most examples and code on the web and from
the android books, DO NOT work.
– lots of subtle errors
• debugging is made more difficult, because
– the camera throws errors
• CameraInput Recording is not ready … frame dropped.
– the AudioFlinger shows constant buffer overflow
warnings.
– And this is when the app is working correctly.
Uses a surfaceView
• Like a taking a picture, we need a view finder
which uses a surfaceView.
• Call for the MediaRecorder
• and setup the encoding.
– both audio and video.
Example SurfaceView
• in onCreate
recorder = new MediaRecorder();
//setup recorder settings Next Slide
SurfaceView cameraView = (SurfaceView) findViewById(R.id.CameraView);
holder = cameraView.getHolder();
holder.addCallback(this);
holder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
• implemented methods.
public void surfaceCreated(SurfaceHolder holder) {
recorder.setPreviewDisplay(holder.getSurface());
recorder.prepare();
}
Example SurfaceView (2)
public void surfaceChanged(SurfaceHolder holder, int format,
int width, int height) {
}
public void surfaceDestroyed(SurfaceHolder holder) {
if (recording) {
recorder.stop();
recording = false;
}
recorder.release();
finish();
}
Recorder Settings
• We need to set the sources for audio and video
recorder.setAudioSource( MediaRecorder.AudioSource.MIC);
recorder.setVideoSource(
MediaRecorder.VideoSource.DEFAULT);
– MediaRecorder.VideoSource.CAMERA should work as well.
• Now we need to setup encoders.
– In android 2.2 we can use profiles instead of setting everything
manually.
– CamcorderProfile.QUALITY_HIGH or QUALITY_LOW
CamcorderProfile cpHigh =
CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);
recorder.setProfile(cpHigh);
• And set the output location
recorder.setOutputFile("/sdcard/videocapture_example.mp4");
Recorder Settings (2)
• Manual settings could look like this:
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);
recorder.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP);
• QUALITY_HIGH settings are MP4 file
• QUALITY_LOW settings are 3gp file
• The manual list is very long, see the android doc’s
or Apress - Pro Android Media Developing
Graphics, Music, Video, and Rich Media Apps for
Smartphones and Tablets, Chapter 11 for a full
list of settings.
Recording
• To record
recorder.start();
• To stop
recorder.stop();
– The file should be there at this point.
• Remember when you are done
recorder.release();
Example code
• The example code
– need an Sdcard to test code and the file will be located
/sdcard/videocapture_example.mp4
• The code uses an extended surfaceView call
captureSurface, instead of just a surfaceView
– The code is all there, but rearranged from the slides.
• Honesty, the code just didn’t work without an extended SurfaceView
• The code starts up with the viewfinder, touch the screen to
start recording, again to stop recording. It will then launch
the native media player to replay the video.
• As Note, using this on API23+ (marshmallow), the video is
likely upside down.
Recording VIDEO
CAMERA V2 APIS
Camera2 APIs.
• Recording video with Camera2 API is very similar
to taking a picture.
• Except…
– Change Capture to a Repeating Request.
• Why? Because we need more then one frame.
– This changes how some of the code works.
– As a note, on at least API 22 (lollipop 5.1) the example
code works once then fails.
– But works great on API 23 without any problems.
MediaRecorder again
• Setup the media recorder with a couple of extras
mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);
mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.SURFACE);
mMediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
mMediaRecorder.setOutputFile(file.getAbsolutePath()); //file is setup else where the
example code.
mMediaRecorder.setVideoEncodingBitRate(10000000);
mMediaRecorder.setVideoFrameRate(30);
mMediaRecorder.setVideoSize(mVideoSize.getWidth(), mVideoSize.getHeight());
mMediaRecorder.setVideoEncoder(MediaRecorder.VideoEncoder.H264);
mMediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
• Setup and fix the orientation, hopefully. See code for getJpegOrientation.
int deviceorientation = context.getResources().getConfiguration().orientation;
mMediaRecorder.setOrientationHint(getJpegOrientation(characteristics,
deviceorientation));
mMediaRecorder.prepare();
Now the “preview” screen
• Basically the same with a couple of changes, where we add the MediaRecorder as a surface as well.
captureBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_RECORD);
outputSurfaces = new ArrayList<Surface>(2);
outputSurfaces.add(mMediaRecorder.getSurface());
outputSurfaces.add(mHolder.getSurface());
HandlerThread thread = new HandlerThread("CameraVideo");
thread.start();
backgroundHandler = new Handler(thread.getLooper());
captureBuilder.addTarget(mMediaRecorder.getSurface());
captureBuilder.addTarget(mHolder.getSurface());
captureBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);
mCameraDevice.createCaptureSession(outputSurfaces, mCaptureStateCallback, backgroundHandler);
CaptureSession
CameraCaptureSession.StateCallback mCaptureStateCallback = new
CameraCaptureSession.StateCallback() {
@Override
public void onConfigured(CameraCaptureSession session) {
try {
mSession = session;
• null for capture listener, because mrecoder does the work here and it’s a repeatingRequest now
session.setRepeatingRequest(captureBuilder.build(), null, backgroundHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
@Override
public void onConfigureFailed(CameraCaptureSession session) { }
};
Recording a video
• Finally to start recording:
mMediaRecorder.start();
• And to stop
mMediaRecorder.stop();
mMediaRecorder.reset();
– The file is now stored.
Example code
• VideoCapture4 uses the camera2 and this
material covered here.
– It will check if you are running lollipop or
marshmallow.
• If lollipop, it will exit, since the app dies anyway.
• CameraPreview is an attempt to put all this
perview, picture taking and video taking
together.
– Again, works great in marshmallow and fails in
lollipop.
References
• http://developer.android.com/intl/zhCN/guide/topics/media/index.html
• http://www.brighthub.com/mobile/googleandroid/articles/43414.aspx (a difficult example to follow
and it’s for 1.6)
• Apress - Pro Android Media Developing Graphics, Music,
Video, and Rich Media Apps for Smartphones and Tablets
– Chapter 2 for taking pictures, chapter 9 for video playback.
• API examples, com.example.android.apis.media
• Camera2
– https://github.com/googlesamples/android-Camera2Video
– http://developer.android.com/reference/android/hardware/ca
mera2/package-summary.html
Q&A