pedroSG94
Update rotation during stream and keep it consistent with preview
Hi @pedroSG94. First of all, thank you for the cool library.
Is it possible to change stream rotation during the stream? I can pass a rotation parameter to prepareVideo() method and it works fine, however, I don't know how to change it during the stream.
I'd like to have the same behavior as in Streamlabs app - when I live stream to YouTube using this app and rotate the phone, the stream updates its rotation.
I also struggle with keeping the same rotation on preview and stream.
I use OpenGlView, have my Activity orientation set to portrait. When I call prepareVideo(roatation) I pass rotation param based on sensor rotation. The stream rotation and preview rotation is fine when I start streaming in portait calling prepareVideo(rotation = 90), but when I start streaming in landscape calling prepareVideo(rotation = 0), then stream rotation is fine but preview rotation is wrong.
I've tried playing with OpenGlView.setRotation() but it affects both stream and preview in a way I can't understand.
Please guide me a bit or maybe update the sample app if you'd have some time. Thanks!
Yes, this is possible but you need modify the library. Like this:
- Add this to ScreenRender class:
private float[] rotationMatrix = new float[16];
public void setRotation(int rotation) {
Matrix.setIdentityM(rotationMatrix, 0);
Matrix.rotateM(rotationMatrix, 0, rotation, 0f, 0f, -1f);
update();
}
private void update() {
Matrix.setIdentityM(MVPMatrix, 0);
Matrix.multiplyMM(MVPMatrix, 0, rotationMatrix, 0, MVPMatrix, 0);
}
- Add set rotation in this line: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/encoder/src/main/java/com/pedro/encoder/input/gl/render/ScreenRender.java#L79
- Add int rotation parameter to draw method: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/encoder/src/main/java/com/pedro/encoder/input/gl/render/ScreenRender.java#L77
- Add rotation parameter to drawScreen method and set it to draw method:
public void drawScreen(int width, int height, boolean keepAspectRatio, int mode, int rotation) {
screenRender.draw(width, height, keepAspectRatio, mode, rotation);
}
-
Set rotation to 0 in OffScreenGlThread
-
Modify OpenGlView to support rotation with sensor (You need calibrate sensor by yourself) For this you need set rotation to 0 when you draw preview (never rotate preview: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/java/com/pedro/rtplibrary/view/OpenGlView.java#L130 Add custom rotation get from sensor in stream result: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/java/com/pedro/rtplibrary/view/OpenGlView.java#L140 For this last you can create a sensor class like this:
public class RotationSensor implements SensorEventListener {
private SensorManager sensorManager;
private Context activity;
private GetRotation getRotation;
public interface GetRotation {
void getRotation(int rotation);
}
public RotationSensor(Context activity, GetRotation getRotation) {
this.activity = activity;
this.getRotation = getRotation;
}
public void prepare() {
sensorManager = (SensorManager) activity.getSystemService(SENSOR_SERVICE);
}
public void resume() {
sensorManager.registerListener(this,
sensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER),
SensorManager.SENSOR_DELAY_NORMAL);
}
public void pause() {
sensorManager.unregisterListener(this);
}
@Override
public void onSensorChanged(SensorEvent sensorEvent) {
if (sensorEvent.sensor.getType() == Sensor.TYPE_ACCELEROMETER) {
getAccelerometer(sensorEvent);
}
}
@Override
public void onAccuracyChanged(Sensor sensor, int i) {
}
private void getAccelerometer(SensorEvent event) {
float[] values = event.values;
// Movement
float x = values[0];
float y = values[1];
float z = values[2];
float accelationSquareRoot = (x * x + y * y + z * z)
/ (SensorManager.GRAVITY_EARTH * SensorManager.GRAVITY_EARTH);
Log.e("Pedro", "x: " + x + ", y: " + y + ", z:" + z + ", ASR: " + accelationSquareRoot);
if (y < 0) {
getRotation.getRotation(270);
} else {
getRotation.getRotation(0);
}
}
}
- Now, add this sensor class to OpenGlview: Instance class:
private RotationSensor rotationSensor = new RotationSensor(getContext(), new RotationSensor.GetRotation() {
@Override
public void getRotation(int rotation) {
rotationResult = rotation;
}
});
Set prepare in constructors. Set resume in surfaceChanged callback Implement surfaceDestroyed callback and set pause.
Remember that you need calibrate sensor as you want
@pedroSG94 Thank you for the quick answer. I was playing with your code today. I've forked the repo and applied this code to your sample, however, with no success. Looks like setting orientation affects preview as well. I've added temporary PR with your changes, could you confirm if everything is fine? https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/pull/532/files PS I had to move to Camera2 API and apply 720p resolution, but don't mind it.
Let me explain the goal I want to achieve. Here is the livestream I recorder using Streamlab app on Android https://www.youtube.com/watch?v=EE1dojK1pGE In the 19th second, you can notice rotation w portrait to landscape. I'd like to do the same and have my preview blocked (either in portrait or landscape), just like in any camera app.
I'll dig more into this, in meantime, if you have some tips for me, then I'd appreciate it
You only have an error in a line. Your error is here: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/pull/532/files#diff-d910747282a21a3999748fbc2bcbd2bbR141 Replace to:
managerRender.drawScreen(previewWidth, previewHeight, keepAspectRatio, aspectRatioMode, 0);
This line draw in preview: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/pull/532/files#diff-d910747282a21a3999748fbc2bcbd2bbR141 And this line draw in stream: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/pull/532/files#diff-d910747282a21a3999748fbc2bcbd2bbR151
@pedroSG94 Thanks, your fix works! Stream orientation is now correct. However, I still had problems with correct stream scaling being in landscape. Here are results for portrait and landscape:
https://cl.ly/30100704a063
https://cl.ly/83ab486f93d0
Luckily, I was able to fix it. I did it like this:
a) For stream rendering in ScreenRenderer class, I've changed viewport to be always landscape
GLES20.glViewport(0, 0, height, width); - swapped width with height
b) Applied scaling to matrix (only when in portrait)
Matrix.scaleM(MVPMatrix, 0, scaleX, scaleY, 1f);
where scaleX in that case is:
final float adjustedWidth = width * (width / (float) height); return adjustedWidth / height;
I'm not sure if there is an easier solution for that? You can check my PR for details. I've also simplified the sensor manager by using Android's OrientationEventListener.
Now I have exactly the same behavior as the Stremlabs app. I think it would be cool to have such a feature by default in your lib, also for portrait mode (as I only support landscape).
Your solution is good. I'm not sure about set a feature for this because calibrate sensor in all devices is not easy cause not all sensor throw same values. I will study your code and try adapt it to landscape.
@pedroSG94 Maybe you're right that putting sensor logic into the lib may not be the best fit for this feature, but I meant more the possibility to set static stream orientation + blocking preview. I think this is what Youtube and Streamlabs do? That way a library user could set it either depending on sensor rotation or user clicking rotation button. This sensor rotation manager could be just a helper class if someone would like to use it.
@pedroSG94 Maybe you're right that putting sensor logic into the lib may not be the best fit for this feature, but I meant more the possibility to set static stream orientation + blocking preview. I think this is what Youtube and Streamlabs do? That way a library user could set it either depending on sensor rotation or user clicking rotation button. This sensor rotation manager could be just a helper class if someone would like to use it.
This is interesting. I did stream rotation implementation and now anyone can create her own sensor to implement this feature. You can rotate stream (without effect in preview) with this method:
rtmpCamera1.getGlInterface().setStreamRotation(rotation);
Commit: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/commit/c9a738abc4d3496a3c618dad33c7e3b0b99dc57f
@pedroSG94 That's great! I've tested it and preview is still and the stream rotation works fine. That's good, however, after rotation, the stream doesn't scale according to rotation. So when you change rotation to landscape the video is squeezed instead of being stretched. You can checkout on my PR where it works well and compare these two behaviors.
Finished: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/commit/2350fd6fc34cc38a9fed266f30e8577db7a8bc3b Landscape and portrait is supported now. Also I added in rtpLibrary module your sensor to do it more easy. You only need use your sensor like:
private SensorRotationManager sensorRotationManager;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
sensorRotationManager =
new SensorRotationManager(this, new SensorRotationManager.RotationChangedListener() {
@Override
public void onRotationChanged(int rotation) {
rtmpCamera1.getGlInterface().setStreamRotation(rotation);
}
});
}
@Override
public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i1, int i2) {
sensorRotationManager.start();
}
@Override
public void surfaceDestroyed(SurfaceHolder surfaceHolder) {
sensorRotationManager.stop();
}
@pedroSG94 I've tried it using opengl sample and I couldn't make it work. Were you able to do so? How do you test it? I'm using the Vimeo live stream as it is quick, free and very stable https://vimeo.com/features/livestreaming.
Here are the screenshots: portrait https://cl.ly/2c31a62b0cd8 landscape https://cl.ly/b19d28fb1bf6
And to be sure we're on the same page here are the screenshots when I use code from my PR a) 1280 x 720 portrait https://cl.ly/cfa9e08b2b74 landscape https://cl.ly/21e4453f8587 b) 640 x 480 portrait https://cl.ly/59057adc8aa1 landscape https://cl.ly/98667e26bd10 Don't mind the frame is different as I was closer to the screen when recording 640x480
I'll try to check code details tomorrow
I tested with local server and VLC player.
Portrait without rotate:
Portrait rotated to landscape:
Landscape without rotate:
Landscape rotated to portrait:
@pedroSG94 Your result is good. Could you show how did you modify the sample code because I have to be missing something?
Download last commit and replace rtmp Opengl example to this (basically I only added sensor and set stream rotation in callback):
package com.pedro.rtpstreamer.openglexample;
import android.graphics.BitmapFactory;
import android.graphics.Color;
import android.media.MediaPlayer;
import android.os.Build;
import android.os.Bundle;
import android.os.Environment;
import android.view.Menu;
import android.view.MenuItem;
import android.view.MotionEvent;
import android.view.SurfaceHolder;
import android.view.View;
import android.view.WindowManager;
import android.widget.Button;
import android.widget.EditText;
import android.widget.Toast;
import androidx.annotation.RequiresApi;
import androidx.appcompat.app.AppCompatActivity;
import com.pedro.encoder.input.gl.SpriteGestureController;
import com.pedro.encoder.input.gl.render.filters.AnalogTVFilterRender;
import com.pedro.encoder.input.gl.render.filters.AndroidViewFilterRender;
import com.pedro.encoder.input.gl.render.filters.BasicDeformationFilterRender;
import com.pedro.encoder.input.gl.render.filters.BeautyFilterRender;
import com.pedro.encoder.input.gl.render.filters.BlackFilterRender;
import com.pedro.encoder.input.gl.render.filters.BlurFilterRender;
import com.pedro.encoder.input.gl.render.filters.BrightnessFilterRender;
import com.pedro.encoder.input.gl.render.filters.CartoonFilterRender;
import com.pedro.encoder.input.gl.render.filters.CircleFilterRender;
import com.pedro.encoder.input.gl.render.filters.ColorFilterRender;
import com.pedro.encoder.input.gl.render.filters.ContrastFilterRender;
import com.pedro.encoder.input.gl.render.filters.DuotoneFilterRender;
import com.pedro.encoder.input.gl.render.filters.EarlyBirdFilterRender;
import com.pedro.encoder.input.gl.render.filters.EdgeDetectionFilterRender;
import com.pedro.encoder.input.gl.render.filters.ExposureFilterRender;
import com.pedro.encoder.input.gl.render.filters.FireFilterRender;
import com.pedro.encoder.input.gl.render.filters.GammaFilterRender;
import com.pedro.encoder.input.gl.render.filters.GlitchFilterRender;
import com.pedro.encoder.input.gl.render.filters.GreyScaleFilterRender;
import com.pedro.encoder.input.gl.render.filters.HalftoneLinesFilterRender;
import com.pedro.encoder.input.gl.render.filters.Image70sFilterRender;
import com.pedro.encoder.input.gl.render.filters.LamoishFilterRender;
import com.pedro.encoder.input.gl.render.filters.MoneyFilterRender;
import com.pedro.encoder.input.gl.render.filters.NegativeFilterRender;
import com.pedro.encoder.input.gl.render.filters.NoFilterRender;
import com.pedro.encoder.input.gl.render.filters.PixelatedFilterRender;
import com.pedro.encoder.input.gl.render.filters.PolygonizationFilterRender;
import com.pedro.encoder.input.gl.render.filters.RGBSaturationFilterRender;
import com.pedro.encoder.input.gl.render.filters.RainbowFilterRender;
import com.pedro.encoder.input.gl.render.filters.RippleFilterRender;
import com.pedro.encoder.input.gl.render.filters.RotationFilterRender;
import com.pedro.encoder.input.gl.render.filters.SaturationFilterRender;
import com.pedro.encoder.input.gl.render.filters.SepiaFilterRender;
import com.pedro.encoder.input.gl.render.filters.SharpnessFilterRender;
import com.pedro.encoder.input.gl.render.filters.SnowFilterRender;
import com.pedro.encoder.input.gl.render.filters.SwirlFilterRender;
import com.pedro.encoder.input.gl.render.filters.TemperatureFilterRender;
import com.pedro.encoder.input.gl.render.filters.ZebraFilterRender;
import com.pedro.encoder.input.gl.render.filters.object.GifObjectFilterRender;
import com.pedro.encoder.input.gl.render.filters.object.ImageObjectFilterRender;
import com.pedro.encoder.input.gl.render.filters.object.SurfaceFilterRender;
import com.pedro.encoder.input.gl.render.filters.object.TextObjectFilterRender;
import com.pedro.encoder.input.video.CameraOpenException;
import com.pedro.encoder.utils.gl.TranslateTo;
import com.pedro.rtplibrary.rtmp.RtmpCamera1;
import com.pedro.rtplibrary.util.SensorRotationManager;
import com.pedro.rtplibrary.view.OpenGlView;
import com.pedro.rtpstreamer.R;
import java.io.File;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;
import net.ossrs.rtmp.ConnectCheckerRtmp;
/**
* More documentation see:
* {@link com.pedro.rtplibrary.base.Camera1Base}
* {@link com.pedro.rtplibrary.rtmp.RtmpCamera1}
*/
@RequiresApi(api = Build.VERSION_CODES.JELLY_BEAN_MR2)
public class OpenGlRtmpActivity extends AppCompatActivity
implements ConnectCheckerRtmp, View.OnClickListener, SurfaceHolder.Callback,
View.OnTouchListener {
private RtmpCamera1 rtmpCamera1;
private Button button;
private Button bRecord;
private EditText etUrl;
private String currentDateAndTime = "";
private File folder = new File(Environment.getExternalStorageDirectory().getAbsolutePath()
+ "/rtmp-rtsp-stream-client-java");
private OpenGlView openGlView;
private SpriteGestureController spriteGestureController = new SpriteGestureController();
private SensorRotationManager sensorRotationManager;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
setContentView(R.layout.activity_open_gl);
openGlView = findViewById(R.id.surfaceView);
button = findViewById(R.id.b_start_stop);
button.setOnClickListener(this);
bRecord = findViewById(R.id.b_record);
bRecord.setOnClickListener(this);
Button switchCamera = findViewById(R.id.switch_camera);
switchCamera.setOnClickListener(this);
etUrl = findViewById(R.id.et_rtp_url);
etUrl.setHint(R.string.hint_rtmp);
rtmpCamera1 = new RtmpCamera1(openGlView, this);
openGlView.getHolder().addCallback(this);
openGlView.setOnTouchListener(this);
sensorRotationManager = new SensorRotationManager(this, new SensorRotationManager.RotationChangedListener() {
@Override
public void onRotationChanged(int rotation) {
rtmpCamera1.getGlInterface().setStreamRotation(rotation);
}
});
}
@Override
public boolean onCreateOptionsMenu(Menu menu) {
getMenuInflater().inflate(R.menu.gl_menu, menu);
return true;
}
@Override
public boolean onOptionsItemSelected(MenuItem item) {
//Stop listener for image, text and gif stream objects.
spriteGestureController.setBaseObjectFilterRender(null);
switch (item.getItemId()) {
case R.id.e_d_fxaa:
rtmpCamera1.getGlInterface().enableAA(!rtmpCamera1.getGlInterface().isAAEnabled());
Toast.makeText(this,
"FXAA " + (rtmpCamera1.getGlInterface().isAAEnabled() ? "enabled" : "disabled"),
Toast.LENGTH_SHORT).show();
return true;
//filters. NOTE: You can change filter values on fly without reset the filter.
// Example:
// ColorFilterRender color = new ColorFilterRender()
// rtmpCamera1.setFilter(color);
// color.setRGBColor(255, 0, 0); //red tint
case R.id.no_filter:
rtmpCamera1.getGlInterface().setFilter(new NoFilterRender());
return true;
case R.id.analog_tv:
rtmpCamera1.getGlInterface().setFilter(new AnalogTVFilterRender());
return true;
case R.id.android_view:
AndroidViewFilterRender androidViewFilterRender = new AndroidViewFilterRender();
androidViewFilterRender.setView(findViewById(R.id.switch_camera));
rtmpCamera1.getGlInterface().setFilter(androidViewFilterRender);
return true;
case R.id.basic_deformation:
rtmpCamera1.getGlInterface().setFilter(new BasicDeformationFilterRender());
return true;
case R.id.beauty:
rtmpCamera1.getGlInterface().setFilter(new BeautyFilterRender());
return true;
case R.id.black:
rtmpCamera1.getGlInterface().setFilter(new BlackFilterRender());
return true;
case R.id.blur:
rtmpCamera1.getGlInterface().setFilter(new BlurFilterRender());
return true;
case R.id.brightness:
rtmpCamera1.getGlInterface().setFilter(new BrightnessFilterRender());
return true;
case R.id.cartoon:
rtmpCamera1.getGlInterface().setFilter(new CartoonFilterRender());
return true;
case R.id.circle:
rtmpCamera1.getGlInterface().setFilter(new CircleFilterRender());
return true;
case R.id.color:
rtmpCamera1.getGlInterface().setFilter(new ColorFilterRender());
return true;
case R.id.contrast:
rtmpCamera1.getGlInterface().setFilter(new ContrastFilterRender());
return true;
case R.id.duotone:
rtmpCamera1.getGlInterface().setFilter(new DuotoneFilterRender());
return true;
case R.id.early_bird:
rtmpCamera1.getGlInterface().setFilter(new EarlyBirdFilterRender());
return true;
case R.id.edge_detection:
rtmpCamera1.getGlInterface().setFilter(new EdgeDetectionFilterRender());
return true;
case R.id.exposure:
rtmpCamera1.getGlInterface().setFilter(new ExposureFilterRender());
return true;
case R.id.fire:
rtmpCamera1.getGlInterface().setFilter(new FireFilterRender());
return true;
case R.id.gamma:
rtmpCamera1.getGlInterface().setFilter(new GammaFilterRender());
return true;
case R.id.glitch:
rtmpCamera1.getGlInterface().setFilter(new GlitchFilterRender());
return true;
case R.id.gif:
setGifToStream();
return true;
case R.id.grey_scale:
rtmpCamera1.getGlInterface().setFilter(new GreyScaleFilterRender());
return true;
case R.id.halftone_lines:
rtmpCamera1.getGlInterface().setFilter(new HalftoneLinesFilterRender());
return true;
case R.id.image:
setImageToStream();
return true;
case R.id.image_70s:
rtmpCamera1.getGlInterface().setFilter(new Image70sFilterRender());
return true;
case R.id.lamoish:
rtmpCamera1.getGlInterface().setFilter(new LamoishFilterRender());
return true;
case R.id.money:
rtmpCamera1.getGlInterface().setFilter(new MoneyFilterRender());
return true;
case R.id.negative:
rtmpCamera1.getGlInterface().setFilter(new NegativeFilterRender());
return true;
case R.id.pixelated:
rtmpCamera1.getGlInterface().setFilter(new PixelatedFilterRender());
return true;
case R.id.polygonization:
rtmpCamera1.getGlInterface().setFilter(new PolygonizationFilterRender());
return true;
case R.id.rainbow:
rtmpCamera1.getGlInterface().setFilter(new RainbowFilterRender());
return true;
case R.id.rgb_saturate:
RGBSaturationFilterRender rgbSaturationFilterRender = new RGBSaturationFilterRender();
rtmpCamera1.getGlInterface().setFilter(rgbSaturationFilterRender);
//Reduce green and blue colors 20%. Red will predominate.
rgbSaturationFilterRender.setRGBSaturation(1f, 0.8f, 0.8f);
return true;
case R.id.ripple:
rtmpCamera1.getGlInterface().setFilter(new RippleFilterRender());
return true;
case R.id.rotation:
RotationFilterRender rotationFilterRender = new RotationFilterRender();
rtmpCamera1.getGlInterface().setFilter(rotationFilterRender);
rotationFilterRender.setRotation(90);
return true;
case R.id.saturation:
rtmpCamera1.getGlInterface().setFilter(new SaturationFilterRender());
return true;
case R.id.sepia:
rtmpCamera1.getGlInterface().setFilter(new SepiaFilterRender());
return true;
case R.id.sharpness:
rtmpCamera1.getGlInterface().setFilter(new SharpnessFilterRender());
return true;
case R.id.snow:
rtmpCamera1.getGlInterface().setFilter(new SnowFilterRender());
return true;
case R.id.swirl:
rtmpCamera1.getGlInterface().setFilter(new SwirlFilterRender());
return true;
case R.id.surface_filter:
//You can render this filter with other api that draw in a surface. for example you can use VLC
SurfaceFilterRender surfaceFilterRender = new SurfaceFilterRender();
rtmpCamera1.getGlInterface().setFilter(surfaceFilterRender);
MediaPlayer mediaPlayer = MediaPlayer.create(this, R.raw.big_bunny_240p);
mediaPlayer.setSurface(surfaceFilterRender.getSurface());
mediaPlayer.start();
//Video is 360x240 so select a percent to keep aspect ratio (50% x 33.3% screen)
surfaceFilterRender.setScale(50f, 33.3f);
spriteGestureController.setBaseObjectFilterRender(surfaceFilterRender); //Optional
return true;
case R.id.temperature:
rtmpCamera1.getGlInterface().setFilter(new TemperatureFilterRender());
return true;
case R.id.text:
setTextToStream();
return true;
case R.id.zebra:
rtmpCamera1.getGlInterface().setFilter(new ZebraFilterRender());
return true;
default:
return false;
}
}
private void setTextToStream() {
TextObjectFilterRender textObjectFilterRender = new TextObjectFilterRender();
rtmpCamera1.getGlInterface().setFilter(textObjectFilterRender);
textObjectFilterRender.setText("Hello world", 22, Color.RED);
textObjectFilterRender.setDefaultScale(rtmpCamera1.getStreamWidth(),
rtmpCamera1.getStreamHeight());
textObjectFilterRender.setPosition(TranslateTo.CENTER);
spriteGestureController.setBaseObjectFilterRender(textObjectFilterRender); //Optional
}
private void setImageToStream() {
ImageObjectFilterRender imageObjectFilterRender = new ImageObjectFilterRender();
rtmpCamera1.getGlInterface().setFilter(imageObjectFilterRender);
imageObjectFilterRender.setImage(
BitmapFactory.decodeResource(getResources(), R.mipmap.ic_launcher));
imageObjectFilterRender.setDefaultScale(rtmpCamera1.getStreamWidth(),
rtmpCamera1.getStreamHeight());
imageObjectFilterRender.setPosition(TranslateTo.RIGHT);
spriteGestureController.setBaseObjectFilterRender(imageObjectFilterRender); //Optional
spriteGestureController.setPreventMoveOutside(false); //Optional
}
private void setGifToStream() {
try {
GifObjectFilterRender gifObjectFilterRender = new GifObjectFilterRender();
gifObjectFilterRender.setGif(getResources().openRawResource(R.raw.banana));
rtmpCamera1.getGlInterface().setFilter(gifObjectFilterRender);
gifObjectFilterRender.setDefaultScale(rtmpCamera1.getStreamWidth(),
rtmpCamera1.getStreamHeight());
gifObjectFilterRender.setPosition(TranslateTo.BOTTOM);
spriteGestureController.setBaseObjectFilterRender(gifObjectFilterRender); //Optional
} catch (IOException e) {
Toast.makeText(this, e.getMessage(), Toast.LENGTH_SHORT).show();
}
}
@Override
public void onConnectionSuccessRtmp() {
runOnUiThread(new Runnable() {
@Override
public void run() {
Toast.makeText(OpenGlRtmpActivity.this, "Connection success", Toast.LENGTH_SHORT).show();
}
});
}
@Override
public void onConnectionFailedRtmp(final String reason) {
runOnUiThread(new Runnable() {
@Override
public void run() {
Toast.makeText(OpenGlRtmpActivity.this, "Connection failed. " + reason, Toast.LENGTH_SHORT)
.show();
rtmpCamera1.stopStream();
button.setText(R.string.start_button);
}
});
}
@Override
public void onNewBitrateRtmp(long bitrate) {
}
@Override
public void onDisconnectRtmp() {
runOnUiThread(new Runnable() {
@Override
public void run() {
Toast.makeText(OpenGlRtmpActivity.this, "Disconnected", Toast.LENGTH_SHORT).show();
}
});
}
@Override
public void onAuthErrorRtmp() {
runOnUiThread(new Runnable() {
@Override
public void run() {
Toast.makeText(OpenGlRtmpActivity.this, "Auth error", Toast.LENGTH_SHORT).show();
}
});
}
@Override
public void onAuthSuccessRtmp() {
runOnUiThread(new Runnable() {
@Override
public void run() {
Toast.makeText(OpenGlRtmpActivity.this, "Auth success", Toast.LENGTH_SHORT).show();
}
});
}
@Override
public void onClick(View view) {
switch (view.getId()) {
case R.id.b_start_stop:
if (!rtmpCamera1.isStreaming()) {
if (rtmpCamera1.isRecording()
|| rtmpCamera1.prepareAudio() && rtmpCamera1.prepareVideo()) {
button.setText(R.string.stop_button);
rtmpCamera1.startStream(etUrl.getText().toString());
} else {
Toast.makeText(this, "Error preparing stream, This device cant do it",
Toast.LENGTH_SHORT).show();
}
} else {
button.setText(R.string.start_button);
rtmpCamera1.stopStream();
}
break;
case R.id.switch_camera:
try {
rtmpCamera1.switchCamera();
} catch (CameraOpenException e) {
Toast.makeText(this, e.getMessage(), Toast.LENGTH_SHORT).show();
}
break;
case R.id.b_record:
if (!rtmpCamera1.isRecording()) {
try {
if (!folder.exists()) {
folder.mkdir();
}
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd_HHmmss", Locale.getDefault());
currentDateAndTime = sdf.format(new Date());
if (!rtmpCamera1.isStreaming()) {
if (rtmpCamera1.prepareAudio() && rtmpCamera1.prepareVideo()) {
rtmpCamera1.startRecord(
folder.getAbsolutePath() + "/" + currentDateAndTime + ".mp4");
bRecord.setText(R.string.stop_record);
Toast.makeText(this, "Recording... ", Toast.LENGTH_SHORT).show();
} else {
Toast.makeText(this, "Error preparing stream, This device cant do it",
Toast.LENGTH_SHORT).show();
}
} else {
rtmpCamera1.startRecord(folder.getAbsolutePath() + "/" + currentDateAndTime + ".mp4");
bRecord.setText(R.string.stop_record);
Toast.makeText(this, "Recording... ", Toast.LENGTH_SHORT).show();
}
} catch (IOException e) {
rtmpCamera1.stopRecord();
bRecord.setText(R.string.start_record);
Toast.makeText(this, e.getMessage(), Toast.LENGTH_SHORT).show();
}
} else {
rtmpCamera1.stopRecord();
bRecord.setText(R.string.start_record);
Toast.makeText(this,
"file " + currentDateAndTime + ".mp4 saved in " + folder.getAbsolutePath(),
Toast.LENGTH_SHORT).show();
currentDateAndTime = "";
}
break;
default:
break;
}
}
@Override
public void surfaceCreated(SurfaceHolder surfaceHolder) {
}
@Override
public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i1, int i2) {
rtmpCamera1.startPreview();
sensorRotationManager.start();
}
@Override
public void surfaceDestroyed(SurfaceHolder surfaceHolder) {
sensorRotationManager.stop();
if (rtmpCamera1.isRecording()) {
rtmpCamera1.stopRecord();
bRecord.setText(R.string.start_record);
Toast.makeText(this,
"file " + currentDateAndTime + ".mp4 saved in " + folder.getAbsolutePath(),
Toast.LENGTH_SHORT).show();
currentDateAndTime = "";
}
if (rtmpCamera1.isStreaming()) {
rtmpCamera1.stopStream();
button.setText(getResources().getString(R.string.start_button));
}
rtmpCamera1.stopPreview();
}
@Override
public boolean onTouch(View view, MotionEvent motionEvent) {
if (spriteGestureController.spriteTouched(view, motionEvent)) {
spriteGestureController.moveSprite(view, motionEvent);
spriteGestureController.scaleSprite(motionEvent);
return true;
}
return false;
}
}
@pedroSG94 I've tried your code using the node rtmp server + vlc and no luck. I've noticed the higher resolution I set the smaller stream view is in the landscape. 1280x720 - https://cl.ly/2ffbf98af0fd 640x480 - https://cl.ly/31516054a208 Same result on YouTube, Vimeo, VLC
- Could you try it using 1280x720?
- Could you make vlc full screen as maybe it just wraps the content?
- Could you try using either Youtube or Vimeo? For Vimeo you just have to sign up with google.
- How do you set
landscapeandportraitmode in code, maybe this is something I do wrong?
I tried with 1280x720 and all is working fine. The right side is my second screen so I'm using VLC full screen in my left screen.
I will try with YouTube but you need wait 24h because I enabled stream access today and YouTube need 24h to do it .In Vimeo I need provide card info and I don't want do it. (The screenshoot is in spanish if you want traduce it to english).
About set landscape and portrait. I only change screen orientation in manifest:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/app/src/main/AndroidManifest.xml#L93
@pedroSG94 You don't have to provide any card to Vimeo.
Open this link https://vimeo.com/features/livestreaming
Click start free demo, then sign up with google, then switch to Connect (RTMP) tab to get stream url and key and that's all.
Try with this app to discard that you are using an old commit or bad code: app.zip You must use OpenGl RTMP If you have a different result, maybe the code is wrong for some devices and we need know where is the problem
Thank you @pedroSG94. Your sample works well. I've noticed that you set activity orientation to landscape and once I did the same it worked well. So this was the issue. Now, the question here is how to make it work when activity orientation is set to portrait? Because currently, the way stream works strongly depends on activity orientation. Can I change it programmatically so I have the same behavior as in your .apk, but having Activity in a portrait? I've played with all possible combinations using prepareVideo() method, passing all possible rotations and setting hardwareRotation to true as well, but no luck.
You only need go to manifest and change screen orientation to portrait. Other way is let device rotate and block rotation before start stream. Remember that start stream in portrait if you are using 1280x720 resolution you are rotating stream to 720x1280 and that could be the problem that you see in vimeo
@pedroSG94 This is exactly my issue, so setting activity orientation to portrait breaks it. The problem is that it is my app's UI requirement to have the activity in portrait and the stream in landscape. And I think it's quite a normal use case. That's why I believe the stream should be independent of activity orientation. It would be cool to have control over it in code and not depend on activity orientation. For now, I can't figure out what to modify to achieve this, but I'll have a deeper look at it tomorrow and maybe I'll propose some solution to that.
This isn't a bug. I shared you that effect with VLC tests images.
The reason about it is that I need keep compatibility with SurfaceView and TextureView options but in that cases you can't use opengl and your case is not possible without opengl. Because you only modify stream result and with SurfaceView or TextureView you must show in preview exactly that same that you are streaming. To fix it you need delete line that change stream resolution in VideoEncoder and modify preview and stream result glviewport/matrix
I know how to do it. The question is how to keep compatibility with SurfaceView and TextureView without create an unmaintainable code. If you have any idea let me know it. I'm looking a solution too.
Also I think that your case and actual code should be optional (I want keep both if possible).
Hello, For now, you can try this (Only working in portrait locked):
-
Go to VideoEncoder and comment lines 104, 105, 106, 107 and 110: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/encoder/src/main/java/com/pedro/encoder/video/VideoEncoder.java#L104
-
Go to SizeCalculator and replace to this:
package com.pedro.encoder.utils.gl;
import android.graphics.PointF;
import android.opengl.GLES20;
import android.opengl.Matrix;
/**
* Created by pedro on 22/03/19.
*/
public class SizeCalculator {
public static void calculateViewPort(boolean keepAspectRatio, int mode, int previewWidth,
int previewHeight, int streamWidth, int streamHeight) {
if (keepAspectRatio) {
if (previewWidth > previewHeight) { //landscape
if (mode == 0) { //adjust
int realWidth = previewHeight * streamWidth / streamHeight;
GLES20.glViewport((previewWidth - realWidth) / 2, 0, realWidth, previewHeight);
} else { //fill
int realHeight = previewWidth * streamHeight / streamWidth;
GLES20.glViewport(0, -((realHeight - previewWidth) / 2), previewWidth, realHeight);
}
} else { //portrait
if (mode == 0) { //adjust
int realHeight = previewWidth * streamHeight / streamWidth;
GLES20.glViewport(0, (previewHeight - realHeight) / 2, previewWidth, realHeight);
} else { //fill
int realWidth = previewHeight * streamWidth / streamHeight;
GLES20.glViewport(-((realWidth - previewWidth) / 2), 0, realWidth, previewHeight);
}
}
} else {
GLES20.glViewport(0, 0, previewHeight, previewWidth);
}
}
public static void updateMatrix(int rotation, int width, int height, boolean isPreview,
boolean isPortrait, float[] MVPMatrix) {
Matrix.setIdentityM(MVPMatrix, 0);
PointF scale = getScale(rotation, width, height, isPortrait, isPreview);
Matrix.scaleM(MVPMatrix, 0, scale.x, scale.y, 1f);
if (!isPreview && !isPortrait) rotation += 90;
Matrix.rotateM(MVPMatrix, 0, rotation, 0f, 0f, -1f);
}
private static PointF getScale(int rotation, int width, int height, boolean isPortrait,
boolean isPreview) {
float scaleX = 1f;
float scaleY = 1f;
if (!isPreview) {
if (isPortrait && rotation != 90 && rotation != 270) { //portrait
final float adjustedWidth = width * (width / (float) height);
scaleX = adjustedWidth / height;
//} else if (!isPortrait && rotation != 90 && rotation != 270) { //landscape
// final float adjustedWidth = height * (height / (float) width);
// scaleX = adjustedWidth / width;
}
}
return new PointF(scaleX, scaleY);
}
}
To keep compatibility with other modes. Now, you need specify to view that you want rotate stream on fly using this params(version 1.8.1): https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/res/values/attrs.xml#L13 https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/res/values/attrs.xml#L14
Thanks @pedroSG94 . What's the difference between them? I can see the logic is the same for both of them? I've tested it and portrait orientation works very well with both portrait and landscape rotation. When setting landscape activity orientation the stream works very well but the preview is clipped.
Is it possible with current approach, as it is actually my concern, to have portrait orientation in preview and landscape on the stream?
Could you tell me which part of the code is responsible for setting stream orientation? I know you can modify it by changing video encoder orientation or size, but it would also affect the preview. I was looking for a place in code where I can modify stream orientation. I've tried changing viewport and adding Matrix.rotateM in ScreenRenderer and CameraRenderer but it only affects the image and not the container in other worlds the video is changing but it's still trimmed to the container. I'm not sure if I'm clear enough?
Hi,
After modify VideoEncoder and use that code in SizeCalculator the result should be this:
- Only work with portrait screen locked (cause I thought, you only wanted portrait. Maybe I'm wrong). No tested with landscape.
- If you are streaming with 1280x720 your stream resolution is always 1280x720 (in portrait and landscape) not 720x1280 like before if you are streaming in portrait.
- Portrait stream result is 1280x720 with black bands in left and right to keep aspect ratio and when you rotate to landscape is full screen stream (assume that your player is 16:9 resolution).
In SizeCalculator I changed:
} else {
GLES20.glViewport(0, 0, previewHeight, previewWidth);
}
Swap width and height to keep viewport full screen. Also updateMatrix and getScale to adapt stream result and keep aspect ratio (avoid distortion).
If you want modify only stream result you only need check isPreview boolean and modify MVPMatrix rotation in updateMatrix method. Remember that if you want keep without changes in preview you should set rotation only when you draw to stream result.
- Draw in preview(set rotation 0 to keep without changes): https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/java/com/pedro/rtplibrary/view/OpenGlView.java#L130
- Draw in stream (set sensor rotation to rotate on fly): https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtplibrary/src/main/java/com/pedro/rtplibrary/view/OpenGlView.java#L141
Thank you @pedroSG94
Basically what I need is to have a preview in portrait orientation and stream in landscape orientation.
So the preview looks like a regular camera app and stream looks like Youtube video (narrowed in portrait and wide in landscape). I can't really decouple stream orientation and activity orientation. Changing viewport(height, width) doesn't result in changing orientation, it only changes the drawing area in the window. So the "window" or "container" is still in portrait but the drawing area is rotated. It's worth setting GLES20.glClearColor(0.0f, 255f, 0.0f, 1.0f); so the drawing area is green. That way it's easier to notice drawing area as it doesn't blend with black stripes :)
But yeah..it's my case, some other people may want to have landscape preview orientation and landscape stream orientation (like Streamlabs app). Anyways these two cases are quite common. I think it would be cool to have it. I guess we'd need to have clear distinctions between: stream orientation, stream rotation (image), preview orientation (activity orientation) and preview rotation (image). Let me know if everything is clear.
Hi,
I did a new branch with that implementation as I understood you:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/tree/orientation
Download that commit, use Opengl Rtmp example and let me know if this is your case or I'm wrong in something.
With device screen locked to portrait (always 1280x720 resolution):
Rotated to portrait:
Rotated to landscape:
If this is your case. The question now, is how to implement it without break SurfaceVIew and TextureView that can't implement it cause opengl is not used.
@pedroSG94 Exactly this is my case :) And indeed, how to implement it to not break other stuff. That's why I was digging in the code and tried to do it. Unfortunately, I don't get how to change orientation dynamically after it's being set in video encoder. Changing viewport and rotation in OpenGL only affects the video image and not orientation. So what is the container so we can manipulate it? I was thinking that's Surface from VideoEncoder (VideoEncoder.getInputSurface), but I've tried to add rotation to OpenGlView.getSurfaceTexture() using getTransformMatrix but it didn't work. I might be doing something wrong though. I definitely lack knowledge here yet :)