From d6f3599d3e745e7b98a4d9a0e0529a8b1fa1f76e Mon Sep 17 00:00:00 2001 From: fduncanh Date: Mon, 28 Feb 2022 12:21:18 -0500 Subject: [PATCH] add option for modifying h264parse in video pipeline --- README.html | 4 +++- README.md | 21 +++++++++++++---- README.txt | 34 ++++++++++++++++++++-------- renderers/video_renderer.h | 4 ++-- renderers/video_renderer_gstreamer.c | 7 +++--- uxplay.1 | 2 ++ uxplay.cpp | 10 ++++++-- 7 files changed, 60 insertions(+), 22 deletions(-) diff --git a/README.html b/README.html index fcf40a0..928fa3e 100644 --- a/README.html +++ b/README.html @@ -18,7 +18,9 @@

UxPlay uses GStreamer Plugins for rendering audio and video, This means that video and audio are supported “out of the box”, using a choice of plugins. AirPlay streams video in h264 format: gstreamer decoding is plugin agnostic, and uses accelerated GPU hardware h264 decoders if available; if not, software decoding is used.

For systems with Intel integrated graphics, hardware GPU decoding with the gstreamer VAAPI plugin is preferable. VAAPI is open-source, and in addition to Intel, can support some AMD GPU’s (the open-source “Nouveau” drivers for NVIDIA graphics are also in principle supported when VAAPI is supplemented with firmware extracted from the proprietary NVIDIA drivers).

For NVIDIA graphics with the proprietary drivers, the nvdec plugin (recently renamed nvh264dec) can be used for accelerated video decoding on the NVIDIA GPU with CUDA. The nvdec plugin is part of gstreamer1.0-plugins-bad, but is generally not included in binary packages, as NVIDIA’s proprietary Video Codec SDK must be downloaded, and three header files from it must be added to the gstreamer source before the plugin can be compiled. Users must do this themselves: see these instructions, and adapt them as necessary for your GStreamer installation. This plugin should be used with the -vd nvdec (or nvh264dec) and -vs glimagesink uxplay options.

-

The Video4Linux2 decoder v4l2h264dec (from gstreamer1.0-plugins-good) is the appropriate choice for the Broadcom GPU in the Raspberry Pi 4, (UxPlay does not run well on the Raspberry Pi if GPU hardware h264 video decoding is not used, as its CPU is not powerful enough for satisfactory software-based decoding without large latency); uxplay options can be used to create GStreamer video pipelines to use non-VAAPI hardware decoders, and a special option -rpi creates a pipeline appropriate for the Raspberry Pi. (This is “Work In Progess”: it does not yet consistently work.)

+

Note to packagers: OpenSSL-3.0.0 solves GPL v3 license issues.

Some Linux distributions such as Debian do not allow distribution of compiled GPL code linked to OpenSSL-1.1.1 because its “dual OpenSSL/SSLeay” license has some incompatibilities with GPL, unless all code authors have explicitly given an “exception” to allow such linking (the historical origins of UxPlay make this impossible to obtain). Other distributions treat OpenSSL as a “System Library” which the GPL allows linking to.

For “GPL-strict” distributions, UxPlay can be built using OpenSSL- 3.0.0, which has a new GPLv3-compatible license.

diff --git a/README.md b/README.md index 4d53312..bc24fbf 100644 --- a/README.md +++ b/README.md @@ -76,11 +76,22 @@ Users must do this themselves: see [these instructions](https://gist.github.com/ adapt them as necessary for your GStreamer installation. This plugin should be used with the `-vd nvdec` (or nvh264dec) and `-vs glimagesink` uxplay options. -The Video4Linux2 decoder v4l2h264dec (from gstreamer1.0-plugins-good) is the appropriate choice for the Broadcom GPU in the -Raspberry Pi 4, (UxPlay does not run well on the Raspberry Pi if GPU hardware h264 video decoding is not used, -as its CPU is not powerful enough for satisfactory software-based decoding without large latency); uxplay options can be used -to create GStreamer video pipelines to use non-VAAPI hardware decoders, and a special option `-rpi` creates a pipeline -appropriate for the Raspberry Pi. _(This is "Work In Progess": it does not yet consistently work.)_ +* **support for Raspberry Pi** (_work in progess_): the R Pi platform has long been + supported by [RPiPlay](http://github.com/FD-/RPiPlay) using the deprecated 32-bit-only omx (OpenMAX) decoder, now + removed from latest Raspberry Pi OS (Bullseye), leading to user interest in getting UxPlay to work + on the R Pi (at least on model 4). UxPlay can work with software h264 video decoding (option `-avdec`), but without hardware decoding + has unacceptable latency on the Pi. The Video4Linux2 GStreamer plugin v4l2h264dec (from gstreamer1.0-plugins-good) is now the + appropriate choice for accelerated + hardware video decoding by the Broadcom GPU on the Pi. UxPlay options can be used to create GStreamer video pipelines to use + non-VAAPI hardware decoders, and a new UxPlay option `-rpi` creates a pipeline that should be appropriate for the Raspberry Pi, + and when it has worked, it has worked well. However, it usually does not work because of a "caps" mismatch between source and + sink, which could be fixable with GStreamer "know-how". The GStreamer + video pipeline is "` (decrypted h264 stream from iOS)! queue ! h264parse ! decodebin ! videoconvert ! autovideosink sync=false`" + and the uxplay options `-vp , -vd, -vc ,-vs` can be used to + replace `h264parse, decodebin, videoconvert, autovideosink` by the user's + choices. (`-rpi` is equivalent to '-vd v4l2h264dec -vc v4l2convert`). The first frame in the h264 video stream sends the "caps" + details (SPS, PPS) to h264parse. UxPlay users with the R Pi (model 4 recommended) are invited to experiment and report any successes. + ### Note to packagers: OpenSSL-3.0.0 solves GPL v3 license issues. diff --git a/README.txt b/README.txt index 5089f2a..e5de038 100644 --- a/README.txt +++ b/README.txt @@ -100,15 +100,31 @@ and adapt them as necessary for your GStreamer installation. This plugin should be used with the `-vd nvdec` (or nvh264dec) and `-vs glimagesink` uxplay options. -The Video4Linux2 decoder v4l2h264dec (from gstreamer1.0-plugins-good) is -the appropriate choice for the Broadcom GPU in the Raspberry Pi 4, -(UxPlay does not run well on the Raspberry Pi if GPU hardware h264 video -decoding is not used, as its CPU is not powerful enough for satisfactory -software-based decoding without large latency); uxplay options can be -used to create GStreamer video pipelines to use non-VAAPI hardware -decoders, and a special option `-rpi` creates a pipeline appropriate for -the Raspberry Pi. *(This is "Work In Progess": it does not yet -consistently work.)* +- **support for Raspberry Pi** (*work in progess*): the R Pi platform + has long been supported by [RPiPlay](http://github.com/FD-/RPiPlay) + using the deprecated 32-bit-only omx (OpenMAX) decoder, now removed + from latest Raspberry Pi OS (Bullseye), leading to user interest in + getting UxPlay to work on the R Pi (at least on model 4). UxPlay can + work with software h264 video decoding (option `-avdec`), but + without hardware decoding has unacceptable latency on the Pi. The + Video4Linux2 GStreamer plugin v4l2h264dec (from + gstreamer1.0-plugins-good) is now the appropriate choice for + accelerated hardware video decoding by the Broadcom GPU on the Pi. + UxPlay options can be used to create GStreamer video pipelines to + use non-VAAPI hardware decoders, and a new UxPlay option `-rpi` + creates a pipeline that should be appropriate for the Raspberry Pi, + and when it has worked, it has worked well. However, it usually does + not work because of a "caps" mismatch between source and sink, which + could be fixable with GStreamer "know-how". The GStreamer video + pipeline is + "`(decrypted h264 stream from iOS)! queue ! h264parse ! decodebin ! videoconvert ! autovideosink sync=false`" + and the uxplay options `-vp , -vd, -vc ,-vs` can be used to replace + `h264parse, decodebin, videoconvert, autovideosink` by the user's + choices. (`-rpi` is equivalent to '-vd v4l2h264dec -vc + v4l2convert\`). The first frame in the h264 video stream sends the + "caps" details (SPS, PPS) to h264parse. UxPlay users with the R Pi + (model 4 recommended) are invited to experiment and report any + successes. ### Note to packagers: OpenSSL-3.0.0 solves GPL v3 license issues. diff --git a/renderers/video_renderer.h b/renderers/video_renderer.h index 1906111..5e9fb95 100644 --- a/renderers/video_renderer.h +++ b/renderers/video_renderer.h @@ -45,8 +45,8 @@ typedef enum videoflip_e { typedef struct video_renderer_s video_renderer_t; -void video_renderer_init (logger_t *logger, const char *server_name, videoflip_t videoflip[2], const char *decoder, - const char *converter, const char *videosink); +void video_renderer_init (logger_t *logger, const char *server_name, videoflip_t videoflip[2], const char *parser, + const char *decoder, const char *converter, const char *videosink); void video_renderer_start (); void video_renderer_stop (); void video_renderer_render_buffer (raop_ntp_t *ntp, unsigned char* data, int data_len, uint64_t pts, int type); diff --git a/renderers/video_renderer_gstreamer.c b/renderers/video_renderer_gstreamer.c index bf1deb3..c026844 100644 --- a/renderers/video_renderer_gstreamer.c +++ b/renderers/video_renderer_gstreamer.c @@ -105,8 +105,8 @@ void video_renderer_size(float *f_width_source, float *f_height_source, float *f logger_log(logger, LOGGER_DEBUG, "begin video stream wxh = %dx%d; source %dx%d", width, height, width_source, height_source); } -void video_renderer_init(logger_t *render_logger, const char *server_name, videoflip_t videoflip[2], const char *decoder, - const char *converter, const char *videosink) { +void video_renderer_init(logger_t *render_logger, const char *server_name, videoflip_t videoflip[2], const char *parser, + const char *decoder, const char *converter, const char *videosink) { GError *error = NULL; logger = render_logger; @@ -123,7 +123,8 @@ void video_renderer_init(logger_t *render_logger, const char *server_name, vide gst_init(NULL,NULL); GString *launch = g_string_new("appsrc name=video_source stream-type=0 format=GST_FORMAT_TIME is-live=true ! "); g_string_append(launch, "queue ! "); - g_string_append(launch, "h264parse ! "); + g_string_append(launch, parser); + g_string_append(launch, " ! "); g_string_append(launch, decoder); g_string_append(launch, " ! "); g_string_append(launch, converter); diff --git a/uxplay.1 b/uxplay.1 index c361353..f533666 100644 --- a/uxplay.1 +++ b/uxplay.1 @@ -37,6 +37,8 @@ UxPlay 1.48: An open\-source AirPlay mirroring server based on RPiPlay .TP \fB\-t\fR n Relaunch server if no connection existed in last n seconds .TP +\fB\-vp\fI prs \fR Choose GStreamer h264 parser; default "h264parse" +.TP \fB\-vd\fI dec \fR Choose GStreamer h264 decoder; default "decodebin" .IP choices: (software) avdec_h264; (hardware) v4l2h264dec, diff --git a/uxplay.cpp b/uxplay.cpp index 9de38bc..67b46e9 100644 --- a/uxplay.cpp +++ b/uxplay.cpp @@ -77,6 +77,7 @@ static std::string audiosink = "autoaudiosink"; static bool use_audio = true; static bool new_window_closing_behavior = true; static bool close_window; +static std::string video_parser = "h264parse"; static std::string video_decoder = "decodebin"; static std::string video_converter = "videoconvert"; static bool show_client_FPS_data = false; @@ -226,6 +227,7 @@ static void print_info (char *name) { printf(" \"-p tcp n\" or \"-p udp n\" sets TCP or UDP ports separately\n"); printf("-m Use random MAC address (use for concurrent UxPlay's)\n"); printf("-t n Relaunch server if no connection existed in last n seconds\n"); + printf("-vp ... Choose the GSteamer h264 parser: default \"h264parse\"\n"); printf("-vd ... Choose the GStreamer h264 decoder; default \"decodebin\"\n"); printf(" choices: (software) avdec_h264; (hardware) v4l2h264dec,\n"); printf(" nvdec, nvh264dec, vaapih64dec, vtdec,etc.\n"); @@ -450,6 +452,10 @@ int main (int argc, char *argv[]) { } else if (arg == "-h" || arg == "-v") { print_info(argv[0]); exit(0); + } else if (arg == "-vp") { + if (!option_has_value(i, argc, arg, argv[i+1])) exit(1); + video_parser.erase(); + video_parser.append(argv[++i]); } else if (arg == "-vd") { if (!option_has_value(i, argc, arg, argv[i+1])) exit(1); video_decoder.erase(); @@ -530,7 +536,7 @@ int main (int argc, char *argv[]) { } if (use_video) { - video_renderer_init(render_logger, server_name.c_str(), videoflip, + video_renderer_init(render_logger, server_name.c_str(), videoflip, video_parser.c_str(), video_decoder.c_str(), video_converter.c_str(), videosink.c_str()); video_renderer_start(); } @@ -569,7 +575,7 @@ int main (int argc, char *argv[]) { if (use_audio) audio_renderer_stop(); if (use_video && close_window) { video_renderer_destroy(); - video_renderer_init(render_logger, server_name.c_str(), videoflip, + video_renderer_init(render_logger, server_name.c_str(), videoflip, video_parser.c_str(), video_decoder.c_str(), video_converter.c_str(), videosink.c_str()); video_renderer_start(); }