HTML5 + JavaScript + CSS3 RGBA video overlays on top of live GStreamer video pipelines

GStreamer comes with a number of plugins that allow rendering of text and/or graphics overlays on top of video: rsvgoverlay, subtitleoverlay, textoverlay, cairooverlay, gdkpixbufoverlay, opencvtextoverlay, etc. However, some of these plugins often allow only static graphics and text, and often do not approach the flexibility and power of dedicated video post-processing software products.

“noweffects” (a play on the name of a popular video post-processing software) is a proof-of-concept of leveraging the power of a modern HTML5 + JavaScript + CSS3 web browser engine to render high-quality, programmable, alpha-aware, animated, vector- and bitmap based content, which is then rendered into an RGBA raw video stream, which can then be transferred via some kind of IPC method to separate GStreamer processeses, where it can be composited with other content via GStreamers regular compositor or videomixer plugins.

Qt was chosen for its ease of integration of modern WebKit (QtWebKit) and GStreamer (qt-gstreamer), and its ability to render widgets to RGBA images. The QMainWindow widget is rendered in regular intervals to QImages in RGBA format, then inserted into a GStreamer pipeline via the appsrc plugin. This pipeline simply uses udpsink to multicast the raw video RTP packets on localhost to allow for multiple ‘subscribers’. A second GStreamer pipleline can then use udpsrc and apply the overlay.

Proof-of-concept code available at: https://github.com/michaelfranzl/noweffects

The following demonstration video was generated with “noweffects”: A website (showing CSS3 animations), rendered to an RGBA video via QtWebKit, then overlaid on top of a video test pattern in a separate GStreamer process.

GStreamer pipeline recipe: Stream file via RTP using rtpbin

Given an audio/video file encoded with

then the following GStreamer pipeline (I’m using version 1.13.1) will stream it via RTP using rtpbin to localhost ports 50000-50003:

The receiver outputting the media to screen and speakers:

Notes/Gotchas

  • The sender uses almost no CPU because the media is not transcoded.
  • Instead of vp9dec you can use avdec_vp9.
  • The sync attributes must be specified exactly as given.
  • When the sender is restarted while the client is running, the client terminates with the error streaming stopped, reason not-linked.
  • media, encoding-name and clock-rate attributes are required.
  • The encoding names specified in the receiver must match those of the sender.
  • It is not necessary to specify RTP payload numbers.
  • If on the receiver  sync=false, audio and video are not in sync.
  • Above example only supports one receiver. To support multiple receivers, you can multicast the UDP packets to the loopback network device with the following modifications:
    • udpsink options: host=225.0.0.37 auto-multicast=true multicast-iface=lo ttl-mc=0 bind-address=127.0.0.1
    • udpsrc options: address=225.0.0.37 auto-multicast=true multicast-iface=lo ttl-mc=0 bind-address=127.0.0.1
    • Note: 225.0.0.37 is just an example multicast address. ttl-mc=0 is important, otherwise the packets will be forwarded across network boundaries. You should be careful with multicasting, and educate yourself before you try it.

Receiver without rtpbin

To receive without using rtpbin:

Here, the sender can be restarted without bringing the receiver down.

PhantomJS alternative: Write short PyQt scripts instead (phantom.py)

For a project of mine, I needed a ‘headless’ script that renders dynamically generated HTML (via JavaScript) to a PDF file. In looking for existing headless browsers, I found PhantomJS, CasperJS and similar projects.

PhantomJS looked most promising, but it had bugs related to the CSS @media type ‘print’ which made the project useless for generating properly formatted PDFs.[1][2] These issues are still open after three years. As of 2017, the official PhantomJS binary download is still at version 2.1 which includes an older downstream QtWebKit version 5.5.1 [3].

In trying to fix the bugs myself, or upgrade QtWebKit, I spent two full days trying to build PhantomJS, but eventually had to give up because the build system seems to be reworked right now. It seems that instead of patching the custom QtWebKit, the development team is moving towards making “PhantomJS a normal Qt application with system-installed Qt and QtWebKit”. [4]

So, I started questioning PhantomJS. What magic does it really do? By inspecting the source code, I found out that its entire concept hinges on just two Qt functions: QWebFrame::addToJavaScriptWindowObject() and QWebFrame::evaluateJavascript(). The former allows exposing Qt objects to the JavaScript space of WebKit. For example, PhantomJS’s JavaScript API method injectJs is simply a wrapper for an underlying evaluateJavascript call inside of a C++ function of a C++ object which has been previously attached via addToJavaScriptWindowObject.

In short, PhantomJS’s custom (and in my opinion rather sparse) JavaScript API simply calls Qt’s C++ functions.

The obvious disadvantage of this concept is that an extension or fixing of the API requires a recompilation of the C++ code.

It is obviously much simpler to directly script Qt, for example via PyQt.

So, I started writing a short PyQt application, and after just 90 lines of Python code, I had what I needed: a headless browser using an up-to-date version of WebKit, which did not have the shortcomings of the version in PhantomJS.

The script is published on my blog and as a Github gist. I don’t expect getting 20,000 Github stars like others, but honestly, following (perhaps too simple) logic, my phantom.py script should get more. 😉

phantom.py: A lean replacement for bulky headless browser frameworks

 

References:

[1]: https://github.com/ariya/phantomjs/issues/12685

[2]: https://github.com/ariya/phantomjs/issues/13524

[3]: https://github.com/ariya/phantomjs/tree/2.1.1/src/qt

[4]: https://github.com/ariya/phantomjs/issues/14458#issuecomment-242391757

phantom.py: A lean replacement for bulky headless browser frameworks

This is a simple but fully scriptable headless QtWebKit browser using PyQt5 in Python3, specialized in executing external JavaScript and generating PDF files. A lean replacement for other bulky headless browser frameworks. (Source code at end of this post as well as in this github gist)

Usage

If you have a display attached:

If you don’t have a display attached (i.e. on a remote server):

Arguments:

  • <url> Can be a http(s) URL or a path to a local file
  • <pdf-file> Path and name of PDF file to generate
  • [<javascript-file>] (optional) Path and name of a JavaScript file to execute

Features

  • Generate a PDF screenshot of the web page after it is completely loaded.
  • Optionally execute a local JavaScript file specified by the argument <javascript-file> after the web page is completely loaded, and before the PDF is generated.
  • console.log’s will be printed to stdout.
  • Easily add new features by changing the source code of this script, without compiling C++ code. For more advanced applications, consider attaching PyQt objects/methods to WebKit’s JavaScript space by using  QWebFrame::addToJavaScriptWindowObject().

If you execute an external <javascript-file>, phantom.py has no way of knowing when that script has finished doing its work. For this reason, the external script should execute  console.log("__PHANTOM_PY_DONE__"); when done. This will trigger the PDF generation, after which phantom.py will exit. If no  __PHANTOM_PY_DONE__ string is seen on the console for 10 seconds, phantom.py will exit without doing anything. This behavior could be implemented more elegantly without console.log’s but it is the simplest solution.

It is important to remember that since you’re just running WebKit, you can use everything that WebKit supports, including the usual JS client libraries, CSS, CSS @media types, etc.

Dependencies

  • Python3
  • PyQt5
  • xvfb (optional for display-less machines)

Installation of dependencies in Debian Stretch is easy:

Finding the equivalent for other OSes is an exercise that I leave to you.

Examples

Given the following file /tmp/test.html:

… and the following file /tmp/test.js:

… and running this script (without attached display) …

… you will get a PDF file /tmp/out.pdf with the contents “foo bar baz”.

Note that the second occurrence of “foo” has been replaced by the web page’s own script, and the third occurrence of “foo” by the external JS file.

Source Code

 

“Open Source” does not imply “less secure”

Sometimes programmers hesitate to make their software open source because they think that revelation of the source code would allow attackers to ‘hack it’.

Certainly there are specific cases where this is true, but not as a general rule.

In my opinion, if inspection of the source code allows an attacker to ‘hack it’, then the programmer has done it wrong. Security primarily comes from writing secure algorithms, independent of their open source nature.

OpenSSL is a case in point: It is open source, but nevertheless it powers HTTPS all over the internet. “But,” you say, “it is only secure because its code is kinda obscure.” Well, no: Cryptographically secure algorithms exhibit very astonishing properties. For example, the One-time pad encryption technique is extremely simple and exhibits “Perfect secrecy” which is defined by Wikipedia as follows:

One-time pads are “information-theoretically secure” in that the encrypted message (i.e., the ciphertext) provides no information about the original message to a cryptanalyst (except the maximum possible length of the message). This is a very strong notion of security first developed during WWII by Claude Shannon and proved, mathematically, to be true for the one-time pad by Shannon about the same time. His result was published in the Bell Labs Technical Journal in 1949. Properly used, one-time pads are secure in this sense even against adversaries with infinite computational power.

Claude Shannon proved, using information theory considerations, that the one-time pad has a property he termed perfect secrecy; that is, the ciphertext C gives absolutely no additional information about the plaintext. This is because, given a truly random key which is used only once, a ciphertext can be translated into any plaintext of the same length, and all are equally likely.

Take for example the following simple implementation of the One-time pad (via XOR) in Ruby (which took me just a couple of minutes to write):

This code is open source, but it nevertheless exhibits the property of perfect (i.e. 100%) security “even against adversaries with infinite computational power” — given that the key is never submitted over insecure channels.

Sure, the One-time pad is not practical, and one could probably exploit weaknesses in Ruby or the underlying operating system. But that is not the point. The point is that, given proper implementation of software, it can be made open source without compromising its security.

To contrast this with an example of (bad) source code which should not be made public because it only creates a false sense of security:

Here, ((msgraw[n] + 7) ^ 99) is equivalent to a hard-coded secret. Sure, the obscured message, when transmitted over a public network, may look random. But the algorithm could easily be reverse-engineered by cryptoanalysis. Also, if the souce code were revealed, it would be trivial to decode past and future messages.

Conclusion

“Open Source” does not imply “insecure”. Security comes from secure — not secret — algorithms (which of course includes the freedom of bugs). What counts as “secure” is defined mathematically, and “mathematics (and in extension, physics) can’t be bribed.” It is not easy to come up with such algorithms, but it is possible, and there are many successful examples.

Needless to say, not every little piece of code should be made open source — ideally programmers will only publish generally useful and readable software which they intend to maintain, but that is a subject for another blog post.

How to install yubikey-manager on Debian

yubikey-manager is a Python application requiring some dependencies for it to be installed from the Python repositories, because it is not yet in the official Debian package repository. Here is how:

Here is the main commandline utility:

WooCommerce Shipping Plugin “External Fetch”

I just wrote the following shipping plugin for WooCommerce because existing plugins would not cover the case I’m working at. It is available on my github account under the permissive MIT license.

https://github.com/michaelfranzl/woocommerce-shipping-external-fetch

woocommerce-shipping-external-fetch

Shipping Plugin for WooCommerce which HTTP PUTs the cart contents in JSON format to an external web service specified by protocol/host/port/URI, and receives the calculated shipping offer (costs, labels etc.) as JSON from that webservice.

Useful when the shipping calculation complexity exceeds the capabilities of other shipping calculation plugins.

The external shipping calculation application is, of course, business specific and thus not included in this repository. However, an example of a JSON request and reponse is shown below.

When the webservice is not reachable, not responsive, or returns HTTP status codes other than 200 (i.e. when it experiences a server error), this plugin offers free shipping to the customer, since technical problems are the ‘fault’ of the store owner and should not prevent a customer from completing an order. A setting to configure this behavior is not included, but it would be easy to add.

This plugin supports WooCommerce shipping zones. So, in theory you could have different web services dedicated to different shipping zones.

To add the plugin to an existing shipping zone:

  1. Go to WooCommerce -> Settings -> Shipping
  2. Click on “Manage shipping methods” below a Zone
  3. Click “Add shipping method” button
  4. Select “External Fetch” and click “Add shipping method” button
  5. Configure the Plugin by clicking on “Edit”
  6. Customize “Method title” and “Method description” and set the “JSON API Endpoint” to, for example  http://localhost:4040/calculate

At this point you can add a product to the cart, set up a simple webserver listening at port 4040 on the same machine, and receive/send JSON as shown in the section “Examples” further below.

This plugin is for DEVELOPERS only and will likely remain in an ALPHA state.

License

Copyright 2017 Michael Franzl

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Examples

Cart contents in JSON format as sent by this plugin:

 

Response in JSON format as expected by this plugin:

This plugin will add the  description values in a  &lt;div class="shipping_rate_description" below each selectable shipping rate entry on the checkout page to give the customer a better idea about the shipping method.

You can put a message into  cart_no_shipping_available_html which will be added to the default WooCommerce message when there are no shipping methods available.

 

vim – Unable to paste with middle mouse button – E353: Nothing in register ” error – Solution

Put set clipboard=unnamed within your ~/.vimrc VIM configuration file.

Re-posted from https://linuxconfig.org/vim-unable-to-paste-e353-nothing-in-register-error-solution

How to compile ezstream from source

Debian Stretch’s version of ezstream is currently a bit out of date. Here is how you compile ezstream from source to get the latest improvements and bugfixes. Not even the INSTALL file in the ezstream repo has all the steps:

Note that the configuration file structure has changed from what can be found on older blog posts on the internet. For example, to pipe OGG Vorbis data into ezstream without re-encoding, you can use something like teststream.xml:

Then, to stream 30 seconds of brown noise with a sine sweep to an Icecast server for testing purposes: