phantom.py: A lean replacement for bulky headless browser frameworks

This is a simple but fully scriptable headless QtWebKit browser using PyQt5 in Python3, specialized in executing external JavaScript and generating PDF files. A lean replacement for other bulky headless browser frameworks. (Source code at end of this post as well as in this github gist)

Usage

If you have a display attached:

If you don’t have a display attached (i.e. on a remote server):

Arguments:

  • <url> Can be a http(s) URL or a path to a local file
  • <pdf-file> Path and name of PDF file to generate
  • [<javascript-file>] (optional) Path and name of a JavaScript file to execute

Features

  • Generate a PDF screenshot of the web page after it is completely loaded.
  • Optionally execute a local JavaScript file specified by the argument <javascript-file> after the web page is completely loaded, and before the PDF is generated.
  • console.log’s will be printed to stdout.
  • Easily add new features by changing the source code of this script, without compiling C++ code. For more advanced applications, consider attaching PyQt objects/methods to WebKit’s JavaScript space by using  QWebFrame::addToJavaScriptWindowObject().

If you execute an external <javascript-file>, phantom.py has no way of knowing when that script has finished doing its work. For this reason, the external script should execute  console.log("__PHANTOM_PY_DONE__"); when done. This will trigger the PDF generation, after which phantom.py will exit. If no  __PHANTOM_PY_DONE__ string is seen on the console for 10 seconds, phantom.py will exit without doing anything. This behavior could be implemented more elegantly without console.log’s but it is the simplest solution.

It is important to remember that since you’re just running WebKit, you can use everything that WebKit supports, including the usual JS client libraries, CSS, CSS @media types, etc.

Dependencies

  • Python3
  • PyQt5
  • xvfb (optional for display-less machines)

Installation of dependencies in Debian Stretch is easy:

Finding the equivalent for other OSes is an exercise that I leave to you.

Examples

Given the following file /tmp/test.html:

… and the following file /tmp/test.js:

… and running this script (without attached display) …

… you will get a PDF file /tmp/out.pdf with the contents “foo bar baz”.

Note that the second occurrence of “foo” has been replaced by the web page’s own script, and the third occurrence of “foo” by the external JS file.

Source Code

 

How to install yubikey-manager on Debian

yubikey-manager is a Python application requiring some dependencies for it to be installed from the Python repositories, because it is not yet in the official Debian package repository. Here is how:

Here is the main commandline utility:

Venus apparent diameter plot for 2017-2020

The apparent size of Venus varies dramatically with the Earth-Venus distance, depending on both orbits around the sun. The apparent angular diameter of Venus is between 0.175 and 1 arcminutes.

Phases of Venus and evolution of its apparent diameter. (Statis Kalyvas - VT-2004 programme)
Phases of Venus and evolution of its apparent diameter. (Statis Kalyvas – VT-2004 programme)

Wikipedia has nice images on that and says that…

“the extreme crescent phase of Venus can be seen without a telescope by those with exceptionally acute eyesight, at the limit of human perception.”

I will definitely try next time with my naked eyes, but if you have an entry-level telescope, you will be certainly be better able to see the crescent phase during closest approach.

The question is, when does Venus come closest to Earth for best observation? Not finding anything on the interwebs, I wrote a quick plotting program with Python’s ephem package and gnuplot.

Venus apparent diameter plot for 2017-2020

Graph of apparent diameter of Venus in arcminutes for the time period 2017-2020
Graph of apparent diameter of Venus in arcminutes for the time period 2017-2020

Here is the very short Python program that calculates the apparent diameter of Venus for 4 years beginning with January 2017:

OpenGL programming in Python: pyglpainter

This was a recent hobby programming project of mine for use in a CNC application, using Python and OpenGL. The source code is available at https://github.com/michaelfranzl/pyglpainter .Simple OpenGL output using pyglpainter library

This Python module provides the class PainterWidget, extending PyQt5’s QGLWidget class with boilerplate code neccessary for applications which want to build a classical orthagnoal 3D world in which the user can interactively navigate with the mouse via the classical (and expected) Pan-Zoom-Rotate paradigm implemented via a virtual trackball (using quaternions for rotations).

This class is especially useful for technical visualizations in 3D space. It provides a simple Python API to draw raw OpenGL primitives (LINES, LINE_STRIP, TRIANGLES, etc.) as well as a number of useful composite primitives rendered by this class itself ( Grid, Star, CoordSystem, Text, etc., see files in classes/items). As a bonus, all objects/items can either be drawn as real 3D world entities which optionally support “billboard” mode (fully camera-aligned or arbitrary- axis aligned), or as a 2D overlay.

It uses the “modern”, shader-based, OpenGL API rather than the deprecated “fixed pipeline” and was developed for Python version 3 and Qt version 5.

Model, View and Projection matrices are calculated on the CPU, and then utilized in the GPU.

Qt has been chosen not only because it provides the GL environment but also vector, matrix and quaternion math. A port of this Python code into native Qt C++ is therefore trivial.

Look at example.py, part of this project, to see how this class can be used. If you need more functionality, consider subclassing.

Most of the time, calls to item_create() are enough to build a 3D world with interesting objects in it (the name for these objects here is “items”). Items can be rendered with different shaders.

This project was originally created for a CNC application, but then extracted from this application and made multi-purpose. The author believes it contains the simplest and shortest code to quickly utilize the basic and raw powers of OpenGL. To keep code simple and short, the project was optimized for technical, line- and triangle based primitives, not the realism that game engines strive for. The simple shaders included in this project will draw aliased lines and the output therefore will look more like computer graphics of the 80’s. But “modern” OpenGL offloads many things into shaders anyway.

This class can either be used for teaching purposes, experimentation, or as a visualization backend for production-class applications.

Mouse Navigation

Left Button drag left/right/up/down: Rotate camera left/right/up/down

Middle Button drag left/right/up/down: Move camera left/right/up/down

Wheel rotate up/down: Move camera ahead/back

Right Button drag up/down: Move camera ahead/back (same as wheel)

The FOV (Field of View) is held constant. “Zooming” is rather moving the camera forward alongs its look axis, which is more natural than changing the FOV of the camera. Even cameras in movies and TV series nowadays very, very rarely zoom.