Running a graphical window program via SSH on a remote machine (with GPU hardware acceleration)

Note 1: Even though it’s mid-2018, this post is still about the X Window System. Things still are in the transition phase towards Wayland, and things might get better or different over time.

Note 2: This post is not about displaying a graphical window of a program running on a remote machine on the local machine (like VNC or X forwarding). It is about running a remote program and displaying its graphical window on the remote machine itself, as if it had been directly started by a user sitting in front of the remote display. One obvious use case for the solution to this problem would be a remote graphics rendering farm, where programs must make use of the GPU hardware acceleration of the machine they’re running on.

Note that graphical programs started via Xvfb or via X login sessions on fake/software displays (started by some VNC servers) will not use GPU hardware acceleration. The project VirtualGL might be a viable solution too, but I haven’t looked into that yet.

Some experiments on localhost

I’m going to explore the behavior of localhost relative to our problem first. You’ll  need to be logged in to an X graphical environment with monitor attached.

The trivial case: No SSH login session

Running a local program with a graphical window from a local terminal on a local machine is trivial when you are logged into the graphical environment: For example, in a terminal, simply type glxgears and it will run and display with GPU hardware acceleration.

With SSH login session to the same user

Things become a bit more interesting when you use SSH to connect to your current user on localhost. Let’s say your local username is “me”. Try

It will output:

This can be fixed by setting the DISPLAY variable to the same value that is set for the non-SSH session:

Glxgears will run at this point.

With SSH login session to another user

Things become even more interesting when you SSH into some other local user on localhost, called “other” below.

You will get the message:

Trying to export DISPLAY as before won’t help us now:

You will receive the message:

This is now a permission problem. There are two solutions for it:

Solution 1: Relax permissions vIA XHOST PROGRAM

To allow non-networked connections to the X server, you can run (as user “me” which is currently using the X environment):

Then DISPLAY=:0 glxgears will start working as user “other”.

For security reasons, you should undo what you just did:

Settings via xhost are not permanent across reboots.

Solution 2: via Xauthority file

If you don’t want or can’t use the xhost program, there is a second way (which I like better because it only involves files and file permissions):

User “me” has an environment variable  env | grep XAUTHORITY

(I’m using the gdm display manager. The path could be different in your case.)

This file contains a secret which is readable only for user “me”, for security reasons. As a quick test, make this file available world-readable in /tmp:

Then, as user “other”:

Glxgears will run again.

To make sure that we are using hardware acceleration, run glxinfo:

This prints for me:

Make sure you remove  /tmp/xauthority_me after this test.

Note that the Xauthority file is different after each reboot. But it should be trivial to make it available to other users in a secure way if done properly.

Application on remote machine

If you were able to make things work on the local machine, the same steps should work on a remote machine, too. To clarify, the remote machine needs:

  • A real X login session active (you will likely need to set up auto-login in your display manager if the machine is not accessible).
  • A real monitor attached. Modern graphics cards and/or BIOSes simply shut down the GPU to save power when there is no real device attached to the HDMI port. This is is not Linux or driver specific. Instead of real monitors, you probably want to use “HDMI emulator” hardware plugs – they are cheap-ish and small. Otherwise, the graphical window might not even get painted into the graphics memory. The usual symptom is a black screen when using VNC.

Summary

If you SSH-login into the remote machine, as the user that is currently logged in to the X graphical environment, you can just set the DISPLAY environment variable when running a program, and the program should show on the screen.

If you SSH-login into the remote machine, as a user that is not currently logged in to the X graphical environment, but some other user is, you can set both DISPLAY and XAUTHORITY environment variables as explained further above, and the program should show up on the screen.

Related Links

https://serverfault.com/questions/186805/remote-offscreen-rendering

https://stackoverflow.com/questions/6281998/can-i-run-glu-opengl-on-a-headless-server#8961649

https://superuser.com/questions/305220/issue-with-vnc-when-there-is-no-monitor

https://askubuntu.com/questions/453109/add-fake-display-when-no-monitor-is-plugged-in

https://software.intel.com/en-us/forums/intel-business-client-software-development/topic/279956

OpenGL programming in Python: pyglpainter

This was a recent hobby programming project of mine for use in a CNC application, using Python and OpenGL. The source code is available at https://github.com/michaelfranzl/pyglpainter .Simple OpenGL output using pyglpainter library

This Python module provides the class PainterWidget, extending PyQt5’s QGLWidget class with boilerplate code neccessary for applications which want to build a classical orthagnoal 3D world in which the user can interactively navigate with the mouse via the classical (and expected) Pan-Zoom-Rotate paradigm implemented via a virtual trackball (using quaternions for rotations).

This class is especially useful for technical visualizations in 3D space. It provides a simple Python API to draw raw OpenGL primitives (LINES, LINE_STRIP, TRIANGLES, etc.) as well as a number of useful composite primitives rendered by this class itself ( Grid, Star, CoordSystem, Text, etc., see files in classes/items). As a bonus, all objects/items can either be drawn as real 3D world entities which optionally support “billboard” mode (fully camera-aligned or arbitrary- axis aligned), or as a 2D overlay.

It uses the “modern”, shader-based, OpenGL API rather than the deprecated “fixed pipeline” and was developed for Python version 3 and Qt version 5.

Model, View and Projection matrices are calculated on the CPU, and then utilized in the GPU.

Qt has been chosen not only because it provides the GL environment but also vector, matrix and quaternion math. A port of this Python code into native Qt C++ is therefore trivial.

Look at example.py, part of this project, to see how this class can be used. If you need more functionality, consider subclassing.

Most of the time, calls to item_create() are enough to build a 3D world with interesting objects in it (the name for these objects here is “items”). Items can be rendered with different shaders.

This project was originally created for a CNC application, but then extracted from this application and made multi-purpose. The author believes it contains the simplest and shortest code to quickly utilize the basic and raw powers of OpenGL. To keep code simple and short, the project was optimized for technical, line- and triangle based primitives, not the realism that game engines strive for. The simple shaders included in this project will draw aliased lines and the output therefore will look more like computer graphics of the 80’s. But “modern” OpenGL offloads many things into shaders anyway.

This class can either be used for teaching purposes, experimentation, or as a visualization backend for production-class applications.

Mouse Navigation

Left Button drag left/right/up/down: Rotate camera left/right/up/down

Middle Button drag left/right/up/down: Move camera left/right/up/down

Wheel rotate up/down: Move camera ahead/back

Right Button drag up/down: Move camera ahead/back (same as wheel)

The FOV (Field of View) is held constant. “Zooming” is rather moving the camera forward alongs its look axis, which is more natural than changing the FOV of the camera. Even cameras in movies and TV series nowadays very, very rarely zoom.