Linux Framebuffers

A Linux framebuffer exposes a display as a file: /dev/fb0 holds the pixels and /sys/class/graphics/fb0 describes them, so no driver code is needed on the Elixir side. RasterExRatatui.Framebuffer.Surface is a complete surface for one, keyboard included; RasterExRatatui.Framebuffer and RasterExRatatui.Input.Devices are the pieces it is made of. This guide covers the device side of both.

Check the device first

Before writing a surface, confirm the framebuffer exists and read its geometry, from an IEx session on the device:

File.ls!("/dev") |> Enum.filter(&String.starts_with?(&1, "fb"))
{:ok, info} = RasterExRatatui.Framebuffer.info("fb0")
#=> {:ok, %{width: ..., height: ..., bits_per_pixel: ..., stride: ...}}

Then paint it red to prove the path end to end. At 32 bits per pixel:

File.write!("/dev/fb0", :binary.copy(<<0, 0, 255, 255>>, info.width * info.height))

and at 16 (<<0, 248>> is red in little-endian RGB565):

File.write!("/dev/fb0", :binary.copy(<<0, 248>>, info.width * info.height))

A write longer than the device fails with :enospc after painting what fits, which is what painting 32-bit pixels on a 16-bit framebuffer looks like.

The depth is whatever the kernel chose, so read bits_per_pixel instead of assuming it. The KMS fbdev emulation on a Raspberry Pi gives the DSI Touch Display 2 16 bits per pixel. RasterExRatatui.Framebuffer.format_for/1 maps 16 to RGB565 and 32 to XRGB8888, and returns {:error, :unsupported} for anything else.

The surface

defmodule MyDevice.Surface do
  use RasterExRatatui.Framebuffer.Surface, app: MyDevice.Dashboard
end

RasterExRatatui.Framebuffer.Surface is the whole thing: it waits for /dev/fb0 (the display drivers are kernel modules that load during boot; on the Pi 4 the DSI panel probed about ten seconds after the application started), reads the geometry, picks RGB565 or XRGB8888 from the depth and a font scale that keeps about a hundred columns on the long side, detaches the framebuffer console, reads the keyboard through RasterExRatatui.Input.Devices, and starts the app again when it quits. Its moduledoc has the option table: framebuffer:, scale:, rotate:, console:, keyboard:, and the rest.

Under the hood

The use gives the module init/1, push/2, handle_info/2, and terminate/2 delegating to RasterExRatatui.Framebuffer.Surface's functions of the same name, and every one is overridable: an override calls the default and adds to it. The state is a map with :fb, the open RasterExRatatui.Framebuffer, and :devices, the RasterExRatatui.Input.Devices.

@impl true
def init(opts) do
  {:ok, config, state} = RasterExRatatui.Framebuffer.Surface.init(opts)
  {:ok, config, Map.put(state, :backlight, MyDevice.Backlight.on!())}
end

Underneath, Framebuffer.open/2 reads virtual_size, bits_per_pixel, and stride from sysfs and opens the device; Framebuffer.write/2 places every patch row at y * stride + x * bytes_per_pixel, so line padding (a stride longer than the visible line) is handled, and a full frame is written the same way. A surface for a panel that is not a framebuffer uses those pieces from a RasterExRatatui.Surface of its own.

A panel on its side

A framebuffer has the panel's native orientation whatever the stand does: the Touch Display 2 is 720×1280 portrait on a landscape stand. The kernel can turn its own console (video=DSI-1:720x1280@60,rotate=90 on the command line, or fbcon's rotate_all), but that never changes what a write to /dev/fb0 means; the device stays portrait. Turning the app is the raster's job: rotate: 90 (or 270, whichever way the stand holds it) on the surface gives the app a 1280×720 image and writes the panel in its native order, so nothing about the framebuffer side changes. See "Rotation" in Building a Surface.

Keep the console off the display

The kernel's framebuffer console draws on the same device, so boot messages, a login prompt, or a blinking cursor can appear on top of the app.

• Unbind fbcon. RasterExRatatui.Framebuffer.unbind_console/2 writes 0 to /sys/class/vtconsole/<console>/bind, which stops the console drawing on the framebuffer. The framebuffer console is usually "vtcon1"; /sys/class/vtconsole/*/name tells them apart. It is best effort: on a kernel without that console it returns an error that can be ignored.
• Move the system console elsewhere. On Nerves, erlinit's ctty option (config :nerves, :erlinit, ctty: "ttyS0") puts IEx on a serial port instead of the display; it stays reachable over SSH.

The keyboard

RasterExRatatui.Input.Devices owns the keyboard: it finds the first /dev/input/eventN that reports letter keys (or reads the path it is given), starts an input_event reader on it with grab: true so keystrokes do not also reach the kernel console, translates what the reader delivers through RasterExRatatui.Input.Evdev into the same %ExRatatui.Event.Key{} structs a terminal would send (shift, ctrl, alt, super, and caps lock tracked; other layouts a layout: map away), and keeps looking, every two seconds, whenever there is no keyboard: at boot before one is plugged in, and again after one is unplugged, starting from a fresh translator so modifiers held on the old keyboard do not stick. The framebuffer surface embeds it; a surface of its own does the same in three lines: Devices.new/1 + Devices.start/1 in init/1, every message through Devices.handle_info/2, Devices.stop/1 in terminate/2 so no reader outlives the surface holding the grab.

input_event is a C port that only builds on Linux, so it is the consumer's dependency, not this library's: add {:input_event, "~> 1.4"} to the project. Without it Devices.start/1 returns {:error, :input_event_missing} and the framebuffer surface logs a warning once and runs without input, so the project still compiles and tests on a host. One thing to know about input_event: InputEvent.enumerate/0 starts and stops a short-lived reader per device from the calling process, so a surface, which traps exits, sees their :normal exits in handle_info/2; Devices.handle_info/2 answers :unknown to those and the surface ignores them.

Touch

touch: true on the framebuffer surface reads the first touch panel RasterExRatatui.Input.Devices finds (a device reporting the multitouch abs_mt_position_x axis, or abs_x with btn_touch), grabbed like the keyboard. RasterExRatatui.Input.Touch turns the first finger into the ExRatatui.Event.Mouse events a terminal would send: "down" and "up" on the cell under it, "drag" when it moves into another cell, nothing while it is in the margins; a second finger is ignored. The controller's axes are read from the device and scaled to the panel's pixels, and the cell comes from RasterExRatatui.Raster.cell_at/2, so a rotated panel maps right without anything else. A controller whose axes do not follow the panel's orientation gets swap_xy:, invert_x:, or invert_y:.

Performance

The surface pushes patches, so the steady-state cost follows what changed rather than the size of the panel. Pixel regions are the exception, since their cost grows with the panel pixels they cover (see Pixel Formats): an animated Viewport3D is cheaper in a moderate rect or rendered less often. Only regions that changed are rasterised, so a still Image beside the animation costs nothing after its first frame. On the Pi 4, a 696×480 Viewport3D region turning five times a second costs about 43 ms to rasterise and 13 ms to write at RGB565, and the surface's mailbox stays empty; RasterExRatatui.Telemetry.probe/3 prints those numbers for any surface.

Panels without a framebuffer

The helpers assume only /dev/fbN and sysfs. A panel the kernel does not expose as a framebuffer (an SPI controller driven from Elixir, an e-ink driver) implements push/2 with its own write instead; see Building a Surface.