@rmader@floss.social
@rmader@floss.social avatar

rmader

@rmader@floss.social

Mostly #wayland stuff around #gnomeshell, #gstreamer, #Firefox, #Pipewire, #libcamera, #linuxmobile - https://floss.social/@collabora

Generally thing that make the Linux desktop use graphics hardware features. And sometimes random boosts.

Based in Berlin.

Member of the German green party.

This profile is from a federated server and may be incomplete. Browse more on the original instance.

rmader, to random
@rmader@floss.social avatar

After some more debugging, here are two videos proving that / video playback with fullscreen GL/VK overlay works on a and / on / , each maxing out their video decoders with 1080p@60 and 4k@60 respectively.

The message here is: no matter what hardware - if you want to make a video/camera app that needs to be highly efficient, you can do it with a proper, yet still lightweight toolkit now. No need for a custom kms/drm backend.

The GTK4 demo video player on a Raspberry 4, playing BBB. The camera moves over to another screen with Weston debug output, showing that the compositor uses hardware planes, which is needed for optimal playback performance.
The GTK4 demo video player on a Pinebook Pro (using a RK3399 SoC), playing BBB. The camera moves over to another screen with Weston debug output, showing that the compositor uses hardware planes, which is needed for optimal playback performance.

rmader,
@rmader@floss.social avatar

The obvious use-case would be video player, multimedia centers and video-chat software.

But there's more: let's say you want to build a FLOSS drone operator software that runs on a dedicated tablet - but can also be used on any laptop, just on the desktop? Some camera software that displays extra info like the temperature over the video? You'll probably want to use / anyway - can now be a pretty decent frontend, on some embedded device just like on the desktop :)

nekohayo, to linux
@nekohayo@mastodon.social avatar

At 4 AM in the night, one of my GPUs decided to freeze up (just like @rmader's in https://gitlab.freedesktop.org/mesa/mesa/-/issues/9939), while the computer was idle.

Nice thing about : it didn't freeze the kernel.
Not so nice: it still crashed & all the apps/data 🫠️

https://floss.social/@rmader/111496200326456441 gives me some hope that someday we might have an architectural solution for https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/5634

rmader,
@rmader@floss.social avatar

@readf0x @nekohayo Yeah, I hope both of these things will find their way into Gnome - unfortunately AFAICS not for 46 :/

rmader, to GNOME
@rmader@floss.social avatar

Got the last bug fixed to make play video with overlay in full zero-copy / hardware plane only mode \o/

This allows highly efficient playback with e.g. subtitles and other overlays, only using the GPU when GTK needs to repaint parts of the main window / the overlay.

I'll talk more about this at my presentation at , see https://fosdem.org/2024/schedule/event/fosdem-2024-3557-the-state-of-video-offloading-on-the-linux-desktop/

The video shows a weston desktop. The gtk4 video player demo is launched, a video started and put into fullscreen. The camera moves to a second screen where weston debug output is printed, showing that weston indeed uses a cursor, overlay and primary plane (and no rendering / compositing) to display the video and overlay (and thecursor).

rmader, to GNOME
@rmader@floss.social avatar

I'll also be at and do a short talk about video offloading related stuff I've been posting about, featuring , , and

https://fosdem.org/2024/schedule/event/fosdem-2024-3557-the-state-of-video-offloading-on-the-linux-desktop/

See you there!

bragefuglseth, to firefox
@bragefuglseth@fosstodon.org avatar

Firefox with its stock look, proper rounded window corners and actual Wayland support feels like jumping 5 years ahead of how it's shipped by default currently.

Screenshot of Firefox with all the window corners rounded and its modern stock theme

rmader,
@rmader@floss.social avatar

@tbernard @bragefuglseth @mozilla Emillio, a mozilla person, picked it up and landed it but it got reverted - I think because some x11 issues. If the wayland rollout works well we can hopefully give it another go (maybe only on wayland then).

rmader,
@rmader@floss.social avatar

@tbernard @bragefuglseth @mozilla Ah right - not aware of anyone working on this yet, could start a bugzilla bug maybe?

baarkerlounger, to linuxphones
@baarkerlounger@mastodon.social avatar

New Jellyfin client Delfin just showed up on Flathub and it looks amazing. Very exciting to have a native client that runs nicely (albeit playback is a little stuttery still) on an L5!

https://flathub.org/apps/cafe.avery.Delfin

rmader,
@rmader@floss.social avatar

@baarkerlounger This looks awesome! FTR., this is exactly the kind of app that I hope will be able to benefit from the offload work in (see https://floss.social/@rmader/111496162059266119)

While the stuttering on the IIUC is likely due to missing upstream V4L2 stateless decoding in - which is the most important thing to get working - the offloading stuff will help to reduce buffer copies to a minimum before showing things on screen. Together hopefully allowing people to stream for hours.

rmader, to GNOME
@rmader@floss.social avatar

Regarding the future of video playback in I'd like to add some more context around current developments in , and in a short 🧵

TL;DR: by making use of more modern hardware features we're finally in the position to catch up to other platforms with regards to energy efficiency. So let's do it!

rmader,
@rmader@floss.social avatar

In a previous thread I wrote about YUV support in Mutter having made its way into #gnome45 (https://floss.social/@rmader/111144687912968833). Recently #gtk4 devs picked things up and implemented compositor offloading, see their nice blog post: https://floss.social/@GTK/111415523629484640

rmader,
@rmader@floss.social avatar

We're now working on filling the remaining gaps so this can become an actual reality on the (and generally / / ) desktop. And what can I say - things actually work out quite nicely! While on modern Intel or AMD systems the effect is mostly about lower resource consumption, on some low-end hardware there are visible differences on what you can play fluently.

rmader,
@rmader@floss.social avatar

Take for example these examples of a (rk3399 / v4l2 stateless) playing 4k content or the 4b (v4l2 stateful / m2m) playing 1080p@60fps (the limit of the hardware decoder) on a 2560x1440 screen. My videos probably don't fully capture the effect, but especially in the later case the difference is pretty visible). This is using the gtk video player demo with some extra patches and .

A Pinebook Pro with Gnome destkop, with the opaque region overlay on, showing the gtk4 demo video player playing a video. The opaque region is on, showing the difference between composited and pass-trough / direct scanout playback.
A screen connected to a Raspberry Pi with Gnome destkop, with the opaque region overlay on, showing the gtk4 demo video player playing a video. The opaque region is on, showing the difference between composited and pass-trough / direct scanout playback.

rmader,
@rmader@floss.social avatar

My hope is that we can ship some initial support for all of this in - if possible with the default video player. Be it Totem, ported to , or some alternative. The goal is of course that as many apps as possible can make use of it without much hassle - be it video streaming apps, video chat apps like @dino or Fractal, clients like Tuba or camera apps like / Snapshot (for a battery friendly viewfinder - important on ). And of course browsers.

rmader,
@rmader@floss.social avatar

Making this all work is a long process and when talking to people that have been working in the field longer than me, I often get the feedback to be not too optimistic regarding time frames. As always when pushing some limits there will be driver bugs, missing APIs and loose ends all over the place. So far every device I tried needed at least some fixes in , the kernel or both. And when things break, we have to be extra careful to not impact stability.

rmader,
@rmader@floss.social avatar

That's why we're very fortunate that recently got funding from the . One of the sponsored projects is to implement GL robustness in / Mutter, so if the driver stumbles when trying to import some unusual buffer you won't lose your session.

rmader,
@rmader@floss.social avatar

My personal vision with all of this is to see desktop technologies not only catching up with what other OSs offer, but becoming leading players - just like what other FLOSS projects already archived (or are in the process of becoming) in their areas. I'm thinking of , , , , the kernel of course, and many others.

rmader,
@rmader@floss.social avatar

The plan is as usual: expose exciting technology to a great community of technical capable people -> improve quality, reliability and features (HDR...) -> get more companies to use it for products and invest into its development -> repeat -> world domination :P

Yeah, quite optimistic. Let's see how it goes. End 🧵

rmader,
@rmader@floss.social avatar

P.S.: totally forgot - if you want to follow the development, try things out or even consider helping, here are some links:

I'm also looking for people to help with caps negotiation in various elements, especially for V4L2 decoders :)

1ace, to random
@1ace@mastodon.gamedev.place avatar

deleted_by_author

    •
    rmader,
    @rmader@floss.social avatar

    @1ace Uh, that sounds bad. Can you just open an issue at https://bugzilla.mozilla.org and cc me (there or ping me)? Without further info it's a bit hard to say who should be pinged best, but probably stransky@redhat.com or emilio@crisal.io.

    rmader, to GNOME
    @rmader@floss.social avatar

    For app folks: IMHO we need to think about what to do with Totem / Gnome Videos. It has not yet been ported to , which is increasingly becoming an issue.

    Apart from not fitting nicely UI wise, it prevents us from using the newly introduced hardware offloading (zero-copy playback) and, crucially, from (properly) supporting HDR content going forward.

    I.e. we either need a port - or should consider making an alternative a core app to focus on.

    Short 🧵

    rmader,
    @rmader@floss.social avatar

    @alatiera Thanks, that pretty good to hear - I wonder if such a port could also make use s GtkVideo / GtkGstSink which are used for the gtk demo player (gtk4-demo --run=video_player)

    That would mean an even bigger part of the technical base could just live in gtk, which possibly would make HDR upbringing easier.

    rmader,
    @rmader@floss.social avatar

    @cassidy @alatiera I wonder if the usefulness of such tools actually increased with (hopefully) becoming a thing (if the app as a whole was fully adaptive) - but in generally I agree that it'd be fine to drop stuff if it helps to make the core more maintainable.

    rmader,
    @rmader@floss.social avatar

    @cassidy @alatiera I think it's worth mentioning that for a long time there wasn't much interesting happening on this front. But now (and thus proper, desktop integrated zero-copy playback) is becoming mature and HDR is around the corner.

    rmader,
    @rmader@floss.social avatar

    @alatiera @cassidy We folks are hard on it I'd say - and given how bad the state is on other OSs it's an area where we have a chance to shine - especially if we manage to get lots of the infrastructure work paid by television vendors, google, amazon or car makers.

    rmader,
    @rmader@floss.social avatar

    P.S.: I was just made aware of glide (https://github.com/philn/glide) which has some compelling features:

    • Written in Rust which is quite popular in the app community I hear.
    • Seems to already support most stuff we need, could just use some UI refresh.
    • Explicitly supported on MacOS. That's interesting because the hardware offloading in gtk4 so far only supports Wayland, but was written in a way to work on other platforms as well.
    rmader,
    @rmader@floss.social avatar

    @agx Interesting - I wonder if we could make a player that works for you as well so you maybe just have to maintain a small fork?

    In any case, I guess the gtk4 hardware plane offloading would be interesting for the librem5 as well - and could increase battery life quite a bit, assuming the video hardware decoder produces something compatible with display hardware (which should be the case).

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • JUstTest
  • tacticalgear
  • DreamBathrooms
  • osvaldo12
  • GTA5RPClips
  • ngwrru68w68
  • magazineikmin
  • everett
  • Youngstown
  • slotface
  • rosin
  • mdbf
  • kavyap
  • modclub
  • provamag3
  • InstantRegret
  • Durango
  • cubers
  • khanakhh
  • ethstaker
  • thenastyranch
  • Leos
  • normalnudes
  • tester
  • cisconetworking
  • anitta
  • megavids
  • lostlight
  • All magazines
    •