Made some progress with 360° video, but small snag where the Vision Pro can't draw the video to canvas fast enough in WebKit (required as video can't be directly snapshotted to my knowledge… right?). Any ideas for optimization?
@christianselig Guessing you’re trying to snapshot the video frame to render on the native side?
I wonder if WebCodecs’ VideoDecoder might work instead. Directly decode the video/frame instead of trying to blit the video to canvas at a certain frame rate. I assume you’d still need to send that frame over to the native side (which I guess is what essentially happens when you take a snapshot, which I guess you’re trying to do at the same interval?).
@tienshiao Capturing it native side once visible is really fast, but I’ve been unable to find any way to encode it into something portable to “send” over, do you think VideoDecoder could do that more efficiently than canvas’ toDataURL? That ones not too fast
@christianselig You seem to be using a ‘2d’ canvas context rather than a ‘3d’. This requires blitting the whole frame to another buffer, which is going to be very slow for a 4K video frame. Try using a WebGL backed canvas (canvas.getContext("webgl")) as that should be able to texture directly from the video frame without hitting main memory.
@christianselig You can also use video.requestVideoFrameCallback() rather than a 40hz interval to be notified when there's a new frame to be displayed. That should mean a 24fps video would only need to be painted at 24hz (about a 40% savings).
@christianselig@jernoble my mistake 😔. Still, setVideoFrameCallback() allows the paint cycle to exactly match the video fps, without having to introspect the container.
@christianselig why are you recreating the canvas and refinding the video inside the setInterval callback? Those both should be able to just do once at init. Not sure if that’s inflicting much overhead though
@christianselig nevermind, somehow my brain just skipped over the if (!canvas) 🤦♂️ but still the context and the video element can just be initialized outside of setInterval
@christianselig Understood but I’m curious if the performance would be better with the animation timer and only capturing when a given FPS interval has passed?
@christianselig@mark At work we're doing 4k video playback through WebGL with good performance, but it needs a lot more infrastructure than what you have in your screenshot.
@erik@mark I know next to no WebGL, what would the most basic flow look like there? I still need the video element to exist in the HTML DOM, but could you draw/capture frames from the video element using WebGL faster?
Add comment