OK, so you know the so-called "source code leak" from Rabbit for the #r1 last week? It wasn't a real leak, but what it showed was different than what I was expecting. So Rabbit Hole, the way you configure your R1 -- it only works on desktops, which I thought was rather odd. And then I went to manage my connection and I figured out why. This whole thing is a fucking VNC session in the web browser served as if it is a modal login
@film_girl The VM-through-web is so finicky that it won't even let you type numbers on a keyboard's number-pad. It's like they're only looking for the keycodes from the keys above the alphabetic keys.
@matdevdug thanks! I saw it and did a spit-take and then had to document it. Because it was one thing to know the apps themselves were running as playwright scripts in a vm. But I stupidly just assumed they’d be passing oAuth tokens or something and not actually using that AS the login method.
@film_girl This is hilarious, but I can't say I'm terribly surprised. There are so many ways that you could possibly "interact" with an app/website and have it not use an API
@lorewanderer truly. And I’m not even that surprised that they are using Playwright scripts for the Uber/Door Dash stuff (Spotify’s API is robust and free so that does surprise me). But I was still a little shocked they used a VM as the auth point. I’d just assumed they’d do auth in a safer way and pass tokens on to the headless browser. But in retrospect that was asking for too much.
So when you go to https://hole.rabbit.tech for your configuration, that i a proper web app. But look at what happens when you click on one of the connect buttons. A VNC session opens up. Notice the URL is /uber-vnc
@film_girl Just tried connecting Uber to mine, and the text rendering fuzziness is a dead giveaway that something is off. Glad you figured out what they're doing here.
The VNC doesn’t have clipboard access and can't access any of your extensions -- this is how I first discovered this (I wasn't looking at the URL at first). So logging into your account takes some effort. Look at this button here. This is what you need to use to pass a password into this VNC.
@d right -- because it is a VNC hiding as a modal/login. I don't know if they are doing web assembly or just serving pure TinyVNC but like, this is sketch as hell for sure.
The whole process is also horribly slow, even though it is trying to trick the user into thinking it is just a normal modal Uber/DoorDash/Spotify login window. I'll add a video in a second
@film_girl this must be part of their whole workaround for not having to use APIs and use their large action models. It’s basically something like selenium and they literally have a VM that your session is running on persistently somewhere. This is a wild design choice because I bet it is incredibly expensive infrastructure.
@Encryptic yeah they are using Playwright instead of selenium (yay TypeScript) and the stuff is hand-coded as far as I can tell. For Uber/DD, I can almost understand this choice. I cannot understand it for Spotify, who has an incredible API and a service for which the VM nature of this approach means there is delay when doing things like pause and resume. Until I see any real examples that aren’t part of their demos, I’m unconvinced the so-called LAM even exists right now.
@Encryptic but right now nothing is using their LAM as far as I can tell. The general queries come from perplexity, which is using GPT-4 under the hood and the app integrations are manually coded in Playwright. So the goal might be to be able to record an action and convert it to code using their LAM, but we’re not there yet.
@chrismasterson exactly. I knew they were doing some of this because of the leaked code. I didn’t quite realize that included how they were doing auth.
@zer0 I understand that. But Spotify does have an API, and a good one. Uber has an API. DoorDash too. It’s possible the Uber/DD APIs wouldn’t do what they want to accomplish (Uber’s prob would). They still shouldn’t do auth flow this way, without making it clear the user is logging into the service on a browser running in a VM on a cloud machine. Uber and Spotify both support oAuth. No reason to login this way.
@film_girl just casually type your password for other services into a (virtual) cloud computer Rabbit controls… yeah… not so sure about that being a good idea.
Add comment