[Issue] Cannot Access ElevenLabs API in `ai-character-chat` (Fixed)

perchance, 15 days ago

Hmm, I wasn’t able to replicate this problem. I added this as character’s custom code, and it does log the response in the console:

<span style="color:#323232;">oc.thread.on(</span><span style="color:#183691;">"MessageAdded"</span><span style="color:#323232;">, </span><span style="font-weight:bold;color:#a71d5d;">async function</span><span style="color:#323232;">() {
</span><span style="color:#323232;">  </span><span style="font-weight:bold;color:#a71d5d;">let </span><span style="color:#323232;">result </span><span style="font-weight:bold;color:#a71d5d;">= </span><span style="color:#323232;">await fetch(</span><span style="color:#183691;">"https://httpbin.org/post"</span><span style="color:#323232;">, {
</span><span style="color:#323232;">    method: </span><span style="color:#183691;">"POST"</span><span style="color:#323232;">,
</span><span style="color:#323232;">    headers: { </span><span style="color:#183691;">"Content-Type"</span><span style="color:#323232;">:</span><span style="color:#183691;">"application/json" </span><span style="color:#323232;">},
</span><span style="color:#323232;">    body: JSON.stringify({foo:</span><span style="color:#0086b3;">3</span><span style="color:#323232;">})
</span><span style="color:#323232;">  }).then(r </span><span style="font-weight:bold;color:#a71d5d;">=> </span><span style="color:#323232;">r.json());
</span><span style="color:#323232;">  </span><span style="color:#795da3;">console</span><span style="color:#323232;">.</span><span style="color:#0086b3;">log</span><span style="color:#323232;">(result);
</span><span style="color:#323232;">});
</span>

Is it perhaps header values you’re missing, or something? Certainly possible that there is a big though and I’m just not hitting it with the above example for whatever reason.

Either way this motivated me to finally get around to creating a CORs-bypassing plugin: perchance.org/fetch-plugin That way I can ensure strong backwards-compatibility and performance instead of just relying on a little glitch.com server, which won’t scale, and doesn’t have good perf/uptime guarantees. I’ll wait to hear back from you before integrating it into ai-character-chat just so we can don’t end up making it harder for you to reproduce the bug you were up against here.

Alllo, 15 days ago (edited 15 days ago)

bottom of page says “Check out more plugins at perchance.org/plugins” twice

edit: and now it doesn’t. must be magic

perchance, 15 days ago (edited 15 days ago)

@Alllo @VioneT Sorry just realized the plugin is not quite ready for prime time! Need to make some breaking changes so I’ve commented out the code for now. The moment I started integrating it into ai-character-chat I realised it’s a bad idea to overwrite the existing fetch - should instead just be a separate “superFetch” or whatever because otherwise if you need to e.g. download a big file that you know doesn’t have any CORs problems, you’re forced to go through the proxy, which will be slower. Will update soon hopefully, if not tomorrow. Sorry for trouble if you’d started playing with it!

perchance, 15 days ago (edited 15 days ago)

@Alllo @VioneT Okay, superFetch it is: perchance.org/super-fetch-plugin … phew - happy I managed to pull the plug on that previous approach quickly, but sorry again for the breaking change! I’d only ever do when a plugin is extremely new - i.e. a few hours old, as in this case. I take breaking changes with the official plugins really seriously, so it was still somewhat painful to do.

VioneT, 15 days ago (edited 15 days ago)

Previously it worked fine, but now it doesn’t. Here’s the code for the ElevenLabs API that I wrote. Here is a HAR file of the requests.

perchance, 15 days ago (edited 15 days ago)

Okay, thanks for the example, I think it’s all fixed now. Hopefully I didn’t break anything. Been up for two days straight tho so I wouldn’t bet on it, but I did some basic tests and it seems good. Will check lemmy messages first thing tomorrow 🫡

VioneT, 15 days ago (edited 15 days ago)

It seems to work now, though there are some inconsistencies with the chunk text arrangement, which causes the text in the stream to be quite jumbled. I’m looking into it now, I’ll update if it is still inconsistent with the order.

EDIT: It is inconsistent with the order of the chunks. Maybe there is a way to parse it in order? Currently I’m having it pushed into an array, then sort that array by the index, then join the sorted array to be a string before pushing it to the t2s, though it is still inconsistent and sometimes the streaming finishes and the text to be spoken is not yet queued up.

Here are the code hacks to re-sort the chunks to order lmao.

perchance, 13 days ago

I wasn’t able to reproduce this when trying it with this code:

<span style="color:#323232;">oc.thread.on(</span><span style="color:#183691;">"StreamingMessage"</span><span style="color:#323232;">, </span><span style="font-weight:bold;color:#a71d5d;">async function </span><span style="color:#323232;">(data) {
</span><span style="color:#323232;">  </span><span style="font-weight:bold;color:#a71d5d;">let </span><span style="color:#323232;">lastChunkI </span><span style="font-weight:bold;color:#a71d5d;">= -</span><span style="color:#0086b3;">1</span><span style="color:#323232;">;
</span><span style="color:#323232;">  </span><span style="font-weight:bold;color:#a71d5d;">let </span><span style="color:#323232;">chunksText </span><span style="font-weight:bold;color:#a71d5d;">= </span><span style="color:#183691;">""</span><span style="color:#323232;">;
</span><span style="color:#323232;">  </span><span style="font-weight:bold;color:#a71d5d;">let </span><span style="color:#323232;">chunks </span><span style="font-weight:bold;color:#a71d5d;">= </span><span style="color:#323232;">[];
</span><span style="color:#323232;">  </span><span style="font-weight:bold;color:#a71d5d;">for await </span><span style="color:#323232;">(let chunk </span><span style="font-weight:bold;color:#a71d5d;">of </span><span style="color:#323232;">data.chunks) {
</span><span style="color:#323232;">    chunks.</span><span style="color:#0086b3;">push</span><span style="color:#323232;">(chunk);
</span><span style="color:#323232;">    chunksText </span><span style="font-weight:bold;color:#a71d5d;">+= </span><span style="color:#323232;">chunk.text;
</span><span style="color:#323232;">    </span><span style="font-weight:bold;color:#a71d5d;">if</span><span style="color:#323232;">(chunk.index </span><span style="font-weight:bold;color:#a71d5d;">!== </span><span style="color:#323232;">lastChunkI</span><span style="font-weight:bold;color:#a71d5d;">+</span><span style="color:#0086b3;">1</span><span style="color:#323232;">) </span><span style="color:#795da3;">console</span><span style="color:#323232;">.</span><span style="color:#0086b3;">warn</span><span style="color:#323232;">(</span><span style="color:#183691;">"OUT OF ORDER CHUNKS!"</span><span style="color:#323232;">, chunks);
</span><span style="color:#323232;">    lastChunkI</span><span style="font-weight:bold;color:#a71d5d;">++</span><span style="color:#323232;">;
</span><span style="color:#323232;">  }
</span><span style="color:#323232;">  </span><span style="color:#795da3;">console</span><span style="color:#323232;">.</span><span style="color:#0086b3;">log</span><span style="color:#323232;">(</span><span style="color:#183691;">"chunks:"</span><span style="color:#323232;">, chunks);
</span><span style="color:#323232;">  </span><span style="color:#795da3;">console</span><span style="color:#323232;">.</span><span style="color:#0086b3;">log</span><span style="color:#323232;">(</span><span style="color:#183691;">"chunksText:"</span><span style="color:#323232;">, chunksText);
</span><span style="color:#323232;">});
</span>

Or have I misunderstood the problem?

VioneT, 13 days ago

You have. On trying your code. it gave me the OUT OF ORDER CHUNKS!:

https://lemmy.world/pictrs/image/9f4522db-67cd-4717-868b-a73c3e1a8785.png
https://lemmy.world/pictrs/image/e352ef84-23d7-4a2d-a6d2-86e0ce887aab.png
Here’s the character I used with the custom code to check the out of order chunks.: Link to Character

perchance, 13 days ago

Ahh, thank you! Was very confused at first because I iteratively made your character closer to the default Assistant to work out why it was happening in yours but not the assistant, and found that the profile pic was the cause lmao, but eventually realised that it was because data url vs normal URL, and the data url (being larger) was making an async IndexedDB request take a few milliseconds longer, which caused the out-of-order-ness. But I shouldn’t have even been doing those db requests in the first place, so I’ve removed them, and this race condition type bug shouldn’t be possible at all now. Thanks again!!

VioneT, 13 days ago

Niiceeeeeeeeeee it is good now. Thanks again!

Alllo, 15 days ago

haha. one and a half days for me and maybe 30+ hours straight just now on something with your wonderful comments update :) can’t wait to share it! No sleeping yet!

BluePower, 15 days ago (edited 15 days ago)

Same 😄

I’ll complete and share the project possibly (and hopefully) as an actual plugin after the post-announcement update of my generator hub page, but I’ll be releasing the “early implementations” somewhere in my experiment generator so everyone can try it right now and give some feedback on it.

VioneT, 16 days ago

@perchance - pinging dev