Back in the 80’s I used to listen to a college station that added reverb to all the music–like it needed it back then. I’d like to add that effect to the music I play on Spotify( for nostalgia of course), is there a free/paid app that can do that? I looked but the only apps I found said it just edited MP3s not streaming...
For a long time, I thought of the blockchain as almost synonymous with cryptocurrencies, so as I saw stuff like “Odyssey” and “lbry” appearing and being “based on the blockchain”, my first thought was that it was another crypto scam. Then, I just got reminded of it and started looking more into it, and it just seemed...
What’s wrong with that though? BTC handles forks just fine. Eventually one fork will win out and life will continue on as usual.
The bigger issue this paper presents is that miners become incentivized to mine empty blocks. But can’t you just enforce a minimum transaction count on blocks?
There’s only incentive to do that if the mempool is empty. If the mempool is full, there will be plenty of transactions for both the first miner and the next miner.
Wait… This entire paper only makes sense if the mempool is near empty. If the mempool is full, then there is no reason to mine an empty/partial block because there will always be transactions left for future miners.
So basically:
Mempool full = miner would mine full blocks just like intended.
Mempool empty = miner would mine empty blocks but that isn’t a problem because there are no transactions to process in the mempool.
I don’t know what a .webp file is but I don’t like it. They’re like a filthy prank version of the image/gif you’re looking for. They make you jump through all these hoops to find the original versions of the files that you can actually do anything with....
It’s still bad compared to modern lossless algorithms. PNG is very old and even though PNG encoders have evolved, it is still fundamentally a decade behind modern lossless compression algorithms.
For example: JPEG XL in lossless mode compresses at least 30% better than PNG.
Also, PNG is not actually lossless in many cases. PNG only supports RGB colorspaces. If you try to store anything that’s not in an RGB colorspace (e.g. a frame of a video) in a PNG, you will lose some color information as colorspace conversion is not lossless. Example of someone running into this issue: stackoverflow.com/q/35399677
JPEG XL supports non-RGB colorspaces so you don’t have this problem.
There’s no real reason why you shouldn’t use PNG for a frame of video. I’m not talking about using it as a video format, I’m talking about extracting a frame from a video and sending it off to an editor for inclusion in another video or image.
As a user, I would expect that I could use the most popular lossless image format if I want to losslessly share a frame from a movie with someone.
Of course I do agree that we need adoption of other image formats. We really should not still be cramming everything in PNGs or JPGs in 2023.
I think we generally agree but I just want to clarify anyways. I’m not saying we should use PNG to store frames from videos.
What I am saying however, is that we should replace PNG with a modern lossless image format that is more flexible so users don’t have to deal with these issues. All this colorspace stuff should be automatically handled and I shouldn’t have to worry about it not being lossless. If I want to save a frame of video, I should be able to do it using an image format that everybody recognizes and accepts, it should not be a huge hassle and it should be fully lossless.
The only app I can’t live without. Except for gboard, all of my applications are Foss. There is no competition for gboard’s swipe typing, not to mention its many capabilities like as searching for gifs, stickers, being able to paste copied images, translating, and so on. I’d like to know how I can use gboard while...
Just got a new phone (OnePlus Nord 3), turned refresh rate HUD in developer settings and I see some parts of the system and some apps display 120 Hz but I have problem noticing any difference, same with my wife’s Redmi Note 12, i have to look very carefully and maaaybe I notice some different, not sure
No, you want it for scrolling. Scrolling feels much more responsive at 120Hz. It does drain battery more but not by enough to be a deal breaker for most people.
it just predicts the next word out of likely candidates based on the previous words
An entity that can consistently predict the next word of any conversation, book, news article with extremely high accuracy is quite literally a god because it can effectively predict the future. So it is not surprising to me that GPT’s performance is not consistent.
It won’t even know it’s written itself into a corner
It many cases it does. For example, if GPT gives you a wrong answer, you can often just send an empty message (single space) and GPT will say something like: “Looks like my previous answer was incorrect, let me try again: blah blah blah”.
And until we get a new approach to LLM’s, we can only improve it by adding more training data and more layers allowing it to pick out more subtle patterns in larger amounts of data.
This says nothing. You are effectively saying: “Until we can find a new approach, we can only expand on the existing approach” which is obvious.
But new approaches come all the time! Advances in tokenization come all the time. Every week there is a new paper with a new model architecture. We are not stuck in some sort of hole.
The issue here is that you are describing the goal of LLMs, not how they actually work. The goal of an LLM is to pick the next most likely token. However, it cannot achieve this via rudimentary statistics alone because the model simply does not have enough parameters to memorize which token is more likely to go next in all cases. So yes, the model “builds up statistics of which tokens it sees in which contexts” but it does so by building it’s own internal data structures and organization systems which are complete black boxes.
Also, going “one token at a time” is only a “limitation” because LLMs are not accurate enough. If LLMs were more accurate, then generating “one token at a time” would not be an issue because the LLM would never need to backtrack.
And this limitation only exists because there isn’t much research into LLMs backtracking yet! For example, you could give LLMs a “backspace” token: news.ycombinator.com/item?id=36425375
Have you tried that when it’s correct too? And in that case you mention it has a clean break and then start anew with token generation, allowing it to go a different path. You can see it more clearly experimenting with local LLM’s that have fewer layers to maintain the illusion.
If it’s correct, then it gives a variety of responses. The space token effectively just makes it reflect on the conversation.
We’re trying to make a flying machine by improving pogo sticks. No matter how well you design the pogo stick and the spring, it will not be a flying machine.
To be clear, I do not believe LLMs are the future. But I do believe that they show us that AI research is on the right track.
Building a pogo stick is essential to building a flying machine. By building a pogo stick, you learn so much about physics. Over time, you replace the spring with some gunpowder to get a mortar. You shape the gunpowder into a tube to get a model rocket and discover the pendulum rocket fallacy. And finally, instead of gunpowder, you use liquid fuel and you get a rocket that can go into space.
First of all, this link is just to C# bindings of llama.cpp and so doesn’t contain the actual implementation. But it also doesn’t refute my criticism of your claim. More specifically, I take issue with this statement that you said: “today’s LLM’s ingest a ton of text [snip] and builds up statistics of which tokens it sees in that context”.
I claim that this is not how today’s LLMs work because we have no idea what LLMs do with the input data during training. We have very little insight into what kind of data structure it builds and how the data structure it built is organized.
No, it’s done because one letter at a time is too slow. Tokens are a “happy” medium tradeoff.
I think I worded my sentence ambiguously, let me re-word it for you: “Going one token at a time is only considered a limitation because LLMs are not accurate enough”
It makes a “break” of the block, which lets it start a new answer instead of continuing on the previous. How it reacts to that depends on the fine tune and filters before the data hits the LLM.
Once again, my sentence was not written well, my bad. I was commenting on the observed behavior, not on how it works from a technical perspective.
I have just said that LLM’s we have today can’t fix the problems with false data and hallucinations, because it’s a core principle of how it operates. It will require a new approach.
You could add a rocket engine and wings to a pogo stick, but then it’s no longer a pogo stick but an airplane with a weird landing gear. Today’s LLM’s could give us hints to how to make a better AI, but that would be a different thing than today’s LLM’s. From what has been leaked from OpenAI GPT4 has scaling issues so they use mixture of experts. Just throwing hardware at it is already showing diminishing returns. And we’re learning fascinating new ways of training them, but the inherent problem is the same.
Alright, we agree here for the most part so I’m just going to skip this.
For example, if you ask an LLM if it can give an answer to a question, it will have two paths to go down, positive and negative. Note, at the point where it chooses that it doesn’t know how to finish it, it doesn’t look ahead.
This is weird though. How do you know LLMs can’t look ahead? When we prompt LLMs, we are basically asking them this question: “What is the next word of your response?” How do you know it hasn’t written out the entire response in memory already after which it only shows you the first word? LLMs are neural networks. Neural networks have working memory. That’s how neural networks work after all, it’s just a vector of data that is repeatedly transformed as it passes through each layer. Of course, if it does write the entire response in memory, it is all thrown away after every word.
As far as the backspace tokens go, you are right to be skeptical but also do not be surprised if it works out. We’ve had LLMs trained to complete and edit text for some time already. They’ve fallen out of use today but they did perform acceptably well.
Wow, very nice! First of all, I will preface by admitting that I have not worked with LLMs to the degree of making a toy implementation. Your explanation of the sampling techniques is insightful but doesn’t clear up my confusion. Why does sampling imply the absence of higher level structure in the model?
For example, even though poker is highly influenced by chance, I can still have a plan that will increase my likelihood of winning. I don’t know what card will be drawn next but I can prepare strategies for each possible card. I can have preferences for which cards I want to be drawn next.
If you have a workspace with dependencies you probably noticed that sometimes cargo seemingly unnecessary recompile external dependencies as you switch between different members of your workspace....
So, I’m kinda new to this Lemmy thingy and the fediverse. I like the fediverse from a technological standpoint. However, I think that, if we gain more and more traction, Lemmy (and by extend the entire fediverse) is a GDPR clusterfuck waiting to happen. With big and expensive repercussions…...
GDPR does not distinguish between public or private data.
GDPR handles public data through propagation. If you download public data that is GDPR covered, the data you downloaded also becomes GDPR covered. You are required to follow all GDPR regulations while handling the downloaded data.
Remember, GDPR covers almost all "collected personal data". It does not matter if the data was originally public, and how/where the data was collected. It's all covered.
Sound App for Bluetoothe Streaming
Back in the 80’s I used to listen to a college station that added reverb to all the music–like it needed it back then. I’d like to add that effect to the music I play on Spotify( for nostalgia of course), is there a free/paid app that can do that? I looked but the only apps I found said it just edited MP3s not streaming...
Is the blockchain an interesting innovation, aside from cryptocurrencies ?
For a long time, I thought of the blockchain as almost synonymous with cryptocurrencies, so as I saw stuff like “Odyssey” and “lbry” appearing and being “based on the blockchain”, my first thought was that it was another crypto scam. Then, I just got reminded of it and started looking more into it, and it just seemed...
what are .webp files and why has my online experience been plagued by them?
I don’t know what a .webp file is but I don’t like it. They’re like a filthy prank version of the image/gif you’re looking for. They make you jump through all these hoops to find the original versions of the files that you can actually do anything with....
Privacy with Google's Gboard (lemm.ee)
The only app I can’t live without. Except for gboard, all of my applications are Foss. There is no competition for gboard’s swipe typing, not to mention its many capabilities like as searching for gifs, stickers, being able to paste copied images, translating, and so on. I’d like to know how I can use gboard while...
not seeing any non negligible difference between 60 and 120 Hz, am I weird?
Just got a new phone (OnePlus Nord 3), turned refresh rate HUD in developer settings and I see some parts of the system and some apps display 120 Hz but I have problem noticing any difference, same with my wife’s Redmi Note 12, i have to look very carefully and maaaybe I notice some different, not sure
Why AI detectors think the US Constitution was written by AI (arstechnica.com)
Fixing feature unification compilation time issues with `cargo-hackerman` (crates.io)
If you have a workspace with dependencies you probably noticed that sometimes cargo seemingly unnecessary recompile external dependencies as you switch between different members of your workspace....
Lemmy.world updated to 0.18.1-rc
Looks like it works....
Jerboa app and Lemmy 0.18
The 0.18 version of Lemmy was announced. This will solve many issues....
Lemmy (and the fediverse) and GDPR: a clusterfuck waiting to happen?
So, I’m kinda new to this Lemmy thingy and the fediverse. I like the fediverse from a technological standpoint. However, I think that, if we gain more and more traction, Lemmy (and by extend the entire fediverse) is a GDPR clusterfuck waiting to happen. With big and expensive repercussions…...
[Guess not...] Installed lemmy-ui 0.18.0 RC-1
I just installed the 0.18.0 release candidate 1 of the lemmy-ui component....
Judge orders sheriff to evict Twitter from Boulder office (www.bizjournals.com)
Edge automatically sends images you view online to Microsoft by default (www.neowin.net)