The open source debate in #ML ( #AI ) is absolutely irrelevant unless all the training data are also made open. Tech reporters are getting lost again because #ML vendors are misleading them. #LLM#MLsec
@cigitalgem Indeed, and by the way, there is a current effort from @osi to define what is #opensource#ai and some people are pointing out that the data must be released as well:
"As OpenAI trains its new model, its new Safety and Security committee will work to hone policies and processes for safeguarding the technology, the company said. The committee includes Mr. Altman, as well as OpenAI board members Bret Taylor, Adam D’Angelo and Nicole Seligman. The company said that the new policies could be in place in the late summer or fall."
When you choose to use an #ML#LLM foundation model, you accept the risk management decisions made by the vendor without your input. Wonder what they are? Read this #MLsec paper from #IEEE computer.
Earlier today, Microsoft released new WizardLM-2 7b, 8x22b, 70b with great benchmark result, (of course, they say as good or almost same as GPT-4), but they removed weights on Huggingface, repo on Github, and their whitepaper. Someone on Reddit joked maybe they released GPT-4 by mistake! lol Quantized. weights from other people are still around on Huggingface! #ML#LLM#AI
@chikim Also, I think we talked about this before, I cannot justify 20 USD per month for either Copilot pro or Chat GPT. They really need to try harder or just lower the price. Make it a Spotify, for example! :)
@vick21@chikim Chat GPT Plus isn't worth it, you can just load up on $6 of developer credits and use an altertnative interface to GPT-4. I'm a fan of the commandline LLLM (https://github.com/simonw/llm), but GUIs do exist. Copilot for VS Code is another matter entirely, I get it for free via the Github Student pack, to which I have access, but I'd probably pay up if I needed to.
I don't think the tech nerds out there understand how upsetting generative AI is to artists. Not because it will replace them, but because there will be a generation of soulless creation devoid of humanity.
Also, how many children are looking at the progress and thinking 'what's the point of becoming an artist?'. Or how many school directors are thinking 'what's the point of a fine art budget'.
@mempko on the second paragraph, i think you’re a little backwards on what draws children to art. i can say fairly authoritatively that 8yo’s aren’t yet thinking about the finer points of what it takes to become a full time artist 😊
i doubt anyone, even adults, were ever drawn to art because they thought it was easy money. i can’t imagine schools ever invested in art because they believed they were setting students up with high paying jobs
"The issue is that [Google] trained up the [Gemini] foundation model on the polluted ocean and now they're trying to stop the pollution from getting out with a filter, and that doesn't work," he said. "These models were built by drinking a data ocean without cleaning it first. And we have to do better than that." And Microsoft has the same problem, he added. #MLsec#ML#AI#LLM
Some counterargue that training on Nazi content allows it to recognize it so that it can be finetuned not to be a Nazi. But it seems to me that making its output match anti-Nazi speech is more effective than making it match Nazi speech.
Prompt "engineering" boils my blood. Can you imagine if you were working on a stream prediction system and the quality of the output depended on prepending a stream of magic numbers? You'd disdain anyone claiming that was a sustainable solution for a business. (I mean, I can imagine it, because that's exactly the kind of crap you see in consulting.) #ML
Allez, petit article qui va bien, tapé à l'arrache, mais qui peut vous intéresser. Comment j'ai utilisé une #IA, locale, pour générer de la data fictive.
Code fourni en bas de l'article. Et n'hésitez pas à réagir dans la section commentaire !
Last week I attended the 6th Perspectives on Scientific Error Conference at @TUEindhoven
I learned so much! About #metascience#preregistration#replicability#qrp questionable research practices, methods to detect data fabrication, #peerreview, #poweranalysis artefacts in #ML machine learning...
I'm impressed by the commitment of participants to improve science through error detection & prevention. Thanks to the organizers Noah van Dongen, @lakens@annescheel Felipe Romero and @annaveer
NVIDIA announced a New LLM: Nemotron-4 15B. Trained on 8T tokens. Training took 13 days with 3,072 H100s. Model is not available yet, but hhere's the paper. #ML#LLM#AIhttps://huggingface.co/papers/2402.16819
Any good sources on what the outputs of the attention blocks in a transformer represent? I expected that for "The bank of the plane took it around the savings bank on the bank of the river", the vectors corresponding to "bank" would diverge -- "rotation things/money things/rivery things" -- but AFAICT that doesn't clearly happen. Here are the dot prods of the normalized vectors (aka "cosine similarity") against themselves after embedding layer and attention block 5: #ML#Transformer
@kellogh Yeah, there’s a linear layer after the attention is applied. I more or less expected that to swizzle things up, which is why the continued correlation between “rotation bank” “money bank” “river bank” surprises. I thought they’d diverge (they don’t in a clear way) but if I swapped in “embankment of the river” that then some vector in the transformer-block output would converge with “river bank”. Haven’t done that code yet.
