mamund,
@mamund@mastodon.social avatar

ChatGPT's odds of getting code questions correct are worse than a coin flip

https://www.theregister.com/2023/08/07/chatgpt_stack_overflow_ai/

"'Our analysis shows that 52 percent of ChatGPT answers are incorrect and 77 percent are verbose,' the team's paper concluded. 'Nonetheless, ChatGPT answers are still preferred 39.34 percent of the time due to their comprehensiveness and well-articulated language style.' Among the set of preferred ChatGPT answers, 77 percent were wrong." --

rabble,
@rabble@mastodon.social avatar

@mamund do any of the other more code focused LLM’s work better?

mamund,
@mamund@mastodon.social avatar

@rabble i don't know of any right now but there are some attempts to create coding AI (not so much conversational as computational). i can't point to one ATM but have seen a few show up in my inbox.

the good news is that higher-level interfaces (e.g. APIs and links/forms in HTML) are much more "converstational" than things like protobuf or graphQL and might yield some interesting solutions.

i don't yet see enough focus on this interface AI, tho.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • ChatGPT
  • rosin
  • thenastyranch
  • ethstaker
  • osvaldo12
  • mdbf
  • DreamBathrooms
  • InstantRegret
  • magazineikmin
  • Youngstown
  • ngwrru68w68
  • slotface
  • GTA5RPClips
  • kavyap
  • cubers
  • JUstTest
  • everett
  • cisconetworking
  • tacticalgear
  • anitta
  • khanakhh
  • normalnudes
  • Durango
  • modclub
  • tester
  • provamag3
  • Leos
  • megavids
  • lostlight
  • All magazines