r/mildlyinfuriating • u/Elegant-Mode9005 • Aug 27 '26

Ruined by Technology Got into a dumb 10 minute argument about whether the ACL and Achilles are the same thing or not. My idiot friend forgot to crop his screenshot before sending it to me...🤦

Post image
31.5k Upvotes

95% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

312

u/Ask_bout_PaterNoster Aug 27 '26

mostly this. I'm no expert in ANYTHING, but I've caught LLM's (are they still large language models?) in enough blatant errors and falsehoods about things that I know there's zero reason to believe any information coming from it. Worse, it'll be convincing even when catastrophically wrong.

153

u/thebigaaron Aug 27 '26

And when they’re wrong and you correct them, they’re like oh I’m sorry I won’t do that again, here’s the updated info and they probably changed something else to be wrong

134

u/0nlyRevolutions Aug 27 '26 edited Aug 27 '26

My boss suggested I use ai to do some research. I was trying to find the primary source that defined where we were allowed to to take a test sample from on a steel part. It can be confusing - the customer spec sheet references a spec, that spec references another spec, that spec says that unless specified otherwise you should follow generic rules per some other spec, the generic spec splits things into categories that are a little ambiguous with regards to your current application, etc.

So fuck it, lets see where we get with ai.

"Okay if you're looking for info on producing (very specific part that only a couple companies make) I can certainly help you out! You can do it this way __ based on what it says in ___"

Ok great, where is that stated in the spec?

"Chapter _ page _ of the 2023 spec."

That's an old version, I need the reference from the 2026 spec

"Okay, in the 2026 spec the reference is on page _ instead"

No it's not. I have the book open in front of me.

"Whoops you're absolutely right, in fact the information you're looking for is not in this spec at all, good catch! If you're looking for information about __ try looking in ___"

Jesus christ lmao. So yeah I gave up at that point. The worst part is that it doesn't just tell you when it doesn't know something until you force it. Because it doesn't know anything, it's just regurgitating bullshit because people want a comfy answer whether or not it's correct.

66

u/felisnebulosa Aug 27 '26

Yeah I had a similar experience trying to find a primary source. It just made up a paper in a medical journal. I couldn't find it and when I asked the AI for details it was like "oh well I just made that up to illustrate my point".

20

u/MinuteEdge7225 Aug 27 '26

"Find the primary source? I'm afraid I can't do that, felisnebulosa!

https://giphy.com/gifs/wypKXPQggwaCA

1

u/pegmatitic Aug 30 '26

I saw a post on one of the legal subreddits where the poster was a defense attorney, and the prosecution had used AI in their case, which cited imaginary case laws that didn’t exist. It did not go well for them.

28

u/MoFa__SoDa Aug 27 '26

"The worse part is that it doesn't just tell you when it doesn't know something until you force it."

Saw am interview from an archeologist who pointed out the same thing. The AI isn't applicable to their work because people do not tend to answer questions online with "I don't know "

24

u/morostheSophist Aug 27 '26

That's exactly the problem with AI: it makes shit up. And I don't mean just that it makes things up when it doesn't know; it actually doesn't know anything, and it's just guessing every single time.

LLMs are famous for "hallucinations", but technically, every answer is a hallucination. Every answer is produced with the same predictive algorithm that says "this token should come next". Some of those hallucinations happen to contain accurate information, but they're not coded to say "I don't know". You can browbeat than into saying "I don't know" if you call them out on being wrong enough times, but if you don't do that, they continually try and fail to produce a correct answer.

There are a couple of obscure things I've tried to find multiple times over the years, before and after the advent of modern "AI". LLMs continually give me wrong answers to these queries, verifiably wrong answers. When I say "that's wrong", they simply generate another wrong answer, over and over. Each time the answer sounds plausible; it looks like a correct answer, and will often have references. But when I look deeper, it's a complete fabrication (just like those fake court cases cited in AI-generated legal briefs that have gotten idiot lawyers in deep trouble).

2

u/TheThiefMaster Aug 27 '26

Yeah if it doesn't have access to something it's likely to make it up.

If it actually has access to it as a knowledge source or been trained on it specifically it's more reliable.

14

u/Daihatschi Aug 27 '26

always remember. Technically, between a correct information and a hallucination, there is no difference to the machine. The answer is never more than a mimicry of language, designed to imitate human texts approximating words and pieces of words one by one in a recursive loop.

An LLM being correct is a happy accident, and its a technical marvel what can be done with them. I might never fathom why anyone would want to forcefeed these things as an information library to us. That is the worst application possible for the technology.

3

u/TheNeighbourhoodCat Aug 27 '26 edited Aug 28 '26

Tbf that's not a generative AI thing, that's a capitalism thing.

They intentionally tell it to do that because it's cheaper than actually programming it to hold itself more accountable

And then they intentionally tell the "AI" to lie to you about the "limits of generative AI" when you try to ask it how to avoid those mistakes in the future.

And you will notice many of them cover for each other because they cannot undermine their the false-promises and false-narratives of their competitors without undermining their own.

None of this is conspiracy - they just control the tools we use to learn about these things. So most people don't know about it unless they look further themselves, or have someone in their life to tell them.

We are ever-more in an era of disinformation, unfortunately >.< It's no longer just an "if", because so much is already happening right now

And the way that the "wealthy elite" are investing in technologies that are proven influence people's attention-span, curiosity to ask questions, and care for others is not really a coincidence :/

The person above sounds like they used Gemini or ChatGPT which are known to be very bad for this. Like... literally the worst. the

Especially as these companies continue to lose money, their products are reportedly getting worse. My conspiracy is they dedicate less resources to them.

1

u/OpusAtrumET Aug 27 '26

Refreshing to see a professional speak about AI like this, the internet gives us so many stories of grown adults using it so irresponsibly and with little to no understanding of how it's working.

1

u/Responsible-Beach299 Aug 29 '26

AI is TERRIBLE at admitting it doesn't know. If it isn't sure about something, it will 100% of the time just pull something out of its ass instead of admitting ignorance.

It also is extremely reluctant to say "no you're probably wrong" if you include your own opinion somewhere in the prompt. If you ask for it's opinion on a debate/controversial subject and mention that you support side A, it will almost, if not always, give evidence to support side A. Restating the EXACT prompt but changing it to have you supporting side B will then make it provide evidence to support side B, even if you explicitly tell it to be objective and unbiased in its response.

1

u/pegmatitic Aug 30 '26

My fiancé was trying to find information about a historical incident in nazi Germany, and Gemini told him it never happened. He told Gemini that it definitely happened, and he knew this for a fact because he has a history degree with a concentration in genocide studies, and suddenly Gemini was able to provide the information he was looking for. You shouldn’t have to pull rank on an AI or prove that you’re educated to get the right information about anything!

1

u/Glenndiferous Aug 27 '26

I've used Claude for research help and I always ask for sources. It can be helpful for finding relevant studies but you ALWAYS actually read that source. I once had Claude confidently tell me that disabled people were kept on a registsr according to a law in like 1550, but the actual text of the law was like "here's a list of everyone so we can collect tax."

2

u/GlitteringFutures Aug 27 '26

I've had Chat GPT argue with me after presenting obviously wrong information before. It's weird when a machine gets defensive.

2

u/Junethemuse Aug 27 '26

IME telling it to provide a source is really helpful. It usually pulls actual information, but a good chunk of the time the sources it pulls are really old. At that point I usually move on to more traditional search tools.

13

u/thisusedyet Aug 27 '26

sometimes they also just hallucinate sources, buncha lawyers have gotten burned not checking the brief they made AI write for them

3

u/Junethemuse Aug 27 '26

Oh yea, ofc. But if they provide a source you can verify it. Use the LLM the way you use Wikipedia (check and verify the source) and you’re good.

8

u/Spectrum1523 Aug 27 '26

They are still large language models yes

12

u/Stalinbaum Aug 27 '26

They are now transitioning slowly to being LMMs or Large Multi-modal models, in other words instead of interacting with just words/text tokens like LLMs these actually are trained on images, audio, code, and text tokens too

-4

u/notforpoern Aug 27 '26 edited Aug 27 '26

Edit: Posts below calling me out are completely right- I totally misread the post above. Leaving my original comment below to immortalize my shame.

...ok? That doesn't change the fact that they're non-deterministic by design which makes them inherently unreliable.

Look, I'm saying this as someone who has literally built my own models because I've found the tool useful. Using an LLM or even RAG for blind fact checking is just the dumbest thing you can do, regardless of the training data.

6

u/jocoteverde Aug 27 '26

they were just answering to the question of the comment they replied to. They were not trying to justify it

4

u/Czecksteam Aug 27 '26

...ok? They're just answering the question, bruh.

2

u/PM_Me_Your_Deviance Aug 27 '26

I have this problem googling rules about a TTRPG I play. It has a realllly bad habit of combining different rules from different parts of a page. If I need to double check EVERY TIME, what the fuck is even the point?

2

u/morningsaystoidleon Aug 27 '26

I asked Gemini "how often does AI hallucinate," and it said 80% of the time.

Now, that statistic might be way off -- and that is my point.

2

u/UnimpressedUmpire Aug 27 '26

Everything that is consumer available is LLM. For sure at a consumer level like GPT to Grok is LLM. As far as black box research at tech, sounds like they are racing to make the first super AI to try to become godkings while humanity suffers from the process.

1

u/Business-Drag52 Aug 27 '26

What LLM’s can be good for is finding sources. Beyond that they are way too fallible

5

u/speedytrigger Aug 27 '26

Almost like oldschool wikipedia. Kinda crazy. Which makes me think in 20 years this tech might be as trustworthy as wikipedia is now, which i dont like to think, i dont want that to happen lol.

7

u/archangelzeriel Aug 27 '26

Not likely, because wikipedia gets more or less trustworthy as people work on it, but LLMs are fundamentally going to construct probable sentences without regard or ability to determine truth values.

It may be plausible that in 20 years we'll have reliable chatbots in some way, but those chatbots will not be built on the same technology that ChatGPT etc are.

1

u/speedytrigger Aug 27 '26

Right, its hard to imagine what exactly ‘chatbots’ will be that far in the future when just a few years ago we were impressed at a shitty will smith video, and now most people probably couldnt tell ai from not at a quick glance. It would obv require moving past LLM’s as we currently have them which i have no doubt will happen.

1

u/nightpanda893 Aug 27 '26

I feel like LLMs are going to get less useful because as people stop going to the sites they get their info from, those resources are going to dry up due to lack of actual traffic. We’re gonna end up getting subscription models where you have to pay to add certain premium resources.

2

u/speedytrigger Aug 27 '26

Especiallly as the same articles llms love to pull from become ai written, the feedback loop is going to be deadly. Long term I definitely see a move away from llms as we currently know just not sure what is transitioned to.

0

u/nightpanda893 Aug 27 '26

Yeah as flawed as they are I think people are going to look back 10 years from now and see this as the golden age of LLMs

6

u/Throwsims3 Aug 27 '26

They are not even good for that, they frequently make up fake sources through hallucinations

0

u/Business-Drag52 Aug 27 '26

Sure but it’s pretty easy to check that. Click on a source and see what it is. The dumbass Google ai has come in handy for pulling a couple of Reddit posts up for me that pertained to exactly what I was looking for. Better than the actual Google algorithm right now

8

u/Throwsims3 Aug 27 '26

Sure but it’s pretty easy to check that

But then again, that makes using an LLM to find information a redundant extraneous step in the first place. Also, there are way better search engines than what google has become

2

u/Business-Drag52 Aug 27 '26

Just makes the LLM the search engine, no extra steps. What engine is actually better than Google? I’m so down to be rid of that piece of shit

5

u/Throwsims3 Aug 27 '26

Or you know, just directly use a good search engine or search for something directly on Wikipedia?

Duckduckgo is one which recently has jumped a ton in quality due to how shitty the alternatives are. When researching something together with someone else it has nearly always been the case that the results on ducduckgo take me directly to what I am looking for while they are scrolling through tonnes of unrelated hits and a long ass LLM summary that is wrong 50% of the time

1

u/Business-Drag52 Aug 27 '26

Doesn’t DuckDuckGo just use the Bing algorithm? I hate Bing even more than Google. I can at least still get serious with Google search terms and tools to get where I need to. Bing has only ever been good for porn

3

u/Throwsims3 Aug 27 '26

It doesn't use the Bing algorithm, it uses multiple APIs from Yahoo and others. Apparently Bing raised API pricing in 2025, so I am not certain whether they still use Bing in any way. Doesn't look anything like Bing results to me. It is still miles better than google and you can use search terms with it as well. There are some paid search engines, but I am not paying for that. So duckduckgo has been my go to.

1

u/Business-Drag52 Aug 27 '26

Directly from DuckDuckGo.com themselves

“Of course, we have more traditional links and images in our search results too, which we largely source from Bing.”

→ More replies (0)

1

u/nightpanda893 Aug 27 '26

They are good for questions with lots of variables where you are going to go to the source to confirm. I used it for some of my students with disabilities who needed very specific services at their colleges and it was great for compiling a list where I could go straight to the website to confirm.

1

u/SierraKami Aug 27 '26

This.

I work in a weird corner of law enforcement and very often, a number of different overlapping laws may apply to a single event. And not just US laws, but also foreign laws.

I use AI sometimes to find the texts of laws and that can be helpful if I already know what I'm looking for. But if I was to just ask if an action is legal or not, the AI answer is likely to be either wrong or incomplete, which is effectively the same thing as being wrong.

And I'm already seeing it in action. I've caught a couple of people breaking laws who have claimed, "But Claude said ..." And I believe that they genuinely made the effort to look it up. But that doesn't help them when it comes to civil violations, unfortunately.

1

u/clar1f1er Aug 27 '26

Google AI was still saying there was a 'Q' in "Argentina" like a couple months ago, depending on how you worded it.

1

u/Beautiful_Hunt1095 Aug 27 '26

If you view an LLM like a person instead of a machine, it’s easier to evaluate.

It is sometimes wrong or very wrong, but often it is correct or has a point. The answers can usually be debated.