r/mildlyinfuriating • u/Elegant-Mode9005 • Aug 27 '26

Ruined by Technology Got into a dumb 10 minute argument about whether the ACL and Achilles are the same thing or not. My idiot friend forgot to crop his screenshot before sending it to me...šŸ¤¦

Post image
31.5k Upvotes

95% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

874

u/Friendstastegood Aug 27 '26

Also ChatGPT can be wrong about things! It is not a trustworthy or reliable source because it isn't even a source at all! It's just a text generator!

312

u/Ask_bout_PaterNoster Aug 27 '26

mostly this. I'm no expert in ANYTHING, but I've caught LLM's (are they still large language models?) in enough blatant errors and falsehoods about things that I know there's zero reason to believe any information coming from it. Worse, it'll be convincing even when catastrophically wrong.

154

u/thebigaaron Aug 27 '26

And when theyā€™re wrong and you correct them, theyā€™re like oh Iā€™m sorry I wonā€™t do that again, hereā€™s the updated info and they probably changed something else to be wrong

132

u/0nlyRevolutions Aug 27 '26 edited Aug 27 '26

My boss suggested I use ai to do some research. I was trying to find the primary source that defined where we were allowed to to take a test sample from on a steel part. It can be confusing - the customer spec sheet references a spec, that spec references another spec, that spec says that unless specified otherwise you should follow generic rules per some other spec, the generic spec splits things into categories that are a little ambiguous with regards to your current application, etc.

So fuck it, lets see where we get with ai.

"Okay if you're looking for info on producing (very specific part that only a couple companies make) I can certainly help you out! You can do it this way __ based on what it says in ___"

Ok great, where is that stated in the spec?

"Chapter _ page _ of the 2023 spec."

That's an old version, I need the reference from the 2026 spec

"Okay, in the 2026 spec the reference is on page _ instead"

No it's not. I have the book open in front of me.

"Whoops you're absolutely right, in fact the information you're looking for is not in this spec at all, good catch! If you're looking for information about __ try looking in ___"

Jesus christ lmao. So yeah I gave up at that point. The worst part is that it doesn't just tell you when it doesn't know something until you force it. Because it doesn't know anything, it's just regurgitating bullshit because people want a comfy answer whether or not it's correct.

64

u/felisnebulosa Aug 27 '26

Yeah I had a similar experience trying to find a primary source. It just made up a paper in a medical journal. I couldn't find it and when I asked the AI for details it was like "oh well I just made that up to illustrate my point".

18

u/MinuteEdge7225 Aug 27 '26

"Find the primary source? I'm afraid I can't do that, felisnebulosa!

https://giphy.com/gifs/wypKXPQggwaCA

1

u/pegmatitic Aug 30 '26

I saw a post on one of the legal subreddits where the poster was a defense attorney, and the prosecution had used AI in their case, which cited imaginary case laws that didnā€™t exist. It did not go well for them.

30

u/MoFa__SoDa Aug 27 '26

"The worse part is that it doesn't just tell you when it doesn't know something until you force it."

Saw am interview from an archeologist who pointed out the same thing. The AI isn't applicable to their work because people do not tend to answer questions online with "I don't know "

24

u/morostheSophist Aug 27 '26

That's exactly the problem with AI: it makes shit up. And I don't mean just that it makes things up when it doesn't know; it actually doesn't know anything, and it's just guessing every single time.

LLMs are famous for "hallucinations", but technically, every answer is a hallucination. Every answer is produced with the same predictive algorithm that says "this token should come next". Some of those hallucinations happen to contain accurate information, but they're not coded to say "I don't know". You can browbeat than into saying "I don't know" if you call them out on being wrong enough times, but if you don't do that, they continually try and fail to produce a correct answer.

There are a couple of obscure things I've tried to find multiple times over the years, before and after the advent of modern "AI". LLMs continually give me wrong answers to these queries, verifiably wrong answers. When I say "that's wrong", they simply generate another wrong answer, over and over. Each time the answer sounds plausible; it looks like a correct answer, and will often have references. But when I look deeper, it's a complete fabrication (just like those fake court cases cited in AI-generated legal briefs that have gotten idiot lawyers in deep trouble).

2

u/TheThiefMaster Aug 27 '26

Yeah if it doesn't have access to something it's likely to make it up.

If it actually has access to it as a knowledge source or been trained on it specifically it's more reliable.

15

u/Daihatschi Aug 27 '26

always remember. Technically, between a correct information and a hallucination, there is no difference to the machine. The answer is never more than a mimicry of language, designed to imitate human texts approximating words and pieces of words one by one in a recursive loop.

An LLM being correct is a happy accident, and its a technical marvel what can be done with them. I might never fathom why anyone would want to forcefeed these things as an information library to us. That is the worst application possible for the technology.

3

u/TheNeighbourhoodCat Aug 27 '26 edited Aug 28 '26

Tbf that's not a generative AI thing, that's a capitalism thing.

They intentionally tell it to do that because it's cheaper than actually programming it to hold itself more accountable

And then they intentionally tell the "AI" to lie to you about the "limits of generative AI" when you try to ask it how to avoid those mistakes in the future.

And you will notice many of them cover for each other because they cannot undermine their the false-promises and false-narratives of their competitors without undermining their own.

None of this is conspiracy - they just control the tools we use to learn about these things. So most people don't know about it unless they look further themselves, or have someone in their life to tell them.

We are ever-more in an era of disinformation, unfortunately >.< It's no longer just an "if", because so much is already happening right now

And the way that the "wealthy elite" are investing in technologies that are proven influence people's attention-span, curiosity to ask questions, and care for others is not really a coincidence :/

The person above sounds like they used Gemini or ChatGPT which are known to be very bad for this. Like... literally the worst. the

Especially as these companies continue to lose money, their products are reportedly getting worse. My conspiracy is they dedicate less resources to them.

1

u/OpusAtrumET Aug 27 '26

Refreshing to see a professional speak about AI like this, the internet gives us so many stories of grown adults using it so irresponsibly and with little to no understanding of how it's working.

1

u/Responsible-Beach299 Aug 29 '26

AI is TERRIBLE at admitting it doesn't know. If it isn't sure about something, it will 100% of the time just pull something out of its ass instead of admitting ignorance.

It also is extremely reluctant to say "no you're probably wrong" if you include your own opinion somewhere in the prompt. If you ask for it's opinion on a debate/controversial subject and mention that you support side A, it will almost, if not always, give evidence to support side A. Restating the EXACT prompt but changing it to have you supporting side B will then make it provide evidence to support side B, even if you explicitly tell it to be objective and unbiased in its response.

1

u/pegmatitic Aug 30 '26

My fiancĆ© was trying to find information about a historical incident in nazi Germany, and Gemini told him it never happened. He told Gemini that it definitely happened, and he knew this for a fact because he has a history degree with a concentration in genocide studies, and suddenly Gemini was able to provide the information he was looking for. You shouldnā€™t have to pull rank on an AI or prove that youā€™re educated to get the right information about anything!

1

u/Glenndiferous Aug 27 '26

I've used Claude for research help and I always ask for sources. It can be helpful for finding relevant studies but you ALWAYS actually read that source. I once had Claude confidently tell me that disabled people were kept on a registsr according to a law in like 1550, but the actual text of the law was like "here's a list of everyone so we can collect tax."

2

u/GlitteringFutures Aug 27 '26

I've had Chat GPT argue with me after presenting obviously wrong information before. It's weird when a machine gets defensive.

2

u/Junethemuse Aug 27 '26

IME telling it to provide a source is really helpful. It usually pulls actual information, but a good chunk of the time the sources it pulls are really old. At that point I usually move on to more traditional search tools.

12

u/thisusedyet Aug 27 '26

sometimes they also just hallucinate sources, buncha lawyers have gotten burned not checking the brief they made AI write for them

3

u/Junethemuse Aug 27 '26

Oh yea, ofc. But if they provide a source you can verify it. Use the LLM the way you use Wikipedia (check and verify the source) and youā€™re good.

8

u/Spectrum1523 Aug 27 '26

They are still large language models yes

12

u/Stalinbaum Aug 27 '26

They are now transitioning slowly to being LMMs or Large Multi-modal models, in other words instead of interacting with just words/text tokens like LLMs these actually are trained on images, audio, code, and text tokens too

-4

u/notforpoern Aug 27 '26 edited Aug 27 '26

Edit: Posts below calling me out are completely right- I totally misread the post above. Leaving my original comment below to immortalize my shame.

...ok? That doesn't change the fact that they're non-deterministic by design which makes them inherently unreliable.

Look, I'm saying this as someone who has literally built my own models because I've found the tool useful. Using an LLM or even RAG for blind fact checking is just the dumbest thing you can do, regardless of the training data.

6

u/jocoteverde Aug 27 '26

they were just answering to the question of the comment they replied to. They were not trying to justify it

3

u/Czecksteam Aug 27 '26

...ok? They're just answering the question, bruh.

2

u/PM_Me_Your_Deviance Aug 27 '26

I have this problem googling rules about a TTRPG I play. It has a realllly bad habit of combining different rules from different parts of a page. If I need to double check EVERY TIME, what the fuck is even the point?

2

u/morningsaystoidleon Aug 27 '26

I asked Gemini "how often does AI hallucinate," and it said 80% of the time.

Now, that statistic might be way off -- and that is my point.

2

u/UnimpressedUmpire Aug 27 '26

Everything that is consumer available is LLM. For sure at a consumer level like GPT to Grok is LLM. As far as black box research at tech, sounds like they are racing to make the first super AI to try to become godkings while humanity suffers from the process.

0

u/Business-Drag52 Aug 27 '26

What LLMā€™s can be good for is finding sources. Beyond that they are way too fallible

5

u/speedytrigger Aug 27 '26

Almost like oldschool wikipedia. Kinda crazy. Which makes me think in 20 years this tech might be as trustworthy as wikipedia is now, which i dont like to think, i dont want that to happen lol.

7

u/archangelzeriel Aug 27 '26

Not likely, because wikipedia gets more or less trustworthy as people work on it, but LLMs are fundamentally going to construct probable sentences without regard or ability to determine truth values.

It may be plausible that in 20 years we'll have reliable chatbots in some way, but those chatbots will not be built on the same technology that ChatGPT etc are.

1

u/speedytrigger Aug 27 '26

Right, its hard to imagine what exactly ā€˜chatbotsā€™ will be that far in the future when just a few years ago we were impressed at a shitty will smith video, and now most people probably couldnt tell ai from not at a quick glance. It would obv require moving past LLMā€™s as we currently have them which i have no doubt will happen.

1

u/nightpanda893 Aug 27 '26

I feel like LLMs are going to get less useful because as people stop going to the sites they get their info from, those resources are going to dry up due to lack of actual traffic. Weā€™re gonna end up getting subscription models where you have to pay to add certain premium resources.

2

u/speedytrigger Aug 27 '26

Especiallly as the same articles llms love to pull from become ai written, the feedback loop is going to be deadly. Long term I definitely see a move away from llms as we currently know just not sure what is transitioned to.

0

u/nightpanda893 Aug 27 '26

Yeah as flawed as they are I think people are going to look back 10 years from now and see this as the golden age of LLMs

5

u/Throwsims3 Aug 27 '26

They are not even good for that, they frequently make up fake sources through hallucinations

0

u/Business-Drag52 Aug 27 '26

Sure but itā€™s pretty easy to check that. Click on a source and see what it is. The dumbass Google ai has come in handy for pulling a couple of Reddit posts up for me that pertained to exactly what I was looking for. Better than the actual Google algorithm right now

10

u/Throwsims3 Aug 27 '26

Sure but itā€™s pretty easy to check that

But then again, that makes using an LLM to find information a redundant extraneous step in the first place. Also, there are way better search engines than what google has become

2

u/Business-Drag52 Aug 27 '26

Just makes the LLM the search engine, no extra steps. What engine is actually better than Google? Iā€™m so down to be rid of that piece of shit

4

u/Throwsims3 Aug 27 '26

Or you know, just directly use a good search engine or search for something directly on Wikipedia?

Duckduckgo is one which recently has jumped a ton in quality due to how shitty the alternatives are. When researching something together with someone else it has nearly always been the case that the results on ducduckgo take me directly to what I am looking for while they are scrolling through tonnes of unrelated hits and a long ass LLM summary that is wrong 50% of the time

1

u/Business-Drag52 Aug 27 '26

Doesnā€™t DuckDuckGo just use the Bing algorithm? I hate Bing even more than Google. I can at least still get serious with Google search terms and tools to get where I need to. Bing has only ever been good for porn

3

u/Throwsims3 Aug 27 '26

It doesn't use the Bing algorithm, it uses multiple APIs from Yahoo and others. Apparently Bing raised API pricing in 2025, so I am not certain whether they still use Bing in any way. Doesn't look anything like Bing results to me. It is still miles better than google and you can use search terms with it as well. There are some paid search engines, but I am not paying for that. So duckduckgo has been my go to.

→ More replies (0)

1

u/nightpanda893 Aug 27 '26

They are good for questions with lots of variables where you are going to go to the source to confirm. I used it for some of my students with disabilities who needed very specific services at their colleges and it was great for compiling a list where I could go straight to the website to confirm.

1

u/SierraKami Aug 27 '26

This.

I work in a weird corner of law enforcement and very often, a number of different overlapping laws may apply to a single event. And not just US laws, but also foreign laws.

I use AI sometimes to find the texts of laws and that can be helpful if I already know what I'm looking for. But if I was to just ask if an action is legal or not, the AI answer is likely to be either wrong or incomplete, which is effectively the same thing as being wrong.

And I'm already seeing it in action. I've caught a couple of people breaking laws who have claimed, "But Claude said ..." And I believe that they genuinely made the effort to look it up. But that doesn't help them when it comes to civil violations, unfortunately.

1

u/clar1f1er Aug 27 '26

Google AI was still saying there was a 'Q' in "Argentina" like a couple months ago, depending on how you worded it.

1

u/Beautiful_Hunt1095 Aug 27 '26

If you view an LLM like a person instead of a machine, itā€™s easier to evaluate.

It is sometimes wrong or very wrong, but often it is correct or has a point. The answers can usually be debated.

45

u/ArokLazarus Aug 27 '26

I've literally had ChatGPT tell me you should buy high and sell low in stocks to make money. As like the typical thing to do. Literally goes against all money making sense but I could not get it to break free of it's delusion.

8

u/pornalt4altporn Aug 27 '26

It doesn't matter, they still generate output that fits the statistical distribution they have learned. They don't consult a declarative memory store and retrieve correct information.

Now perhaps someone will invent as way to have them do that, but training bigger neural networks on larger datasets doesn't do that.

5

u/thisusedyet Aug 27 '26

I mean, statistically speaking, it IS the typical thing to do. You just don't make any money off it

14

u/MRCHalifax Aug 27 '26

I remember one argument I had on Reddit, where someone insisted that Quebec was the only fortified city in Canada, and posted ChatGPT quotes as proof. As someone who once had a summer job involving working in one of the forts that encircled and defended Halifax, I begged to differ.

9

u/VivaZeBull Aug 27 '26

It constantly tells me I am misquoting people ā€¦ I am not you just donā€™t know the episode, jerk.

My biggest pissoff was when it was new and google kept telling me that Nancyā€™s Mom didnā€™t say ā€œIs Steve your boyfriend now?ā€

Yes she did bitch. Yes she did.

3

u/Themoosemingled Aug 27 '26

I was writing back in the winter how upset I was by Rob Reinerā€™s murder.
Chat stats saying ā€œRob Reiner is alive and well. It may seem like bad things have happened but heā€™s fineā€.
Really fucked up.

5

u/bekahed979 Aug 27 '26

Isn't this literally a fake reference? I don't understand how anyone sees AI as anything but awful.

3

u/The-Singular Aug 27 '26

So, this could be real all along! Imagine AI hallucinating a correct answer as wrong.

1

u/TheReturnofPanda Aug 27 '26

OPā€™s friend is influencing GPT to be giving out more false information. This is exactly how it gets trained

1

u/burnerX5 Aug 27 '26

I was taking a recert exam and asked Gemini a question. It defiantly told me that yep, a test question was correct.

...It was incorrect and I had ot restart the exam as it's the type where you have to get 100% correct. Damn near made me look for my screenshot file of all the test questions Ifrom last year!!!!

1

u/MaiT3N Aug 28 '26

"sorry I just hallucinated everything above, and nothing I said is actually true"

1

u/AzazelsAdvocate Aug 27 '26

For basic information like this, modern models don't ever really get it wrong. Where they tend to hallucinate is when you ask them about information that isn't readily available or easily searchable. For example, if I try to ask Claude about a niche mechanic in a video game, it'll often come up with something that "sounds correct" rather than finding the real answer.

Of course you can work around this by requesting sources for the things AI finds, which is good practice in general.

2

u/instanding Aug 27 '26

Thatā€™s not my experience. Judo info is easily searchable for instance, yet it will claim people beat people they never beat at weight divisions 3 categories above what they actually competed at.

And there are websites and blogs and all sorts with the correct infoā€™.

2

u/shaurya_770 Aug 27 '26

If u ask for it to provide credible sources it's rarely wrong. So it works really well as a summarozer