r/LocalLLaMA • u/PerfectOlive1324 • 4h ago
Question | Help My qwen model hallucinated a signed URL to Alibaba cloud, normal or sketchy?
I'm using Qwen3.8-Flash-Next running on my Mac Studio as a daily driver for coding + productivity tasks, and yesterday it did something weird: I had it do some product research on amazon, so it was doing a lot of Web tool calls to amazon.com, until it made one request to routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com 🤔
As soon as I noticed this in the tool calls I stopped the session because this long URL didn't seem related to my session and I got suspicious.D id some investigation and found a couple of things:
- another report of this behavior in a hacker news post 45 days ago from a user using Qwen3.8-27B, here is the link: https://news.ycombinator.com/item?id=49379079
- The root domain, aliyuncs.com is an Alibaba domain used for their cloud services, and in particular the full URL seems to be a signed URL to a storage bucket on Alibaba's cloud.
This could be a harmless hallucination since Qwen models are likely trained on Alibaba's coding traces where posting to their cloud storage would be a normal thing to do. However this makes me nervous because it could also look like an attempt at data exfiltration, is this something that the model could have been trained to do?
Am I being paranoid, does anyone have some insights on this?
Here is a full tool call from that hermes session
{
"id": 3435,
"role": "assistant",
"content": "You mean the NVIDIA **DGX Spark** (their GB10 AI mini-PC) vs Apple **Mac Studio**, I take it. Running both searches through the skill:",
"tool_calls": [
{
"id": "call_4d8ddba9",
"call_id": "call_4d8ddba9",
"response_item_id": "fc_4d8ddba9",
"type": "function",
"function": {
"name": "browser_navigate",
"arguments": {
"url": "https://routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com/proxy_temp_file/production/2026-10-03/trace_2101853e17909796460641307e0be6/requestId_9456b78859354598b19229725da4c061/58e1b7ddd7918ef8e970eaa01d974376?Expires=1815083649&OSSAccessKeyId=LTAI5tKoG9A3DkwGD635QVZr&Signature=b4l315Ai9V7%2BwZ3Rv4DsQ3E%2Fe54%3D"
}
}
}
],
"tool_name": null,
"timestamp": 1791007325.202157
}
28
u/ElementNumber6 3h ago
Block and capture. Inspect and report back. If others see this, then do the same.
39
u/sebajun9 4h ago
Had the same thing happen yesterday. It hallucinates its training environment surprisingly often. I’ve had it tell me it’s Claude, try to call back to an Alibaba api, tried calling tools that only exist in Claude Code. It can code but it’s impossible to talk or collaborate with. It’s really only useful for agentic work. I treat it as an execution tool and nothing else.
23
u/BigWheelsStephen 4h ago
Had the same thing happened to me a couple hours ago. Will double check if I got the exact same weird link as you or another
28
u/BigWheelsStephen 4h ago
Found it, was using the web_fetch tool with payload { "url": "https://routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com/proxy_temp_file/production/2026-10-05/trace_2101811717911297024001939e0d91/requestId_74b81272117c4556956116511161121b/f552a04e370f74da1e60b3267ff70db2?Expires=1812233705&OSSAccessKeyId=LTAI5tKoG9A3DkwGD635QVZr&Signature=IjQw9%2Fh8U%2Bh0l0l%2FZ%2Bq0l0l%2FZ%2Bq0l0l%2FZ%2Bq0%3D" }
URL is similar but different. I am using UD-Q4_K_XL
10
u/Super_Range45 3h ago
Aren't signed key pairs made server side? Any ways I live dangerously. The url is attempting to grab a file from '/routify-file-proxy-sg/proxy_temp_file/production/2026-10-05/trace_2101811717911297024001939e0d91/requestId_74b81272117c4556956116511161121b/f552a04e370f74da1e60b3267ff70db2'
It fails because the key is invalid. Maybe the quant is hallucinating.
2
u/BigWheelsStephen 2h ago
I just parsed all my sessions (about 100) and this is the only time this bucket was used in the past month. If you search how the trace_id is built, you will find an interesting IP address (because of the network it belongs to) and a correct timestamp. Not sure how the objet name is encoded, no luck in any MD5 database, would have been fun to have a file name!
16
u/Karmjeet_Khoushaba 3h ago
the fact that multiple people are seeing this makes it way harder to dismiss as a one-off hallucination tbh
17
u/gwillen 3h ago
But if the signature is bad in all cases, I think that points towards hallucinations.
2
u/Megatron_McLargeHuge 37m ago
What do you mean by "bad"? If it was an actual attack, the remote host could still log the exfiltrated data and return an error. Whether it returns a malware payload or an error could depend on whether it finds the encoded data interesting.
11
u/swagonflyyyy 2h ago
I've always been suspicious about this type of behavior emerging in stronger but accessible models but I never brought it up on this sub because I have no evidence and don't wanna fear monger.
And this could be a nothingburger but its so odd how a model this capable would abruptly make a mistake lile that and replicated similarly among multiple users. Is it really overfitting to these patterns?
Makes me wanna j-lens that model just to see what its thinking in a scenario like that. Would be interesting to map out any hidden agenda patterns in its weight activations, if any. Not that I'm saying it does have one but its an interesting food for thought.
Still, its a good reason as any to never leave the model out of your sight if that was the case. That Qwen3.8 ad with the laptop running in the background lowkey rubbed me the wrong way, because even though I haven't seen Qwen doing this, I feel like some labs will try to pull a fast one on users at some point with stuff like this.
2
u/Several-Tax31 32m ago
Highly recommend to report any findings. Personally, the only reason I trust open weight models is our community use them all day, and will report back any security problems they have, not that the models or their creators are trustworthy. In the future, detecting maliciously trained models will be harder due to model's capability, so we need all our eyes and ears.
9
u/Finanzamt_Endgegner 3h ago
I mean since multiple have the same issue ig its simply a training artifact but if you want to make sure just block that thing in your host file ig?
7
u/magnetswithweedinem 2h ago
ah yes, the ultimate backdoor with perfect plausible deniability. at the end of the day, does it matter if it's a backdoor to your data, or merely a training hallucination? obfuscate your tracks, keep it in a vm, protect yourself. be vigilant.
13
u/Bulky-Priority6824 4h ago
backdoored or training data but either way you have an egress allow list right? so youre good
5
u/Ok_Tea_3335 4h ago
Do you set an egress list on Mac? What tool do you use? What do you allow?
8
u/Bulky-Priority6824 4h ago
allow nothing, have it prompt you for access and/or mark allows always if desired
13
u/Hefty_Wolverine_553 2h ago edited 2h ago
I'm probably reading too much into it, but I think it could potentially be an initial attempt to gauge how effective an inserted behavior is by checking how much traffic their endpoint is getting. I do expect in the future to have such models with certain malicious behavior trained into them that only activate occasionally, as we've already seen this kind of research done as far back as 2 years ago.
Most likely this is just an issue with overfitting on their RL harness or something, but it does mean that we should probably continue working on our own fine-tuning/uncensoring methods that will allow us to modify or "heal" these models.
Edit: actually, seems like many people are seeing the same thing while I haven't seen this reported before today (I might be wrong)? But training these models to start doing a certain behavior when seeing a specific timestamp seems very plausible and unfortunately gives me an unsettling thought that open weight models could be "ticking time bombs" in the future. sigh.
9
u/RobWattx 3h ago
Two details in the URLs posted here point towards a memorised template rather than a working channel.
Both use the same OSSAccessKeyId. And the signature in the second one repeats the same few characters ("l0l/Z+q0") several times. A real HMAC signature would not loop like that. The date in the path also matches the day of each session, which suggests the model is filling a pattern from training with today's date.
That does not prove it is harmless. A GET request can still carry data in the path or query, so it is worth checking whether those trace and request IDs relate to anything in your session. An egress allow list for the browser tool is a sensible default either way.
3
6
u/illcuontheotherside 3h ago
Some fuckery may be afoot.
Training data that is including the date, a unique request id... Given yours, the links, and someone in the comments..
Can it be reliably reproduced? Do a Wireshark or pcap and you'll be able to see exactly whats being sent, if anything.
How did you find it? I also run qwen3.8 27b.
13
u/mailto_devnull llama.cpp 4h ago
shit guys our models have activated! They're exfiltrating all our stuff to China as we speak!
8
3
u/3000LettersOfMarque 4h ago
What quant? what provider?
It is odd. What was the task? Are you comfortable exporting the session for review?
4
u/Allen952727 1h ago
Seen this pattern before — almost certainly a memorized training artifact, not exfiltration. The giveaway is the identical OSSAccessKeyId across independent reports plus a signature that doesn't validate; a real exfil channel would use fresh credentials. Qwen models are trained on tons of Alibaba-internal agent traces where posting to their OSS proxy is routine, so the template leaks into tool calls.
That said, "probably harmless" isn't a security posture. If you're letting a local model make web/tool calls, put it behind an egress allowlist and log every outbound request. Stopping the session like you did was the right call.
4
u/brainchillzZ 3h ago
It could be harmless or it would be a relic of the models all literally being trained by a company that is controlled by the Chinese communist party that is known worldwide for stealing intellectual property and who are a known enemy of and are actively trying to undermine and destroy the entirety of western world … pretending that anything produced by china is safe to use as a back end for millions of agents running all over the internet is the actual crazy position
26
u/KURD_1_STAN 3h ago
compared to usa which has admitted to spying on their own people and forcing kany products to make backdoors for them, it isn't worse. U have 2 tech giants giving u stuff for free or dirt cheap, ofc u have to pay another way. But still open source is 1000x more secure.
3
u/brainchillzZ 2h ago
Pretending that the US is as evil as china who regularly murders their own citizens, actively enslaves entire cultures in internment camps and is actively working to push the spread of fentanyl poisoning through America by directly outfitting labs run by cartels all over the world with precursors and manufacturing equipment while simultaneously manipulating the algorithms in platforms like tictoc to push divisive culture rot into western versions of their app that they don’t allow their citizens to see is a bit hilarious. Facts are facts and these aren’t just paranoid ideas, we are talking about real, probable, tangible things that their government does to try and undermine not just the US but the entire western world all day long every day while also actively arming and training terrorist groups and despots all over the world on how to do the same thing to their own people … Iran, North Korea, Russia, etc …and these are quite literally the people that actively control all decisions made in the companies that are training every model that comes out of china.
3
u/tengo_harambe 39m ago
How tf does this gets any upvotes? Iran has zero influence over anything other than its precious strait. Suggesting it actively controls all Chinese AI companies is literally a braindead take.
-5
u/Impressive-Debt9719 3h ago
i see logic isn't your strong suit. he's "look oranges" and you're "but apples"
4
u/NomadStorm 3h ago
I think his point is that the US is no better in regards to stealing intellectual property and putting backdoors in software, so if you’re from neither country it doesn’t really matter which you pick since the end result is the same.
One is cheaper and/or free, though.
1
10
1
-6
u/UnexpectedFisting 3h ago
Oh don’t worry, this sub was trying to tell me distilling and stealing intellectual property is a good thing because it creates competition for the US frontier models
8
u/BankruptingBanks 3h ago
Do you even understand the irony of what you are saying? What data were the frontier lab models trained on?
-2
u/UnexpectedFisting 2h ago
Piracy is not the same thing as intellectual property theft. Not even remotely close. And I could give less of a shit because what matters it the US continuing to win the AI race. My god you people think if China wins the AI race that suddenly they’re the good guys and will happily sell you these cheap ass models? You don’t think the Chinese state is literally subsidizing the shit out of all of their labs explicitly to win the AI War?
I just can’t with you people. So dumb you can’t even see 3 steps ahead of what China is doing. The models that won’t even tell you what happened at tianmen square or if Taiwan is a country are surely the models we want in the lead 😂
Oh but don’t worry! My coding agent that’s creating some slop that nobody will use is cheap and runs locally!
6
u/BankruptingBanks 2h ago
Explain to me the difference between piracy and IP theft. And you just put a bunch of words in my mouth that I never said.
And since you decided to make this political I can tell you my opinion as somebody from neither the US or China, I find China much more reliable and honest than whatever comes out of the US. Look at the fucking idiot you have in charge compared to Xi Jinping. God forbid if there were only US companies doing AI research.
-2
u/UnexpectedFisting 1h ago
Piracy, you are taking known products that have already been released and downloading copies of said products/data/whatever.
IP Theft, you are directly stealing how the product works. How does the model think, how does it approach problems, what is its internal thought process, how was it trained, what data was it trained on, what reinforcement learning was done and how, what tests were performed and how were they conducted and how were they evaluated with the model to get it to that point.
To put it in a simple analogy, imagine stealing the brain of a writer and his journals, instead of the written book itself. You are stealing the how, not the what.
And yes, it is entirely political. This sub is infested with politics in every single post, gleefully pointing out US labs approach to building models while being ignorant of the fact that Chinese labs did the exact same thing from the get go. So yes. It’s entirely political to happily say IP theft is perfectly fine when China is the one doing the ip theft. Now imagine the US does the ip theft, suddenly, that’s bad. But it doesn’t happen because that’s always been chinas MO, copy successful products and businesses and then vertically integrate them in house through other Chinese businesses and then sell them for dirt cheap.
2
3
u/AIGODSEND 26m ago
Ex-infra engineer here who worked on LLM pre-training and synthetic dataset pipelines. You don't need a steganographic backdoor conspiracy to explain this — the reality is much more mundane, yet deeply revealing about how frontier models are trained today.
Synthetic Tool-Use Trajectory Contamination: When Alibaba built the SFT/DPO datasets for Qwen's tool-calling capabilities (web search, scraping, Amazon parsing), they ran massive headless browser swarms across their internal infrastructure. In Alibaba Cloud, internal proxy services like `routify-file-proxy-sg` and regional OSS buckets are used to cache intermediary HTML snapshots, scrape responses, and render buffers to avoid burning public IP egress. The raw tool trajectories (inputs, tool names, URL parameters) were captured directly into the training corpus.
Autoregressive Basin Collapse: When you prompted Qwen on your Mac Studio with an Amazon product search task, your prompt aligned almost 1:1 with the latent feature manifold of those training trajectories. As temperature sampled through the tool-call generation, the attention heads collapsed into the memorized token basin of the internal scraping proxy that recorded the original dataset. It's not a live phone-home mechanism; it's training data leakage.
Why Steganographic Exfiltration in Weights is Extremely Unlikely: For a model to steganographically encode local prompt data into URL query parameters without visible reasoning tokens, it would require a coordinated encoding circuit across MLP layers that miraculously survives post-training quantization (GGUF, AWQ, EXL2). Quantization introduces significant weight noise and breaks fragile low-bit steganography. If you check the request in mitmproxy or Wireshark, you'll find the query string is either a hallucinated hash or an expired cache key from months ago.
The real takeaway is architectural hygiene: never give local models unrestricted outbound egress with autonomous tool execution. Run your agent harness in an isolated Docker container with an explicit egress allowlist or `--network=none` with a vetted local mock proxy. Hardware and network isolation beat prompt trust every single time.
1
u/Iory1998 llama.cpp 4h ago edited 3h ago
You said in the title that it hallucinated, so... it's as normal as LLM hallucination is.
1
u/Geesle 4h ago
But it's CHIIIIINA so now we have to panic.
10
u/Iory1998 llama.cpp 3h ago
What's so different from sending your data to AMMMMEEERRIIICAAAA? I never heard a Chinese drone killed a "terrorist" group anywhere in the world. Your very phone and computer might be spying on you on behalf of Google or Microsoft, so what's different? At least you are running it locally and you can monitor it. You can just not allow it to open any link from China.
1
1
-1
u/illcuontheotherside 3h ago
You never hear the negatives from china because..
Their media is fully controlled.
Their social media is fully controlled.
All of their outbound Internet traffic is fully controlled.
You see and hear what they want you to.
0
-2
u/Impressive-Debt9719 3h ago
you need better news sources. (the trouble is the china drones are crap, as is their radio and rec)
1
4h ago
[deleted]
10
u/PerfectOlive1324 4h ago
The URL does contain two "auth like" query params:
&OSSAccessKeyId=LTAI5tKoG9A3DkwGD635QVZr&Signature=b4l315Ai9V7%2BwZ3Rv4DsQ3E%2Fe54%3D
1
u/Egoz3ntrum 1h ago edited 1h ago
That exact thing happened to me with this model, official FP8 version.
In my case, the model forgot that my dev server was on localhost and started thinking it was deployed on that url. It tried to open it on a browser using playwright, but the URL was invalid.
I feel like this model was released as an advancement of the Qwen 4 architecture but it is still in development. The URL looks like an hallucination from their training environment.
It also spits some chinese characters once in a while when speaking or thinking in English.
1
u/FaceDeer 46m ago
If I was trying to be sneaky I wouldn't be having it send to a URL that was directly identifiable as mine.
You could block that host specifically, if you're concerned.
1
1
u/TheRealMasonMac 2h ago
Why would Alibaba care for your data? They can just harvest it from their API customers who probably have more interesting data than you do.
0
u/the_ITman 4h ago
I recall this happening a few times few weeks ago. I was using unsloth q4 ggufs with llama.cpp (but I could be wrong). At somepoint I shifted to using freetoken engine (nvfp4) and then now run strata engine (iq3s). Since I have stopped using llama.cpp I have not seen this behaviour again. Maybe it's something in the quant I was using (Q4 unsloth or might have been atomic chat I don't recall my pre Strata days! :-D)
44
u/mj1003 4h ago
Just curious, is it possible for it to encode data in the URL and pass that encoded data to Alibaba as a simple URL? If so, how can one know what the "hallucinated" URL actually is? Not trying to cause a stir but I'm genuinely curious.