r/technology • u/Miserable_Phase_2519 • 14h ago

Artificial Intelligence Florida woman used Claude as a diary, then Anthropic reported an entry to police

https://www.techspot.com/news/114091-florida-woman-used-claude-diary-anthropic-reported-shoot.html
13.5k Upvotes

97% Upvoted

1.4k comments sorted by

View all comments

Show parent comments

4

u/Khayrum117 11h ago

There's plenty of 1-2gb local models on HuggingFace. A 1-2gb model will run perfectly on a 8gb GPU. Also while slower, if 8gb is all you have you can offload some slightly larger models to partially work off CPU and RAM. But for what you want, there's plenty of small models youll be able to run. Hell, my Pixel 9 Pro XL can run a 4gb local model a 9 tokens a second.

2

u/calle04x 11h ago

Thanks. I should look into it.

5

u/overfloaterx 11h ago

Easy starting point is LM Studio. Desktop app laid out in a similar way to ChatGPT/Claude with a built-in index of local LLM models that you can download in a single click.

I only recently starting dipping my toes in because I haven't kept up with the big LLM players, and wanted to dabble and familiarize myself with prompting before I started sharing info with them or paying anything.

Have to admit I'm not sure what performance is like if you're running on a laptop or business-focused machine without a dedicated GPU, though.