r/hacking • u/Thin-Bobcat-4738 • 11d ago
great user hack My local AI pentesting stack is dailed in!
"Qwen3.6 35B + custom tools + docker"
Just wanted to share a local setup I finally got working smoothly for my pentesting workflow.
Im running Qwen3.6 35B locally via ollama, wrapped inside an open webui docker container on kali. To make it easily accessible with local https, im routing everything through caddy using nip.io for dns management.
instead of using the llm as a glorified search engine I bundled a unified python script directly into the open webui tools framework to give the model real capabilities.
right now it can:
scrape github repos natively + parse nmap logs + isolated python sandbox.
all local no api, self hosted, no data leaks.
honestly this wasn't a easy task for me, never touching a llm model locally and learning on the fly.
it was a lot of fun setting up (3 days straight) but now it's time to have some fun with it.
if you want to do this or something similar lmk, I would like to see others use ai like this.
7
u/-Krotik- 7d ago
ditch ollama for llama.cpp
1
u/Thin-Bobcat-4738 5d ago
yeah? whats the benefit? like I said before I'm new to the self hosted ai space.
3
u/Own_Picture_6442 7d ago
Very cool. I’m also learning and have some similarity in setup. I’m running Ubuntu server and vLLM on a desktop hosting my GPU. That server backends Open WebUI and a home assistant voice assistant attached to robotic eyes (still being borned). I just stood up caddy today, still need to finish configuration. I’m running an RTX 4060ti. I started with Qwen 14B but too quickly maxed out tokens. I switched to IBM Granite 8B and it’s SOOO much faster. About to start adding features like doc creation and tooling. Anyways, very cool! Keep up posted!
2
u/Thin-Bobcat-4738 6d ago
dude, sounds like you are having just as much fun as I am. I know all too well what you mean with qwen 14b maxing out. the more and more I invest focus in "ai homelab" the more I want to grab a 3090 24gb. Yeah caddy gives me peace of mind for the most part. Keep on keeping on man!
1
u/Thin-Bobcat-4738 6d ago
dude, sounds like you are having just as much fun as I am. I know all too well what you mean with qwen 14b maxing out. the more and more I invest focus in "ai homelab" the more I want to grab a 3090 24gb. Yeah caddy gives me peace of mind for the most part. Keep on keeping on man!
3
u/Skater_Bruski 6d ago
KOTIS pairs perfectly with this and now I want a cyberpunk skin for Kotis on arena.
3
u/Thin-Bobcat-4738 6d ago
lol it does right, one of my fav commanders;)
1
u/Skater_Bruski 6d ago
Bane of my existence. I play brawl to play stupid tribal decks and start sweating when I see it.
1
2
u/Acceptable-Gold-9771 10d ago
35b is good for pentesting 👀 pretty cool
1
u/Thin-Bobcat-4738 10d ago
since it's just MoE not the best but good enough, I also have qwen3.6-27B which I think is MUCH better at big prompts of data dump parsing like nmap scans.
1
u/Thin-Bobcat-4738 10d ago
it handles function calling pretty well, so far I've successfully used it to chain 3 tools together. the results were pretty spot on as far as what I was looking for. I'm gonna keep including tools until I push it to the limits. i need to add a base64 decoder next.
2
u/VeryFortniteOfYou 7d ago
Take into consideration when configuring security that containers like docker have had various escapes released in recent months. If you develop a more permissive version you may want to upgrade to an actual factual vm, or add some supervisory layer. ymmv. Cool concept.
1
2
u/Highfivesghost 7d ago
Thanks for sharing. Might have some more questions as I’m in the middle of doing something similar
2
u/Zaxtie 6d ago
If you’re interested in taking this to an actual career/home lab situation for an interview all companies would likely do these actions with a SOAR solution i.e. a container with python or similar on it. Create an agent with your LLM by defining a scope and scanning data. Data is security. Maybe create a mitre att&ck agent that attaches ttps to your pen test results or create an investigation package to create more scripts for your soar solution (like patching or similar). Your odds of getting a container with root python privs in a legit enterprise is already low but allowing it to pentest is 0.
If you’re doing it for lols and fun more power to you.
1
u/Thin-Bobcat-4738 6d ago
yeah it's all for fun tbh. With ai being a buzz I just want to know how capable a little ol local llm can be. I'll have to be honest I've been the type to avoid ai in the past, that's changing as of lately. I've been really having fun with it.
1
1
10d ago
[removed] — view removed comment
1
u/Thin-Bobcat-4738 10d ago
my custom python exe with root equivalent access inside my open webui docker container. basically getting full sec context of that container.
1
u/BSheep14 6d ago
Is the only way to get an ai model to actually assist with pentesting (even on your own stuff) without it putting up the “I can’t help with that” guard rails to host it locally? I was trying to get Gemini and Claude to become a teacher of sorts and I tested metasploitable and Kali in my homelab to learn and it gets to a point where they just start to believe your intent is malicious
So my assumption is a self hosted llm I can remove all the safety bs right?
1
u/Thin-Bobcat-4738 6d ago
yeah I've gotten past guardrails with using uncensored models or good prompts. googles public gemini model loves hooking me up with prompts to bypass some blocks i may face using my local llm. not good enough to bypass EDR or anything crazy, but some lighter stuff like giving me vulnerabilities to services on open ports to exploit.
2
u/BSheep14 6d ago
Im curious as to what models are worth spinning up that are super locked down and still quality enough data sets/ responses
My 3090ti is by far the best GPU I own but I wouldn’t want to run the majority of my work from that machine I rather just steal its compute and feed it into the VMs on my proxmox cluster just for local llm
2
u/Thin-Bobcat-4738 5d ago
what's your vram on the 3090? if it's 24gb then you can run a REALLY good llm like qwen-3.8-27b-uncensored with no issues locally. the most secure and private way to run it is locally ollama<open webui<docker<caddyfile<nip.io and if you decide to give it tool functionality like web search you can also set that up to be private. you can configure it in a way that chat history ect gets wiped after every session. I haven't ran this stack in a vm only on bare metal so i'm not to sure how it good it would work, it might have a little latency or bottleneck doing GPU pass through but i'm not sure. your hardware is capable no doubt!
1
1
u/Thin-Bobcat-4738 5d ago
yes. I suggest qwen3.8-27b-uncensored, you can DL it and run it via ollama. its already had its "guardrails" removed. Basically the only thing you need is a beefy gpu with 16gb-24gb of vram. nvidia tesla v100 32gb hits the sweet spot if you can afford a $600 gpu.
1
u/MrYellowHeart 6d ago
Cool build but heads up — that PCIe adapter on the Pi is only running 1 lane instead of the 16 a normal motherboard slot gives you. So no matter how strong the GPU is, you're force-feeding it data through a straw. Great for things like mining or light inference where it loads once and just crunches, bad for LLMs where you're constantly moving data to/from the GPU. Pi's CPU is also pretty weak for everything happening around the GPU (tokenizing, serving requests, etc), so you end up bottlenecked twice over. Works as a proof of concept, just don't expect anywhere near the GPU's real performance.
1
u/Thin-Bobcat-4738 5d ago
that's not a pi. it is running on 16x lane. my only bottleneck is the thunderbolt4. this you are looking at is a eGPU dock.
1
32
u/reddit_god 11d ago
The tasks you're giving it are very simple. 35B is way overkill. A 4B model should be able to do the same thing 10x faster. The model you're using is extremely capable, but like.. 30 steps after whatever you're doing right now.
Honestly, what you're describing isn't even a job for an LLM. Python can do what you're describing, and instantly. If you really want to improve what you're doing, stop thinking about it in terms of "how can I make my LLM do what python does" and more like "how can I use python to most efficiently deliver data to my LLM".