Write-up: https://zhebrak.io/posts/brokenml/

Repo: https://github.com/zhebrak/brokenml

I built a benchmark to determine the access level agents need to successfully fix broken ML workloads (PyTorch over Gloo on a two-node Slurm cluster or a llama.cpp server running in Docker Compose). A separate verifier re-runs the job on a fresh cluster to score each solution. The faults were designed to reflect what you see in practice, including shape mismatches across ranks or hanging collectives, a straggler node, OOM errors, broken model checkpoints, scheduling issues, poor serving throughput, and silent loss divergence.

Nine models ran every task at three access levels: config and code only, plus live read-only logs, or root on the cluster. The gap between open and closed models is mostly in what an agent can do from the config files and code alone, without full cluster access. Main caveats are CPU over Gloo (not GPUs) and only one run per task per model.

I built a spotlight search for windows. It has an embedding model and a MoE models to go with it.

The MoE

It's 2 Qwen 3.5 4B models, I fine tuned one for coding work and another for agentic work, and then I combined them both into an MoE. I didn't want to shove it into everything so you can use it to fix code and that's about it

Embedding Model

it uses embeddinggemma 2 (270M) It currently doesn't have image support.

---

I used Opus 5.5 to write it

The model is here: https://huggingface.co/scutoid/2x4B but if you have 16GB of ram or more, the app will ask you if you want to download it and it'll download it without you needing the link

The app source is here. The exe is under releases: https://github.com/blueprintnerd/LSearch

A few weeks ago we posted the open-source Graphsignal profiler here: wrap vllm serve / sglang serve / whatever in graphsignal-run and it serves everything it measures as JSON, mainly so an agent can read it in a loop.

Turns out we kept opening the JSON ourselves to see what the GPU was doing. So now there is graphsignal-top:

graphsignal-run vllm serve <model> --port 8000 # one terminal
graphsignal-top # another

What you get: per-GPU util over time, memory, power, throttling; the engine's running/waiting requests, tok/s and TTFT/TPOT p50/p95; a GPU time split (kernels / graphs / memcpy / sync); and every kernel ranked by time over the last 5 s with a readable name and category, not just the mangled symbol. Tabs for CUDA graph replays, transfers and syncs.

Screenshot is Qwen3-32B on 2x RTX PRO 6000, tp 2, over PCIe. NCCL all-reduce is a quarter of GPU time and half the wall time is host sync waiting on it. You see that in one glance.

Works remotely too: graphsignal-top --host gpu-box against a profiler started with --listen-host, or port-forward over ssh. --once prints a single frame if you just want it in a log.

Repo: https://github.com/graphsignal/graphsignal

I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL.

There's been a ton of talk of model and hardware specific inference engines showing crazy performance numbers on models and hardware that have no business doing what they do. Inspired, I've decided to pull out Mr. Opus and see where I could push things, targeting the P100 and this model specifically, and found some pretty great gains. The work I'm about to outline is pretty shamelessly vibe coded. Blah blah, directing the model with my vision, blah blah. Vibe coded. Opus did the nitty gritty. I am wildly unqualified to explain specifics for what was done, only results and a general overview of the work. Tell opus to lay out what these cards are good at and what llama.cpp is doing relative to this model that wildly misuses the hardware, optimize optimize optimize. That's how it started at least. Then there was that post yesterday about a 3090 Qwen3.8 27b engine that was doing persistent megakernels and a reduced draft vocab, so we ended up incorporating those ideas.

Truthfully, I was targeting my step father's performance in decode. He has a 306012GB+3090 setup, running Ornith 1.5. Not quite the same model, not apples:apples, but at Q4_K_M it's close enough that I'm willing to call results comparable enough to get a good understanding of where things stand. He gets ~125 t/s, unknown settings. I believe he's running through ollama. Isn't sure if he has spec decode, doesn't know his prefill. That decode number is all I have to go off of, and it was my target for this hardware.

There's the semi-well-known shinbunbun p100 patches (since have been updated quite a bit so my base may not be reflective of current day performance with those patches alone) for llama.cpp that improves performance a good bit, claims +~70% on decode. This is what I started with as a baseline.

My system specs as as such:

Ryzen 7 5700x, stock clocks/boost

48GB DDR4@3200mhz

2x Tesla P100, limited to 175 watts each, one card stuck at PCIe gen3 x4, and the other at x16

Notably, sub-optimal. On a proper PCIe setup, performance may be slightly higher.

After a few rounds of improvements, this is where we got on my system as of this early afternoon:

And Perplexity and KLD numbers:

I think there is some work to be done with regards to some model quality. I don't think I'd run this as your main setup just yet, at least not without some testing and tuning. The numbers on paper look fine, but the model does feel a little bit different in practice. Hard to really pin it down. Exploratory? More squirrel brained. Tightening up the sampler seems to help a lot (previously temp 1.0, min p 0.05, top k 40. Is now temp 0.7 min p 0.25, probably too tight but it's working so that's where I'll keep it. Same top k), so the model can still be made to work well.

I do have to note, these patches are hyper focused on the P100 running this specific model. I'm sure it'll help some other models, but I can't even guarantee it'll run other models properly at all. It is entirely vibe coded, so I cannot speak to stability. It's held up in my limited testing, but I make no guarantees. My main goal with this is to get these out there for the couple dozen other people running these cards and this model, and to maybe provide a base of patches to build on for others. Also, these patches run this still very capable model fast as fuck on decade old datacenter cards that are still kinda cheap, so I think this should be available.

I'm going to be going through and making Opus do a ton of porting work if an MoE of a similar class/size comes out for Qwen4. The vast majority of the work is indeed P100 specific, and that work can be done for any model's llama.cpp implementation I'm sure.

If you want to download and read the massive readme create by Opus, who understands what it made more than I do, behold the link: https://github.com/Mrinohk/qwen36-p100-patches

It's been 4 years since c.ai first hallucination model, and yet, we're nowhere good enough at LLMs in terms of spontaneity/interesting hallucination features. Fortunately, last year brought new architecture that allow those within 10-16GB VRAM range to run something usable up to 64k Context Windows via Kobold or LLAMA.

Requirements: 10-16GB VRAM - 26-32GB RAM.

How I Evaluated:

• Speed; If tokens per second are at least 10t/s.
• Coherency: If dialogue/monologue/POV makes any internal sense asides from basic logic (robotic feeling) or behaviorism.
• JP/PT-BR/US-EN: If it can handle/interpret typos, nuance or harsh verbiage.
• Slop Patterns: From "casting long shadows" to "smells like ozone and burnt sugar" frequency.
• MOE vs FULL: If rather pick a MOE instead of a FULL model.
• Censorship; All these models have either none or too low refusal rates even at NSFL topics, worry not.
• Instruction/Engineering: I've tested WITH and WITHOUT Pandora Instructions, if a model can handle 23 intertwined rules it can handle anything/not need a prompt at all.

Side Note: I've ignored all CoT models, waste of tokens, slower, not needed, not at all. Do prompt nicely and it's the same. Just pick any quant higher than Q_4_K, size doesn't matter ( ͡° ͜ʖ ͡°).

The Catch: I've done a small knowledge test involving animes/manga/movies/music on 70B, 120B and 300B models and found out only models above 200B actually have this sort of data, even a 120B will rarely have the information about the media you're interest at. But who can even run it?

The List (Best to Worse/Why):

1. Gemma 4 26B-A4B: Qwen 35B-A3B but with better dialogue/monologue, directive, slightly smarter/precise and funnier.
2. Qwen 3.6 35B-A3B (Abliterated): Fast, accurate, and decently fun, yet may overfit previous inputs/output format/patterns without proper directive instructions, or speak gibberish/ask too many questions because of directive.
3. Gemma 4 Instruct 19B-A1B Heretic (Abliterated): 26B-A4B but a bit more rigid, plain, yet keeps accuracy and logic.
4. The Omega Directive 14B: Follows instructions, has low slop pattern rate and writes nicely, yet might become predicable after some context fill.
5. GLM-4.7-Flash 31B-A3B (Abliterated): Performs similar to Qwen 35B-A3B but more straightforward, less interpretative of nuance/verbiage.
6. Qwen3 30B A1.5B 64K High Speed NEO MAX: Basically 35-A3B with increased reasoning.
7. MN CaptainErisNebula Chimera Heretic Uncensored 12B (Abliterated): Its weird, a bit rushy trying to fill everything at once, full of patterns, but its the most proactive of them all, unlike to get stuck in a loop, has good dialogue/monologue yet ain't suitable for Q&A roles.
8. Mistral Nemo Instruct 2407 HERETIC HI Claude Opus 12B: It's decent, slop seems to have been rephrased properly, yet may overfit impersonation POV, or try to rizz you up mostly, not suitable for Q&A roles.
9. Gemma 3 12b GLM-4.7 Flash INSTRUCT Hybrid Heretic (Uncensored): Interestingly impersonative, accurate, yet may overfit previous inputs' formatting/style with frequent slop patterns or lack humour/too prim.
10. Ministral Instruct 2512 Absolute Heresy 14B: Mistral sucks at following complex rules (regex, status panel, commands, lorebook, etc), or turn {{char}} into a rizzler, despite being the best one regarding dialogue/monologue/spontaneity, ironically.
11. GLM 4.6V Flash Hybrid ?B: UD but with decreased slop frequency, even less refusals.
12. GLM 4.6V Flash UD 9B: Can feel robotic/repetitive depending on the personality of your {{char}} as making too many questions, despite high accuracy at following all PI 23 rules + regex commands.
13. Gemma 4 E4B UD: For the desperate, it's tiny, fast, accurate and smart, but will prolly repeat itself/have slop patterns.
14. LFM2 12B A1B SpeedDemon The Deckard II HERETIC (Uncensored): Lighting fast, highly accurate and smart, but lacks sauce.

Overall Review: GLM tends to be pessimist and too logical, Qwen tends to be too inquisitive and fancy, Gemma tends to be funny and creative, Mistral tends to be flirty and oddly specific, Llama tends to be sloppy and predicable, this happens across all of their finetunes regardless if Omega, Heretic, Deckard, Dark, Horror, RP, Style, etc.

Settings: Kobold can't load these, use LLAMA for QWEN 35B (20,6GB) and GLM 31B (17,7GB). Other models all running at fastest speed possible with 40-64k - FlashAttention/FastFowarding/SWA/SmartCache + MMPROJ + TTS if any in Kobold.

Models I'm aware but discarded after trial cuz either too slow (massive), poor or have better alternatives: Mistral 24B, GPT 20B OSS, Kimi K2, Phi-4, Skyfall, DeepSeek, Cydonia, Magistry, Magnus Cydons, etc.

Not Tested: Goetia 26B-A4B v1.7, Gemma 4 26B-A4B StyleTune V2, Promethean Dawn 26B-A4B and Fenrir-X 26B-A4B.

What if I have 24GB VRAM (e,g, RTX 3090)? If going lower try Omega Darker Gaslight The Final Forgotten Fever Dream 24B, Qwen3 24B-A4B Freedom HQ Thinking Heretic NeoMAX-D_AU, Cydonia 24B, Magistry 24B, Magnus Cydons 24B, Erebus 12B Instruct 2608, Mistral RP 24B, DeepSeek 33B, Velvet Eclipse 2x12B or Velvet Eclipse 4x12B.

END

I built a GPU compiler and wrote an article about it, focusing on concepts shared by modern compilers like TVM, Modular, and PyTorch Inductor. The compiler works with vLLM for full end-to-end model serving, so it may be useful if you're interested in building LLM serving engines.

The article breaks CUDA kernel generation into a sequence of passes and shows how a common operation (RMSNorm, Matmul, Attention) is progressively lowered to generate a CUDA kernel. Each pass introduces its own IR that is optimized for specific tasks like kernel fusion or schedule enumeration.

Each example is reproducible, so you can write PyTorch code and generate code at a specific IR level, which makes the flow easier to understand.

% emmy compile -c 'nn.Linear(3, 2, bias=False)(torch.randn(4, 3))' --ir loop 
for a0 in 0..4         
  for a1 in 0..2             
    for a2 in 0..3                 
      in0 = load input[a0, a2]
      in1 = load linear_wt[a2, a1]
      v0 = multiply(in0, in1)
      acc0 <- add(acc0, v0)
   linear[a0, a1] = acc0

On the benchmarked workloads, many outperform Torch Inductor and cuBLAS. Generated kernels use modern features like tensor cores and TMA.

The field is very dynamic, with many competing approaches, so it's hard to pin down the "core concepts"; this is my take on them.

YouTube: https://youtu.be/KK6TWf4Pp6A?si=0hCuOMEan4mauJjN

Article: https://riftstack.ai/research/inside-a-gpu-compiler-from-pytorch-to-cuda

GitHub: https://github.com/cloudrift-ai/emmy

Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, including ancient hardware.

I originally built it because Open WebUI kept feeling heavier than the models I was running.

One of the main criticisms I got from you was the lack of Agents, MCP and RAG.

Well, that's changed now. All three are integrated.

I put a lot of effort into making the agentic system as token-efficient as possible. Thanks to several optimizations, it can work smoothly with large files even when using smaller context windows.

There's also an integrated RAG system, which works great with embedding models like Qwen3 (setup instructions included).

And a couple of extras:

• A standalone alternative to NPX Supergateway.
• Image editing through ComfyUI, with automatic binding detection for all major image models.

All of this without turning LLC into the heavyweight application I originally wanted to get away from.

I'd love to hear what you think, especially if you've tried earlier versions!

Download:
https://www.locallightai.com/llc

GitHub:
https://github.com/srware-net/LocalLightChat

Dual A100X PCIe (link at Gen4 x8, weird DPU cards) in a 2U EPYC 7402P box, 256GB DDR4.

Proxmox with the GPU's passed through to debian 13, 610.43.02 drivers.

llama.cpp, qwen3.8 flash next UD-Q4_K_XL, Q8_0 MTP head, draft max 2, mmproj enabled.

-sm layer -fa on --parallel 7 --ctx-size 1835008 -ub 2048 --cache-ram 32768

(7 slots x 262k), q8_0 K/V

Real world single user performance sits around 50-70tk/s. When bumping things up to 2-3 users/sessions, generation drops like a rock, 10-15tk/s. Seeking advice on how to better serve my team at work. We need models worth running and performance that doesn't have us waiting around.

I'm very partial towards MoE models, so no 27B. Should we swap over to vLLM, or run an instance of qwen3.6-35b-a3b on each card?

Use case is long session agentic 'coding' (more running linux commands, probing hardware devices). I'm frustrated and not sure how to squeeze more performance out of these. Thanks in advance!

Disclaimer: I work for NuMind, the company behind this plugin.

TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-)

I've been working on making it easier to obtain embeddings from a LLM (preferably multi-modal) that is already being served by vLLM. The idea is simple (and not mine as we'll discuss): the model already computes internal representations (1 hidden state per decoder layer) of your prompt to predict the first output token. We expose the last hidden state vector through the chat API (compatible with the openai-api and openai client even though it's not standard)

There are some interesting papers on this: PromptEOL, PromptReps and E5-V. To briefly summarize the E5-V work (which is the paper I based most of my work on): They use this prompt

Summary above image in one word:
And use the last hidden state (the hidden state of the last decoder block) as an embedding. And it works quite well. The intuition is that asking for a one token/word summary encourages the model to concentrate relevant information at that position.

At NuMind, we build NuExtract3, an open-weight 4B model under Apache-2.0, for extracting structured data and converting an image/document into Markdown. We also have a NuExtract platform and API for using document extraction with a much larger (pro) model. (see my NuExtract 3 anouncement post for more detailed information). Since our model mainly outputs json (except in the Markdown mode), the prompt we use is

{"short description": "string"}

This plug-in could be useful for local RAG, semantic search, clustering documents or images, or training a small classifier on top of the vectors.

Of course, a hidden state isn't automatically a good embedding. Quality depends on the model, prompt, and task, and a dedicated embedding model may be faster or better. The plugin just makes these experiments easier (and faster than the manual PyTorch + transformers set-up we used to determine these are viable embeddings)

I'd really like to get this merged into vLLM: PR #57185. If you have any review/opinion on my code, know anyone that I could talk to to get it merged, it would be wonderful. I know it's kind of an edge case but I would still say it's useful.

In the meantime, we have an Apache-2.0 plugin for vLLM. You opt in on a one-token, non-streaming chat request and get the vector in the response. Ordinary chat and streaming still work on the same server. The README has examples and the current support/validation limits.

Would there be interest in me porting this workflow to llama.cpp? I know most people here use llama.cpp rather than vLLM so I might try to push a PR there too. No idea whether it's easier to have code accepted in vLLM or in llama.cpp.

If you have any feedback, please share :-)Disclaimer: I work for NuMind, the company behind this plugin.TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-)
I've been working on making it easier to obtain embeddings from a LLM (preferably multi-modal) that is already being served by vLLM. The idea is simple (and not mine as we'll discuss): the model already computes internal representations (1 hidden state per decoder layer) of your prompt to predict the first output token. We expose the last hidden state vector through the chat API (compatible with the openai-api and openai client even though it's not standard)

There are some interesting papers on this: PromptEOL, PromptReps and E5-V. To briefly summarize the E5-V work (which is the paper I based most of my work on): They use this prompt

Summary above image in one word

And use the last hidden state (the hidden state of the last decoder block) as an embedding. It works fairly well. The intuition is that asking for a one token/word summary encourages the model to concentrate relevant information at that position.

At NuMind, we build NuExtract3, an open-weight 4B model under Apache-2.0, for extracting structured data and converting an image/document into Markdown. We also have the NuExtract platform and API for using document extraction with a much larger (pro) model. (see my NuExtract 3 anouncement post for more detailed information).

Since our model mainly outputs json (except in the Markdown mode), the prompt we use is

{"short description": "string"}

This plug-in could be useful for local RAG, semantic search, clustering documents or images, or training a small classifier on top of the vectors. Quality will depend on the model, prompt, and task, and a dedicated embedding model may be faster or better. The plugin just makes these experiments easier (and faster than the manual PyTorch + transformers set-up we used to determine these are viable embeddings)

I'd really like to get this merged into vLLM: PR #57185. If you have any review/opinion on my code, know anyone that I could talk to to get it merged, it would be wonderful. I know it's kind of an edge case but I would still say it's useful.

In the meantime, we have an Apache-2.0 plugin for vLLM. There is a extra argument to add, max_tokens=1 and the embeddings will be in the additional response arguments. Ordinary chat and streaming still work on the same server. The README has examples and the current support/validation limits.

Would there be interest in me porting this workflow to llama.cpp? I know that's what most people here use so I might try to push a PR there too. No idea whether it's easier to have code accepted in vLLM or in llama.cpp (seems to be quite long for vLLM).

If you have any feedback, please share :-)

1. I originally installed llama cpp via their install script on their page; it used Vulkan by default. The GPU (NVIDIA) would "disappear" and I'd have to restart the machine.
2. I re-built llama cpp for CUDA, and had to remove that original install, but the llama cpp built with/for CUDA is much more stable.
3. If I also get an AMD GPU and want to have both in my PC, do I also have to build a separate llama cpp "instance" for HIP usage?

I'm not planning on using the same instance of llama cpp with the NVIDIA and AMD card at the same time (not trying to tensor split or parallelize the same LLM across those GPUs, or whatever).

I might have one model running on one GPU, a second model running on the second GPU, for different purposes, not always simultaneously, so in that sense, would it make sense to keep the current version of llama for CUDA and build a second version of it that I could alias as llama-hip (different ports, etc)?

If any of you have done this, I'd love to read your thoughts.

If it does make sense to split a model in a 1:2 ratio (16GB CUDA:32GB HIP), I'm happy to experiment, but that wasn't my intention.

Initial LLM chat shows that I could try something like this:

```sh Do you need a separate binary for HIP?

Yes — a HIP build is a distinct build. HIP is not "added at runtime" to your CUDA binary; it has to be compiled in with -DGGML_HIP=ON, using the ROCm toolchain. The good news: the two builds can coexist happily. Just use separate build directories:

your existing NVIDIA build stays as-is

cmake -B build-cuda -DGGML_CUDA=ON cmake --build build-cuda --config Release -j

a sibling HIP build

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \ cmake -B build-hip -DGGML_HIP=ON -DGPU_TARGETS=gfxXXXX -DCMAKE_BUILD_TYPE=Release cmake --build build-hip --config Release -j ```

The rest was basically warnings about why trying to use the same instance of llama cpp (built for both CUDA and HIP) could be problematic, ending with:

sh What I'd recommend 1. Keep your CUDA build for the 5060 Ti. Add a separate HIP build for the Radeon. Two binaries, two build dirs. 2. Run two server instances (different ports), one per card. Best stability, best per-card performance, and you use both GPUs simultaneously. 3. Treat "one model spread across CUDA+HIP" as experimental — try it if curious (build-both, -sm layer -ts 16,32, --list-devices), but don't build your workflow on it. If a single model genuinely needs >32GB and won't fit on the Radeon alone, that's the case where a mixed build is the only option short of Vulkan — and it may not deliver the memory benefit due to the duplication bug.

EXL3-QLoRA
I've built this over the last few months and I think it's stable enough to share now. A fork of ExLlamaV3, this is a native ExLlamaV3 training implementation with plain PyTorch, without transformers.

exl3-qlora on github

Why?
With it, you can train 2-8bpw exl3 quantized models, rather than the old BNB 4 bit lock-in. EXL3 models are currently the highest precision models for their size, which leads to lower loss. Adapters are saved in standard PEFT format and load in TabbyAPI, PEFT, and can be merged back to the base model.

Model Support
Llama3, Mistral Small/Medium, Qwen3 dense and MoE, Qwen3.5/3.6/3.8 hybrid models, Gemma3/4 including MoE, Arcee Trinity. Each family is checked against the inference forward before training. Want more? I'm open to requests.

Features:

• SFT, DPO, KTO
• rsLoRA, PiSSA
• Optional LoRA on routed MoE experts, embedding, and LM head
• Single GPU, multi-GPU layer split, and DDP
• Sample packing, chat rendering through jinja, including tool calls and reasoning
• Gradient checkpointing, activation offload, chunked cross entropy, 8-bit and paged optimizers, Liger kernels
• Live run monitor in the browser (wandb at home)
• Real-time training, where the loaded model serves chat and trains its adapter in place

Benchmarks:

Qwen3.5-4B

All on the same dataset and hardware. bnb is about twice as fast, but exl3 is more flexible, vram efficient, and produces lower loss in my tests.

This is an unofficial community project, and a work in progress. Any feedback or feature requests are welcome, you can post them here, the github repo, or join us in the ExLlama Discord, where we have a thread in the #community-projects channel.

I have already produced several successful models with this training pipeline and released them, you can test them here.

I wanted to track grocery spending per item without sending my receipts to anyone, so I built an Android app around a small vision model that reads a photo of a receipt into JSON: store, date, total, every line item. (iOS will take a couple more days to be checked by the Apple team.)

Also, most apps only allow broad categories like “Food” or whatever, but I wanted to see a more detailed breakdown, e.g. 150 euros for coffee...

I fine-tuned Qwen3.5 0.8B and OvisOCR2 (which is itself a Qwen3.5 0.8B finetune) on receipt data.

I wanted to publish this as a personal project but chose to go through my employer (less hassle).

The app is free and will stay free. If you want to contribute to the project, you can donate your receipt images for better models.

There is also an option to connect to your own OpenAI API-compatible server if on-device processing is too memory-intensive for you. (The models used by the app will be published as open source (to use with your own Server) )

I would appreciate feedback from different devices !

Playstore Link

What is 'Small', what is 'Large'? We're still over the place, years later we still can't reach consensus, causing endless confusion, so in infinite wisdom and generosity I took it upon myself to fix it.
I've arbitrary decided from this day forth we shall all follow this table:

Tiny - fits in 2GB (v)RAM
Small - fits in 8GB (v)RAM
Medium - fits in 32GB (v)RAM
Large - fits in 128GB (v)RAM
Huge - fits in 512GB (v)RAM

Anything larger than 'Huge' is beyond my comprehension and beyond my financial means for at least the next 4 lifetimes, so I'll leave that for the future generations to fight over.
You're welcome!

Inspired by DigUp for macOS, VectorDash aims to be a full Windows 11 search replacement, powered by EmbeddingGemma 2.

• Full indexing: search by meaning or exact words across photos, screenshots, audio, video, PDFs and Office documents. Find “images of dogs,” a podcast discussion, or a specific PDF clause—with page and timestamp matches. Supports multilingual and optional code search. This uses EmbeddingGemma 2.
• Quick indexing: rapidly index filenames and folders across entire drives, without EmbeddingGemma 2. Combine both modes and filter by source or file type.
• Universal launcher: find and launch Windows apps, open Windows settings, and calculate math directly in search.
• Quick search: customizable global hotkey, thumbnails, previews, and familiar file actions.
• Flexible setup: one-click model/tool setup, GPU, iGPU or CPU inference, and optional remote llama-server.
• Background indexing: scheduled updates, CPU/GPU idle detection, pause/cancel controls, and exclusions for Windows cache folders.
• Agent access: optional read-only MCP server with LAN access
• Local storage: disk-backed SQLite, encrypted by default

Around 200 MB of app RAM in testing with roughly 3,500 fully indexed files and a million quick-indexed files. Options between low-memory, balanced, and performance for search performance. See performance measurements.

Models and processing helpers use additional memory and resources.

File support · Security details

Yep, vibe-coded this. Tested extensively, and I'm just happy I never have to use Windows search again! Feel free to send me bug reports (especially for different GPUs and CPUs) or feature suggestions!

Write-up, serve command and raw data: https://github.com/lukolszewski/vllm

This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users in your VRAM without the performace hit of fp8. Speeds in comparison to my earlier llama-multigpu performance fixes https://www.reddit.com/r/LocalLLaMA/comments/1x023z4/qwen38flashnext_on_6x3090_6x4090_without_nvlink/

Upstream vLLM does not start this model on that machine at all: the model refuses --pipeline-parallel-size > 1, TP has to be 1/2/4/8 so six cards are useless for it, and a 24 GB card cannot hold it at TP=1. Putting the 102 GB n-gram (PLE) table on SSD has just showed up as PR on upstream today (keeping it in RAM is the typical setup), and fp8 KV crashes on Ampere.

Mashine is a consumer PC, no epyc/threadripper, no P2P, still pretty usable (6x3090, 7950X, 187 GB RAM, four cards behind Thunderbolt), GPTQ-INT4 checkpoint (nvfp4 was slower), one streaming request per cell, tokens/s:

Things that did not pan out: NVFP4 checkpoints load with no changes but are 10-12 % slower than GPTQ on these cards, both go through Marlin weight-only, there is nothing for FP4 to accelerate on Ampere. And per-user decode under pipeline parallelism falls with the user count (57 -> 44 -> 38 -> 29 at 1/3/5/7 users) while llama.cpp's grouped decode stays nearly flat (34 -> 30 at 5); the sums are similar at five users. So: vLLM for prefill, single-user speed and seat count; llama.cpp if you want every user to see the same speed (I posted a similar set of fixes for llama earlier).

The AMD side exists too: u/alanoo81 ran the model at PP=3 on Radeon V620s earlier and that is where the PP idea came from; on current upstream only the guard removal is still needed and the branch keeps a partition check instead. If you are on AMD start there.

Caveats: one model family, NVIDIA only, tested on exactly one machine (plus kernel unit tests on a 5060 Ti). Batched outputs are not byte-identical to solo runs, vLLM's usual batch nondeterminism. Load to healthy is 4-5 minutes. As with the llama.cpp work, the code was written with an LLM and validated by measurement and output checks (needle tests, kernel outputs against oracles), not by review; each change is one commit, take what you need. This is a fixing branch, not a fork I intend to maintain.

tl;dr: Are there other models I should try? Is there something better than Kimi K3 for a code review? Is GLM-5.3 better at finding/patching security vulnerabilities? Are skills like https://github.com/alibaba/open-code-review worth using? Any other things I should be doing to help mitigate the monstrosity of the codebase I'm slowly growing?

I've programmed an app that helps me manage a bunch of things, let's say file management with some extras for calls to local models that do stuff on the files, doesn't really matter. It's nothing very complicated in principle, just a database that keeps track of things, and loads of files with tags, descriptions, dates, etc, etc. Currently, excluding comments and docs, it's about 40k lines of code (DB, the whole CLI pipeline with a bunch of commands, a pretty complicated GUI...).

It's not public, and it may never be, I just want it to streamline my work. It also doesn't manage any critical data or anything particularly sensitive. Because of that, I decided early on that I want to see how it goes if I don't ever look at the code. I've mostly been using DeepSeek (different versions over the last 3 or 4 months) through their API. Sometimes, I'll get Qwen3.8 Flash Next (which I'm able to run at Q5 with quite slow performance and 262k context on 2x5080s and 128GB DDR5), and ask it to do a code review. Every once in a while, when I really want to clean things up, I drop 10-20€ and ask Kimi K3 to do an audit to the whole codebase. In all of these cases, I use them in Hermes as orchestrators and use Qwen3.8 27B for sub-agents.

Wether it's Qwen or K3, when I do a code review, I tell them to find bugs, security vulnerabilities, and improve the codebase from a design/efficiency point of view, specifying to use SOLID, DRY, KISS, YAGNI, etc principles. So far, the app is working fantastically and I find that the current Deepseek Flash is perfectly capable of solving any problems I find in the GUI during use.

I'm wondering, is there anything I should be doing differently?

Are there other models I should try? Is there something better than Kimi K3 for a code review? Is GLM-5.3 better at finding/patching security vulnerabilities? Are skills like https://github.com/alibaba/open-code-review worth using? Any other things I should be doing to help mitigate the monstrosity of the codebase I'm slowly growing?

We’re building a phone voice agent for Algeria. Current models don’t handle Algerian Darija well, so we want to fine-tune our own. We have a lot of real call recordings, but we’ve never trained a speech model before.

Looking for guidance on 4 things:

1. Which model to pick? For speech-to-text and text-to-speech, with a commercial license.
2. How to train it? Fine-tune vs LoRA, what hardware, any good guides or notebooks.
3. What data do we need? How many hours, what format, do transcripts need to be perfect?
4. How to handle different dialects? Algerian Darija isn’t one single accent. It changes a lot between regions: the east (near Tunisia), the west (near Morocco), the capital, and the south all sound different, with different words and pronunciation. Many people also mix in French, and some regions mix in Berber (Amazigh) words. How do we make sure the model understands and speaks all of these? Do we need balanced data from each region, or is one big mixed dataset enough?

I upgraded to OpenCode 2.0.26 today, and even with competent bots it seems to hang up where the bot will be like, "Great, I'll implement that." And I have to prompt it with, "Yes, go ahead." as if it has no agenticness.

Bot: "Fixing the test:" <stops>
Me "T-t-t-today, junior!"

I stumbled across this as I was currently having glm5.3 flash run only around 10tok/s basically unusable. With Maya i am running 30 tok/s now. GLM5.3 feels even with this low quant like a much more enjoyable model than qwen 3.8 flash next so far. Still testing .

This is very exciting allowing many to run GLM5.3 flash.

https://github.com/mw00/project-maya

Project Maya

A 321-billion-parameter AI model on your own GPU
GLM-5.3-Flash, private and fast · one NVIDIA GPU or up to 16 · AMD and Windows (experimental) · chat, pictures, OpenAI- and Anthropic-compatible API

Models this size usually need a datacenter. Maya runs GLM-5.3-Flash (zai-org, MIT license) on the hardware you already have: 321 B parameters, about 18 B active per token, and a context of up to 1 M tokens. Its engine is built for this one model. It keeps the experts your conversation uses most on the GPU, the next ones in RAM and the rest on your NVMe SSD, and moves them as you chat. It measures your GPUs, CPU, RAM and SSD and tunes itself to them. Nothing leaves your machine.

• Fast: up to 118 tokens/s on four RTX 4090s, 30+ on a single RTX 5090. Real numbers from real users below.
• Close to the original: Maya's own quants keep 97.7-99.2% of the FP8 model's zero-shot accuracy.
• Plugs into your tools: OpenAI- and Anthropic-compatible API with tool calls, ready for coding agents.
• Everything in the box: a chat and a live monitor in the browser, pictures, thinking levels, a one-command setup.

It's a tiny 6M parameter text to image model, and it can make recognizable images at 64x64.

The 6M accounts for everything, there's no oversized text encoder or VAE.

Check out the model card for samples, a detailed explanation about it's architecture and training, and a quick start!

Why do we need SKILLS.md? Isn't it better for LLMs to be trained with skills - instead of each installation having their own skills directory?

Title.

Not sure what is considered to be a well-respected/reputable test suite for benchmarking the abilities of local tiny models. Figured people here would know more about it.

I was running Qwen3.8-Flash-Next (base and later Swift-1.5 NVFP4) on a paid pod seat for about a month, and it started answering with short strings that echoed its own earlier reasoning. I assumed the context was getting too big for it. But it wasn't.

Across almost 13,000 turns the echo appeared 266 times, and the more copies of it already in context, the likelier the next turn produced it. The culprit was `preserve_thinking: true` feeding prior reasoning back into every prompt. Removing the echo removed the problem.

Once I trusted Flash-Next again, I decided to check out the latest buzz of Flash-Next on Strata but had to see if it would work on AMD. 176B total parameters, 6B active per token, 66.4 GB of weights on a 23.8 GiB resident in VRAM, 31.6 GB of expert weights and a 28.8 GB PLE table in system RAM.

Results:

Decode is 105–160 tok/s. Context is 500,000, and I measured it actually holding 506,849 tokens.

Quality held between 81% and 84% with thinking off and 88–94% with it on across 4k, 64k and 256k, and nothing truncated.

Decode fell 18% over that range — slower at depth, not worse. An incredible improvement over my llama.cpp and Swift 1.5 27b dense.

Set to 500k context in pi and now my daily driver. This has been another huge leap forward for me locally.

Full writeup https://t0mj.github.io/holler/posts/flashnext-q2-strata-amd-daily-driver/

I plan to do more writeups about my experiences with flash-next and strata on AMD for local development on a gaming rig.

Disclaimer: My writing is a mix of hand written with ai guidance, pushback, probing, and evidence gathering. (Yes, I wrote this statement.)