https://www.anthropic.com/research/distillation-defenses

I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive.

As a struggling AMD Radeon owner I'd love to see AI generated open source CUDA implementation.

Not only do I dislike NVIDIA, but there's delicious irony in using CUDA cores to do it.

Anyone up for the challenge?

The newly named D-Series GPU has up to 480GB of LPDDR5X memory, 1.5TB/s of memory bandwidth

Intel rates it for up to 1.3 PetaFLOPS of FP8 performance, with support for formats ranging from FP4 to FP64. It features a total of 32 Xe3P cores, 256 XMX engines, 32 MB of unified L2 cache.

No cost estimates currently, I hope they launch a generic blower style variant at a later date for prosumer workstations.

Edit: Memory bandwidth added to the description, added a table via STH.

Introducing QwenGyre, bringing elastic RL to extreme-long-horizon (XLong) agent training.

It launches with two core designs:

- Elastic Scheduler: reallocates GPUs between rollout & training as demand shifts—live executions never interrupted. Up to 1.85× speedup over Colocate, 1.78× over Async.

- Trajectory Processor: turns branching executions (compaction, sub-agents, retries) into prefix-shared trajectory trees with bounded, deduplicated training samples. Partial progress after timeout still gets scored.

Scaled to Qwen3.8-2.4T-A95B with ~700K tokens per rollout, QwenGyre lifts NL2RepoBench from 52.5% to 58.5% in 48 steps, cutting end-to-end training time by ~44% vs Async.

🔗 Read the paper:

arxiv.org/abs/2609.33848

gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf

"MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 35B-A3B variant, with 35B-A3B being the first MoE model in the MiniCPM-V series. It builds on the family's single-image, multi-image, and video understanding capabilities, combining stronger foundation capabilities with efficient visual encoding and on-device inference. Notable features of MiniCPM-V 4.7 include:

• 📱 The First Open-Source MoE Model with Lossless, Smooth On-Device Inference. With our Lookahead Expert Scheduling (LES) technique introduced alongside MiniCPM-V 4.7, combined with fine-grained layer-wise quantization and DSpark speculative sampling integration, MiniCPM-V 4.7 35B-A3B becomes the first open-source MoE model to run smoothly on-device without compromising performance, achieving a decoding speed of 13+ tokens/s on iPhone 17 Pro Max.
• 🔥 A New Pareto Frontier in Performance and Efficiency. As model capabilities continue to advance, inference efficiency has become an important measure of a model's value. On resource-constrained devices, delivering the strongest possible capabilities within a limited resource budget calls for higher capability density. The 35B-A3B variant of MiniCPM-V 4.7 advances the performance–efficiency Pareto frontier for MLLMs. In Instruct mode, 35B-A3B achieves a higher overall score in less inference time than Qwen3.6 35B-A3B.
• 🚀 Efficient Architecture. Based on the intra-ViT early compression technique from LLaVA-UHD v4, MiniCPM-V 4.7 compresses visual tokens early within the vision encoder, reducing computation in subsequent encoder layers. It also supports mixed 4x/16x visual token compression, allowing developers to balance visual detail and inference cost across tasks.
• 💻 Broad Platform and Chip Support. MiniCPM-V 4.7 supports iOS, Android, and HarmonyOS, together with open-source on-device deployment support. It also provides day-0 support for chip platforms from Rockchip, Houmo, Intel, AMD, and Arm, extending deployment options across mobile and local computing devices.
• 🛠️ Developer Friendly. MiniCPM-V 4.7 supports inference frameworks including SGLang, vLLM, llama.cpp, and Ollama, along with fine-tuning frameworks including SWIFT and LLaMA-Factory. It provides GGUF, BNB, AWQ, and GPTQ quantized variants and adds FP8 quantization support. We also release Self-Replay Fine-Tuning, a fine-tuning approach that helps developers extend model capabilities at low cost while reducing forgetting of existing capabilities, making further customization easier."

My window will have a sign that says "we don't call tech support" with a simplified pic of a v100 on it

I know this reads like a shitpost and I'm half joking and half serious

Edit: I'd sell these on Etsy if Jensen wouldn't hit me with a C&D

Edit2: if you're from Europe pretend I said knife or blackjack or something

I saw a lot of posts here struggling to run 27B models on 8GB VRAM or complaining about phones turning into space heaters. I hit the exact same OOM wall on my phone, so I decided to build a hardware-split workaround.

I wrote a custom C++ engine (fork of llama.rn) to cluster two Android phones over a simple USB cable.

The Setup & Performance:

• Model: Ternary-Bonsai-2-27B (compressed to 2.06 bpw / 6.52 GiB). Perplexity: 14.26 (vs 14.05 reference).
• Cluster Mode (Xiaomi redme 100 pro max + ROG 7): Layers are split 4.36 GiB / 2.15 GiB over the USB cable via custom RPC. Speed: ~2.8 tok/s (bottlenecked by the slower phone, but perfectly synced).
• Solo Mode (Xiaomi 15 only): 5.6 tok/s text generation.

The Secret Sauce (Why it doesn't melt the phone):
I didn't use standard quantization. The weights are ternary (-1, 0, 1) in the Hadamard-rotated basis. Instead of un-rotating offline (which destroys the 2-bit primal), I kept the weights folded in the GGUF and wrote a patch to apply the Hadamard transform at runtime on the device's ARM CPU.
Result: Only additions/subtractions. Zero MAC operations. Max temperature sits around 30°C.

Proof & Links:
🎥 Raw Demo (2m 45s, one take):

• 🎬 https://youtube.com/shorts/CH1im2JR4yM
• 📊 Engine Logs: https://gist.github.com/figkcoci-debug/76f41c409fc5afe5a431094cc228ddc0
• 📄 Methodology: https://gist.github.com/figkcoci-debug/11871f60812d2d18de5214fda3d94160

I'm a high school student currently looking for a lab placement/grant to push this further (planning to write custom Vulkan compute shaders for the Hadamard transform to boost t/s even more).

Would love to hear what the compression and edge AI community thinks about this approach

After running a benchmark test on Exo in Opencode and being amazed by the result, I thought I'd try other models on Opencode. Many were surprisingly terrible. Especially some mid level models.

Earlier posts:
https://www.reddit.com/r/opencode/comments/1wzm1ah/exo_free_frontier_level_wow/
https://www.reddit.com/r/opencode/comments/1ww5o7y/animated_horse_space_bunny/

Today I installed Strata on https://github.com/Niko1221/Strata

I connected up my AMD R9700 32gb with 64gb RAM and used the model Qwen3.8-Flash-Next IQ3_S. Admittedly I was expecting terrible results considering it's a small Q3. It acheived 90 tk/s decode approximately 3x compare to 27b.

I tried the usual prompt from the other posts, but the output wasn't great. So I tried iterating with a few more prompts in the hope it would improve it. The major leap was when I asked it for a plan, then asked it to build it and when I said "low poly" style but with many polygons. FYI I'd done similar with other models and the result would get worse rather than better.

Anyway I thought I'd share the output.
Qwen 3.8 Flash Next IQ3 S - https://web.esuyo.com/horse-strata-qwen-3.8-q3-s-1/

Qwen 3.8 Max - https://web.esuyo.com/horse-qwen-3.8-max/

With the vLLM Semantic Router team we released Decision 3.0 (d3), our new SOTA decision model family ranging from 0.6B to 27B, Apache-2.0, all reading text, images and video.

How it works

- Built on Qwen3.5 (0.8B / 2B / 4B / 9B) and Qwen3.8-27B. The parameter counts include the vision encoder: 0.33B in the 4B, 0.46B in the 27B.

- Three decision types: choice, yes/no, score. Probability for every answer, with no text generation. Several questions about the same input are answered in one call, each from its own forward pass over the input.

- Images: multiple per request, read at up to 1.6 MP.

- Video: multiple per request (MP4, WebM, MOV, MKV), sampled at 2 fps, at most 32 frames over the whole clip, each at up to 0.2 MP.

- BF16 weights, load with Transformers (trust_remote_code=True), trained on AMD Instinct MI325X.

Results (our evaluation, official Jev 0.3 kit on the released weights)

- #1 on both boards: the 27B scores 64.7 text and 71.6 vision, ahead of every listed model, Jev's own included.

- Every size leads its class: the 0.6B, 0.8B, 4B and 9B each lead their size class. The 9B (d3-flash) beats our previous-generation 27B (Decision 2.0).

- Video zero-shot: no model saw video in training. On all 19,140 Perception Test validation questions, the 27B scores 77.4 and the 4B scores 70.3.

HF collection: https://huggingface.co/collections/vllm-sr/decision-30

Didn't realize the V100 were still so fast, anyone have first hand experience on these?

We're launching Precision Numerics by releasing Boole-1.7B, a 1-bit version of Qwen3-1.7B.

• 247 MB packed weights: 13.9× smaller than BF16.
• 85.9% of the BF16 model’s average score across our 10-task evaluation - outperforms 1-bit Bonsai-1.7B and nearly matches 1.58-bit Bonsai-1.7B.

Try it locally in your browser (it's for low-latency tasks using English and non-thinking mode), or use it with vLLM or llama.cpp.

We'll ship more capable models for the open-source community. Let us know what you what models/capabilities you want to see next.

• Blog: precisionnumerics.ai/blog/boole-1p7b
• Weights: huggingface.co/precision-numerics/Boole-1.7B
• Demo code (for vLLM, llama.cpp): github.com/PrecisionNumerics/boole-demo
• Local chat in browser: precisionnumerics.ai/chat/

Gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-gguf

"MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 1B variant. It builds on the family's single-image, multi-image, and video understanding capabilities, combining stronger foundation capabilities with efficient visual encoding and on-device inference. Notable features of MiniCPM-V 4.7 include:

• 🔥 A New Pareto Frontier in Performance and Efficiency. As model capabilities continue to advance, inference efficiency has become an important measure of a model's value. On resource-constrained devices, delivering the strongest possible capabilities within a limited resource budget calls for higher capability density. The 1B variant of MiniCPM-V 4.7 advances the performance–efficiency Pareto frontier for MLLMs. In Instruct mode, the 1B variant continues to improve foundation capabilities over MiniCPM-V 4.6 and substantially outperforms Qwen3.5 0.8B in both overall score and inference efficiency.
• 🚀 Efficient Architecture. Based on the intra-ViT early compression technique from LLaVA-UHD v4, MiniCPM-V 4.7 compresses visual tokens early within the vision encoder, reducing computation in subsequent encoder layers. It also supports mixed 4x/16x visual token compression, allowing developers to balance visual detail and inference cost across tasks.
• 💻 Broad Platform and Chip Support. MiniCPM-V 4.7 supports iOS, Android, and HarmonyOS, together with open-source on-device deployment support. It also provides day-0 support for chip platforms from Rockchip, Houmo, Intel, AMD, and Arm, extending deployment options across mobile and local computing devices.
• 🛠️ Developer Friendly. MiniCPM-V 4.7 supports inference frameworks including SGLang, vLLM, llama.cpp, and Ollama, along with fine-tuning frameworks including SWIFT and LLaMA-Factory. It provides GGUF, BNB, AWQ, and GPTQ quantized variants and adds FP8 quantization support. We also release Self-Replay Fine-Tuning, a fine-tuning approach that helps developers extend model capabilities at low cost while reducing forgetting of existing capabilities, making further customization easier."

I’m planning to get a Mac mini mainly to run local LLMs and AI agents 24/7. I plan to run models up to 27B parameters with Q4 quantization, and maybe not for coding, maybe just for everday task and model exploration.

I’m trying to figure out whether I should prioritize more RAM or a more powerful chip.

For example, I’m considering:

M4 with 32GB RAM

M4 Pro with 24GB RAM

For this kind of workload, which matters more: having more memory to run larger models, or better CPU/GPU performance for faster inference?

I’d love to hear from anyone running a similar setup!

Here're results from my final test of coding, coding (panda/numpy), data analysis, data-driven decision-making, and local RAG domains I interact with most often.

This test, which is refined from a 500+ pack, now provides a differentiating signal between models after having weeded out tests all models consistently pass.

It’s a specific test developed based on my workflow and not at all an endorsement or damning of the models for all types of use cases. Aim is to identify models for heavy spreadsheet, rounding precision, array, coding type work that I can then trust the models to make decisions and act on. Verdict is review of any work I do locally has to be reviewed by Claude or gpt.

When Qwen4 or others get released, if they do better on this, gives me real good feel for my use case. Also good canary to when I can go fully local and invest in pricey gear.

• Models on OpenRouter sometimes disclose their quantization and sometimes not.
  • or-qwen-qwen3.8-27b is stock at bf16
  • or-qwen-qwen3.8-flash-next didn't disclose its quantization. In this test, it is the model that gets truncated the most (max token per test 16,384). Even after raising the max token for it from 16k to 32k, the number of truncations due to overthink didn't disappear entirely.
• iq3 GSQ-RCO is decent performer even placed among Q4s
• mradermacher's fine-tunes continue to standout for my type of work.
• Muse Glimmer, Qwen3.6-35B-A3B etc. included for comparison.

A lot of projects have come up recently consisting of specific hardware config specialized forks of existing conferencing engines, but they all seem to be made using Claude or OpenAI's biggest frontier models.

So given that open source is about 4-6 months behind, can we reasonably expect that local runnable models will start having the ability to dissect a big repo like llama cpp and optimize it for a given set of hardware sometime starting next year?

Or maybe are models like deepseek4.1 or glm5.3 already able to do that? What about their smaller twins? v4fvision and 5.3flash? Maybe even qwen flash next is capable of doing this?

Qwen 4 is coming out end of October, anyone expect that model to be able to pull self-optimization off?

I get a lot of hate whenever I tell people I run a Q4 quant of Qwen3.8 27B with 262k KV 80t/s with vision, all while running 2 screens on windows on the same card. People seem to think I am lying or making it up. I am really just trying to help people. I am saddened whenever I see fellow 16GB VRAMers stuck on 60k context running q3 or lower quants or running weaker models because they think 27B it is out of their reach. I really just want to help them.

I get told it's impossible to fit the weights let alone the KV, or that the KV must be q1 and that I am definitely not fitting vision.

The trick is llama.cpp-adaptive-kv-streaming. I love this fork and by default it only supports q8_0/q4_0 but I modified it for my own purposes to support q4_0. But you can still do q8_0/q4_0 if you lower the context a bit or reduce the batch and ubatch to 128 or drop the vision maybe and increase kv-stream-arena-mib. Alternately ask you favorite LLM to add q4_0 support, it's just a few lines of code to a few files.

My MTP bench normally gets around 80t/s but is slower due to the video capture, which also uses more VRAM.

I did not run llama-benchy at full context in the video as it takes a few minutes, but at 262k I get around 20t/s and pp 400t/s. But the speed drop is linear and normally I compact at around 180-200k in pi anyway. But for many even those speeds are considered usable as I have seen many people satisfied with just 10-20t/s.

cache-ram you can set to whatever you like it does not matter so ignore my 20GB setting.

The only caveat is watch how many apps you have open that consume VRAM. For me a browser with a few tab, notepad++ and pi is enough. But nothing preventing you from dropping to the next lower quant if you like a little more breathing room. Windows will push active VRAM into shared VRAM when it runs out and your speeds will tank if you open too many apps.

Some will also argue that in my video my shared VRAM goes up. That's normal as it's the KV being push to system RAM which is then streamed to the kv-stream-arena (configured to 1.5GB).

For those interested I just set batch sizes to 128 and set q8_0/q4_0 KV and it slows pp by about 100t/s at high and low context on llama-benchy. Come to think of it I might just go with this KV setting myself and discard my own fork.

The fork developer does not test on windows so sometimes there are compiler/linker errors but easily fixed by pointing an LLM at it.

As for the frontend, don't worry about that, that's just my launcher I vibe coded. llama.cpp-adaptive-kv-streaming is command line just like llama.cpp

Hope this helps at least one person.

Hey everyone, I made whatmodel.sh

To use it,

1. Enter your RAM and GPU
2. See what models you can run locally with its rank and the command to run it
3. Compare them with API models on the same score

Building it made me realise a 32GB Mac can now run a model that scores on-par with GPT-5, completely locally and for free (e.g. Qwen3.8 27B at 4-bit).

Each model gets one score fitted across 19,731 published benchmark results, and every result links to its source. The method is at whatmodel.sh/method, and the data, scoring code and bench script are open source: github.com/jakemanger/whatmodel

It works from the terminal too: curl whatmodel.sh

If you'd like to contribute, there's a bench script to collect real-world tok/s and you can also submit any new models and benchmark scores along with their source on the github.

Cheapest one I see at MC is the asus pro-art version for 6499. https://www.microcenter.com/product/715662/asus-proart-p14-14-wqxga-120hz-oled-laptop

Hey everyone,

Sharing an educational tool called AskCyph Health Explorer to explore how single-letter DNA mutations lead to disease, how they alter 3D anatomy, inheritance probabilities, and available treatments.

To make it interactive while protecting user privacy, the architecture splits the workload between two models: AlphaGenome for molecular biology predictions, and MedGemma 1.5 4B running 100% locally in your browser over WebGPU as a private explainer

How the Pipeline Works

1. Predicting Molecular Impact (AlphaGenome): Instead of asking an LLM to hallucinate codon mechanics, the app connects to Google DeepMind’s AlphaGenome API to predict regulatory variant effects and splicing disruptions (e.g., contrasting a direct coding change in Sickle Cell against a non-coding Beta-Thalassemia mutation breaking RNA splicing).
2. Private Client-Side Explainer (MedGemma 1.5 4B via WebGPU): All user interaction, chat follow-ups, and multimodal inspection run entirely on your local hardware using ONNX Runtime Web. When you ask questions about hereditary risks or drop in an X-ray/histology diagram, no prompts, images, or chat tokens are sent to any external server.
3. Deterministic Grounding: Variant details, OMIM IDs, inheritance modes, and drug approvals are pulled directly from NCBI ClinVar, openFDA, and ClinicalTrials.gov and injected into MedGemma’s prompt. The local model synthesizes the verified facts and AlphaGenome scores in plain language without making up clinical details.

To make it interactive while protecting user privacy, the architecture splits the workload between two models: AlphaGenome for molecular biology predictions, and MedGemma 1.5 4B running 100% locally in your browser over WebGPU as a private explainer.

How the Pipeline Works

1. Predicting Molecular Impact (AlphaGenome): Instead of asking an LLM to hallucinate codon mechanics, the app connects to Google DeepMind’s AlphaGenome API to predict regulatory variant effects and splicing disruptions (e.g., contrasting a direct coding change in Sickle Cell against a non-coding Beta-Thalassemia mutation breaking RNA splicing).
2. Private Client-Side Explainer (MedGemma 1.5 4B via WebGPU): All user interaction, chat follow-ups, and multimodal inspection run entirely on your local hardware using ONNX Runtime Web. When you ask questions about hereditary risks or drop in an X-ray/histology diagram, no prompts, images, or chat tokens are sent to any external server.
3. Deterministic Grounding: Variant details, OMIM IDs, inheritance modes, and drug approvals are pulled directly from NCBI ClinVar, openFDA, and ClinicalTrials.gov and injected into MedGemma’s prompt. The local model synthesizes the verified facts and AlphaGenome scores in plain language without making up clinical details.

Technical Specs & Performance

• Model: MedGemma 1.5 4B (INT4 ONNX with FP16/FP32 execution fallback).
• Runtime: WebGPU via ONNX Runtime Web.
• Multimodal: Vision projection layers execute on-device to inspect uploaded clinical diagrams.
• Storage: ~2.8 GB compressed weights, cached locally for instant subsequent loads.
• Zero friction: No login required, and free.

Links & Demo

• Live Tool: AskCyph Health Explorer

Curious to hear what WebGPU throughput you get on your machines, and how others are pairing specialized foundation models with client-side SLMs for sensitive domains!

I can't find anything in the Docs. I see that the setup.sh file mainly pulls a big ass unaudited binary and, with all simpathy for the repo author, that's not going to run on my machine.

I am sure the documentation is somewhere, but I can't locate it. (Yes, I could ask an LLM to search the repo, but from time I want to do things the old way, reading myself, and asking for help to the FOSS community). Where are the docs to build strata from source?

I built a spotlight search for windows. It has an embedding model and a MoE models to go with it.

The MoE

It's 2 Qwen 3.5 4B models, I fine tuned one for coding work and another for agentic work, and then I combined them both into an MoE. I didn't want to shove it into everything so you can use it to fix code and that's about it

Embedding Model

it uses embeddinggemma 2 (270M) It currently doesn't have image support.

---

I used Opus 5.5 to write it

The model is here: https://huggingface.co/scutoid/2x4B but if you have 16GB of ram or more, the app will ask you if you want to download it and it'll download it without you needing the link

The app source is here. The exe is under releases: https://github.com/blueprintnerd/LSearch