Building strata from source

[u/Expensive-Paint-9490]

I can't find anything in the Docs. I see that the setup.sh file mainly pulls a big ass unaudited binary and, with all simpathy for the repo author, that's not going to run on my machine.

I am sure the documentation is somewhere, but I can't locate it. (Yes, I could ask an LLM to search the repo, but from time I want to do things the old way, reading myself, and asking for help to the FOSS community). Where are the docs to build strata from source?

[u/ketosoy]

Glad you’re mentioning this.  It makes me uncomfortable that “wget this *.sh file and pipe it to bash” is becoming the default installation path

[u/La_doc → u/ketosoy]

Exactly this. I don't understand why this is so prevalent especially among AI-related tools. I can live with the neckbeardish "here is the necessary documentation, use cmake yourself" approach, I would prefer a system package or a docker container. A Flatpak or AppImage is also acceptable. But telling people to wget/curl download a script and pipe that right into bash??? That's psychopath behavior.

[u/Cautious_Chicken_604 → u/La_doc]

People are lazy AF, that's why. It's not any more complicated than that.

[u/La_doc → u/Cautious_Chicken_604]

Well, now if we had some kind of automatic tool to aid us in integrating a well documented dockerfile and/or docker-compose stack for example into our public repository... But alas, this kind of automatic helper sadly doesn't exist, since it would need a certain degree of AGEN(T)cy.

/s

[u/draconic_tongue → u/La_doc]

MOST people don't want to fuck around with docker

[u/Reasonable-Height704 → u/La_doc]

Blame JavaScript and node, that community started it years ago.

[u/ChristopherRoberto → u/Reasonable-Height704]

They brought it back, really. Shar archives were a plague in the '90s and caused various disasters that have been forgotten as they were in the before-times, but they'd finally been stomped out.

[u/remielowik → u/ketosoy]

Why, it's a great way to get stuff installed on target pcs as an attacker. Why use a system like a package manager, snaps or even docker containers if you can also have your target simply give you the keys to his system for free.

[u/FastHotEmu → u/remielowik]

AaaS aka ass: attack as a service

[u/-p-e-w- → u/remielowik]

Because there is no such thing as a cross platform package manager that people actually have installed.

[u/NortySpock → u/ketosoy]

"UnbiasedBob" on YouTube had the same concerns and forked it to clean up the installation process.

I have not tested it but it's on the list of things to try

https://github.com/robertkarl/UnbiasedStrata

[u/saltyourhash → u/ketosoy]

https://nocurlbash.com/ is a resource all should read

[u/duyntnet]

This is what i run, works for me (help from Gemini, Windows):

cmake -G "Visual Studio 17 2022" -A x64 -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_RUNTIME_LIBRARY=Shared && cmake --build build --config Release --target strata -j 10

Or you can provide LLM with CMakeLists.txt and ask it for help with build command.

[u/Gold_Coconut9777]

I just audited the setup.sh installation flow both myself and with AI.

TLDR: As long as you pass --build argument to setup.sh:

./setup.sh --build

basically everything is built locally. Except for llama.cpp pulled from GitHub, Python dependencies pulled from PyPi, and the model itself pulled from HuggingFace

To pin down the PyPi dependencies you can also do (with venv activated)

pip download -r requirements.txt -d /srv/strata/wheels

And then build like this:

PIP_NO_INDEX=1 PIP_FIND_LINKS=/srv/strata/wheels ./setup.sh --build

If you are super paranoid and want to also handle the llama.cpp dependency, you can find the correct version to download like this:

grep LLAMA_CPP_COMMIT setup.py

Download it manually, rename to

llama.cpp-<7-char-commit>.zip

and put the zip file in

third_party/llama.cpp/

You could also download GGUF files and pass the directory with --gguf-dir argument. So the command becomes

./setup.sh --build --gguf-dir <your_directory>

I don't think you can do more than this. And obviously you will have to manually keep track of all the dependencies this way (Python, Llama, HuggingFace).

[u/jcy → u/Gold_Coconut9777]

banger post

[u/Agile_Discussion4164]

Related question: Is there a consolidated list / documentation of all the available parameters ?

- setup.sh --help doesn't give much

• there is /docs but everything is scattered in there and does not seem to have all the opt-in parameters / env vars. Morever those docs seem almost 100% IA generated and are not exactly didactic.
  
• there is an insane amount of code adding tens of new options merged each day.

- pointing an agent at the git and asking to optimize may work to a certain degree, but it takes time (and gets really messy if you want your local IA to do it itself ^^)

Thinkering with the parameters I managed to get significantly better perf than the automatic mode (older ddr4 server, multiple but varied gpus,...), but I'm wondering what i'm still missing.

This software is insanely usefull, but the growth speed of this project does not seem sustainable to me ...

[u/Expensive-Paint-9490 → u/Agile_Discussion4164]

When I read "this software was built with AI assistance" I assume the vast majority of it wasn't ever reviewed, or read at all. The projects growing too fast are becoming the norm. The thing I don't get is why people use agents to write code but seem unable to make them generate docs.

[u/epictunasandwich → u/Agile_Discussion4164]

what did you find, i have a similar setup :D

[u/Agile_Discussion4164 → u/epictunasandwich]

automatic mode was failling at distributing the data accross my GPUs (now 16,16,16,8 GB respectively, the first also having 2x the compute compared to others.).
forcing the order with

"gpu": [ 0, 2, 3, 1 ],

and the split with

"layer_split": "14,28,42",

gave me a large boost (~60tok/s gen -> 90tok/s)

also using

"args": [ "--trim-stage-weights", ... ]

I recently started using concurrent slots. It does not speed things much in my case (i'm probably CPU bound on decode with my 2x xeons 6240 ...), but it makes working with agents (mostly hermes) smoother

"parallel": 4

And as I have spare ram, I use it to cache conversations (helps tremendlously with agents)

"--conversation-cache-mib", "20000",

"--conversation-cache-slots", "10"

(there is an option to not cache very small conversations that I did not try yet ... probably usefull to not constantly evict larger conversations with very small agent rounds)

I have yet to tune the vision aspect ... my workflows often include a lot of "advanced" ocr (handwritten and shematics that dedicated tools like tessaract cannot do reliabily) and I may dedicate one of the GPU to this task

[u/mycocoon]

Take a look at the Dockerfile

[u/x_ci]

Check out this YT Video and you're 100% right to question this.

[u/ionsago → u/x_ci]

To save y'all some time watching this great video

https://github.com/robertkarl/UnbiasedStrata

followed with make build

[u/Osi32]

I haven’t messed with strata yet, but this is good to know. I just assumed strata’s GitHub was source code and could be built. Am I mistaken?

[u/miversen33]

Ya I was having issues with this too. It's also a pretty annoying mess trying to create an AMD dockerfile around strata.

If the maintainer of this project is here in this sub, please spend a bit of time cleaning up the docs. I get that the speed this project blew up is really hard to manage, but the documentation, frankly, sucks.

[u/Strix-Halo-9004]

on 0.14.2
I still can't remember how to install it on fedora 44, but qwen 3.6 35B local can handle it. you just make it to read manual and .py to decode it.
and it's very strange, because my local ai don't use setup.sh. it recommended me to install with his new bash shell on pi.dev.
because I want to use mradermacher Qwen3.8-Flash-Next-Uncensored.i1-IQ4_XS.gguf with B16 mtp from hf and convert to q2 mtp (btw from my benchmark It's accurate better than unsloth UD IQ4_XS, may be from imatrix quant )

after that my local ai recommend me to use local vocab to create custom mtp. very fun and very strange.

I confirmed mradermacher Qwen3.8-Flash-Next-Uncensored.i1-IQ4_XS.gguf + custom mtp q2 . it can run on strix halo 128GB

[strata] done: 671 tokens in 19 s (40.0 tok/s) (stop, cancel=False)
[strata] reading the prompt: 47,288 of 47,293 tokens, 2 s so far
[strata] thinking: 40 of max 4096 tokens, 49.2 tok/s, 3 s
[strata] thinking: 80 of max 4096 tokens, 43.9 tok/s, 4 s
[strata] thinking: 134 of max 4096 tokens, 46.5 tok/s, 5 s
[strata] thinking: 183 of max 4096 tokens, 46.2 tok/s, 6 s
[strata] thinking: 221 of max 4096 tokens, 44.3 tok/s, 7 s
[strata] thinking: 262 of max 4096 tokens, 43.4 tok/s, 8 s
[strata] thinking: 296 of max 4096 tokens, 41.6 tok/s, 9 s
[strata] thinking: 334 of max 4096 tokens, 41.0 tok/s, 10 s
[strata] thinking: 367 of max 4096 tokens, 40.1 tok/s, 12 s
[strata] thinking: 399 of max 4096 tokens, 39.2 tok/s, 13 s
[strata] thinking: 438 of max 4096 tokens, 39.1 tok/s, 14 s
[strata] answering: 473 of max 4096 tokens, 38.7 tok/s, 15 s
[strata] answering: 505 of max 4096 tokens, 37.9 tok/s, 16 s
[strata] writing a tool call: bash: 550 of max 4096 tokens, 38.3 tok/s, 17 s
[strata] writing a tool call: bash: 602 of max 4096 tokens, 39.2 tok/s, 18 s
[strata] writing a tool call: bash: 654 of max 4096 tokens, 39.9 tok/s, 19 s
[strata] writing a tool call: bash: 702 of max 4096 tokens, 40.2 tok/s, 20 s
[strata] writing a tool call: bash: 762 of max 4096 tokens, 41.2 tok/s, 21 s
[strata] writing a tool call: bash: 816 of max 4096 tokens, 41.7 tok/s, 22 s
[strata] writing a tool call: bash: 869 of max 4096 tokens, 42.1 tok/s, 23 s
[strata] writing a tool call: bash: 915 of max 4096 tokens, 42.3 tok/s, 24 s
[strata] writing a tool call: bash: 957 of max 4096 tokens, 42.2 tok/s, 25 s

I think if you can run something like qwen 35B or above that. you can open it.

[u/hideo_kuze_]

I haven't run it myself. One option could be to run it on a locked down container or wait for llama.cpp

IIRC llama.cpp has a PR for MoE caching. but IDK how good it is compared to strata

[u/Jacalyn_Nivins]

The working CMake command is in these comments, not the repo, while setup.sh just fetches an unaudited binary. That's the real documentation gap.

[u/bluekingmuch]

have you tried

./setup.sh --build

[u/1A655A9CEC05B28E04 → u/bluekingmuch]

That's exactly the problem being discussed here..

[u/bluekingmuch → u/1A655A9CEC05B28E04]

then i am glad i could help! with --build, no "big ass unaudited binary" is pulled, but source will be compiled then.

[u/No-Wall6427]

My hermes agent builds it each time, it takes a few minutes.

[u/Timely_Impression_92]

run some agent inside repo and ask about it - gpt sol created build script for me

[u/Gold_Coconut9777 → u/Timely_Impression_92]

Why not share it then?

[u/Gold_Coconut9777 → u/[deleted]]

It will just follow the setup.sh path which is exactly what we are trying to tweak