It remains to be seen once it's released, let's say theoretically unsloth quantizises it to their own version of Q8-XL, and it's 2TB in size. But we don't know what speed it will run on a dual or quad socket xeon server with, let's say 48 * 64GB DIMMs, 3TB of RAM. Enough room for the model and its full default context size. 10 tokens/s? What kind of speed will it run at when context fill is 200,000+?

The ability to run it fast enough to go on a recursive nested attack of finding an entry point into something and then proceeding with lateral movement/privilege escalation and such will require more speed, like 40-50 tok/s at least, unless you're prepared to wait weeks.

Same that some people are right now running GLM5.2 in its 850GB version on CPU-only and a pile of DDR4 or DDR5 server RAM, yeah it runs, but not very fast. Good enough to give it "build this piece of something and wait a few hours" tasks, come back later and see what it's done. Yeah, you can do that under $20-30k for sure. Even with something like a used Dell R940 with 1536GB RAM bought on eBay.

Because you need GPUs to run it fast enough for an attack to be effective. 8 GPU servers with enough VRAM are that expensive.

FWIW, you can get a 16x RTX 6000 Pro setup running for significantly less than that. Even considering the electrical hookup fees (it’s a lot of power and cooling). Home ownership might push you into the original quote though.

Kimi K3 is in no way going to fit in 96GB RAM * 16 units (1536GB) unless badly quantized and with a small amount of context.

Why not? The model is 2.8T parameters with native MXFP4, which is 1400GB.

I would figure it needs a lot of kV cache and context size which can be much bigger than the model itself

To run it "at useful speed"