Running Deepseek R1 IQ2XXS (200GB) from SSD actually works

prompt eval time = 97774.66 ms / 367 tokens ( 266.42 ms per token, 3.75 tokens per second)

eval time = 253545.02 ms / 380 tokens ( 667.22 ms per token, 1.50 tokens per second)

total time = 351319.68 ms / 747 tokens

No, not a distill, but a 2bit quantized version of the actual 671B model (IQ2XXS), about 200GB large, running on a 14900K with 96GB DDR5 6800 and a single 3090 24GB (with 5 layers offloaded), and for the rest running off of PCIe 4.0 SSD (Samsung 990 pro)

Although of limited actual usefulness, it's just amazing that is actually works! With larger context it takes a couple of minutes just to process the prompt, token generation is actually reasonably fast.

Thanks https://www.reddit.com/r/LocalLLaMA/comments/1icrc2l/comment/m9t5cbw/ !

Edit: one hour later, i've tried a bigger prompt (800 tokens input), with more tokens output (6000 tokens output)

prompt eval time = 210540.92 ms / 803 tokens ( 262.19 ms per token, 3.81 tokens per second)
eval time = 6883760.49 ms / 6091 tokens ( 1130.15 ms per token, 0.88 tokens per second)
total time = 7094301.41 ms / 6894 tokens

It 'works'. Lets keep it at that. Usable? Meh. The main drawback is all the <thinking>... honestly. For a simple answer it does a whole lot of <thinking> and that takes a lot of tokens and thus a lot of time and context in follow-up questions taking even more time.