Improving DeepSeek R1 reasoning trace
This post is about my journey to make DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf answer correctly the following prompt:
"I currently have 2 apples. I ate one yesterday. How many apples do I have now? Think step by step."
Context: I noticed in the past by looking at the logits that Llama 3B Q3 GGUF should be able to answer correctly that prompt if it's guided in the right direction in certain key moments.
With the release of DeepSeek models, now I have a new toy to experiment with because these models are trained with certain phrases (like "Hmm", "Wait", "So", "Alternatively") meant to enhance reasoning.
Vgel made a gist where </think> is replaced with one such phrase in order to extend the reasoning trace.
I adapted Vgel's idea to Backtrack Sampler and noticed that DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf can't answer the prompt correctly even if I extend the reasoning trace a lot.
What seems to be happening is that once it gets to the wrong conclusion too early, it starts outputting other ways to get to the same wrong conclusion and the "Wait" phrase doesn't really trigger a perspective that that even considers the right answer or takes into account the timing.
So I decided that instead of just replacing "</think>", to also replace "So" and "Therefore" with " But let me rephrase the request to see if I missed something." in order to help it not draw the wrong conclusion too early.
Now the reasoning text was good, but the problem was that it just didn't stop reasoning. It takes into account today/yesterday as key elements of the prompt and it understands that the correct answer might be "2", but it's really confused by this and can't reach a conclusion.
So I added another replace criteria in order to hurry up the reasoning: after 1024 tokens were reached, I wanted it to replace "Wait" and "But" with "\nOkay, so in conclusion".
This actually did the trick, and I finally managed to get a quantized 'small' model to answer that prompt correctly, woohoo! 🎉
Please note that in my experiments, I'm using the standard temperature in llama.cpp Python (0.7). I also tried using a very low temperature, but the model doesn’t provide a good reasoning trace and starts to repeat itself. Adding a repeat penalty also ruins the output, as the model tends to repeat certain phrases.
Overall, I’m fine with a 0.7 temperature because the reasoning trace is super long, giving the model many chances to discover the correct answer. The replacements I presented seem to work best after multiple trials, though I do believe the replacement phrases can be further improved to achieve the correct result more often.
