The most I could squeeze out of Bonsai’s output speed was around 40 tokens per second; the average was between 10 and 20. Disabling thinking boosted the output speed, and cut down the time to first token (and the overall output time), but sometimes that came at the cost of accuracy or coherency. For instance, the William Gibson question worked best with thinking on. With thinking off, I got a reply that was outrageously false. (No, William Gibson did not write Blood Meridian; Cormac McCarthy did.)
Thinking costs also had a major impact in the coding-related prompts. The code modularity question, for instance, took longer than six minutes just in the thinking phase, and the actual response took longer than two minutes to generate at 4.1 tokens per second. With thinking off, the results came more quickly (although there was a fair amount of overhead for tokenizing the input), and Bonsai generated a shorter but still useful reply at 7.7 tokens per second.
One intriguing capability in Bonsai 27B is its ability to use draft models for speculative decoding, a technique for speeding up token generation. This it does in place of using multi-token prediction, or MTP. However, speculative decoding requires loading both the main model and the draft model into GPU memory, so it’s only well-suited for cases where you have a lot of VRAM to throw at the problem. (MTP is integrated into the model itself.) My setup did not afford enough memory to use speculative decoding with Bonsai.
