Firehose

This paper studies how lossy verification schemes can improve the efficiency of speculative decoding in large language models, but may also degrade generation quality. Practitioners may care about understanding the trade-offs between speed and quality when using these schemes.