self-critique · code-generation · codegen, starcoder
recommended confidence: moderate no check recorded
Fine-tuning code LMs to self-refine faulty generations with execution feedback lifts pass rates by up to 63.5% relative
- After Cycle's self-refinement training, iterative refinement with execution feedback raises pass rates by up to 63.5% relative over one-time generation, consistently across four model sizes on HumanEval, MBPP-S, and APPS. arXiv:2403.18746
- Without this training, prompting code LMs to refine their own faulty generation largely fails, with the model producing an exact copy of the faulty code in 42.2% of CodeGen cases and 64.8% of StarCoder cases on HumanEval. arXiv:2403.18746
CYCLE: Learning to Self-Refine the Code Generation · arXiv preprint, 2024 not read in full