BePilot: An AI Programming Assistant for Compiler Backend Development

This project provides the fine-tuned BePilot-1.5B/7B based on QWen2.5-Coder-1.5B/7B aiming for enhancing manual compiler backend development efficiency.

1. Dependency

conda env create -f bepilot.yml

2. Fine-Tuning Process

We performed LoRA fine-tuning on QWen2.5-Coder-1.5B/7B using the train and validation set in ComBack++.

3. Inferencing

Our lora fine-tuned BePilot-1.5B is saved in ./BePilot-1.5B and BePilot-7B is saved in ./BePilot-7B.

Please specify the parameter and batch size for inferencing in ./run_inference.sh.

Run following command for inferencing:

bash ./run_inference.sh

The BePilot-generated code will be saved in ./Res.

The accuracy of BePilot-1.5B/7B on the test set of ComBack++ surpasses that of some mainstream LLMs for code with similar parameter counts.

• GCC

  	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Code. Gen.	Code. Gen.	Code. Gen.	Pro. Rep.	Pro. Rep.	Pro. Rep.	Pro. Rep.	Pro. Rep.	Pro. Rep.
  	C++	C++	C++	MD	MD	MD	C++	C++	C++	MD	MD	MD	C++	C++	C++	C++	C++	C++	MD	MD	MD
  Model	EM	ED	R-L	EM	ED	R-L	EM	ED	R-L	EM	ED	R-L	CB	ED	R-L	EM	ED	R-L	EM	ED	R-L
  DeepSeekCoder-1.3B	0	5.44	0.01	0	5.01	0	0	7.57	0.01	0	7.28	0.01	19.57	26.61	0.06	0	13.96	0.05	0	11.77	0.03
  YiCoder-1.5B	0	1.03	0.00	0	1.50	0.01	0	1.63	0.01	0	2.38	0.01	23.22	5.58	0.01	0	1.63	0.01	0	1.63	0.01
  QWen2.5-Coder-1.5B	0	3.76	0.02	0	5.50	0.03	0.04	4.11	0.02	0	6.20	0.03	25.74	19.68	0.04	0	7.36	0.02	0	9.03	0.02
  CodeLLaMA-7B	0	4.63	0.10	0	4.61	0.14	0	6.25	0.05	0.05	11.87	0.13	16.65	22.93	0.09	0	15.68	0.17	0	9.91	0.21
  CodeGemma-7B	0	5.36	0.03	0	5.81	0.02	0	9.45	0.02	0	8.72	0.01	25.74	26.89	0.08	0	18.84	0.09	0	18.58	0.10
  QWen2.5-Coder-7B	0	10.06	0.03	0	16.41	0.07	0	18.08	0.06	0	22.01	0.13	22.96	27.89	0.07	0	18.63	0.04	0	16.94	0.03
  CodeLLaMA-13B	0	4.42	0.04	0	4.89	0.13	0	5.10	0.04	0	4.71	0.08	22.42	25.60	0.06	0	13.67	0.21	0	6.90	0.33
  QWen2.5-Coder-14B	0.42	1.75	0.25	0	0.68	0.34	0	0.90	0.37	0	1.10	0.45	13.45	27.41	0.19	0	15.01	0.26	0	12.59	0.11
  QWen2.5-Coder-32B	0	0.88	0.23	0	0.57	0.19	0	1.19	0.44	0	1.22	0.32	15.02	26.55	0.21	0	14.63	0.27	0	12.14	0.09
  CodeLLaMA-34B	0	9.76	0.21	0	12.01	0.41	0.13	13.07	0.19	0.69	20.53	0.35	16.98	25.93	0.11	0	18.62	0.44	0	9.86	0.28
  BePilot-1.5B	57.55	77.98	0.87	74.03	88.45	0.94	60.11	74.24	0.76	58.70	84.81	0.88	40.09	48.77	0.50	29.10	71.62	0.74	29.31	78.77	0.80
  BePilot-7B	69.26	84.78	0.88	78.07	90.62	0.94	68.82	80.45	0.81	68.58	88.35	0.89	51.97	59.50	0.60	35.77	73.21	0.77	35.46	80.98	0.83

• LLVM

  	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Stmt.Comp.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Next. Sugg.	Code. Gen.	Code. Gen.	Code. Gen.	Pro. Rep.	Pro. Rep.	Pro. Rep.	Pro. Rep.	Pro. Rep.	Pro. Rep.
  	C++	C++	C++	TD	TD	TD	C++	C++	C++	TD	TD	TD	C++	C++	C++	C++	C++	C++	TD	TD	TD
  Model	EM	ED	R-L	EM	ED	R-L	EM	ED	R-L	EM	ED	R-L	CB	ED	R-L	EM	ED	R-L	EM	ED	R-L
  DeepSeekCoder-1.3B	0	5.03	0	0	8.48	0.01	0	7.45	0.01	0	7.99	0.01	22.36	25.22	0.06	0	14.47	0.05	0	13.23	0.06
  YiCoder-1.5B	0	1.56	0.01	0	6.09	0.03	0	2.93	0.01	0	5.21	0.03	23.69	6.59	0.01	0	2.08	0.01	0	0.94	0.01
  QWen2.5-Coder-1.5B	0	3.59	0.02	0	7.52	0.05	0.02	4.57	0.01	0	5.50	0.03	27.17	18.96	0.04	0	7.79	0.02	0	11.33	0.05
  CodeLLaMA-7B	0	4.42	0.09	0	8.93	0.11	0.04	5.60	0.03	0.29	9.80	0.09	19.29	17.44	0.05	0	10.47	0.10	0	8.90	0.14
  CodeGemma-7B	0	5.02	0.02	0	7.93	0.03	0	10.72	0.02	0	11.20	0.01	29.37	25.35	0.08	0	20.18	0.08	0	18.79	0.11
  QWen2.5-Coder-7B	0	8.75	0.02	0	17.84	0.06	0	19.16	0.07	0	27.45	0.20	25.20	28.80	0.08	0	20.11	0.05	0	22.68	0.08
  CodeLLaMA-13B	0	2.98	0.02	0	6.40	0.12	0.02	5.30	0.02	0	12.71	0.31	19.05	18.00	0.08	0	16.17	0.29	0	3.06	0.06
  QWen2.5-Coder-14B	0.05	1.39	0.23	0	2.39	0.15	0	0.98	0.36	0	1.18	0.52	13.21	27.91	0.17	0	15.78	0.25	0	18.71	0.26
  QWen2.5-Coder-32B	0	1.01	0.18	0	2.08	0.12	0	1.28	0.42	0	1.68	0.55	16.25	27.47	0.20	0	15.74	0.26	0	19.77	0.25
  CodeLLaMA-34B	0.36	10.31	0.25	0	26.50	0.37	0.30	15.50	0.19	1.01	22.91	0.55	21.30	19.84	0.07	0	18.20	0.36	0	6.10	0.14
  BePilot-1.5B	62.11	80.79	0.90	60.46	83.81	0.89	57.39	73.16	0.75	51.30	70.60	0.76	41.12	49.25	0.49	26.64	71.63	0.76	7.02	74.44	0.74
  BePilot-7B	73.90	87.53	0.91	68.09	86.89	0.90	64.69	78.34	0.79	57.51	75.97	0.81	47.08	55.18	0.57	33.19	71.68	0.78	7.23	70.72	0.73