Aider LLM Leaderboards
Metrics
Izigaba
Izindikimba
I-Aider LLM Leaderboards iwuhlelo lokulinganisa olwakhelwe ukuhlola amamodeli olimi amakhulu (LLM) ngokukwazi kwawo ukubhala nokuhlela ikhodi ngaphandle kokungenelela komuntu. Isebenzisa izivivinyo zokuhlela ezingama-225 ezitholakala ku-Exercism, ezisabalala ezilimini eziyisithupha zokuhlela: i-C++, i-Go, i-Java, i-JavaScript, i-Python ne-Rust.
Amamodeli ahlolwa kusetshenziswa amamethrikhi ahlanganisa izinga lokudlula, amaphesenti okulungile, namaphesenti amacala akheke kahle, okunikeza umbono onobukhulu obuningi bamakhono okwenza ikhodi. Lolu hlelo lufaneleka kakhulu kubathuthukisi nabacwaningi abafuna ukuqhathanisa ukusebenza kwe-LLM emisebenzini yokuhlela enengqondo, enezilimi eziningi, futhi lusebenza njengebhodi labahamba phambili elibuyekezwa njalo elibonisa isimo samamodeli samanje kulezi zinselelo.
Isizinda Nemvelaphi
Ibhentshimakhi ye-Aider LLM Leaderboards yasungulwa njengengxenye yephrojekthi ye-Aider, ithuluzi lokuhlela amakhodi elisizwa yi-AI, ukuze linikeze indlela ejwayelekile futhi engaphindwa kalula yokulinganisa ukuthi amamodeli amakhulu olimi angawenza kahle kangakanani umsebenzi wokuhlela amakhodi ongokoqobo. Ibhentshimakhi ithola izivivinyo zayo ku-Exercism, inkundla esekwe kahle yomthombo ovulekile enikeza izinselelo zokuhlela amakhodi eziningi ezilimini ezahlukene. Ngokuthola kuleyo ndawo, ibhentshimakhi izuza iqoqo lezinkinga elikhethwe ngokucophelela elibonisa izinselelo zangempela zokuhlela amakhodi kunokuba kube izivivinyo zokwenziwa noma ezilula.
Ibhentshimakhi igxile ngokukhethekile endleleni yokuhlola i-polyglot, ihlanganisa izilimi eziyisithupha zokuhlela amakhodi: C++, Go, Java, JavaScript, Python, ne-Rust. Lo mklamo wezilimi eziningi ubonisa iqiniso lokuthi abasizi bokuhlela amakhodi abakwazi kufanele basebenze ngokungaguquguquki kuzo zonke izinhlelo zemvelo zezilimi ezahlukene, hhayi kuphela kolunye ulimi oluvelele njengokuthi Python.
Isakhiwo Nendlela Yokuhlola
Ibhentshimakhi iqukethe izivivinyo zokuhlela amakhodi ezingama-225 ezikhethwe ngokusekelwe kubunzima bazo nobubanzi bazo phakathi kwezilimi eziyisithupha ezisekelwayo. Amamodeli ayahlolwa ngokuthi ayakwazi yini ukubhala noma ukuhlela ikhodi edlula amasethi okuhlola azenzakalelayo ngaphandle kokungenelela komuntu ngesikhathi senqubo. Le ndlela ilingisa isimo sangempela sokuhlela amakhodi esebenzisa i-agent lapho imodeli kufanele ihumushe imiyalelo, ikhiqize ikhodi elungile, futhi ibhekane namaphutha aseceleni ngokuzimela.
Ukusebenza kukalwa kusetshenziswa amamethrikhi amaningana:
- Pass Rate 1: Ukuthi umzamo wokuqala wemodeli uyadlula yini yonke imivivinyo.
- Pass Rate 2: Ukuthi imodeli iyadlula yini imivivinyo phakathi kwenani elilinganiselwe lemizamo.
- Percent Correct: Isilinganiso sezivivinyo ezixazululwe ngendlela efanele ngokuphelele.
- Percent Cases Well Formed: Isilinganiso sempendulo ezivumelekile ngokolimi nangokwesakhiwo, noma zingaphelele ngokufanele.
Ndawonye, la mamethrikhi anikeza umbono onobukhulu obuningi bamakhono emodeli, ehlukanisa phakathi kwamamodeli akhiqiza okukhiphayo okuhlanzekile nokuhlelekile nalawo athola ukunemba ngokwethembeka noma ngokusebenza kahle.
Izimo Zokusetshenziswa Nokubaluleka
I-leaderboard isebenza njengereferensi engokoqobo kubathuthukisi, abacwaningi, nezinhlangano ezihlola ukuthi yimaphi amamodeli amakhulu olimi afaneleka kakhulu imisebenzi yokukhiqiza nokuhlela ikhodi. Njengoba le bhentshimakhi isebenza njenge-leaderboard ebuyekezwa njalo, ibonisa isimo samanje sokusebenza kwamamodeli njengoba kukhishwa amamodeli amasha nezinguqulo ezibuyekeziwe. Lokhu kuyenza ibe insiza ephilayo kunokuba kube isithombe esingashintshi.
Kubathuthukisi abahlanganisa ama-LLM emisebenzini yokuhlela amakhodi noma abakha amathuluzi phezu kwama-API emodeli, i-leaderboard inikeza idatha esebenzisekayo ngokusebenza kokuhlela amakhodi okungokoqobo kuzo zonke izilimi eziningi. Abacwaningi abafunda amakhono okwenza ikhodi bangayisebenzisa ukulandelela intuthuko ngokuhamba kwesikhathi futhi bathole izindawo lapho amamodeli ehlala ehluleka khona, njengokuthi izilimi ezingajwayelekile ukuqeqeshwa ngazo njenge-Rust noma i-Go uma kuqhathaniswa ne-Python.
Ububanzi Nemikhawulo
Nakuba le bhentshimakhi ihlanganisa ububanzi obunenjongo bezilimi zokuhlela amakhodi namazinga obunzima bezivivinyo, ububanzi bayo bunqunyelwe yizivivinyo ezingama-225 ezikhethiwe nezilimi eziyisithupha ezimele. Ukusebenza kule bhentshimakhi kungase kunganwebeki ngokugcwele kuyo yonke imisebenzi yokuhlela amakhodi, ikakhulukazi leyo ebandakanya ama-codebase amakhulu, ukwakheka kwesistimu okunzima, noma izilimi ezingafakiwe eqoqweni lokuhlola. Ukwengeza, njengoba izivivinyo zitholakala ku-Exercism, le bhentshimakhi ibonisa isitayela nemikhuba yaleyo nkundla, okungenzeka ihluke kwezinye izinhlobo zezinselelo zokuhlela amakhodi noma izindawo zokusebenza zekhodi. Lezi zici kufanelekile ukuzicabangela lapho kuchazwa imiphumela ye-leaderboard kumongo obanzi wamakhono okuhlela amakhodi e-LLM.