Izinhlangano kuzo zonke izimboni zisebenzisa ukufinyezwa kombhalo okuzenzakalelayo ukuze ziphathe kahle kakhulu inani elikhulu lolwazi futhi zenze izinqumo ezingcono. Emkhakheni wezezimali, amabhange otshalomali afinyeza imibiko yenzuzo yehlele ezintweni ezibalulekile ezithathwayo ukuze ahlaziye ngokushesha ukusebenza kwekota. Izinkampani zemidiya zisebenzisa ukufingqa ukuqapha izindaba nezinkundla zokuxhumana ukuze izintatheli zisheshe zibhale izindaba ngezindaba ezisathuthuka. Ama-ejensi kahulumeni afingqa amadokhumenti enqubomgomo ende nemibiko ukusiza abenzi bezinqubomgomo bahlele amasu futhi babeke phambili imigomo.

Ngokwenza izinguqulo ezifingqiwe zamadokhumenti amade, ayinkimbinkimbi, ubuchwepheshe bokufingqa buvumela abasebenzisi ukuthi bagxile kokuqukethwe okubaluleke kakhulu. Lokhu kuholela ekuqondeni okungcono nasekugcinweni kolwazi olubalulekile. Ukonga isikhathi kuvumela ababambiqhaza ukuthi babuyekeze izinto eziningi ngesikhathi esincane, bathole umbono obanzi. Ngokuqonda okuthuthukisiwe kanye nemininingwane ehlanganisiwe kakhudlwana, izinhlangano zingenza izinqumo zamasu ezinolwazi olungcono, zisheshise ucwaningo, zithuthukise ukukhiqiza, futhi zandise umthelela wazo. Amandla okuguqula amandla okufinyeza okuthuthukile azoqhubeka nokukhula njengoba izimboni ezengeziwe zisebenzisa ubuhlakani bokwenziwa (AI) ukuze zisebenzise imifudlana yolwazi echichimayo.

Kulokhu okuthunyelwe, sihlola izindlela eziholayo zokuhlola ukunemba kwesifinyezo ngokuqondile, okuhlanganisa amamethrikhi e-ROUGE, METEOR, kanye ne-BERTScore. Ukuqonda amandla kanye nobuthakathaka balawa masu kungasiza ukuqondisa imizamo yokukhethwa nokuthuthukisa. Umgomo usuwonke walokhu okuthunyelwe uwukwenza uhlolo lwesifinyezo lungabonakali ukuze sisize amaqembu angcono ukusebenza kwebhentshimakhi kuleli khono elibalulekile njengoba efuna ukukhulisa inani.

Izinhlobo zokufingqa

Ukufingqa ngokuvamile kungahlukaniswa ngezinhlobo ezimbili eziyinhloko: isifinyezo esithathelwanayo kanye nesifinyezo esingabonakali. Zombili lezi zindlela zihlose ukufingqa izingcezu ezinde zombhalo zibe izinhlobo ezimfushane, zithwebule ulwazi olubucayi kakhulu noma ingqikithi yokuqukethwe kwasekuqaleni, kodwa zikwenza ngezindlela ezihluke kakhulu.

Ukufingqa okukhiphayo kuhilela ukukhomba nokukhipha imishwana eyinhloko, imisho, noma amasegimenti embhalweni wokuqala ngaphandle kokuwashintsha. Uhlelo lukhetha izingxenye zombhalo othathwa njengonolwazi kakhulu noma omele wonke. Ukufingqa okukhiphekayo kuyasiza uma ukunemba kubalulekile futhi isifinyezo sidinga ukukhombisa ulwazi oluqondile oluvela embhalweni wokuqala. Lezi kungaba izimo zokusebenzisa njengokugqamisa imigomo ethile yomthetho, izibopho, namalungelo ashiwo emigomeni yokusebenzisa. Amasu ajwayeleke kakhulu asetshenziselwa ukufinyeza okukhishiwe yi-term frequency-inverse document frequency (TF-IDF), amaphuzu emisho, i-algorithm yezinga lombhalo, nokufunda komshini ogadiwe (ML).

Ukufingqa okungaqondakali kuqhubela phambili ngokukhiqiza imishwana emisha nemisho ebingekho embhalweni wokuqala, ngokuyisisekelo ibeka ngamafuphi futhi ifinyeze okuqukethwe kwasekuqaleni. Le ndlela idinga ukuqonda okujulile kombhalo, ngoba i-AI idinga ukuhumusha incazelo bese iyiveza ngendlela entsha, emfushane. Amamodeli ezilimi ezinkulu (ama-LLM) afaneleka kangcono ukufinyezwa okungaqondakali ngoba amamodeli e-transformer asebenzisa izindlela zokunaka ukuze agxile ezingxenyeni ezifanele zombhalo wokufakwayo lapho enza izifinyezo. Indlela yokunaka ivumela imodeli ukuthi yabele izisindo ezihlukene kumagama ahlukene noma amathokheni ngokulandelana kokufaka, okuyenza ithwebule ukuncika kwebanga elide nolwazi oluhlobene komongo.

Ngaphezu kwalezi zinhlobo ezimbili eziyinhloko, kunezindlela ezixubile ezihlanganisa izindlela zokukhipha kanye nezokungabonakali. Lezi zindlela zingase ziqale ngokufinyeza okungeziwe ukuze kuhlonzwe okuqukethwe okubaluleke kakhulu bese zisebenzisa amasu abstractive ukuze zibhale kabusha noma zifinyeze lokho okuqukethwe kube isifinyezo esisheshayo.

Inselele

Ukuthola indlela ephelele yokuhlola ikhwalithi yesifinyezo kuseyinselele evulekile. Njengoba izinhlangano ziya ngokuya zithembela ekufinyezweni kombhalo okuzenzakalelayo ukuze zichithe ulwazi olubalulekile olusuka kumadokhumenti, isidingo siyakhula samasu amisiwe okulinganisa ukunemba kokufingqa. Ngokufanelekile, lawa mamethrikhi okuhlola angalinganisela ukuthi izifinyezo ezikhiqizwe ngomshini zikukhipha kahle kangakanani okuqukethwe okubaluleke kakhulu emibhalweni yomthombo futhi zethule izifinyezo ezihambisanayo ezibonisa incazelo nomongo wangempela.

Nokho, ukwenza izindlela zokuhlola eziqinile zokufingqa umbhalo kuletha ubunzima:

• Izifinyezo zereferensi ezigunyazwe ngumuntu ezisetshenziselwa ukuqhathanisa ngokuvamile zibonisa ukuhlukahluka okuphezulu okusekelwe ekunqumeni okubalulekile kokubaluleka
• Izici ezicashile zekhwalithi yesifinyezo njengokuqephuza, ukufundeka, nokuhambisana kubonakala kunzima ukulinganisa ngokohlelo
• Ukwehluka okubanzi kukhona kuzo zonke izindlela zokufingqa kusuka kuma-algorithms ezibalo kuya kumanethiwekhi we-neural, okwenza kube nzima ukuqhathanisa okuqondile

I-Recall-Oriented Understudy for Gisting Evaluation (ROUGE)

ROUGE amamethrikhi, njenge-ROUGE-N kanye ne-ROUGE-L, idlala indima ebalulekile ekuhloleni ikhwalithi yezifinyezo ezikhiqizwa umshini uma kuqhathaniswa nezifinyezo zereferensi ezibhalwe ngabantu. Lawa mamethrikhi agxile ekuhloleni ukugqagqana phakathi kokuqukethwe kwezifinyezo ezikhiqizwe ngumshini nezidalwe umuntu ngokuhlaziya ama-n-grams, okungamaqembu wamagama noma amathokheni. Isibonelo, i-ROUGE-1 ihlola ukufana kwamagama ngamanye (amayunigramu), kanti i-ROUGE-2 ibheka amapheya amagama (ama-bigrams). Ukwengeza, i-ROUGE-N ihlola ukulandelana okude kakhulu kwamagama phakathi kwemibhalo emibili, okuvumela ukuguquguquka kokuhleleka kwamagama.

Ukubonisa lokhu, cabangela lezi zibonelo ezilandelayo:

• ROGUE-1 metric - I-ROUGE-1 ihlola ukugqagqana kwama-unigram (amagama angawodwa) phakathi kwesifinyezo esikhiqiziwe nesifinyezo sereferensi. Isibonelo, uma isifinyezo sereferensi siqukethe "Impungushe ensundu esheshayo eyeqa," futhi isifinyezo esikhiqiziwe sithi "Impungushe ensundu igxuma ngokushesha," imethrikhi ye-ROUGE-1 izocabangela "okunsundu," "impungushe," kanye "nokugxuma" njengokugqagqene. ama-unigrams. I-ROUGE-1 igxile ebukhoneni bamagama ngamanye ezifingqini, ukulinganisa ukuthi isifinyezo esikhiqiziwe sithatha kahle kanjani amagama ayisihluthulelo esifinyezweni sereferensi.
• ROGUE-2 metric - I-ROUGE-2 ihlola ukugqagqana kwama-bigrams (amapheya amagama ancikene) phakathi kwesifinyezo esikhiqiziwe nesifinyezo sereferensi. Isibonelo, uma isifinyezo sereferensi sinokuthi "Ikati lilele," futhi isifinyezo esikhiqiziwe sithi "Ikati lilele," i-ROUGE-2 izohlonza ukuthi "ikati lilele" futhi "ilele" njenge-bigram egqagqene. I-ROUGE-2 inikeza ukuqonda kokuthi isifinyezo esikhiqiziwe sikugcina kahle kanjani ukulandelana nomongo wokubhanqwa kwamagama uma kuqhathaniswa nesifinyezo sereferensi.
• ROUGE-N imethrikhi - I-ROUGE-N ifomu elijwayelekile lapho u-N emelela noma iyiphi inombolo, evumela ukuhlolwa okusekelwe kuma-n-grams (ukulandelana kwamagama angu-N). Uma kucatshangelwa i-N=3, uma isifinyezo sereferensi sithi "Ilanga likhanya ngokugqamile," futhi isifinyezo esikhiqiziwe sithi "Ilanga likhanya ngokugqamile," i-ROUGE-3 ingabona "ilanga elikhanya ngokugqamile" njenge-trigram efanayo. I-ROUGE-N inikeza ukuguquguquka kokuhlola izifinyezo ngokusekelwe kubude obuhlukene bokulandelana kwamagama, inikeze ukuhlola okubanzi kokugqagqana kokuqukethwe.

Lezi zibonelo zibonisa indlela amamethrikhi e-ROUGE-1, ROUGE-2, kanye ne-ROUGE-N asebenza ngayo ekuhloleni isifinyezo esizenzakalelayo noma imisebenzi yokuhumusha ngomshini ngokuqhathanisa izifinyezo ezikhiqiziwe nezifinyezo zereferensi ezisekelwe kumazinga ahlukene wokulandelana kwamagama.

Bala isikolo se-ROUGE-N

Ungasebenzisa izinyathelo ezilandelayo ukubala isikolo se-ROUGE-N:

1. Faka ithokheni isifinyezo esikhiqiziwe nesifinyezo sereferensi kumagama angawodwana noma amathokheni usebenzisa izindlela eziyisisekelo zokwenza amathokheni njengokuhlukanisa ngesikhala esimhlophe noma imitapo yolwazi yokucubungula ulimi lwemvelo (NLP).
2. Khiqiza ama-n-grams (ukulandelana okuhambisanayo kwamagama angu-N) kusukela kukho kokubili isifinyezo esikhiqiziwe kanye nesifinyezo sereferensi.
3. Bala inani lama-n-grams agqagqene phakathi kwesifinyezo esikhiqiziwe nesifinyezo sereferensi.
4. Bala ukunemba, ukukhumbula, kanye nesikolo se-F1:
   • Ukwenza kahle hle - Inani lama-n-grams agqagqene ahlukaniswe ngenani eliphelele lama-n-grams esifinyezweni esikhiqiziwe.
   • Khumbula - Inani lama-n-grams agqagqene ahlukaniswe ngenani eliphelele lama-n-grams esifinyezweni sereferensi.
   • F1 isikolo - Incazelo ye-harmonic yokunemba nokukhumbula, ibalwa njengokuthi (2 * ukunemba * khumbula) / (ngokunemba + khumbula).
5. Isilinganiso sesikolo se-F1 esitholwe ekubaleni ukunemba, ukukhumbula, kanye nesikolo se-F1 kumugqa ngamunye kudathasethi sithathwa njengomphumela we-ROUGE-N.

Ukulinganiselwa

I-ROGUE inemikhawulo elandelayo:

• Ukugxila okuncane ekugqagqaneni kwe-lexical - Umqondo oyinhloko ngemuva kwe-ROUGE uwukuqhathanisa isifinyezo esikhiqizwe uhlelo nesethi yereferensi noma izifinyezo ezidalwe ngumuntu, futhi ulinganise ukugqagqana kwamagama phakathi kwazo. Lokhu kusho ukuthi i-ROUGE inokugxila okuncane kakhulu ekufaneni kwezinga legama. Empeleni ayihloli incazelo ye-semantic, ukuhambisana, noma ukufundeka kwesifinyezo. Isistimu ingathola amaphuzu aphezulu e-ROUGE ngokumane ikhiphe imisho igama negama embhalweni wokuqala, ngaphandle kokukhiqiza isifinyezo esihambisanayo noma esifushane.
• Ukungazweli ekuchazeni amagama - Ngenxa yokuthi i-ROUGE incike ekufanisweni kwe-lexical, ayikwazi ukubona ukulingana kwe-semantic phakathi kwamagama nemishwana. Ngakho-ke, ukuchaza amagama kanye nokusetshenziswa komqondofana ngokuvamile kuzoholela kuzikolo eziphansi ze-ROUGE, ngisho noma incazelo igciniwe. Lokhu kwenza kube nokubi amasistimu abeka ngamafuphi noma afingqe ngendlela engacacile.
• Ukuntula ukuqonda kwe-semantic – U-ROUGE akahloli ukuthi ingabe uhlelo lwaluziqonda ngempela yini izincazelo nemiqondo embhalweni wokuqala. Isifinyezo singafinyelela ukugqagqana kwezichazamagama nezikhombo, kuyilapho siphuthelwa imiqondo eyinhloko noma siqukethe ukungqubuzana kwangempela. U-ROUGE ubengeke azikhombe lezi zinkinga.

Isetshenziswa nini i-ROUGE

I-ROUGE ilula futhi iyashesha ukubala. Yisebenzise njengesisekelo noma ibhentshimark yekhwalithi yesifinyezo ehlobene nokukhetha okuqukethwe. Amamethrikhi e-ROUGE asetshenziswa ngempumelelo kakhulu ezimeni ezibandakanya imisebenzi yokufingqa engaqondakali, ukuhlolwa kokufingqa okuzenzakalelayo, ukuhlolwa kwama-LLM, nokuhlaziya okuqhathanisayo kwezindlela zokufingqa ezihlukene. Ngokusebenzisa amamethrikhi e-ROUGE kulezi zimo, ababambiqhaza bangakwazi ukuhlola ngobuningi ikhwalithi nokusebenza kwezinqubo zokukhiqiza isifinyezo.

I-Metric Yokuhlola Ukuhumusha Ngoku-oda Okucacile (METEOR)

Enye yezinselelo ezinkulu ekuhloleni amasistimu okufingqa ukuhlola ukuthi isifinyezo esikhiqiziwe sihamba kahle kangakanani ngokunengqondo, kunokukhetha amagama afanele nemishwana embhalweni womthombo. Ukukhipha kalula amagama angukhiye afanelekile nemisho akusho ukuthi kukhiqizi isifinyezo esibumbene nesibumbene. Isifinyezo kufanele sihambe kahle futhi sixhumanise imibono ngokunengqondo, ngisho noma ingethulwanga ngendlela efanayo neyedokhumenti yokuqala.

Ukuvumelana nezimo zokumatanisa ngokunciphisa amagama emsukeni wawo noma esimweni sesisekelo (Isibonelo, ngemva kokunquma, amagama anjengokuthi “gijima,” “gijima,” kanye “gijima” wonke aba “gijima”) kanye nomqondofana asho METEOR ihlobana kangcono nezinqumo zomuntu zekhwalithi efingqiwe. Ingakwazi ukubona ukuthi okuqukethwe okubalulekile kugcinwa yini, ngisho noma amagama ehluka. Lena inzuzo eyinhloko ngaphezu kwamamethrikhi asekelwe ku-n-gram afana ne-ROUGE, ebheka kuphela ukufana kwamathokheni okuyiwonawona. I-METEOR iphinde inikeze amaphuzu aphezulu ezifinyezweni ezigxile kokuqukethwe okubaluleke kakhulu okuvela kusithenjwa. Amaphuzu aphansi anikezwa ulwazi oluphindaphindayo noma olungabalulekile. Lokhu kuhambisana kahle nomgomo wokufingqa ukugcina okuqukethwe okubaluleke kakhulu kuphela. I-METEOR iyimethrikhi enengqondo ngokwezibalo enganqoba eminye imikhawulo yokufanisa i-n-gram yokuhlola ukufingqwa kombhalo. Ukufakwa kwesiqu kanye namasinonimu kuvumela ukuhlolwa okungcono kolwazi oludlulele kanye nokunemba kokuqukethwe.

Ukubonisa lokhu, cabangela lezi zibonelo ezilandelayo:

Isifinyezo Sereferensi: Amaqabunga awela ngesikhathi sekwindla.

Kwenziwe Isifinyezo 1: Amaqabunga awela ekwindla.

Kwenziwe Isifinyezo 2: Ishiya eluhlaza ehlobo.

Amagama afanayo phakathi kwereferensi nesifinyezo esikhiqiziwe esingu-1 agqanyisiwe:

Isifinyezo Sereferensi: Leaves iwe ngesikhathi sasekwindla.

Kwenziwe Isifinyezo 1: Leaves phonsa phakathi iwe.

Noma elithi “ukwindla” kanye “nekwindla” kuyizimpawu ezihlukene, i-METEOR iwabona njenganomqondo ofanayo ngokumadanisa kwawo amagama afanayo. “Ukuwisa” kanye “nokuwa” kukhonjwa njengokufana okuneziqu. Esifinyezweni esikhiqiziwe 2, akukho okufanayo nesifinyezo sereferensi ngaphandle kokuthi “Amaqabunga,” ngakho lesi sifinyezo sizothola amaphuzu aphansi kakhulu e-METEOR. Uma okufanayo okunengqondo kakhulu, kuya phezulu amaphuzu we-METEOR. Lokhu kuvumela i-METEOR ukuthi ihlole kangcono okuqukethwe kanye nokunemba kwezifinyezo uma kuqhathaniswa nokumatanisa kwe-n-gram okulula.

Bala isikolo se-METEOR

Qedela izinyathelo ezilandelayo ukuze ubale isikolo se-METEOR:

1. Faka ithokheni isifinyezo esikhiqiziwe nesifinyezo sereferensi kumagama angawodwana noma amathokheni usebenzisa izindlela eziyisisekelo zokwenza amathokheni njengokuhlukanisa ngesikhala esimhlophe noma imitapo yolwazi ye-NLP.
2. Bala ukunemba kwe-unigram, ukukhumbula, kanye nesilinganiso sika-F-mean, unikeze isisindo esiningi sokukhumbula kunokunemba.
3. Faka inhlawulo ngokufana ngqo ukuze ugweme ukukugcizelela ngokweqile. Inhlawulo ikhethwa ngokusekelwe kuzici zesethi yedatha, izidingo zomsebenzi, kanye nebhalansi phakathi kokunemba nokukhumbula. Susa le nhlawulo kumphumela we-F-mean obalwe esinyathelweni sesi-2.
4. Bala isikolo se-F-mean samafomu aneziqu (ukunciphisa amagama emsuka noma efomini lempande) kanye nomqondo ofanayo wamayunigram lapho kufanele khona. Hlanganisa lokhu ngesikolo esibalwe ngaphambilini se-F-mean ukuze uthole amaphuzu wokugcina we-METEOR. Isikolo se-METEOR sisukela ku-0–1, lapho u-0 ebonisa ukuthi akukho ukufana phakathi kwesifinyezo esikhiqiziwe nesifinyezo sereferensi, futhi u-1 ebonisa ukuqondana okuphelele. Ngokuvamile, amaphuzu okufinyezwa awela phakathi kuka-0–0.6.

Ukulinganiselwa

Lapho kusetshenziswa imethrikhi ye-METEOR yokuhlola imisebenzi yokufingqa, kungase kuphakame izinselele ezimbalwa:

• Ubunkimbinkimbi be-Semantic – Ukugcizelela kwe-METEOR ekufananeni kwe-semantic kungaba nzima ukuze kuthwebule izincazelo ezinencanyana nomongo emisebenzini eyinkimbinkimbi yokufingqa, okungase kuholele ekungalumbeni kahle ekuhlaziyeni.
• Ukuhlukahluka kwereferensi - Ukwehluka kwezifinyezo zereferensi ezikhiqizwe ngumuntu kungaba nomthelela kuzikolo ze-METEOR, ngenxa yokuthi umehluko kokuqukethwe okuyisithenjwa ungase uthinte ukuhlolwa kwezifinyezo ezikhiqizwa umshini.
• Ukuhlukahluka kwezilimi - Ukusebenza kwe-METEOR kungase kwehluke ngezilimi eziningi ngenxa yokuhlukahluka kwezilimi, ukuhluka kwe-syntax, nama-nuances we-semantic, okuletha izinselele ekuhloleni ukufingqa kwezilimi eziningi.
• Ukungafani kobude - Ukuhlola izifinyezo zobude obuhlukahlukene kungaba inselele ku-METEOR, ngoba ukungafani ngobude uma kuqhathaniswa nesifinyezo sereferensi kungase kubangele izinhlawulo noma amaphutha ekuhloleni.
• Ukushuna ipharamitha – Ukuthuthukisa amapharamitha e-METEOR kumadathasethi ahlukene nemisebenzi yokufingqa kungase kudle isikhathi futhi kudinga ukushuna ngokucophelela ukuze kuqinisekiswe ukuthi imethrikhi inikeza ukuhlola okunembile.
• Ukuchema kokuhlola - Kukhona ubungozi bokuchema kokuhlola nge-METEOR uma kungalungiswanga kahle noma kulinganiselwe izizinda ezithile zokufingqa noma imisebenzi. Lokhu kungase kuholele emiphumeleni esontekile futhi kuthinte ukwethembeka kwenqubo yokuhlola.

Ngokuqaphela lezi zinselele futhi uzicabangele lapho usebenzisa i-METEOR njengemethrikhi yemisebenzi yokufingqa, abacwaningi nodokotela bangazulazula ngemikhawulo engaba khona futhi benze izinqumo ezinolwazi kakhulu ezinqubweni zabo zokuhlola.

Isetshenziswa nini i-METEOR

I-METEOR ijwayele ukusetshenziselwa ukuhlola ngokuzenzakalelayo ikhwalithi yezifinyezo zombhalo. Kungcono ukusebenzisa i-METEOR njengemethrikhi yokuhlola lapho ukuhleleka kwemibono, imiqondo, noma amabhizinisi kusifinyezo sibalulekile. I-METEOR ibheka ukuhleleka futhi ifanise ama-n-grams phakathi kwesifinyezo esikhiqiziwe nezifinyezo zereferensi. Iklomelisa izifinyezo ezigcina ulwazi olulandelanayo. Ngokungafani namamethrikhi afana ne-ROUGE, ancike ekugqagqaneni kwama-n-grams nezifinyezo zereferensi, i-METEOR ifana neziqu, amagama afanayo, nama-paraphrases. I-METEOR isebenza kangcono uma kungaba nezindlela eziningi ezilungile zokufingqa umbhalo wokuqala. I-METEOR ihlanganisa amagama afanayo e-WordNet kanye namathokheni aneziqu lapho imesha ama-n-grams. Ngamafuphi, izifinyezo ezifanayo ngokwezibalo kodwa ezisebenzisa amagama ahlukene noma imisho zisazozuza kahle. I-METEOR inenhlawulo eyakhelwe ngaphakathi yezifinyezo ezinama-n-grams aphindaphindayo. Ngakho-ke, akukhuthazi ukukhishwa kwegama negama noma ukuntuleka kokukhipha. I-METEOR iyisinqumo esihle lapho ukufana kwe-semantic, ukuhleleka kwemibono, nemisho eqephuzayo kubalulekile ekwahluleleni ikhwalithi yesifinyezo. Ayifaneleki kangako imisebenzi lapho kuphela ukugqagqana kwezichazamazwi nezifingqo zereferensi ezibalulekile.

BERTScore

Izilinganiso zesichazamazwi sezinga eliphezulu njenge-ROUGE kanye ne-METEOR zihlola amasistimu wokufingqa ngokuqhathanisa igama eligqagqana phakathi kwesifinyezo sekhandidethi nesifinyezo sereferensi. Nokho, bathembela kakhulu ekufanisweni kwentambo phakathi kwamagama nemishwana. Lokhu kusho ukuthi bangase baphuthelwe ukufana kwe-semantic phakathi kwamagama nemishwana enezakhiwo ezihlukile kodwa izincazelo eziyisisekelo ezifanayo. Ngokuthembela kuphela ekufanisweni kwendawo, lawa mamethrikhi angase abukele phansi ikhwalithi yezifinyezo zesistimu ezisebenzisa amagama afanayo noma imiqondo ephimisela amagama ngendlela ehlukile kuzifingqo zereferensi. Izifinyezo ezimbili zingadlulisela ulwazi olucishe lufane kodwa zithole amaphuzu asezingeni eliphansi ngenxa yomehluko wamagama.

BERTScore iyindlela yokuhlola ngokuzenzakalela ukuthi isifinyezo sihle kangakanani ngokusiqhathanisa nesifinyezo esiyireferensi esibhalwe ngumuntu. Isebenzisa i-BERT, indlela edumile ye-NLP, ukuqonda incazelo nomongo wamagama kusifinyezo sekhandidethi nesifinyezo sereferensi. Ngokucacile, ibheka igama ngalinye noma ithokheni kusifinyezo sekhandidethi futhi ithola igama elifanayo kakhulu kusifinyezo sereferensi esisekelwe ekushumekiweyo kwe-BERT, okuyizethulo zevekhtha zencazelo nomongo wegama ngalinye. Ikala ukufana kusetshenziswa ukufana kwe-cosine, okutshela ukuthi ama-vector asondele kangakanani komunye nomunye. Kugama ngalinye kusifinyezo sekhandidethi, lithola igama elihlobene kakhulu kusifinyezo sereferensi kusetshenziswa ukuqonda kwe-BERT kolimi. Iqhathanisa konke lokhu kufana kwamagama kuso sonke isifinyezo ukuze uthole amaphuzu aphelele wokuthi isifinyezo sekhandidethi sifana kangakanani nesifinyezo sereferensi. Uma amagama nezincazelo ezishuthwe i-BERT zifana kakhulu, kuphakama i-BERTScore. Lokhu kuyivumela ukuthi ihlole ngokuzenzakalelayo ikhwalithi yesifinyezo esikhiqiziwe ngokusiqhathanisa nereferensi yomuntu ngaphandle kokudinga ukuhlolwa komuntu isikhathi ngasinye.

Ukufakazela lokhu, ake ucabange unesifinyezo esikhiqizwe ngomshini: “Impungushe ensundu esheshayo igxuma phezu kwenja evilaphayo.” Manje, ake sicabangele isifinyezo sereferensi esenziwe ngumuntu: “Impungushe ensundu esheshayo igxuma phezu kwenja elele.”

Bala i-BERTScore

Qedela izinyathelo ezilandelayo ukuze ubale i-BERTScore:

1. I-BERTScore isebenzisa ukushumeka komongo ukuze imele ithokheni ngayinye kuyo yomibili imisho yekhandidethi (ekhiqizwe ngomshini) kanye nereferensi (eyenziwe ngumuntu). Ukushumeka kokuqukethwe kuwuhlobo lokumelwa kwamagama ku-NLP oluthwebula incazelo yegama ngokusekelwe kumongo walo ngaphakathi komusho noma umbhalo. Ngokungafani nokushumeka kwamagama okuvamile okunikeza i-vector engaguquki egameni ngalinye kungakhathaliseki umongo walo, ukushumeka kwesimo kucabangela amagama azungezile ukuze kukhiqize ukumelela okuhlukile kwegama ngalinye kuye ngokuthi lisetshenziswe kanjani emshweni othile.
2. I-metric ibe isihlanganisa ukufana phakathi kwethokheni ngayinye emushweni wekhandidethi nethokheni ngayinye emshweni oyireferensi isebenzisa ukufana kwe-cosine. Ukufana kwe-cosine kusisiza ukuthi silinganisele ukuthi amasethi amabili edatha ahlobene eduze kangakanani ngokugxila lapho akhomba khona endaweni enezinhlangothi eziningi, okuyenza ibe ithuluzi elibalulekile lemisebenzi efana nama-algorithm okusesha, i-NLP, nezinhlelo zokuncoma.
3. Ngokuqhathanisa ukushumeka komongo kanye nezibalo zokufana zekhompuyutha zawo wonke amathokheni, i-BERTScore ikhiqiza ukuhlola okuphelele okuthwebula ukuhlobana kwe-semantic nomongo wesifinyezo esikhiqiziwe uma kuqhathaniswa nereferensi eyenziwe ngumuntu.
4. Okukhiphayo kokugcina kwe-BERTScore kunikeza amaphuzu afanayo abonisa ukuthi isifinyezo esikhiqizwe umshini sivumelana kanjani nesifinyezo sereferensi ngokwencazelo nomongo.

Empeleni, i-BERTScore idlulela ngalé kwamamethrikhi endabuko ngokucabangela ukuhluka kwe-semantic nomongo wemisho, inikeze ukuhlolwa okuyinkimbinkimbi kakhulu okubonisa eduze ukwahlulela komuntu. Le ndlela ethuthukisiwe ithuthukisa ukunemba nokuthembeka kokuhlola imisebenzi yokufingqa, okwenza i-BERTScore ibe ithuluzi elibalulekile ekuhloleni amasistimu okukhiqiza umbhalo.

Ukulinganiselwa:

Yize i-BERTScore inikezela ngezinzuzo ezibalulekile ekuhloleni imisebenzi yokufingqa, futhi iza nemikhawulo ethile edinga ukucatshangelwa:

• Amandla wokubala - I-BERTScore ingaba namandla ngokwezibalo ngenxa yokuthembela kumamodeli olimi aqeqeshwe kusengaphambili njenge-BERT. Lokhu kungaholela ezikhathini ezinde zokuhlola, ikakhulukazi lapho kucutshungulwa umthamo omkhulu wedatha yombhalo.
• Ukuncika kumamodeli aqeqeshwe ngaphambilini – Ukusebenza kwe-BERTScore kuncike kakhulu kwikhwalithi kanye nokuhambisana kwemodeli yolimi oluqeqeshwe ngaphambilini olusetshenzisiwe. Ezimeni lapho imodeli eqeqeshwe kusengaphambili ingase ingathwebuli ngokwanele ama-nuances ombhalo, imiphumela yokuhlola ingase ithinteke.
• Ukungafinyeleli - Ukukala i-BERTScore kumadathasethi amakhulu noma izinhlelo zokusebenza zesikhathi sangempela kungaba inselele ngenxa yezidingo zayo zekhompyutha. Ukusebenzisa i-BERTScore ezindaweni zokukhiqiza kungase kudinge amasu okuthuthukisa ukuze kuhlinzekwe ukusebenza kahle.
• Ukucaciswa kwesizinda – Ukusebenza kwe-BERTScore kungase kwehluke ezizindeni ezihlukene noma ezinhlotsheni zombhalo ezikhethekile. Ukuvumelanisa imethrikhi nezizinda ezithile noma imisebenzi kungase kudinge ukucushwa kahle noma ukulungiswa ukuze kukhiqizwe ukuhlola okunembile.
• Ukutolika – Nakuba i-BERTScore inikeza ukuhlola okuphelele okusekelwe ekushumekeni komongo, ukuhumusha izizathu ezithile ezibangela amaphuzu afanayo akhiqizwe ngethokheni ngayinye kungaba yinkimbinkimbi futhi kungase kudinge ukuhlaziywa okwengeziwe.
• Ukuhlola okungenazithenjwa – Nakuba i-BERTScore inciphisa ukuthembela kuzifinyezo zereferensi ukuze zihlolwe, le ndlela engenazithenjwa ingase ingazithwebuli ngokugcwele zonke izici zekhwalithi yokufingqa, ikakhulukazi ezimeni lapho izinkomba eziklanywe umuntu zibalulekile ukuze kuhlolwe ukuhambisana nokuhambisana kokuqukethwe.

Ukwamukela le mikhawulo kungakusiza wenze izinqumo ezinolwazi lapho usebenzisa i-BERTScore njengemethrikhi yokuhlola imisebenzi yokufingqa, inikeze ukuqonda okunokulinganisela kwamandla ayo nezingqinamba.

Isetshenziswa nini i-BERTScore

I-BERTScore ingahlola ikhwalithi yokufingqa kombhalo ngokuqhathanisa isifinyezo esikhiqiziwe nesifinyezo sereferensi. Isebenzisa amanethiwekhi e-neural afana ne-BERT ukukala ukufana kwe-semantic ngale kokumataniswa kwegama noma imisho ngqo. Lokhu kwenza i-BERTScore isebenziseke kakhulu uma ukwethembeka kwe-semantic kugcina incazelo ephelele nokuqukethwe kubalulekile emsebenzini wakho wokufingqa. I-BERTScore izonikeza amaphuzu aphezulu ezifinyezweni ezidlulisa ulwazi olufanayo njengesifinyezo sereferensi, ngisho noma zisebenzisa amagama ahlukene nezakhiwo zemisho. Okubalulekile ukuthi i-BERTScore ilungele imisebenzi yokufingqa lapho ukugcina incazelo egcwele ye-semantic hhayi nje amagama angukhiye noma izihloko kubalulekile. Ukulinganisa kwayo kwe-neural okuthuthukisiwe kuyivumela ukuthi iqhathanise incazelo ngale kokumataniswa kwamagama kwezinga eliphezulu. Lokhu kuyenza ifaneleke ezimweni lapho umehluko ocashile emagameni ungashintsha kakhulu incazelo kanye nemithelela. I-BERTScore, ikakhulukazi, yenza kahle kakhulu ekuthwebuleni ukufana kwe-semantic, okubalulekile ekuhloleni ikhwalithi yezifinyezo ezingabonakali njengalezo ezikhiqizwe amamodeli we-Retrieval Augmented Generation (RAG).

Izinhlaka zokuhlola eziyimodeli

Izinhlaka zokuhlola amamodeli zibalulekile ekulinganiseni ngokunembile ukusebenza kwamamodeli ahlukahlukene okufingqa. Lezi zinhlaka ziwusizo ekuqhathaniseni amamodeli, zihlinzeka ngokuhambisana phakathi kwezifinyezo ezikhiqiziwe nokuqukethwe komthombo, nokukhomba ukushiyeka ezindleleni zokuhlola. Ngokwenza ukuhlola okuphelele kanye nokulinganisa okungaguquki, lezi zinhlaka zithuthukisa ucwaningo lokufinyezwa kombhalo ngokumela izinqubo zokuhlola ezisezingeni futhi zivumele ukuqhathaniswa kwamamodeli anezici eziningi.

Ku-AWS, i Umtapo wezincwadi we-FMeval Ngaphakathi I-Amazon SageMaker Cacisa iqondisa ukuhlolwa nokukhethwa kwamamodeli ayisisekelo (ama-FM) emisebenzi efana nokufingqa kombhalo, ukuphendula imibuzo, nokuhlukanisa. Ikunika amandla okuhlola ama-FM ngokusekelwe kumamethrikhi afana nokunemba, ukuqina, ubuciko, ukwenzelela, kanye nobuthi, okusekela kokubili ukuhlola okuzenzakalelayo nokwe-loop komuntu kuma-LLM. Ngokuhlola okusekelwe ku-UI noma okuhleliwe, i-FMEval ikhiqiza imibiko enemininingwane enokubonwayo ukuze ilinganise ubungozi bemodeli njengokungalungi, ubuthi, noma ukuchema, ukusiza izinhlangano ukuthi zihambelane nemihlahlandlela yazo ekhiqizayo ye-AI. Kulesi sigaba, sibonisa indlela yokusebenzisa umtapo wezincwadi we-FMEval.

Linganisa uClaude v2 ngokunemba kokufingqa usebenzisa i-Amazon Bedrock

Amazwibela ekhodi alandelayo ayisibonelo sendlela yokusebenzisana nemodeli ye-Anthropic Claude usebenzisa ikhodi yePython:

import json
# We use Claude v2 in this example.
# See https://docs.anthropic.com/claude/reference/claude-on-amazon-bedrock#list-available-models
# for instructions on how to list the model IDs for all available Claude model variants.
model_id = 'anthropic.claude-v2'
accept = "application/json"
contentType = "application/json"
# `prompt_data` is structured in the format that the Claude model expects, as documented here:
# https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html#model-parameters-claude-request-body
prompt_data = """Human: Who is Barack Obama?
Assistant:
"""
# For more details on parameters that can be included in `body` (such as "max_tokens_to_sample"),
# see https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-claude.html#model-parameters-claude-request-body
body = json.dumps({"prompt": prompt_data, "max_tokens_to_sample": 500})
# Invoke the model
response = bedrock_runtime.invoke_model(
body=body, modelId=model_id, accept=accept, contentType=contentType
)
# Parse the invocation response
response_body = json.loads(response.get("body").read())
print(response_body.get("completion"))

Ngamagama alula, le khodi yenza lezi zenzo ezilandelayo:

1. Ngenisa imitapo yolwazi edingekayo, okuhlanganisa json, ukusebenza nedatha ye-JSON.
2. Chaza i-ID yemodeli ngokuthi anthropic.claude-v2 bese usetha uhlobo lokuqukethwe lwesicelo.
3. Dala prompt_data okuguquguqukayo okwakha idatha yokufaka yemodeli ye-Claude. Kulokhu, ibuza umbuzo othi "Ubani uBarack Obama?" futhi ilindele impendulo evela kumodeli.
4. Yakha indikimba yento ye-JSON enegama elihlanganisa idatha yokwaziswa, futhi ucacise amapharamitha engeziwe njengenani eliphakeme lamathokheni azokhiqizwa.
5. Cela imodeli kaClaude usebenzisa bedrock_runtime.invoke_model namapharamitha achaziwe.
6. Hlunga impendulo kumodeli, khipha ukuqedwa (umbhalo okhiqiziwe), bese ukuphrinte.

Qiniseka ukuthi Ubunikazi be-AWS Nokuphathwa Kokufinyelela (IAM) indima ehlobene ne- I-Amazon SageMaker Studio iphrofayili yomsebenzisi inokufinyelela ku- I-Amazon Bedrock amamodeli akhethiwe. Bukela ku Izibonelo zenqubomgomo esekelwe kubunikazi be-Amazon Bedrock ukuze uthole isiqondiso sezinqubo ezihamba phambili nezibonelo zezinqubomgomo ezisuselwe kubunikazi be-Amazon Bedrock.

Kusetshenziswa umtapo wezincwadi we-FMEval ukuhlola okukhiphayo okufingqiwe okuvela ku-Claude

Sisebenzisa ikhodi elandelayo ukuze sihlole okukhiphayo okufingqiwe:

from fmeval.data_loaders.data_config import DataConfig
from fmeval.model_runners.bedrock_model_runner import BedrockModelRunner
from fmeval.constants import MIME_TYPE_JSONLINES
from fmeval.eval_algorithms.summarization_accuracy import SummarizationAccuracy
config = DataConfig(
    dataset_name="gigaword_sample",
    dataset_uri="gigaword_sample.jsonl",
    dataset_mime_type=MIME_TYPE_JSONLINES,
    model_input_location="document",
    target_output_location="summary"
)
bedrock_model_runner = BedrockModelRunner(
    model_id=model_id,
    output='completion',
    content_template='{"prompt": $prompt, "max_tokens_to_sample": 500}'
)
eval_algo = SummarizationAccuracy()
eval_output = eval_algo.evaluate(model=bedrock_model_runner, dataset_config=config,
prompt_template="Human: Summarise the following text in one sentence: $featurennAssistant:n", save=True)

Kumazwibela ekhodi esandulele, ukuhlola ukufingqwa kombhalo kusetshenziswa umtapo wezincwadi we-FMEval, siqedela izinyathelo ezilandelayo:

1. Dala ModelRunner ukwenza isicelo ku-LLM yakho. Umtapo wezincwadi we-FMEval uhlinzeka ngosekelo olwakhelwe ngaphakathi lwe I-Amazon SageMaker iziphetho kanye I-Amazon SageMaker JumpStart Ama-LLM. Ungakwazi futhi ukunweba i- ModelRunner isikhombimsebenzisi sanoma yimaphi ama-LLM asingathwa noma yikuphi.
2. Sebenzisa okusekelwe eval_algorithms njengobuthi, ukufinyezwa, ukunemba, i-semantic, nokuqina, ngokusekelwe ezidingweni zakho zokuhlola.
3. Enza ngendlela oyifisayo imingcele yokucushwa kokuhlola esimweni sakho esithile sokusebenzisa.
4. Sebenzisa i-algorithm yokuhlola enamadathasethi akhelwe ngaphakathi noma angokwezifiso ukuze uhlole imodeli yakho ye-LLM. Idathasethi esetshenziswe kulesi simo ithathwe kokulandelayo GitHub repo.

Bheka ku umhlahlandlela wonjiniyela nezibonelo ukuze kusetshenziswe okuningiliziwe kwama-algorithms okuhlaziya.

Ithebula elilandelayo lifingqa imiphumela yokuhlola.

Bheka isampula ibhuku lokubhala ukuze uthole imininingwane eyengeziwe mayelana nokuhlolwa kokufingqa esixoxile ngakho kulokhu okuthunyelwe.

Isiphetho

I-ROUGE, METEOR, kanye ne-BERTScore zonke zikala ikhwalithi yezifinyezo ezikhiqizwa umshini, kodwa zigxila ezicini ezihlukene ezifana nokugqagqana kwe-lexical, ukushelela, noma ukufana kwe-semantic. Qiniseka ukuthi ukhetha imethrikhi ehambisana nalokho okuchaza "okuhle" esimweni sakho sokusebenzisa isifinyezo esithile. Ungasebenzisa futhi inhlanganisela yamamethrikhi. Lokhu kunikeza ukuhlolwa okuhlanganiswe kahle futhi kuvikele ebuthakathakeni obungaba khona banoma iyiphi imethrikhi ngayinye. Ngezilinganiso ezifanele, ungakwazi ukuthuthukisa izifinyezo zakho ngokuphindaphindiwe ukuze uhlangabezane nanoma yimuphi umbono wokunemba obaluleke kakhulu.

Ukwengeza, ukuhlolwa kwe-FM ne-LLM kuyadingeka ukuze ukwazi ukukhiqiza lawa mamodeli ngezinga eliphezulu. Nge-FMEval, uthola isethi enkulu yama-algorithms akhelwe ngaphakathi kuyo yonke imisebenzi ye-NLP, kodwa futhi ithuluzi elibukhali neliguquguqukayo lokuhlola ngezinga elikhulu lamamodeli akho, amasethi edatha, nama-algorithms. Ukukhuphula, ungasebenzisa le phakheji kumapayipi akho e-LLMOps ukuze hlola amamodeli amaningi. Ukuze ufunde kabanzi mayelana ne-FMEval ku-AWS nokuthi ungayisebenzisa kanjani ngempumelelo, bheka Sebenzisa i-SageMaker Clarify ukuze uhlole amamodeli olimi amakhulu. Ukuze uthole ukuqonda okwengeziwe kanye nemininingwane ngamakhono e-SageMaker Cacisa ekuhloleni ama-FM, bona I-Amazon SageMaker Clarify Ikwenza Kubelula Ukuhlola Nokukhetha Amamodeli Esisekelo.

Mayelana Ababhali

Dinesh Kumar Subramani i-Senior Solutions Architect ozinze e-Edinburgh, eScotland. Ugxile kwezobuhlakani bokwenziwa nokufunda ngomshini, futhi uyilungu lomphakathi wenkundla yezobuchwepheshe e-Amazon. U-Dinesh usebenzisana eduze namakhasimende kaHulumeni Omkhulu wase-UK ukuze axazulule izinkinga zawo esebenzisa izinsiza ze-AWS. Ngaphandle komsebenzi, u-Dinesh ujabulela ukuchitha isikhathi esihle nomndeni wakhe, edlala i-chess, nokuhlola izinhlobonhlobo zomculo.

Pranav Sharma ungumholi we-AWS wokushayela ubuchwepheshe kanye nezinhlelo zokuguqula amabhizinisi kulo lonke elaseYurophu, eMpumalanga Ephakathi, nase-Afrika. Unolwazi lokuklama nokusebenzisa izinkundla zobuhlakani bokwenziwa ekukhiqizeni ezisekela izigidi zamakhasimende futhi zilethe imiphumela yebhizinisi. Udlale ezobuchwepheshe kanye nezindima zokuhola abantu ezinhlanganweni ze-Global Financial Services. Ngaphandle komsebenzi, uthanda ukufunda, ukudlala ithenisi nendodana yakhe, nokubuka amafilimu.

• I-SEO Powered Content & PR Distribution. Khuliswa Namuhla.
• I-PlatoData.Network Vertical Generative Ai. Zinike Amandla. Finyelela Lapha.
• I-PlatoAiStream. I-Web3 Intelligence. Ulwazi Lukhulisiwe. Finyelela Lapha.
• I-PlatoESG. Ikhabhoni, I-CleanTech, Amandla, Environment, Ilanga, Ukuphathwa Kwemfucuza. Finyelela Lapha.
• I-PlatoHealth. I-Biotech kanye ne-Clinical Trials Intelligence. Finyelela Lapha.
• Source: https://aws.amazon.com/blogs/machine-learning/evaluate-the-text-summarization-capabilities-of-llms-for-enhanced-decision-making-on-aws/