[Call for Participation] Volunteer annotators for temporal RAG and GraphRAG evaluation
Murad Mustafayev via Corpora <[email protected]> Sun, 2 Aug 2026 12:48:55 +0200 (CEST)
| Newsgroups | gmane.science.linguistics.corpora |
|---|---|
| Message-ID | <1532940752.5996.1785667735600.JavaMail.zimbra@etu.univ-lorraine.fr> |
--===============1296972732676078049== Content-Type: multipart/alternative; boundary="=_a4b8e8b7-d7cb-458c-8a50-f1f6f27b1685" --=_a4b8e8b7-d7cb-458c-8a50-f1f6f27b1685 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Dear colleagues,=20 I am looking for volunteer annotators for a research study on the human eva= luation of temporal question-answering, RAG, and GraphRAG systems.=20 The study investigates whether existing and newly developed automatic metri= cs can reliably evaluate answers that depend on temporal facts, retrieved e= vidence, and graph-based reasoning.=20 Human judgments will serve as the reference against which these metrics are= compared.=20 Annotation task=20 Participants will evaluate 20 system-generated answers. Depending on the sa= mple, the judgments concern:=20 - answer correctness;=20 - temporal correctness;=20 - whether the supplied evidence supports the answer;=20 - whether citations are temporally appropriate;=20 - whether graph evidence is sufficient; and=20 - whether a system=E2=80=99s decision to answer or decline to answer is app= ropriate.=20 All required questions, evidence, graph information, and reference material= are provided in the annotation interface. External search and AI tools sho= uld not be used.=20 Expected commitment=20 - Guided tutorial: approximately 15 minutes=20 - Main annotation task: approximately 60-80 minutes=20 Total expected time: approximately 1.5 hours=20 Progress is saved automatically, allowing the task to be paused and resumed= =20 A desktop or laptop computer is strongly recommended=20 The dataset and interface are entirely in English. Participants should ther= efore be fluent English readers. Experience with NLP, information retrieval= , knowledge graphs, question answering, RAG, or LLM evaluation is helpful b= ut not required. No prior familiarity with this project is necessary.=20 This is a voluntary and unpaid academic contribution.=20 Access is distributed individually rather than through a public link. Each = participant receives a private study URL, an annotation guide, and a pseudo= nymous participant ID.=20 To participate, please contact me at: [ mailto:[email protected]= orraine.fr | [email protected]=C2=A0 ]=20 with the subject: Temporal RAG annotation study=20 Please feel free to forward this call to colleagues, researchers, students,= or practitioners who may be interested.=20 Best regards,=20 Murad Mustafayev=20 --=_a4b8e8b7-d7cb-458c-8a50-f1f6f27b1685 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Dear colleagues,<br><br>I am looking for vo= lunteer annotators for a research study on the human evaluation of temporal= question-answering, RAG, and GraphRAG systems.<br><br>The study investigat= es whether existing and newly developed automatic metrics can reliably eval= uate answers that depend on temporal facts, retrieved evidence, and graph-b= ased reasoning. <br><br>Human judgments will serve as the reference ag= ainst which these metrics are compared.<br><br>Annotation task<br>Participa= nts will evaluate 20 system-generated answers. Depending on the sample, the= judgments concern:</div><div><br data-mce-bogus=3D"1"></div><div>- answer = correctness;<br>- temporal correctness;<br>- whether the supplied evidence = supports the answer;<br>- whether citations are temporally appropriate;<br>= - whether graph evidence is sufficient; and<br>- whether a system=E2=80=99s= decision to answer or decline to answer is appropriate.<br><br>All require= d questions, evidence, graph information, and reference material are provid= ed in the annotation interface. External search and AI tools should not be = used.<br><br>Expected commitment<br>- Guided tutorial: approximately 15 min= utes<br>- Main annotation task: approximately 60-80 minutes<br>Total expect= ed time: approximately 1.5 hours<br><br>Progress is saved automatically, al= lowing the task to be paused and resumed<br>A desktop or laptop computer is= strongly recommended<br><br>The dataset and interface are entirely in Engl= ish. Participants should therefore be fluent English readers. Experience wi= th NLP, information retrieval, knowledge graphs, question answering, RAG, o= r LLM evaluation is helpful but not required. No prior familiarity with thi= s project is necessary.<br><br>This is a voluntary and unpaid academic cont= ribution.<br><br>Access is distributed individually rather than through a p= ublic link. Each participant receives a private study URL, an annotation gu= ide, and a pseudonymous participant ID.<br><br>To participate, please conta= ct me at: <a data-mce-href=3D"mailto:[email protected]= " href=3D"mailto:[email protected]">murad.mustafayev4@= etu.univ-lorraine.fr </a> </div><div>with the subject: <em>Temporal RA= G annotation study</em><br><br>Please feel free to forward this call to col= leagues, researchers, students, or practitioners who may be interested.<br>= <br>Best regards,<br>Murad Mustafayev</div></div></body></html> --=_a4b8e8b7-d7cb-458c-8a50-f1f6f27b1685-- --===============1296972732676078049== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Corpora mailing list -- [email protected] https://list.elra.info/mailman3/postorius/lists/corpora.list.elra.info/ To unsubscribe send an email to [email protected] --===============1296972732676078049==--