A Colab notebook rebuilds the final file end to end: load data and cached embeddings, load the four trained models from Drive, run the builder. A local ROUGE harness matches the organiser's scorer, and a simulator weights local results by the test mix.
Zindi · 2026
17th of 1,651
Open-source models only
Multilingual Health QA
Find the vetted answer. Don't invent one.
Health question answering in Akan, Amharic, Luganda, Swahili and English, across Ethiopia, Ghana, Kenya and Uganda. For each question, the system finds the closest questions that health professionals have already answered, then chooses the answer with a strategy picked for each language. It placed 17th of 1,651 participants.
Role
Sole author
Context
Zindi competition; final project for Machine Learning Techniques I, African Leadership University
Data
36,501 question–answer pairs in 8 language–country subsets
Stack
AfriE5, FAISS, XLM-RoBERTa, Qwen2.5-7B with LoRA
17thof 1,651
Final rank on the Zindi leaderboard, out of 1,651 participants.
Final rank
0.6908vs 0.4908
Public score against my own TF-IDF retrieval baseline, 41% higher.
Public leaderboard
44
Experiments logged with a verdict each. 24 were rejected, and they stay in the log.
Public ledger
0
Paid APIs in the final solution. Every model is open-source and the pipeline reruns in Colab.
Reproducible
01 · Problem
Health answers in the language people ask in
Five languages, four countries, and very few NLP tools for most of them. The answers are judged against references written by health professionals.
The competition asked for answers to 2,618 health questions across eight language–country subsets. Each answer is scored against a reference: ROUGE-1 and ROUGE-L, which reward word overlap, count for 37% each, and an LLM judge of factual accuracy, completeness and language quality counts for 26%. Overlap scoring punishes paraphrase, and in health a fluent answer that is wrong is worse than no answer.
Training data per subset
Question–answer pairs in the training and validation files. Amharic has the least, under a quarter of Ugandan English.
Show as table
| Subset | Train | Validation | Total | Test questions |
|---|---|---|---|---|
| English, Uganda | 7,624 | 1,688 | 9,312 | 744 |
| Akan, Ghana | 4,455 | 1,114 | 5,569 | 492 |
| English, Ghana | 4,443 | 1,104 | 5,547 | 491 |
| English, Ethiopia | 3,915 | 564 | 4,479 | 60 |
| Luganda, Uganda | 3,383 | 846 | 4,229 | 374 |
| Swahili, Kenya | 2,070 | 518 | 2,588 | 229 |
| English, Kenya | 2,080 | 390 | 2,470 | 167 |
| Amharic, Ethiopia | 1,845 | 462 | 2,307 | 61 |
| All | 29,815 | 6,686 | 36,501 | 2,618 |
Source: competition Train, Val and Test files
02 · System
Retrieve, then choose per language
The final pipeline treats the training answers as a library. A question is matched to similar questions that already have expert answers, and the answer is chosen from the best fifteen. No single strategy was best everywhere, so each subset uses the one that scored best on its own held-out data.
Input
Test questionOne of eight language–country subsets
Retrieve
AfriE5-Large-instructAfrican-language embedder, fine-tuned question to question with hard negatives
FAISS, top 15Cosine search within the same subset; a second fine-tune blended in per language
Choose
Per-language routingConsensus vote, reranker and stitcher, or generation
Output
One answerThe same text in all three scored columns
| Subset | Strategy | Why |
|---|---|---|
| Akan, Amharic | XLM-RoBERTa cross-encoder reranks the candidates, then an extractive stitcher joins the best sentences | Single retrieved answers missed too much of the reference; verbatim sentences recover overlap without paraphrase |
| English, Ghana | Qwen2.5-7B, fine-tuned with LoRA, writes the answer | The one subset where generation clearly beat the best possible retrieved answer |
| Luganda | A Luganda-specific retriever adapter, then a consensus vote | Trained on 541 answer-similarity triplets; a small but measured gain |
| All other subsets | Consensus vote (MBR): the candidate that agrees most with the others | Dense top-1 was already strong; consensus filters out odd matches |
03 · Results
Results
Every submission was scored on the public leaderboard. The chart shows the path from my baseline to the final file, including the two generation models that scored well below it.
Final · 17th of 1,651
Public score, step by step
Each row is one submission. The line marks the TF-IDF baseline; generation models fell below it.
Show as table
| Submission | Public score | Key change |
|---|---|---|
| TF-IDF character n-gram retrieval | 0.4908 | Baseline |
| mT5-base seq2seq | 0.2396 | Generation loses overlap |
| AfriTeVa V2 seq2seq | 0.2971 | Better than mT5, still below retrieval |
| E5-base semantic retrieval | 0.5742 | Dense embeddings |
| AfriE5-Large + hard negatives | 0.6545 | African-language embedder |
| + MBR consensus selection | 0.6597 | Consensus over top-1 |
| + Extractive stitcher | 0.6650 | Verbatim sentence extraction |
| + Unicode fix and identical columns | 0.6843 | Correct Amharic and Akan scoring |
| + Embedding interpolation (V4) | 0.6898 | Second selected final |
| + Cross-encoder reranker (V7) | 0.6908 | Best, selected final |
Source: score progression in the project README and experiment log
Experiment log
44 experiments, one verdict each
Each square is one experiment in the public log.
Kept, 15Rejected, 24Partial or superseded, 2Not run, 3
Kept includes the two selected final files. Not run: ideas logged with the reasoning behind them, left for later.
Biggest gains
Where the score came from
Gain in public score from each step, largest first.
The scoring fix, highlighted, gained more than the last three model changes together.
04 · What failed
Reported in full
Most of the 44 experiments lost. The log keeps each one with its result and the reason it was dropped, so the final pipeline can be traced back to evidence.
0.24
Generation lost to my own baseline
mT5-base scored 0.2396, AfriTeVa V2 0.2971 and free-form Qwen-7B answers 0.3505, all below 0.4908 for simple retrieval. Paraphrased answers lose word overlap with the reference.
Retrieve expert-written answers; generate only where it measurably wins, for Ghanaian English.
−0.0014
A better ROUGE that lost overall
A later file raised Ghanaian English ROUGE on held-out data by 0.017, but the LLM judge column fell by 0.0063, and the leaderboard score dropped.
Rejected. The final file is the one that scored best on all three columns.
Leak
A bug in my own tuning
While re-tuning the stitcher, a held-out question could retrieve itself as its own best match, which inflated local scores.
Caught and fixed before submission; the final file was checked clean.
Forgetting
One retriever for all languages hurt the strong ones
Fine-tuning the retriever on every subset at once lifted the weak languages and degraded the strong ones.
Blend the base and fine-tuned retrievers per language, with weights chosen on held-out data.
+0.01 given up
A paid API that helped, removed
Refining answers with the Gemini API added about 0.01 to the public score, but the final solution had to use open-source models only. Gemini was used in early experiments and is not in the final pipeline.
The final pipeline is fully open-source, so anyone can rerun it.
05 · Engineering
A scoring fix worth more than a model
The organiser's scorer splits text into words with Unicode-aware rules. My first local scorer did not, so Amharic and Akan answers were being scored wrongly on my machine, and every tuning decision for them rested on that.
0.04 → 0.21
local Amharic score once the tokeniser matched the organiser's
+0.019
public score after the fix, with tuning redone and one answer in all columns: 0.6650 to 0.6843
15
candidates per question, the depth that scored best in a sweep
2,618
test answers, one per question, the same text in all three columns
Reproducible
Responsible
Answers come from a corpus written by health professionals, and the strongest languages get no free-form generation at all. ROUGE disadvantages morphologically rich languages such as Akan and Amharic, so their numbers are read with that in mind.
PythonPyTorchsentence-transformersAfriE5-Large-instructFAISSXLM-RoBERTaQwen2.5-7BPEFT / LoRABGE-M3rouge-scoreGoogle Colab
06 · Limits and next
What the score does not show
Known limitations
Most answers are retrieved, not written, so quality depends on the corpus holding a similar question. ROUGE rewards overlap with one reference and can undervalue a correct answer phrased differently, especially in Akan and Amharic. Scores come from automatic metrics and an LLM judge, not from clinicians.
Next
Three ideas are logged but not run: a larger Qwen3 embedding model as an extra retrieval leg, answer length as a reranker feature, and pseudo-labelling the reranker on test questions. Each is logged with the reasoning behind it.
In health, the safest answer is often one an expert already wrote.