On Dec. 31 2024 I will use the OpenAI API to query the highest capability model available at the time following the methodology outlined here and here. I will use the same prompts and sentences. Will at least five languages among those I tested today achieve correct translation for at least 9 out of 10 sentences in both the technical and non-technical categories?
Currently no language passes this bar. The list of languages is as follows: Swahili, Hausa, Igbo, Yoruba, Berber, Oromo, Amharic, Fula, Malagasy, Somali, Zulu, Xhosa, Shona, Wolof, Tigrinya, Kinyarwanda, Afrikaans, Kikuyu, Chewa, Tswana, Luganda, Sotho, Mandinka, Ewe, Lingala, Bambara, Kpelle, Akan, Serer, Krio, Kikongo, Mossi, Nuer, Mende, Dinka, Temne, Sango, Luba-Katanga, Gikuyu, Dholuo, Soninke, Gun, Zarma, Hadiya, Tsonga, Kanuri.
Note that some widely spoken languages such as Arabic and French are not part of this list. The list is meant to include languages spoken mostly in Africa and not elsewhere. Some language names are questionable (e.g. too broad, exonyms, etc.) but I will use these names for consistency.
If the OpenAI API is no longer accessible at the time I will resolve NA.
Custom GPTs count if I will be able to query them using the regular API. I will expend a reasonable amount of effort to test a number of them as they appear. This will introduce an element of subjectivity, so I will not trade in the market myself. Here is and interaction with a custom GPT that seems to do better in Hausa: https://chat.openai.com/share/85a4c171-e1c7-4be4-a1eb-f05762024aec https://chat.openai.com/share/477bfb97-f788-4c86-9fbe-c7c2c02854f4 Check it out here: https://chat.openai.com/g/g-Cq1CBjjxo-hausagpt You can use articles from the bbc in Hausa to test it: https://www.bbc.com/hausa
