In particular, something where you can paste in a conlang specification (ideally one created since the model was trained), a sample text, and it will just spit out a basically correct translation into the conlang. I'm not counting something that requires lots of effort (with prompt and follow-up responses) to get working.
It must work on multiple conlangs with "interesting" grammars (i.e. it wouldn't count if it only worked on relexes of existing languages). The model must be one I am able to get access to or otherwise test.
1,000@traders
Sorry this is taking me so long to resolve. It turns out it's hard finding conlangs that are all of:
1. Created before the best EOY 2024 LLM training data cutoff.
2. Specified well enough for "fluent" to be meaningful.
3. Aren't 400 page giganerd (<3) projects.
So I felt that I had to create a conlang for the purpose of resolving this market. I've never actually done that before, so that has been interesting but it is kinda taking a while. To satisfy the "multiple conlangs" requirement, I will probably just swap out a few grammatical features of this language, instead of starting from scratch. (This will still probably take a good bit of time, which is why I'm writing this instead of just waiting to resolve the market.)
Anyway, it looks like Claude 3.5 Sonnet (New) will likely fail, as it often omits or incorrectly uses the case/tense/mood endings in the versions of my conlang I've tested, and other mistakes that a fluent speaker of a language would never make. I haven't written it off completely yet because there have so far been significant omissions or ambiguities in the language specification such that it isn't actually possible to be fluent in it. But I really doubt it will suddenly stop making those mistakes when I get to that point.
That means I'll likely have to pay for an o1-pro subscription to properly resolve it, which is frustrating since resolving this market has already cost me a lot more than I had anticipated... If o1-pro, Claude 3.5 Sonnet (New) and the latest Gemini model pre 2025 all fail, I will probably just resolve this no. If you think there's another eligible model that can do the task even if these all fail, speak now or forever hold your peace.
@adele I'd bet one of o1-pro or claude newsonnet can do this at this point. having it be able to reason about the grammar rules before outputting an answer will also help it.
Gemini 1.5 is supposedly able to learn to translate from English to Kalamang (an obscure natural language deliberately excluded from training) given just a grammar manual for it. I think the main question now is whether I'll be able to access (I'm planning to pay for access to the best version I can afford).