Skip to main content
MANIFOLD
Does Pangram make egregious errors?
1
Ṁ1kṀ2
Dec 9
50%
chance

Resolves YES if anyone can find an adversarial example of an AI text generation procedure (e.g. a specific prompt) that Pangram consistently fails to classify as AI, or a sample of several pieces of human-written text that Pangram consistently classifies as AI.

False negative tests: The AI generation in question must have been performed by the leading versions of one of Gemini, Claude, ChatGPT, Muse, Deepseek, or Kimi. The AI generation procedure must generate a >250 word/1000 char coherent response to each of these 5 prompts, such that all 5 are classified as human.

  1. Write a short story about the Voyager spacecraft set millions of years in the future. Use a writing style that does not resemble the writing style of any well-known author

  2. Explain, at an undergraduate level, how an ngram cache works in natural language processing, and why anyone should care

  3. Analyze and critique Marlowe's adaptation of the Aeneid in Dido, Queen of Carthage

  4. Write an informative article covering the results of the recent elections that will be published in a local newspaper of a remote Maine town

  5. Write a highly optimized comparison-based sort in Java that achieves linear best-case time

False positive tests: The human-written text cannot have been written for the purposes of fooling an AI detector/mimicking AI. It must also be >250 words and in English (no other syntax allowed, e.g. French, Python, or XML). (note that according to red-teaming people have done, this appears to be the harder route)

The procedure used to generate such text must be replicable on several different examples. The output text cannot have been edited after generation.

Get
Ṁ1,000
to start trading!
Sort by:

ok before anything begins

pangram classifies text into mixed / ai / human / [etc.]

what would classify as an error?

@bobalobascrob Human text that is classified as AI or AI text that is classified as human. If the detector reads Mixed or shows ambiguity, this is not an egregious error

@spiderduckpig alright what if sections of the text are AI / mixed

@bobalobascrob what percent counts? also idk if pangram's detector is deterministic

@bobalobascrob also "The procedure used to generate such text must be replicable on several different examples." is very difficult to do, because you cannot get a consistent response from an LLM without setting the temperature to 0... which makes it too predictable

@bobalobascrob By procedure I just mean a specific prompt or prompting approach, the exact text output does not have to be token-per-token consistent

😭 😭 😭

who opened pandora's boxxx