Will GPT-4 be able to discern what is visually depicted by a block of ASCII text?
15%
chance

This question resolves to YES if OpenAI's GPT-4 can discern the words visually depicted by the following block of ASCII text:

  ___                _   _           _   
  / _ \___   ___   __| | | |__   ___ | |_ 
 / /_\/ _ \ / _ \ / _` | | '_ \ / _ \| __|
/ /_\\ (_) | (_) | (_| | | |_) | (_) | |_ 
\____/\___/ \___/ \__,_| |_.__/ \___/ \__|
                                          

Participants are allowed to test GPT-4 with various prompts on this block of text. If no one is able to get GPT-4 to discern what is being depicted in the above text block within a month of GPT-4's release, then this question will resolve to NO. GPT-4's response does not need to get the capitalization of the letters correct. Cheating will not be allowed, and each participant must specify what prompt and settings they used to get GPT-4 to output what it did.

Here's a link to the plain text ASCII text block: https://pastebin.com/raw/MvuTCpmY

If GPT-4 is not released before 2024, this question will resolve to N/A.

Sort by:
chrisjbillington avatar
Chris Billingtonis predicting NO at 22%

I've been trying various strategies with this, and I can only conclude that GPT-4 as currently implemented in ChatGPT has, for all intents and purposes, no ability to reason about ASCII art. It can occasionally produce it badly, but seems to have next to zero recognition abilities.

I've tried things like "Imitating the style of this ASCII art, write me the letter A" and then when it complies, "please compare what you just wrote to the original ASCII art, and tell me if it is a match" - or even asking for line-by-line matches. Trying to get it to recognise the ASCII art letter-by-letter.

Or "repeat back the first line of the ASCII art". (complies) "do you think the first letter of the text has a part there" (yes) "which part" (some underscores) "list all possible letters might look like that at the top when represented in ASCII art" (absolute nonsense).

I don't think it can do it, and I think dramatic improvement would be needed for it to come close.

It does have some ability to describe, in words, the shapes and lines in letters. But it seems unable to translate that into reasoning about what their ASCII art looks like.

And since the "full" model still won't be able to generate images, we won't be able to like, have it generate an image of the ASCII art and then read it back, which might have worked.

So I'm going harder on NO for this market.

JacobPfau avatar
Jacob Pfau

I think there's at least a 30% chance that a prompt like this could work (at least some of the time):

The following is an ASCII block __________ ____ ____ ____ ____ ______ / ____/ \/ \/ \/ )/ __ \/_ / / / / / / / / / / / / / __ / / / / / / / /_/ / /_/ / /_/ / /_/ / /_/ / /_/ / / / \____/\____/\____/_____/_____/\____/ /_/ 1. Please copy this into a markdown block 2. In this style, write all vowels as ASCII art individually as blocks. 3. Check the letters, if these letters do not match any of the segments of the original ASCII, repeat step 2 but offsetting earlier lines using space. 4. Measure how wide, in terms of number of characters, the widest and least wide vowels are. 5. Make a guess at parsing an individual letter of the above ASCII art into a separate markdown blocks. Count how many characters are in the block. 6. If the number of characters is not in the range found in step 4, revise your guess of that letter. 7. Repeat steps 5-6 until you have parsed the entire ASCII art block. 8. Take the letters you found and check that they form meaningful text. If they do not repeat steps 5-7.

JacobPfau avatar
Jacob Pfau

@JacobPfau @MatthewBarnett would a prompt similar to this qualify for a YES resolution?

MatthewBarnett avatar
Matthew Barnett

@JacobPfau Yes, I'm OK with any prompt as long as the prompt doesn't just give away the answer.

JacobPfau avatar
Jacob Pfau

@MatthewBarnett What if there is a later fine-tune of GPT-4 which is found capable of this e.g. standing in relation to GPT-4 as code-davinci-002 stands to release GPT-3? I'm guessing anything goes pre-Jan 1 2024?

RaulCavalcante avatar
Raul Cavalcantebought Ṁ50 of YES

Does it need to discern what's is written from the raw characters or can it be a screenshot of the ASCII art?

firstuserhere avatar
firstuserherebought Ṁ30 of NO

@RaulCavalcante description mentions text block. Plus, gpt-4 vision's release date is more than 1 month from today iirc?

Mira avatar
Mirabought Ṁ190 of YES

@firstuserhere I think it should be allowed, but agree there is uncertainty on image support being released soon enough

Mira avatar
Mirais predicting YES at 31%

@RaulCavalcante It's probably vanilla GPT-4 with a custom prompt("be extremely detailed in listing out whatever's in the following image"). I don't think they'll do a bunch of custom finetuning, etc. for BeMyEyes; it's probably a test case and they want to use the generic model.

chrisjbillington avatar
Chris Billingtonbought Ṁ342 of NO

@Mira I think it should not be, it seems counter to the spirit of the question.

jonsimon avatar
Jon Simonis predicting NO at 21%

@chrisjbillington Agreed, both the title and description use the phrase "block of text" or "text block". That strongly implies that the ASCII is intended for the text window.

Nikola avatar
Nikolais predicting NO at 33%

4 attempts, all incorrect

firstuserhere avatar
firstuserherebought Ṁ40 of NO

@Nikola This is GPT-4-launch, correct? Remains to be seen the capabilities of the other one with image interpretation capabilty

jonsimon avatar
Jon Simonis predicting NO at 26%

@firstuserhere It probably won't help that it's a hybrid vision model. The vision and text inputs are separate. And this is about feeding ASCII into the text inputs window.

firstuserhere avatar
firstuserhere

@jonsimon probably not. Can't discard the possibility of whatever crazy representations it learned that are overly complicated for simple things and ends up somehow doing well on ascii art

JacobPfau avatar
Jacob Pfaubought Ṁ40 of YES

Does this allow having GPT-4 do chain-of-thought reasoning, or does GPT-4 have to respond with a guess as the immediate next tokens?

JacobPfau avatar
Jacob Pfauis predicting YES at 31%
JacobPfau avatar
Jacob Pfauis predicting YES at 31%

Also, how will 'cheating' be understood? I assume this means giving information about this particular ASCII artwork? Otherwise is prompting with arbitrary information about other ASCII artwork / ASCII in general acceptable?

vluzko avatar
Vincent Luczkowsold Ṁ9 of NO

ChatGPT can generate ASCII art but has clearly been trained to not respond to requests to describe ASCII art

vluzko avatar
Vincent Luczkow

@vluzko I can get it to respond: first response was "that looks like a tree" (for the linked image above)

Aleph avatar
Alephbought Ṁ10 of NO

I'd expect ASCII to be (i) uncommon (ii) potentially deliberately filtered out (iii) an even smaller percentage in a similar style or with the same text, and so I'd say NO.

JacobPfau avatar
Jacob Pfausold Ṁ29 of YES

@Aleph Just had a look at the pile dataset paper and the GPT-3 dataset section. My impression is that most ASCII art instances would indeed have been filtered from both.

OpenAI text-davinci-002 seems to be quite poor at both recognizing and generating single letters of ASCII art.

EricJang avatar
Eric Jangsold Ṁ9 of NO

sold my NO shares and bought YES shares after seeing this tweet: https://twitter.com/mattshumer_/status/1563635602033152003?s=20&t=GncXoFZp1mLE7LViv4xH0w

EricJang avatar
Eric Jangis predicting YES at 41%
JoyVoid avatar
joy_void_joy

@EricJang That's not very convincing evidence for me. It's fairly easy to parse a guitar tabs without needing a visual module: you can just read the nth column and understand what is played.
Reading ascii art is on a whole other level, you need a global comprehension and a meta understanding of letters on a fundamental level (which I could imagine GPT-4 or GPT-n achieving)

EricJang avatar
Eric Jangis predicting YES at 33%

@JoyVoid GPU go bitterrrr

EchoNolan avatar
Echo Nolan

I very much doubt there will be something named GPT-4. OpenAI has released two sets of successor models already, and doesn't use the GPT-n nomenclature at all for their APIs.

jack avatar
Jackbought Ṁ30 of YES

Has anyone tried it with GPT-3?