Just like we cannot explain how the human body works and how our brain works, there is a possibility that we won't be able to fathom how the AI makes decisions. This is central to the issue of losing control to AI and in some ways the issue of concentration of power. Understanding how AI would use arguments to influence opinions could be one of the ways we could understand it. I don't look at this finding just to be helpful for social networks, rather a finding that aligns with out major priorities of AI alignment and control theory. And my vote is yes, because I believe that such curiosities must be funded in order to have a breakthrough on understanding how to control AI in the future.

As Stuart Russell mentions on his book Human Compatible that AI should follow the following principles:

1. The machine’s only objective is to maximise the realisation of human preferences.

2. The machine is initially uncertain about what those preferences are.

3. The ultimate source of information about human preferences is human behaviour.

But, we are not yet sure how AI could use arguments to influence humans on changing their realisation of preferences in the first place, which is a central assumption to that above principles (That human preferences will not be influenced by AI)

So, heading in this direction no matter how narrow the research is a addition to the body of knowledge.

Please note that this account bought some shares in this market in error. Once this error was noticed, we then sold them all. This account has a policy of not betting in its own markets.

I’d be willing to bet this results in inferior results (and less publicity inside or outside the AI community) than a few hours of Yannic hacking while streaming:

https://m.youtube.com/watch?v=efPrtcLdcdM

Vet this with anyone who knows anything about machine learning and they’ll tell you the approach is nonsensical.

Whatever experience the team has in “use of language” this not only intends to produce nothing (a risk report about people making “AI-driven propaganda”?) but the methods proposed are facially wrong.

Bizarre mix of linguistics, hand-wavy appeals to AI, and pretending to understand LLMs or RL and the like, but not in any way equipped to do what they propose—leaving aside that even if achieved, it will end up in a file drawer and aims to solve zero issues actually faced by social media companies, AI research departments, or people actually working in the area.

@Gigacasting I'm not convinced. I'm more excited about the step of empirically studying LLM arguers than developing models of Bayesian reasoning, but reading through the proposal it seems like the team is broadly qualified to execute the work and the proposal isn't obviously bad.

You say that if achieved it won't have much of an effect but I'm not sure this is true. The impact of research is heavy-tailed and there seems like a non-negligible chance this work could lead to very useful insights regarding pushing online discourse in a good direction, even if the modal outcome isn't useful.

One thing I would be curious about is how the team understand their work in relation to https://truthful.ai/

(FWIW, I've published some stuff in NLP https://scholar.google.com/citations?user=Q33DXbEAAAAJ&hl=en and worked on applied NLP for about a year, https://elicit.org/)

^ legit ML background

There are ongoing major competitions in Deepfake detection, usually with adversarial models (some teams develop, others detect)

None of their nonsense about strength of arguments or the linguistics and Bayesian this and that or “agents” or “neural language” is particularly coherent and all indicates they probably have zero ability to even attempt basic GANs or adversarial training — the fact the paper is so focused on word-salad is a huge red flag.

If someone wanted to figure out how well LLMs can be trained to fool people and persuade, and quantify current and future capabilities (a learning curve across parameter sizes/years) and work on strategies to mitigate this, zero of them would have this background:

The FTX competitions in Trojan detection are an example of money well spent, but there’s a reason no important machine learning paper has come from academic lab in half a decade—and every reason to think the grant writers are not even in the 10,000 most competent people to work on what they propose.

@Gigacasting Thanks for elaborating!

Here are some things I agree with you about:

1. I'm generally pretty pessimistic about the direct impact of work in academic labs compared to more practical work in industry or even perhaps by reasonable people doing independent applied projects. A lot of academics are still way too skeptical of LLMs and deep learning in general (in the vein of Gary Marcus). I'd probably be more excited about the proposal if it was less theoretical and more applied.

2. The Center for AI Safety (CAIS) style competitions such as Trojan Detection have a higher expected value per dollar spent than this proposal.
   

Here are some things I'm open to having my mind changed on, but think I disagree with you about:

1. The more theoretical Bayesian, linguistics, pragmatics etc. stuff in the proposal isn't word salad and while I'm not super excited about more of it, it doesn't provide strong evidence that the proposal authors won't be able to do more applied work as well (and pushing the proposal authors toward more applied work could potentially be good).

2. I'm not sure if they're in the top 10,000 people in the world to execute the empirical portion of their plan, but they're probably in the top 100,000.

3. The fact that CAIS competitions have higher expected value per dollar spent doesn't in and of itself mean we shouldn't also fund this proposal, as the funding bar may be significantly lower than CAIS. This proposal still feels near the borderline to me.

Also:

there’s a reason no important machine learning paper has come from academic lab in half a decade

I'm guessing you are intentionally being hyperbolic? If not I think I can find some examples of at least a few obviously important papers from academic labs, even though I agree with you directionally.

“I'm not sure if they're in the top 10,000 people in the world to execute the empirical portion of their plan, but they're probably in the top 100,000.”

I laughed.

—-

I’m sure they’re very good at word-celing, but their mention of irrelevant garbage is a sign the money is just being burned. They are pretending to be capable of ML, but all the methods mentioned show they aren’t.

I’ll pro bono set up a contest to do the “project” — we will use a Twitter dataset and see if teams can create fake tweets and if others can detect them, and use memory/compute tiers to establish a scaling curve to predict the year in which offense overtakes defense, and tips to better detect bots.

We’ll sell it to Elon for $5M which the foundation can keep.

(Obviously being hyperbolic about academics, but it’s this detachment from results and grant-grift process that leads to them being so useless—at AI specifically.)

@Gigacasting There aren’t even 10,000 real Bayesian statisticians in the world. There probably won’t be for at least another 20 years. I haven’t read the full proposal yet, but the FDA will only accept clinical trials that use a couple specific Bayesian clinical trial designs and they require the statisticians who developed the methodology to sign off on the final design. I was fortunate to work with Yuan Ji of University of Chicago for awhile who is arguably the best Bayesian statistician in medicine. https://www.cytel.com/news/cytel-acquires-laiya-consulting?hs_amp=true

This is just horrible. in every way.

@Gigacasting why? seems non-obvious to me

@Gigacasting "to influence opinions, even those of pure rational interpreters" did it for me