Will the project "The Adversarial Collaboration Project" receive any funding from the Clearer Thinking Regranting program run by ClearerThinking.org?
Remember, betting in this market is not the only way you can have a shot at winning part of the $13,000 in cash prizes! As explained here, you can also win money by sharing information or arguments that change our mind about which projects to fund or how much to fund them. If you have an argument or public information for or against this project, share it as a comment below. If you have private information or information that has the potential to harm anyone, please send it to clearerthinkingregrants@gmail.com instead.
Below, you can find some selected quotes from the public copy of the application. The text beneath each heading was written by the applicant. Alternatively, you can click here to see the entire public portion of their application.
Why the applicant thinks we should fund this project
The Adversarial Collaboration Project supports scholars with clashing theoretical-ideological views to engage in best practices for resolving scientific disputes. There are many ongoing debates in the social and behavioral sciences that influence policy and organizational decision-making in which both sides have become entrenched, research findings have become politicized, and scientific progress has come to a halt. We seek to stimulate a culture shift among social and behavioral scientists whose work touches on polarizing topics with policy significance by encouraging disagreeing scholars to work together to make scientific progress.
As originally conceived by Economics Nobel Prize Laureate, Daniel Kahneman, adversarial collaborations call on scholars to: (1) make good faith efforts to articulate each other’s positions (so that each side feels fairly characterized, not caricatured); (2) work together to design methods that both sides agree constitute a fair test and that they agree, ex ante, have the potential to change their minds; (3) jointly publish the results, regardless of “who wins, loses or draws” on which topics. Each collaborator serves as a check on their adversary to confirm that the hypotheses are falsifiable, the scientific tests are fair, and the interpretations accurately characterize the findings. Because adversarial collaborations restrict scholars’ abilities to rig methods in favor of their own hypothesis and to dismiss unexpected results, adversarial collaborations are likely to advance debates faster and generate more reliable knowledge than traditional approaches.
This initiative aims to discover best practices for participating in adversarial collaborations and to normalize such practices in order to improve the accuracy and efficiency of the social sciences and its reputation among policy makers and the public.
Here's the mechanism by which the applicant expects their project will achieve positive outcomes.
We will persuade our peers that ACs are a more rigorous and scientifically sound approach to resolving empirical disagreements than current norms. We will elevate the prestige of ACs by engaging prominent scholars in ACs, publishing ACs in top journals, and getting endorsements from prestigious journals and scholars (e.g., Daniel Kahneman recently gave a lecture on the benefits of ACs: https://www.edge.org/adversarial-collaboration-daniel-kahneman). Scholars who participate in ACs will gain reputational benefits for their open-mindedness, willingness to engage adversaries, and commitment to rigorous research. Once scientists see that participation in ACs is a route to top tier publications and prestige, they will opt in to participate.
How much funding are they requesting?
$486,535
What would they do with the amount just specified?
$160,000 would go toward all research expenses, including all data collections, statistical software, paying research assistants, and if needed, hiring expert coders, programmers, or analysts. $101,100 would go toward salary and benefits for the Postdoctoral Scholar through 06/24 (note the majority of his salary from now through 06/23 is covered by a postdoctoral award received by Paul Connor), so this $101,100 would cover the remaining 13.5 months. $225,435 would go toward salary and benefits for the Executive Director through 12/24 (note her salary is covered by a grant through 06/23, so this $225,435 would cover the remaining 18 months).
Here you can review the entire public portion of the application (which contains a lot more information about the applicant and their project):
https://docs.google.com/document/d/17e3rjjcK8ywipBVu7rz7iG2o7xwUoaiceg6WWiwu_Gw/edit#
Sep 20, 3:44pm:
Related questions
🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ1,810 | |
| 2 | Ṁ930 | |
| 3 | Ṁ734 | |
| 4 | Ṁ561 | |
| 5 | Ṁ533 |
I believe this project's outcomes aren't likely enough to justify a 70%+ valuation.
The core work to be done is absolutely reasonable and achievable:
(1) support a series of ACs,
(2) promote ACs in relevant spaces
(3) obtain endorsements for ACs
(4) encourage public invitations for ACs
But the outcomes are dependent on variables outside the team's control:
(1*) "prestigious and admired scholars" must engage, and "top tier journals" must publish
(2*) audiences must respond to promotion. how persuasive will the evangelism be? marketers often struggle to know true conversion rates
(3*) similar to 1*, "top journals" and "top scholars" required to endorse
(4*) "so scholars who initiate and participate in ACs gain respect in the scientific community" is not readily measurable or provable.
The core mechanic of "We will persuade our peers" is useful in a pinch - it's like passing a Persuasion check with UPenn and Tetlock as modifiers. But to persuade a significant bloc of academics, journals, and funding organizations that AC is worth pursuing strikes me as a hard check to pass, even with $500k
@512yinz Hi Ian--I agree we have ambitious goals, but I do think they are attainable, and we are starting to see early signs of success in the some of the outcomes you mention.
For example, just in the past year and a half, we have persuaded dozens of scholars to participate in ACs, including Jon Haidt (NYU), Dave Rand (MIT), Anna Dreber (Stockholm School of Economics), Eric Uhlmann (INSEAD), Jay Van Bavel (NYU), Rich Petty (Ohio State), Katrin Auspurg (LMU Munich), Sander van Der Linden (Cambridge), Jan-Willem Van Prooijen (Vrije), and Greg Mitchell (UVirginia) (among others), and none of these folks had participated ACs before.
We persuaded the top methods journal in psychology (AMPPS; IF=15.8) to put out a call for a special issue on ACs (here: https://www.psychologicalscience.org/publications/ampps/adversarial-collaborations ).
Two Cornell professors emailed me just 12 days ago (cc'ing 30+ other scholars) announcing that they will be writing a paper on why they think ACs are the most important advance in social science (more important than Open Science and other initiatives). Both Dave Rand (MIT) and Steven Pinker (Harvard) recently tweeted positively about our initiative.
And I am increasingly invited to speak to new groups about the benefits of ACs both inside and outside of the behavioral sciences (e.g., I will be speaking to the American Society for Nutrition in early Nov).
I am sure most initiatives receive pushback at some point (change can be uncomfortable), but so far, the response has been largely positive. And I think these early indicators give us good reason to be optimistic.
This is more a comment on the tournament structure, but this particular question accentuates the issue well. The goal of the tournament seems to 1) have markets indicate community opinion of projects, and 2) encourage insightful analysis for and against in the comments. I think its generally pretty good at both aims, but it is still somewhat flawed.
My position on this grant is that it will probably get funded and I don't agree with that choice. If the goal of the market is to reflect sentiment toward projects, then my incentive should be to place a large "No" bet to reflect my disagreement.
But I don't think that's the right strategy. On one hand, I can comment "No" but bet "Yes", but that risks undermining my own argument by implying I don't believe it's strong enough to fully persuade Clearer Thinking. Or I can comment "No" and bet "No", likely shooting myself in the foot for the prediction side of the tournament. It feels like a lose-lose, with the net result for me being a muted "No" bet, far smaller than it would otherwise be. But that discrepancy between bets and thoughts (here and on several other questions) means the market isn't optimally reflecting what it's meant to reflect.
(Just to be crystal clear, I'm not at all saying this project is high just because the system is biased. I'm only saying my incentives aren't fully aligned with the apparent goals of the tournament.)
I'm curious how others have interpreted their incentives in this tournament, and especially what the impact has been on their bets. I've seen @Adam vote directly against his read in a different take on the incentives at play.
@GregJustice I bet what I think will happen, comment when I think I have interesting input on what will happen- I don't expect the probability alone, rather than the comments, to be taken as a strong direction in a way that would lead to self-fulfilling prophecies, and I think they'll listen to the arguments on the merits more than whether I'm prepared to bet they'll be persuasive.
@GregJustice I think the rational way to maximize your potential winnings is to:
- predict the projects that will receive funding, regardless of your opinion
- comment on projects that you feel you have a persuasive or insightful angle on
and I think if we all approach it that way, the predictions will have a bit of 'playing the ref' bias. However, the commentary may be the more valuable outcome for Clearer Thinking? And there's also an unknown overlap between "they should fund this" and "i think they will fund this" that I think makes it impossible to draw out the discrete drivers of investment.
All that to say, thanks for the comment on the metagame, I'd been thinking about it as well.
Adversarial collabs are pretty hot in the open science movement right now too (see this pretty cool talk by Dr Tim Rakow with variety of examples) and my impression is they can be done at much cheaper scale. I think there's a lot of value here still but it's a big buy-in. Like Nuño I'm not too big on the funding structure. Shouldn't there be funding for post-docs from other grants anyway in most cases? And I guess I'd be more excited if the topic of the AC was set in stone. Anti-Black implict bias is important (a lot of the literature on implicit bias is methodologically messy) but existential risk is understudied in these frameworks.
I have participated in two adversarial collaborations, there is a reason Scott is not hot on them, that the adversarial collaboration website is dead, and that it's Discord died two years ago, etc. As an intentional joint product, it's missing key parts of what we might call the "social architecture which creates interesting and useful results". i.e. there are too many failure modes.
Personally, though, I like them a lot and am always down to do another!
@JohnBuridan Oh man I really want to suggest an AC on the value of this grant, but I'm not sure I have the time lol maybe someone else in favor can do it with you.
If anyone's skeptical of AI alignment and interested in an AC, send me briefly your reasons and experience at [firstname].d.[lastname]@gmail.com and I might be down (I have become less skeptical over the years and am still very curious to explore disagreements in the space).
@JohnBuridan I'd briefly say that I don't defer to Scott much, I don't think one (?) failed attempt at something should update us that much on the feasibility of future attempts with a different angle, and I'm skeptical of the issues with an "intentional joint product". Seems like double cruxing is really useful, I do it all the time in conversation, and an AC is a longer version of a double crux that gets written and shared for the benefit of the rest of the world. I also don't think it needs to appeal to that much of the world for the subset of people who do ACs to create a lot of value.
@EliLifland I am a big believer in Double-crux and am developing a high school program for teaching it, doing it, and scoring performance (the Goodharting dangers are huge; first deployment next week!). However, the value of double-cruxing is mostly internal to the participants. I believe that creating a public write-up of the process and updating in a double-crux is a different product and requires a different set of skills.
@EmmaRolls Right. There is no way this should be funded. Well, unless the applicant is Daniel Kahneman.
@EmmaRolls Hi Emma--Let me see if I can clarify (I am the applicant). They requested we delete specific salary information from the public facing application (Clearer Thinking has the full details), but they said we could reply to comments. We have three primary members of our team. The PI (Phil Tetlock) has his salary covered entirely by Penn. My salary (the executive director) and our postdoc's salary are entirely grant covered, and both of our funding will run out next year. The 225k covers a year and a half of my salary and includes the 33% that Penn allocates to benefits. The salary numbers are roughly based on the minimum Penn can pay employees based on their current salary + the small raise they give each year. The 160k will go primarily to data collections. These can be a bit costly because we pay our participants a fair wage (usually ~$10-15/hour), and we strive for high quality samples (demographically representative and sufficiently powered), but 160k will still allow us to conduct several very high quality studies, which will be led by me and our postdoc. I have used grant funds to pay RAs in the past (we pay them $15/hour), but I am often able to get small awards through undergraduate funding schemes at Penn to cover these costs. I have also used grant funds to hire expert coders in the past, but this is rare, and typically experts (who are full time employed at other universities) will join our projects as co-authors rather than as paid outside consultants. Thanks very much for your interest in our project!
@GeorgeVii Dude, I have done this like 15 times in the last few days. I am trying to be quick draw McGraw so people don’t beat me to the punch but I can’t seem to remember that I can’t seem to remember that I screw this up every time when betting NO.
@GeorgeVii This is a warning message I could get behind!! “Are you sure you want to be an idiot AGAIN?” LOL.
@BTE I noticed this and had assumed this was some clever trick involving exploiting bot behaviour or limit orders that my brain was simply operating on insufficient dimensions to understand!
@jbeshir You flatter me by thinking I was being clever. I bet it’s because I called the Musk-Twitter court case 4 months before it even existed. 😎😉
@MichaelDickens Thomas Peterffy is a genius, but has the ethics of a payday loan shark. No chance that was not intentionally manipulative.
@GeorgeVii More like impatient. Like making bigger bets but that is clearly not the way to go when everybody is day trading the market because money is on the line. Haha.
@sketch I see. idk big bets seem fine if you believe in your prediction & hold it, higher variance tho so use if you want chance of being top. swing trading is the safer/lower variance choice (if you just want a prize but not the top then that might be the way to go)