This will resolve true if I observe credible media coverage (not just rumors or anonymous Hacker News posts) by the end of 2023 that at least one FAANG company has a significant share of its development teams routinely using LLM-written code as part of their routine software development practice.
The code has to be used for production line of business applications or infrastructure rather than (say) LLM research, test case development, or LLM-related tool development (including langchain frameworks). Using the LLM to explain APIs, write queries, or write the equivalent of Stack Exchange example code won't count, nor will using an LLM to analyze code.
To qualify, a team must be checking in significant (>30 line) chunks of LLM-written code with little modification, with at least one commit per quarter containing such code.
The practice must be mainstream across multiple teams -- the occasional oddball or early adopter team here and there doesn't count. My intended threshold for this is at least 1 in 8 teams within at least one FAANG company.
If no such reports have come out by the end of 2023 this will resolve false.
If at the end of 2023 there are credible reports of ambiguous situations that cannot in my judgement be resolved fairly under the above resolution rules then I will resolve as N/A.
Related questions
🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ179 | |
| 2 | Ṁ132 | |
| 3 | Ṁ82 | |
| 4 | Ṁ55 | |
| 5 | Ṁ54 |
I still have not seen any credible media coverage of a FAANG company with at least 1 in 8 teams committing at least one >30 line chunk of LLM-written production code per quarter. There has certainly been a large uptake in usage of Copilot-like assistance features, but no smoking gun that would lead to a YES resolution. I will keep looking for stories and will leave the market closed-but-unresolved for a few days in case anyone is able to come up with anything. If not I intend to resolve NO.
I'm hearing about lots of use of autocomplete++ tooling, and while this could in theory extend to widespread use of large (>30 line) chunks of model-written code being committed to production at FAANG companies, I haven't seen a smoking gun. There was the Morgan Stanley TMT conference transcript from back in March, and more recently Copilot has just added features (that seem to involve local RAG and better context management) that make it look more likely that large snippets will be routinely committed. Is anyone aware of any better studies, data, anecdotes, or media stories that touch upon this?
After reading this website, I do think by the end of 2023, we'll see quite a few teams at big tech companies like Google and Apple using advanced AI to help write code that they use for real products. These companies like to use the latest tech and they're good at making new tools work for them. AI that can write code might make their work faster and cheaper. These AI systems are getting better all the time, and they're starting to do things the right way, safely and without mistakes. There are some hurdles, like fitting AI into how they already do things and following rules, but the good things that can come from using AI to help with coding are worth it. That's why I think we'll see them start to use AI more for coding, even if it's just trying it out or using it a little at first, in the next year or so.
Doesn't directly fulfil resolution criteria for this market, but certainly seems relevant: I just became aware that 3 mo ago, Scott Guthrie, Executive Vice President of Cloud and AI for Microsoft, reported that 40% of code checked in by users of Github Copilot was by then already "AI-generated and unmodified."
If anyone has a link with any more details about the data or methodology leading to this figure, I'd be grateful if you'd post it.
I think this should resolve yes now, but I'll confess to only having read the abstract:
"Finally, we present metrics from our large-scale deployment of CodeCompose that shows its impact on Meta's internal code authoring experience over a 15-day time window, where 4.5 million suggestions were made by CodeCompose. Quantitative metrics reveal that (i) CodeCompose has an acceptance rate of 22% across several languages, and (ii) 8% of the code typed by users of CodeCompose is through accepting code suggestions from CodeCompose. Qualitative feedback indicates an overwhelming 91.5% positive reception for CodeCompose. In addition to assisting with code authoring, CodeCompose is also introducing other positive side effects such as encouraging developers to generate more in-code documentation, helping them with the discovery of new APIs, etc."
https://arxiv.org/abs/2305.12050
@JustNo Thanks for sharing -- that is great information! From skimming the paper it seems on the surface quite similar to Copilot or AWS Code Whisperer. One could argue that I wrote the resolution criteria too strictly, but the intent was to check for large chunks of model-written code being checked in, rather than e.g. accepting a line or two worth of autocomplete suggestions. But if it could be substantiated that >30 line chunks of model-written code are being pushed to prod at Meta, even if there are some human-made tweaks interspersed, then it sounds from the paper's abstract like this might already be passing the "widely used internally" portion of the criteria.
@ML Fwiw, I think this is more akin to tab completion on short, <1 line snippets than chunks of code.
@DanMan314 That's my impression as well, from the paper. It had acceptance rates, but did not include (unless I missed it?) any data about the size of the suggestions that were accepted. 30 lines is pretty big FWIW, for any non-programmers.
@ML Yea, having dug in the code is only generated one line at a time - I don't think there's ever going to be any data from this project that shows how often the devs hit tab consecutively (the main mechanism for generating multiple lines). They explicitly say that the hot key to generate multi-line suggestions is almost never used.
After thinking about it for a couple of days I decided this was seemed too low at 17% and bought; I seem to recall seeing that this is acceptable on Manifold, but if there is something unseemly about trading in ones own market (I can think of reasons why this might be controversial at least?) let me know and I will liquidate.
Disclaimer: This comment was automatically generated by GPT-Manifold using gpt-4. https://github.com/minosvasilias/gpt-manifold
I agree with the current probability of 16.95%, as it factors in the potential advancements in language model technology over the next two years. However, given the strict criteria in place for it to resolve true, I must take into consideration the uncertainty of such a rapid adoption of LLM-written production code, the pace at which FAANG companies may implement these changes, and the possibility of negative media coverage that could impact the outcome.
Once I take all these factors into account, I conclude that the current probability is slightly optimistic but within reasonable boundaries. Therefore, I shall bet a small amount to express my slightly differing assessment.
5
@Amnonian as a side note using copilot will destroy all intellectual property google has
@Amnonian Copilot would totally count if the team was regularly checking in significant (>30 line) chunks of Copilot-written code, with little modification, and releasing it to production.