The advancement of computational power has been guided by various metrics, including FLOPS (Floating Point Operations per Second). Zettascale computing represents a major leap, equivalent to 10^21 FLOPS. It goes well beyond the current exascale systems, opening new horizons in fields like climate modeling, medical research, and artificial intelligence. Minifloat precision refers to the use of reduced precision floating-point numbers, which can be a trade-off between computation speed and accuracy. Having at least 8 bits for the minifloat precision can help ensure that the system has a reasonable level of accuracy while still enabling high-speed computation.
Will reliable sources reveal the existence of a zettascale computer with a minifloat precision of at least 8 bits before January 1, 2025?
Resolution Criteria:
This question will resolve to "YES" if, before January 1, 2025, a zettascale computer meeting the following criteria is revealed by reliable sources:
Computational Power: The computer should be capable of achieving zettascale computing, equivalent to 10^21 FLOP/s or higher. This will refer to theoretical peak performance FLOP. Reported sparse FLOP will not count.
Contiguous computer: the computer must be a single contiguous supercomputer, for example inside of a single data center, rather than being a set of computers loosely connected across space or time.
Minifloat Precision: The FLOP/s will be measured in minifloat precision of at least 8 bits, allowing for a balance between computational speed and accuracy. For example, FP8 and FP16 are acceptable binary interchange formats to use for resolution.
Independent Revelation: The computer’s capabilities must be revealed by reliable source, such as statements from a major tech firm, reporting from knowledgable journalists, or highly reputable third party bloggers. If there is credible reason to dispute the veracity of a claim about a particular computing system, this question will wait until more evidence has come to light before resolving.
I will use my discretion when resolving this question, possibly in consultation with experts in the field of high-performance computing, to ensure that the criteria are met and that the zettascale computer with at least 8 bits of minifloat precision has indeed been revealed by a reliable source.
@MatthewBarnett Does the reveal have to happen within 2025, or just the construction of the supercomputer?
@NoaNabeshima This should mean that this market resolves YES, right?
300k GPUs × 4k TeraFLOPs > 10^21 FLOPS
@NoaNabeshima Who was the single entity that ordered the 300k GPUs? A redistributor? Or a compute-centric company like Microsoft/OpenAI/Department of Energy/etc?
@NoaNabeshima You'd want about 300MW for that many GPUs and some datacenters with that much exist now I believe. That being said the h100 only has 2k dense flops, not 4k, so 300k H100s doesn't get you to a zettaflop even if you wanted to use them all as one supercomputer.
(ThoughI think if someone gets to a zettaflop in 2024 it might instead be on the B100 which comes out later this year.)
I'm guessing H100s are the most likely way for a supercomputer to have lots of 8-bit FLOP/S before 2025.
An H100 SXM gets 4k teraFLOP/S or 15.6 OOM of FLOP/S. [0]
Zettascale is 21 OOM. So there's 5.4 OOM to cover.
An H100 is $30K. So without discount from Nvidia it′s $10B to reach zettascale. That's about the size of OpenAI's Microsoft investment. It would be a substantial chunk of the H100s Nvidia sells per year at current rates, like 20%? [1]
I think if it exists it's not that likely that it'll be revealed in a reliable way before 2025. I think the probability that it will exist is 35%. And revealed is 20%? Feels like the probabilities could move quickly with info.
Some other facts:
- Sometimes the top 1 supercomputer on TOP500 uses mostly Nvidia GPUs. The most A100s currently in a TOP500 supercomputer is Leonardo with 14K which is with high probability smaller than the largest private A100 supercomputer, at least going by rumours ab the computer to train GPT-4..
- TOP500 uses FP64 for FLOP.
- I think FLOP/S in TOPS-500 is always calculated with GPUs but I'm not sure.
- The current top-1 supercomputer cost $600M.
- The FLOP/S of the top-1 supercomputer increases by an OOM every ~4.3 years [2]
[0] - https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet
[1] - https://finance.yahoo.com/news/nvidia-sold-900-tons-h100-140613078.html
[2] - https://ourworldindata.org/grapher/supercomputer-power-flops