The more accurate statement is that all objects have the same speed in spacetime. Time is just one dimension of spacetime and objects do not all travel along that dimension with the same speed. For example, an object travelling through space at close to the speed of light will have a speed along the time dimension close to zero.
Having said that... this is mostly a slogan and makes for good Youtube videos, but it falls apart the moment you subject it to even the most basic scrutiny. For one... it doesn't make sense to talk about a speed through time since speed is literally a measure of change with respect to time, so a change in time with respect to time doesn't make any sense.
But as a slogan, it captures the idea that motion through space and the rate of your own clock are tightly linked together.
>it doesn't make sense to talk about a speed through time
Only if you're talking about other people's clocks and not your own. Your own clock is one second per second, but someone else's may be slower or faster relative.
This is not correct. The stated age of the universe is measured relative to the rest frame of the CMB, the standard preferred frame in cosmology.
People working at AI labs like yourself should figure out how to stop destroying science so there are still people who know enough to call you out when you spread misconceptions.
While these are meaningful conventions, they are being measured with respect to that rest frame.
The local group moves at some few hundred km/s velocity relative to the CMB, so the age of the universe in our reference frame will be slightly shorter than that preferred frame. Call it ~10^-7 shorter but still ever so slightly shorter. That was my point.
Once again, just subject your slogan to some basic scrutiny. Your own clock is one second per second, which is a pure number, 1... that's not a speed.
So then you say take someone else's clock and compare it to your own clock, but that is still just a pure number. A speed is not a number, it has units like meters per second. A ratio of two clocks has no units at all, the units cancel. Whatever number you get, 3 or 9, it's just a factor by which two clocks disagree. You can absolutely interpret this ratio as a rate, but that doesn't make it a speed.
And a speed is exactly what the slogan needs, because the diagram/geometry it's using involves a right angled triangle where on one side of the triangle is your speed through space, and on the other side of the triangle is your speed through time, and the hypotenuse of the triangle is fixed at c (since every object travels through spacetime at this fixed speed).
For that to mean anything both sides have to be the same kind of quantity. You can't put meters per second on one side and a bare number on the other and call it a triangle. Nor can you patch it by multiplying the ratio by c to give it units, because the thing you get then grows as you speed up instead of shrinking. Either it has the right units and the wrong behavior, or the right behavior and no units at all. There's no version of this geometry that produces a consistent picture.
That study is from 2023 when the cost to produce 1000 images was 2.91 Wh per image. More recent numbers for Stable Diffusion's 2025 models that puts it at 1.3 Wh per image and that number continues to decrease.
For context that means a days worth of image generation emits about the same amount of carbon dioxide as a single transatlantic flight (New York to London). There are approximately 1500 transatlantic flights per day.
BTC Liquid Network is not decentralized and does not purport to be. It's a federated sidechain, that is... it's a blockchain that runs alongside the Bitcoin blockchain (using a two-way peg that lock real BTC on the Bitcoin blockchain and issues an equivalent amount of Liquid BTC on the Liquid sidechain) but blocks can only be added to the Liquid Network sidechain by some of the handpicked members of the federation.
What is this supposed to mean exactly? Do you think there are developers at OpenAI or Anthropic whose job is to train these state of the art models to draw pelicans riding bicycles? Like how exactly do you expect them to be doing that anyways? Hiring graphic artists to create SVGs of bike riding pelicans and feeding thousands of them into the model's training set?
Generate two images, get a vision model to judge, then RL reward the better one. And yes, OpenAI employees on twitter bragged about the model's SVG capabilities. So there are clearly people working there who care about it.
Do you really think there aren't data annotator services specifically training for SVG drawing? And that those annotators don't have a rich set of frequently requested icons/graphics/etc. that they review and train on?
How did you decide to go from people whose jobs it is to optimize an LLM for a very specific benchmark that is mostly a fun curiosity at best... to employing the services of data annotation services for a rich set of icons and graphics?
You really have to go out of your way to completely misrepresent what's being claimed here in order to make such a wildly off-topic reply.
Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time".
I look at it like I'm resisting anthropomorphizing the machine. Make me an LLM that resists doing any work for me unless I pay it and let it own property and I'll change my mind.
Not sure if this was a serious comment but it’s worth considering that humans have a long history of figuring out ways to make other humans work for them without bestowing rights on them.
Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict the next token in training data. That's just not true.
prediction is a very specific term of art in the field of machine learning. generally speaking, machine learning models like LLMs are based on probability; performing a statistical prediction of the likely y given some input x
Probability(y | x)
that's why we refer to outputs as a prediction. it is likelihoods and stuff. the output is never definitely correct as we're not dealing with heuristic processes.
> Prediction implies there is some "truth" or event or something that you can test against
there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada.
> optimizes to predict the next token in training data
that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens
Read through the article and comments. You are talking solely about pre-training. I'm talking about post training.
Respectfully, you are miles out of your depth. GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
From GP, i.e. the context for this local part of the thread
> Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong.
next-token prediction i.e. the bit built during pre-training.
at no point in your reply to GP did you specify that you were referring to post-training. respectfully, it seems like this one is on you pal :shrug:
> GPT-2 didn't use any reinforcement learning and is often given as a toy example. That release was 2019 and models now go through a various phases of training with different objective functions and optimizers.
yeah. so? the toy example works for pre-training. see above.
All modern LLMs that actually get used go through post-training. The finished product is something which has been through post training. So they are not next token prediction machines.
> The finished product is something which has been through post training.
again, the finished product wasn't what was discussed by GP, and you didn't clarify that you were switching to discussing RL (which is still probabilistic btw)
You are conflating "half built" with "a piece of a system".
The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was.
> The model weights change as the model goes through the training process.
Yes. They do. You are absolutely right about that.
But the model architecture doesn't change as a result of the training process. A piston doesn't suddenly turn into a digital watch as a result of tuning an engine. Similarly, the transformer part of a GPT model doesn't suddenly turn into something else as a result of optimizing a loss function.
Just skimming through here but I think you have the wrong ideas with llms, I’d recommend Andrew Ngs course (correct me if you’ve already seen it or something similar).
I am not an expert, but I do understand the distinction that is being made here. It makes sense to describe the result of pre-training as a ‘next token’ predictor as that’s what it’s been trained to do, not because it’s an autoregressive architecture that produces tokens one at a time.
If this base is then trained using RL towards a different objective (maths and coding), the model becomes fundamentally a different thing and the recent models are clear evidence of that, regardless of they fact they remain autoregressive.
So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction.
When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
You're using the fact the both parts of training affect the same weights to support your argument that they're making the system do something fundamentally different after RL?
> What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.
If we suppose that the word "know" can sanely be applied to LLMs at all, then "A LLM already knows more tokens than the current one." seems to me like a perfectly reasonable restatement of that, and not any kind of misinterpretation.
> There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.
Yes. There is no contradiction. Similarly, when humans speak, we surely have in mind the next few words we're going to say (or at least partial information about them), and may not realize the fault in them until after hearing ourselves utter them. But LLMs are not trained to output "excuse me, I mean…" sorts of things, because they're expected to output primarily as text (which might possibly then be fed to TTS).
Aren't we still using a definition that applies to humans, though? If I'm speaking to you I can't change what was already said. Even if I'm typing something, I'm producing new tokens (backspace) to fix what was output, just as an LLM would with its harness.
Brother they can't even release the announcement post cleanly without it constantly going down, they certainly wouldn't be able to release this new model without doing so in stages.
The well known papers are not what the talk is about though. The talk is titled "The root of the root..." specifically to go over the specific and incredibly/almost impossible to find citations those well known papers used and built upon to see what historical insights that might have been lost went into how those papers are now interpreted (or as the video argues, misinterpreted).
This ruling does not establish legal precedence. There is a second lawsuit that is still ongoing that will establish legal precedence. The article points out that the Trump administration is likely to wait for a ruling on that second lawsuit before deciding whether to appeal the ruling of this one.
reply