I was recently able to get a workload of mine optimized enough on Zen 5 to hit a sustained 6.0 IPC/core (3.0 / thread) at 5.1 GHz. Seeing the a > 99.8% branch prediction rate and a > 99.99% L2 cache hit rate retiring > 1T instructions every 3 seconds feels amazing.
Zen5 is incredible when you're able to make the most of it. I’m super excited about Zen6.
For someone who has never touched workload optimization, can you share a little about how you measure this? Does IPC mean inter-process communication in this context?
I know nothing about workload optimization, and I'd like to know more - right now I feel like the Good Burger gif. "Yeah, I know some of these words".
I've never done workload optimisation myself, but have looked into it just-in-case. It's a really interesting field and unfortunately, you need to know about both your compiler and target CPU/GPU to get the big gains. I believe most modern CPUs have registers for cache use. You have to do some guesswork to get IPC numbers, since core frequency can vary.
Increasing instructions-per-clock is all about minimising program branching (essentially 'if' statements). Because a CPU core can execute instructions faster than main memory can fetch em. It's a fun game to look at an 'if' statement and figure out how you could instead make it an arithmetic operation :).
Maximising cache hits is all about how you structure and access data. For example, if a cache entry is n bytes long, you want to ensure your struct is smaller than n bytes. Having very consistent access patterns can also help (e.g. arrays-of-structs vs structs-of-arrays).
This field is super deep, it's very fun to learn about!
Though you should always ask what your compiler/optimizer is doing for you. Research into how to turn your ifs into arithmetic is ongoing. If the compiler can optimize your simple ifs into the complex arithmetic then you should go with the simple if and let the compiler do that. (unfortunately in many cases the compiler cannot be sure because of some special case - even though odds are you don't care about it)
Agreed! Although I like the idea of considering each logical branch added to my programs. It kind of forces me to think through how variance is being treated in my call stack.
Thank you. Does workload optimisation usually refer to the CPU/GPU alone, or would it include disk or network IO? Or would that be more accurately called something else, like profiling?
in the context of CPU performance / architechture `IPC` pretty much always means "Instructions per clock". That's a measure of the internal parallelism a given CPU core is achieving. Most of the time code is not able to get very close to the theoretical maximums a core can achieve for very long, so the numbers the OP is quoting are incredibly impressive.
I'm not too familiar with the CPU world of performance measurement, but in GPU land, and I suspect for CPUs too, there are a slew of hardware "performance counters" which are registers that increment every time the event they measure happens.
So there could be a "instructions completed" counter and a "cycle" counter, and before starting the benchmark, you record the current value of both counters, then after the benchmark, record the final values, and compute the Instructions Per Cycle (IPC) as the difference in instructions completed over the difference in cycle count.
As for optimization, at a high level it's about minimizing the amount of time any part of the CPU is waiting for other parts of the CPU. The specifics require a lot of background knowledge about how modern CPUs work, more than can fit in a post, but if you're interested, topics to read about include:
### CPU cache hierarchy
CPUs store copies of data from RAM in smaller, faster memory physically closer to where the computation happens, so it's available more quickly. The CPU decides what values to store in the cache, and gives only limited control to the program, so an optimal program needs to be careful to not make the CPU make bad caching decisions (including for synchronizing the cache between multiple threads of execution).
### Instruction pipelining and out-of-order execution (a.k.a. "superscalar" execution)
Modern CPUs operate like an assembly line. A new instruction can start executing before the previous instruction(s) finishes. CPUs also have redundant hardware, so multiple instructions can be in progress at the same step of the pipeline. But there are limitations; sometimes the input to one instruction depends on the output of the previous instruction, so the whole pipeline stalls until the result is ready. An optimal program orders its operations to avoid these stalls as much as possible.
### Branch predicition
When the code to execute depends on the result of a computation, like in an `if` statement, we call it a branch. Branches can stall the pipeline, because the CPU doesn't know what instructions to execute next until the current result is ready. However, to mitigate this, modern CPUs predict which code-path will be taken when a branch is reached and begin executing the associated instructions immediately. If the prediction is right, the pipeline stall is avoided, but if it's wrong the pipeline state has to be restored to what it was before the wrong branch started executing, which is even more expensive than a stall. Usually, the CPU predicts branches correctly, so branch prediction is a net gain. Optimal programs need to understand how the CPU makes these predictions and make their branches as predictable as possible.
### Single Instruction, Multiple Data (SIMD)
Some programs do the same operations to each item of a set of data. CPUs have so-called SIMD instructions to accelerate this by, unsurprisingly, performing the same operation to multiple items at once. For example, they could add 4, 8, or even up to 64 pairs of numbers at once, depending on the instruction set and the range of the inputs. Suitable programs are optimized by arranging their inputs and operations so that SIMD instructions can be used — either directly or by being written in a way that a compiler can translate individual operations to SIMD operations.
Thanks. Is there any convenient tool you'd recommend for watching those registers?
I've said in another comment that any optimisation I'd be doing would be looking at the entire stack, and CPU/GPU optimisation would only be as a learning exercise. I'm a tiny bit familiar with caching and instruction pipelines thanks to college, and branch prediction thanks to Spectre-class bugs. I want to find some time to dive deeper, now.
Their successors had more tangible performance improvements in everyday applications, so yes they were quickly forgotten. Also, it was a reduction in max core count from 10 to 8, intel changes sockets too often, and 14nm+++++.
It’s a backtracking search program looking for integer solutions to a specific problem. I’ve tuned a series of bloom filters to fill my L2 cache so that I rarely have to touch main memory (this alone took my IPC from 0.1-0.3 to 3.0 per thread). Without SMT it’s 4.6 IPC/core.
I think it’s only able to exceed 4.0/thread with SMT off because of a uops cache? From what I’ve read the Zen5 front end only had a 4-wide instruction decode per thread.
How do you explicitly tell the CPU to fetch the next batch of data before you start processing the current batch so that the cache is hot when you need it? I imagine there are instructions for that.
There is a prefetch instruction but you also don't need to in obvious loops like that because the CPU has hardware to detect sequential access patterns. IIRC one test showed a CPU could prefetch up to 11 interleaved sequential access patterns with different strides.
For a long time (probably 2002-2010) I really wanted to do LFS. But I've been on Gentoo since 2004 which, over the last ~22 years, has probably exposed me to almost everything LFS would have, plus a lot more it wouldn't have. I still wish I'd did it way back when. I probably would stumbled through fewer issues had I.
I had the same experience. I think the only thing that LFS would add to that experience would be the package-manager-less experience (building without portage's help)
Calling vuln scanning from a non-profit a felony is a bit of a stretch. Many for-profit companies do similar vuln scanning and then threaten companies with security "scorecards". That is borderline extortion.
If the non-profit was walking down the road and rattling everybody’s door lock to see which are unlocked, and having a look around the windows to see if any are open, would that be a crime?
Because that is exactly what all of these vulnerability scanning companies are doing, and all of us sort of just… let them.
If the neighborhood was constantly being canvased by criminals checking doorknobs, so your concerned neighbor went over to your house to check your doorknob, and then let you know if you accidentally left it open, would you also accuse your neighbor of being a criminal trying to break in?
This is why the checking doors / neighborhood analogy isn't a good one.
Having one person with poor computer security negatively impacts everyone. Hacked sites turn into phishing landing pages, exploit kit hosting, stolen data dumps, and launching off points for attacks on everyone else. The vuln scanning ShadowServer is doing is meant to be a public good, which is why they share the info with ISPs and governments.
Security is too intertwined to stand by and say other people's vulnerabilities aren't your problem.
It is a good analogy because thieves stealing from one house successfully gives them resources and incentivizes them stealing from the same area again.
And even if you remove the analogy, ShadowServer means good, but good intentions doesn’t necessarily make their action moral or legal.
Yes, compromised servers can be used by hackers as means to commit crimes. But when these groups scan the entire internet, they do cause harm as well, as shown in the original linked article. Much less harm than a black hat, but they still waste time and resources from innocent third parties.
It’s fair to ask if the harm they cause is worth the good they do.
> It’s fair to ask if the harm they cause is worth the good they do.
Researcher disclosures, even with POCs, have moved the industry to action incumbents would have rather buried. I’m thinking of CPU and memory exploits, and stuff like log4j, as examples.
Frontier AI is enabling the cyber arms race more than anything past, and certainly more than some bot slowly crawling web servers for old vulnerabilities.
If we’re talking harm, it should be in the broader context of internet history, imo.
Ok, but there is a massive difference between "I hammered my own CPU and found a bug that affects everyone" and "I hammered THE ENTIRE INTERNET and found some poor persons website that is not super secure and then I snitched on them to their ISP".
The first is a lot more legal and ethical than the second.
They didn't hammer the entire internet. That would be like a multi-terabit DDOS. In contrast, the scanner in question sent a trickle of web requests for old vulns, nearly-innocuous at this point.
Fear & the danger aren't the same. anyway, I'll let it go.
It’s a fact that these scanners send unauthorized requests to servers whose owners did not consent to scanning.
If we think this is acceptable, which of the following is also acceptable, and if some is not, where is the difference:
1) Detecting the scan and notifying the ISP of the scanner that they are abusers
2) Notifying law enforcement and filing a formal complaint
3) Implementing passive defense measures and responding with a reverse slow loris attack where you trickle 1 response packet per minute down the pipe to attempt to DoS the scanner. But only to prevent them from accidentally hitting some innocent third party’s homemade wiki system that deletes content on GET requests. Not with any malice, just to help prevent harm to others.
4) Implementing active defense where you return progressively more hostile response starting with zip bombs and ending with whatever sota cyberweapon latest AI can give you. But only to shut down the scanner and not cause any damage what so ever. Simply to check if their security is up to par. You would of course report any detected vulnerabilities to the ISP of the scanner, to prevent criminals from using the same.
5) Actively responding with a scan of your own to any incoming scans.
I just checked my own webserver logs (I also run a sever in the ntp pool) and I too see some hits in my webserver logs.
They look to all be log4j vuln scanning activity (CVE-2021-44228), and the volume isn't that high (a few a day, and not every day). They just have some overzealous vuln scanning. And yes, they shouldn't have the NTP pool under their DNS name.
I've had all sorts of strange things happen because of my ntp pool membership, this one is pretty far on the benign end of things.
A set of permutation fair dice work for any subset of dice and players. So a 4-dice perm-fair set allows any three dice to be used and is guaranteed to be perm-fair for 3 as well.
The “Go First” name is catchy for laypeople, but permutation fairness is the strongest and most interesting property.
There are sets we call “all subset place fair” which means any subset of the dice can be used and can fairly choose 1st, 2nd, and so forth, but this property is slightly weaker and doesn’t always make every ordering equally likely for every subset.
The big open question is whether a set of 5 (permutation fair) 30-sided dice exists.
I’ve been working on that on and off since 2012. I picked it up again about a month ago and have made dramatic speed improvements to my search, but exhausting the whole space I’m searching will still take my computer an estimated 70 years.
Well I’m first exhausting some promising regions of the search space which should complete in less than 2 weeks. After that I strongly suspect a solution doesn’t exist (for the column grouped space I’m searching).
If it were just a matter of a few thousand dollars of computer time (say, less than $5000) the money would already be spent and I’d have an answer.
We’ll see, I may build the tooling to distribute the search and enlist help from others interested.
It’s only been about 2 weeks since I was able to drop the runtime from “age of the universe” levels to just decades.
If you're being serious, i assume its because its primarily a novelty problem, and the economic investment of 4000 computers isn't worth it just for the bragging rights of solving the problem.
> GPT‑6 Astra brings together years of research and big bets across pre-training
Do we know if they’ve finally completed another pre-training run, or is this building off the same pre-training base they’ve been using since the GPT-4 days?
At first I was going to say this is just the tetrahedra trick dressed up in slightly different clothes, and some sense it is, but there is a nice cancellation in the y and z coordinates which leads to less calculation in practice.
I saw some news last week saying the project had reached 100%. Looking at the git repo (https://gitlab.com/kholdfuzion/goldeneye_src) that claim is a bit harder to verify, but maybe look again?
I just don't understand how Alpöge can get Claude to construct so many complex mathematical objects. He must already have some deep insight into the structure these things are likely to take. Naive prompting, or naive search would not have found pretty much any of the object he's found in the past month.
He has an AB in Math from Harvard with a senior thesis "The average elliptic curve has few integral points", and a PhD in Math from Princeton with thesis "Points on Curves", so it's fair to say he has some deep insight. (https://alpo.ge/cv.pdf)
Okay this result may well be in his expertise wheelhouse but what about the recent smooth Carathéodory conjecture disproof? Or the Jacobian, or the Hadamard matrices? It seems to span a pretty wide range. I know others are getting good results too, but his really stand out.
It wouldn't surprise me if he is given access to a more advanced model or more compute, since he is an Anthropic employee. Add to that the fact that this makes great PR and that he has not shown any of his prompts, it's more than likely in my opinion that this is the case.
Zen5 is incredible when you're able to make the most of it. I’m super excited about Zen6.
reply