The "whistleblower"'s admission that he "did not have evidence to support his suspicion of fraud" is pretty damning. He fails to meet a much, much lower bar than direct observation -- he admitted he had no evidence at all to even support his suspicion of fraud.
> the most damning evidence you could possibly have is that you cannot reproduce the results
Seems like the "whistleblower" didn't even have that. From the paper by the AlphaChip authors: "We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline. We still do not know how Markov and his collaborators produced the numbers in their paper."
> [whistleblower] stated he did not have evidence to support his suspicion of fraud, that he needed to cross a much larger threshold to prove his suspicion
is exactly saying what I was saying.
But in addition, this is hearsay, "quoted" only by Google's rep. It was never actually mentioned by the whistleblower. It has exactly 0 value. Using this quote at face value is intentionally misleading no matter which way you put it. They're literally the defendants - they're basically quoting themselves.
> Seems like the "whistleblower" didn't even have that.
Before he was fired?
Also, I find it funny that for all the talk of the crisis of reproducibility, anyone would trust for a second the authors of the paper more than the attempts done by a 3rd party (and literally done by one of the most important names of the entire floorplanning academic community, to begin with). At least the EDA community has used some benchmarks that have been often used by other papers, allowing some resemblance of a comparison, and a criticism that "these ancient benchmarks do not reflect our holy ways or whatever" is a criticism that maybe I also share; but it's a hoop that everyone who has ever published any such paper (including all the big names) has had to pass in order to be published, unless apparently if you are called Google and publish in Nature.
Frankly, at this point I don't even know why would anyone bother with Google's paper. It feels as if they've managed to alienate the entire floorplanning academic community, and whenever I read one of Google's "responses" I see why.
Google infrastructure is weird and takes significant work to disentangle from a given project, so I'm not surprised it took them six months to open-source it.
> Before he was fired?
I don't know how long someone should expect to remain employed when making baseless allegations of scientific misconduct against his colleagues instead of doing actual work. Again, he did not have evidence to support his suspicion of fraud, and he admitted this at the time.
> most important names of the entire floorplanning academic community
If the old guard struggles with ML basics, what can the AlphaChip authors be expected to do about this? This pattern is unfortunately common when ML comes for a new field -- some researchers adapt and build, and others fail and complain (or worse, don't really even try).
> it's a hoop that everyone who has ever published any such paper (including all the big names) has had to pass in order to be published
If the hoop doesn't match what modern chip design needs, we shouldn't expect researchers to hop through it. No one is comparing Vision Transformers against AlexNet on MNIST. Meanwhile, AlphaChip is already used in production to make real layouts for real chips.
> Google infrastructure is weird and takes significant work to disentangle from a given project, so I'm not surprised it took them six months to open-source it.
No, this is ridiculous. Then don't publish, damn it. But they publish first, claiming that all the data is there, claiming that their papers satisfy all the rules required for reproduceability... and only actually release the source once they're caught lying . It is not the first time it happens.
This seriously damages Google's credibility and very well deservedly so.
> I don't know how long someone should expect to remain employed when making baseless allegations of scientific misconduct against his colleagues instead of doing actual work.
My question was whether this supposed data was available to the whistleblower before he was fired or not. It is kind of important (see my first point).
> Again, he did not have evidence to support his suspicion of fraud, and he admitted this at the time.
Why are you ignoring my counterargument? Where did he admit this exactly? The only source for this quote so far is hearsay from the defendant itself in a civil lawsuit, i.e. 0 value.
> If the old guard struggles with ML basics, what can the AlphaChip authors be expected to do about this? This pattern is unfortunately common when ML comes for a new field -- some researchers adapt and build, and others fail and complain (or worse, don't really even try).
This is as much of an ad-hominem, childish and stereotyping attack as it gets, holier-than-thou attitude, and, frankly, I resent the implications. Imagine if I said "If the morons at Google struggle with chip design basics, what can we do?". Does this encourage conversation?
There is no "old guard", EDA has been using AI quite effectively since before Google was even a thing, and likely _right now_ there are more experts in RL employed by EDA companies than there are AI experts at Google entirely.
> If the hoop doesn't match what modern chip design needs, we shouldn't expect researchers to hop through it.
The difference in "needs" is nowhere near great enough yet to be worth the effort of changing it, for reasons that should be evident by now.
> My question was whether this supposed data was available to the whistleblower before he was fired or not. It is kind of important (see my first point).
If you read the court case, you can see quite clearly that he never obtained any evidence, and simply hoped to go on a general fishing expedition.
For him, it was like the AlphaChip authors performed a magic trick. Magic is impossible, and so something must be up. Even if he doesn't know how they did it, there must surely be fraud somewhere. Did it ever occur to him that he might just be wrong?
The closest thing he had to "evidence" of his belief was his study where he says he tried AlphaChip and got worse results than SA. But even that study was flawed, and known to be flawed at the time. From "That Chip Has Sailed":
"In 2022, it was reviewed by an independent committee at Google, which determined that “the claims and conclusions in the draft are not scientifically backed by the experiments” [33] and “as the [AlphaChip] results on their original datasets were independently reproduced, this brought the [Markov et al.] RL results into question” [33]. We provided the committee with one-line scripts that generated significantly better RL results than those reported in Markov et al., outperforming their “stronger” simulated annealing baseline. We still do not know how Markov and his collaborators produced the numbers in their paper."
([33] is the sworn statement from Jon Orwant, head of the independent resolution committee.)
How large do you imagine the conspiracy is here? The method is secretly bad, but the TPU team uses it in production anyway? MediaTek built on it anyway? The TF-Agents team claimed to reproduce it, but they're actually lying? Jeff Dean put his reputation behind it but he's either fooled or lying?
Please hold off on replying further on this thread until you have read OP's linked paper. I understand you want to defend your field against the encroachment of people you perceive to be bad actors, but I believe you have things backwards.
> Please hold off on replying further on this thread until you have read OP's linked paper.
This continues the holier-than-thou attitude, but also violates this site's rules.
Not only I have read the entire court transcript, I was following it with interest as it happened. My point still stands. It is sleazy, to say the least, to misquote the whistleblower. It is even more sleazy to base such quote on "something the defendant claimed that the plaintiff claimed in a civil lawsuit".
This is point #1 on this "critique" and already rings all alarm bells, which is what my comment was about.
Google also did many, many evils during the entire trial (e.g. through the entire SLAPP thing) that no wonder they did not want to expose to the public. But that's another story.
> How large do you imagine the conspiracy is here?
Not large. Google continuously publishes papers where the results cannot be reproduced, at least by the time the paper is published, and barely gets a slap on the wrist from Nature, which is supposed to enforce these things.
> The method is secretly bad, but the TPU team uses it in production anyway? MediaTek built on it anyway? The TF-Agents team claimed to reproduce it, but they're actually lying?
Neither Tensor nor Mediatek are examples of really high-stages design; Tensor is not know for beating any records and Mediatek specializes in value.
While my area is frontend rather than backend, I have a passing familiarity with automatic macro placement (due to the fact that the barriers between flow steps keep getting thinner every day). Almost every other chip house of large-enough size has a macro placer. Almost every PhD student has implemented one. It's an easy problem, and with minimal knowledge about your design style you can likely do a better job than the generic ones from major vendors. However, it doesn't matter if you have the best macro placement engine in the world, your results would still not be ground-breaking overall. You can in fact ALSO do a terrible job and still get a pretty decent result since the flow will compensate. This was already my informed opinion even before the Nature article, but for Google's method, which _requires_ another placement tool to refine its work, would be even more true. And Google's method even requires a GPU farm for something that is, while important, still a relatively short step in the entire placement flow (and for not very push-through performance designs like Google's or MediaTek's , which are like the bread and butter of the world, the entire flow would not have required GPUs before). It's no wonder than the whistleblower thought the entire point of this Nature article was for Google to sell more Google Cloud GPU time. They may have convinced Mediatek, but not the academic community.
I don't know how long someone should expect to remain employed when making baseless allegations of scientific misconduct against his colleagues instead of doing actual work. Again, he did not have evidence to support his suspicion of fraud, and he admitted this at the time.
I'm sorry that the "most important names of the entire floorplanning academic community" are struggling with ML basics, but it is what it is. The "Chip Has Sailed" paper makes this pretty clear. This pattern is unfortunately common when ML comes for a new field -- some researchers adapt and build, and others fail and complain (or worse, don't really even try).
> it's a hoop that everyone who has ever published any such paper (including all the big names) has had to pass in order to be published
If the hoop doesn't match what modern chip design needs, we shouldn't expect researchers to hop through it. No one is comparing Vision Transformers against AlexNet on MNIST. Meanwhile, AlphaChip is already used in production to make real layouts for real chips. TPU is a big deal!
I think the one thing we agree on is that this field desperately needs large public benchmarks that are representative of modern chip design.
Synopsys disavowed Markov's paper:
"Regarding the CACM article that Igor Markov's comments and writings do not represent Synopsys views or opinions in any way. Synopsys is also aligned with you on the potential of Reinforcement Learning AI for chip design"
(https://x.com/JeffDean/status/1859431937640665474)
There's a lot of... passionate discussion in this thread, but we shouldn't lose sight of the big picture -- Google has used AlphaChip in multiple generations of TPU, their flagship AI accelerator. This is a multi-billion dollar project that is strategically critical for the success of the company. The idea that they're secretly making TPUs worse in order to prop up a research paper is just absurd. Google has even expanded their of AlphaChip use to other chips (e.g. Axion).
Meanwhile, MediaTek built on AlphaChip and is using it widely, and announced that it was used to help design Dimensity 5G (4nm technology node size).
I can understand that, when this open-source method first came out, there were some who were skeptical, but we are way beyond that now -- the evidence is just overwhelming.
I'm going to paste here the quotes from the bottom of the blog post, as it seems like a lot of people have missed them:
“AlphaChip’s groundbreaking AI approach revolutionizes a key phase of chip design. At MediaTek, we’ve been pioneering chip design’s floorplanning and macro placement by extending this technique in combination with the industry’s best practices. This paradigm shift not only enhances design efficiency, but also sets new benchmarks for effectiveness, propelling the industry towards future breakthroughs.”
--SR Tsai, Senior Vice President of MediaTek
“AlphaChip has inspired an entirely new line of research on reinforcement learning for chip design, cutting across the design flow from logic synthesis to floor planning, timing optimization and beyond. While the details vary, key ideas in the paper including pretrained agents that help guide online search and graph network based circuit representations continue to influence the field, including my own work on RL for logic synthesis. If not already, this work is poised to be one of the landmark papers in machine learning for hardware design.”
--Siddharth Garg, Professor of Electrical and Computer Engineering, NYU
"AlphaChip demonstrates the remarkable transformative potential of Reinforcement Learning (RL) in tackling one of the most complex hardware optimization challenges: chip floorplanning. This research not only extends the application of RL beyond its established success in game-playing scenarios to practical, high-impact industrial challenges, but also establishes a robust baseline environment for benchmarking future advancements at the intersection of AI and full-stack chip design. The work's long-term implications are far-reaching, illustrating how hard engineering tasks can be reframed as new avenues for AI-driven optimization in semiconductor technology."
--Vijay Janapa Reddi, John L. Loeb Associate Professor of Engineering and Applied Sciences, Harvard University
“Reinforcement learning has profoundly influenced electronic design automation (EDA), particularly by addressing the challenge of data scarcity in AI-driven methods. Despite obstacles including delayed rewards and limited generalization, research has proven reinforcement learning's capability in complex electronic design automation tasks such as floorplanning. This seminal paper has become a cornerstone in reinforcement learning-electronic design automation research and is frequently cited, including in my own work that received the Best Paper Award at the 2023 ACM Design Automation Conference.”
--Professor Sung-Kyu Lim, Georgia Institute of Technology
"There are two major forces that are playing a pivotal role in the modern era: semiconductor chip design and AI. This research charted a new path and demonstrated ideas that enabled the electronic design automation (EDA) community to see the power of AI and reinforcement learning for IC design. It has had a seminal impact in the field of AI for chip design and has been critical in influencing our thinking and efforts around establishing a major research conference like IEEE LLM-Aided Design (LAD) for discussion of such impactful ideas."
--Ruchir Puri, Chief Scientist, IBM Research; IBM Fellow
Unfortunately, commercial EDA companies generally have restrictive licensing agreements that prohibit direct public comparison.
Still, the fact that Google uses it for TPU is pretty telling - this is a multi-billion dollar, mission-critical chip design effort, and there's no way they'd make TPU worse just to prop up a research paper. MediaTek's production use is also a good indicator.
In the blog post, they announce MediaTek's widespread usage, the deployment in multiple generations of TPU with increasing performance each generation, Axion, etc.
Chips designed with the help of AlphaChip are in datacenters and Samsung phones, right now. That's pretty neat!
You are now using multiple new accounts based on the name of one of the authors (Anna Goldie) and her husband (Gabriel). First this one ('gabegobblegoldi'), and then 'anna-gabriella'.
I think it is time for you to take a deep breath and think about what you are doing and why.
You seem to be obsessed with the idea that this work is overrated. MediaTek and Google don't think so, and use it in production for their chips, including TPU, Dimensity, Axion, and others. If you're right and they're wrong, using this method loses them money. If it's the other way around, then using this method makes them gain money.
Chatterjee settled his case. He has moved on. This is not some product being sold -- it is a free, open-source tool. People who see value in it use it; others don't, and so they don't. This is how it always works, and it's fine.
You made two accounts, at the same time, using the same toxic naming scheme. You have even admittedly openly to reposting to avoid prior flaggings (e.g. https://news.ycombinator.com/item?id=41683073).
From what I saw in the rebuttal papers, the Google cost-function is wirelength based. You can still get good TNS from that if your timing is very simplistic -- or if you choose your benchmark carefully.
They optimize using a fast heuristic based on wirelength, congestion, and density, but they evaluate with full P&R. It is definitely interesting that they get good timing without explicitly including it in their reward function!
The odd thing is that they don't compute timing in RL, but claim that somehow TNS and WNS improved. Does anyone believe this? With five circuits and three wins, the results are a coin toss.
Prior to AlphaChip, macro placement was done manually by human engineers in any production setting. Prior algorithmic methods especially struggled to manage congestion, resulting in chips that weren't manufacturable.
Definitely a big part of it. Chips enable better EDA tools, which enable better chips. First it was analytic solvers and simulated annealing, now ML. Exciting times!
https://deepmind.google/discover/blog/how-alphachip-transfor...
And they've responded to the absurd degree of skepticism that followed:
"That Chip Has Sailed: A Critique of Unfounded Skepticism Around AI for Chip Design" - https://arxiv.org/abs/2411.10053