This is a new blog about making connections between different media and ideas. You can subscribe by email or RSS! Those links are below.
This week, famed quirky science communicator Hank Green admitted to using ChatGPT for "research" and also overrelying on it personally in many ways. (In my own opinion, creators who cop to using LLMs in a really specific way are lying and likely using it in more controversial ways too. I am not talking about actual working scientists. Hank is an entertainer.)
I'm happy about the way Hank handled this overall, because he mentioned loved ones intervening with him, reflecting on his actions, and taking a break. It sucks that he's been relying on ChatGPT in a science communication role, there's no doubt that he should not have done it, but his actions after that were more rational and responsible.
The Underling Defense
I am also a science communicator, but I'm the underling in an organization where someone else might be the harried Hank Green person. Hank hasn't been a traditional worker for a long time, and may never have been a professional staff writer, so it's possible he doesn't know busy people have relied on and then blamed underlings for all of human history probably. Historically, and now in the global economy of the internet, these underlings were also low paid and/or exploited. ChatGPT is the culmination of the idea that you deserve underlings who do whatever you say at the drop of a hat.
Okay. But this isn't really a political argument otherwise, but what I said is true and it does play a part in this genre of story. People who cheat, lie, plagiarize, and so forth love to say they're just so exhausted that they "misread their own notes" or "forgot the quotation marks" or have a bad assistant or whatever. As Keith Mars once said, "The buck stops... there...?"
LLMs as General Research
I'm in a unique position because I worked in nonfiction book publishing as part of a legacy that went back decades, not in a prestigious way but in the sense that we had databases and existing physical copies of trivia books from many years prior. Misinformation didn't start with the internet, and it was never limited to some of the higher profile examples we still know about.
People took listed facts and stories from books, assuming that something in a book had been fact checked and must be true. You can follow the blame tree all the way to the roots here as you think about the same underling or person relying on an underling, trying to make the same deadline as now, except they had to use the landline phone or send a telegram or do flag semaphore.
All that is to say, I saw firsthand that a lot of incorrect stuff had trickled through the hands of dozens of my very smart and not overly busy colleagues, from original sources that were also intended as the truth. And that entire body of information got fed into the Fargo wood chipper of LLM implementation. 99% of the body may be true things, but 1% is a lot when you're trying to reliably communicate the truth.
Wikipedia infamously came under fire because people trying to correct popular misconceptions were being counter-edited out in order to preserve the popular impression of many facts, even though that popular impression was wrong. This presaged LLMs, because Wikipedia runs on a consensus model that mimics probabilistic models in some ways. Wikipedia is not, they say over and over, the place for "original research."
I think people's idea of LLMs is that, because of the huge volume of text these models have chowed down on, the factual truth must simply float to the top by likelihood alone. The thing is, some factual truths were never able to float to the top of anything, and that's before we even get into the power dynamics and so forth of certain areas of the truth.
LLMs as Echo Chambers of SEO
By now, you've likely seen how Google's "AI Preview" widget was made to tell people to glue their pizza together. Google has held such a dominant place over internet users since the 2000s that the internet itself has been written in order to conform to Google's style requirements. Yes, we see that in AI slop now, but we have seen it in cynical listicles for decades, and we see it now in circuitous pages that promise to answer simple questions but take 1,500 words to do so.
(That's because Google ranks pages containing these full question search strings, more highly than those that just answer the question by giving information that answers the question.)
Wikipedia ingested a body of information with factual errors, but much of that original bulk was from sources like the public domain versions of the Encyclopaedia Britannica or transparently factual government websites. LLM companies trained their models to choke down the spiny and succulent Google SEO ecosystem. This was never vetted in any way except by Google's proprietary PageRank algorithm, and that is not a fact check at all.
Scientists Have Good Reasons Not to Use LLMs
I cover emerging science research in my career. The overall field of science publishing is experiencing a consumer LLM crisis at the moment, just like everyone else. There are fake data, fake images and charts, I'm sure there are fake writings around real data sometimes, and all of this is enabled by a very low regulation climate in science publishing.
In the past, science journals were printed on paper, and their overall size and number of accepted papers depended on actual physical room inside the planned journal. You can increase your page count within reason, but that costs more in materials, takes more time to manufacture, takes more time to ship, and costs more to actually mail to subscribers and universities. It also increases labor cost in at least half a dozen ways, like the time required to coordinate a bunch of peer reviewers plus the time many more peer reviewers must spend.
Instead, journals were just more selective. Academic publishers typically offer a range of journal types, so a scientist with a brief new idea can submit that to Science Company Letters. Someone with a short paper instead of an experiment or mathematical proof can submit to Science Company Reviews. People with full experimental regalia can submit to the appropriate journal for that. Many of the most important scientific ideas were written as brief letters to these journals. It is genuinely cool to see some luminary be like "What if gravity was real. Okay, talk soon. Sincerely, Isaac Newton" type of vibes.
Today, physically printed journals still have limitations, but online hyperjournals can just do whatever they want all the time. I don't want to guess how much of allegedly "peer reviewed" research is literally never read by anyone, because it appears in these low quality journals where everyone just tacitly agrees to pass everything along.
This is also not as cut and dry as it sounds, unfortunately. People in academia around the world face many types of extreme pressure to publish in order to be promoted or even to be considered for academic positions at all. In more constrictive or totalitarian nations, the pressure must be worse and may even bring physical danger. So I'm not judging this industry, just describing it for you.
Scientists May Have Good Reasons to Use LLMs
Falsifying data, writing or broad background research using LLMs, those are easy to identify as bad behaviors in science. What Hank Green was doing is not scientific research in any way -- he is a summarizer and entertainer -- so this section is not at all about him or me.
Working research scientists are in a position now where they can use LLMs and other forms of AI to support their research meaningfully. I'm not a scientist and I'm not vetting or claiming to represent any of these reasons. I'm just sharing what I observe, which is that scientists whose work I trust are finding that they can use LLMs to make their research better.
It's not very chic or punchy to say that individual results vary and context matters a great deal. In 2024, a scientist I like told me that algorithms for finding new molecules were mostly useless to him, because many of their suggestions were infeasible. But medical researchers may compile hundreds of papers and realize that (I am making this up as an example) cancer patients who were also taking statins for their cholesterol experienced some kind of reduction in their tumor volume. That kind of thing.
I think of this as, like, "Search Plus" mode. For many reasons, there is just not a human being or a university department of "reading every scientific paper in order to find mention of cancer and statins together."
This is just one use case I know about, and when credible scientists talk openly about probing LLMs for something they can use, I believe them.
But Hank Green is not a working scientist and neither am I. We aren't in this group at all in any sense. I would not ever use an LLM (I do not use them at all to be clear!) to try to find some interesting throughline for me to report on or something. I don't have the qualifications, resources, or desire to vet that. It isn't my job or my place to do that.
Science Communicators Should Definitely Not Use LLMs
I will make this section brief. Sorry this is so long.
Hank is a generalist science communicator for the most part. He takes on an entire topic, like knitting or CPAP machines, and has kind of a "debunking" approach overall. That means he and his team are likely ingesting stuff that contains the types of legacy errors I mentioned above. The people most likely to notice these errors are specialists, like knitters or doctors. An LLM will never know it's telling you a popular misconception as the truth. It cannot vet these ideas.
There are a bunch of great videos by knitters who explain all the problems in Hank's controversial video about knitting. To be honest, these errors run the gamut I laid out here of all the ways LLM use can go wrong and how that is profoundly, inextricably linked with our terminally Google-ized search culture. Someone with no expertise, who feels a time crunch, can't tell that a stock photo with keyword spam like "knit knitting knitted crochet crocheted knots weaving woven" is just a closeup of a man's woven shirt from Kohls or something. (Stock images are also the victim of unregulated U.S. monopolizing, because Getty slowly rolled up almost all their competitors. It's problems all the way down, sorry John Green.)
I cover emerging science research, meaning brand new papers that suggest new approaches or novel formulations of things like the birth of the universe or a quantum computing setup. LLMs are also never appropriate for me, because they can't interpret new research that does not yet have any broader consensus. And frankly, I don't want to be responsible for checking up on my own version of an underling.
In Conclusion
I don't like that Hank Green chose to rely on LLMs (I would say "overrely on," but I don't believe they have any place in his overall workflow to begin with), but I like how he handled the reaction. In the piece I linked above, Kotaku shares Hank's comments that he's taking a break from his work, seeing the unhealthy dopamine relationship he has with ChatGPT, and speaking with his wife and brother about his actions.
I really value having a concrete example to point to of someone whose livelihood and reputation are in danger because of his choice to use LLMs, and who turned to his real life loved ones and treated their opinions with respect. There are always factors at play that we can and can't see. I don't care enough about Hank Green to judge him harshly, except by the parameters I laid out here, because our work context is similar.
I do know that sponsorship and ad revenue for YouTube and other platforms has dried up in an extreme way, and I know that even very famous TV and movie stars are now doing gross commercials for freemium mobile games. It's rough out there.
Thanks for reading. I'd love to hear your polite and engaged feedback if you have any!