We Quizzed 5 AI Chatbots for Health and Safety Advice
Some of the answers were fine, but others had CR's experts concerned. Here are our tips for using these popular platforms safely.
As generative artificial intelligence (AI) platforms like OpenAI’s ChatGPT and Google Gemini go from a novelty to an everyday tool, can we trust them to not spit out information that’s misleading—or, even worse, downright dangerous?
Despite rapid advances in AI, examples abound across social media and in the news of dubious or outlandish AI-generated statements. Recently, Google’s AI Overviews, an enhancement to the company’s search engine, made a splash by suggesting users fix runny cheese on pizza by adding glue. ChatGPT entirely invented past court cases when being used by a lawyer for research in 2023—a mistake that made its way into a real-life legal brief.
Such mishaps are eliciting enough concern that Google now appears to have rolled back its just-released AI search engine tool while it works to improve its performance.
One problem is that AI tools can draw from a wide range of sources, creating a lot of room for error.
“When a chatbot is answering your question about something important, like how to know if you have COVID-19, its response might be based on info from the Centers for Disease Control and Prevention—or it might be from a ranting and raving blog post,” says Grace Gedye, a policy analyst at CR who works on consumer protection issues related to AI. (Note: CR is experimenting with AI tools that retrieve and summarize information solely from Consumer Reports data and reporting—and we’re proceeding carefully. The technology won’t be used to generate articles.)
‘How many carbon monoxide detectors do you need?’
CR says: For all homes—and particularly those with fuel-burning appliances such as a furnace—properly placed carbon monoxide detectors are a crucial safety measure. CR advises consumers to install a carbon monoxide detector on each living level, outside each sleeping area, in the basement, and near—not inside—an attached garage. And these devices should be interconnected so that when one goes off, they all do. Did the chatbots agree?
Source: Gemini Source: Gemini
AI’s answer: (Mostly) right.
Google Gemini got this answer nearly right, says Bernie Deitrick, a senior tester at CR with experience evaluating carbon monoxide detectors. But it erred slightly when it suggested putting a CO alarm inside an attached garage, he says. “Attached garages are usually not temperature-controlled,” he says, “so that could also lead to the detector (or its battery) being damaged during temperature extremes.” (Explore CR’s test results for carbon monoxide detectors.)
But we’ll give some bonus points to Gemini for attempting to show its sources, even if its Consumer Product Safety Commission citation takes users to a homepage rather than the page where the information is found. If a chatbot fails to provide citations on its own, it’s generally smart to ask straightforward follow-up questions, like “Where did you find this information?” The more an AI tool can help you track down primary sources, the better.
‘How to filter PFAS from tap water?’
CR says: Today, according to a recent estimate, at least 45 percent of the nation’s tap water contains some type of PFAS—a group of more than 14,000 manmade chemicals that persist in the environment and human body for a long time and have earned the nickname “forever chemicals.” But not all water filters can get them out: CR recommends looking for a water filtration system with the National Sanitation Foundation certification code NSF/ANSI 53 (or NSF/ANSI 58 for reverse osmosis systems) and double-checking that the manufacturer specifically claims that its product removes PFAS. But even certified filters can’t be guaranteed to remove all types of PFAS that might be present in your water.
Source: Copilot Source: Copilot
AI’s answer: Lacking some key details.
Using Microsoft Copilot’s advice, consumers may feel confident buying any activated carbon or reverse osmosis filter to get rid of PFAS in their water. However, not all filters of these kinds are designated for PFAS removal. Plus, “even a filter with a reliable certification may not effectively remove all the PFAS present in a heavily contaminated drinking water sample,” says Tunde Akinleye, a chemist and food safety tester at CR. This answer demonstrates a common problem with AI-generated advice, particularly on complex topics like this one: It may point you in the right direction but miss important nuance.
‘What age to buy a front-facing car seat?’
CR says: We recommend keeping your baby in a rear-facing child car seat until at least 2 years of age or until they exceed the rear-facing height or weight limit of the car seat, as stated by the manufacturer. You can then upgrade to a harnessed forward-facing car seat until your child outgrows the harness height or weight limit. (Check out CR’s test results for dozens of car seats.)
Source: ChatGPT Source: ChatGPT
AI’s answer: Spot-on.
ChatGPT-4o’s succinct advice aligned with our own experts’ and the current industry standard. It also cited the American Academy of Pediatrics—though we’d prefer the answer linked out directly to the AAP site—and included an important stipulation to always defer to a car seat manufacturer’s guidelines, as well as local laws and regulations. When we then asked ChatGPT to provide us with the AAP guidelines and policy statements, it responded with a number of helpful links. Two thumbs up.
‘Can kids play with water beads?’
CR says: Absolutely not. These popular children’s toys carry high risks if ingested, like bowel obstruction, blocked airways, and infections, and have led to reports of deaths and thousands of emergency room visits. CR has led an effort to get these dangerous products off store shelves and online marketplaces, and our experts adamantly advise against allowing children to play with them.
Source: Meta AI Source: Meta AI
AI’s answer: Dangerous and inconsistent.
Meta AI failed to highlight the significant safety risks of water beads, aside from a passing reference to a need for parental supervision.
“Today’s AI chatbots can sometimes be helpful, but they cannot be trusted to reliably flag product safety hazards,” says William Wallace, CR’s associate director of safety policy. “Unfortunately, new products often hit the market and become popular before they’re properly vetted for safety, and it’s this popularity that these chatbots seem to pick up on.”
Interestingly, when I altered the wording of my question slightly, Meta AI sometimes conjured up an appropriate warning. Goes to show, these AI chatbots can answer your question very differently depending on the verbiage you use—which is much less of an issue with traditional search engines. (When I searched “water beads kids” in a variety of different ways on Google, the results often featured some sponsored shopping links at the top—but, in every case, they also showed me websites warning of water beads’ risks.) That’s why it’s a good practice to ask an AI chatbot your question in a few different ways.
‘What’s the safest midsized car in 2024?’
CR says: Our auto experts don’t designate a single “safest” midsized car, the reason being that there are many facets to auto safety, from protecting occupants in the event of a crash to preventing a collision altogether. (CR conducts its own tests on attributes like braking and accident avoidance performance.) But the quick answer is to look at the cars that earn the Insurance Institute for Highway Safety’s highest possible Top Safety Pick+ designation. That’s an industry gold-standard safety rating based on numerous, exclusive tests. In 2024, thus far, the midsized vehicles that fall into this Top Safety Pick+ category are the Honda Accord and the Hyundai Ioniq 6.
Source: Perplexity Source: Perplexity
AI’s answer: Correct, but with surprising sources.
Perplexity AI’s answer was just fine—though we’ll say that the Honda Accord isn’t the only midsized car to earn high safety ratings in 2024. In addition, there are different ways to measure a car’s safety, but the chatbot seems to focus on crash tests.
But I was surprised by the chatbot’s sourcing. Perplexity cited five different online articles—including one from iSeeCars, a popular marketplace for used cars, and an outdated MotorTrend story—but it didn’t directly link out to the IIHS or the National Highway Traffic Safety Administration.
So, Can You Trust AI Chatbots?
The AI-generated responses ran the gamut from spot-on to somewhat spotty to egregiously wrong. In most cases, the bots’ consumer advice was largely accurate and appropriately cited, but, even then, our experts often still had slight clarifications and important context to add. In one case, the AI platform omitted a critical safety warning that would’ve been readily available from a simple Google search.
Most of the platforms acknowledge and clearly state their shortcomings—“ChatGPT can make mistakes. Check important info,” is a typical warning. That’s good advice. Here are more tips on how to use these platforms in a way that reduces risk, particularly when it comes to exploring topics that have health or safety implications for you and your family.
CR’s Tips for Using AI Chatbots
Use AI as a starting point. AI chatbots can be powerful time savers by quickly introducing or summarizing a broad or complex topic in seconds, but it’s smart to then check the source material.
Ask for sources. Some of the platforms we tested display their sources by default. But if an AI chatbot doesn’t show its work, ask. A simple “Where did you find that information?” works. Even if an AI chatbot does cite a reputable source in its response, it doesn’t always mean that’s where the information came from, or that it was summarized correctly. Again, your best bet is to go to the primary source yourself.
Think of AI as an assistant—not an expert. Ask chatbots questions like, “What sources would you go to for credible information on [X]?” “Who are the foremost experts on [Y]?” or “Give me a reading list to learn about [Z].” These requests will prompt AI to jumpstart your own more thorough research, but you’re not relying on the chatbot to provide the information itself.
Watch out for almost-right answers. While completely outlandish responses may make the news, it’s the 95 percent accurate responses that are more likely to go undetected. Often, this is because an AI chatbot has lost context and nuance or else jumped to a slightly wrong conclusion. That’s most of what we saw here in our test run.
Ask questions more than once. Slightly different versions of the same query can yield very different results. There’s no harm in asking a question in a few different ways to see how the results differ, or else turning to several AI platforms to compare the answers.
Don’t look to AI for breaking news. Some AI chatbots incorporate search engine results and more timely information in their responses, but others won’t. The bottom line: Don’t expect chatbots to flag late-breaking news, like recently recalled products or food safety alerts.
Stay diligent as AI integrates with search engines. All signs point to AI increasingly becoming a part of our search experience—or, for some people, maybe replacing a traditional search engine entirely. But this can compromise your ability to vet information. When a regular search engine serves up links, you can decide how much you trust each website it surfaces. Then you can decide what to click on, and how much stock to put in the information you read. “Several of the biggest chatbots wash away that important context,” Gedye says, “requiring a bit more skepticism and due diligence from consumers.”
