4. Deferring to Authority and Expertise

Fact checkers often defer to authority to confirm the accuracy of statements. We defer to certain documents to record the "official" version of events; data and scientific evidence to describe the world as it is; experts to explain facts and place them in context; and individuals to tell us about their own experiences. No one can know everything independently; we all rely on one another's knowledge, and this is especially true for people who are in the business of knowing things (and checking them).

It may sometimes feel like, because of our rigorous methods and authoritative tone, fact checkers are able to access some realm of objective facts that others can't. But journalists still operate within the same social fabric as everyone else. What makes our work different isn't the purity of the sources we turn to for knowledge but rather the integrity and careful skepticism with which we approach them and evaluate their authority and relevance.

It is the responsibility of the entire editorial team to determine to whose authority journalism should defer, when, and why—but it is the fact checker's job to confirm the appropriateness of the team's decision. As fact checker, you are responsible not only for checking that the facts in a story are accurate but also for explaining how we know they're accurate and why we can stand by the authority of the sources used. Doing so requires being able to navigate different kinds of sources and forms of knowledge, including oral records, expert testimony, and scientific consensus.

We begin this chapter by explaining why assessing the authority of a source is not always as simple as it seems. Then we provide guidelines for assessing traditionally trusted sources, such as official documents and data and statistics, and traditionally overlooked sources, such as social media and oral records. In Identifying Experts, we discuss the notion of expertise, and in Scientific Consensus, we turn to science reporting as a special case study in navigating and expertise.

The Turtle Problem

When journalists report a story—whether breaking news, someone's personal narrative, or the result of a years-long investigation—they do so by relying on sources. The fact checker enters the picture afterward, and their job is to ensure that what the reporter has prepared for publication lines up with what the sources say. But how far back do a fact checker's responsibilities go? How can we know that the sources the reporter used are reliable, that what they say is correct?

The answer, to a large extent, is that journalistic practice is grounded in trust. A fact checker working on a science story can't run an experiment again, on their own, to make sure the results of the study they're citing are "really true," nor will a fact checker working on a story about a politician's speech always be able to consult a video of it to make sure other sources' descriptions match. The entire fact-checking operation relies on the editorial team's confidence in the sources they have chosen for a given topic.

However, this doesn't mean that fact checkers should trust indiscriminately. On the contrary, we should be interested in confirming the accuracy of our sources as well as the relative accuracy of the work we publish based on those sources. But this can't go on forever—at some point, a fact checker's search for trustworthy sources necessarily bottoms out.

In a conversation with us, one editor at a national magazine suggested drawing an analogy between the source that confirms a fact and the turtle on whose back the earth rests according to some creation stories. People ask: "But what does the turtle stand on?" And the standard (humorous) response is: "It's turtles all the way down." Here is the fact checker's Turtle Problem: to avoid an infinite series of sources (the first confirming the fact, the second confirming the reliability of the first, the third confirming the reliability of the second, and so on), we are required to consider some sources reliable enough to stand on their own. The question is how a fact checker can recognize those sources.

As outlined in Chapter Three, the solution we propose to the Turtle Problem is that of authority: a fact checker's search bottoms out once they've reached a sufficiently authoritative source. The notion of authority is at the foundation of most fact-checking work: it allows us to explain why there is something wrong with attributing a scientific fact about the climate crisis to a single study financed by a fossil fuel company or a fact about someone's personal identity to someone else who doesn't know them personally. Those sources are not authoritative for the facts at hand.

In this chapter, we deconstruct this idea. Relying on authority is inevitable, and most of the time, operating with journalism's traditional distinctions between primary and secondary sources is useful and reasonable. But that reasonableness is determined in the context of each story. A fact checker should be ready to debate, reconsider, discuss, and assess each source used in a story, and should learn not to default to the usual sources without reasoned consideration—especially in the context of problematic power dynamics, such as the historical marginalization of certain communities by others.

Consider a story that includes historical demographic data about the Mississauga First Nation—for example, how many people lived on the territory 100 years ago. Standard fact-checking methodology dictates that these facts would be confirmed by primary government sources, such as the records of Statistics Canada or Indigenous Services Canada. But we know that, in the first half of the twentieth century, government agents destroyed at least hundreds of thousands of files (likely many more) pertaining to Indigenous peoples, and as a result, the existing data is inaccurate. A journalist could find additional data by relying on sources that have traditionally been considered less valid, such as the oral records kept by the First Nation itself. If a fact checker refuses to consider a demographic fact grounded in oral records "checked" because they could not find a government record to confirm it and they do not consider community-based data to be authoritative, then they are arguably perpetuating the Canadian government's erasure of Indigenous history.

Fact checkers therefore have considerable responsibility, alongside the rest of the editorial team, to think critically about the sources we defer to. We are not categorically beholden or restricted to any one source; instead, our job is to understand where a source's authority comes from and when (if ever) it is justified, from a fact-based-journalism point of view. As we will emphasize in this chapter (and subsequent ones), alternative sources often do exist to confirm or refute facts for which traditional "primary sources" would be inappropriate—finding them just sometimes requires more creativity and flexibility on the part of the reporter and the fact checker. If you can't find truly authoritative sources to confirm or refute a fact (which certainly does happen), what you can find, and the reason you can't find more, often becomes an interesting part of the story itself.

Writers give you sources, but should you go beyond their sources? The answer is yes, when it makes sense to do so. Evaluating sources is not a science; it's an art.

—Peter Canby, former head of fact-checking, The New Yorker

It is your job to judge the authority of the sources provided by the writer. Any checker can simply go over the same territory the writer covered and catch errors of transcription or misunderstanding. A good checker recognizes when to go beyond the writer's sources. If the writer's source seems dubious, find another one, or two.

—Cynthia Brouse, After the Fact: A Guide to Fact-Checking for Magazines and Other Media (2007)

Assessing the Authority of Sources

Something most fact checkers know and act on instinctively is that the authority structures we work with in journalism are fact-relative: any division of authoritative and non-authoritative sources that doesn't vary based on context is far from useful or conducive to accuracy (which is why we chose to stray from the standard "primary sources" formulation). Government documents are most authoritative when used to confirm the government's stance on the issue at hand. Similarly, policy papers published by an advocacy organization are most authoritative when used to confirm the organization's stance. Someone's childhood memories can tell you about their memories, not about their relatives' memories. An expert in a certain field can tell you about the consensus in their domain, not in that of their colleague.

In this section, we expand on this idea of fact-relativity. As discussed in Chapter One, we will rely on the Internal/External Fact Distinction and the Authority Principle, which holds that a source is always authoritative regarding its internal facts but only sometimes regarding its external facts, depending on how those facts were gathered and the quality of the source. Two important tasks of this guide are to help you identify which facts are internal and external to any particular source and to help you decide for which external facts that source can nevertheless be considered authoritative. This provides a solution to the Turtle Problem.

This chapter focuses on gathered sources (as opposed to interviewed sources): records that have been created by people at an earlier stage and accessed by the reporter later on. Gathered sources can be published documents, online media, or journal articles, to cite just a few examples. What distinguishes them is that they have been recorded; the fact checker can consult them independently and repeatedly. (Oral records can be thought of as a special case of gathered sources, since they can be accessed only via conversation with others, but they are still records in the sense that interests us here.)

It can be helpful to think about fact-checking as a skill that requires you to navigate between different forms of knowledge as they exist in different kinds of sources—printed records, oral records, lived experience, data and statistics, scientific research—and choose the most appropriate for confirming the fact at hand. In an abstract sense, a fact checker is someone who translates information from various forms of knowledge into fact-based journalism, which is its own form of knowledge with its own standards and norms of transmission. As the Knowledge Principle tells us, a good fact checker should understand the context in which a fact is stated, and they should know how to operate within that context as well as outside of it. This chapter will include a number of examples along this vein, since this is especially important to keep in mind while fact-checking with gathered sources.

Some facts are situated in certain forms of knowledge, so they are best confirmed using certain kinds of sources. The fact of Missing and Murdered Indigenous Women and Girls (MMIWG), for example, is poorly documented in official records due to a lack of tracking and coordination between police departments, local governments, and other authorities. But we still know that the problem exists, in part because of the significant work done by community organizations to gather oral records, create crowdsourced databases, and compile testimonials of lived experience. The lack of officially published records doesn't make the fact that thousands of Indigenous women, girls, and Two-Spirit people are missing any less true—or any less checkable. Fact checkers just need to know which sources can help check a certain fact, which sources cannot, and why.

A general rule for assessing the authority of gathered sources is that the more transparent the source is—the more you can find out about how its content was compiled and what kind of verification process it underwent—the better. As we outline below, it's always worth asking some basic questions about a source's reliability, and if you can't answer them, you can always call experts and ask for help. The beauty of fact-checking, when you have the time to do it right, is that you're never on your own.

Official Documents and Publications

Reports distributed by institutions, documents published by government or legal bodies, content shared by established publications, and other official public documents are often taken as the archetype of credible sources. Indeed, documents of this kind are helpful for fact-checking: an official document such as a minister's briefing is a good place to start, and it would certainly be appropriate for confirming the ministry's stance on an issue. But official documents are not always where fact-checking bottoms out.

There's an important caveat to the reliability of officially published documents: like all other documents, they were created by people—in this case, by people whose aim is to lay down the "official story" for the public or for their internal organizational needs and priorities. To create the document in question, they chose some facts, left others out, and arranged them in a particular order to make a particular point. This is not necessarily problematic—it's just how writing works, including in journalism—but it means that, sometimes, the documents are less helpful for confirming external facts and more helpful for providing us with internal facts about themselves and the people who published them.

For any published document, the internal facts are those that are transparent—for example, the fact that the document says what it says, regardless of whether it's true or not. If the reporter writes The government report says that the minister will resign tomorrow, then your checking that the report does say that can be done by simply looking at the document and finding the relevant phrase. But, if the reporter writes The minister will resign tomorrow and attributes that external fact to the government report, you may no longer find that document authoritative—you may want to find a second, independent source (or perhaps call the minister directly) before considering the external fact confirmed. As a general rule, if it's possible for the document to be wrong or misleading about a fact, then that fact should be considered external to the document, and you should think about whether the document should be treated as authoritative. (Sometimes it can, but not always.)

If you want to figure out whether an officially published document is an authoritative source for external facts, you can put it through a series of tests, as listed below.

Four Tests for Assessing the Authority of an Official Document

(1) Author Test: Who created and published this document?

• Are they qualified to speak on this topic?
• Do they have a good track record in terms of accuracy?
• If the document contains information about a certain community, have the author's past publications been well received by that community?
• Do they have a stake in the issue at hand that would reasonably compromise their credibility?
• Was this document created at the direction of a person or entity with a clear stake or interest?

(2) Quality Test: What is the research quality of the document?

• Does it explain its methodology and cite sources for external facts?
• Are its claims accurate and consistent, as far as you can tell?
• Are there internal contradictions that aren't acknowledged or addressed?

(3) Integrity Test: What language and contextualizing information is used?

• Are facts conveyed with the necessary and appropriate contextualizing information?
• Is the language appropriate for and approved by the relevant communities?
• Is it careful to articulate nuance and complexity?
• Does it make explicit or implicit claims about causation without evidence?

(4) Comparison Test: How does it compare with other sources?

• Does it contradict other sources that are authoritative?
• Are these contradictions acknowledged and explained?

If the document fails one of these tests, it may still be authoritative for the fact(s) at hand, but there is reason for you to dig a little further—you should be able to justify why the document is an exception to the general rule. The Author Test checks whether the document's publisher is motivated to skew facts in a particular direction, the Integrity Test checks whether their presentation of the information is (involuntarily) biased, and the Quality Test and Comparison Test check whether their work is reliable. When in doubt, try to speak with the authors of the document and discuss it with them.

Press releases are a common case of non-authoritative published documents—the archetypal example of failing the Author Test. Despite the official veneer of these documents, we advise against using them to confirm external facts most of the time. A press release can certainly confirm uncontroversial facts about the company or person that issued it, such as the spelling of a company's name, its location, and its founding date; it can also be useful to indicate the point of view of a person or organization and to establish a timeline. But, for anything more complicated, like financial information, it's better to use more direct sources, such as company records. And, for anything else, even something as seemingly innocuous as "This is what happened in our lobby yesterday," it's even less authoritative; it really just provides one version of the events, and other sources may have their own versions.

This lesson can be applied broadly: reports are not necessarily good sources of information about people who were not involved in writing them (i.e., their external facts).

When reporting on a mining project that happened 100 years ago, for example, your primary sources are typically government documentation and company documentation—the people who lived in the nearby community may be really hard to find, maybe because they've passed away, were displaced by the mine, or were written out of official accounts of the mine. So, if you were considering government and company documents to be your only primary sources for the story, you would be deferring to the official version of events...We obviously want to counter this mindset. It's similar to the situation when, if someone accuses the police of violence today, a journalist refers to the official police report, where it says nothing happened, and then writes that the allegation 'could not be confirmed.'

—Saima Desai, Editor, Briarpatch

Using official documents and police accounts as an objective source of truth is more than inaccurate—it's a source of harm, of oppression. It allows the media to be used to further narratives that oppress, exclude, and discriminate against marginalized and criminalized people. So it's an ethical issue, an accuracy issue, and also a boring story...Here's my broad principle: the people who oppress or harm a group of people are not the objective source of information about those people.

—Chanelle Gallant, activist, organizer, and writer

If a document about a certain community uses language that community doesn't use or approve of, then that document likely fails the Integrity Test. For example, a document about the harms of drug use that repeatedly uses the term addict when most people prefer the descriptor person who uses drugs likely isn't an authoritative source on the perspectives of that community. Similarly, an investigative report about someone's claim of sexual harassment at work that repeatedly uses the word tryst to describe the incident likely is not an authoritative source on the complainant's experience. (See Chapter Five for more discussion on this point.)

In conclusion: when trying to assess the authority of a publicly released document, you should first assess the kinds of facts that document is qualified to confirm or refute. Then, see how it handles those facts—whether the information is coherent, placed in the right context, and well presented. Finally, during triangulation, see where other documents line up and don't with that one (on the facts that they are qualified to confirm). If all these things look good, you're usually safe.

Most of the time, official reports are an important part of your fact-checking work. But they can, and sometimes should, be balanced with other types of reporting. A fact checker (like the reporter who came before them) should be open-minded about the kinds of sources that can be relied on to confirm facts. Once you understand the context behind an "official document, there's no reason to consider it necessarily more reliable than people's quotes or lived experience about the external world. As usual, it all depends on the fact at hand.

When reporting on homelessness in the Bay area, reporters often don't go out and get the stories from the people within the community, and often rely on local authorities or a press release; it's a big problem, because we do need to go to members of that community and see what happened from their perspective and do our due diligence to corroborate what they're saying. And, if you have time, it's important to put that to the authority figure in that situation.

—Meghan Herbst, Research Editor, Wired

Digital Sources and Social Media

Online, there is often no clear delineation between "official" and self-published information, save for obvious cases such as government and company websites. Nefarious agents sometimes try to fool reporters into taking the information they publish as authoritative when the opposite is true. The sheer accessibility of information on the internet can also be problematic for fact-checking: spend enough time online, and you can find supporting evidence for almost any "fact."

Online sources certainly require care and skepticism. But, beyond the technical skills required to navigate the internet, there's nothing particularly new about this type of source from a fact-checking perspective. In short, the internet is not more or less reliable than any other publication platform. The fundamentals of verification still apply: whether online or offline, it's the context in which something is published that matters most in assessing its reliability. What's exceptional about the internet is that assessing the context of online content may require more work and specialized skills on the part of the fact checker.

Before we had the Internet, fact-checkers used the telephone and the library to find information. The first question they asked themselves when confronting a fact was: "What would be the most authoritative source for this fact?" The second was: "Where would I find that source?" Today Google encourages us to put the cart before the horse, to hunt for sources before we've asked ourselves that first question. If we're not sure what the most authoritative source might be, it should not be surprising if we don't find it when we look.

—Cynthia Brouse, After the Fact: A Guide to Fact-Checking for Magazines and Other Media (2007)

The internet has democratized knowledge in a way that has helped fact-checking more than it has harmed it. It provides the opportunity to find and speak with people who might have otherwise been unreachable, and often to verify information that may have been omitted from official records. Some communities have been excluded from or do not often participate in official or mainstream forms of knowledge production, and some people have limited accessibility to certain forms of communication. A rejection of online sources can also be a rejection of certain people's ways of expression and identity—and a dismissal of facts that are just as reliable as what has been published on paper.

Social media plays a really big role in LGBTQ communities because that tends to be one of the only places where people can comfortably present the way they want to and have the freedom to speak, be referred to the way they want to be referred to, and establish those norms with their community.

—Sid Drmay, freelance journalist

Digital literacy and a keen critical eye are important for fact-checking digital sources; just googling a fact you're trying to check without having a sense of the kind of source you're looking for is not a good strategy. Some facts are best confirmed using official websites and company pages. But social media and forums can also be quite helpful for confirming details about people's identities or community experiences or for sourcing images, videos, and eyewitness accounts of an event. As a general rule, Wikipedia should never be used to confirm facts about anything other than Wikipedia and its users, but it can still be a useful starting point for finding other references in your research.

For any gathered source, you should ask yourself: Is it the original? Who produced it, when, where, and why? How did they come to know the information they're now publishing as fact, and are their methods reliable? For online sources, finding answers to these questions means visiting a website's "About Us" page or scrolling to see if they have a copyright notice at the bottom of their home page, checking social media accounts, searching domain registration records, and investigating what other reliable sources have found about that source—a variation of what you should do in any fact-checking situation, whether online or off. The main difference about working with the internet is that there are more sophisticated tools for conducting your check. Craig Silverman gives a helpful overview in his Verification Handbook series, which we recommend consulting. These tools can be useful if you're fact-checking an investigative or "very online" piece, but most of the time, healthy skepticism is all that is required.

It's helpful to think of the content of social media and forum posts as equivalent, fact-checking-wise, to the content of any other documentation about a person with whom you're not interacting live. If the story you're fact-checking includes facts obtained via Twitter, Facebook, LinkedIn, Pinterest, Reddit, or any other such platform, use the same standards that you would use for fact-checking information you found in, say, a written report. That means, first, trying to speak directly with anyone named or identifiable in the piece to check the facts that concern them; second, assessing the source's reliability (checking for red flags and triangulating with other sources as much as possible); and, third, being transparent with readers about where the fact comes from.

If facts were taken from a public social media post, ask yourself: How do I know this person is who they claim to be? How recent was the post? Could their opinions or circumstances have changed? Is the post missing any information or context? If you couldn't confirm the statement with the person directly, ask yourself whether you can reasonably still attribute it to them. If not, you might suggest a correction to the story making it clear to readers that the statement comes from a social media profile in their name and not from them directly. It's good practice to attribute a quote to someone only if you can confirm with them that they actually said it—which, again, applies beyond the internet. (In fact, it is a consequence of the Personal Principle, which we discuss in more detail in later chapters.)

One additional consideration comes from the special relationship between social media and people's identity and expression: the privacy of someone's post should be reflected by its treatment in a work of journalism. If the post is public, it's fair game. If facts were taken from a private post, however, contact the source directly, confirm that they consent to being quoted and/or identified in the story, and if they do, find out how they want to be identified. If the reporter attributes a quote from a private post to someone's digital pseudonym, make sure it's okay to use that pseudonym. You might think that identifying the source by their handle protects their identity, but they could be easily recognizable within their online community.

Data and Statistics

In Chapter Three, we wrote that there are three pertinent kinds of facts about data: (1) what the raw data represents and how it was collected, (2) how that data was subsequently interpreted and analyzed (for example, by data analysts or by the authors of a scientific paper), and (3) how the journalist represented and used the data in their own story. We then provided guidelines for making sure that the reporter has understood and conveyed numbers properly in their story. Those guidelines can help you check facts of kind 3, which are your responsibility as fact checker. But facts of kind 1 and 2 are equally important; just because the reporter has used the numbers properly in the story doesn't mean the numbers themselves are as reliable as they seem. Fact checkers benefit from knowing how to dig deeper into the numbers in order to assess a source's authority.

Numbers sometimes give sources the appearance of utmost authority. People "often take statistics as objective facts that exist without any context," says Tahu Kukutai, a Professor of Demography at the University of Waikato. But acting on this assumption as a fact checker can lead to errors; according to Kukutai, an overly deferential attitude toward the accuracy of numbers is "highly problematic." There are several different layers in the data-production process, including its collection, aggregation, classification, dissemination, and interpretation, and each layer can introduce errors and biases. The "fact" produced at the end (published in a survey, academic paper, institutional report, or elsewhere) can hide its errors behind the apparent authority of hard numbers.

It is important to understand what the data you're trying to fact-check represents; if you can't find out anything about how a number was reached or what precisely it means, then it may not be reliable. If you discover during fact-checking that a significant group of people was excluded from a survey or study, that survey questions were asked with a particular bias or in unclear language, or that the data is contested by the people it's about, then you also have good reason to suspect that the results are not as reliable as they may have initially seemed.

The list of questions below should help you to think critically about numbers and data beyond assessing whether the journalist has represented them accurately:

Questions to Ask When Assessing Data and Statistics

• If the data is interpreted, aggregated, and published as part of an official document, is that document authoritative?

  See Four Tests for Assessing the Authority of an Official Document.

• Is the source itself presenting the data accurately?

  See Guidelines for Reporting Accurately on Data and Statistics.

• Who gathered and analyzed the data?

  Are they qualified to do so? What is their relationship to the data? Who funded their work? (If you can't find anyone "behind" the data, that's not a good sign.)

• How was the data gathered?

  If it was self-reported (for example, in a survey), is that a problem? What was the participation rate? What is the error rate? Who or what was included in the sample, and are there any significant exclusions?

• Is the data missing context?

  Are there any important limitations? What is the data not covering?

Data is certainly helpful if you can understand and qualify it well. It can provide concrete evidence and ground qualitative facts in an important way. The inclusion of COVID-19 infection data in journalism stories about the pandemic, for example, provided evidence for the seriousness of the situation and likely helped readers assess health risks more accurately. However, COVID-19 data and statistics from different sources were rarely consistent or clear, and there were instances—especially early in the pandemic—when governing bodies spread incomplete or incorrect information, thereby contributing to confusing and divisive messaging. In those cases, the apparent authority of numbers was used to mask the uncertainty lingering underneath—which goes to show why fact checkers need to understand the story behind the numbers before they can assess their credibility.

In data journalism, the classic example of how hard numbers can be a cover for messy context is government-gathered census data about race and ethnicity. Any time you encounter data about a group of people who were not involved in producing that data, as is the case for most ethnic census data, Kukutai emphasizes the importance of "doing your homework." You may still end up using that data, but you should make sure that the community it describes gets a chance to contribute their own information if they have it.

Take this example from the Toronto Star. The 2006 Canadian census found that there were 29,245 people who identified as Tamil in the Toronto Census Metropolitan Area. However, Statistics Canada also found that 98,265 people in the same region described Tamil as their mother tongue. And, in 2009, one of the newspaper's immigration reporters wrote based on expert interviews that there were 200,000 ethnic Tamils in the GTA—a significantly higher number. As a fact checker, how can you tell which number is correct for describing the Torontonian Tamil community? Although the Star's library team argued that Statistics Canada is likely most authoritative, the reporter disagreed: "In this case, I think the academics have made a good case for why a community made up of war-scarred refugees with an intense loyalty to Canada might not identify themselves as just Tamils," she reportedly told the newspaper's public editor at the time, Kathy English.

English concluded that both the experts' number (200,000) and the Statistics Canada number (roughly 30,000) should be cited, with explanation, in the newspaper's reporting on Toronto's Tamil population. "The Star's library makes a good case for consistently citing Statistics Canada's 'official' data in the Star's reporting on populations," she wrote. "But for the widest, most accurate picture of any population, other relevant data should not be ignored."

We always say that you have to take Statistics Canada numbers with a grain of salt. They are usually the only official numbers we have that reflect the entire Canadian population. But we also know those numbers have limitations. Many people are not comfortable with providing information to the government. Some people may not provide honest answers to poll takers. Because of that, we may not get a clear picture of certain groups and communities from official sources like Statistics Canada or from national surveys. Therefore, it's important to balance official numbers with context and insight provided by individuals working in or with certain communities.

—Astrid Lange, Library and Research Specialist, the Toronto Star

As much as you can, try to determine whether the numbers in a story are an authoritative source for the fact at hand. Whenever you find it difficult to answer the questions about data listed above and you suspect there is cause for skepticism, it's a good idea to call the author of the report, the person who gathered the data, or any data analyst and ask them to walk through the numbers with you.

If you can't find an answer, leave a comment in your checking copy noting that you weren't able to confirm the reliability of a number. As always, transparency is key: explain exactly why you feel there may be reason for skepticism so that the handling editor and reporter can decide on reasonable next steps.

You can't always take numbers at face value...My experience with using data sets is that journalists have to go into the technical documentation that describes how the data was collected and characterized in order to really understand. And you may need to speak to some other people that may have used the data before to get a sense of whether it's good data. Not all data that's published is actually good data.

—Tai Huynh, Editor-in-Chief and Publisher, The Local

If there is a data set that isn't within your expertise, humility is always a great quality. Instead of trusting a source that you believe to be problematic, you might look for a topic expert whom you know will be less partisan to help you interpret the data.

—Denise Balkissoon, Ontario Bureau Chief, The Narwhal

Oral Records

Historians and academic researchers have long known what fact checkers are just beginning to acknowledge today: that reliance on a traditional notion of "primacy sources" is not always conducive to accuracy. One important lesson we can learn from historians is that historical research cannot be separated from historical problems. If a person or group of people were marginalized or oppressed by others, any official historical records about them are probably unreliable or at the very least incapable of telling the whole story. Sourcing expectations for a story about a high-ranking Confederate general in nineteenth-century Virginia, for example, cannot be the same as those for a story about an enslaved person who lived in the same region at the same time.

Although journalists may feel that there is always an objective "fact of the matter" about a certain historical event, we must admit that it is sometimes frustratingly out of reach. The standards we have for confirming a historical fact will vary based on what kinds of records were created at the time, by whom, for what purpose, and which of them can still be accessed today. Historians have adapted to this challenge by developing a practice of reading "against the grain"—that is, working with the limited resources they have from a certain time period while remaining critical of the material and conscious of what may be missing from the official narrative. They have also balanced a flexibility in sourcing—drawing on various sources such as artwork, handicraft, and oral records—with a rigorous internal process of verification. We argue that fact checkers and journalists in general should adopt a similar attitude.

The challenges of historical research reflect the way that marginalization produces knowledge problems. And I don't think it's journalists' job to solve all of those large problems. I think it's their job to expose whose fault it is that there are those problems in the first place, and to say, 'Look, this is just a problem.'

—Cassius Adair, Assistant Professor of Media Studies, The New School

The period I write about is a time when city leaders, officials, politicians, citizens and historical records gatekeepers did not value African Americans' perspective or voices, so they did not record the things they said or preserve their documents...Although there are a few exceptions, for the most part, the lives, political views, accomplishments, thoughts, intellect, and cultural practices of the masses of everyday African Americans were not deemed valuable. So historians like me, who are committed to African American history, must seek to find them in the sources left by whites. Some of the sources are dripping with vile racism and white supremacy, which was difficult for me to stomach....

You have to "read against the grain" the sources you do have. For example, if all you have are slaveowners' journals and no diaries written by African Americans, then you cannot just throw up your hands and say a book on African Americans in slavery cannot be written because they did not leave primary sources. Much info can be gleaned from the slaveowner's journal or other sources.

—Nikki M. Taylor, Professor of History, Howard University, in an interview with the Cincinnati Enquirer (2022)

For years, erroneous Western beliefs that the written word is more trustworthy than oral histories have threatened and damaged traditional ways of passing down knowledge...Information found in Euro-Canadian historical documents, such as the records, notebooks, diaries, letters or photographs of missionaries and settlers, can also be useful in understanding the past. As in any situation, however, the points of view of the sources' creators need to be taken into account, as many of these sources were created by colonial government employees or potential supporters of colonization.

—Caroleen Molenaar, "Indigenous Oral Histories and Primary Sources," The Canadian Encyclopedia (2020)

Most of the time, fact checkers are provided with the sources they need by the reporter. But they should not make the mistake of thinking that the kinds of sources journalists are most used to working with (namely people, official documents, and websites) are the only ones available. The Knowledge Principle holds that fact checkers should be capable of navigating between different forms of knowledge, which are confirmed with different kinds of sources. How you carry out your fact-check will depend on the kinds of facts you're working with: you shouldn't confirm an oral fact by looking for a documentary source, for example.

In the next section, we will look at oral records as an example of a different form of knowledge—one that can be as useful for journalists and fact checkers as it has been for historians. Understanding oral records as the sources of facts opens up many possibilities in fact-checking: if a source tells you, "I have no printed record of this, but everyone in the community knows it," that doesn't mean the fact is uncheckable. It also doesn't mean you have to take them at their word. It means you will need to understand what checking that fact within that community looks like, which entails knowing how community knowledge is passed on.

In what follows, we differentiate between knowledge preserved and passed down orally in various communities, which we call an oral record, and history passed down in an Indigenous community based on certain established cultural protocols, which is a specific kind of oral record that we call an oral history (though alternative names, such as oral tradition or oral narrative, are also sometimes used). Both constitute oral forms of knowledge, but they are grounded in different norms and traditions, of which the fact checker needs to be aware.

The Collaboration Principle is applicable here: to determine how to appropriately check facts using oral records, a fact checker will almost always need input from the interviewed sources and possibly from other community members. You must be open to a variety of possible answers while maintaining your critical fact-checking mindset.

Stories about marginalized communities may not come with the same kind of paper trail as those based in the dominant culture. In journalism, this requires reporters (and consequently fact checkers) to rely on oral records and testimonials more than they otherwise might. Journalists working on a story about slavery in North America, for example, can rely on records of oral testimony made by (formerly) enslaved people. LGBTQ2S+ communities also have an established practice of oral record-keeping. And most research on the Holocaust is based on testimonials gathered after the fact.

You should fact-check oral records orally—if the reporter attributes a fact to oral records, don't expect to find a written record, and if you do, don't expect that it will necessarily be accurate or authoritative. Instead, try to speak with the person or people who can confirm the content of the oral record. Information passed down in oral form is just as reliable as information passed down in printed form—it's just a different form of knowledge. Fact checkers should approach the oral information presented in a story just like they would anything else: verify that what the reporter has included in the story accurately represents the facts as they were preserved in the original source, then confirm that the source itself is authoritative for the fact at hand. Assessing the authority of an oral record will require you to understand enough about the fact to know which questions to ask, what verification of that fact would look like in an oral framework, and how to communicate those details transparently to your audience if needed.

Not all information passed down orally can count as an oral record, however. Family stories, which are discussed in Chapter Six, are a good example: they are oral records in a rough sense, but they typically lack the kinds of verification and collaboration processes that make oral records reliable.

Of course, the information found in an oral record can be triangulated with information found elsewhere, such as in documentary sources. For example, if the oral record mentions the date of a specific event, and printed records at the time also mention that event, then you should check to see whether the details line up and include any findings in your fact-checking comments. But finding inconsistencies between oral and printed records doesn't necessarily mean that the printed record is the correct one; you might have to do some more investigating to find out why they don't line up, and if the answer is not clear, you might recommend including both versions in the final version of the story.

For the sake of transparency, it sometimes makes sense to specify in the final version of the story that a certain fact is grounded in oral records. But it's not always required that you do so—just like it's not always necessary to specify that a fact comes from a printed document. It depends on whether saying so will tell the audience something that is pertinent to the reliability of the published statement.

Imagine that a reporter has written a story about a small dinner party that took place in Washington, DC, in 1992, and everyone who was at that dinner remembers that an AIDS and gay rights activist named Peter Staley was there. Yet the host's diary of the day does not include Staley's name on the guest list. It would be reasonable to mention both of these facts in a reported story: oral records of the event all line up to place Staley at the dinner, yet the written record does not confirm it. (You would have to confirm that both of these statements are accurate, of course.)

However, the fact-checking situation would be different if the written record that contradicted the oral record were not authoritative or credible—say, because the dinner host was then senator Jesse Helms, whose home was wrapped in a fifteen-foot condom by Staley and other activists in 1991 to protest Helms's opposition to gay rights. That fact would make Helms unlikely to admit to having invited Staley to a dinner party—though it would explain why all the other guests you interviewed remember his attendance thirty years later. In this purely hypothetical case, the oral records would likely trump the written one, though it would still be important to cite the kinds of sources used in the text of the story and in your fact-check.

Another example of disagreement between oral and written records comes from treaty negotiations between Indigenous peoples and the British Crown over the past few centuries. As Cree scholar Sharon Venne has written, Cree oral records give a detailed account of what was agreed to during the signing of Treaty 6, which is different from what is stated in the government's official written records. Cree Elders "have long disputed many aspects of the government of Canada's version of Treaty 6," Venne writes. "The main criticism of the written version has to do with the language used about the lands. The written version contains the wording cede, surrender and forever give up title to the lands. The Elders maintain that these words were not included in the original treaty." Here, as the Power Principle tells us, there is no reason to take the government's records as more authoritative than the Elders'.

Indigenous Oral Histories

The guidelines above apply to all oral records, but things are a little more complicated in the context of Indigenous oral history, which is usually regulated by strict cultural protocols and standardized processes for passing down knowledge.

The term oral history is used in many different contexts, and it is not always given the same significance. If a reporter attributes a fact to oral history, it's important to know exactly what they meant by that before you try to fact-check it. In many Indigenous cultures, oral histories are concerned not with individual ancestries but with community history. It's usually cultural protocol that the provenance of the oral history is well tracked and preserved over the course of transmission: an explicit part of the oral history is information about where it comes from, who told the story, etc. (Many of the quotes and details that follow in this section are specific to Indigenous communities in North America, but the point we aim to make is broader: you must learn about any community's knowledge-transmission protocols before you can reasonably verify information about and from that community.)

When checking a story that involves a community's oral history, here are the questions you should be asking along the way. (Ideally, the reporter will already have answered them.) We will elaborate on each question later in this chapter.

Checking Oral History in Six Steps

• Step 1: What community is the fact or story from?

• Step 2: Is oral history the right term for describing the kind of knowledge involved?

• Step 3: What are the community's cultural protocols around transmitting and sharing that oral history (or whatever the right term is to describe the fact in question)?

• Step 4: Are you following those cultural protocols during your fact-check? In particular, are you checking the story with the right person, and do the reporter and publication have permission to share it?

• Step 5: Is there a printed record the community would prefer that you cite in your journalism instead of the oral record?

• Step 6: If the version of the fact or story that you have confirmed contradicts other versions, are those contradictions problematic?

If something is being described as Indigenous oral history in a piece you're fact-checking, you should first make sure that oral history is the right descriptor. Keep in mind that a wide variety of terms are used across cultures: different Indigenous communities may use different terminology to refer to their oral history (such as traditional knowledge, oral narrative, and oral tradition), and you should stick to their wording as much as possible. You consequently won't be able to assess whether a statement can be appropriately described as oral history until you know exactly which community it comes from. When presented with something that is described as oral history, you should ask members of that community what kind of cultural protocols are involved in the transfer of that knowledge and what term should be used to describe it.

For example, Jennifer Wemigwans, an author and researcher from the Wikwemikong First Nation, helpfully differentiates between sacred teachings, which are passed on only to Elders and traditional teachers through ceremonial protocols, and personal knowledge, which is gained through educational pursuits and spiritual and empirical processes. Personal knowledge isn't bound by any cultural protocols, though sacred teachings are. This means a fact checker would need to approach these two categories differently: personal knowledge can be verified by speaking with the personal knowledge holder, but sacred teachings and oral history may require a more involved process.

The more specific you can get about the provenance of information based on oral history, the better. In Indigenous Writes: A Guide to First Nations, Métis and Inuit Issues in Canada, Métis writer Chelsea Vowel suggests asking three questions to validate Indigenous stories: Which nation is this story from? Which community is it from? And who from that community told it? If these questions cannot be answered, then the story may not be authentic. Similarly, Tara Williamson, research director at the University of Victoria's Indigenous Law Research Unit, advises asking sources how they came to know the oral history they're sharing. If the answer is "horizontal"—meaning they were told the story from someone else of the same generation—then it's likely that "that's just community talking." But, if the answer is "vertical"—meaning the information was passed down through generations—then that sounds more like oral history. It's also a good idea to ask sources outright whether they would describe the information they are sharing as oral history.

In an oral culture, the means of passing on information is via story telling. That is not to say someone makes up a story, but a story is the manner in which the information is told...Storytelling is an art and a skill among the Cree Peoples, as it is for all peoples who use the oral method to convey stories. A skilled storyteller is a master of the language and of the history of the peoples. It is through telling stories that the histories of the peoples, as well as important political, legal, and social values, are transmitted.

—Sharon Venne, "Understanding Treaty 6: An Indigenous Perspective" (1997)

Stephen Puskas, an Inuk cultural consultant, points out that, for many Inuit communities and First Nations, especially in the Pacific Northwest, you have to know which family a story comes from. Many Inuit stories differ from family to family. According to Robert Jago, a journalist from the Kwantlen First Nation, First Nations families in the Pacific Northwest often hold ownership over their stories. In this case, you should always fact-check that there is consent from a family to publish their story or oral history in a work of journalism.

The idea is not to memorize all of these different protocols for different communities. Instead, be aware of the wide variety of forms of knowledge that you may be asked to fact-check in a reported story. Fact-checking oral history will look different in each case, and the first step in your research should always be to find out exactly what form of knowledge you're dealing with. Then you can work within whatever framework is most appropriate, making sure to respect the local protocols.

For example, in some communities, certain stories can be told only during specific seasons or by particular people. If the oral history protocol of a community is that a story should be passed down only by Elders, you should confirm with the source that they are indeed an Elder (the definition of which differs between nations and communities). And you should further confirm that they are comfortable with your publishing their oral history in a work of journalism despite the fact that you are not part of that tradition.

When Elders begin to tell a story, they will describe in great detail the history which gave rise to the story. For example, Elder Charles Blackman of the Cold Lake Reserve begins: "I'm 68 years old. Of what has been said, I will elaborate some things too, regarding the first signing of Treaty 6. I have heard in many places that I have been, and I've heard a lot the time the Elders assembled at Duck Lake, Saskatchewan. Elders were called up from many parts of the country. The expression the Elders gave then was the same as to what we have been saying." If the listener wants to verify the words of the Elder, they are made welcome to check with persons who were present at the meeting in Duck Lake, Saskatchewan.

—Sharon Venne, "Understanding Treaty 6: An Indigenous Perspective" (1997)

It's important to verify that the community whose oral history you're fact-checking is comfortable with that transfer of knowledge. (As usual, the reporter will ideally already have done this and recorded it in their research package.) Once information from oral history is recorded in a work of fact-checked journalism, it exists in a new form—and it's reasonable to expect that not every community will want to see their knowledge transformed in this way (see Chapter Eight). Williamson notes that, sometimes, the Indigenous Law Research Unit will include only the already-published version of a First Nation's oral history in their reports (usually the version recorded by a visiting anthropologist sometime in the past century or so), even if it's not the most accurate version, because they don't want to be publishing oral stories that have not yet been recorded in printed form. Journalists could reasonably come up with the same kind of arrangement when reporting on Indigenous oral history, while remaining transparent toward their readers about such decisions.

There are many examples of non-Indigenous anthropologists during the 1960s, 1970s, and 1980s publishing collections from Oral Traditions in breach of Protocols. It's important to know this, so you don't republish all or parts of these works without first seeking advice. Just because it's in a book—or especially if it's in an academic book from the 1960s, 1970s, and 1980s—doesn't mean the content was appropriate to publish in the first place, or that it has been published with consent, or that it has been published accurately. In the past, Indigenous cultural material has been subject to interpretation by non-Indigenous people. Today, as Indigenous Peoples seek to reclaim control over their cultural property, Indigenous interpretation is a way of enhancing the cultural significance of the content.

—Gregory Younging, Elements of Indigenous Style: A Guide for Writing By and About Indigenous Peoples (2018)

Finally, don't expect oral history to be perfectly consistent and complete in the way that journalists have come to expect from official records. If you find a direct contradiction within an oral history, you can always ask about it. And, most of the time, if the oral story contradicts a printed version recorded by an anthropologist or the like, the oral version has priority. But you may also find that different knowledge keepers tell the same story slightly differently or include different details in their recounting of the story, and that may simply be part of the nature of oral knowledge in that community.

No one Elder knows the complete story. The information is spread among a wide group of people for a variety of reasons....It is through continuous contact with the Elder that one will hear the complete story known by that Elder. It is not a process of sitting for one hour or an afternoon...The Elders have within their memories a collective history. No one Elder has all the information about a particular event; each has a personal memory which embraces their parents' or grandparents' memory of the details and circumstances of events that took place.

—Sharon Venne, "Understanding Treaty 6: An Indigenous Perspective" (1997)

The story is alive, and we want it to be alive. We don't want it fixed in history that this is the truth forever and evermore. There's a lot of power in the lack of fixed truth in oral history.

—Tara Williamson, Indigenous Law Research Unit, University of Victoria

Identifying Experts

We've established that, although operating within the realm of traditional primary sources is reasonable sometimes, it is only within the context of a particular story that reasonableness can be determined. Fact-checking does not happen in isolation; the kind of verification that a fact requires varies based on its context, its form, and its history—and, sometimes, the fact checker won't be able to determine these nuances independently. In particularly challenging cases, the fact checker can benefit from reaching out to an expert who can help them navigate questions of accuracy and authority.

Finding someone who can speak with authority on a subject is useful and important. This is especially true in the first days of a new and difficult fact-check, when you're trying to get a sense of the story or to resolve technical difficulties you don't feel qualified to address on your own.

When conducting a fact-checking call with an expert on background, follow the instructions given in Chapter Three. Make sure the expert understands that you are not doing original reporting for a story but rather hoping to verify information reported by someone else with them, given their expertise. Carry out the fact-checking interview as usual, creating a document with numbered questions, assigning it a letter, and writing down their answers, but first, explain clearly that the fact-checking call is being conducted "on background": they are helping you with your work, not being interviewed in order to be quoted in the story.

Before you rely on an expert in your research, you need to make sure the person you choose to rely on is appropriate for the job. Essentially, to adapt the words of historian Steven Shapin, a fact checker needs to know who knows. Even in the simplest cases, you will need to assess the expertise of quoted sources. The reporter attributes a fact to someone, an "expert" on the topic at hand, and the fact checker verifies that this person is an appropriate source. This is a relatively easy thing to do for domains of fact where authority and expertise are not a contested issue, and conducting basic background research before calling someone is usually enough. But, if you think that more skepticism is warranted, or if you are looking to find your own expert on a complicated issue, the first question you should ask is what kind of expertise is required for the fact(s) at hand. It's helpful to think of experts as belonging to two general categories: subject-matter experts (or specialists) and community representatives. (Here, we think of "community" in a broad sense—any community that produces knowledge and/or shares experiences or identity, such as the global community of quantum physicists or the community of Tamil residents in Toronto.)

A subject-matter expert can provide an analysis of a topic or community and their history; the archetypal example of a subject-matter expert is a qualified academic researcher. A community representative, on the other hand, is an individual with intimate knowledge of the community to which they belong; crucially, they need to have been designated (officially or otherwise) or accepted as a representative by their community. The leader of an advocacy organization created by members of the community counts as a community representative, for example. It's important to note that community representatives are not always subject-matter experts, and subject-matter experts aren't always community representatives, though of course one person can play both roles in certain communities.

Which type of expertise is appropriate will depend on the nature of the facts at hand. If the fact is framed as something that a community would say about itself, confirming it with a community representative would be appropriate; if the fact is framed as something about a community that members of that community could in theory not even be aware of (for example, historical details or trends), then a subject-matter expert is appropriate. (It can also sometimes be appropriate to look for both types of experts in order to ensure that what the community says about itself lines up with what others say about it. If you find that it doesn't, that's an important thing to note in your fact check.)

Imagine that you are fact-checking a story about housing conditions in the Vancouver Downtown Eastside. Nicolas Leech-Crier, a local who has been involved in the community for years, suggests an academic researcher in health and justice at the University of British Columbia as a subject-matter expert who "has the years of academic experience and knowledge and the credentials required." On the other hand, Leech-Crier notes, "an academic may not know where to start or what to ask from whom" when it comes to telling the stories of people within the community; for that task, a representative such as Leech-Crier himself would be more appropriate.

I will always believe that journalists speaking with appropriate community representatives is really a simple question of respect. In Vancouver's Downtown Eastside, we have some of the types of social issues which, of course, mainstream media has historically represented in generally negative ways...So, then, to be Storytellers...by using the access we have as journalists to honour the voice of that community, according to the personal contexts of people who live in that community, from their intergenerational traumas to their struggles in their disadvantage of socio-economic stratification inspiring revolution...and to do it through the power of language that many do not have that same access to, is a responsibility we as apparently responsible members of the media should really take seriously—just as seriously as medical doctors should take the Hippocratic oath.

—Nicolas Leech-Crier, Reporter and Storytelling & Community Networking Liaison, Megaphone

Journalists have an obligation to ensure that their work is as truthful and well-informed as possible, and covering gun violence requires an additional layer of sensitivity and nuance. When fact-checking stories about gun violence, it's important to consult not just gun violence experts—the people who create scholarship and policy—but also "on-the-ground" experts: community members and leaders whose lives are affected by gun violence.

—Ruth DeFoster, Assistant Professor, Hubbard School of Journalism and Mass Communication, University of Minnesota

Scientific and academic research communities typically have transparent organizational structures and formal institutions that can help you identify the relevant subject-matter experts. Identifying representatives for marginalized communities can be more challenging (and sometimes impossible for a fact checker to do on their own, as we discuss in Chapter Eight). If you are too far removed from the social, cultural, or intellectual context in which a fact is situated, you may not be able to determine the right expert—and that's okay, you just need to acknowledge that fact in your research package and discuss possible next steps with the head of research.

Of course, you are always limited by the resources you've been given to work with, including the original sources used by the reporter. If the reporter didn't do the legwork to find an appropriate expert or community representative, you may not be able to make up for that during fact-checking. But you should definitely do the work required to identify when a reporter has relied on improper representation and expertise—even if you can't solve that problem yourself. The guidelines below should help.

Tips for Assessing Subject-Matter Experts and Community Representatives

• Determine your domain.

  Find an expert in the specific field or community you're hoping to discuss. Just as an electrician likely can't explain to you the details of a plumbing problem, a researcher specializing in Latin American music likely cannot describe the history of Japanese jazz, and someone studying the Wolastoqey language likely cannot confirm Inuktitut spellings.

• Determine their credentials.

  Has the person in question been cited as an expert on this topic in the past? If yes, that's generally a good sign—but it's not foolproof. As Rinaldo Walcott, a professor of gender studies at the University of Toronto, points out, news media tend to return to the same experts or community representatives for interviews even when they are not the best for the job, mostly for the sake of convenience. If you're in doubt about someone's credentials despite their extensive media experience, there's nothing wrong with looking for a second opinion.

• Ask questions about their credentials.

  Does the person in question identify themselves as an expert or community representative (or both), and can they explain why? It sometimes helps to ask the expert directly. Most people will answer honestly and be happy to discuss nuances. (Though keep in mind that some will be too humble to call themselves "expert" and will prefer a term like "specialist" or "researcher"—it's better to discuss someone's credentials than to talk about their specific title(s).)

• If possible/appropriate, ask others about them.

  Do other sources in the same community or domain agree that this person is representative of them or their community? It's important to make sure that the way someone presents themselves is the way they are viewed by the people or organization they claim to represent. A self-proclaimed spokesperson is not necessarily a true voice for the community or field.

• Check for controversies.

  It's important to make sure that the reporter hasn't given a platform to someone who will promote inaccurate information (such as a science denialist or a false community representative). Is the expert making controversial claims, and if so, can they back them up? If, when conducing background research on an expert, you find reason to doubt their credentials or you think that others have done so, don't immediately discount the source: speak with their community and/or other members in their field, and do your own independent research. It may be appropriate to also speak to the source themselves about any controversies. If the issue cannot be resolved, be prepared to find an alternative expert.

• Check for conflicts of interest.

  An expert can of course belong to the community in which they specialize or which they represent, and they may work for an organization that has a specific stance on the subject at hand. There is nothing necessarily wrong with this—in most cases, their position and involvement are what make them an expert. However, if they are receiving funding or benefiting in some way by your coverage of the issue, due to their association with a particular group, organization, or company, then this should, at the very least, be cause for consideration.

Usually, using these guidelines along with your common sense is enough. Remember, though, that this is a lot of work. In practice, it can take a long time to assess the reporter's experts and find your own. It will depend on your own judgment and practical considerations how deep you dig into each expert you rely on during fact-checking.

Here's an example from a few years ago: a fact checker was working on a reported story about an ethnic community in a large Canadian city. The reporter had cited a nurse in a local medical clinic who gave substantial testimony about the sexual health of community members. (She would count as a subject-matter expert, since she was not part of the community herself.) The nurse refused to answer any requests for fact-checking, but the fact checker had an audio recording of her interview and could confirm that all the quotes in the story lined up with what she had told the reporter. But the fact checker was skeptical about the information she had given, and they found it hard to independently confirm the facts that the nurse had provided. After doing some digging into her credentials (by conducting online research and calling the clinic where she supposedly worked), the fact checker discovered that the source was in fact not a nurse but a physical therapist who had worked independently with some of the patients at the clinic. She was therefore unqualified to speak to the sexual health of members of that community. Her quotes were removed from the story.

Scientific Consensus

When fact-checking a science story, it can be helpful to think of scientists as a community of people who share particular norms of research, communication, and verification, and who produce a particular form of knowledge: scientific knowledge. Thinking of science in this way allows you to apply many of the lessons from this chapter to a science fact check, including those about finding authoritative sources, working with different forms of knowledge, and deferring to appropriate experts and community representatives.

Science is an interesting case: because many journalists don't have a science background, we are usually not part of the scientific community and, for the most part, we don't know much about how knowledge is produced within that community. At the same time, scientific results are often debated and discussed in the public sphere, including by non-experts and a minority of so-called science denialists. Journalists are asked to mediate these conflicts—and they often attempt to do so without understanding how the scientific facts in question were produced or what makes science authoritative in the first place. As a fact checker working on a science story, you will get thrown into this mess and will have to know how to navigate it.

Misunderstanding the source or nature of scientific authority can lead to inaccuracy, confusion, and manipulation by others. The scientific research community prioritizes transparency of methodology and clear admission of uncertainty in its work. But people trying to manipulate the situation to their advantage will point to the realities of scientific methodology as reasons to discredit it: they will ask for "balance" in science reporting and argue that journalists should cite more opposing views—climate denialists, for example. That's a problem you may have to deal with as a fact checker, and it can be helpful in these cases to step back and think about the bigger picture in which you're operating—even if you don't understand all the scientific details in question.

Contemporary philosophers of science have proposed to define scientific knowledge not as a collection of stand-alone objective facts but as information that is socially constructed and agreed upon by a diversity of experts. Scientific knowledge is built by collaboration among a huge community of experts and researchers with different perspectives and experiences: it is representative of the scientific community as a whole, even if individual members disagree about the details. That doesn't mean science is foolproof. Rather, what makes it special and authoritative is that it operates via collaboration and consensus. And that's a good thing, from a fact checker's perspective, because it means we should do what we do best: call a bunch of people and ask questions.

Below are general principles for fact-checking a science story. If you're fact-checking a story where a reporter has misunderstood the nature of scientific knowledge, perhaps because they are relying on an individual instead of the community or because they are criticizing the knowledge for its uncertainty, you may not be able to fix the story for them—and, regardless, it's not your job to do so—but you should be able to point out errors in scientific fact and explain where the reporter went wrong.

• Understand how science works.

  As Kathleen Hall Jamieson, co-founder of FactCheck.org, has written, it's important for journalists to honour the "norms and ways of knowing" of science when communicating within and about it. The first thing you need to know is how science works—i.e., what kinds of sources, certainty, and communication you should expect while fact-checking a science story.

  Scientists' methodologies, tools, and results vary widely based on the field of scientific research. How science works depends largely on which kind of science you're talking about. But it's important to acknowledge that, usually, scientific research can't produce absolute certainty, nor does it aim to; rather, it is focused on turning observations into generalizations and predictions. This should be reflected in the kind of language you use to check scientific facts.

  Before fact-checking a complex science story, we recommend that you read this helpful primer on science fact-checking, written by the Knight Science Journalism team at MIT.

  It's important to understand how science functions, because at least in the US, I don't think our education system really explains what science is. It's just not the thing that says, "This is true." It's a process to find out things that are probably true about the world. And it's a very long process, and it's dynamic, and it changes.

  So I think that it's very easy for a fact checker or a reporter—and I've done this in my career too, especially early on, when you're a little bit more naive to how you should be approaching these things—to find a fact in a paper in a well-respected scientific journal and think, well, it says it here, so here's my source. Instead, you need to look at the entirety of the literature on that topic and understand where that paper sits, especially if it's literature in certain areas of science. If it's something that is very well established at this point, like gravity or evolution, then yeah, it's okay to use fewer sources. But, with emerging science that hasn't been as well-established, you need to see where that paper lies in the broader universe of other research on that topic and see if it's all pointing in a certain direction or whether it is an outlier. You need to do a little bit more work to make sure that there aren't conflicts of interest for the authors, and so on and so forth.

  —Brooke Borel, Articles Editor, Undark

• Read scientific papers carefully.

  Here are some guidelines for reading scientific articles (inspired by this helpful guide):

  • First, read the introduction of the paper. Then, read the discussion, interpretation, and conclusion sections. Only after having read them should you read the abstract, which usually omits important information for the sake of space. Finally, look at the acknowledgements section to see who funded the research.

  • Ask yourself how the results and conclusion of the paper answer the specific questions that were asked in the beginning of the paper. How does everything connect? Do you understand how the conclusion was reached?

  • Read as much background research and literature reviews as necessary in order to (at least superficially) understand the context in which a paper was written. Literature reviews are a science fact checker's best friend.

  • Watch for retractions and corrections to scientific papers. You can do so using resources such as Retraction Watch and by visiting the digital edition of the academic journal where the paper in question was published.

  When appropriate, you should also follow the guidelines provided earlier in this chapter on Official Documents and Publications and Data and Statistics.

  A classic mistake is not understanding absolute versus relative risk. So, when a paper says a certain food or environmental factor increases cancer by a certain percent, it's usually pointing out a change in relative risk, or how much of an increase of a risk happens between two specific groups of people. This is misinterpreted as an absolute risk, or the increase in risk that a specific person might have if they are exposed to the food or the environmental factor. Just understanding the language of science and how it functions is really important for a science reporter and fact checker.

  —Brooke Borel, Articles Editor, Undark

• Speak with people.

  It's almost always a good idea to call the author of a paper and ask them to confirm the reporter's characterization of their work, especially for complicated scientific findings. You can also, depending on the circumstance (and with approval of the head of research), send technical language and descriptions to experts verbatim in order to confirm that they are accurate. (You should not read anything verbatim to PR people, however—even if they're doing PR for, say, NASA.)

  If you're looking for extra help or assurance, you can also speak with a scientific expert who isn't part of story and ask them to help you confirm that the reporter has placed the scientific findings in the right context.

  Instead of just citing a scientific expert on a fact, you want to know how they know it. You can call an expert, and after fifteen minutes of chatting, ask, "What are the papers that support this? What are the best papers on this like? Who is doing the research on that?"

  —Shannon Palus, Senior Editor, Slate

• Understand how science denialists operate and know how to identify them.

  Climate denialists crowd out professional scientists in the media: a study conducted by researchers at the University of California in August 2019 found that climate "contrarians" were featured in about 49 percent more media articles than climate scientists. (This is mostly because of the prevalence of online blogs. But, even in mainstream media, contrarians continue to be quoted slightly more than scientists.) You need to be aware of this when checking controversial scientific stories and make sure, to the best of your abilities, that the scientific experts being cited in a story are representative of the scientific community in question.

• Place findings in context.

  Make sure the broader characterization of scientific knowledge in the article is correct, beyond representing the results of a single article. (This is where literature reviews and conversations with experts can be very helpful.)

  Are there any problematic oversights in terms of data or methodology, or any notable open questions in the literature? You can always check with the reporter to see whether you're missing context about why they chose a particular study in their reporting.

  If fact checkers only check a story against the sources a journalist provides and don't dig deeper into the literature, they may miss mistakes or scientist bias.

  —Brooke Borel et al, The State of Fact-Checking in Science Journalism, Knight Science Journalism at MIT (2018)

  A common error made by journalists is to read only one health study and then write a story about its findings as authoritative. In the scientific world, they rely on multiple studies to generalise something... As a journalist, I ask myself, Are we cherry-picking one study that might be a very weak study or an anomaly? When in doubt, I check with people who are in that industry, who know the literature.>

  —Tai Huynh, Editor-in-Chief and Publisher, The Local

• Check the language in the story.

  Make sure the language used by the reporter to describe scientific knowledge is specific, appropriate, and representative of the science (including in the headline and display copy of the story). Of course, the reporter may have done some appropriate paraphrasing and simplification, but you still want to confirm that the scientists will not object to the reporter's descriptions of their work for valid factual reasons.

  If the story includes jargon and technical language, make sure that it means what you (and the reporter) think it means.

  If the reporter is using metaphors or analogies to explain science, run those metaphors by a scientist to make sure that they are reasonable. You will have to use your judgment, informed by your understanding of the science and your conversations with scientists, to determine how much correction to a scientific metaphor is appropriate or warranted.

• Don't claim that science "says" or "knows" anything.

  The words science says are nearly always a red flag in a story: it's rarely true. There is usually reasoned debate and uncertainty among scientists about the details of any particular scientific fact, and there's also no single way of "saying" or "knowing" in science. Shannon Palus advises replacing such phrases with more specific language, such as "The scientific consensus is..." or "Experts agree that..."

  Once you have determined a better phrasing, you'll need to check that that statement is accurate too.

  "Follow the science" is a ridiculous phrase. Science is amorphous and dynamic, involving thousands of people who argue.

  —Ed Yong, Staff Writer, The Atlantic