This chapter is designed to help you understand what tertiary materials are including scientific publications, field diaries from participant observation, as well as tables and stats that can be obtained when looking at sources like census data. You will learn how to locate, find and credibly read tertiary documents while acknowledging that there are limits in how this information can be used in your research project. Finally, you will be instructed on the necessary steps to engage in participant observation if your research requires it.
1. Understanding Tertiary Materials
- Tertiary materials are intended to PROVIDE INFORMATION and in and of themselves, offer no explanations, no theory and very little explicit analysis
- Examples include tables and graphs as well as the census info, large survey data, and statistics often found in government reports
- Tertiary data also includes the raw data (research) in natural and social science papers and even the references and footnotes in bibliographies and newspapers
- Finally, field notes gathered in participant observation can also be classified as tertiary documents
- Participant observation refers to systematic observations and note-taking on site and for a specific period of time in connection with situations such as community interactions, meetings, and conferences (more details below)
- These observations are usually recorded in something called a FIELD DIARY which involves a specialized form of writing designed to capture observations in a way that allows the researcher to use these observations later when writing up their research
- It is the opposite of action research where the emphasis is on participation as opposed to observation
- The key issue to understanding tertiary documents is credibility – while you may want to use this data as evidence for your own research, you must not misread what is in the original documents which is a common thing to do, especially when it comes to interpreting scientific materials
2. Reading Science
A. Interpreting Scientific Materials
- Remember that scientific writing is tertiary insofar as the findings from scientific observations or applications from scientific techniques (e.g. surveys, experiments, modeling) are concerned
- Despite the fact that scientists often talk about their work as being “objective,” it is important to understand that scientific data rarely provides definitive results and no research is ever entirely free of perspective
- Instead, think of science as a body of strategies used for systematically observing, learning about and explaining phenomena
- Different scientific disciplines may have different strategies (e.g. theories vs. case studies vs. observational data)
- Before you report an any research, you must first establish its credibility, and in the social and natural sciences, this is done in two ways: a) any study is discussed in the context of other studies, as part of a body of material that is subject to debate but is otherwise connected, and b) emphasis is placed on the methods used to collect and analyze information
- This means that credibility is essentially established by INSIDERS (i.e. those already familiar with the kind of research and topics under study)
- Ways to properly assess scientific materials include determining reliability (a term used to indicate other researchers focusing on the same problem and following the same methods would reach similar results), validity (indicates conclusions are sound and properly reflect what has been observed and/or calculated), and probability (based on statistical analysis and used to indicate the degree to which research findings are not likely to be due to chance)
B. Common pitfalls when reading science
- While it is perfectly reasonable for a non-scientist to read scientific materials and draw conclusions, you still need to avoid certain pitfalls if your own discussion of science is to be credible
i. Not recognizing the obvious
- Since scientific findings come from many fields of study, you need to understand the methodology and conclusions in relation to the specific discipline itself
- Often, this means you will not know how to read the studies of different disciplines, and the solution here is to rely on science journalism as opposed to original research or to consult someone knowledgeable in the discipline
ii. Don’t rely on one single study
- In the sciences especially, information from a single research study is rarely useful on its own
- Usually, you need multiple studies or corroborating research to produce information considered useful and trustworthy
iii. Don’t depend on anecdotes and intuition
- Don’t use anecdotes or intuition as sources of data as science often produces results that seem counter-intuitive
- While anecdotes and intuition may draw attention to ideas that might otherwise be ignored, in and of themselves, they do not produce credible conclusions.
I am a bit confused. What you mean by anecdotes and intuition?
Sometimes, we are prone to draw conclusions based on single story or on information that seems obvious. For example, the members of a community located near a power plant make observations about the rising number of cancers in their area. On the basis on these observations they intuit that the emissions plant is responsible for this excessive number of cancer clusters. Well, while community members should definitely following up on their intuition, it might be the case that the cancers and emissions have nothing to do with each other. One would need a much more detailed analysis and sophisticated statistical techniques to know if their intuition was right.
iv. Know the difference between probabilities and certainties
- Science is always more or less uncertain and this means that most natural (and some social) sciences trade in probabilities as opposed to absolute certainties
- This does NOT mean that once scientific conclusion is as good as another; it just means that while some conclusions may be stronger and more reliable than others, they are rarely uncontroverted
- Probability is a common measure in a lot of scientific work and probability estimates are based on the idea that if left to chance, there will be a clustering of results around the middle or average with fewer results at the more extreme ends
- Usually, a study is considered noteworthy because the findings deviate from that which would be expected if the result was simple left due to chance
v. Lots of things can’t be proven with science
- Many things cannot be “proven” by research in the natural and social sciences
- For example, lots of chemicals we have banned day were initially considered “safe” for public use, and this change in status was only possible thanks to new research studies which presented new evidence that was absent before – in other words, science can’t always provide us with definitive conclusions no matter how important these issues are to the public
vi. Be careful about mapping one research study directly onto another
- Despite the fact that scientists often talk about replicating research, it is rare for the detailed findings of one study to directly map onto the findings of another study
- This means that there are usually differences between one study and another and the importance of these differences is debated whenever conclusions are being drawn
C. Using findings from scientific studies
i. Find science material of interest to you
- There are many good science databases (aggregators) that will help you obtain useful articles which contain the findings of scientific work
- Examples: Web of Science, Science News, Scintilla, Science Daily, Science Magazine, Reddit Science, Research Blogging, Science Seeker (links to websites required here)
ii. Find science journalism that deals with your public issue
- Sometimes, you will find it far more useful to consult science journalism materials as opposed to original research articles
- In science journalism, information is summarized, the points of controversy noted and the significance of the material is discussed
- This means that scientific journalism tends to give you a broad overview of topics without many of the details found in hard science journals
- You can find science journalism in places like the science section of a newspaper or in general news (e.g. Time, Newsweek, The Economist) or scientific magazines (e.g. National Geographic, Scientific American, Science, Nature, Popular Mechanics, Physics Today)
iii. Consult Professional Publications
- Many professional organizations often report on scientific studies
- Examples include Institute of Electrical and Electronics Engineers (IEEE), Canadian Medical Association, American Medical Association
iv. Consult someone knowledgeable
- Even with diligence, there are limits to what you can learn about scientific research, especially if it concerns a discipline you know little about
- Consulting an expert in the field (even with a specific point of view) will allow you to become more aware about the uncertainties and points of debate about the topic under consideration
3. Understanding Participant Observation
- While participant observation provides a richness of information likely unavailable from any other method, there is no way to avoid the heavy imposition of values, preconceptions and personal predilections of the observer
I find the idea of participant observation fascinating but how to do I ensure my own observations aren’t clouded by my biases, values or personal beliefs?
Those who do participant observation have adopted strategies for handling the problem of observer bias and predisposition. These include: a) lengthy and different periods of observation; b) identification of questions prior to research which predispose the observer to see what otherwise might be missed or discounted; c) using a field diary that include both situational and self-observations; d) searching for corroborating materials such as documents and interviews; e) having the observer play “devil’s advocate” which involves seeking evidence of counter-intuitive insights and observations that fail to confirm expectations and e) using a “triangulation” method which involves locating different points of observation such that different perspectives can be brought to bear.
i. What situation is worth observing?
- The first step in participant observation is to determine which situations are usefully observed in person
- This might include situations officially designated as deliberations such as meetings, legislative committees, inquiries and regulatory proceedings, trials, conferences, public events but may also include places like clinics, libraries, university classrooms or even a hospital waiting room
- Depending on the situation, you may be able to witness power relations, institutional functions and the purpose and role of the setting itself
ii. When to get consent
- Next, you must get consent for observation – this is essential if the situation is not one that is open to the public
- In public situations, the guidelines for getting consent are less clear and somewhat impractical, as it would be difficult to obtain consent from everyone in a hospital, university classroom or public hearing
- In these cases, while you still need permission from someone in charge to engage in your research, you don’t need to obtain consent from everyone involved if no one will be identified personally and no one will be directly quoted
- In situations where people are making on-the-record formal presentations (e.g. legislative assembly or council meeting), you do not need consent as participants expect to be observed and recorded
- However, if you do intend to use their comments for your research, you should get consent first
- In terms of casual conversations, if the person you speak with is unidentifiable and no quotes are used, then the conversation is simply a part of your observations – if not, the conversation then has to be treated more like an interview (see next chapter)
- When in doubt, OBTAIN CONSENT!
Do I need to obtain consent if I take someone’s picture?
If you take someone’s picture and intend to make use of that photo in your research, you need to obtain that person’s consent first. However, if no individual can be personally identified from your photo and you seek no commercial gain from it, you do not need consent although it is courteous to seek it.
iii. Prepare before observing
- Make sure to collect as much material as possible before starting your observations
- This can include items such as agendas, speaker and/or participant lists, background papers, CVs, past media coverage, and meeting rules and practices
iv. Visiting ahead of time
- Try and visit your place of observation ahead of time
- This will allow you to draw a rough map of the arrangements of the room
- This is important, as often, where the table and chairs are placed can say a lot about what is supposed to happen
v. Draw up a list of questions
- It is very important to draw up a list of questions ahead of time that can be answered through your observations
- The purpose of these pre-determined questions is to allow alert yourself about what should be observed closely
- These questions will provide you with guidance about the most important aspects of the situation for your research and will also allow you to challenge whatever preconceptions you have going into your situation
vi. The observation process
- During the observation process, try and sit as far away from the main stage of action as possible without it impeding what can be observed
- It is better for you to not participate actively or draw attention to yourself as your goal is not to observe others’ reactions to you
- While it is important to focus your attention on what is being said, you also want to look around and considered other things such as “who isn’t here?” or “what isn’t being discussed?”
- Your answers to these and similar questions are part of the your observations as well
- Other things to consider include observing body language as well as the expertise and interests of those you are observing
- While you want to refrain from making stereotypes, you may also want to guess at the class and status of participants, and this can include questions such as “Is diversity evident in this situation, or do the people all look the same?” or similarly, “what kinds of people have been invited to speak and what elements are included in their presentations?”
I am feeling a bit overwhelmed by the idea of making all of these observations. What if I miss observing something?
We understand that the list of possible observations is very long, and we don’t expect you to observe everything. After all, paying attention to one thing will mean not paying attention to something else. For example, if you are looking at what is going on in front of the room, it means you can’t see what is happening behind you. Again, that is why having a well-thought out list of questions beforehand is important, as it will go a long way to making sure your observations are sufficient for your research. In fact, you will probably only observe about 15% of your situation in total. That said, limited as it seems, your observations are a crucial element in your research, and what you see is at least as important as what someone else says or what is stated on a document.
vii. Note-taking (Field Diaries)
- The notes you take during participant observation are called a field diary
- These notes should be no more than jottings such as a single word, phrase, scribble-type drawing or shorthand symbol in addition to exclamation or question marks in the margins
- Your notes taken during the event will be incomplete but that is okay because if you spent all your time taking notes, it takes your mind and focus away from the actual observations
viii. Filling in your Field Diary
- Once you have completed observing your situation, immediately go to a quiet place and fill in the details of what you saw and heard
- You will be surprised at how much you recall, but this is only true if you ensure there is a very small window between your observations and completing your notes
- If you go for lunch, check your email, or text a friend in-between, you are creating distractions which will impact your recall abilities
- You need to allocate several hours after each observation session to expand on your notes and jottings
ix. Add “notes to myself” in your field diary
- In addition to your observation notes, don’t forget to include your personal comments, judgements or feelings about what has been said or observed in your field diary
- It is likely you will remember your own personal thoughts and judgements even more than your observations and this will help lead towards analysis in your research
- However, make sure to demarcate these personal comments to distinguish them from your observational data
x. Tie observations to interviews
- Make sure to tie your observations to your interviews, as your interviews will fill in many gaps and act as a check on the undue influence of your personal reflections
4. Understanding Census Data
Steps in using census data
i. Find the data
- Most countries maintain statistical bureaus that carry out a census of their population and provide you with an overview of industrial activities in their jurisdiction
- In Canada, you can find census data through Statistics Canada (LINK REQUIRED)
- You will find a wide array of information ranging from topical census data to community related information such as housing costs, income, language usage and population
- To access detailed census data in Canada, go to one of the depository libraries (e.g. university, legislative or select public libraries)
- that receive all Stats Canada and/or Federal Government publications
- Census data from other countries can be found through the following links: US federal government (LINK REQUIRED), Britain’s Office of National Statistics (LINK REQUIRED) or National Archives (LINK REQUIRED), Nigeria’s National Bureau of Statistics (LINK REQUIRED), Brazil’s Insitituto Brasileiro de Geografia e Estatistica (LINK REQUIRED), or China’s National Bureau of Statistics of China (LINK REQUIRED)
- Depending on what country you are dealing with, a local consulate, embassy or trade office may be able to provide you with census data and reports in addition to academic and business journals and various institutional websites such as the United Nations, World Trade Organization, OECD, World Health Organization, International Monetary Fund and World Bank (LINKS REQUIRED)
ii. Carefully examine the data
- While the data or table stands on its own as a representation of information that has been amassed and organized you need to look at what is really being said so that you don’t read into the data more than you should
- The tables you find in census data and elsewhere summarize information and present it in such a way that comparisons can be made
- In the text surrounding a table, conclusions are usually drawn, and data are organized so that you will see how and why these conclusions are justified
- Read the titles carefully as far too often, tables are presumed to say one thing and actually say another, or the info in the table is applied to situations that are not depicted in the table
- Examine the notes accompanying the table which indicate the limiting conditions under which info is gathered, the sources of info, and any constraints that need to be factored while reading the table
- If the table is going to be used in the context of your own study, it is especially important to pay attention to the information sources and the original source of the table itself so that you are confident using the data moving forward
- This includes going to the original publication to see how the table was originally constructed and asking yourself the following questions: was the publication peer-reviewed? Was the info taken from a reputable source? Was any information censored? Were there any check and balances in preparing or publishing the material?
- Next, pay close attention to the categories used to organize the information in the table, especially the column titles
- Column titles represent indicators, which means only some aspect of the phenomena being described has been measured in a particular way to capture the picture as a whole
- For example, the title “children’s health” in a table might only be represented by the number of visits taken to a hospital
- It is in the choice of indicators that information is shaped, skewed and sometimes biased and that is why each indicator in a table should be examined to see what is actually being measured or referred to in organizing and making sense of the information
iii. Pay attention to why table was constructed
- You need to have some understanding about why the table was constructed and what the author was trying to show
- The title and text of the table will give you some indication about the purpose of the table, and ask yourself questions such as: What is being compared? What has been included and left out?
iv. Extracting info from the table
- To summarize, before you extrapolate data from your table, have you paid attention to the title, notes, definitions and categories being used?
- Without understanding all the information surrounding a table, your interpretation of the data will likely be subject to many misunderstandings!
5. Understanding Statistical Information
- Stats often portray comparisons using numbers that are usually in the form of percentages
- Stats are another way of representing information and are sometimes quite similar to information found in a table
- Sometimes, statistical information has a purpose beyond the numbers represented in a table
- In such cases, conclusions from the data are applied to a larger group or category
- For example, if you conducted a survey of 100 small and large universities and came to the conclusion that “small universities have a better teacher-to-student ratio than larger universities but less choice in subject areas,” you may have only examined these 100 schools, but the conclusions are being applied to the bigger categories of small and large-scale universities
- To ensure you use stats correctly, it may thus be a good idea to find someone knowledgeable to consult with who is familiar with the discipline and/or methods being used
- This will allow you to learn whether the methods and materials are considered trustworthy, or whether there is some controversy with the data itself
Once you have read this chapter, you should be able to:
- Understand what tertiary documents are, and the differences between primary, secondary and tertiary materials.
- Recognize the challenges that come from reading tertiary documents including scientific materials, field diaries, census data and statistical information.
- Find and locate science and science publication resources using databases.
- Perform the research steps required to engage in participant observation if your research project requires it.
- Locate, examine and extrapolate information from census data and other sources which use tables and statistics while making sure you don’t read more into the data than you should.