A Toolkit for Educators to Explore Qualitative Data in Schools - Create an Observation Tool

Given the availability of quantitative data in our education system, it’s likely that most of your school improvement goals are also quantitative in nature. Using math engagement as a demonstration of this process, examples of improvement goals may include the following:

• Increase the percentage of students meeting or exceeding math standards by X percent within two years.
• Increase the use of evidence-based instructional strategies in classrooms by X percent, based on observation data.

These goals seem straightforward until you begin considering how to achieve them. How should you go about increasing the use of evidence-based instructional strategies? What are some reasons why students might not be meeting or exceeding math standards? Qualitative data can help answer these questions, thereby providing a deeper understanding of the overall situation and helping you determine where to spend your energy and what steps to take to reach your goals.

Specifically, the two examples of school improvement goals above can benefit from an observation tool.

A classroom observation tool is a document used to gather information and document things like instructional practices, student engagement, and classroom climate. The tool includes a list of criteria, often in the form of a rubric or checklist with space for the observer to take notes about what they see happening in the classroom related to each criterion. In the context of school improvement, insights gathered through an observation tool are often referred to as fidelity measures, as they help assess whether instructional practices are implemented as intended. These tools provide helpful information about where teachers might need more coaching to reach quantitative goals or where students need more support for these goals to be met.

This section will walk you through how to create and use an observation tool to identify ways to work toward quantitative goals.

Create an observation tool.

Purpose

The first step is to determine the purpose of your classroom observation tool. In this example, we will focus on learning about classroom practices to help students meet math standards in a middle school setting.

Determine criteria

Next, you must determine the criteria included in your classroom observation tool. This step alone can include multiple sub-steps. You might start by considering where you can pull your criteria from rather than starting from scratch. For example:

• Are there existing metrics or existing language related to your purpose that most teachers will be familiar with?
• What tools exist on this topic outside of your setting?

When creating a classroom observation tool to learn about instructional practices to help students meet math standards, we could start by asking what math standards middle school students need to meet.

Most assessments focus on the Common Core Math Standards. These standards include such skills as doing operations with fractions, simplifying algebraic expressions, and calculating surface area or volume of 2D and 3D objects. None of these lend themselves well to a tool used to observe instructional practice. But the Standards for Mathematical Practice, thought of building blocks for a students’ foundation for doing math, could be a good place to start. The Standards for Mathematical Practice include:

1. Make sense of problems and persevere in solving them.
2. Reason abstractly and quantitatively.
3. Construct viable arguments and critique the reasoning of others.
4. Model with mathematics.
5. Use appropriate tools strategically.
6. Attend to precision.
7. Look for and make use of structure.
8. Look for and express regularity in repeated reasoning.

These standards will require some modification to become clear criteria that an observer can look for in a teacher’s instructional practices, but they are a good start.

Define criteria

The language you use to name and define your criteria should be clear and easy to understand. You may need multiple rounds of refining this language. To ensure that criteria are clear and easily understood, look for existing documentation of these criteria that might be familiar to teachers.

Let’s say your school has peer learning communities for math teachers. In the past, teachers have been asked to reflect on how their instructional practices support students in both their engagement and their math mastery. You can map these reflection questions onto the Standards for Mathematical Practice to help you define your criteria in a way teachers will already be familiar with and likely to understand. You can also borrow language from the Standards for Mathematical Practice themselves if this language is clear and familiar to your teachers. Remember to credit sources of information that inform your tool. Crediting sources allows teachers to learn about the source materials and invites them to dive deeper into any of the instructional practices they’re going to observe.

Applying these ideas, here are two example criteria and definitions based on the Standards for Mathematical Practices:

[table]

For these examples, we used items 1 and 7 from the list of Standards for Mathematical Practice and developed criteria by turning these student practices into actions teachers can take to help students develop the practice. Then we determined what a teacher can do that is observable to determine whether the criteria are present.

Decide on the structure or format of the observation tool

It is also important to decide what you want your observation tool to look like. Often, such tools are a rubric or a checklist. Before you decide or choose the easiest structure, ask yourself what information you want to collect. Is it important to have a number associated with how strongly each criterion is observed, or would it suffice to know whether that criterion was observed at all? Do you want to provide space for the observer to write notes or would you like them to reflect on the overall observation at the end?

In the example observation tool above, the intention is to develop a baseline and learn about what teachers most and least commonly do. For the observation tool below, we only care about whether a criterion was observed at all, and we encourage notes and thoughts from the observer along with one overall reflection question.

[table]

Use the observation tool.

Testing the observation tool

Before fully implementing the classroom observation tool, you should test it to make sure it is easy to use and captures information that matches its purpose. Test it yourself in a classroom and ask a few other people who will act as observers to test it in a similar setting. After testing, ask them for feedback via a survey, an email following their completed observation, or a follow-up conversation. Based on this feedback, refine the observation tool to make it as useful as possible. Then you’re ready to begin using it!

Train the observers

Now that your final observation tool is ready to implement, you should decide who your observers will be. A fellow math teacher, for instance, will complete the observation tool differently than an instructional coach or school administrator. Each observer brings their own expertise and experiences, so whom you select may impact data collection.

With your observers selected, you should host a brief training to review the observation tool and answer any questions they might have. This training is beneficial to prepare observers for implementation and is also a form of professional development! The opportunity to observe another classroom allows an educator to see different teaching strategies and classroom cultures, gain fresh perspectives, and reflect on their own practices.

Implement the observation tool

Now the observation tool is out and being used! Next, you need to determine the frequency and duration of implementation for your observation tool. It’s best to align your implementation plans with the scope of your school improvement goals. Let’s return to our example goals:

• Increase the percentage of students meeting or exceeding math standards by X percent within two years.
• Increase the use of evidence-based instructional strategies in classrooms by X percent, based on observation data.

An observation tool aligned with either of these goals would ideally be implemented in as many math classrooms as possible. To see an increase in the percentage of students meeting or exceeding math standards within two years, the observation tool could be implemented once a semester over those two years. To address the second goal of evidence-based instructional strategies, multiple observations in the same math classrooms over a year might be necessary.

Whatever your goals might be, consider the additional work required both for the observer and for yourself as the leader of this tool. You will need to balance the desire for robust data with time available.

Using the results of the observation tool

It’s important to know what you’ll do with the data collected from your classroom observation tool. In the examples we’ve used throughout this section, the data can be used to identify the strengths of math teachers in your school, determine what type of professional development can improve instructional practices that help students meet math standards, and provide instructional coaches with information to use as they coach and mentor teachers. However, before determining what to do with the data, you will first have to make meaning of the data. You can read more about this in our next section.