Let's talk for a moment about an unpopular topic: treatment fidelity. For those not familiar, treatment fidelity is the term used for the correct implementation of instruction and assessment that is specifically designed to help struggling students.
For a number of reasons, some teachers have come to think of fidelity as the “F” word in education. Why? Because the documentation of steps used when providing instruction or assessment as part of a multi-tiered system of supports (MTSS) is often very challenging.
In this post, I'll unpack the origins of the term fidelity, suggest an alternative, and describe the steps needed to document tiered supports.
Fidelity or Integrity?
There are multiple terms used for the accurate implementation of instructional materials and assessments. Two of the most common are fidelity and integrity. Each of these has a slightly different meaning:
The word fidelity comes from the Latin root fides which means faith.
The word integrity comes from the Latin root integer which means complete or whole.
Although both of these words have been used to describe the procedures required to implement instruction or assessment correctly, fidelity’s origin in faith connotes a desire or intention but might not be measurable. In contrast, integrity suggests completeness. And if there is a complete description of something, it can more easily be measured.
It is my professional belief that integrity is the better word because it focuses on completing all the instructional or assessment steps.
Documenting Treatment Integrity
Instructional Integrity
In order to know if the instruction has been implemented as intended, some record of what instructional steps were intended is needed. The starting point for documenting instructional integrity is a checklist of the steps in the lesson. Many publishers include these with instructional materials. Below is an example of the integrity checklist for the FastBridge reading intervention.
If an integrity checklist is not available for a published intervention, it might be available from an online resource such as the RTI Network. It is also possible to create an integrity checklist for new, teacher-made lessons.
To create a checklist:
- Write out a list of each step in the lesson.
- Next, check to see whether each step is something that can be easily observed.
- As needed, revise the wording in each step to reflect behaviors that can be observed.
Assessment Integrity
Checklists can also be helpful in documenting assessment integrity. Like interventions, many publishers include assessment procedure checklists. The FastBridge system includes a checklist for every FAST™ assessment. These checklists are known as the Observing and Rating Administrator Accuracy (ORAA) forms. (The ORAA forms are found in the Resources section of the online courses for each assessment.)
Here is the ORAA for CBMreading:
Two Methods for Documenting Treatment Integrity
While checklists provide the tool for collecting integrity data, there are two different ways these tools can be used. One approach is for a teacher or examiner to fill out the checklist after completing the lesson or assessment. The other is to have an observer watch a lesson or assessment, and check each step that the teacher does according to the checklist.
Self-Report
This self-report method is the easiest to implement because it does not require additional staff. And, this method will likely not intimidate or overwhelm the teacher. That being said, self-reported integrity could be inaccurate because a teacher might not recall the completed steps accurately.
Observations
Naturally, the observation method has the benefit of providing more objective information. However, they require additional staff and teachers might be anxious when observed.
Self-reports and observations can be used together as part of a comprehensive integrity data plan. A good way to initiate integrity data collection is with self-reports. Using these as the starting point helps teachers become comfortable with collecting integrity data. After a teacher has completed a number of self-reports, an observation can be conducted using the same checklist.
The observation format provides two types of integrity data. First, it documents the teacher’s accuracy with a specific lesson or assessment. Second, the observation can also be compared with the self-reports to give an idea of how accurately the teacher reported prior sessions. The combination of self-reports and observations provides additional integrity data without requiring intensive staff support.
Target Level of Treatment Integrity
For both self-reports and observations, the target level of integrity is 80% or higher. The percentage can be calculated by totaling the number of required steps in the lesson or assessment and the total number that were completed accurately. Then, the sum of the accurate steps is divided by the sum of the total steps to determine the percentage for the session.
Here is an example based on the 12 steps in the CBMreading™ assessment shown in the above ORAA:
Total number of possible steps = 12
Total steps recorded as correct = 10
10/12 = 83% accuracy
While 80% accuracy is the minimum typically expected or required in research and other applications, higher percentages are always better. That said, it is not expected that integrity will always be 100%. Some errors or interruptions are normal and therefore teachers should not be alarmed if integrity is not perfect every time.
When the percentage is less than 80%, the lesson or assessment should be redone with higher accuracy. When the integrity is more than 80%, the session does not need to be re-done. In those cases, teachers can consider which, if any, steps were omitted and work to improve those in the future.
Integrity and Data: A Case Study
The usefulness of any data is dependent on the integrity of the instruction and progress monitoring scores. The following three scenarios for the same data demonstrate how instructional and assessment integrity play a crucial role in score interpretation.
Below is a sample CBMreading progress monitoring graph for a student participating in additional reading instruction.
As revealed in the following three scenarios, how to interpret the data will depend on integrity information.
Scenario 1. In this scenario, the instructional integrity data indicated an accuracy of 65% and revealed that the instruction was not being implemented according to the published directions. In this case, the teacher should be provided with preparation to conduct the lessons according to the published steps, and additional progress data from sessions when the instruction is done according to the directions should be collected.
Note that frequent student absences would constitute a lack of instructional integrity because instruction cannot be implemented as planned if the student is rarely at school. If the instructional integrity is 80% or higher when the student is present, but the student’s attendance is below 80%, then the next steps would be to work to improve the student’s attendance.
Scenario 2 In this case, the instructional integrity data showed 93% accuracy, but the progress assessment integrity indicated 45% accuracy. This suggests that the assessment was not administered correctly. The person conducting progress monitoring needs to learn how to conduct it accurately and then more data collected. Usually, there must be at least 6 data points before progress data can be interpreted.
Scenario 3 If both the intervention and progress assessment integrity data are 80% or higher, then the data can be interpreted. In this case, data from a graph such as the one shown above can be interpreted. This graph suggests that the instruction is not working and different instruction is needed.
Summary
Rather than using the potentially loaded term "treatment fidelity" I would encourage schools to try the alternative term "treatment integrity," which has a root meeting of whole and complete. When viewed from this different perspective, the process of correctly implementing instruction and assessments may begin to feel less overwhelming and more achievable.
For more information about the role of instructional and assessment integrity in an MTSS, check out the RTI Network or the book on this topic by Sanetti, and Collier-Meek (2019).
