A Meta-Analysis of Outcomes Related to High-Order Processing (Executive Summary)
Background and Purpose
As children go through adolescence, striking changes occur in their problem-solving abilities. As a result, adolescents are generally more efficient and sophisticated learners than younger children, and they are able to more easily cope with advanced topics in difficult academic areas.
Adolescents with learning disabilities (LD), however, have difficulties in problem solving, also called high-order processing. The problems begin in elementary school, when a child with LD experiences difficulties with tasks such as word recognition, spelling, and computation. These difficulties become generalized as the child ages, and adolescents with LD typically have difficulties with skills of high-order processing.
The search for effective methods of improving problem-solving skills in adolescents with LD has become an important focus of research in recent years, especially as it has become clearer that deficits in problem-solving skills are superimposed on lower-order processing problems in areas such as spelling and computation.
The purpose of this report was to synthesize and summarize the research on interventions conducted between 1963 and 1997. Fifty-eight intervention studies were analyzed according to the age and intelligence of the adolescents, the characteristics of the intervention (e.g. number of instructional interventions, components of instruction) and the methods used by the original investigators. It was hoped that this synthesis of the research literature would help identify instructional models for adolescents that predict success in improving problem-solving skills.
It was clear from earlier research that not all interventions work equally well in this population of students, and two instructional methods seemed superior to others: direct instruction and strategy instruction. It was considered important to distinguish the points of commonality and distinction of these two approaches and identify the components of instructional models that predict the best outcome for adolescents with LD.
For purposes of this study, direct instruction was categorized as those that employed the following techniques:
- Breaking down a task into small steps
- Administering probes
- Supplying repeated feedback
- Providing students with diagrammatic or pictorial presentations
- Allowing independent practice and individually paced instruction
- Breaking instruction down into simpler phrases
- Instructing in a small group
- Teacher modeling of skills
- Providing set materials at a rapid pace
- Providing instruction for individual children
- Having the teacher ask skill-related questions
- Having the teacher provide new materials
Studies included in this synthesis that were categorized as strategy instruction had the following components:
- Elaborate explanations
- Teacher modeling of processes
- Reminders to use certain strategies
- Step-by-step prompts
- Teacher-student dialogue
- Teacher asks process-oriented questions
- Teacher provides only necessary assistance
Findings
Three main questions were posed as the core of this review of the literature on problem solving in adolescent learners:
Do studies that use direct and/or strategy instruction produce better effects on problem-solving skills than those that do not?
- In contrast to other methods, interventions were more effective when studies included derivatives of cognitive and/or direct instruction.
- Effective instruction can be either a bottom-up or a top-down approach, as long as certain components are included in the intervention.
- No statistical advantages of the direct instruction or strategy instruction were apparent.
- A clear orientation to task, drill-repetition-practice, sequencing, teacher modeling, and systematic probing may supersede effects related to distinctive qualities of either the strategy or direct instruction methods.
Is the magnitude of the beneficial effect related to certain components of the intervention?
- Regardless of the overall model of instruction, only a few specific instructional components increased treatment effectiveness.
- The following components predicted the magnitude of treatment outcomes: sequencing (eg, breaking down the task), step-by-step prompts; drill-repetition-practice; directed questioning and responses; individualization combined with small group instruction; segmentation (breaking down a targeted skill into smaller units and then synthesizing the parts into a whole); technology (e.g., computer use); and small interactive group instruction.
- The drill-repetition-practice-review component was an important variable in predicting effectiveness.
Do studies that vary in their definition of LD yield different results?
- Studies that allowed for cut-off scores (reporting standardized IQ scores at 85 and above and reading scores below the 25th percentile) yielded more effective treatments than other studies.
- Studies that produced the highest treatment outcomes enrolled students whose intelligence scores (84 to 91) and reading scores (84 to 91) were in the same low range.
- Less effective interventions were reported in studies in which discrepancies existed between intelligence scores and reading scores.
- Studies that included students with LD who have aptitude profiles similar to generally poor achievers or slow learners (low IQ and low reading) generally showed more results of the intervention than did studies that included learners with IQ/reading discrepancies.
Recommendations
Researchers
Six methodologic variables have a clear influence on treatment outcomes. These variables must be controlled for, however, before one can generalize about treatment outcomes. In short, studies of high internal validity yield smaller effect sizes, and perhaps more realistic outcomes, than studies with low internal validity. Investigators in this field should remember that methodology and sample description matter when interpreting treatment effects.
When interpreting treatment outcomes, an agreed upon taxonomy of dependent measures reflecting high-order skills should be in place. Unfortunately, several of the studies we included in this meta-analysis stated that they had focused on high-order skills but, in fact, the investigators did not include or utilize dependent measures that directly assessed high-order processing.
It makes no sense to teach high-order processing and assess students on low-order skills like fact recall. Inter-vention research needs to identify high-order skills that operate independently and influence and/or interact with low-order processing.
We need studies that empirically validate components of instruction that are the common core of effective procedures. Once those procedures are identified, it will be possible to systematically test advances in instructional models for adolescents with LD.
Practitioners
Consumers should be cautioned against studies with low internal validity, as described above, because they inflate treatment outcomes. In addition, it is important to include the following components when designing material and instruction for adolescents with L breaking down the task into smaller steps, using step-by-step prompts, drill-repetition-practice, individual and small group instruction, focusing on isolated processing components and then synthesizing components, and using computer-mediated instruction.
Although it may be difficult to distinguish low achievers from students with LD by using discrepancy scores, responsiveness to instruction may help make the distinction. In general, students with low reading and low IQ scores are more responsive to treatment than adolescents with clear discrepancies in reading and IQ scores.
Very few studies have targeted high-order processing in adolescents with LD. It cannot be assumed that mastery of low-order skills, such as word recognition, computation, spelling, and grammar, will correct high-order processing difficulties.
Funding Agencies
Direct funding is needed for further study of high-order processing and learning disabilities. Most funding to date has focused on low-order processing such as word recognition skills.
This document was prepared for the Keys to Successful Learning Summit held in May 1999 in Washington, D.C. Keys to Successful Learning is an ongoing collaboration sponsored by the National Center for Learning Disabilities in partnership with the Office of Special Education Programs (US Department of Education) and the National Institute of Child Health & Human Development (National Institutes of Health).
The purpose of this initiative is to translate research and policy on learning disabilities into high standards for learning and achievement in the classroom, and to take action at the local, state and federal levels to ensure that all students, including those with learning disabilities, are afforded the highest quality education.
Keys to Successful Learning is supported by a coalition of national and regional funders as well as a broad range of participating education organizations.
