Hawaii State Department of Education
Please email your questions to sbac@notes.k12.hi.us
How schools were selected to participate in our upcoming Pilot Test.
The Smarter Balanced Assessment Consortiums has been receiving questions about how 6,000 schools selected to participate in our upcoming Pilot Test were chosen to be part of the scientific sample for Pilot Tests.
Understanding how schools were identified starts with knowing the purposes of the Pilot Test. The Pilot Test is our first chance to evaluate how the items we are developing are working—are they free from bias; are they stable measures of our intended academic content areas; do they assess growth across grades as intended; if we have more than one approach on how we present items/tasks, which one is better; etc. Conducting the analyses necessary to answer these important questions requires us to have the students who participate in the sample accurately reflect the characteristics of students from across the entire Consortium. For example, it would not be a good idea for the schools in the Pilot Test to all be suburban schools. Clearly, if we did that, we would not be able to generalize our findings to schools across the districts. So we created a "scientific sample" of schools to represent the entire Consortium. We also invited all schools in the Consortium to indicate if they wanted to participate in the Pilot Testing. Schools with interest in being in the Pilot Test who are not in the scientific sample are in what is called the "voluntary sample," and will be able to participate near the end of the Pilot Testing window.
Schools were selected to be part of the "scientific sample" using a pre-determined set of characteristics (e.g., percent free/reduced price lunch, percent white/non-white students) designed to match the distribution of those characteristics across the Consortium.
The number of items and tasks we are evaluating requires us to sample 10% of the students in the Consortium, and resulted in about 6,000 schools that were identified as being in the scientific sample. These identified schools have been notified either by phone or email. When we find a school that cannot participate, we turn to a second list of schools to find a "matching" school that has characteristics similar to the school that has refused. We give priority to inviting a matching school that has already indicated it wants to be in the "voluntary" sample of schools. Keep in mind, however, that matching the characteristics of the refusal school is the first criterion.
To date, many of our schools in the scientific sample remain unresponsive about their participation. We have asked our state Chiefs and K-12 state leads to ask their respective schools to confirm their participation. The success of our Smarter Balanced Pilot Test is based on strong state and district participation of our scientific sample. Without adequate school participation, Smarter Balanced will not be able to calibrate the number of items needed for the Field Test next spring. This will dramatically hinder our ability to produce a fully adaptive test.
To this end we have extended our recruiting window for the scientific sample until mid-February to allow our non-responsive schools to confirm their participation. If your school is among this group I encourage you to confirm participation. Schools wanting to participate in the voluntary Pilot should know this deadline has also been extended to mid-February 2013.
The section below goes into more detail about the sampling, if you are interested.
The sample was drawn from all eligible schools/districts selected to be representative of all Smarter Balanced states. A multiple-stage, stratified sampling with nested cluster sampling was used as the primary approach to ensure the representativeness of the selected sample. Stratified sampling entails assignment of schools to strata that have similar characteristics. Schools are then sampled from the stratification cells in proportion to their size.
The states that make up the Smarter Balanced Assessment Consortium were used to conduct the first-stage stratification to ensure that each state is adequately represented in the sample. Target sample size from each state was determined so that state representation was proportional to state school enrollment population. Within each state, additional strata were defined to increase sampling efficiency. For grades 3 to 9 sample selection, percent proficient on state assessment was used as a stratification variable. And for grades 9 to 11 sample selection, percent Title I was used as a stratification variable. As the next step, random samples of schools were selected as clusters within each stratum. Schools not selected to participate as part of the scientific sample were able to volunteer to participate through an online survey.
I hope this helps explain how the selections of schools came about. We are looking forward to a very successful Pilot Test, and to learning a lot more about our items, our item types, and about the kinds of technology supports and challenges that our analyses illuminate. Thanks in advance to all those schools who have agreed to participate.
© Hawaii State Department of Education, P.O. Box 2360, Honolulu, HI 96804; Physical address: 1390 Miller St, Honolulu, HI 96813; phone: 808-586-3230; fax: 808-586-3234. All rights reserved. For problems/questions concerning this web site, please email the webmaster. Links to other web sites should not be considered an endorsement. DOE is not responsible for the content of external web sites.