Ola Jachtorowicz

Formative evaluation for expanding Science News in High Schools (now Science News Learning) program into community colleges. Conducted as part of T523: Formative Evaluation for Educational Product Development with Professor Christine Reich at the Harvard Graduate School of Education.

Team members: Allison Williams, Ji Su Lee

Prepared for: Science News, Society for Science

Project Synopsis

Science News Learning [formerly known as Science News in High Schools (SNHS)] is an educational program by the Society for Science that provides educators with classroom resources and professional development opportunities based on the publication Science News.

SNHS was seeking to understand the landscape in community colleges as part of a five-year expansion plan.

Through conversation with Society for Science stakeholders and a literature review, we determined appropriate evaluation questions and designed an approach to evaluate the product in this new setting. We delivered out findings and recommendations to stakeholders through both a presentation and written report.

Evaluation Process

First, we determined appropriate evaluation questions that would allow us to deliver actionable insights to the SNHS team about expansion into the community college space:

1. What are the curriculum needs of community college STEM faculty and learners?
2. In what way, if any, is institutional/structural support necessary to obtain and implement new resources, including SNHS, in community colleges?
3. What is the fit and what are the gaps between SNHS and community colleges?

Next, we determined what process and evaluation instruments would allow us to answer these questions. We settled on a three-pronged approach:

A mixed-methods approach allowed us to obtain both standardized conclusions available from quantitative methods and the flexibility and sensitivity of qualitative methods. We first conducted informational interviews which were reviewed using a quick-method, inductive, thematic analysis with affinity-diagramming techniques. This analysis informed an online survey and structured interviews with a think-aloud protocol.

We inspected our survey data with exploratory data analysis (EDA), while using a deductive, thematic analysis with affinity-diagramming techniques for the structured interview, which was captured in a matrix of resource features and inductive themes.

Findings

Key findings uncovered during each phase of the evaluation were as follows:

To capture the richness of the qualitative data uncovered during the think-aloud protocol and structured interview, we coded all comments as either positive (+1), neutral (0), or negative (-1) in order to compile a sentiment analysis. I created a heat map that showed the sentiment of comments across themes (Y-axis) and product features (X-axis):

Overall, we determined that Science News programming and resources are a great fit for community colleges in terms of pedagogy and usability.

We found:

1. The potential to meet existing curricular needs for free instructional resources that align with existing learning objectives.
2. A lack of institutional/structural barriers, and
3. An overall good fit in terms of usability and pedagogy, with some specific recommendations for overcoming gaps, such as in content specificity, rigor, connections to lab skills, and use of multimedia.