IDEAS Framework Toolkit

Additional Resources

• • IDEAS Framework Toolkit
  • Valley Settlement
  • Raising a Reader

Transcript

Corey Zimmerman: Welcome to the Brain Architects, a podcast from the Center on the Developing Child at Harvard University. I’m Corey Zimmerman, the Center’s Chief Program Officer. Our Center believes that advances in the science of child development provide a powerful source of new ideas that can improve outcomes for children and their caregivers. By sharing the latest science from the field, we hope to help you make that science actionable, and apply it in your work in ways that can increase your impact.

With that goal in mind, the Center recently released the IDEAS Impact Framework Toolkit—a free online resource designed to help innovators in the field of early childhood build improved programs and products that are positioned to achieve greater impact in their communities. The Toolkit is self-guided, self-paced, and provides a structured and flexible approach that facilitates program development, evaluation, and fast-cycle iteration, including resources to help teams develop and investigate a clear and precise Theory of Change.

In April, we hosted a webinar about the toolkit, where we provided an overview of the site and had the opportunity to hear from teams at several organizations in the field about how they leveraged the toolkit and its resources to shape their work. We’re excited to share those discussions with you here on this episode of the Brain Architects podcast. If you’re interested in hearing a full walk through of the toolkit, by the Director of our Pediatric Innovation Initiative, Dr. Melanie Berry, please head over to our YouTube channel to view the full webinar recording. You’ll also hear from Dr. Melanie Berry during the Q&A portion.

The full IDEAS toolkit we’ll be talking about today can be found at ideas.developingchild.harvard.edu. And now, without further ado, here’s Dr. Aeshna Badruzzaman, the Center’s Senior Project Manager for Instructional Design and the moderator for our panel discussion.

Aeshna Badruzzaman: Hello, everyone. Welcome. My name is Dr. Aeshna Badruzzaman. I am a Senior Project Manager for Instructional Design at the Center on the Developing Child at Harvard University or HCDC, and I’m part of the development team of the IDEAS Impact Framework Toolkit. And today, I’ll be your host. So, you may hear me come off mute, and help guide presenters, and I’ll be facilitating our question and answer period. So, we are so pleased to be talking to you today about this resource. The IDEAS Impact Framework was born out of more than a decade of the Frontiers of Innovation Initiative or FOI. And some of you may have been partners in that effort. So, while our team no longer offers live training on the framework, we are so excited to be introducing it to you as a free open access resource. And we really hope that this format is going to help make IDEAS accessible to innovators in the field of early childhood development moving forward. The framework was developed in partnership with the University of Washington College of Education, and the University of Oregon Center for Translational Neuroscience. With support from the Gates Foundation, The Lego Foundation, Porticus and the Hemera Foundation. I encourage you to check out our history and acknowledgments page of the toolkit for more information about our various collaborations and supporters throughout time as well.

Now I’ll go ahead and introduce our first set of speakers from folks at Valley Settlement. We have with us Karla Reyes, who is the program manager of the El Busesito mobile preschool program at Valley Settlement, which is a nonprofit that works to create opportunities for the Latino community in the Aspen to Parachute region of Colorado. Karla joined Valley Settlement in March 2015, as a preschool teacher for El Busesito until June 2021 when she took on a leadership role. And we also have Sally Boughton who is the Director of Development and Communications at Valley Settlement, a nonprofit again, serving the rural Aspen to Parachute region Colorado with six to generation programs designed by and for local Latina immigrant families. And Sally has been with Valley Settlement for over five years and began managing the organization’s evaluation function in 2021. Thank you so much Karla and Sally look forward to hearing from you.

Karla Reyes: Good afternoon, everyone. Thank you for inviting us to share our work with you all and how we have used the framework. I’m going to talk a little bit about it, we’ll see the program and how I kind of started. The idea that it will succeed though began in 2011. We had two bilingual and bicultural community organizers, who met one on one with about 300 families from the Aspen to Parachute region of Colorado. And they learned about their lives and the barriers that they faced within our community. One of the findings from the initial listening tour was that only 1% of Latino children in our community were enrolled in preschool. We also learned that three of the biggest barriers for families to participate in preschool programs were language, cost and transportation resulting in lack of access. Now we have all this information. And we started thinking creatively of different ways that we could bring more access to preschool education to our community. I have also seen those one of the first two generation programs that we launched in Valley Settlement to address the needs of preschool education. And throughout the years Valley Settlement has continued to learn, evolve and co-design programming to respond to community needs. Now, our program has four mobile preschool buses that have been retrofitted into small preschool classrooms. We have two teachers on the bus, and we serve eight children at a time, we provide families with about five to 10 hours of free preschool education. We have about 96 children that we serve annually between 40 to 50 children graduating at the end of the school year and moving on to kindergarten. And currently right now we serve different five different neighborhoods within our community. And we strive to build close relationships with families. So, our program really is designed to meet families at where they are, are at and start breaking down those barriers. We host family nights; we have home visits with our families. We have parent teacher conferences; we have different ways that families can volunteer within our program. We provide a lot of materials for families to use that home so that they can do home activities and homework packets with their students. And we really try to engage with the families. So, each one of our teachers speak Spanish, is bilingual and bicultural. So, this really allows that bond and that relationship to build with each one of our families. I’m going to hand it off to Sally Boughton, and she’s going to talk a little bit more about how we’ve used the framework.

Sally Boughton: Thanks, Karla. So, several years ago, we started working with the team at Frontiers of Innovation to refine and evolve our evaluation practices. This work included creating theories of change for each of our programs, researching and recommending observational assessments to measure participant progress towards our program targets and outcomes, and creating implementation guides for our programs to detail the critical components of our work and ensure that future staff can implement programs with fidelity, while still continuing to listen to and evolve alongside families. Since the early days of our programming, Valley Settlement has invested in evaluation to measure understand and strengthen the changes that children and families create in their lives through our programming. Working with the team at FOI really brought this to the next level. Over the last few years, we’ve been working to be more inclusive and participatory in our evaluation process. So now our entire staff gathers for three days every summer, in what I call an evaluation retreat, where we review our annual program data as a team and then try to answer those questions ¿qué? ¿por qué? ¿Y ahora qué? or as the toolkit outlines: What does the data say? Why? Why might the data say this? Or what does that mean? And finally, now what do we do to tweak or change in our evaluation approach or in our programming, based on what we see in the data? Our teams then create action plans to outline those changes that they want to make. We’re usually tweaking one or more program components for the upcoming year. On day two of the retreat, teams then go in and refine and evolve their theories of change. So, we really see that theory of change as a living document that breathes and grows alongside our programming. They identify what targets and outcomes they’re interested in measuring for the coming program year. And then after that evaluation retreat, we work together with our evaluation consultant to refine our measurement tools. And then I always try to call out and highlight that I am in the minority of Valley Settlements. Our staff are largely of the community that we work with. Most of our staff are immigrants or children of immigrants, many have grown up in this community, or immigrated to the community as adults with young children. And so, they have those shared lived experiences with the participants in our programs. And many of our staff have actually been former participants in our programs. Having the entire team participate in this process is incredibly valuable. It really places the experts in the work our staff in the evaluator seat, and we gained so much more by having that inclusive, participatory process. And we’re really so grateful to have our work shared in the online toolkit because, you know, I am not an expert in the IDEAS framework by any means. And that’s kind of the whole point is that it’s very usable, you can go in, you can click through this toolkit, you can see how it all is structured and works. And it just makes for a really kind of manageable, useful process that you can engage in. Thanks so much.

Aeshna: Thank you, Karla, and Sally, really appreciate you taking the time to share your experiences with us. Now we’ll hear from folks from Raising a Reader. So Raising A Reader supports families to build, practice and grow reading routines at home. Their award-winning evidence-based program helps caring adults set their children up for success by creating shared reading routines, fostering social emotional learning, healthy family relationships and learning skills needed to thrive in school and beyond. And first, we’ll hear from Michelle Sioson Hyman, who is Senior Vice President, program and partnerships. And in her role, Michelle is responsible for overseeing program development, growth and impact. And then we’ll hear from Andres Garcia Lopez, who is a Senior Project Manager at the Center on the Developing Child. And in his role, he’s coached many early childhood development entrepreneurs, including Raising a Reader in developing strategies to maintain their science-based impact, while scaling their ventures. Welcome Michelle and Andreas.

Michelle Sioson Hyman: Thank you so much for having me. I’ll start with a brief overview of Raising a Reader and how we’ve used the framework and then fundraising, engaging some conversation. So Raising a Reader is a national family engagement and early literacy organization through our network of affiliates and partners across 34 states and both rural and urban communities. We engage and support parents and the other caring adults in children’s lives help strengthen the bonds with their children, while building critical early reading and social emotional skills. So along with our award winning multicultural and multilingual book collection, we provide easy to use materials and guides that are really designed to make the most of that shared reading time in the home. So our work really does begin though, with partnering with local agencies who become members of our affiliate network, a community of practice in which we can share best practices and build connections. And we provide professional development, technical assistance and capacity building support to this network of affiliates and partners, who really work across the intersection of systems, supporting children and families at the various points throughout their educational and developmental journey. So that’s, you know, in ECD, K, 12, Health and Human Services. And we’re really able to meet families in the spaces and places where they are involved in how the framework has really impacted our work was that we were introduced to Andres. And the framework is a really critical inflection point in our history. So we’re over 23 years old and Raising a Reader had 39 independent evaluation that prove the success of our Classic Red Bookbag Program and its impact on improving and sustaining home literacy environments. But one thing that we realized through our work with Andres was that there were critical aspects to our work that we weren’t capturing in our theory of change. And just maybe I’ll stop there, and then we can chat. Does that work?

Andres Garcia Lopez: Sounds good. That works. Michelle. Thanks so much for that overview. And I’m so excited to be part of this panel, and it’s an honor to share it with you, Michelle, and with the Valley Settlement team. So I’ll just add a few things. I was working with Michelle as part of a fellowship that the Center partners with the Promise Venture Studio. And as was mentioned before, the theory of change on the IDEAS framework really helps you think about what are the key ingredients that my organization in my program works on or provides to families and

or maybe two partners that get to the targets that that move the needle towards my outcomes. One thing that was different about Raising a Reader was that they weren’t with partners. So I wanted to mention that sometimes the IDEAS framework can be, and the theory of change could be flexible, and adaptable to meet your needs. Originally, there are three columns in the theory of change. But we’re working with Michelle, we thought we should have an extra column because they wanted to look at how working with partners and affiliate organizations, what their strategy is that was doing racing, a reader was getting into the targets in partners and affiliates and how that was getting to the outcomes with families. And that was a key component on identifying precisely the actions that get to the targets and the outcomes. And I’ll pause there so that Michelle can share more about the specifics of what some of those strategies were and how that helped the organization.

Michelle: Thanks, Andres. Yeah, so one of the things including that additional column that Andres was talking about, it really helped us think through, how are we really building that educator capacity? And how are we really providing professional development around early childhood development. And another aspect to it is that we knew we did it all the time. And we had stories about how we did it all the time. But using the theory of change, the framework to really make it much more precise, is really helping us think through how we are doing it. And so, it also helped us think about how our program is impacting early relational health through strengthening healthy family bonds. And so, it really has made us to be better poised to effectively test and evaluate how we are doing this work and what isn’t what is not working.

Andres: Thank you, Michelle. And one comment that I add, as we have about 90% of the participants that are now in the webinar are new to the to the framework, sometimes you may use the framework as a program developer, or somebody who’s implementing a program like your shell, but you could also use it to help other organizations like the way I have used it as part of the center in the developing child, or as promised venture studio has also used it with social entrepreneurs in their organization, the framework is really helpful in helping you think through your impact strategy. I mentioned a brief comment, if you’re familiar with other frameworks that innovators use, like the lean startup or business model canvas or other ones, it these helps you think through in a very clear way, in a simplified way, what are your strategies and how I am I get into the outcomes. And because of its it’s simple, and it can fit in one page, it also facilitates communication, communicating internally and externally, with the families you work with, with their funders, potential funders and with potential partners. But I’ll pause there.

Michelle: One more thing I just wanted to add about the framework is how it really helped us think about our innovations and new programming to into our theory of change and help facilitate that fast cycle iteration. Because it’s over the last few years, we’ve developed to new programs and explored how we were success and exploring how we can integrate technology into our programming. And we didn’t have that we didn’t have that in our previous theory of change. And so being able to build that into a using the framework to build that into our theory of change, thinking about the evaluation, how do we get that feedback loop? It was really beneficial and helpful for us as we’re continuing to innovate and develop new programs to

Andres: Thank you, Michelle.

Aeshna: Thank you so much for sharing your experiences. And we’re going to go ahead and pull some questions from the chat. And we will start with kind of a somewhat broad question. And that came from Aaron Soto. Is that are there any prerequisites for an organization to implement the ideas framework? And it was Melanie, you might want to speak to this?

Melanie Berry: Sure. I would say there aren’t necessarily any standard prerequisites. But I do think having worked with a lot of different organizations around this framework, there are some conditions that set you up to be more or less successful or effective using the framework. One thing I would say is it’s important to have all the right people at the table. So, I mentioned that one of the principles of the framework is co creation and this idea that, you know, bringing together a group of people who have multiple perspectives on the program can be really valuable and that might include leadership, people who are responsible for developing or implementing the program, people who will lead on the evaluation or research efforts, but equally, importantly, you might invite people to contribute who have a role in actually delivering the service working directly with kids, families and caregivers. And better yet invite a representative from the community that you serve to be part of this process. So that’s the first piece is just having the right people at the table. And then the second thing I would say is timing can be important. So this framework is really designed to help you prepare for a fast cycle iteration process. So to prepare for a round of collecting data, reviewing that data, interpreting and analyzing it and making sense of it, and putting what you learn into practice. So the timing there can be important, you know, are you set up and prepared to actually put this plan into action? Do you have the resources you need? Is everyone bought in? Etc? Yeah, and having the authority to put what you learn into practice. So if you’re implementing a program that was developed by someone else, do you have sort of the leeway to make changes to how you’re implementing that program based on what you learned? Or are there kind of more strict parameters around how you implement that program?

Aeshna: Thanks, Melanie. The next question we have came from Nicolas, and it says a question for Miss Reyes from El Busesito. Were there any outcomes or benefits that happened unexpectedly from developing this program? i.e. unintended consequences that happened, which you did not expect, yet?

Karla: Yes. So there, it’s that’s definitely been a learning curve, we’ve definitely had to modify and just evolve the program. One of the biggest changes that we’ve made just recently is changing the program from a five-hour week model, where children receive two and a half hours of preschool twice a week, to offering five hours of preschool twice a week. So in total, they’re receiving 10 hours a week. And this really came from listening and taking the time to listen to parents and hear what their needs were, for years, or parents had been asking for more time on the bus, we’re really trying to make an impact on how many children we served. And like I said, we have, we have the capacity to serve 96 children in our valley. So that’s 96 children that otherwise wouldn’t be receiving preschool, you know, in a traditional preschool setting. And we’ve noticed recently, we’ve had a decrease in our enrollment. So it’s been a little bit harder to enroll children into our shorter classes. And I think that has now impacted our school district and our other centers that have grown their capacity in their centers, which was the ultimate goal to get more children into preschool and enrolled. So we’ve now looked at how we can because we’re mobile, we can now take our program and start serving communities that don’t have that access. So it’s, it’s been playing out lately, that we’ve noticed these trends.

Aeshna: Thank you. And actually, I just realized that this question is, the way it got segmented in the question answer section, I didn’t realize that the Nicholas who asked the question has the same question for folks at Raising a Reader. So were there any outcomes or benefits that happened unexpectedly from developing the program?

Michelle: Sure. So I’ll say that we have had 39 independent evaluation that showed, then we knew that Raising Reader helped improve home literacy environments, and which is like increased shared reading time, increased duration, and frequency, improve the number of books in the home. But one thing that we were hearing from folks was, oh, well, it’s helping me build confidence and supporting my child’s early learning in the home. It’s really providing a sense of comfort and support for our families, this daily really reading routine. And so creating a new theory of change, with coaching support from Andres, to really make a much more precise, we were able to build those kinds of things into our theory of change, which then led us to improving our measurement tools, so asking specific questions so that we could actually get some more data around, While their stories are great,it’s also helpful to have our surveys also reflect some of that more quantitatively as well.

Aeshna: Thank you, Michelle. And we have one question here. This asks if someone could speak to how this theory of change framework can inform logic model use and development, ensuring that the information is complimentary and not duplicative for programs who choose to create both types of resources.

Melanie: I’m happy to field that; I get asked that question a lot. From what I understand. Those two terms theory of change and logic model are often used actually interchangeably in the field. And there really isn’t solid consensus on how the two are similar and different. The best guidance on that that I’ve found is that logic models tend to be more standardized, they often include inputs, activities, outputs, and then short term and long-term outcomes are variations on that theme. And they’re really, their descriptive theories of change, are a bit less standardized. So if you Google the term theory of change, you’ll find many, many, many different approaches to theory of change. But in general, they’re intended to be causal models that really explain how and why the expected changes come about. That’s one way of thinking about that, then logic models are more descriptive and theories of change are really intended to be explanatory causal models. Honestly, in practice, I find that they’re, you know, when someone asks you for a logic model or a theory of change, you really have to follow up to ask what they mean, specifically what they’re looking for there. Because I think expectations vary widely. Our approach to theory of change is really, it zooms in on the point of service delivery, or the point of contact with kids, families, caregivers, and follows that through to the ultimate outcomes that you’re hoping to see, which are typically child level in the field of early childhood. And it can be really helpful to set you up to make a plan for gathering data to better understand whether you’re having the impact that you’re hoping to achieve. I hope that’s helpful. Sally, do you want to add to that?

Sally: Yeah, so I actually have a real-life example of how we’ve used both at the same time. So we recently worked with the team at Mathematica to create a Two Gen logic model, they did a project with different to join organizations across the country. And so we have been using theories of change for years in our programs and our six different programs to really identify, you know, what is the what are our strategies? What are the targets? What are the outcomes we’re trying to have in each of our programs, so it’s kind of granular, and then we worked with the team at Mathematica to create this, like, what is our whole organization do? And what is the whole change we’re trying to make in the community in in children and families. And so having that overarching logic model, that our theories of change then kind of feed up into, you can see how they, how they interact, how they’re incorporated in that larger logic model has been just really interesting, but we’re not duplicating. So we don’t have like a logic model and a theory of change for each of our programs.

Aeshna: Does anybody else want to speak to that before I move to the next question? Okay. So we have a question here from Eric Marlowe and asks, In your experience, I think this could be open to anyone here. How long are the typical iteration cycles? How long do you recommend evaluating and adapting elements of a given program so that changes are made neither too soon? nor too late?

Melanie: I can take a first pass that then I’d be curious to hear from our colleagues at Raising a Reader and Valley Settlement, if I understood correctly. So that was the question. Like, how often or how long does it take? Or maybe a little bit of both?

Aeshna: My interpretation was a little bit of both.

Melanie: Okay. I think so the way this question gets asked to us often is how fast is fast cycle? Like, are we talking something you can do in days, weeks, months, years? And the answer there, I think is unfortunately, it depends. It really depends on the nature of the program, or service or product that you’re looking to evaluate and improve. If it’s, let’s say, a 10 week parenting group, then a single cycle could take, you know, you’d want a couple of months to plan and prepare, to identify to develop your theory of change to identify your questions to figure out your study design, figure out the tools you want to use prepare for data collection, then obviously, you need the 10 weeks to go by where you’re actually delivering the program to kids and families. And then you’ll need some time afterwards to analyze interpret and make sense of that data.

But that timeframe is obviously going to be really different if it’s a program that’s implemented on a school year calendar, for instance, or if you develop something like an app or a website that families can engage with, as they choose, and maybe dosage and engagement looks really different from parents or parents. So there’s no right answer for how long a cycle can take, you really just need to be thoughtful about what you’re hoping to learn. And then in terms of how often, I think that really varies, again, from organization to organization, and what the appetite and bandwidth is to engage in this kind of iterative learning. I know that valley settlement, for instance, has really built this into your kind of culture and your routines as an organization. And it seems like you’ve developed a really nice kind of annual rhythm. So maybe I could pass the baton to Sally and, Karla, to talk more about that.

Sally: Yeah, so as you say, Melanie, we really do our evaluation on an annual basis. So we most of our programs happen during the school year. We do pre and post surveys, we also do pre mid and post TS GOLD assessments for Busesito preschool. And then we really we do the bulk of data analysis in June and July. And then every July, we gather and do that evaluation retreat with our entire team. And then in August, we’re kind of refining our evaluation and planning the next evaluation cycle. But what I will say is that when we’re piloting new initiatives, we are trying to be a little more like eyes on as the as the initiative is happening. So for example, we worked a few years ago to implement a Child Development Associate course, for family, friends and neighbors, providers, and also for high school students who are Spanish speaking. And we were we were doing little pre and post assessments throughout at the beginning and end of kind of each section of the quarter module. Just to understand like, was our approach working? How could we pivot and adjust. So as we’re piloting new things, we do try to be a little more rapid, if you will, I would just echo the same thing that we do have an annual, like an annual cycle, I guess, where we do an annual evaluation, annual check in with our affiliates is what we call it. And then we do have a couple of pilots right now where we are calling them mini learning phases. After each learning phase, then we’ll take a look at the feedback and then see what tweaks or what modifications we need to do to improve the program. And then we have our second learning phase, and things like that.

Aeshna: A quick follow up from Nicolas to this question was so does that mean that the theory of change is different with each iteration?

Sally: So we definitely like we can evolve our theory of change every year, we go in as a team and look at the strategies and say like, you know, are you are you still doing these things? How are you doing them? You know, we definitely change and evolve in our programming, we’re not doing the same thing every year, because programs, neighborhoods, communities change and evolve. And we learn new things every year. So we do go in and tweak our theory of change every year.

Michelle: And I’ll say we just revised ours so we haven’t changed it. But one thing I think it is helping us think about though, as we are developing our program, how to stay focused on what it is that we really want to do. Because there are so many needs, and we serve so many different communities that have different needs. How do we stay true to this theory of change that for our mission, and things like that, so it really helps us identify are clearly like, where is our most unique impact and helps us stay there and not kind of stray just because there might be a funding source over there or something like that.

Melanie: I’ll just add that. It’s a great question. And it gives us the chance to underscore that. Absolutely. A Theory of Change can and should evolve over time, as you learn. So we call them living documents, right? It’s not a one-time exercise that you do, and you make a PDF of it. And it’s done. It’s a living document that you come back to after every round of learning. And you say, you know, what did we learn? How can we refine our theory of change? How can we refine our actual program or product or service? And how could we refine how we’re gathering data and learning going forward?

Aeshna: Thank you. So now we have a question from Megan Crystal, asking if we have any examples if we know of any state level policy or programs that have used this framework.

Melanie: I have a couple that come to mind. So a while back as part of the Frontiers of Innovation Initiative that Aeshna mentioned, there was a project team who implemented a video coaching program to support childcare and early learning professionals. And that project was done in partnership with the State Department of Early Learning. And this framework was used to sort of articulate theory of change for that approach, and to actually work with partners at the University of Washington to craft the evaluation plan. So that’s one example. And then another more recent example that comes to mind is our Center worked with partners at the Massachusetts Department of Early Education and Care to create an initiative wide theory of change, actually, similarly for an effort underway to bring early care and support organizations to build the capacity of actually childcare directors across the state. So we worked with them to create an initiative wide theory of change. And then each organization who was providing that service, use that initiative, wide theory of change as a template and kind of tailored it for their particular approach. And I think that’s still being used right now, as the initiative continues to kind of facilitate learning and improvement over time. There might be other examples, but those are two that come to mind.

Aeshna: Thanks, Melanie. Does anybody else-

Melanie: Can I squeeze in one more just in case folks are looking for examples. We worked with an organization called TOPS, which is based in the Netherlands. And they provide if I’m remembering correctly, services to families with newborn children. And I think that’s actually like a nationwide program that’s used ideas to drive towards greater impact. And there are resources or references about the tops program in the resources section of the toolkit. I think there’s a research article there that talks about their experience.

Aeshna: Alright. Great. Thanks, everyone. I was hoping we might have time for one more question, but actually see that we’re pretty close to the end there. So I just wanted to thank you all for joining us. thank our panelists for sharing your experiences and your learning, we really appreciate it. And thank you all so much for joining, we really, really hope that this resource is useful for you all, and we wish you the best in your continued work, supporting kids and families.

Corey: The Brain Architects is a product of the Center on the Developing Child at Harvard University. You can find us at developingchild.harvard.edu, where we’ll post any resources that were discussed in this episode. We’re also on Twitter @HarvardCenter, Facebook at Center Developing Child, and Instagram @DevelopingChildHarvard. Our music is Brain Power by Mela from freemusicarchive.org.