For T127, Teaching and Learning Lab Practicum, my team developed and piloted a simulation using a platform that faculty has indicated interest in: Viewpoint.
Our understanding of simulations is that they are a constructivist approach, aiming for authentic learning. Schaffer and Resnick (1999, MIT Media Lab) categorized authentic learning experiences into four “kinds”:
- Activities aligned with outside world
- Assessment aligned with instruction
- Topics aligned with what learners want to know
- Methods of inquiry aligned with discipline
We wanted to evaluate how well our simulation participants’ affect, behavior, and understandings aligned with those expert practitioners in the Teaching and Learning Lab. My partner, Lauren conducted interviews with members of the TLL and designed evaluation instruments to collect simulation participant responses for comparison.
Authentic Emotions
The two top emotions for experts during a Kick-Off Meeting are nervousness and excitement, which were ranked highly by our simulation participants as well. However, the number one emotion experienced by participants was uncertainty, with 11 our of 14 respondents reporting experiencing this emotion during the simulation. Uncertainty was not reported by experts at all.
The data don’t tell us how much of this uncertainty stems from not feeling adequately prepared by our simulation design (pre-work and Viewpoint structure) versus uncertainty inherent in experiencing a kick-off meeting for the first time.
I wonder if with continued iteration on our prototype, we could reduce participant feelings of uncertainty?
Authentic Kick-Off Meeting Focus
Table 1. Participant Mentions of Trust and Relationships Compared to Expert Interviews
| Expert Interviews | Simulation Participants |
| [E-CL] My role “develops a relationship of trust with [new faculty members]… take the time to develop the relationship, have them see us as a solution partner, not there to tell them what to do. We’re here to enhance what they’re already doing, not fixing anything.” [E-LXD1] “I’m not here to change what they do… relationship building…I’m on your side and will make sure the students love it “ [E-LXD2] “Very focused on relationship building, getting to know FM, letting them get to know me, letting them understand our process and where they’re at, understanding what they need “ [E-LXD3] “Establishing relationships and building trust, asking questions, demonstrating I’ve put in the time to learn about the course ahead of time. … so reassuring [faculty members] that you have done this before” | [CL1 – POST] “prioritising relationship-building and creating trust between the faculty and the TLL team:…we need to respect their expertise with the course content while also providing useful learning design recommendations and support with conducting the class online” [CL2 – POST] “building relationships with SMEs (reassuring the faculty member and addressing her concerns)” [CL3 – POST] “building relationships of trust” [LXD1 – POST] “to have a solid relationship with the professor.” [LXD1 – POST] – “I made sure to introduce myself as an ally to the faculty both in my email and in person, shaking *his* hand and assuring him that we would be working very closely together on all ‘deadlines’ and meeting regularly and that this was a familiar process with pre-set steps that we are very comfortable using.” [O3 -MCB] “Observe a sense of psychological safety and trust being built among the four parties” [PM1 – POST] “Building connections between stakeholders” [PM3 – POST] “It is also important to gain the faculty member’s trust at this initial meeting. Each of us presented our expertise in helping the faculty member in the transition to the residential course, and we welcome them to contribute their ideas and insights. So I’d say we also have the trust by the end” [PM4 – POST] “established a positive working relationship” [FM1 – POST] “creating a trusty and comfortable working environment,” |
The theme of trust and relationship-building emerged prominently for 8 out of 14 survey respondents. I wonder if continued iteration on our prototype could make this theme even more prominent for all participants?
Perceptions of Authenticity and Immersion
Halfway through the simulation, we asked them to step out of their role for a “metacognitive break.”
In addition to reflecting on how they were contributing to the meeting and what challenges they still needed to address, we asked them directly how immersed they were feeling and if anything felt inauthentic.
7
Immersed / Authentic
vs.
5
Not Immersed / Inauthentic
For the metacognitive break data, we were able to split participant responses by whether they were under the virtual or in-person condition. Surprisingly, the ratio of comments coded as “immersed / authentic” vs “not immersed / inauthentic” across both conditions was identical: 7 to 5.
Table 2a. How immersed do you feel in this kickoff meeting? What, if any, feels authentic to a kickoff meeting? What, if any, feels inauthentic? [Virtual Participant Responses]
| Immersed/Authentic (7) | Not Immersed/Inauthentic (5) |
| “The kinds of questions we are asking each other about learning goals and objectives, as well as understanding the needs of a professor, feel authentic. “[CL1] “Real-life issues and even sudden changes (e.g. Faculty’s relative’s wedding)” [PM1] “It was very authentic. I could see this happen in the real life” [FM1] “Very immersed and authentic. I would imagine working in a similar setting in my future practice.” [CL2] “Very immersed and authentic” [PM2] “very immersed.“ [FM1] “I feel immersed because of participant investment in remaining true to character as well as prior work to understand their characters’ responses/behaviours” [O2] | “Some things that might feel a bit inauthentic are determining when it is appropriate to speak up and facilitate as the course lead.” [CL1] “Lack of prior communication with the team, more confusion about the roles in the simulation than the actual participation” [PM1] “I feel less engaged than I did in the in-person pilot version. I think we’re approaching it very professionally and cordially but that takes away some of the fun (like laughing at ourselves roleplaying). I also don’t see so much of the private beliefs emerging, but perhaps that is an accurate for a team of people who don’t know each other well and are just meeting for the first time.” [LXD1] “Given that we know it’s not an actual kickoff, I am struggling to step into the role.” [LXD2] “Using our real names felt slightly inauthentic, although I can see how taking on a different name could be uncomfortable” [O2] |
Table 2b. How immersed do you feel in this kickoff meeting? What, if any, feels authentic to a kickoff meeting? What, if any, feels inauthentic? [In-Person Participant Responses]
| Immersed / Authentic (7) | Not Immersed / Inauthentic (5) |
| “Pretty immersed, I would say! Everyone is playing their roles well and I feel this is quite similar to some of my prior experiences as a project manager.” [PM3] “I felt very immersed (love the private message feature!!!) until this sim break.” [LXD3] “I can tell that everyone is doing their work and responsible for their roles. Nothing is inauthentic at this point.” [O3] “Feels pretty authentic to me!” [CL4] “I feel well immersed in the meeting. I feel as if I am advocating for my role’s needs, and I am enjoying it very much.” [PM4] “It feels authentic because we are talking about the project phases, deadlines, and course details. The question from the professor also feels real as there is a real concern.” [LXD4] “Very immersed in this kickoff meeting in terms of each roles we are representing at a low-stake level;” [FM4] | “I am not super familiar with this context or the relationships between different roles beyond the preparation materials. It’s difficult to navigate it in an invested and authentic way when I feel like a fish out of water that’s mostly improvising and working on limited information. I wouldn’t carry myself this way in an authentic work setting because I would be more thoroughly prepared.” [CL3] “the setting could be more authentic if we spent a couple minutes more on icebreaker questions to build comfort and set norms.” [PM3] “The note taker’s typing is quite loud and I was really happy in a prior sim when it was just us two, no typing, no metacognitive breaks, and it was voice recorded with the exception of a few quick hand-written notes.” [LXD3] “Because I am not actually a project manager in the TLL in some cases I don’t know what my role *actually* is, and I’m not sure if documents can be shared with outside members etc.” [PM4] “maybe in reality it is more intense” [FM4] |
These responses highlighted elements that made the simulation less authentic/immersive – exactly what we look for in a formative evaluation as feedback for a subsequent iteration. In a subsequent iteration, addressing the following areas could improve participant perception of authenticity/immersion:
- More prep time for the TLL team to build team cohesion and collaborative determine who is the leading the meeting, goals, etc. prior to meeting the faculty member
- More scaffolding prior to the simulation for participants to engage more deeply with the pre-work, their role, and the simulation content
- No note-taking / observer role’
It’s interesting to note that there don’t seem to be major differences in participant perceptions based on the virtual or in-person simulation condition.
But What About Assessment?
In Week 6 of T127, we learned about authentic assessment. Authentic assessments examine student’s performance on stimulating, life-related tasks and real-world scenarios.
We also read about an approach tied to authentic assessment called cognitive apprenticeship as described by Allan Collins (2005). The author explains that he and his colleagues wanted to provide students with apprentice-like experiences by focusing on close attention to cognitive processes and immediate response.
Simulations can be used for both formative and/or summative assessments. In fact, Collins writes that “video and computer technology has enhanced the ability to create simulation environments where students are learning skills in context,” when citing research in support of cognitive apprenticeship.
As I reflected on our pilot of the Learning Design Kick-Off meeting, I wonder how to give participants more feedback. I wonder if it could be incorporated with cognitive apprenticeship in some way. There are three ways that come to mind by reviewing Collins’ recommendations:
- Have TFs act as coaches during the simulation, giving feedback immediately afterwards or even over private message in Viewpoint during the simulation
- Run the simulation over a longer period of time (perhaps a week with most components being asynchronous, but with each TLL team must meeting prior to the simulation in character). Participants submit their rolling agenda action items using Viewpoint’s “decision point” feature. Then, Viewpoint itself could provide feedback about proper sequencing (building rapport before jumping to timeline) to scaffold the process. Collins writes that “computers can offer scaffolding without stigma.”
- Provide a more structured reflection for participants for how they did in the kick-off meeting by allowing them to compare their “performance to a set of criteria for evaluating performance” or even comparing to a video of a real (successful) kick-off meeting
Leave a Reply