Evaluation Methods and Data Collection

This evaluation process is somewhat based off of Collaborative Inquiry by Dr. Jennifer Donohoo, published by the Greater Essex County District School Board (2011). This guide was chosen because of the evaluation's emphasis upon participatory evaluation.

The first step of the evaluation process will be to introduce it as a formal and collaborative inquiry into the TAC program. Having already discussed the nature and content of the evaluation with administrative staff, it will be introduced to other stakeholder groups as follows:

Step One: Gathering representatives from each stakeholder group

As this program intends to gather representatives from each stakeholder group to facilitate a participatory evaluation approach, the first obvious step is to identify willing participants.

Our school, though young, has a developing and enthusiastic faculty, staff and student body. The best way to find representatives is through a voluntary program of professional/personal development.

Administrative, Canadian and Chinese staff, as well as students can look forward to the professional/personal development opportunities offered by the evaluation process, as well as the potential for publication of the completed evaluation.

Administrative staff will be able to work collaboratively with staff and help to develop relationships and a collegial atmosphere of school development.

Canadian teachers can work to develop their professional skills by engaging in the program and bolster their resume with this experience.

Chinese staff can develop their resume and engage with the Canadian staff and students. Often, they have expressed feeling isolated from the projects undertaken by the Canadian staff members and have previously voiced a desire to join our professional development activities.

Students can take this opportunity to gain volunteer hours, contribute meaningfully to the school, develop their resume, and get experience with formalized data collection and evaluation organization. We do not anticipate a difficulty finding representatives from each stakeholder group.

Step Two: Peer workshops to gauge climate and perception

Administration:

Focus: Macro perspectives, school engagement with the program, academic/quantitative data resources

  • Perceptions of school buy-in with the program
  • What were the original aims of the program (establishing quantitative data sources to be evaluated)?
  • How the program has developed, met/challenged expectations

Canadian Teachers:

Focus: Enforcement and perceived value within the classroom, school community

  • Open-forum reflections on how they use the TAC program
    • In the classroom
    • Outside the classroom
    • As punishment/reward
    • As language development tool
  • Perceived efficacy of the TAC program
    • How can it be used to develop positive learning environments?
    • How are students perceiving the program? How do they engage with it?
    • In what ways is it helping to develop students?
  • How does the TAC program contribute towards the 'school community'?

Chinese Staff:

Focus: Engagement and secondary support of the program, developing an inclusive school community

  • What effect have you noticed the TAC program having upon the student body?
  • Do you actively engage in the TAC program?
  • What are the benefits of an English immersion environment?
  • How can we encourage staff to engage more in the program?
  • How can we help to develop school community through the TAC program?
    • Would it be valuable to assign a Chinese staff 'team leader' to each TAC group?

Students:

Focus: Perceptions of the program, perceived efficacy

  • What has contributed most to developing your English skills?
  • What connection do you have with your TAC group?
  • Do you like the rewards for winning?
  • Do you enjoy the songs that the losing group sings?
  • How has the TAC program contributed to your English skills?
  • Do you feel incentivized to speak English?

Step Three: Collecting evidence

The evaluation coordinator will record and organize the data gathered from peer groups.

Afterwards, a selection of two to three representatives from each stakeholder group will reconvene to look at the preliminary records and identify the type of data that can be collected to substantiate this information.

Because of the diverse participatory group, different roles and responsibilities can be assigned to different group members. For example, students will have different opportunities and limitations than Chinese and Canadian staff as well as administration.

When considering where/how/when to collect evidence, we will follow the steps outlined in the Ontario Guide (2011) and design the process collaboratively.

The participating representatives will then be able to get assistance from other volunteers within their stakeholder groups.

Possible quantitative data sources:

  • Student grades since program implementation
  • Amount of TAC cards collected daily
  • Success on assessment tasks and standardized evaluations
  • Participation marks in class

Possible qualitative data sources:

  • Interviews and surveys
  • Anecdotal perception of program efficacy and student development

Step Four: Perceptions vs. reality

Upon receipt of the data, evaluators and volunteers will work to compile the different data sources. A formal report will be presented to a limited audience of chosen representatives (the same as in step two) from each stakeholder group. These representatives will then disseminate the findings to the larger group.

Step Five: Interactional workshop: celebrating and sharing

All volunteers will reconvene in a large workshop.

Ideally, after the interactional practice of various stakeholder groups in steps two and three, participants will feel comfortable to engage with each other as peers.

The evaluation's completion and success will be celebrated as well as the development of school community through the process of the evaluation.

Participants will engage with different groups reflecting on the TAC program. Each group will consist of members from differing stakeholder groups and conduct an analysis of program development. Groups will be introduced to the sphere of concern vs. influence to analyze the enabling and constraining factors of the evaluation process (Saunders, 2011) (Donohoo, 2011).

Groups will present their ideas to the workshop.

Follow up steps:

  • Evaluators will work to develop a final report of the evaluation with intentions of publication. Contribution will be voluntary among stakeholder groups. All volunteers and participants will be recognized in the report
  • Reorganizing and developing the TAC program based on the findings, conclusions and decisions made during the evaluation