EMU | COE NCATE 2003 Standard 2.B Data Collection, Analysis, and Evaluation

Standard 2-B
Data Collection, Analysis, and Evaluation

Target: The unit is implementing its assessment system and providing regular and comprehensive data on program quality, unit operations, and candidate performance at each stage of a program, including the first years of practice. Data from candidates, graduates, faculty, and other members of the professional community are based on multiple assessments from both internal and external sources. The unit maintains a record of formal candidate complaints and documentation of their resolution. Data are regularly and systematically collected, compiled, summarized, analyzed, and reported publicly for the purpose of improving candidate performance, program quality, and unit operations. The unit is developing and testing different information technologies to improve its assessment system.

Reporting of assessment data at each stage. As shown in Exhibit 2.1.A, the EMU Professional Educator Assessment System has six major stages. (1) Admission to the university is handled through the university's admissions office, both for undergraduate and graduate students. Admission requirements are stated in the catalog. Click here then to page 8, for undergraduate admission requirements and click here, then to page 9, for graduate admission requirements. In general, the reporting is to the applicant, as to whether accepted or not.

(2) Admission to the initial teacher preparation program is handled through the College of Education Office of Academic Services. Admission requirements are given here. Admission to advanced programs is handled through the relevant department. Admission requirements are given under the department materials in the Graduate Catalog. In those few instances (e.g., special education programs) where there is a third-level admission, that process is handled within the relevant department. In all these instances, the reporting is to the applicant, as to whether accepted or not.

For admissions to the initial teacher preparation program, there is a monthly reporting of the applications processed, by teaching field, that is distributed to College of Education administrators. This reporting includes the number of students accepted, by broad demographic category (gender, teaching field level, etc.). These data are incorporated into a monthly report that is provided to all academic administrators directly involved with the preparation of teachers. See Exhibit 2.11. For admissions to the advanced programs that are housed in the College of Education, there is a report received from central administration. (Reporting has fallen behind as a result of conversion to a new system at the central administrative level.) This is condensed and shared with all College of Education administrators. See Exhibit 2.12.

(3) Retention in the initial teacher preparation program is managed through the College of Education Office of Academic Services. Criteria are given here. When a candidate is not retained, that reporting is to the candidate, along with counseling about academic alternatives. For those initial teacher preparation programs that have additional retention requirements (e.g., special education), retention is managed by the department housing the program, criteria are given in the department section of the catalog, and reporting is to the candidate. For the advanced programs, retention requirements are given in the program section of the Graduate Catalog, retention is managed by the department, and reporting is to the candidate.

(4) Mid-program assessments of candidates are described in Exhibit 2.1.A. In general, these are relatively new to our programming. Pilot study data have been reported within the Department of Teacher Education. The Exhibits in the Exhibit 2.3 series represent the first wide-scale reporting of these data. However, Exhibit 2.1.B documents the plan for accumulating this data, under the leadership of COE Associate Dean Jim Berry and the Data and Assessment Advisory Committee. The plan also indicates the timetable for reporting aggregated data to decision-making groups, (a) either the Basic Programs Committee or the Advanced Programs Committee, (b) the College of Education Council, and (c) all relevant academic administrators and other faculty bodies.

Mid-program assessments of programs consist primarily of student ratings of instructors and courses. These data are reported to the instructors. In aggregate, these are published approximately once every two years and are distributed in hard copy to academic administrators. Copies of the Fall 1998-Summer 2000 Student Ratings of Instructors and Courses and the Fall 2000-Summer 2002 Student Ratings of Instructors and Courses are available in the EMU Library, the COE dean's office, or departmental offices.

(5) Near-exit assessments of candidates at the initial level fall into several categories. Prior to Winter 2003, student teaching evaluations, conducted by the cooperating teacher and university supervisor, were reported only to the director of student teaching and to the candidate. However, effective with Winter 2003, student teaching unit results have been gathered and they are reported in Exhibit 2.3.E. The Implementation Plan, shown in Exhibit 2.1.A, shows how these data will be combined with related data and reported, on the timetable given, to decision-making groups.

For many years, the quarterly report of results on the Michigan Tests for Teacher Certification has been widely circulated on campus, especially to the chairs of decision-making groups and the heads of departments where the content is taught. The data in this report give a "rolling" 16-quarter pass rate, by test area. The executive summary of this quarterly report may be seen in Exhibit 2.13.

Near-exit assessments of advanced programs vary widely among programs and include student choice. See Exhibit 2.1.A for the current requirements. Reporting is to the candidate and to the appropriate faculty groups within the administering department. Exhibit 2.1.B includes the plan for greater use of these data.

Near-exit assessment of the initial program by candidates is accomplished through use of the Educational Benchmarking, Inc. survey form. Results may be seen in Exhibit 2.3.F. Reporting has been on a targeted basis, primarily focused on advising and career services. However, Exhibit 2.1.B provides the plan for incorporating these data with other related indicators and the timetable for reporting these to the decision-making groups identified above.

(6) Post-completion assessments of the program are accomplished largely (supplemented in a few instances where programs, especially at the advanced level, have conducted separate post-completion assessments of the program) through the survey of program completers. Results may be found in Exhibit 2.3.G. Exhibit 2.1.B provides the plan and the timetable for incorporating these data with other related indicators and the timetable for reporting these to the decision-making groups identified above.

Post-completion assessment of new professionals is accomplished largely through the survey of immediate supervisors of the new professionals. Results may be found in Exhibit 2.3.H. Exhibit 2.1.B provides the plan and the timetable for incorporating these data with other related indicators and the timetable for reporting these to the decision-making groups identified above.

Data from a variety of sources, internal and external. Data sources are numerous, as may be seen in Exhibit 2.1.A. These include, but are not limited to, grade point averages, standardized test scores, course grades, interviews, auditions and other performance measures, projects, writing samples, lesson plans, faculty-made examinations, written analyses, unit and curriculum plans, media products, observations of work with children/youth, professional behaviors, surveys, reports from immediate supervisors, and many others. Assessors not only include a variety of faculty members but P-12 personnel and immediate supervisors "on the job."

Candidate complaints. Informal complaints are almost always resolved between the faculty or staff member involved, sometimes with the assistance of the department head or comparable administrator. In the rare instances where a candidate alleges a difficulty related to gender, disability, race, national origin, etc., under university policy/procedure that student is referred to the appropriate central administrative office, which assumes jurisdiction for resolution. The university has a well-defined, generally smooth functioning, "grade grievance" procedure. See Exhibit 2.5 to review the policy and procedure. It is rare that a formal student complaint cannot be handled under the "grade grievance" policy and procedure. If the grievance is resolved at Step 1, under the policy the records remain in the department head's office. If the grievance is carried to Step 2, the candidate has a choice of bringing the matter to a department grade grievance committee or the relevant college grade grievance committee. Records of Step 2 grievances are, under the policy, maintained in the dean's office. See Exhibit 2.4 for a brief description of the Step 2 grievances that have been processed in the College of Education in recent times. Records of Step 3 grievances are maintained in the Provost's office.

Reporting of summary data, candidate performance, program quality, unit operations. The current and planned reporting of summary data and candidate performance has been discussed above and is summarized in Exhibit 2.1.B. Program quality, as measured (incompletely and inadequately) by the pass rates on the Michigan Tests for Teacher Certification, is reported quarterly. See Exhibit 2.13. Program quality, as measured by the reports of employers of first-year professionals, is given in Exhibit 2.3.H. Program quality, as measured by the reports of SPA's and state review, is given in the Overview of Programs.

Unit operations are assessed in part by EMU's program review process. In effect, this is a review of the programming of each academic department, conducted on an established schedule. To date, within the College of Education, only the Department of Health, Physical Education, Recreation and Dance (including professional educator programs in physical education [initial and advanced levels] and health education) has undergone this review. However, the content majors for approximately 13 teaching fields have been reviewed as part of the departments in which they are housed. Some of these have received "continuation" ratings; others have received "continuation with specific changes" ratings. No professional education program has received a rating lower than these two. See the Program Review Annual Report.

Information technologies. A very high priority for the College of Education for 2003-2004 is to assimilate a large quantity of assessment data into an electronic format where various data elements can be combined with each other--either for a single candidate or for groups of students or by category of data--such that trends can be discerned and strengths and weaknesses noted. Traditionally, the assessment data described in Exhibit 2.1.A have been maintained on a decentralized basis in a variety of offices, sometimes in paper files, and sometimes on the hard drives of the computers of individual administrators and faculty members.

Under the leadership of Associate Dean Jim Berry, working with the COE Data and Assessment Advisory Committee, conversations began during 2002-2003 with the leadership of EMU's Office of Institutional Research and Information Management with an eye toward creating a web-based data warehousing system. An Enterprise database will be the hub into which data will be collected for the College of Education. A data mart will be developed at the institutional level based on the information needs identified in Exhibit 2.1.B, as well as information needs for program, department, and college management. During the 2002-2003 a full-time COE professional/technical staff member was reassigned to serve as the coordinator of data management for the COE, effective with the Fall Semester of 2003. It is anticipated that there will be data input from most COE units. Many details remain to be resolved.