Item Analyses | Item Analyses | Scoring Multiple Choice Tests | TSQS | Academic Technologies

Item Analyses

TSQS supports two programs that may be used for test (item) analysis:

    • Itemanal
    • Examtab

These programs are designed to be used with the GPSCOR scanner program and the General Purpose Answer sheets that can be processed by GPSCOR. In addition, the Itemanal program can be used with data files produced by the MRSCOR scanner program by treating each response to each item as a True/False item. Appropriate output will be returned along with your processed answer sheets if you indicate that you want one of these programs run by indicating your choice under the Data Processing Required: option on the Request for Service sheet.

A popular practice in a number of departments is to create multiple forms of a test for purposes of administration in large (crowded) classes. This is usually done by reordering the items in the test. In order to analyse all forms of a test as a single set, mapping instructions must be provided so that the items can be aligned to correspond from one form to another before proceeding with the item analysis. If these mapping instructions are not provided, a separate analysis will be performed for each version of the test.

The answer sheets from the different forms of the test do not need to be sorted into separate groups but, if they are submitted as one group, a version number must be coded on each key and the students must be instructed to code the same (appropriate) number on their answer sheets using the Special Codes section on the answer sheet. We recommend that the numbers assigned to the different versions be several digits in length in order to guard against students making undetected errors when coding this number. During scanning, the answer sheets are scored against the keys for all forms of the test. When the subsequent analyses are done, the appropriate version of the test and the corresponding score are selected for the reports.

Itemanal:

Of the packages available at TSQS, Itemanal produces the most comprehensive test and item analysis. It begins by displaying the key that was used to score the test. A value of * is used to indicate that no correct answer was supplied for an item on the key that was used.

The following pages display the FREQuency of occurrence of each SCORE, the Z-score that would be associated with that raw SCORE if the distribution was normally distributed, the PERCENTILE associated with the SCORE (which is the CUMulative PERCENTAGE of scores occurring below the current score plus half of the current scores) and the CUMulative PERCENTAGE of examinees obtaining scores at or below the current value. Finally, the cutting points are reported which will be used to divide the class into high, middle and low groups. By default, the program attempts to place 27% of the class into each of the high and low groups with the remainder in the middle. The success in identifying 27% will depend on the size of the class and the presence of tied scores at the cutting points.

After reviewing the distribution of scores in the frequency table and histogram and the overall test statistics that appear in the sample output, you may find it useful to start at the back of the report where a scattergram is displayed which summarizes the biserial correlations and difficulty levels of all the items. Items having low values of either (say, below .3) and, particularly, items having negative correlations should then be given special attention in examining the item-wise reports. If the class-size is large enough, plots are also provided that display the success rates (difficulty levels) of quintile groups for each item. These, too, can alert you to items that are behaving strangely. In the sample output, the plot for Item 34 illustrates a nicely behaving item while those for Items 13 and 37 indicate a need for closer examination. In the item-wise reports, items for which the HIGH group chooses a particular incorrect response more often than they choose the correct response should, especially, be examined more closely. Also, items for which the LOW group apparently gives the correct response more frequently than the HIGH group or for which the HIGH group chooses a particular incorrect response more often than does the LOW group should also be closely examined.

 


Examtab:

Examtab produces significantly less information than Itemanal. It has none of the graphical output, the frequency distribution does not include the percentile column, and the itemwise report consists of only one line per item. As may be discovered by comparing the output from the two programs, the Point Biserial reported by Examtab is the correlation between the score on the item and the score on the remainder of the test (rather than the total test score).

Main Parameters 'E' and 'F' on the Request for Service sheet should both be zero if you intend to use this program.

 

SCORE       FREQ.       PERCENT       CUM-%
2 1 .75 .75
3 0 .00 .75
4 0 .00 .75
5 0 .00 .75
6 1 .75 1.50
7 2 1.50 3.01
8 2 1.50 4.51
9 1 .75 5.26
10 4 3.01 8.27
11 5 3.76 12.03
12 8 6.02 18.05
13 6 4.51 22.56
14 10 7.52 30.08
15 10 7.52 37.59
16 6 4.51 42.11
17 7 5.26 47.37
18 11 8.27 55.64
19 7 5.26 60.90
20 1 .75 61.65
21 4 3.01 64.66
22 4 3.01 67.67
23 6 4.51 72.18
24 8 6.02 78.20
25 4 3.01 81.20
26 6 4.51 85.71
27 3 2.26 87.97
28 3 2.26 90.23
29 4 3.01 93.23
30 1 .75 93.98
31 1 .75 94.74
32 5 3.76 98.50
33 0 .00 98.50
34 1 .75 99.25
35 0 .00 99.25
36 0 .00 99.25
37 0 .00 99.25
38 1 .75 100.00
TEST MEAN = 18.83
TEST VARIANCE = 46.29
TEST STANDARD DEVIATION = 6.804

 


 

ITEM      POINT BISERIAL      ALTERNATIVES      ANSWERED CORRECT
A B C D
8 .302 22.6 26.3* 13.5 36.8 35
9 .201 5.3* 19.5 17.3 45.9 7
12 .309 51.1* 13.5 18.0 16.5 68
13 .034 13.5 17.3 29.3 27.8* 7
34 .408 12.0 45.9* 24.1 17.3 61
35 .367 26.3 9.8 24.8 37.6* 50
36 .415 14.3 10.5 11.3 63.9* 85
37 -.031 21.1 39.1* 21.8 16.5 52
(PERCENT) (FREQUENCY)

Academic Technologies Sitemap

Classrooms

Book a Classroom
Classroom Listing

Computer Labs

Book a Lab
Lab Listing

Software

N/A

Test Scoring & Questionnaire Services
General Data Collection
Instructor Designed Questionnaire (IDQ) System
Scoring Multiple Choice Tests
Universal Student Ratings of Instruction (USRI)
Web Surveys
High Performance Computing

Shared Computing
Visualization