8. FORMATIVE ASSESSMENT

Attach here with the link of a pop quiz.

http://www.proprofs.com/quiz-school/story.php?title=NjEwOTA0

Everything About Pretesting » ProProfs Quiz Maker

6. SUMMARY

7. FREQUENTLY ASKED QUESTIONS (FAQ)

1. What is the difference between pretesting and trialling?

They both involve trying out test materials on a representative sample of the test group to gather various types of information about their performance and measurement characteristics.

·         Pretesting is a general term for this kind of activity, but it is also use more specifically to refer to occasions when test materials are administered to large groups of test-takers in order to carry out a range of statistical studies on the scores produced.

·         Trialling is often used to refer to a form of pretesting involving much smaller groups of test-takers who can provide useful feedback on different performance aspects of the test materials.

2.      How can pretesting help teachers and learners?

It helps learners to know which areas they need most practice in, and gives them experience and confidence in taking tests. For teachers, it helps highlight areas where their learners need more help.

3. Are items/questions pretested? If yes, how are students selected?

Most of the examination boards did pretesting; as most of them think that it has pretested items on a few students “as security of material is essential” and the other said test-types and difficult items were pretested informally by settlers/examiners in their own school using secure candidates.

4.If items/questions are pretested, how are they being analyzed?

Majority wise, for objective tests, there are two methods being applied to calculate for each objective test item.

(i) The facilty value (F.V.)

(ii) The discrimination index (D.I.)

However, for subjectively marked tests like summaries, essays and oral interview, there will be no definite statistic can be used. However, there will be marking guideline and criteria to ensure the trial marking session should follow the general patterns.

5.What happens if trial items/ questions are unsatisfactory?

If items proved to be unsatisfactory, they were either rewritten or discarded. However, there are some testing centres like UCLES which have said that any revised items will have to go through another pretesting cycle. Every time an item is rewritten it should be tried out again, as there is no guarantee that an edited item will be any more successful than the previous item. In reality, retrialling of items seems to be impossible, for that case, item analysis should be carried out after the live test has been administered, and faulty items should be omitted from the final scores.

6.What steps, if any, are to be taken to monitor the quality of individual item/ test writers        

·The designers or the objectively marked papers were involved in editing sessions and were monitored by the overseeing chair and the subject officer.

·A training course for item writers is provided.

·Draft tests were monitored and feedback was given to writers before the exam.

· Markers provide feedback after the administration of an exam, and the item writers’ work was then monitored.

·Those item writers with “moderation revealed quality” while unsatisfactory writers were “discontinued”.

4. TABULATION & ANALYSIS OF DATA

HOW TO CALCULATE AND TABULATE STATISTICS

In Statistics, the most important calculations are the mean, mode, median, variance and standard deviation. Statistics begins with a set of numbers which are called the Sample. The set of all possible numbers is called the Populations. Let’s say that we have 20 students in a class with their English Test score scale from 50-100. Then the sample size is 20 and the population is the set of all people who have take or will take the test.

The Table shown below is an example of the tabulation of students’ score.

Student	Score
Student 1	55
Student 2	77
Student 3	90
Student 4	66
Student 5	71
Student 6	67
Student 7	66
Student 8	81
Student 9	66
Student 10	88
Student 11	73
Student 12	66
Student 13	85
Student 14	82
Student 15	66
Student 16	76
Student 17	64
Student 18	66
Student 19	90
Student 20	87

To calculate the mean, you sum up all the numbers in the sample and then divide by the sample size. The sum is 1475. The mean is 1475/20 = 73.75.

CALCULATING THE MODE

The mode is the number that appears the most often in the sample.

To calculate the mode, we count the number of times each rating is made. In the sample above, the number that appears the most often is “66” that appeared for 6 times in the table. Hence, the mode is 66.

But what would happen if we have the following sequence : 66,66,70,77,77

In this case, a conclusion that can be draw is that there is no unique mode. A mode is unique if and only if ONE number is more frequent than all others.

What if the sample size is too big such as > 50 ? It would cost a lot of time and may contain inevitable human mistake in arranging the data. Hence, we can make use of Microsoft Excel in calculating the median for us. Firstly, the score is entered into the Microsoft Excel and let’s use the tabulation above as example. As the result, the data entered would range from row A2 to A21. After that, at the column below the last data insert this formula in order to get the MODE ; =Mode(A2:A21) and press enter.

CALCULATING THE MEDIAN

The median is the value we get when we order all of our scores and then find the one in the middle. But what happens if the sample size is even? In this case, we can add the two middle numbers and then divide by 2. For the tabulation above, the sample size is 20 which is even and hence the two middle numbers need to be divided by 2 and the result is 72.

***BEAR IN MIND THAT ALL THE SCORE MUST ARRANGE IN A ASCENDING ORDER BEFORE CALCULATING THE MEDIAN IN ORDER TO GET THE CORRECT RESULT.***

Similarly to the mode, we can use Microsoft Excel to calculate the median of a set of data. Firstly, the score is entered into the Microsoft Excel and let’s use the tabulation above as example. As the result, the data entered would range from row A2 to A21. After that, at the column below the last data insert this formula in order to get the median ; =Median(A1:A20) and press enter.

CALCULATING VARIANCE

The variance is a measure of the variation of the sample data. The larger the variance, the more random the answers appear. Most people find that standard deviation is a more useful way of measuring variability.

The method for calculating the variance is different depending on whether we are calculating the variance of a population (everyone) or the variance of a sample. In order to calculate the variance of a given sample data, one must follow the steps provided below :

1.     Calculate the mean of the sample data.

2.     Calculate the difference between each value and mean. (Negative values might be obtained and it is perfectly normal)

3.     Multiply the answer in Step 2 to the power of 2 or in mathematical term “Square” it.

4.     Sum up all the values calculated in step 3.

5.     For the sample variance, we divide the sum in step 4 by the (sample size -1).

The formula for the variance is :

 = Variance

X = respective sample value

 = Sum of

X bar = Mean or average

N = Sample size

Similarly, Variance could be calculated in Microsoft Power Point by inserting the appropriate formula. Firstly, the score is entered into the Microsoft Excel and let’s use the tabulation above as example. As the result, the data entered would range from row A2 to A21. After that, at the column below the last data insert this formula in order to get the variance; =Var(A2:A21) and press enter.

CALCULATING THE STANDARD DEVIATION

The standard deviation, like variance, is a measure of the variation of the sample data. The larger the standard deviation, the more random the answers appear. Standard deviation is a more popular way of measuring variation compared to the variance. The method of calculating standard deviation is the same as the variance but with one extra step. The steps of calculating standard deviation are :

1.     Calculate the mean of the sample data.

2.     Calculate the difference between each value and mean. (Negative values might be obtained and it is perfectly normal)

3.     Multiply the answer in Step 2 to the power of 2 or in mathematical term “Square” it.

4.     Sum up all the values calculated in step 3.

5.     For the sample variance, we divide the sum in step 4 by the (sample size -1).

6.     Lastly, we take the square root of the value in step 5.

The formula for calculating standard deviation is:

Again, similarly,standard deviation could be calculated in Microsoft Power Point by inserting the appropriate formula. Firstly, the score is entered into the Microsoft Excel and let’s use the tabulation above as example. As the result, the data entered would range from row A2 to A21. After that, at the column below the last data insert this formula in order to get the Standard Deviation; =STDEV(A2:A21) and press enter.

Preferences

2. THE PRETESTING PROCESS

The commissioning of exam questions must be carried out before they go live and tested on students. Once pre-editing and editing takes place, questions are placed into a "pretest bank". A pretest is a test paper that consist of a number of sections, similar to the real exam. Once the paper are distributed to the centres, administered, and returned back to the test constructors, the sample is marked, analysed and reviewed. The test constructors could therefore assesses the level of difficulty of the questions constructed in the test sample in relation to the accepted level for that particular exam. Apart from that, test constructors can also see how successfully each question discriminates between candidates whose overall performance varies. 
Any editing done to the answer key, the multiple choice questions or the structure and content of the question may necessitate returning the question or section to another round of pretest to reevaluate and reviewed for its acceptance in the school and Examination Council standard and context.

WHAT IS INCLUDED IN A PRETEST
A complete set of pretesting sample should consists of following materials:

1. Instructions to administrators
2. A student register
3. Well printed question papers
4. The anchor test with Candidate Information Sheet
5. Answering sheet, blank paper and any other relevant materials 
6. A feedback form for invigilators.

Pretest vary in length, but are designed to be administered within the teaching period. Each pretest usually consists of sections that are similar to the real examinations. Some pretests contain material intended for use in a revised exams and may contain question types that will not appear in a live examinations for a period of time. A revised examinations, however, is designed to test the same areas of language as the exam it replaces and indirectly provided students with the opportunity to try the new questions types.

Feedbacks From Invigilators

Feedbacks form about the pretests filled by the invigilators are extremely important as it provides a lot of information to the test constructors. This information and data are always analysed when considering whether or not the pretested materials is suitable for uses in a future live examinations.

The Anchor Test and Candidate Information Sheet

Live examinations are usually a combinations from a number of pretests obtained from the "live exam bank". It is therefore important for us to identify the difficulty of pretests and use some time to compare and contrast all the pretests we administer.

The Candidate Information Sheet, completed prior to the Anchor Test, allows us to compare the sample of candidates taking each pretest, as we need to ensure each pretest is administered to a range of candidates.

Answer Sheet

Most answer sheet for Multiple Choice Questions are scanned by the computers. Each answer sheet is personalised by the school examination board and it is important to ensure that the test takers or candidates maintain the same candidate number throughout all the pretest they participate in. The invigilator should also need to ensure that candidates must retain the same candidate number when they complete the student register.

How Should Pretest Be Administered

It is common that pretests may not always be administered in a live examinations hall, and are not invigilated by the officials of examination council. It is normally invigilated by the school teachers. In order to simulate the conditions of a real live examinations as well as to maintain the credibility of the data of the pretests, it is essential that every pretest should be administered in exactly the same way as live examinations with the same level of invigilation. For example, candidates are not allowed to talk during pretests and are not allowed to use any tools and materials that is considered as cheating in the process of answering the pretests. Pretest materials are intended to be used in the future live examinations and should therefore be treated as highly confidential. Photocopying or any other means of copying the pretests materials are strictly prohibited.