I model this on statistics I've often taught as SAMPLING THEORY or TESTING HYPOTHESES.In ACCEPTANCE SAMPLING THEORY, you wish to decide if your supplier has given you a good product. It's too expensive (and perhaps destructive) to test every item supplied, so you draw a RANDOM SAMPLING. Given an ORDERING of the ITEMS, you consult a RANDOM NUMBER TABLE. If the number is 11, you inspect the 11th item on the list. Presumably, you've means for telling if it's nondefective or defective. When completed, you consider the percentage of items passing inspection.
A problem arises.
- Suppose the BATCH is "mostly nondefective", but you happen to have drawn most of the "defectives". That is, THE BATCH IS BETTER THAN THE SAMPLE. But SAMPLING leads you to REJECT a "good" BATCH. Statisticians call this "The Producer's Risk".
- Suppose, instead that the BATCH is "loaded with defectives", but you happen to have drawn most of the "nondefectives". THE BATCH IS WORSE THAN THE SAMPLE. But SAMPLING leads you to ACCEPT a "bad" BATCH. Statistician's call this "The Consumer's Risk".
A similar situation arises in "Testing Hypotheses". The hypothesis might have many confirmations "in the long run", but the available sample disagrees, so a "true hypothesis" is rejected: ERROR OF THE FIRST KIND. Or, it can be "false in the long run", but the sample seems to confirm it: ERROR OF THE SECOND KIND.
There are means for reducing these risks, but that does not concern us here.
When I studied this in statistics, it reminded me of my experiences as a student. And, later, as a teacher, I thought much about it.As undergraduate at Columbia University, I worked part-time and I had husbandly duties. Later, as graduate student, I worked full-time and went to school at night. In the last years, I also had responsibilities as parent of two small sons. These responsibilities limited my study time.
Often, I'd be able to study thoroughly most of the material. But the Final concentrated on the little I didn't have time to study. So it did not adequately test what I knew. This, in general, I call "The Sudent's Risk".
And sometimes I'd found time to study only a part of the material, and go into the Final with shivers, only to find that the test concerned the little I studied. So it did not test what I didn't know. This, in general, I call "Society's Risk".
Before stating my "solution" to this problem, I'll briefly recount three extreme cases.One course at Columbia was in Electromagnetism, given in an amphitheater with perhaps 100 students. In the second semster, the prof mentioned the subject of waveguides, which we knew to be one of his specialies. But at the end of the scheduled period on the last day of the Course, he still hadn't mentioned waveguides. Ten minutes after official ending of the Course, he started on waveguides and lectured for nearly an hour. I stayed for this, because I had a hunch. But about half of the class departed for other classes. On the FINAL, this was THE ONLY SUBJECT EXAMINED! Many protested that the subject had not been discussed and were rebutted. One fellow going in with an "A" tore up his test paper in tears and left.
The second case involved a 6 credit graduate physics course in "Analytical Dynamics", given once a year in the Spring, by a particular professor, who shall be nameless. The Course, for 2 hours a day, 3 days a week, was always given in the same Amphitheater, holding 100+ students, with 9 sliding Blackboards down front. There was a tough and confusing text book, A Treatise on the Analytical Dynamics of Particles and Rigid Bodies, by E. T. Whittaker. Professor Nameless had written a 100-page book of "Notes" to study, and a 100-page book of "Problems". BUT THAT WASN'T ALL!!! Grad students advised us to get to the Class, in the Amphitheater, an hour ahead of time. Three Assistants would enter with notes and cover the 9 blackboards with information, which was erased before Professor Nameless entered the room to lecture. And THAT INFORMATION NEVER APPEARED IN THE COURSE AGAIN. Some poor snooks never could figure out where some of that info came from!
The failure rate was 90%. So, you took it as an Undergrad, taking "Incomplete". Sat in again the next Spring. Failed. Then tried it again the following Spring. I escaped that by transferring to NYU.
The other case concerned an undergraduate mathematics course in "Modern Algebra" with "classic" text-book, A Survey in Modern Algebra, by G. Birkhoff and S. MacLane. My prof concentrated on only one chapter of this book, "Determinants and Matrices". A friend, Dick Rosen, became annoyed at the prof's habits and quit the course early. Dick took ostensibly the same course that summer with another prof, who concentrated on a different chapter, "Groups'.
The next fall, I took a course in Computing at Watson Laboratory, run by IBM and Cooumbia Univeristy. (This was before the first programming language. We programmed by plugboards and punched cards.) I described the above to some of the graduate math majors in this course. One rainy day, in the Lab Library, 7 of us compared our notebooks for ostensibly the same course. NOT ONE TOPIC IN COMMON! (This might be called "Truth in College Catalogs".)
When I became a teacher, I worried about this often. When I became Department Head, I "solved" the problem. The students were given a list of everything they should know after passing a given course. A student was examined on a representative problem of each topic. 100% SAMPLING. No Student's Risk because the student knew what to prepare for and was examined on all of the requirements. No Society's Risk because of teacher's sampling.I did this in the days before computers were available for such purposes. It would be easy now. Any teacher who cares can do this.