Specifying SEEM and WEEM exams

Specifying outsourced commodities

    Whether we are in favor of it or not, outsourcing is a reality in today's business world. In our American economy, this is just as true of communication services as it is of manufactured products. In my own city of 1.7 million people, local telephone answering services were quite common ten years ago. When you called your doctor's office, you were probably talking to an answering service which had their employees working in an entrepreneur's private home. Now, (as I was told by one of the remaining three) all but three of these local answering services have closed or been absorbed by 800 servers somewhere else in — or outside — the U.S. Your "doctor's office" may have moved to India!

    Outsourcing has at least one fundamental requirement which is necessary before it can succeed. For a manufactured part to be acceptable, it must meet predetermined specifications (usually called "specs" in the manufacturing world). It can be measured in thousandths of an inch or some other verifiable parameter. These specs are defined in the contract. If the manufactured part is "in-spec" it passes. If it is "out-of-spec," it is unacceptable.

    That is well and good for a manufactured part. What do you do about the answering service from your "doctor's office"? How can you "spec" the communication skills of the people who are answering the phone?

    The unbelievable answer today is that prospective employees overseas are given written English tests to determine whether or not they will be able to verbally communicate on the telephone. A written exam for verbal communication grading is about as effective as giving a quality control inspector a yardstick in order to determine if the shipment of parts just received are within the specified ±.001 of an inch tolerance.

Specifying SEEM exams

    It appears as though other test providers have said,

    It is difficult to objectively grade spoken English tests. However, since we already have "objective" grading standards for written English tests, we will use the objective grading system we already have and change the test from a spoken English exam to one which is written.

    We have taken a different point of view. We said,

    In the past it has been difficult to objectively grade spoken English tests. Therefore, we will develop a new method of scoring so that we can objectively grade spoken English exams.

Isolating the objective

    The first thing we did to make SEEM as objective as possible was to isolate those parts of English fluency which can be easily measured. In listening to almost any foreign speaker of English, we immediately notice that incorrect use of person and tense of the verb are probably the most frequent errors. Therefore, as a first step we can objectively measure use of person and tense as a means of scoring spoken English. (For examples, see the sample grader's work sheet.)

    We then did the same with sentence construction. We can objectively determine a score based on the speaker's use of vocabulary and syntax in a sentence. Unlike written exams, we force the examinee to develop his or her own sentence structure. That is, written exams typically use incomplete sentences with fill-in-the-blank answers. Incomplete sentences give numerous clues to the correct answer. When using SEEM, the examinee is given only pictures (or key words in the Supplementary Vocabulary Sections). Therefore, the examinee must structure the entire thought and sentence for the response. This allows us to apply our objective standard to an entire sentence rather than merely a portion of a sentence.

    We also added bonus words. In the example from Section II, the examinee was able to use the word "blonde" rather than "yellow" when describing hair color. The examinee was given additional credit for the word "blonde" because it indicates greater language fluency.

    We can then assume (and it is generally quite accurate) that if speakers can use person, tense, basic vocabulary, and syntax correctly in sentences they create themselves, they have enough experience with the English language to use it effectively in normal (non-technical) communication.

Quantifying the subjective

    Although we isolated some important elements we can now objectively test, we still have a hurdle to cross. We still need to assign an objective grade to the examinee's pronunciation. Yet, we need not throw our hands in the air over the impossibility of testing pronunciation objectively. We can create a screen for pronunciation so that it can be quantified. Very simply, if we train our graders to "listen with only normal attention," we have established a way of converting the subjective problem of pronunciation into an objective score. This "moderate attention" exactly duplicates the degree of attention a future American client will give an examinee over the telephone.

    We will instruct our graders not to pay undue attention while listening to the examinee's recorded response. If the grader can easily understand the examinee's statement the first time it is played, the examinee is given a grade of 8. If the grader must pay particular attention to each word on a second play-back in order to understand what was said, the examinee is given a grade of 4. If each word cannot be understood on the second play-back, the examinee receives a 0. When all of the scores for the entire test are added together, this procedure will give a very satisfactory "objective" grade of the examinee's ability to pronounce English.

    Please look back at the sample grader's work sheet. Notice that pronunciation resulted in a grade of 8 of a total of 29 points. This question took approximately five minutes for the examinee to complete. That means that in a forty-minute time period, we could expect approximately eight complete responses. Just for the sake of illustration, let's say that this examinee scored a total of 192 points of which 60 points were given for pronunciation. We could show the score as 60/192/8 in order to emphasize both the pronunciation and English syntax components of this exam. When compared with data provided by the testing agency, this would give good indication of the examinee's ability to pronounce English. (60 would be a high score for pronunciation inasmuch as a perfect score of 8 for eight responses would be 64.) Notice that our score has three components — 60/192/8. The final "8" represents the number of completed responses. Obviously, 60/--/8 represents a higher pronunciation score than another student with a 64/--/10 score. Thus, the first examinee has a 7.5 pronunciation score (60/8) while the second examinee has only a 6.4 score (64/10). In the same way, the English syntax score can be reduced to an independent number. The first examinee with a 60/192/8 scored 24.0 (192/8) in English syntax. We did not give an English syntax score for the second examinee.

    We might also add that we are not testing another country's English. We are testing "American" English. If the examinee will be answering telephones for U.S. clients, it is appropriate that the employer expect "American" English fluency. (We have suggested elsewhere that a United Kingdom grading branch be established to grade exams where British English is used.) However, the very purpose of a spoken English exam is to spare the prospective employer uncertainty when an examinee can write English acceptably but uses "pidgin" English when speaking. We are not passing judgment on what is acceptable in other countries. We are merely offering an exam which assures American (and possibly United Kingdom) employers that the people they are selecting can communicate with the employer's clients.

Expanding the base

    In most cases, we now need to go a step further and establish that the examinee can use specialized English vocabulary within a specific field. For example, if the examinee is applying for work in computer software support (as we are suggesting in our illustrations) then we must test his or her ability to use the specialized English vocabulary in that field. In order to do this, we would establish a large number of Supplementary SEEM exams. There would then be a Supplementary SEEM exam for any specialized English vocabulary required in an employment field. With these supplementary exams, we can assure the future employer that the examinee would be graded for his or her understanding of the specialized English terminology in that given field.

    Since SEEM does not require any security prior to testing, all Supplementary SEEM exam vocabularies could be posted on a website for evaluation. This would aid an American company in selecting the supplementary section they wanted to use when specifying Supplementary SEEM exams in a contract. This obviously becomes an excellent marketing tool inasmuch as it gets American companies involved with SEEM and encourages them to use it in contract negotiations. (See the SEEM exam page regarding unsecured use of SEEM for study prior to the exam.)

Using SEEM with supplementary SEEM vocabulary sections

    The entire SEEM exam is modular. A single section, or any combination of sections, could be used for a particular test. For example, a software development company in the United States may contract with a telephone-based computer support company in India to handle their technical support. The American company wants to ensure that the general English fluency of the support personnel is adequate. The American company also wants to ensure that the support company's employees can communicate effectively in the specific fields which are required for the job.

    The American software development company may specify the following SEEM exams for various telephone operators. (Please note that the scores do not have statistical relevance yet. They are being used for the purpose of an example only. In all probability, the score would be given as both a pronunciation and English syntax score. In this illustration, we are merely using a raw cumulative score.)

    SEEM has the great advantage of offering a spoken English exam which can be defined as a specification within an overseas contract. An American company can specify minimum scores for those who will be speaking with their clients. Because SEEM would be administered (graded) in the United States, the specified score becomes a true indicator of spoken English fluency.

    Further, because the exam is modular, the specification can be tailored to match the specific job position, as suggested in the table above.

What does the grade mean?

    Early in the development of SEEM, a grading curve would be established which would show future employers the level of English fluency represented by a given score. Initially, this would be done by comparing grades of early examinees with their ability to communicate using spoken English. It should not be difficult to establish a satisfactory grading standard before SEEM would be published.

    However, as SEEM became more widely used, a large database would accumulate. The program written for SEEM should include an ever-growing body of data which can be used to show average performance.

    There is another useful option in addition to using statistical data. If an appropriate statement was included in the exam's registration allowing use of the recorded exam without additional examinee notification, spoken examples of grading could be published on a demonstration website. For example, in the above illustration we cited two scores of 60/192/8 (a 7.5 pronunciation score) and 64/192/10 (a 6.4 pronunciation score). As data was accumulated, the SEEM website could use actual exam recordings (without identifying the examinee) to demonstrate the actual pronunciation difference between a 7.5 and 6.4 recording. A wide range of actual scores could be demonstrated, allowing the American business requesting the SEEM exam to realistically specify a score which met their expectation. (We could guess that dissatisfaction would quickly surface with written exams. Thus, the use of a truly objective spoken English exam would drive the standard much higher than it is today. That shift would do exactly what we desire. It would bring even more business to the one institution which was able to objectively score spoken English.)

Public Domain notice: All descriptions of SEEM and WEEM are freely given without either stated or implied copyright protection. Spoken Language International does not have the financial means to develop or administer a spoken or written English exam. However, it would be to Spoken Language International's advantage that such exams be extensively used because of the demand it would place on our Spoken English Learned Quickly language study course. Thus, we present the SEEM and WEEM exams on this website with the hope that another institution will further develop and market them under any name. We relinquish any claim for remuneration of any kind as a result of another institution using this material for their own profit. Future publication of this material would carry the full copyright protection of the publisher and will then be subject to full enforcement of that copyright protection.