Field Tests

FIELD TEST OF THE WordMAP™ WRITING AIDS SYSTEM

Lincoln County School District

Panaca, Nevada

May, 1986

By

Dr. Eldon G. Lytle

Linguistic Technologies Inc.

Dr. Neldon C. Mathews

Lincoln County School District

DISTRICT OBJECTIVES

Long before the advent of A Nation At Risk, the Lincoln County School District had established committees to analyze the curriculum and make recommendations for a stronger and more effective educational program. As a result of the hearings and studies conducted by these committees, one of the priority concerns that was identified was in the area of writing competencies. Subsequent to the study, the administration established a set of goals and objectives whereby the writing program in the district could be directed and evaluated. Basically, the adopted competencies included the following criteria:

1. The ability to select, organize, and relate ideas, and to outline and develop these ideas in coherent paragraphs.

2. The ability to write standard English sentences with correct: sentence structure; verb forms; punctuation; capitalization; possessives; plural forms; word choice and spelling; and other matters of mechanics.

3. The ability to improve one’s own writing by restructuring, correcting errors and rewriting.

4. The ability to gather information from primary and secondary sources, to write and report using this research; to quote, paraphrase, and summarize accurately; and to cite sources properly.

Although these guidelines did not encompass all aspects of the writing curriculum, they did provide a point of departure and a reasonable guide.

The major positive consequence of the formulation of the writing competencies precipitated in the form of greater awareness and increased emphasis on writing.

Unfortunately, the assessment of the program and the evaluation of students remained on a rather subjective basis. Coupled with the problem of a lack of objective measurement, there was no provision for longitudinal tracking to determine growth, maturity, and sophistication of the student’s writing development.

The ability of the district to definitively appraise the outcomes with respect to student growth and program effectiveness, particularly in a historical perspective, was not insured or provided by the competencies and process adopted by the district.

THE WordMAPWRITING-AIDS PROGRAM

The acute need to generate a writing program that would provide student reinforcement, evaluation of writing proficiency, establishment of norms, program analysis and tracking of linguistic progress, piqued the district administration until the WordMAP™ program was demonstrated by Linguistic Technologies. Since this software appeared to respond to all of these needs in a flexible “user friendly” format, approval was granted for a field-test in the district. The paragraphs below describe the WordMAP™ software, review the field test experience, and summarize its outcome.

THE NATURE OF THE SOFTWARE

WordMAP™ is a software system that analyzes English material written by students (or anyone else), checks spelling and certain aspects of grammar, and carries out a variety of other tests relating to both competence and style.

The tests carried out in the current configuration are:

1) GROWTH LEVEL (linguistic maturity)

2) WORD BUILDING (use of prefixes and suffixes)

3) ADVANCED WORDS (as opposed to everyday vocabulary)

4) VOCABULARY DENSITY (repeated use of same words)

5) DESCRIPTION (use of descriptive modifiers)

6) COMPLEX SENTENCES (as opposed to simple sentences)

7) GRAMMAR CHECK (summary index for spelling/grammar)

8) OVERALL (summary index for all of the above)

Other tests can be readily devised, depending upon the needs and desires of English faculty.

WordMAP™ has an expandable database of linguistic patterns for famous authors and accumulates a similar database for students whose compositions are analyzed by the system. The software also includes statistics modules for author and/or group identification and comparison.

WordMAP™ generates a variety of listings, including grammar check listings which highlight errors in English text, comparative performance listings that actually assign scores to student papers based on test results, and listings that relate student performance to peer group and/or age-group norms.

Computerized tutorials for vocabulary building and part-of-speech recognition were also offered as a complement to WordMAP™.

PURPOSE OF THE FIELD TEST

The test was undertaken to determine:

1) what adjustments or extensions to the software might be required for use in the public schools;

2) whether students in the various age-groups could successfully type their own English compositions into the system;

3) what problems might be encountered so far as integration with current English curricula is concerned; and

4) how much training would be required for teachers to become competent in the use of the software.

DEFINING LINGUISTIC NORMS FOR THE DISTRICT

In January, 1985, the WordMAP™ software was installed on IBM PC-XT work-alike units at Lincoln County High School, Panaca, Nevada; and Pahranagat Valley High School, Alamo, Nevada. A district-wide writing assignment (“The Importance of Education”) was made to all students in the district in grades 4-12. In all, six schools participated in the writing project. Teachers conducted a discussion on the assigned theme topic in their respective classrooms in advance of the writing itself so that all students would have something in mind to write about the subject. Students were then asked to write 250-500 words, depending upon age, on the selected theme in a class-room setting without aid or influence of teachers or other students.

Student themes were subsequently typed on the WordMAP™ units by adult typists who eliminated spelling and grammatical errors which might otherwise have interfered with correct linguistic analysis by the software. The writing samples thus obtained were then processed and used to establish initial linguistic norms for grades 4-12 in the Lincoln County School District.

The norms in question were defined by the software in terms of linguistic variables (developed through prior research) which are correlated with the linguistic maturation process. In this case, the resulting norms were found to be approximately equivalent to results obtained previously for the same age groups in the Wasatch School District of northern Utah. The norms extracted from this initial data were stored by WordMAP™ in its database to be used as a basis for subsequent evaluation of linguistic maturity.

With respect to individual students, it was found that some high school students were still using vocabulary and syntax characteristic of elementary school children. Conversely, some elementary school students were using vocabulary and syntax well beyond the norms for their respective age levels. The listings generated by WordMAP™ contained information relating to specific strengths and weaknesses of individual students and groups of students in accordance with design specifications.

STUDENT ENROLLMENT ON WordMAP

Due to the potentially sensitive nature of WordMAP™ test listings, confidential ID numbers were assigned to each student. These numbers were initially constructed and assigned manually on the basis of class roll data already in use. However, when this approach turned out to be tedious and error-prone, a computerized method was implemented and the corresponding modules incorporated into the WordMAP™ software ensemble.

THE ENGLISH CLASSROOM AS A COMPUTER LAB

Due to budget and space limitations, a decision was made to acquire inexpensive laptop computers for the secondary schools which could be used by students at their desks. The units purchased contained 24K memory and built-in word-processing and communications software. Two sets of thirty (30) units were purchased, each set sufficient to provide for an English class of thirty (30) students.

In one school, a mobile unit with a work surface and storage bays was constructed to house the WordMAP™ computer, printer, portable units and supplies. This cart was moved from classroom to classroom, as the various English sections participated. In another school, one English classroom was selected to house all of the equipment, and the various English sections took turns using the room. Both of these arrangements proved to be workable.

In another arrangement, the WordMAP™ computer was installed in the library and students went there to transfer files to it from the portable units. This arrangement turned out to be awkward, since the teacher could not be present in both the library and the classroom.

Elementary schools in the district were already equipped with a substantial number of Commodore-64 units. Additional numbers of these units were purchased for the elementary schools. WordMAP™ communications software was expanded to receive text from either the portables or the Commodore units.

INITIAL TRAINING FOR STUDENTS AND TEACHERS

Students were given basic instruction in the operation and care of the units provided for them. Additionally, a special seminar was conducted for teachers and other interested staff to familiarize them with the rationale of the WordMAP™ linguistic testing modules, the listings generated by the software, and how to transfer files from the portables units to the WordMAP™ computers. Arrangements were made with The University of Nevada, Las Vegas, to award credit to teachers taking the seminar.

STUDENTS DO THEIR OWN TYPING

Next, a second standard theme assignment was made on a district-wide basis, with students required to type their own compositions. Students reacted enthusiastically to the opportunity to use computers for an English assignment. Students who had prior typing experience finished before those who had not. In some cases, typing assistance was provided to slow typers by fast typers. It became apparent that the efficient implementation of computer-aided writing would require that all participants take typing and that this instruction be introduced early in the district curriculum.

The various sections of English in the secondary schools took turns using the portable units until all sections had typed in their theme assignments.

The themes were processed by turn through WordMAP™ and reports generated showing both student and class performance in relation to the district norms previously generated.

INTRODUCTION OF THE WRITING CYCLE

In August 1985, prior to the start of school, a second seminar was conducted in the use of WordMAP™ for English faculty. The purpose of the seminar was to evaluate progress, provide additional instruction, and plan for expanded use of the software.

At the outset of the fall term, 1985, a two-phrase writing cycle was introduced.

In the first cycle, students wrote a rough draft, transferred it to the WordMAP™ computer, and received back a spelling/grammar check listing. In the second cycle, this listing was used by the students to revise and improve the rough draft, which was then transferred again and processed as a final draft. Final drafts were checked by WordMAP™ again for spelling and grammar, as well as for performance on the specific tests identified on page 2 of this article.

Standard topics were used for these writing assignments, and each section of English took its turn using the portable units. In order to control and organize the use of the portable units, each portable was marked clearly with an inventory number, and students were assigned to specific machines. With regard to typing efficiency, it was found that slow typers could minimize time on the portables by first writing their compositions out in longhand so that they could then concentrate on the typing process.

REVISING AND EXPANDING THE GRAMMAR CHECK

The initial format of the grammar check listings turned out to be confusing for some of the students, since certain words were decomposed during analysis and/or converted into base forms. The listing was accordingly revised to display text in its original form, as typed by the student, with error flags inserted.

Additionally, grammar check listings were routed through the WordMAP™ formatter so that they would be printed in finished form, except for error flags. Provision was also made to use special fonts to signal certain types of errors (e.g., misspellings were printed in bold-face type). The number and type of checks made by WordMAP™ was expanded to comprise the following:

Sentence initial AND, subject/verb agreement, possessive pronoun agreement, Be verbs, passives, sexually biased expressions, sequential IF clauses, sequential BECAUSE clauses, homonym pairs, colloquial expressions, trite expressions, crude or profane language, slang, dangling prepositions, split infinitives, double comparisons, double negatives, sentence fragments, long sentences, comma splices, run-on sentences, suggested punctuation, ‘s check, THAT omission, corrections for common misspellings and faulty contractions.

Graphic reports were added as options to the Grammar Check listing to depict: Check item, check type (SPELLING, GRAMMAR, etc.), print format, margin of error, weight, number of occurrences, score by item, score by type, and accumulative scores by item and type.

The option to activate part-of-speech flags was also incorporated, so that, for example, all adjectives and adverbs could be flagged in a particular listing and included as items in the corresponding graphic reports.

It was found that younger students were not prepared to handle the entire inventory of error checks made by WordMAP™. To solve this problem, the Check module was redesigned to allow individual teachers to selectively activate or deactivate specific checks. Eventually, four distinct levels of check were formulated, each successive level using more checks than the preceding level. Provision was also made for reassigning checks among the levels via a simple editing process.

MODIFICATIONS AND ADDITIONS TO REPORTS

The summary test reports generated for final drafts were improved and expanded to depict year-to-date performance as well as performance on the current assignment. Computer scoring options on these reports were expanded to include bonus points, percentages, or letter grades. An additional report was designed for return to the individual student showing performance in relation to the class average for each linguistic test.

BATCH SETUP AND RUN PROCEDURES

Initially a general purpose utility was used to select processing options and reports, and to submit batch jobs for processing. Later, a special Job Setup utility was prepared which allowed teachers to select most-used processing options and reports with a few keystrokes. This utility also listed a JOBSHEET showing the names of all students having papers in a particular batch, the length of each paper, the processing options used, the backup diskette(s) reserved for those papers, etc. In some cases, students learned to transfer files without teacher assistance and to setup and run batches of papers through WordMAP™.

In general practice, papers transferred to the WordMAP™ computer were accumulated during the day and left to run as a batch overnight. The following morning, the teacher or an aid would distribute the listings to the students. Owing to intermittent failures of the power supply at one school, a power backup unit was purchased and used during all-night batch jobs.

CURRENT STATUS OF THE PROGRAM

The WordMAP™ system was initially installed on a contractual basis for a period of one term in order to conduct a field test of the system. Under the contract, the district was to receive a subsequent license to use the software on a continuing basis if it so desired. At the conclusion of the initial test period, the proprietors of WordMAP™ and the district mutually agreed to extend the test period in order to implement the two-phase writing cycle discussed above.

As of the current date, the district has purchased additional equipment and is exercising its option to use the software on a continuing basis. Equipment allocated and/or purchased for the writing program in the district includes six (6) IBM PC-XT work-alike units, six (6) printers, sixty (60) laptop portables, 52 Commodore-64 units, one mobile unit, transfer cables, null modems, and other supplies.

BENEFITS OFFERED BY THE PROGRAM

Benefits offered by the WordMAP™ Writing Aids Program as implemented in the district may be stated as follows:

(1) The program strengthens computer literacy in a general way since all students are required to take English and all English classes grades 5-12 have the opportunity to use either portable units or Commodore-64 units.

(2) Typing and word-processing skills are strengthened in a general way since the same students must also acquire some degree of typing skill and facility in the use of a word processor.

(3) Student compositions typed on the computers and listed on the printer are more pleasant for teachers to review than their corresponding long-hand versions.

(4) The spelling/grammar check facility of the software provides valuable feedback to the students which teachers often do not have time or the inclination to give.

(5) The checks made by the software and the corresponding flags in the listings provide a practical framework for teaching grammar and some elements of style. In short, the program provides an excellent opportunity to devise and evaluate new teaching strategies.

(6) The supplementary linguistic tests made by the software relating to linguistic maturity, word-building skill, etc., provide new information to both teachers and students which can assist in identifying strengths and weaknesses.

(7) Auxiliary drill and practice modules for vocabulary building and part-of-speech mastery are offered in conjunction with WordMAP™ for use by students found to be weak in these areas.

(8) Reports generated by WordMAP™ and added to student files can be used to track the linguistic progress of students from term to term and from year to year.

(9) The capability of WordMAP™ to define performance norms for groups of arbitrary size makes it possible to comparatively evaluate the linguistic performance of age-groups, English classes, schools and even districts with the software.

(10)WordMAP™ is designed to classify library books for the various age-groups. This is done with reference to the database of linguistic patterns for age-groups resident on the system.

(11)The software can be used to test text-books and other printed materials for compatibility with the age-groups for whom they are intended.

(12)WordMAP™ has an expandable database which contains linguistic patterns for many famous authors. The data in this library is available for use by students of comparative literature and/or stylometrics.

(13)The statistical facilities of WordMAP™ in conjunction with its linguistic database make it an ideal tool for experimental or actual projects in author or group identification.

(14)WordMAP™ provides unique opportunities for English-related science fair projects and research activities for both students and faculty.

(15)The intelligent use of WordMAP™ promises to upstep language-arts programs which have become tradition-bound and to capture the interest of students who may otherwise neglect English in favor of math or computer science.

AREAS FOR ADDITIONAL WORK BY THE DISTRICT

The district has not utilized all of the above capabilities of the system during the field test. In terms of what has been accomplished to date, the following recommendations seem appropriate:

(1) Lack of typing skill on the part on some students often delays the program. Remedial work in typing for stragglers would be helpful.

(2) Some students do not make corrections to rough drafts even though they are indicated by the Grammar Check listings. A stiff penalty system for negligence in this regard may be helpful.

(3) The special test reports are not yet fully utilized by teachers to counsel students and make individual assignments in areas of weakness. A file of such reports should be created for each student, and systematic use of the information they contain should be made by students, teachers, and administrators. Reference to these reports and listings during parent-teacher consultations would also be beneficial. Additional training for teachers in this area appears needful.

(4) Facilities of WordMAP™ for coordinating text-books and other materials with age-group norms were used on one occasion. Expanded use of this capability is a logical next-step.

(5) There is some question on the part of English faculty with regard to how the computer-aided writing program is to be integrated with the traditional program. (A committee of teachers is being appointed to study and make recommendations.)

(6) One science fair project was undertaken using WordMAP™ to comparatively evaluate the English performance of sophomore males and females and to determine whether there may be distinctive linguistic patterns which typify the sexes. Expanded use of the system for student/teacher research should be encouraged.

(7) Students at one school tested samples of their own writing for proximity to persons in the famous author database. Literature classes could well expand the library with samples for modern authors that they are currently studying and undertake some exercises in stylometrics.

CONCLUSIONS

The field testing described in this narrative has provided more than sufficient data to alleviate the assessment constraints normally encountered in the traditional classroom approach to writing. The WordMAP™ process does in fact appear to provide a vehicle whereby writing excellence can be achieved, provided that there is willingness to:

(1) Acquire or allocate the necessary equipment and organize space for its use.

(2) Become familiar with the capabilities of WordMAP™ and use it frequently as an integral part of the writing curriculum.

(3) Make systematic and intelligent use of the listings and reports generated by the software.

(4) Exercise patience while students learn typing and word-processing skills.

Leave a Reply

Your email address will not be published.

*