Update on the citizenship test redesign project

from Lynne Weintraub

In December, USCIS announced that is it working on revisions to the 2008 citizenship test. Here is a summary of what we have learned about the test redesign plans to date, my analysis of the changes, and recommendations for the redesign project.

USCIS’s rationale for the changes

We’re told that these changes are in response to stakeholder feedback and new options in technology, and that the overall objectives are “to align with best practices in the field of test development and to further standardize the naturalization test.” What this actually means is open to conjecture. It is unlikely that stakeholders have been demanding new testing technology or standardized testing per se. However, advocates have for years pointed out that citizenship applicants in one field office can have an entirely different interview/testing experience from applicants in another field office, and results frequently vary even from one examiner to another. It is conceivable that the current administration is reengineering the test procedure to make it more efficient and remove it from the interview process (where adjudicators determine whether the applicant meets the legal requirements for citizenship). If this is their intention, we have reason to be optimistic that applicants will see an improvement in the fairness and uniformity of their experience in regard to testing. But as is often the case, the devil is in the details, and USCIS has not divulged many details thus far.

The redesign project

Revisions are being made to two parts of the test: the English speaking test, and the civics test. The new test items are currently being developed by in-house “subject matter experts” and will be reviewed by a committee of outside experts in the fields of “language acquisition, U.S. history and civics, and test development.” So far, we have no information on who these experts will be and when their work will begin. Later in 2023, USCIS plans try out the new test items to see how well they perform with various types of students in adult ESL programs throughout the country. The test is expected to be operational by the end of 2024.

No change in the English literacy test

I was relieved to learn that no changes are being proposed for the English literacy (reading and writing) test, since USCIS has determined that this component is already standardized and aligns with best practices. I have never heard a citizenship teacher or student complain about the level or fairness of the literacy test, and I am pleased to learn that elements of the literacy test are being used as a model for the proposed new speaking test.

The proposed speaking test

USCIS plans to discontinue the practice of using responses to personal background questions from the N-400 citizenship application to assess oral proficiency. I strongly support this change because vocabulary on the N-400 includes legal terms and concepts that are far beyond what we normally teach, even in the most advanced ESL classes (e.g. “genocide,” “Nazi,” “totalitarian” etc.) and I believe the practice does not conform with USCIS regulations which state that applicants are only required to demonstrate the ability to understand “words in ordinary usage in the English language.” I think USCIS is right to be concerned that the current procedure for assessing oral competency lacks uniformity and fails to align with best practices in proficiency testing.

Since the current literacy test is pegged at the “high beginning” level (NRS level 3), the intention is to design a new speaking test at the same level. According to NRS level descriptors, level 3 students “understand common words, simple phrases, and sentences containing familiar vocabulary” and “respond to simple questions about personal everyday activities...using simple phrases or short sentences.”

Teachers who are familiar with the BEST Plus test will recognize the format for the proposed speaking test. The applicant will be shown a color photograph of an ordinary event (related to everyday life topics such as weather, food, or daily activities) and instructed to: “tell me about this picture.” Applicants pass the test if they respond with relevant content vocabulary using simple words and phrases. Just as applicants are given three chances to perform correctly in the literacy test, for the speaking test, applicants will have three chances with three different photos on the new speaking test. Since level descriptors of low beginning-level students indicate only limited control of grammar, I presume that grammatical accuracy will not be a scoring consideration.

What’s tricky about a picture-based test is that people from different cultures can have very different interpretations of what they are seeing, and it may be difficult to find images that have universal meaning for a diverse range of applicants. It may also present challenges for applicants who have some degree of visual impairment, and we don’t have answers yet on how USCIS will handle such cases. However, the fact that applicants will be given three chances with three different pictures gives me hope that this test will turn out to be a fair and reliable measure of speaking skills. Once we see the scoring rubric and results of the trial testing with students like mine (older, low education, low English proficiency) we will have a better sense of whether this new format is an improvement.

The proposed civics test

Based on the information we’ve received, the content of the new civics test is likely to retain many of topics in the 100 questions we teach now. But USCIS has previously announced that the test needs a “refresh” every ten years or so, and is now telling us that (unnamed) subject matter experts within USCIS are developing new items, based on a design framework which will subsequently be reviewed by an external group of subject matter experts. It’s likely that there will be some shifting of the content to bring it up to date and perhaps even include history related to under-represented groups. The number of study questions will remain at 100, and we are told that applicants will be expected to answer six questions out of ten correctly, just as they do with the current test.

Where citizenship educators have major concerns with the proposed new test is the change in format. The current civics test is an open response oral test—the examiner asks a question, and the applicant must provide a spoken response. Our students rehearse the questions and answers as an oral exercise. USCIS proposes to shift to a written multiple-choice test, with no audio support provided. Each question will have one clearly correct answer option and three distractors (incorrect options). Vocabulary for the distractors will be taken from other parts of the study questions/answers to minimize the number of unfamiliar words that students will be confronted with.

We have pressed USCIS to explain the need for this change in format, and we keep getting the same response: “to align with best practices in the field of test development and to further standardize the naturalization test.” I take issue with this response because, to my knowledge, an oral open response test does align with current testing practices, and the administration and scoring of responses seems adequately standardized for all applications. While I have heard many complaints about unfair and disproportionate scoring of the current speaking test, the only concern I hear about the civics test is that occasionally students have trouble understanding a strong accent on the part of the examiner reading the questions. My guess is that the real motivation here is to make test administration more efficient—less time spent with a USCIS adjudicator during an interview.

So, what’s wrong with a written multiple-choice test? For the majority of citizenship applicants, it will be no problem at all. Younger, well-educated immigrants are quite familiar with multiple-choice questions and read well enough to demonstrate their knowledge of civics facts without a problem. But the older, low-literate, less proficient students in my program will suddenly find themselves in trouble with this format—regardless of how well they understand the 100 civics questions. Students who can currently demonstrate their knowledge of civics by listening to questions and reciting answers will now have to achieve a level of reading proficiency that far exceeds that demanded of the official reading test. My fellow textbook author, Bill Bliss, estimates that the new civics test will be two NRS levels higher than the actual reading test. In effect, applicants will be presented with both a beginning-level reading test and an advanced-level reading/civics test. For some, achieving this level of reading proficiency will require several additional years of study. I predict that many will give up on their quest for citizenship because they do not have the time or access to instruction that would allow them to become proficient readers.

The Citizenship Test Working Group has brought up this concern with USCIS and the response we got was that applicants generally prepare for the civics test using study materials that are written at a fairly high reading level. But what the test redesign team has failed to take into account is that many students successfully prepare for the civics test as a listening and speaking task, without reading any official study materials. We were also told that it is less demanding for test-takers to demonstrate receptive language skills (that is, recognizing the correct answer) than it is to demonstrate productive skills (remembering and producing the correct answer). It is true that recognizing written vocabulary (a receptive skill) is easier than writing it (production). And recognizing spoken vocabulary (a receptive skill) is easier than speaking the words (a productive skill). The trouble with the USCIS response is that it ignores the difference between written and oral skills. It suggests that receptive reading tasks are necessarily easier than productive speaking tasks. I don’t think many language acquisition experts would agree. A case in point: children are able to produce oral language for many years before they develop the skills to recognize the same words and phrases in written form.

Another challenge that the new test will present for less educated students is familiarity with multiple choice questions. Students in my literacy classes need explicit instruction to help them make sense of a written question presented together with four abstract written phrases. While the multiple-choice format is ubiquitous to folks with some degree of formal education, it does not come naturally to those confronting it for the first time, and if this new civics test format is implemented, students like mine will be spending a good deal of time learning test-taking strategies.

My recommendations

What do I think should happen with this test redesign effort?

First and foremost, USCIS must ensure that the push to standardize testing procedures and employ new technologies does not leave behind vulnerable populations.

I encourage USCIS to:

• enlist well-qualified adult ESL educators in their external review committee, including experts who have direct classroom experience with students at the lower ranges of English proficiency and students with limited formal education.
• ensure that the trial collects results from a wide range of students, including older students, those with limited education, and those with low levels of English proficiency. Note: USCIS has announced that it will be collecting data on the age range and ESL level of students in the trial testing, but it has not announced plans to collect data on their level of formal education.
• make a firm commitment to stakeholders that the new test will not have a negative effect on overall pass rates.
• proceed carefully with development of a picture-based speaking test.
• proceed with revisions to the civics test content to ensure that it is up to date, inclusive of underrepresented groups in US history and free of ideological bias.
• ensure that any new civics test questions are simply worded and are based on concepts that are accessible to students with limited formal education.
• either scrap plans for a written multiple-choice test or give applicants a choice of how they will demonstrate their understanding of the new civics questions: in written multiple-choice format or in the long-standing oral open response format.

As always, you can rely on me to keep you posted on any new developments I hear about regarding the test redesign project. Stay tuned!