Assignment 2
Fairness

DEAKIN UNIVERSITY

ECL 776 - Language Testing and Assessment
Unit Chair: Dr. Rod Neilson

Adam Scott
500148222

Introduction

Testing is a fundamental aspect of human nature and society. Humans have used tests for psychological and physical inclusion since history began. The importance is evident in early literature such as Homeric text where allusions to testing are personified as gods challenging man. Undoubtedly, tests in those times had their own high-stakes nature, perhaps being the life of an individual, and while this may seem extreme and unfair in the twenty-first century, in its context was perhaps necessary to ensure the survival of a group. This protectionism no doubt persists today, evident in social policy and language norms. Although the stakes have changed, entry to a society is no less high contextually in terms of financial and emotional currency. Now as the dominant species the survival of the fittest no longer seems relevant, so attention must be drawn to current gate-keeping practices and not only their necessity but also whether or not they are fair. While language testing has become standard practice in many countries it has also become politicized especially as a means of power (Hamp-Lyons, 1997a; Shohamy, 2001). In this case the term fairness becomes problematic because determining whether it is society or the political class who has set the standards has many implications. Language testing systems employed by governments may be relevant to determine types of language skills and their validity is certainly justified in some circumstances such as entry to an industry, but it does not mean that test takers are prepared for the communicative reality of a society. Consequently, testing can be unfair for both test taskers and for the community in which they hope to enter and perhaps unethical for those who have the power to employ them. This essay will discuss current issues relating to fairness in language testing. First, key concerns will be explored from different perspectives before examining the Pearson PTE Academic Test. While I believe testing is necessary I will argue that the employment of the PTE challenges some of the assumptions regarding fairness especially in its recent uptake by the Australian government for migration purposes. Concluding, I will comment about the future of testing and fairness as determined by society.

Part One: Key concerns of Fairness

Lack of fairness in the testing of language is by no means a new issue (Shakouri & Sheikhy, 2014). It is a social construct and with the values of society in a constant state of flux and varying across regions it is a difficult term to define (Fulcher & Davidson, 2007). In part one I will unpack some of the major issues concerning fairness. After establishing links to validity, the social dimension of fairness will be discussed in terms of ethics and how this relates to professionalism, power and the voices of the underrepresented.

The centrality of validity to language testing has long since been determined (e.g. Bachman and Palmer, 1996; Chapelle, 1998; Cronbach & Meehl, 1955; Messick, 1989). Jin (2011) outlines the relationship between validity and fairness, with many in the testing community debating whether it is a separate quality or rather the same thing but further along the continuum. Nonetheless it is agreed that a test that is not valid cannot be fair as much as a test has to fair to be valid (Jin, 2011; Shakouri & Sheikhy, 2014). From this point of view fairness subsumes a central role in a language testing.

The importance of ethics and its position within society cannot be overstated. Socially, unbiased and fair tests must be created for ethical reasons (Shakouri & Sheikhy, 2014). The core issue of an ethical approach is professionalism, a concept devised by Davies (1997a, cited in Fulcher & Davidson, 2007). To guide fairness and ethical practice a framework must be established. These are an essential point of reference for test writers and other stakeholders and are critical for accountability. There have been many standards, Fulcher and Davidson (2007) highlight the AERA (1999) as the most frequently cited. Despite the standards allowing testers to frame their work within professionalism, Davies (2003, cited in Jin) points out the inevitable politicization as tests are used to select and sort to meet society’s purpose. In effect tests have the power to be extremely unfair which raises the issue of relativism or philosophical views that shape policy and justify a position. A study with the language testing community (Fulcher & Davidson) demonstrated the significant variability of the understanding of the standards, which suggests that standards can be manipulated to some degree. Nonetheless governments have a responsibility but how fair their application of language testing is depends on one’s point of view. Not only is fairness seriously questioned if governments or other administrative bodies use a test beyond its original purpose but also the ethical implications. Beyond industry and beyond the test it becomes a matter of trust, for example in Australia with the government as gatekeepers, that they uphold standards and attempt to work as ethically as possible on behalf of the community.

Test takers are the central stakeholders in testing yet they are the most underrepresented voice considering their polar position to the governments and institutes that test language. Of course ethical practice must involve listening to and considering contradicting views and new evidence (Shakouri & Sheikhy, 2014). In a study by Puspawati (2014) it was found that test takers identify topics and task as unfair, especially field-of-study and culture specific items. This has serious implications on a majority of major high-stakes test especially if the new direction for fairness involves incorporating the view of test takers to a far greater extent than what is currently done (Puspawati, 2014).

From the teachers perspective it is difficult to ascertain how they can influence or maintain levels of fairness. Obviously they are accountable to the students of test training to provide them with the information necessary to complete the tasks. East (2014) considers teachers are uniquely placed to provide essential information on test validity and fairness. Unquestionably teachers are the bridge between test takers and the society into which entry is desired. In positions of power as the gate keepers of knowledge teachers must also look to ethical practice and professionalism to ensure pedagogy is implemented to provide fair training and opportunity for all students to receive the best training opportunities.

Overall, it can be seen here how difficult it is to define fairness. With links to validity and ethics fairness treads the line between the humanistic and technical. As a social construct, fairness may be undermined by power ideologies yet at the same time testing could be justified as tool for cultural stability (Fulcher & Davidson, 2007).

Part Two: Discussion

The Pearson PTE Academic is a computer based test that is machine scored designed for testing university entrants. Pearson claims that it is one of if not the most reliable tests because of its scoring system. For example one hundred and twenty different languages from one hundred and fifty eight countries comprise the ten thousand responses that optimize the speech recognizer, so national accents are treated equally. Similar optimal work has been invested in developing the writing response bank. In this way the test aims at attainment of reliability and fairness. The integrated test items reflect skills required for academic life, including such tasks as note taking, retelling lectures, summarizing and essay writing. As most universities have or are transitioning to online or flipped classrooms, the PTE online format is purposefully valid according to Messick’s (1989) construct validity. In addition test training would also provide adequate preparation for studying at university and does not raise any significant washback concerns. The only bias it pertains to have is some American English idioms as well as American college terminology which may differ from other higher education systems throughout the English speaking world. These threaten validity, especially concerning fill-in-the-blanks, short answer and some listening items. Despite this the test in general stands up to validation.

The PTE has been accepted by Australian institutes for some time but concerning migration IELTS has monopolized the field for a long time. However this situation has recently changed with PTE becoming available as an option for migration. It is this junction where fairness and validity are immediately compromised. It confirms the first of Bishop’s (cited Fulcher & Davidson, 2007) ethical issues when the work of test developers is being used for other purpose beyond their original intent. The implications of washback for teachers, professionalism and ethics for writers, expectations of society and most importantly unfairness for test takers are prevalent. The strength of its academic format is actually what makes it unfair for numerous reasons when considering its usage for migration. While there is an American bias this is subordinate to the higher education sub-culture. A proportion of test takers have never been to university and never intend on doing so, consequently they are unfairly disadvantaged (Wagner, 2006). Additionally, the academic tasks are to be completed on a keyboard which also should draw criticism. Young test takers that are computer literate with academic experience are at a distinct advantage. While the test designers have developed it for English skills within an academic context the Australian government is using it for general English purposes. Consequently this unfairness is also a threat to validity as the test is not only restrictive but also does not represent the ability and content domain that it was designed to assess (Wagner).

Regarding training programs they move away from communicative competence which was at the basis of early Cambridge tests. Theoretically PTE training can be done by one-self online, further removing the test taker from the socio linguistic aspects. The IELTS for example at least involves human interaction in the speaking test, requiring the test taker to interact with another person. Training programs for PTE focuses on a small niche therefore any training a student undertakes leaves them no more prepared for society and developing communicative competency. Contrary to Shohomy’s (2004) notion that tests should maintain, preserve and represent the knowledge of diverse groups by screening for young computer literate migrants the government is reinforcing power ideologies by allowing entry to those with distinct literacy abilities. Essentially these are the new requirements for the 21st century accepting those that will not be a liability but rather offer economic potential to the workforce. No doubt Pearson is complicit in this process with substantial economic gains to be made with the employment of the PTE. McNamara (2000) highlights the expanded responsibility test developers must have especially taking responsibility for the effects of the test. Pearson undermines its ethical positioning by disregarding its own writers’ sense of professionalism as well as fairness for test takers.

Within the test taking community in the Australian context, the appearance of the PTE seems a panacea to the ills of acquiring the much desired score. A recently perceived notion among test taker is that PTE is easier. That may be the case for those with the academic literacy skills but for those who do not possess these skills it may only prolong the economic and emotional burden of the testing taking cycle. I have seen numerous students switch from IELTS training to PTE and back again once realizing the format of the test. Furthermore because PTE training is specific they are generally no better prepared for the IELTS. In my opinion this creates a new type of power within the test training centers furthering their symbolic role as gate keepers of knowledge.

In conclusion

It can be seen that the introduction of the PTE Academic test as a tool for migration in Australia raises serious ethical questions for those in positions of power, the owners of the test and those employing it. It is clear that fairness is a significant link between beliefs in society and language testing. Australian history is rooted in immigration yet the nature of it remains divisive in the community. The standards that society must hold the governments accountable to should be made much more transparent and test designers must acknowledge that despite their best intentions their tests will undoubtedly be used as tools of power. With this in mind features that make the test fairer need to be incorporated in the instrument. As the digital revolution takes a firm hold, it widens the gap between countries that have had the capacity to implement its full use and those that are yet to do so. Technology also heightens the generational divide. Both these concerns must be addressed because until the divide is bridged as users of technology will have a distinct advantage especially if computer based language testing continues to proliferate. Fairness will continue to be debated as relative positioning and standards jostle for prominence between the different stakeholders who have vested in language testing. The future of fairness must consider the views of the test takers yet whether test designers or those who implement tests take note of these views will remain to be seen and only if those in power are willing to challenge their own philosophical viewpoints and consider relinquishing some level of power. References

Bachman, L., & Palmer, A. (1996). language testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University press

Chapelle, C. (1998) ‘Construct definition and validity inquiry in SLA research.’ In Bachman, L.F and Cohen, A.D. (eds). Interfaces between Second Language Acquisition and Language Testing Research (pp. 32-70). Cambridge: Cambridge University Press.

Cronbach, L.J. & Meehl, P.E. (1955). ‘Construct validity in psychological tests.’ Psychological Bulletin 52, 281-302.

East, M. (2014). Coming to terms with innovative high-stakes assessment practice: Teachers’ viewpoints on assessment reform. Language Testing, 32 (1), 101 - 120

Hamp-Lyons, L. (1997a). Ethics in language testing. In Clapham, C. and Corson, D. (eds.) Encyclopedia of Language and Education, Vol. 7: Language Testing and Assessment. Dordrecht: Kluwer Academic Publishers, 323-333.

Jin, Y. (2011). Fundamental concerns in high-stakes language testing: The case of the College English Test. Pan-Pacific Association of Applied Linguistics 15(2), 71-83

McNamara, T. (1998). Policy and social considerations in language assessment. Annual Review of Applied Linguistics, 18, 304–319.

McNamara, T. (2000). Language Testing. Oxford: Oxford university press.

Fulcher, G. and Davidson, F. (2007). Language Testing and Assessment: an advanced resource book. New York: Routledge.

Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18 (2), 5-11

Puspawati, I. (2014). Fairness issues in a standardized English test for nonnative speakers of English. TESOL Journal 5 (3), 555-572

Shohamy, E. (2001). Symbols and ideologies. The power of tests: A critical perspective on the uses of language tests Harlow, UK: Pearson Education, 117–128.

Shohamy, E. (2004). Assessment in multicultural societies: Applying democratic principles and practices to language testing. In B, Norton & K Toohey (eds.), Critical Pedagogies and Language Learning. Cambridge: Cambridge University Press, 72-92.

Shoukri, N and Sheikhy, R. (2014). Revisiting ethics in language testing system: An ignis fatuus! Modern Journal of Language Teaching Methods, 4 (4), 332-337

Wagner, E. (2006). Can the search for fairness be taken too far? Teachers College, Columbia University Working Papers in TESOL & Applied Linguistics 6, (2), 1-4