L2 Writing Assessment in the Greek School of Foreign Languages

Transcription

1 ISSN Jounl of Lnguge Teching nd Resech, Vol. 4, No. 4, pp , July 013 Mnufctued in Finlnd. doi: /jlt L Witing Assessment in the Geek School of Foeign Lnguges Despoin Pnou Deptment of Eduction, Univesity of Leiceste, UK Abstct In the pst two decdes, thee hs been n incesed inteest in the ssessment of L witing since the esults of such evlutions e used fo viety of dministtive, instuctionl nd esech puposes. One of the pimy issues petining the ssessment of witing qulity is the type of scoing pocedue which will be used; dmittedly subject of get del of esech nd discussion in the lnguge testing litetue. The im of the pesent ppe is fistly, to biefly descibe the type of scoing used in the Geek School of Foeign Lnguges fo ssessing L witing pefomnce nd secondly, to clculte the inte-te elibility of five witten smples. The esults obtined indicte quite high coeltions, thus demonstting the evlutos unifomity in the ppliction of ssessment citei. Index Tems L witing ssessment, scoing pocedue, inte-te elibility I. INTRODUCTION In the lst twenty yes thee hve been developments in testing eflected in the publictions now on the mket (Bchmn, 1990; Fulche, 010; Fulche nd Dvidson, 007; O Sullivn, 011) nd by the ivl of jounls such s Lnguge Testing nd Assessing Witing. Iespectively of the theoeticl ppoch dopted by ech schol, thee is common consensus tht testing is not situted solely within the clssoom but should be viewed s socil phenomenon, thus being plced in lge eductionl context. Guidnce on how to conduct ssessment nd to build good knowledge tests implies knowledge of the lge socio-cultul context of both lnguge lenes nd institutions. Of the mny issues involved in lnguge testing, nd in pticul, second-lnguge ssessment, elibility hs been of pticul concen to both lnguge testes nd teches. II. LITERATURE REVIEW A fundmentl concept in lnguge testing is elibility in the sense tht in ode fo test to chieve its intended pupose, the test esults must be elible (Fulche 1997, 1999). Accoding to Jones (1979) one fcto tht significntly influences elibility in pefomnce test is the consistency of tings. In fct, fou types of elibility hve been poposed: (1) inte-exmine elibility, () int-exmine elibility, (3) inte-te elibility, nd (4) int-te elibility.this section focuses on the issue of inte-te elibility. Accoding to McNm (1996) the issue of elibility tends to be moe complicted becuse of the subjectivity of humn tes in ssessing pticul test. In othe wods, no mtte how unequivocl the set of citei by which the test is to be judged, it could be the cse tht thee e diffeences o even disgeements in the tings llotted to the sme set of tests by goup of humn tes. In fct, Hmp-Lyons (1990) gues tht the elibility of ting lgely depends on the te s ttitudes nd conceptions. In this espect, inte-te elibility is ment to efe to the consistency between two o moe tes who ssess the sme lnguge test (Lombd et l. 005; Gmoff, 000). Thus, inte-te elibility enbles the tes to stnddize thei scoing nd to mintin check on the level of consistency of thei mking. It should be noted tht Tinsley nd Weiss (000, p.98) pefe the tem inte-te geement but hee, the tem inte-te elibility will be used efeing to its widely ccepted sense, tht is, the coeltion between two ting sets. In this sense, te-elibility is clculted by finding the coeltion between two o moe tes s will be subsequently discussed. III. METHODOLOGY Being Deptment of the Univesity of Athens, the School of Foeign Lnguges dhees to the egultions imposed by the Univesity. In pticul, the School of Foeign Lnguges cn only ccept dult lenes. Given tht, it is ssumed tht foeign lnguge lenes e, in pinciple, peceptully, cognitively, linguisticlly nd socilly mtued though thei L1. With espect to scoing pocedues, holistic ting is dopted bsed on numeicl scle nging fom Its ultimte im is to ssess the ovell poficiency level eflected in given smple of student witing nd it is genelly consideed elible scoing method, povided guidelines petining to te tining nd ting dministtion e fithfully dheed to. As Pekins (1983) gues holistic scoing is the most elible when the constuct ssessed is ovell witing poficiency (p.65). Holistic scoing ubics consist of 10 levels in the School of Foeign Lnguges

2 650 JOURNAL OF LANGUAGE TEACHING AND RESEARCH with mks 0- given to off topic essys, 3-4 to miniml evidence of poficiency, 5 to some evidence of poficiency, 6-7 to developing poficiency, 8 to poficient, 9 to vey poficient nd 10 to outstnding pefomnce. Along with the bovementioned scoing nge, the School of Foeign Lnguges mkes use of ubics in ode to nlyze students competency in cetin fetues of lnguge use nd composition skills, such s content, ogniztion, vocbuly, gmm, nd mechnics. As f s content is concened, ttention is dwn to to the following spects, nmely, () the min ide is dequtely developed, (b) elevnt suppoting detils e povided nd (c) thee is sufficient knowledge of the subject mtte. With espect to ogniztion, the following pmetes e tken into considetion: () thee is no choppy o bupt expession of ides, (b) thee is logicl sequencing, (c) cohesiveness, (d) cle nd succint ticultion of the min points nd suppoting detils, nd (e) ides e not pesented in confusing wy. Now tuning to vocbuly, teches e encouged to look fo: () coect nd effective use of wods in given context, (b) dvnce use of vocbuly, nd (c) ppopite use of egiste whees when it comes to gmm, coect use of tense, numbe, wod ode, ticles, ponouns, pepositions nd complex constuctions s well s bsence of gmmticl eos is highly vlued. Lstly, mechnics efe to the dheence of spelling, punctution nd cpitliztion conventions, coect use of pgphing nd legible hndwiting. IV. DATA ANALYSIS AND DISCUSSION Hving outlined the ubics on which L witing ssessment is bsed on in the Geek School of Foeign Lnguges, I will now poceed with the clcultion of the inte-te elibility of the witing test in question. In pticul, the following witing test ws given to 1 dult students, 5 mle nd 7 femle, of English poficiency level. Assessos wee sked to put emphsis on the use - coect o incoect - of the Futue tenses since the below topic invites the use of such tenses. Essy Topic Will ou lives be bette o wose in 50 yes fom now? ( wods). Fo the puposes of the pesent study five ndomly chosen essys mounting t lest 40% of the totl numbe of essys wee selected s epesenttive smple s cn be seen below: Smple essy of the fist student I think tht scientists will find cues fo mny diseses. Mybe the doctos will cue AIDS. With technology the food will be biologicl. So the food will be moe helth. Also, the quntity of food will be moe nd diffeent. Reseches estimete tht thee will be nothe.5 bilion in the plnet in fiftty yes o so. Thus, we will need to povide food fo them. It could be the cse tht new technologies govements will hve in ode to podoce foods. Moeove, the foods we et will hve diffeent fomt. Mny people tlk on tificil met, which looks nd feel like met but it is not met fom n niml but fom stem cells. We my even be eting tificil hmbuge in few yes fo ll we know. To mke things even moe compicte, we my even hve to expeient et insects, such s spides, wsps, woms, nts, gshopes nd beetles. Scientists will hve invented envionmentlly fiendly cs, so the pollution levels will hve decesed. The communiction will be vey simple though the intenet. We cn cll Ameic fom Geece chepe nd fste. Also though the Intenet will know ou eltionships. All these cn mke people lives longe. Howeve, the pessimistic people sid tht people life ws wost in 50 yes fom now. They sid tht noml cities become too cowded so we will live in cities unde the se. The wte thee will not be moe becuse we will destoy foests cycle. The humn eltionships will be gone becuse ll dy we will ply in intenet nd will not ed books. We will not wnt to meet othe people. We will be fid of meeting new people. We will not hve fiends nd ou life wil be miseble. We will wok vey hd in ode to mke ends meeting but ou qulity of ou live will not necessily be bette. The eltionships in ou wok evioment will be hostile nd vey competiton. We wil not be blle to tust ou colleggues becuse we will not believe tht they e sying the tyth. As esult, we will fel lonely nd isolte, nd even sometimes secluded fom the est of the wold. Smple essy of the second student I think people will be living longe becuse scientists will hve found cues fo mny diseses like cnce. Also, childen will be coming to the wold helthy becuse scientists will modify the genes. Futhemoe, I think in 50 yes fom now obbots will be doing homewok. In my view, the wy of eduction will be chnged lot nd childen will be studying though the intenet, due to will lose humn contct. School wil not hve the fomt tht hs it in 50 yes fom now. Thee will be no moe textbooks, ppe, pens, o pencils nd thee wil be e-book only fo students to ed. Thee wil bi no ned to hve teches since lessons will be mde though the intenet. In this wy, school will not exist but thee wil bi e-cls fo potentil students. The eduction system will hve dvnce significntly so s to bi ble to help people of l ge nd lening bilitis to knowledges cquie in evey filed of study. People will be ble to do esech nd univesities will open to eveyone. Public eduction wil not be dem byt elity. Howeve, ccoding to scientists, pollution levels in cities will incese nd noml cities will become too cowed. As esult people will constuct cities below se. On the othe hnd climte will be chnged ll ove the wold nd thee won t be infoests. Moeove, globl wm wil hve melted Anttic ice. In ddition to tht, people my be of dnge becse of depletion of the ozone lye. Deths fom i pollution will be up nd life of qulity will be down. Unless we chnge the wy we consume wte nd enegy, thee will be huge poblem to fce in the ne futue. Thee wil be

3 JOURNAL OF LANGUAGE TEACHING AND RESEARCH 651 getest demnd fo othe nd ext esouces becuse thee will be two bilion people moe in ou plnet in 50 yes fom now. How e we going to cope t such immense numbes? The nswe to tht qestion is not eisie one to give no stetfowd but I guess one fist step tht could mke things bette is people s weness of the dnges tht lie hed of us. Smple essy of the thid student I think tht in fifty yes fom now, the imge of ou plnet, will be quite diffeent of how we know it. Tying to conceptulize Eth in 50 yes fom now, I imgine people living in cities unde the se becuse noml cities will hve become too cowded. Also, I think tht pollution levels in cities will hve decesed since people will dive envionmentlly-fiendly cs. Cs my even dive themselves in smt highwts. Compnies will hve developed tetchnology to such n extent so tht c ccidents will hve being significntly educed. Moeove, cs will be smlle in size but moe effectively mnipulted since they will hve bins. In ddition to tht fuel economy will hve incesed s vehicles will be less thisty fo petol. It could be the cse tht we my hve the luxuy to tlk to ou electified c which will be be ble to do much moe thn just povide us with simple diving instuctions. Fo exmple, you my not need to len how to pk you c, becuse pking will hve become n utomted tsk inseted in the bin of you c. Futhemoe, the qulity of ou lives will significntly impove becuse scientists will hve found cues fo mny diseses. Howeve, no mtte how optimistic somebody is, we cnnot ignoe some seious emeging poblems s the yes go by. The scy thing bout ll this technologicl dvncement is tht people my not even hve the money to ffod c, let lone high-tech c. Moeove, people will be too concened with how mny Fcebook fiends they hve, insted of ceting new fiens nd socilizing in el-life contexts insted of intenet ones. Vlues such s tuth, fiendship, love nd feedom will hve cquied diffeent mening nd nothing will be the sme in the lives of young people who stuggle to find thei wy in thei society. Moeove, I cnnot stop thinking tht we will hve bunt ll the foests to constuct moe nd moe cities. Ou food will hve diffeent fomt nd genelly ou plnet will lose its ntul hbitt nd will become moe tificil. I believe tht we must show ou love to ou plnet s it is, befoe it is too lte. Qulity of life is not solely ssocited with mteil goods but thee e othe things, much simple nd down to eth, tht cn help us impove ou lives. We e the ones tt decide wht is impotnt fo us. A smile, fiendly hndshke, pomise tht is kept nd smll fvou tht is mde e only few to mention. Smple essy of the fouth student I think tht in 50 yes fom now mny things will bi chenge, s the exmple the wy of living, will bi diffeent. Nowdys cities will bi becoming too cowded, young people will leve fom the smll town nd sech fo bette live will go to big cities. The eltionships lso wil chenge dmticly becuose the technologiy nd the fst hythms won t help, they won t hve fee time fo seinig nd meeting new fiends. Evey kind of contct wil bi mde though the Intenet. Fcebook will hve diffeent fomt nd thee will be no need to meet someone outside. Evey fom of socilizin wil bi mde though Fcebook. People wil bi isoleted nsd sceed nd thee wil bi no qulity in eltionship becuse thee wil not be ny el-life eltionship mong people. We will become of ou fellowmtes even moe suspicious nd wil be fid to communicte to othe people becuse we wil not know thei intentions. As esult, we will spend moe time lone wtching T.V. nd doing nothing poductive. Howeve, I believe tht scieintists fiend cues fo mny diseses, so people wil live longe. Also, scieintists invent envionmentl fiendly cs, thing tht will help to keep the cities clenest. Cs will hve bin. We will not hve to divi ous c becuse cs will be ble to dive themselves. We will not hve to len how to pk ou c becuse cs wil be utomted nd will do the pking by themselvs. Also, we will not need to hve ny fuel becuse cs will be electicl. Thee will be no c ccidents becuse people will hve become moe skiled dives. They will be moe ceful nd concened bout ohe people s lives. They will tke get ce with pedestens nd pets s well. Smple essy of the fifth student I think tht in 50 yes fom now, people will live in cities unde the se, s noml cities will hve become too cowded. In ddition to tht, they will hve invented envionmentlly-fiendly cs, so pollution levels in cities will hve decesed. Moe impotntly, people will hve bette qulity of life becuse mchines will do ll the hd wok since technology will hve significntly impoved. It my be the cse tht computes will outpefom humn beings. Of couse, computes hve long wy to go to mtch humn stenghts but consideble pogess hs been mde in the pocessing powe nd memoy cpcity of the computes. Humnlike obots will hve pobbly ppeed, thus implementing new wy of life. Computes my even outpefom humn beings in most of thei ctivities. Advncing compute pefomnce will incese stedily nd mchines will begin to do well in vious es. In ddition to tht, I think tht people will live longe in 50 yes fom now becuse scientists will hve found cues fo mny diseses. In pticul, scientists will be ble to do expeiments with humn embyonic stem cells nd find cues fo mny seious medicl conditions such s cnce. Stem cell-bsed humn thepies will hve become outine pocess nd people will not be toubled by illnesses tht seemed to hve the cost the lives of mny people in the pst. Howeve, I lso believe tht thee will be no moe infoests becuse we will hve destoyed them ll. Even though foests still cove 35 pecent of the wold s lnd e, the qulity of lnd my hve suffeed such sevee dmge in the futue so tht thee will be no foests. The most dmtic impct of this defoesttion will be the loss of hbitt fo thousnds of species. Anothe side-effect will be the climte chnge. Foest soils will quickly dy out if they e not

4 65 JOURNAL OF LANGUAGE TEACHING AND RESEARCH potected by tees nd foest lnds will be tnsfomed into desets. Futhemoe, people will hve become isolted since the only fom of communiction will be the intenet. The end-esult will be mny socilly-isolted people tht e fid to go out in the public nd ttemp socil intections. Consequently, they will become emotionlly-isolted people tht my hve psychologicl dissodes leding to n unhppy nd miseble wy of life. In spite of the bove shotcomings, I tend to believe tht ou lives ll in ll will be bette in 50 yes fom now fo the simple eson tht we will know how to hndle good nd bd things in bette wy. The foementioned 5 witten smples wee ssessed by five English lnguge instuctos, teching in the School of Foeign Lnguges. All of them hd B.A. in English Lnguge nd Litetue fom the Univesity of Athens nd thee of them hd n M.A. in Applied Linguistics fom U.K. Univesity whees the othe two hd n M.A in TESOL gin fom U.K. Univesity. It should be mentioned tht the tes wee explicitly instucted not to signl mistkes in the essy ppes to be mked. Hence, thee wee no identifying mks on the ppes in ode to ensue the mximum objectivity by leving the tes completely uninfluenced. The esults obtined e shown in Tble 1 below: TABLE 1: ASSESSMENT OF FIVE WRITTEN SAMPLES Rte 1 Rte Rte 3 Rte 4 Rte 5 Tget Student Tget Student Tget Student Tget Student Tget Student To hve confidence in the tings, we need infomtion on inte-te elibility. Since the students finl scoe is the combintion o vege of the tings, elibility depends on the numbe of tes. Accoding to Htch nd Lzton (1991) to compute inte-te elibility fo moe thn two tes, ll the ting (poducing Peson coeltion mtix) must be fistly coelted nd then n vege of ll the coeltion coefficients must be deived (p.533). Given the bove, to clculte the inte-te elibility fo set of tings mde by 5 diffeent tes we must fistly clculte ten coeltions: the coeltions between Rte 1 nd Rte, Rte 1 nd Rte 3, Rte 1 nd Rte 4, Rte 1 nd Rte 5, Rte nd Rte 3, Rte nd Rte 4, Rte nd Rte 5, Rte 3 nd Rte 4, Rte 3 nd Rte 5, Rte 4 nd Rte 5. The genel fomul fo clculting the coeltion coefficient is the Peson Poduct-moment coeltion: ( ) / n xy / ( ) / n Y ( Y ) n whee X epesents the tings of the fist te, Y epesents the ting of the second te, nd n equls the numbe of tget students ted by ech te (5 in ou cse). The symbol Σ indictes tht you need to sum the elevnt vlues. Thus, ΣΧ indictes tht you need to sum the vlues of X, nd ΣΧΥ indictes tht you need to sum the vlues of X multiplied by Y (Mclennn, 1993). Hence, by substituting these specific tings into the genel fomul fo coeltion, we come up with the following Peson coeltion mtix: TABLE : PEARSON CORRELATION FOR RATERS R1 R R3 R4 R5 R R R R4.54 R5 Now, using these coeltions, we cn clculte inte-te elibility using the Spemn-Bown fomul: nab 1 ( n 1) AB whee b is equl to the vege coeltion between the tings mde by ech te. The vege fo these coeltions would be: The inte-te elibility is: 7.43 b 10 b.743

5 JOURNAL OF LANGUAGE TEACHING AND RESEARCH 653 (5)(.743) 1 (5 1)(.743) (4)(.743) Thus, the ovell inte-te geement fo the five mkes is.93. Accoding to Lndis nd Koch (1977) vlues less thn 0.00 indicte poo geement, vlues between 0.00 nd 0.0 indicte slight geement, vlues between 0.1 nd 0.40 indicte fi geement, vlues between 0.41 nd 0.60 indicte modete geement, vlues between 0.61 nd 0.80 indicte substntil geement, nd vlues between 0.81 nd 1.00 indicte lmost-pefect geement. Thus, vlue of 0.80 nd bove is lmost pefect. Consequently, this is vey stisfctoy vlue which leds us to infe tht the elibility is vey high s well s the level of te geement. The min fctos tht hve contibuted to the test s elibility e fist nd foemost, the fct tht the im of the witing ssessment ws stightfowd, nmely, judges wee sked to ssess students use of Futue tenses, nd secondly, thee ws cle nd economicl scoing scle s well s ubics to sfely guide ssessos in thei mking of the essys in question. Thus, clity of instuctions nd cle, unmbiguous mking scheme e essentil steps in ioning out diffeences in teche intepettion of the ssessment citei pplied to given test. V. CONCLUSION In this ppe, n ttempt ws mde to clculte the inte-te elibility of 5 witten smples of poficiency-level students ttending the Geek School of Foeign Lnguges. Results hve shown vey stisfctoy level of inte-te geement between the five judges. Given these esults, it cn be concluded, thn on bode esech font, simil, longe-scle studies cn offe useful insights into the witing constuct, with the identifiction nd development of citei which e both useful nd elevnt fo the ssessment of both L1 nd L witing. REFERENCES [1] Bchmn, L. F. (1990). Fundmentl Considetions in Lnguge Testing. Oxfod: Oxfod Univesity Pess. [] Fulche, G. (010). Pcticl Lnguge Testing. London: Hodde Eduction. [3] Fulche, G. nd Dvidson, F. (007). Lnguge Testing nd Assessment. London nd New Yok: Routledge. [4] Fulche, G. (1999). Assessment in English fo cdemic puposes: putting content vlidity in its plce. Applied Linguistics 0 (), [5] Fulche, G. (1997). Assessing witing. In Fulche, G. (ed.) Witing in the English Lnguge Clssoom (pp ). Pentice Hll Euope in ssocition with the Bitish Council. ELT Review Seies. [6] Gmoff, R. (000). Rte elibility in lnguge ssessment: the bug of ll bes. System 8, [7] Hmp-Lyons, L. (1990). Second lnguge witing: ssessment issues. In Koll, B. (ed.) Second Lnguge Witing (pp.69-87). Cmbidge: Cmbidge Univesity Pess. [8] Jones, R. (1979). Pefomnce testing of second lnguge poficiency. In Biee, E. nd Hinofotis, F. (eds.) Concepts in Lnguge Testing (pp ). Wshington, DC: TESOL. [9] Htch, E. nd Lzton, A. (1991). The Resech Mnul: Design nd Sttistics fo Applied Linguistics. Boston, Msschusetts: Heinle & Heinle Publishes. [10] Lndis, J.R. nd Koch, G.G. (1977). The mesuement of obseve geement fo ctegoicl dt. Biometics 33 (1), [11] Lombd, M., Snyde-Duch, J. nd Cmpnell-Bcken, C. (004) Pcticl esouces fo ssessing nd epoting intecode elibility in content nlysis esech pojects. Retieved on Septembe 3, 01 fom: [1] Mclennn, R.N. (1993). Inte-te elibility with SPSS fo windows 5.0. The Ameicn Sttisticin 47 (4), [13] McNm, T. (1996). Mesuing Second Lnguge Pefomnce. London: Longmn. [14] O Sllivn, B. (011). Lnguge Testing: Theoies nd Pctices. Bsingstoke: Plgve Mcmilln. [15] Pekins, K. (1983). On the use of composition scoing techniques, objective mesues, nd objective tests to evlute ESL witing bility. TESOL Qutely 17, [16] Tinsley, H.E.A. nd Weiss, D.J. (000). Intete elibility nd geement. In Tinsley, H.E.A. nd Bown, S. D. (eds.) Hndbook of Applied Multivite Sttistics nd Mthemticl Modeling. (pp ). Sn Diego, CA: Acdemic Pess.

6 654 JOURNAL OF LANGUAGE TEACHING AND RESEARCH Despoin Pnou is Ph.D. student t the Univesity of Leceste. She hs B.A. in English Lnguge nd Litetue (with distinction) nd n M.A. in Tnsltion-Tnsltology (with distinction) fom the Univesity of Athens. She lso hs n M.A. in Linguistics-TESOL (with meit) fom the Univesity of Suey. He esech inteests e foeign lnguge teching, tnsltion, nd the use of figutive/idiomtic lnguge in business discouse.