New rating methodology for contests: segment-based approach Thread poster: Henry Dotterer
|
|---|
A number of the language pairs in the current translation contest have just been moved into the "qualification" phase. That means you can review and rate translations submitted by contestants in various pairs. (For a list of pairs in which that is possible, go to the contest page and scan the list of languages in the box to the right, looking for the word "Qualification" beside the language name.)
The "qualification" phase is the mode a pair enters when more than seven (7) entries h... See more A number of the language pairs in the current translation contest have just been moved into the "qualification" phase. That means you can review and rate translations submitted by contestants in various pairs. (For a list of pairs in which that is possible, go to the contest page and scan the list of languages in the box to the right, looking for the word "Qualification" beside the language name.)
The "qualification" phase is the mode a pair enters when more than seven (7) entries have been received. The idea in this mode of the contest is to allow peer rating, as a means of selecting which of the high number of entries will advance to a finals round.
Until now, qualification round rating has been carried out on an entry-by-entry basis. This time around an additional (optional) approach is available: rating on a segment-by-segment basis. Member José Henrique Lamensdorf expressed one of the intended benefits of this approach when he observed: "It becomes an interesting learning experience to compare how different translators handled the very same segment."
If you work in any of the pairs now in the qualification round, please give the new rating system a try. Your feedback on the approach, and suggestions for improvement, would be appreciated here. ▲ Collapse
| | | | | The segment-based approach | Oct 20, 2012 |
Hi all,
I strongly favour the segment-based approach; in fact I'm totally unable to evaluate/rate any entry as a whole, as none are 100% perfect: there are some in my language pair where 10 to 12 segments are translated absolutely brilliantly, and then there comes a huge blunder (either a wrong or poorly-worded translation, or a dissapointing typo).
BTW, how are we supposed to treat typos and misspellings? Is it a minus one point in 'quality' or 'accuracy'? ))
| | | | | Weaknesses of segment-based apporach | Oct 31, 2012 |
Hi,
I see at least two problems with rating segments instead of complete entries:
1. The average ratings can be skewed by people only rating good segments or only rating bad segments.
2. Someone with only one great idea for only one of the segments could theoretically reach the finals, even if their other translations were substandard. Example: In the recent Yogi-Berra-Contest my total ratings were based on the ability of the contestant to deal with all of ... See more Hi,
I see at least two problems with rating segments instead of complete entries:
1. The average ratings can be skewed by people only rating good segments or only rating bad segments.
2. Someone with only one great idea for only one of the segments could theoretically reach the finals, even if their other translations were substandard. Example: In the recent Yogi-Berra-Contest my total ratings were based on the ability of the contestant to deal with all of the about five really difficult quotes satisfactorily. A contestant having a great idea for only one of the quotes could get high ratings for only this and end up in the finals, even though they might have misunderstood a few of the other segments.
I see the benefit José mentioned ("It becomes an interesting learning experience to compare how different translators handled the very same segment"), but this (and also the group commentary function IMO) moves the contest in a different direction: From a competition enabling good translators to gain exposure to a general translation discussion board.
I propose to get rid of the segment-wise rating for future contests. ▲ Collapse
| | | | | Thanks, Dr. Matthias Schauen | Nov 1, 2012 |
Dr. Matthias Schauen wrote:
1. The average ratings can be skewed by people only rating good segments or only rating bad segments.
Insightful. I don't have a reason to believe that results have been skewed, but we do see that a non-trivial percentage of raters seem to be rating only the segments that they find particularly good or particularly bad. (Fortunately, enough people rate enough of the segments that we were able to require a certain minimum number of ratings as a basis for finalist selection... so maybe it doesn't matter. Still, it would stand to reason that a person who skips a segment has an opinion on that segment, and capturing that opinion could lead to more reliable selection. Any ideas?)
2. Someone with only one great idea for only one of the segments could theoretically reach the finals, even if their other translations were substandard.
That's not correct. The average for each entry is calculated first on a per-segment basis, and then the averages for all segments are averaged to get the overall average for the entry. So with the source text we used this time, for example, which had 14 segments, one segment can carry only 1/14th of the overall weight.
If anything, the argument could be made that some segments (the challenging ones, ie. those with a high "degree of difficulty") ought to be weighted more highly. We plan to experiment with that.
| | | | To report site rules violations or get help, contact a site moderator: You can also contact site staff by submitting a support request » New rating methodology for contests: segment-based approach | Wordfast Pro | Translation Memory Software for Any Platform
Exclusive discount for ProZ.com users!
Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value
Buy now! » |
| | TM-Town | Manage your TMs and Terms ... and boost your translation business
Are you ready for something fresh in the industry? TM-Town is a unique new site for you -- the freelance translator -- to store, manage and share translation memories (TMs) and glossaries...and potentially meet new clients on the basis of your prior work.
More info » |
|
| | | | X Sign in to your ProZ.com account... | | | | | |
Login to reply/comment 
