Automatic answer selection algorithm - please fix the discrepancy (Staff: done) विषय पोस्ट करनेवाला व्यक्ति: Katalin Horváth McClure
|
|---|
There are two cases when the "robot" would grade a KudoZ question.
Case 1: Asker leaves the question open, ignoring all reminders.
Here is the description of what happens then:
http://eng.proz.com/faq/2731#2731
An 'autograding robot' makes its selection based on the net number of peer 'agrees' received by each answer, according to the following rules:
- To be considered, an answer must have at least two "net agrees" (agrees-disagrees).
- The answer with the highest number of net agrees is selected. If two answers have the same number of net agrees, the last one entered is selected. The idea is that if two answers gather the same number of net agrees, the one that did so in a shorter period of time may be better.
Case 2: Asker wants to close the question within 24 hours, but blocked by the system's new feature, and chooses to "leave grading the the community".
Here is the description of what happens in this case:
http://eng.proz.com/faq/5449#5449
About 72 hours after this submission, the system will grade the question in accordance with the following criteria:
a) If none of the answers has a net peer grading (agrees minus disagrees) of 2 or more, then the option selected by the asker will be used.
b) If at least one answer has a net peer grading higher than 1 and one of the answers has a net peer grading higher than the others, then this answer will be selected.
c) If at least one answer has a net peer grading higher than 1 and two or more answers share the same top peer grading, the answer posted first will be selected.
Bold emphasis is mine in both quotes.
The bold text shows the difference in the algorithm used when two answers have the same net peer grading. There is no reason to have a different algorithm, there is no logic that supports this difference.
Case 1 is OK, I have a problem with the bold part in Case 2.
I think there is no perfect algorithm, (there can be various scenarios how two answers can end up with the same number of net agrees), but the algorithm used in Case 1 is logical, represents the most probable scenario (that is, all those agreed to the second answer have seen the first one and still voted for the second, therefore preferring the second answer, while those that voted for the first one may not have seen the second answer when voted) and it has already been used by the system for years.
I would like to ask to make the selection algorithm for Case 2 to be the same as Case 1, when choosing between two or more answers that share the same top peer grading. In other words, select the last answer entered.
Thank you
Katalin
[Edited at 2009-08-17 22:28 GMT]
[Subject edited by staff or moderator 2009-08-25 18:54 GMT]
| | | | | Good thinking | Aug 17, 2009 |
Katalin Horvath McClure wrote:
all those agreed to the second answer have seen the first one and still voted for the second, therefore preferring the second answer, while those that voted for the first one may not have seen the second answer when voted
Hi Katalin,
There is a logic behind this difference and it had to do with time spans (over two weeks in the first case, 72 hours in the second), but I find your statement above to be very logical.
Thanks,
Enrique
| | | | | Time span difference doesn't matter, IMHO | Aug 18, 2009 |
Enrique wrote:
There is a logic behind this difference and it had to do with time spans (over two weeks in the first case, 72 hours in the second)
I don't think it matters how long the time span is. In both cases, the answer posted later had less time until grading to collect the same amount of agrees.
(Besides, even if a question is open for 2 weeks, there is hardly any serious activity on them after 72 hours - in most cases activity ceases much earlier.)
I find your statement above to be very logical.
I am glad you find it logical. So, can we expect the algorithm change for Case 2?
Katalin
| | | | | Implemented, thanks for the suggestion | Aug 25, 2009 |
[quote]Katalin Horvath McClure wrote:
Enrique wrote:
I find your statement above to be very logical.
I am glad you find it logical. So, can we expect the algorithm change for Case 2?
Katalin
Thanks Katalin, the algorithm used for grading when a help question was pre-graded before 24 hours and also for GBK questions was modified as suggested. Now in all cases of robot-grading, when all other criteria are tied, the question posted last will be selected.
Regards,
Enrique
| | |
|
|
|
Sorry for not responding earlier - I was on vacation.
Thanks for the modification in the algorithm and also in the corresponding FAQs.
Katalin
| | | | To report site rules violations or get help, contact a site moderator: You can also contact site staff by submitting a support request » Automatic answer selection algorithm - please fix the discrepancy (Staff: done) | CafeTran Espresso | You've never met a CAT tool this clever!
Translate faster & easier, using a sophisticated CAT tool built by a translator / developer.
Accept jobs from clients who use Trados, MemoQ, Wordfast & major CAT tools.
Download and start using CafeTran Espresso -- for free
Buy now! » |
| | Pastey | Your smart companion app
Pastey is an innovative desktop application that bridges the gap between human expertise and artificial intelligence. With intuitive keyboard shortcuts, Pastey transforms your source text into AI-powered draft translations.
Find out more » |
|
| | | | X Sign in to your ProZ.com account... | | | | | |
Login to reply/comment 
