On Wed, May 18, 2016 at 7:29 PM, Nick Chase <nchase@mirantis.com> wrote:
We wrote about another suggestion here: https://www.mirantis.com/blog/fixing-openstack-summit-submission-process/ (Look after "... and here's mine".) I know there there some plusses and minuses in the interviews we did about it, but here it is:
Borodaenko suggests a more radical change. “I think we should go more in the direction used by the scientific community and more mature open source communities such as the linux kernel.” The process, he explained, works like this:
All submissions are made privately; they cannot be disclosed until after the selection process is over, so there’s no campaigning, and no biasing of the judges. The Peer Review panel is made up of a much larger number of people, and it’s known who they are, but not who reviewed what. So instead of 3 people reviewing all 300 submissions for a single track, you might have 20 people for each track, each of whom review a set of randomly selected submissions. So in this case, if each of those submissions was reviewed by 3 judges, that’s only 45 per person, rather than 300. Judges are randomly assigned proposals, which have all identifying information stripped out. The system will know not to give a judge a proposal from his/her own company. Judges score each proposal on content (is it an interesting topic?), fit for the conference (should we cover this topic at the Summit?), and presentation (does it look like it’s been well thought out and will be presented well?). These scores are used to determine which presentations get in. Proposal authors get back the scores, and the explanations. In an ideal world, authors have a chance to appeal and resubmit with improvements based on the comments to be rescored as in steps 3 and 4, but even if there’s not enough time for that, authors will have a better idea of why they did or didn’t get in, and can use that feedback to create better submissions for next time. Scores determine which proposals get in, potentially with a final step where a set of publicly known individuals reviews the top scorers to make sure that we don’t wind up with 100 sessions on the same topic, but still, the scores should be the final arbiters between whether one proposal or the other is accepted.
“So in the end,” he explained, “it’s about the content of the proposal, and not who you work for or who knows you or doesn’t know you. Scientific conferences have been doing it this way for many years, and it works very well.”
This has some characteristics that are congruent with my own proposal (random selection of a subset being offered to reviewers being one of them), which I obviously agree with. Others, not so much. - While I love the idea of giving detailed feedback and possibly even an option to refine and resubmit talks, even for "just" 45 talks that is a daunting task that would likely keep a reviewer occupied for a week, if the feedback is expected to be any good. - Anonymizing submissions is a double-edged sword. Yes, of course it removes all bias related to an individual speaker's person or company affiliation. However, when you talk to attendees of technical conferences, most people will much prefer to listen to a knowledgeable person who is also a good presenter, than to *the* best expert in their field who can't present worth a damn. Judging the quality of the oral presentation from the written submission is impossible (not least because there is no way of telling whether the submission was written by the presenter), and past speaker performance is a good indicator of future speaker performance. Anonymizing submissions means the latter is ignored, and the former will be attempted. If we determine the benefits to outweigh the downsides that's fine, but this downside should be considered. Then again, at the risk of repeating myself, why not take this to its logical conclusion and make the review "panel" identical to the group of submitters, in other words, with submitting a talk comes the responsibility to review others? This also has the added benefit that rather than divvying up talks with every talk being reviewed by just one (naturally flawed) reviewer, you get multiple eyeballs on each and every talk. Cheers, Florian