[openstack-community] OpenStack Summit proposal voting - not a fan
Hi all, Rather than just complain into the ether, I wanted to let people know why I don't like the voting process for conference proposals and see if I'm the only one. I don't think that the voting process is the best way to gauge whether proposals will be good for the conference. There are a few reasons for that: * Having to hawk & promote proposal(s) is kind of unseemly, and makes us look small, I think. Hundreds of people going "vote for me!" doesn't make us look good. * Some people don't want to pitch themselves, others don't have access to as big a platform to promote * The same issues exist with this system which exist with board voting - there is a possibility that people will vote for their colleagues, not out of any corruption, but just because no-one has time to rate all the proposals, and they're more likely to rate those submitted by people they know more highly * Also, it's a self-selecting group of people who rate proposals - I don't think voters will be representative of summit attendees * After all is said and done, the proposals which are chosen by the voters are guidelines to the people who choose the talks for the tracks, the track leaders I have been a track leader for the last number of summits, and I've seen first hand great presentations get very low numbers of votes, while others which are not as interesting get very high numbers of votes and high ratings. Personally, I would be happy if we could change the system to remove the "pimp my talk" aspect for Summits. Cheers, Dave. -- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13
Lots of conferences do this to boost interest and engage with the community.... but you're absolutely right. A good compromise would be to give track leaders some magic votes to boost certain talks and veto others. One way to implement this would be to allow tracks to have their own algorithms. Openstack is too broad to have one method for all the talks and I think it's totally reasonable to have a default method which track chairs can override. -Adam -- Kili - Cloud for Africa: kili.io Musings: twitter.com/varud <https://twitter.com/varud> More Musings: varud.com About Adam: www.linkedin.com/in/adamcnelson On Thu, Feb 20, 2014 at 6:51 PM, Dave Neary <dneary@redhat.com> wrote:
Hi all,
Rather than just complain into the ether, I wanted to let people know why I don't like the voting process for conference proposals and see if I'm the only one.
I don't think that the voting process is the best way to gauge whether proposals will be good for the conference. There are a few reasons for that:
* Having to hawk & promote proposal(s) is kind of unseemly, and makes us look small, I think. Hundreds of people going "vote for me!" doesn't make us look good. * Some people don't want to pitch themselves, others don't have access to as big a platform to promote * The same issues exist with this system which exist with board voting - there is a possibility that people will vote for their colleagues, not out of any corruption, but just because no-one has time to rate all the proposals, and they're more likely to rate those submitted by people they know more highly * Also, it's a self-selecting group of people who rate proposals - I don't think voters will be representative of summit attendees * After all is said and done, the proposals which are chosen by the voters are guidelines to the people who choose the talks for the tracks, the track leaders
I have been a track leader for the last number of summits, and I've seen first hand great presentations get very low numbers of votes, while others which are not as interesting get very high numbers of votes and high ratings.
Personally, I would be happy if we could change the system to remove the "pimp my talk" aspect for Summits.
Cheers, Dave.
-- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
I tend to agree. Perhaps voting could be limited to those that have RSVPed and are actually going to the Summit as well? In an effort to do my best and look at all the submissions, I voted for a combined several hours in a few goes yesterday and still haven't reached any kind of end. It did seem like presentations came past multiple times and the volume of talks with a tenuous link to OpenStack made it a challenge. There were also talks in there that were submitted for HKG that had NOT been submitted to ATL. I confirmed this with a couple of submitters. It also seemed like some folks submit several talks that are almost the same, and some people just submit too many talks. Perhaps a one or two talk per person limit. Or one talk and one panel. This time I do hope we don't see the combination of talks like we had in HKG which I thought was a good idea as a track chair beforehand, but when I went and saw them on some occasions I felt for the speakers. Am going to have another crack at finishing my voting today. Wish me luck :-P *From:* Adam Nelson [mailto:adam@varud.com] *Sent:* Friday, 21 February 2014 4:04 AM *To:* Dave Neary *Cc:* community@lists.openstack.org *Subject:* Re: [openstack-community] OpenStack Summit proposal voting - not a fan Lots of conferences do this to boost interest and engage with the community.... but you're absolutely right. A good compromise would be to give track leaders some magic votes to boost certain talks and veto others. One way to implement this would be to allow tracks to have their own algorithms. Openstack is too broad to have one method for all the talks and I think it's totally reasonable to have a default method which track chairs can override. -Adam -- Kili - Cloud for Africa: kili.io Musings: twitter.com/varud <https://twitter.com/varud> More Musings: varud.com About Adam: www.linkedin.com/in/adamcnelson On Thu, Feb 20, 2014 at 6:51 PM, Dave Neary <dneary@redhat.com> wrote: Hi all, Rather than just complain into the ether, I wanted to let people know why I don't like the voting process for conference proposals and see if I'm the only one. I don't think that the voting process is the best way to gauge whether proposals will be good for the conference. There are a few reasons for that: * Having to hawk & promote proposal(s) is kind of unseemly, and makes us look small, I think. Hundreds of people going "vote for me!" doesn't make us look good. * Some people don't want to pitch themselves, others don't have access to as big a platform to promote * The same issues exist with this system which exist with board voting - there is a possibility that people will vote for their colleagues, not out of any corruption, but just because no-one has time to rate all the proposals, and they're more likely to rate those submitted by people they know more highly * Also, it's a self-selecting group of people who rate proposals - I don't think voters will be representative of summit attendees * After all is said and done, the proposals which are chosen by the voters are guidelines to the people who choose the talks for the tracks, the track leaders I have been a track leader for the last number of summits, and I've seen first hand great presentations get very low numbers of votes, while others which are not as interesting get very high numbers of votes and high ratings. Personally, I would be happy if we could change the system to remove the "pimp my talk" aspect for Summits. Cheers, Dave. -- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13 _______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Personally, I found the wide range of talks very interesting and there are several that I would be following up on even if they are not selected for the summit. Thus, a full list of talk proposals is valuable. However, the track chairs should be able to balance the vote with their knowledge of the community. The vote should only be advisory rather than mechanical. Selecting the top votes risks being dominated by the current topic du jour rather than a balance of interest of the community. I would not be in favour of limiting the vote to those attending since the videos of the talks are much more widely viewed (and helps for multiple parallel bookings for summit attendees) Overall, I think the importance is that the track chairs feel empowered to choose the appropriate balance of talks for the tracks. There is always a risk of selection bias from the track chairs so there should also be a reasonable turnover in those roles too and a feedback loop for attendees to rate the tracks for interest. How do other big conferences organise this ? We can't be the only ones with over subscribed summits. Tim From: Tristan Goode [mailto:tristan@aptira.com] Sent: 20 February 2014 22:06 To: Adam Nelson; Dave Neary Cc: community@lists.openstack.org Subject: Re: [openstack-community] OpenStack Summit proposal voting - not a fan I tend to agree. Perhaps voting could be limited to those that have RSVPed and are actually going to the Summit as well? In an effort to do my best and look at all the submissions, I voted for a combined several hours in a few goes yesterday and still haven't reached any kind of end. It did seem like presentations came past multiple times and the volume of talks with a tenuous link to OpenStack made it a challenge. There were also talks in there that were submitted for HKG that had NOT been submitted to ATL. I confirmed this with a couple of submitters. It also seemed like some folks submit several talks that are almost the same, and some people just submit too many talks. Perhaps a one or two talk per person limit. Or one talk and one panel. This time I do hope we don't see the combination of talks like we had in HKG which I thought was a good idea as a track chair beforehand, but when I went and saw them on some occasions I felt for the speakers. Am going to have another crack at finishing my voting today. Wish me luck :-P From: Adam Nelson [mailto:adam@varud.com<mailto:adam@varud.com>] Sent: Friday, 21 February 2014 4:04 AM To: Dave Neary Cc: community@lists.openstack.org<mailto:community@lists.openstack.org> Subject: Re: [openstack-community] OpenStack Summit proposal voting - not a fan Lots of conferences do this to boost interest and engage with the community.... but you're absolutely right. A good compromise would be to give track leaders some magic votes to boost certain talks and veto others. One way to implement this would be to allow tracks to have their own algorithms. Openstack is too broad to have one method for all the talks and I think it's totally reasonable to have a default method which track chairs can override. -Adam -- Kili - Cloud for Africa: kili.io<http://kili.io/> Musings: twitter.com/varud<https://twitter.com/varud> More Musings: varud.com<http://varud.com> About Adam: www.linkedin.com/in/adamcnelson<https://www.linkedin.com/in/adamcnelson> On Thu, Feb 20, 2014 at 6:51 PM, Dave Neary <dneary@redhat.com<mailto:dneary@redhat.com>> wrote: Hi all, Rather than just complain into the ether, I wanted to let people know why I don't like the voting process for conference proposals and see if I'm the only one. I don't think that the voting process is the best way to gauge whether proposals will be good for the conference. There are a few reasons for that: * Having to hawk & promote proposal(s) is kind of unseemly, and makes us look small, I think. Hundreds of people going "vote for me!" doesn't make us look good. * Some people don't want to pitch themselves, others don't have access to as big a platform to promote * The same issues exist with this system which exist with board voting - there is a possibility that people will vote for their colleagues, not out of any corruption, but just because no-one has time to rate all the proposals, and they're more likely to rate those submitted by people they know more highly * Also, it's a self-selecting group of people who rate proposals - I don't think voters will be representative of summit attendees * After all is said and done, the proposals which are chosen by the voters are guidelines to the people who choose the talks for the tracks, the track leaders I have been a track leader for the last number of summits, and I've seen first hand great presentations get very low numbers of votes, while others which are not as interesting get very high numbers of votes and high ratings. Personally, I would be happy if we could change the system to remove the "pimp my talk" aspect for Summits. Cheers, Dave. -- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13 _______________________________________________ Community mailing list Community@lists.openstack.org<mailto:Community@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Sorry for the late reply. There is an interesting model (Swiss-system tournament (http://en.wikipedia.org/wiki/Swiss-system_tournament)) to be used in the selection of presentations without that the voter can be influenced by the proximity with a person, or with some subject of interest or the amount of votes. This model consists in distributing some cards for each participant in a Summit or a past Summit. When the vote occurs, the system automatically selects two proposals for presentations to be voted (duel between the two presentations). The person who is voting can choose one of them, or vote neutral. In a ranking order is created with the proposals that have won more. This ensures that all proposals have the same number of evaluations while also ensuring that each person can not choose which presentation you want to vote. Cheers, Marcelo Em 20-02-2014 18:15, Tim Bell escreveu:
Personally, I found the wide range of talks very interesting and there are several that I would be following up on even if they are not selected for the summit. Thus, a full list of talk proposals is valuable.
However, the track chairs should be able to balance the vote with their knowledge of the community. The vote should only be advisory rather than mechanical. Selecting the top votes risks being dominated by the current topic du jour rather than a balance of interest of the community. I would not be in favour of limiting the vote to those attending since the videos of the talks are much more widely viewed (and helps for multiple parallel bookings for summit attendees)
Overall, I think the importance is that the track chairs feel empowered to choose the appropriate balance of talks for the tracks. There is always a risk of selection bias from the track chairs so there should also be a reasonable turnover in those roles too and a feedback loop for attendees to rate the tracks for interest.
How do other big conferences organise this ? We can't be the only ones with over subscribed summits.
Tim
*From:*Tristan Goode [mailto:tristan@aptira.com] *Sent:* 20 February 2014 22:06 *To:* Adam Nelson; Dave Neary *Cc:* community@lists.openstack.org *Subject:* Re: [openstack-community] OpenStack Summit proposal voting - not a fan
I tend to agree. Perhaps voting could be limited to those that have RSVPed and are actually going to the Summit as well?
In an effort to do my best and look at all the submissions, I voted for a combined several hours in a few goes yesterday and still haven't reached any kind of end. It did seem like presentations came past multiple times and the volume of talks with a tenuous link to OpenStack made it a challenge.
There were also talks in there that were submitted for HKG that had NOT been submitted to ATL. I confirmed this with a couple of submitters.
It also seemed like some folks submit several talks that are almost the same, and some people just submit too many talks. Perhaps a one or two talk per person limit. Or one talk and one panel.
This time I do hope we don't see the combination of talks like we had in HKG which I thought was a good idea as a track chair beforehand, but when I went and saw them on some occasions I felt for the speakers.
Am going to have another crack at finishing my voting today. Wish me luck :-P
*From:*Adam Nelson [mailto:adam@varud.com <mailto:adam@varud.com>] *Sent:* Friday, 21 February 2014 4:04 AM *To:* Dave Neary *Cc:* community@lists.openstack.org <mailto:community@lists.openstack.org> *Subject:* Re: [openstack-community] OpenStack Summit proposal voting - not a fan
Lots of conferences do this to boost interest and engage with the community.... but you're absolutely right.
A good compromise would be to give track leaders some magic votes to boost certain talks and veto others.
One way to implement this would be to allow tracks to have their own algorithms. Openstack is too broad to have one method for all the talks and I think it's totally reasonable to have a default method which track chairs can override.
-Adam
--
Kili - Cloud for Africa: kili.io <http://kili.io/>
Musings: twitter.com/varud <https://twitter.com/varud>
More Musings: varud.com <http://varud.com>
About Adam: www.linkedin.com/in/adamcnelson <https://www.linkedin.com/in/adamcnelson>
On Thu, Feb 20, 2014 at 6:51 PM, Dave Neary <dneary@redhat.com <mailto:dneary@redhat.com>> wrote:
Hi all,
Rather than just complain into the ether, I wanted to let people know why I don't like the voting process for conference proposals and see if I'm the only one.
I don't think that the voting process is the best way to gauge whether proposals will be good for the conference. There are a few reasons for that:
* Having to hawk & promote proposal(s) is kind of unseemly, and makes us look small, I think. Hundreds of people going "vote for me!" doesn't make us look good. * Some people don't want to pitch themselves, others don't have access to as big a platform to promote * The same issues exist with this system which exist with board voting - there is a possibility that people will vote for their colleagues, not out of any corruption, but just because no-one has time to rate all the proposals, and they're more likely to rate those submitted by people they know more highly * Also, it's a self-selecting group of people who rate proposals - I don't think voters will be representative of summit attendees * After all is said and done, the proposals which are chosen by the voters are guidelines to the people who choose the talks for the tracks, the track leaders
I have been a track leader for the last number of summits, and I've seen first hand great presentations get very low numbers of votes, while others which are not as interesting get very high numbers of votes and high ratings.
Personally, I would be happy if we could change the system to remove the "pimp my talk" aspect for Summits.
Cheers, Dave.
-- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com <http://community.redhat.com> Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13
_______________________________________________ Community mailing list Community@lists.openstack.org <mailto:Community@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/community
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Hi, On 03/09/2014 04:40 AM, Marcelo Dieder wrote:
Sorry for the late reply. There is an interesting model (Swiss-system tournament (http://en.wikipedia.org/wiki/Swiss-system_tournament)) to be used in the selection of presentations without that the voter can be influenced by the proximity with a person, or with some subject of interest or the amount of votes.
This model consists in distributing some cards for each participant in a Summit or a past Summit. When the vote occurs, the system automatically selects two proposals for presentations to be voted (duel between the two presentations). The person who is voting can choose one of them, or vote neutral.
In a ranking order is created with the proposals that have won more. This ensures that all proposals have the same number of evaluations while also ensuring that each person can not choose which presentation you want to vote.
I think I saw this in the Social Network... isn't that the same system used in Hot or Not and chess ladders? Cheers, Dave. -- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13
Hi,
On 03/09/2014 04:40 AM, Marcelo Dieder wrote:
Sorry for the late reply. There is an interesting model (Swiss-system tournament (http://en.wikipedia.org/wiki/Swiss-system_tournament)) to be used in the selection of presentations without that the voter can be influenced by the proximity with a person, or with some subject of interest or the amount of votes.
This model consists in distributing some cards for each participant in a Summit or a past Summit. When the vote occurs, the system automatically selects two proposals for presentations to be voted (duel between the two presentations). The person who is voting can choose one of them, or vote neutral.
In a ranking order is created with the proposals that have won more. This ensures that all proposals have the same number of evaluations while also ensuring that each person can not choose which presentation you want to vote. I think I saw this in the Social Network... isn't that the same system used in Hot or Not and chess ladders? Yes, is the same system used in chess and maybe on "Hot or Not". At the FISL (Open Source Forum in Brazil), the same system is used to select
Hi Dave, On 10-03-2014 13:00, Dave Neary wrote: presentations from about 900 proposals.
Cheers, Dave.
Cheers, Marcelo
Just to be clear, track chairs have always been encouraged to use their best judgment, using the voting data as one form of input which is by no means absolute or the only form of input. Ultimately the track chairs make the best decision they can. That will continue in Atlanta. On Feb 20, 2014, at 11:04 AM, Adam Nelson <adam@varud.com> wrote:
Lots of conferences do this to boost interest and engage with the community.... but you're absolutely right.
A good compromise would be to give track leaders some magic votes to boost certain talks and veto others.
One way to implement this would be to allow tracks to have their own algorithms. Openstack is too broad to have one method for all the talks and I think it's totally reasonable to have a default method which track chairs can override.
-Adam
-- Kili - Cloud for Africa: kili.io Musings: twitter.com/varud More Musings: varud.com About Adam: www.linkedin.com/in/adamcnelson
On Thu, Feb 20, 2014 at 6:51 PM, Dave Neary <dneary@redhat.com> wrote: Hi all,
Rather than just complain into the ether, I wanted to let people know why I don't like the voting process for conference proposals and see if I'm the only one.
I don't think that the voting process is the best way to gauge whether proposals will be good for the conference. There are a few reasons for that:
* Having to hawk & promote proposal(s) is kind of unseemly, and makes us look small, I think. Hundreds of people going "vote for me!" doesn't make us look good. * Some people don't want to pitch themselves, others don't have access to as big a platform to promote * The same issues exist with this system which exist with board voting - there is a possibility that people will vote for their colleagues, not out of any corruption, but just because no-one has time to rate all the proposals, and they're more likely to rate those submitted by people they know more highly * Also, it's a self-selecting group of people who rate proposals - I don't think voters will be representative of summit attendees * After all is said and done, the proposals which are chosen by the voters are guidelines to the people who choose the talks for the tracks, the track leaders
I have been a track leader for the last number of summits, and I've seen first hand great presentations get very low numbers of votes, while others which are not as interesting get very high numbers of votes and high ratings.
Personally, I would be happy if we could change the system to remove the "pimp my talk" aspect for Summits.
Cheers, Dave.
-- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Dave Neary <dneary@redhat.com> 2014/02/20 23:51
To
"community@lists.openstack.org" <community@lists.openstack.org>,
cc
Subject
[openstack-community] OpenStack Summit proposal voting - not a fan
Hi all,
Rather than just complain into the ether, I wanted to let people know why I don't like the voting process for conference proposals and see if I'm the only one.
I don't think that the voting process is the best way to gauge whether proposals will be good for the conference. There are a few reasons for
I don't like vote and promote myself either. But suggestions are better than complains. I think, the vote result is useful for track owners to understand audiences interests. We just need to simplify the way to "vote". Other than using single link for a single presentation, listing all the presentations of one track in a single page will help the voter to understand the overall image of this track, and save the voter's time to click "next page". Maybe only listing titles and a short description. If the voters want to understand details, provide a link for them. Using only a tick to mark the topics that voters are interested, other than 4 levels of scores. It's also a way to save voters' time. Then maybe a voter only needs 10 minutes to finish the vote of his/her interested tracks. Then more people would like to vote. And then no people would bother to promote himself/herself. Best regards Ying Chun Guo (Daisy) Dave Neary <dneary@redhat.com> wrote on 2014/02/20 23:51:24: that:
* Having to hawk & promote proposal(s) is kind of unseemly, and makes us look small, I think. Hundreds of people going "vote for me!" doesn't make us look good. * Some people don't want to pitch themselves, others don't have access to as big a platform to promote * The same issues exist with this system which exist with board voting - there is a possibility that people will vote for their colleagues, not out of any corruption, but just because no-one has time to rate all the proposals, and they're more likely to rate those submitted by people they know more highly * Also, it's a self-selecting group of people who rate proposals - I don't think voters will be representative of summit attendees * After all is said and done, the proposals which are chosen by the voters are guidelines to the people who choose the talks for the tracks, the track leaders
I have been a track leader for the last number of summits, and I've seen first hand great presentations get very low numbers of votes, while others which are not as interesting get very high numbers of votes and high ratings.
Personally, I would be happy if we could change the system to remove the "pimp my talk" aspect for Summits.
Cheers, Dave.
-- Dave Neary - Community Action and Impact Open Source and Standards, Red Hat - http://community.redhat.com Ph: +33 9 50 71 55 62 / Cell: +33 6 77 01 92 13
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
participants (7)
-
Adam Nelson
-
Dave Neary
-
Marcelo Dieder
-
Mark Collier
-
Tim Bell
-
Tristan Goode
-
Ying Chun Guo