[openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
Hi everyone, For the past few Summits, we've received mixed feedback about having the community vote on proposed sessions as part of the call for speakers process. Historically, after the call for speakers closes, we publish all submitted sessions for community voting before the track chairs review them. The track chairs then choose how much weight to put on the voting resuls, if any, because they make the ultimate decision about which sessions are selected. More info on Track Chairs can be found at the bottom of this email. With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case). We would like to propose removing voting from the selection process for the October 2016 Barcelona Summit, but want to get your input before making a final decision. Our thinking is that by removing voting from the process, we will: - Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle - Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes - Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting We initially started the voting process for good reasons and we do think there's value, but we're reaching a point where the costs are starting to outweigh the benefits. We'd like to get your input before we open the call for speakers in early June for the Barcelona Summit. Thanks, Claire Track Chair Info Track Chairs are subject matter experts who review submissions to their particular track, for example "storage" or "cloud applications." There are typically 3-4 chairs per track who review and collaboratively decide which presentations are ultimately accepted for inclusion on the final agenda. The Foundation strives to recruit Track Chairs from a diverse set of companies, regions, roles in the community (i.e., contributing developers, users and business leaders) and areas of expertise. Information on how to nominate yourself or someone else to serve as a track chair for the Barcelona Summit will be published when the call for speakers goes live in early June. For reference, here's information on the Track Chairs from the Austin Summit: https://etherpad.openstack.org/p/Austin_Summit_Track_Chairs <https://etherpad.openstack.org/p/Austin_Summit_Track_Chairs> and https://www.openstack.org/summit/austin-2016/categories/selection-process <https://www.openstack.org/summit/austin-2016/categories/selection-process>.
Hi everyone, having had the privilege to serve as both as a track chair and as a speaker at previous Summits, I'd like to offer my views here. [Claire]
Hi everyone,
For the past few Summits, we've received mixed feedback about having the community vote on proposed sessions as part of the call for speakers process. Historically, after the call for speakers closes, we publish all submitted sessions for community voting before the track chairs review them. The track chairs then choose how much weight to put on the voting resuls, if any, because they make the ultimate decision about which sessions are selected. More info on Track Chairs can be found at the bottom of this email.
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case).
I would agree that there was some concern on - ballot stuffing, - people voting along "party lines" (i.e. supporting their colleague's talks based more on shared company allegiance rather than talk merit), - companies with a strong social media reach doing their part to drive their own votes up, and such promoted talks being over-voted. [Claire]
We would like to propose removing voting from the selection process for the October 2016 Barcelona Summit, but want to get your input before making a final decision.
I have previously suggested a different approach: improve the *quality* of voting drastically, rather than abolish it altogether. Now what follows isn't a voting process of my own design. We owe the idea to a couple of professors of astronomy, who explain their approach in a paper and a couple of interviews, all of which are linked from here: https://plus.google.com/+FlorianHaas/posts/MsswaBHramG (Please note: I have deliberately disabled comments on that post, in an attempt to keep the discussion here on this list, where it belongs.) If you care about the process, please take a few minutes to review at least both videos linked from that post. If you're *really* interested, read the article as well -- but for the purposes of this discussion, the videos should suffice. When I first floated this idea on the track chair mailing list a few months ago, Duncan Thomas made this point: [Duncan]
I think limiting votes only to people who submit talks would lead to people/companies submitting poor talks just to get a vote (gaming the system).
To which I then replied: [Florian]
That's a fair point. However, reviewers could separately flag proposals that don't meet certain quality criteria. (*Some* formal criteria could even be checked by computers, not humans.) And there could be a rule that if, say, the majority of a talk's (anonymous) reviewers flag foul play, all the proposer's proposals *and* all and the proposer's votes would be invalidated. I think that would be a fairly strong deterrent. And in order to deter abuse of *that* system, the event of a proposer being thus sin-binned should probably be reviewed by a panel of some description.
(Just quoting this here to paint a clearer picture.) [Claire]
Our thinking is that by removing voting from the process, we will: - Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle - Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes
- Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting
While all the above intentions are noble and I agree with them, I do believe that the alternate approach which I'm humbly proposing has all those merits as well. And I do understand that this also means additional work on the voting toolchain. So even in case there is agreement on the approach, I am not suggesting for it to be implemented for Barcelona. Instead, my suggestion would be for the voting process to be unchanged for the upcoming Summit in spite of its imperfections, and for the new approach to be considered for Boston. [Claire]
We initially started the voting process for good reasons and we do think there's value, but we're reaching a point where the costs are starting to outweigh the benefits. We'd like to get your input before we open the call for speakers in early June for the Barcelona Summit.
May I offer a couple of additional thoughts on why I believe killing voting altogether is bad: - Firstly, we've seen some track chair lobbying in the run-up to the Austin cycle already, and in our track tried to nip it in the bud, but I think this would be far worse if track chairs were the only people influencing the talk selection. They already have the final say, but it's a different quality if they have the *only* say. - Secondly and more importantly, I do think of OpenStack as a community, and I much appreciate the fact that the Foundation has chosen a model involving broad individual membership. We as a community ought to have a stake in the Summit selection process; rather than weakening or eliminating that stake, strengthening it would be prudent. What do others think? Cheers, Florian
On May 17, 2016, at 2:13 PM, Florian Haas <florian@hastexo.com> wrote:
I have previously suggested a different approach: improve the *quality* of voting drastically, rather than abolish it altogether.
Now what follows isn't a voting process of my own design. We owe the idea to a couple of professors of astronomy, who explain their approach in a paper and a couple of interviews, all of which are linked from here:
https://plus.google.com/+FlorianHaas/posts/MsswaBHramG <https://plus.google.com/+FlorianHaas/posts/MsswaBHramG>
(Please note: I have deliberately disabled comments on that post, in an attempt to keep the discussion here on this list, where it belongs.)
If you care about the process, please take a few minutes to review at least both videos linked from that post. If you're *really* interested, read the article as well -- but for the purposes of this discussion, the videos should suffice.
Florian, I appreciate your comments in this thread and agree with them nearly entirely. I think those who want to completely eliminate the vote, while well intentioned, would be doing a disservice to the community and ultimately OpenStack itself. I very strongly support the idea of evolving the voting process to try to enhance its value rather than doing away with it entirely. The method you’ve proposed seems like a reasonable place to start. Regards, Richard
On 19/05/16 03:10, Richard Raseley wrote:
On May 17, 2016, at 2:13 PM, Florian Haas <florian@hastexo.com <mailto:florian@hastexo.com>> wrote:
I have previously suggested a different approach: improve the *quality* of voting drastically, rather than abolish it altogether.
Now what follows isn't a voting process of my own design. We owe the idea to a couple of professors of astronomy, who explain their approach in a paper and a couple of interviews, all of which are linked from here:
https://plus.google.com/+FlorianHaas/posts/MsswaBHramG
(Please note: I have deliberately disabled comments on that post, in an attempt to keep the discussion here on this list, where it belongs.)
If you care about the process, please take a few minutes to review at least both videos linked from that post. If you're *really* interested, read the article as well -- but for the purposes of this discussion, the videos should suffice.
Florian,
I appreciate your comments in this thread and agree with them nearly entirely.
I think those who want to completely eliminate the vote, while well intentioned, would be doing a disservice to the community and ultimately OpenStack itself.
I very strongly support the idea of evolving the voting process to try to enhance its value rather than doing away with it entirely. The method you’ve proposed seems like a reasonable place to start.
Regards,
Richard
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house. Perhaps, instead of limiting it to talk submitters, maybe make it available to a different subset: people who have attended previously, maybe? Lana -- Lana Brindley Technical Writer Rackspace Cloud Builders Australia http://lanabrindley.com
Hi Lana! On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com> wrote:
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it available to a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :) If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake. Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process. There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s (tl;dw: have public, anonymized free-form comments available to all reviewers). At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers). What are your thoughts on that? Cheers, Florian
So let me get this straight... You're proposing that only speakers vote for talks yeah? Because hey, fuck the audience right?
-----Original Message----- From: Florian Haas [mailto:florian@hastexo.com] Sent: Thursday, 19 May 2016 6:00 PM To: Lana Brindley <openstack@lanabrindley.com> Cc: community <community@lists.openstack.org> Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
Hi Lana!
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to the main conference for the talks I absolutely don't want to miss. I think by
On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com> wrote: limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it available to
a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :)
If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake.
Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process.
There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s (tl;dw: have public, anonymized free-form comments available to all reviewers).
At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers).
What are your thoughts on that?
Cheers, Florian
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On Thu, May 19, 2016 at 3:19 PM, Tristan Goode <tristan@aptira.com> wrote:
So let me get this straight... You're proposing that only speakers vote for talks yeah?
Because hey, fuck the audience right?
Firstly, no need to get all sweary. Secondly, what the Foundation is proposing is that no-one gets to vote anymore at all. I wonder what that means for the audience. Thirdly, yes what I am proposing is effectively that speakers, *all* prospective speakers, vote for talks. I believe a large group of speakers, all of whom are strongly rooted in the community, in aggregate is a better judge of talk quality for the Summit than both an entire audience whose votes have no bearing on anything, and a small group of hand-picked track chairs. Fourthly, as I've tried to explain before, the review method I'm advocating does _not_ allow reviewers/speakers to review carelessly, or on a whim. In fact, it enforces that rather than thinking about your own preference of what you would like to see and listen to, you need to think along the lines of "what would everyone *else* be interested in". So no, no f*** the audience at all. Be the audience's advocate. Cheers, Florian
Le 19/05/2016 15:19, Tristan Goode a écrit :
So let me get this straight... You're proposing that only speakers vote for talks yeah?
Because hey, fuck the audience right?
I'd be totally opposed to that idea that would induce a clear bias. Let me explain : while the Foundation is trusting a different set of people at every Summit for each track, the above would create a define list of people that would be quite the same for each Summit - because we know that people naturally tend to prefer their own close relations. Don't blame me, but I'm seeing this as an argument about how much we currently trust the track chairs as non-biased people. FWIW, if we agree with the fact that track chairs are good for their duty, why should we change how we select them ? Maybe the proposal is to leave the track chairs, and only allow votes from previous speakers ? If so, that's even more terrible : we're moving from a representative democracy (the track chairs) to an oligarchy. As a regular contributor helping my peers to submit talks, I can just say : kill the votes, they're unnecessarly creating confusion about the selection process, leave how the Foundation pick the track chairs because it has been proven that the talks were high-quality, and just improve the feedback process so that track chairs can use that for their own judgment. -Sylvain
-----Original Message----- From: Florian Haas [mailto:florian@hastexo.com] Sent: Thursday, 19 May 2016 6:00 PM To: Lana Brindley <openstack@lanabrindley.com> Cc: community <community@lists.openstack.org> Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
Hi Lana!
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to the main conference for the talks I absolutely don't want to miss. I think by
On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com> wrote: limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it available to a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :)
If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake.
Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process.
There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s (tl;dw: have public, anonymized free-form comments available to all reviewers).
At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers).
What are your thoughts on that?
Cheers, Florian
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On Thu, May 19, 2016 at 3:40 PM, Sylvain Bauza <sbauza@redhat.com> wrote:
Le 19/05/2016 15:19, Tristan Goode a écrit :
So let me get this straight... You're proposing that only speakers vote for talks yeah?
Because hey, fuck the audience right?
I'd be totally opposed to that idea that would induce a clear bias. Let me explain : while the Foundation is trusting a different set of people at every Summit for each track, the above would create a define list of people that would be quite the same for each Summit - because we know that people naturally tend to prefer their own close relations.
How is that a factor if out of hundreds of talks you only ever review 8-10, randomly selected? What are the odds that your own close relations will even be in the subset you review?
Don't blame me, but I'm seeing this as an argument about how much we currently trust the track chairs as non-biased people. FWIW, if we agree with the fact that track chairs are good for their duty, why should we change how we select them ? Maybe the proposal is to leave the track chairs, and only allow votes from previous speakers ? If so, that's even more terrible : we're moving from a representative democracy (the track chairs) to an oligarchy.
Nope, that's not my proposal. Mine isn't about previous speakers, it's about current talk submitters. Also, minor point. Track chairs are appointed, not elected. I'm not saying that this invalidates your point, but the analogy is off. Not as much a representative democracy as an aristocratic fiefdom. :) Cheers, Florian
Le 19/05/2016 16:04, Florian Haas a écrit :
On Thu, May 19, 2016 at 3:40 PM, Sylvain Bauza <sbauza@redhat.com> wrote:
Le 19/05/2016 15:19, Tristan Goode a écrit :
So let me get this straight... You're proposing that only speakers vote for talks yeah?
Because hey, fuck the audience right?
I'd be totally opposed to that idea that would induce a clear bias. Let me explain : while the Foundation is trusting a different set of people at every Summit for each track, the above would create a define list of people that would be quite the same for each Summit - because we know that people naturally tend to prefer their own close relations.
How is that a factor if out of hundreds of talks you only ever review 8-10, randomly selected? What are the odds that your own close relations will even be in the subset you review?
Because you're not randomly picking a list of talks when reviewing. You naturally tend to review the ones you feel good at, where you built your network and where you know people. Or, on the other hand, you naturally tend to select talks made either by your company peers, or the social network relatives you're close to. Anyway, my point is that you're already biased when picking a subset of talks to vote to (because you can't, as an human, vote for 800+ talks). The vote itself is another bias, where you tend to +3 the ones you trust, because you know the proposer, and not because the proposed abstract is great.
Don't blame me, but I'm seeing this as an argument about how much we currently trust the track chairs as non-biased people. FWIW, if we agree with the fact that track chairs are good for their duty, why should we change how we select them ? Maybe the proposal is to leave the track chairs, and only allow votes from previous speakers ? If so, that's even more terrible : we're moving from a representative democracy (the track chairs) to an oligarchy. Nope, that's not my proposal. Mine isn't about previous speakers, it's about current talk submitters.
Fair point, that doesn't really change my opinion. My paranoid mind also leads me to consider a Foo company wanting to promote a certain talk by adding more talks hence more voters for that specific talk.
Also, minor point. Track chairs are appointed, not elected. I'm not saying that this invalidates your point, but the analogy is off. Not as much a representative democracy as an aristocratic fiefdom. :)
Fair point.
Cheers, Florian
On Thu, May 19, 2016 at 4:37 PM, Sylvain Bauza <sbauza@redhat.com> wrote:
Le 19/05/2016 16:04, Florian Haas a écrit :
On Thu, May 19, 2016 at 3:40 PM, Sylvain Bauza <sbauza@redhat.com> wrote:
Le 19/05/2016 15:19, Tristan Goode a écrit :
So let me get this straight... You're proposing that only speakers vote for talks yeah?
Because hey, fuck the audience right?
I'd be totally opposed to that idea that would induce a clear bias. Let me explain : while the Foundation is trusting a different set of people at every Summit for each track, the above would create a define list of people that would be quite the same for each Summit - because we know that people naturally tend to prefer their own close relations.
How is that a factor if out of hundreds of talks you only ever review 8-10, randomly selected? What are the odds that your own close relations will even be in the subset you review?
Because you're not randomly picking a list of talks when reviewing. You naturally tend to review the ones you feel good at, where you built your network and where you know people. Or, on the other hand, you naturally tend to select talks made either by your company peers, or the social network relatives you're close to.
As I have tried to point out repeatedly, I am proposing a system where the randomized talk selection *is done for you*, so you never see all talks up for review, you only see those allotted to you. You review those, and your review is done. You don't get to see any others. Someone else gets a different set, which may or may not overlap with yours. If your group of reviewers is big enough (a given at the Summit), that means you get multiple eyeballs on every single proposal, no proposal goes unreviewed, and no-one has to deal with a daunting pile of submissions to review.
Anyway, my point is that you're already biased when picking a subset of talks to vote to (because you can't, as an human, vote for 800+ talks). The vote itself is another bias, where you tend to +3 the ones you trust, because you know the proposer, and not because the proposed abstract is great.
Don't blame me, but I'm seeing this as an argument about how much we currently trust the track chairs as non-biased people. FWIW, if we agree with the fact that track chairs are good for their duty, why should we change how we select them ? Maybe the proposal is to leave the track chairs, and only allow votes from previous speakers ? If so, that's even more terrible : we're moving from a representative democracy (the track chairs) to an oligarchy.
Nope, that's not my proposal. Mine isn't about previous speakers, it's about current talk submitters.
Fair point, that doesn't really change my opinion. My paranoid mind also leads me to consider a Foo company wanting to promote a certain talk by adding more talks hence more voters for that specific talk.
Yes, this is Duncan Thomas' point from several months ago that I've already addressed upthread. Cheers, Florian
Excerpts from Florian Haas's message of 2016-05-19 10:00:25 +0200:
Hi Lana!
On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com> wrote:
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it available to a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :)
If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake.
Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process.
There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s (tl;dw: have public, anonymized free-form comments available to all reviewers).
At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers).
What are your thoughts on that?
Cheers, Florian
I feel like this system assumes bad faith on the part of the contributor (speaker, reviewer, and voter), and tries to enforce good behavior through rules and technology. I would rather we have a more public way of selecting track chairs and then have faith in them to evaluate talks objectively for relevance and quality, sharing guidance and feedback as part of the process. Doug
This attendees-only idea isn't entirely without merit, but the audience isn't just people who attend. That is, unless the travel support program decides to go webscale. One can't exclude voters because they can't afford to spend a few thousand dollars to fly to Austin for a week. Some may say such a proposition would be be racist in effect if not intent, but I couldn't possibly comment. :P There are two fundamental choices: democracy or not (which includes curation, oligarchy and dictatorship). If we want to have a reduced franchise we may as well exclude everyone that is affiliated to a submitter (perhaps outside of the marketing talks, and who goes to them?). That would certainly solve the inherent bias issues whilst also minimising dross. Roland On 19 May 2016 11:32 pm, "Doug Hellmann" <doug@doughellmann.com> wrote:
Hi Lana!
On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com> wrote:
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to
Excerpts from Florian Haas's message of 2016-05-19 10:00:25 +0200: the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it
available to a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :)
If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake.
Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process.
There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s (tl;dw: have public, anonymized free-form comments available to all reviewers).
At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers).
What are your thoughts on that?
Cheers, Florian
I feel like this system assumes bad faith on the part of the contributor (speaker, reviewer, and voter), and tries to enforce good behavior through rules and technology. I would rather we have a more public way of selecting track chairs and then have faith in them to evaluate talks objectively for relevance and quality, sharing guidance and feedback as part of the process.
Doug
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On May 19, 2016, at 10:06 AM, Roland Chan <roland@aptira.com> wrote:
This attendees-only idea isn't entirely without merit, but the audience isn't just people who attend. That is, unless the travel support program decides to go webscale. One can't exclude voters because they can't afford to spend a few thousand dollars to fly to Austin for a week. Some may say such a proposition would be be racist in effect if not intent, but I couldn't possibly comment. :P
That's a good point. On the other hand, anyone in the world, whether they’re normally part of the community or not, can watch the videos. So optimizing to allow every potential viewer to have input into the program, over the people who do attend in person, doesn’t really have any guarantee of giving us a good outcome either. That said, I threw out the piece of information as a data point. I still think having track chairs empowered to build the best program they can will give better results than any amount of crowd-sourcing. Doug
There are two fundamental choices: democracy or not (which includes curation, oligarchy and dictatorship).
If we want to have a reduced franchise we may as well exclude everyone that is affiliated to a submitter (perhaps outside of the marketing talks, and who goes to them?). That would certainly solve the inherent bias issues whilst also minimising dross.
Roland
On 19 May 2016 11:32 pm, "Doug Hellmann" <doug@doughellmann.com <mailto:doug@doughellmann.com>> wrote: Excerpts from Florian Haas's message of 2016-05-19 10:00:25 +0200:
Hi Lana!
On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com <mailto:openstack@lanabrindley.com>> wrote:
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it available to a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :)
If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s <https://youtu.be/7c0CoXFApnM?t=6m25s> — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake.
Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process.
There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s <https://youtu.be/bplncn4xC74?t=1m48s> (tl;dw: have public, anonymized free-form comments available to all reviewers).
At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers).
What are your thoughts on that?
Cheers, Florian
I feel like this system assumes bad faith on the part of the contributor (speaker, reviewer, and voter), and tries to enforce good behavior through rules and technology. I would rather we have a more public way of selecting track chairs and then have faith in them to evaluate talks objectively for relevance and quality, sharing guidance and feedback as part of the process.
Doug
_______________________________________________ Community mailing list Community@lists.openstack.org <mailto:Community@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/community <http://lists.openstack.org/cgi-bin/mailman/listinfo/community>
Hi Lana!
On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com> wrote:
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to
So long as the track chairs are not just the usual cheer squad. I'd really like to see some hard core critics of openstack able to take part. *From:* Doug Hellmann [mailto:doug@doughellmann.com] *Sent:* Friday, 20 May 2016 12:11 AM *To:* Roland Chan <roland@aptira.com> *Cc:* community <community@lists.openstack.org> *Subject:* Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit On May 19, 2016, at 10:06 AM, Roland Chan <roland@aptira.com> wrote: This attendees-only idea isn't entirely without merit, but the audience isn't just people who attend. That is, unless the travel support program decides to go webscale. One can't exclude voters because they can't afford to spend a few thousand dollars to fly to Austin for a week. Some may say such a proposition would be be racist in effect if not intent, but I couldn't possibly comment. :P That's a good point. On the other hand, anyone in the world, whether they’re normally part of the community or not, can watch the videos. So optimizing to allow every potential viewer to have input into the program, over the people who do attend in person, doesn’t really have any guarantee of giving us a good outcome either. That said, I threw out the piece of information as a data point. I still think having track chairs empowered to build the best program they can will give better results than any amount of crowd-sourcing. Doug There are two fundamental choices: democracy or not (which includes curation, oligarchy and dictatorship). If we want to have a reduced franchise we may as well exclude everyone that is affiliated to a submitter (perhaps outside of the marketing talks, and who goes to them?). That would certainly solve the inherent bias issues whilst also minimising dross. Roland On 19 May 2016 11:32 pm, "Doug Hellmann" <doug@doughellmann.com> wrote: Excerpts from Florian Haas's message of 2016-05-19 10:00:25 +0200: the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it
available to a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :)
If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake.
Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process.
There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s (tl;dw: have public, anonymized free-form comments available to all reviewers).
At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers).
What are your thoughts on that?
Cheers, Florian
I feel like this system assumes bad faith on the part of the contributor (speaker, reviewer, and voter), and tries to enforce good behavior through rules and technology. I would rather we have a more public way of selecting track chairs and then have faith in them to evaluate talks objectively for relevance and quality, sharing guidance and feedback as part of the process. Doug _______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Hi Lana!
On Thu, May 19, 2016 at 3:23 AM, Lana Brindley <openstack@lanabrindley.com> wrote:
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to
There are three choices. Democracy or not, or the barrel. *From:* Roland Chan [mailto:roland@aptira.com] *Sent:* Friday, 20 May 2016 12:06 AM *To:* Doug Hellmann <doug@doughellmann.com> *Cc:* community <community@lists.openstack.org> *Subject:* Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit This attendees-only idea isn't entirely without merit, but the audience isn't just people who attend. That is, unless the travel support program decides to go webscale. One can't exclude voters because they can't afford to spend a few thousand dollars to fly to Austin for a week. Some may say such a proposition would be be racist in effect if not intent, but I couldn't possibly comment. :P There are two fundamental choices: democracy or not (which includes curation, oligarchy and dictatorship). If we want to have a reduced franchise we may as well exclude everyone that is affiliated to a submitter (perhaps outside of the marketing talks, and who goes to them?). That would certainly solve the inherent bias issues whilst also minimising dross. Roland On 19 May 2016 11:32 pm, "Doug Hellmann" <doug@doughellmann.com> wrote: Excerpts from Florian Haas's message of 2016-05-19 10:00:25 +0200: the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it
available to a different subset: people who have attended previously, maybe?
Hmmm. Well I'm afraid limiting talk votes to talk submitters is exactly what makes the proposed approach meaningful. :)
If you have a minute, please consider reviewing Prof. Merrifield's remarks in the video when Brady asks his question starting with "call me a cynic", about https://youtu.be/7c0CoXFApnM?t=6m25s — this is exactly the part that makes this system self-policing, and it goes out the window if your own proposal isn't at stake.
Side note, if your assessment badly disagrees with what everyone else has been thinking about a proposal, then this is not necessarily because you're naughty and you want to game the system — you may just be a shoddy reviewer who went over their reviewed proposals in a rush whereas everyone else gave them more time. That, too, is something that the system *should* penalize, because it ensures the quality of the review process.
There is one other criticism to this, which is the opposite: what if I'm being *extremely* diligent and I detect an issue that no-one else detects? This is addressed here: https://youtu.be/bplncn4xC74?t=1m48s (tl;dw: have public, anonymized free-form comments available to all reviewers).
At any rate though, I can't think of a way to do this that does *not* make the group of reviewers identical with the group of submitters. And quite frankly, I quite like it as it is, considering the fact that the proposed system forces everyone not only to think "how would I rank this", but also "how would *others* rank this", which is exactly what you want for the benefit of the much greater group of conference attendees (as opposed to speakers).
What are your thoughts on that?
Cheers, Florian
I feel like this system assumes bad faith on the part of the contributor (speaker, reviewer, and voter), and tries to enforce good behavior through rules and technology. I would rather we have a more public way of selecting track chairs and then have faith in them to evaluate talks objectively for relevance and quality, sharing guidance and feedback as part of the process. Doug _______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Hi, On 05/19/2016 09:38 AM, Doug Hellmann wrote:
I feel like this system assumes bad faith on the part of the contributor (speaker, reviewer, and voter), and tries to enforce good behavior through rules and technology. I would rather we have a more public way of selecting track chairs and then have faith in them to evaluate talks objectively for relevance and quality, sharing guidance and feedback as part of the process.
This pretty much summarises my view also. Transparency and accountability of those who choose, and trust that the senior, qualified members of our community who are chosen will do what is in the greater good. Thanks, Dave. -- Dave Neary - NFV/SDN Community Strategy Open Source and Standards, Red Hat - http://community.redhat.com Ph: +1-978-399-2182 / Cell: +1-978-799-3338
On Thu, May 19, 2016 at 3:38 PM, Doug Hellmann <doug@doughellmann.com> wrote:
I feel like this system assumes bad faith on the part of the contributor (speaker, reviewer, and voter), and tries to enforce good behavior through rules and technology. I would rather we have a more public way of selecting track chairs and then have faith in them to evaluate talks objectively for relevance and quality, sharing guidance and feedback as part of the process.
Making the track chair selection process more public and open is certainly a good thing; I also wouldn't mind for the individual selections (shortlists) to be public. Doesn't really address Lana's point that the sheer number of submission makes it a challenge to review all submissions in a track diligently and carefully, but would be an improvement. Cheers, Florian
On 2016-05-19 09:38:24 -0400 (-0400), Doug Hellmann wrote: [...]
I would rather we have a more public way of selecting track chairs [...]
There's a separate demon hidden in that idea though. We don't (or didn't in the past AFAIK) publish the names of track chairs prior to conclusion of the talk selection process. The last thing I would want, as a track chair, is to suddenly be faced with hundreds or more E-mails from random speakers and their fans and coworkers asking me to please, please, pretty-please pick their proposal. -- Jeremy Stanley
On Thu, May 19, 2016 at 6:23 PM, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2016-05-19 09:38:24 -0400 (-0400), Doug Hellmann wrote: [...]
I would rather we have a more public way of selecting track chairs [...]
There's a separate demon hidden in that idea though. We don't (or didn't in the past AFAIK) publish the names of track chairs prior to conclusion of the talk selection process. The last thing I would want, as a track chair, is to suddenly be faced with hundreds or more E-mails from random speakers and their fans and coworkers asking me to please, please, pretty-please pick their proposal.
I think all track chairs, past and future, will join you in that sentiment. But secret track chairs aren't an option either. I've always published my track chair status. I was nagged on Twitter exactly once. I publicly responded asking the person to please stop, and that I thought their behavior left a less-than-professional impression. They promptly stopped. Cheers, Florian
Excerpts from Jeremy Stanley's message of 2016-05-19 16:23:27 +0000:
On 2016-05-19 09:38:24 -0400 (-0400), Doug Hellmann wrote: [...]
I would rather we have a more public way of selecting track chairs [...]
There's a separate demon hidden in that idea though. We don't (or didn't in the past AFAIK) publish the names of track chairs prior to conclusion of the talk selection process. The last thing I would want, as a track chair, is to suddenly be faced with hundreds or more E-mails from random speakers and their fans and coworkers asking me to please, please, pretty-please pick their proposal.
Sure, it's a balance. I think it would be fair to have a blanket rule that harassing the track chairs could be seen as automatic grounds for rejection, though we would want to define that carefully. Doug
Excerpts from Lana Brindley's message of 2016-05-19 11:23:04 +1000:
On 19/05/16 03:10, Richard Raseley wrote:
On May 17, 2016, at 2:13 PM, Florian Haas <florian@hastexo.com <mailto:florian@hastexo.com>> wrote:
I have previously suggested a different approach: improve the *quality* of voting drastically, rather than abolish it altogether.
Now what follows isn't a voting process of my own design. We owe the idea to a couple of professors of astronomy, who explain their approach in a paper and a couple of interviews, all of which are linked from here:
https://plus.google.com/+FlorianHaas/posts/MsswaBHramG
(Please note: I have deliberately disabled comments on that post, in an attempt to keep the discussion here on this list, where it belongs.)
If you care about the process, please take a few minutes to review at least both videos linked from that post. If you're *really* interested, read the article as well -- but for the purposes of this discussion, the videos should suffice.
Florian,
I appreciate your comments in this thread and agree with them nearly entirely.
I think those who want to completely eliminate the vote, while well intentioned, would be doing a disservice to the community and ultimately OpenStack itself.
I very strongly support the idea of evolving the voting process to try to enhance its value rather than doing away with it entirely. The method you’ve proposed seems like a reasonable place to start.
Regards,
Richard
I'm actually really starting to like Florian's proposed method, with one exception: I don't like the idea of limiting it to talk submitters. The reason I say this is that, before I was a PTL, I submitted talks to every summit. That stopped when I became a PTL because, quite simply, I spend most of my time running sessions in the Design Summit, only popping over to the main conference for the talks I absolutely don't want to miss. I think by limiting the voting to only people submitting talks, you will miss the voice of people who deliberately *don't* submit a talk, because they're massively invested in other aspects of the conference, especially those on the more technical side of the house.
Perhaps, instead of limiting it to talk submitters, maybe make it available to a different subset: people who have attended previously, maybe?
Lana
EuroPython limits voting to conference attendees. I don't know how much the votes factor into talk selection, but I could ask someone on the program committee if that would be useful. Doug
On Thu, May 19, 2016 at 3:30 PM, Doug Hellmann <doug@doughellmann.com> wrote:
EuroPython limits voting to conference attendees. I don't know how much the votes factor into talk selection, but I could ask someone on the program committee if that would be useful.
For clarity: is that _previous_ conference attendees, or do they require people to sign up while the conference program is still completely blank? The latter would seem to me like making people buy the cat in the bag, which would be fine for a grassroots conference but strikes me as a bit inappropriate for the huge event that the Summit has become. Cheers, Florian
Speaking for myself and not necessarily for my team, this sounds interesting despite potentially controversial. I agree that large organizations voting for their colleagues is something start-ups cannot compete against, I also believe that the community voice is important. I'm onboard to experimenting to find the right balance. //adam *Adam Lawson* AQORN, Inc. 427 North Tatnall Street Ste. 58461 Wilmington, Delaware 19801-2230 Toll-free: (844) 4-AQORN-NOW ext. 101 International: +1 302-387-4660 Direct: +1 916-246-2072 On Tue, May 17, 2016 at 12:40 PM, Claire Massey <claire@openstack.org> wrote:
Hi everyone,
For the past few Summits, we've received mixed feedback about having the community vote on proposed sessions as part of the call for speakers process. Historically, after the call for speakers closes, we publish all submitted sessions for community voting before the track chairs review them. The track chairs then choose how much weight to put on the voting resuls, if any, because they make the ultimate decision about which sessions are selected. More info on Track Chairs can be found at the bottom of this email.
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case).
We would like to propose removing voting from the selection process for the October 2016 Barcelona Summit, but want to get your input before making a final decision. Our thinking is that by removing voting from the process, we will:
- Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle - Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes - Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting
We initially started the voting process for good reasons and we do think there's value, but we're reaching a point where the costs are starting to outweigh the benefits. We'd like to get your input before we open the call for speakers in early June for the Barcelona Summit.
Thanks, Claire
Track Chair Info Track Chairs are subject matter experts who review submissions to their particular track, for example "storage" or "cloud applications." There are typically 3-4 chairs per track who review and collaboratively decide which presentations are ultimately accepted for inclusion on the final agenda. The Foundation strives to recruit Track Chairs from a diverse set of companies, regions, roles in the community (i.e., contributing developers, users and business leaders) and areas of expertise. Information on how to nominate yourself or someone else to serve as a track chair for the Barcelona Summit will be published when the call for speakers goes live in early June. For reference, here's information on the Track Chairs from the Austin Summit: https://etherpad.openstack.org/p/Austin_Summit_Track_Chairs and https://www.openstack.org/summit/austin-2016/categories/selection-process.
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On 05/17/2016 12:40 PM, Claire Massey wrote:
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case).
I have always considered the public voting a celebration of the success of the summit and nothing else. It's a ritual for the OpenStack community: twice a year we ask speakers to propose their talks and we celebrate all of the submissions. In this celebration, people from anywhere (the crowd) give their opinions... The rituals and the celebrations contribute to define communities, online and not. I've argued at length that the crowd's votes are a by-product of the celebration and are to be discarded by the track chairs. The crowd has no wisdom in this context, it can and will be easily manipulated (think what happened when Pilates run a popularity contest asking the crowd to vote). Track chairs should be trusted instead, therefore they need to be picked carefully (like they've been).
[...]Our thinking is that by removing voting from the process, we will:
- Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle
These considerations by Foundation staff are very practical and it's hard to counter these. Time is not compressible and it's quite clear that development efforts are already stretching the team thin. My vote is to keep the celebrations pre-summit but not at the expense of dev team's sanity. The executives at the foundation are the best judges for this.
- Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes
Twitter *is* fatigue by another name, honestly... this is a thin objection, can be fixed with little investment from the Foundation staff.
- Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting
This can be easily fixed by recommending the track chairs to ignore the votes (even hide the results from track chair tool) and communicate clearly that the crowd is not 'voting' on anything, just participating in a ritual. It's a small fix to documentation and recommendations to chairs. /stef
On 05/18/2016 12:24 AM, Stefano Maffulli wrote:
On 05/17/2016 12:40 PM, Claire Massey wrote:
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case).
I have always considered the public voting a celebration of the success of the summit and nothing else. It's a ritual for the OpenStack community: twice a year we ask speakers to propose their talks and we celebrate all of the submissions. In this celebration, people from anywhere (the crowd) give their opinions...
The rituals and the celebrations contribute to define communities, online and not.
I've argued at length that the crowd's votes are a by-product of the celebration and are to be discarded by the track chairs.
And for what it's worth I have objected to this and will continue to do so. I believe we agree that track chairs should not consider the community vote to be binding. I believe we disagree, however, in that you're arguing the community vote should be discarded, while I very much believe track chairs should consider it.
The crowd has no wisdom in this context, it can and will be easily manipulated (think what happened when Pilates run a popularity contest asking the crowd to vote).
I agree with the second part of this sentence (unless we're discussing historical accuracy ;) ), but I very much disagree with the first.
Track chairs should be trusted instead, therefore they need to be picked carefully (like they've been).
[...]Our thinking is that by removing voting from the process, we will:
- Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle
These considerations by Foundation staff are very practical and it's hard to counter these. Time is not compressible and it's quite clear that development efforts are already stretching the team thin.
My vote is to keep the celebrations pre-summit but not at the expense of dev team's sanity. The executives at the foundation are the best judges for this.
If I may ask one provocative question. Pretty much every Summit I get an email from a new person at the Foundation I haven't met before, and/or who didn't work for the Foundation the Summit before. So I am guessing that ramping up headcount isn't an issue for the Foundation, neither for funding nor for organizational considerations. [Please correct me if I'm wrong here.] If the dev team is overloaded to the point of insanity, is it really an issue to give them more help? Yes, I've read Brooks. I'm not arguing that hiring (or contracting out to) additional developers will have a short-term positive impact. But if this wasn't addressed after Tokyo to benefit Barcelona, then it should be addressed now, to benefit Boston and Sydney.
This can be easily fixed by recommending the track chairs to ignore the votes (even hide the results from track chair tool) and communicate clearly that the crowd is not 'voting' on anything, just participating in a ritual. It's a small fix to documentation and recommendations to chairs.
Which means that track chairs would have to completely guess what the audience might find interesting. In a community that is constantly growing, diversifying and spreading. Don't you think you're making the track chairs' work significantly more difficult that way, and more importantly, that this is detrimental to the quality of the tracks overall? Cheers, Florian
On 18/05/16 09:14, Florian Haas wrote:
On 05/18/2016 12:24 AM, Stefano Maffulli wrote:
On 05/17/2016 12:40 PM, Claire Massey wrote:
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case). I have always considered the public voting a celebration of the success of the summit and nothing else. It's a ritual for the OpenStack community: twice a year we ask speakers to propose their talks and we celebrate all of the submissions. In this celebration, people from anywhere (the crowd) give their opinions...
The rituals and the celebrations contribute to define communities, online and not.
I've argued at length that the crowd's votes are a by-product of the celebration and are to be discarded by the track chairs. If this is only for publicity - then it could be considered a complete and total waste of time to even start the voting process.
Promote the sessions. Asking people to vote and then throw their efforts and opinion away could actually be considered an insult. For what it is worth - I personally think we should do away with it all together
And for what it's worth I have objected to this and will continue to do so. I believe we agree that track chairs should not consider the community vote to be binding. I believe we disagree, however, in that you're arguing the community vote should be discarded, while I very much believe track chairs should consider it.
The crowd has no wisdom in this context, it can and will be easily manipulated (think what happened when Pilates run a popularity contest asking the crowd to vote). I agree with the second part of this sentence (unless we're discussing historical accuracy ;) ), but I very much disagree with the first.
Track chairs should be trusted instead, therefore they need to be picked carefully (like they've been).
[...]Our thinking is that by removing voting from the process, we will:
- Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle These considerations by Foundation staff are very practical and it's hard to counter these. Time is not compressible and it's quite clear that development efforts are already stretching the team thin.
My vote is to keep the celebrations pre-summit but not at the expense of dev team's sanity. The executives at the foundation are the best judges for this. If I may ask one provocative question. Pretty much every Summit I get an email from a new person at the Foundation I haven't met before, and/or who didn't work for the Foundation the Summit before. So I am guessing that ramping up headcount isn't an issue for the Foundation, neither for funding nor for organizational considerations. [Please correct me if I'm wrong here.] If the dev team is overloaded to the point of insanity, is it really an issue to give them more help?
Yes, I've read Brooks. I'm not arguing that hiring (or contracting out to) additional developers will have a short-term positive impact. But if this wasn't addressed after Tokyo to benefit Barcelona, then it should be addressed now, to benefit Boston and Sydney.
This can be easily fixed by recommending the track chairs to ignore the votes (even hide the results from track chair tool) and communicate clearly that the crowd is not 'voting' on anything, just participating in a ritual. It's a small fix to documentation and recommendations to chairs. Which means that track chairs would have to completely guess what the audience might find interesting. In a community that is constantly growing, diversifying and spreading. Don't you think you're making the track chairs' work significantly more difficult that way, and more importantly, that this is detrimental to the quality of the tracks overall?
Cheers, Florian
-- Best Regards, Maish Saidel-Keesing
On 05/18/2016 12:07 PM, Maish Saidel-Keesing wrote:
On 18/05/16 09:14, Florian Haas wrote:
On 05/18/2016 12:24 AM, Stefano Maffulli wrote:
On 05/17/2016 12:40 PM, Claire Massey wrote:
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case). I have always considered the public voting a celebration of the success of the summit and nothing else. It's a ritual for the OpenStack community: twice a year we ask speakers to propose their talks and we celebrate all of the submissions. In this celebration, people from anywhere (the crowd) give their opinions...
The rituals and the celebrations contribute to define communities, online and not.
I've argued at length that the crowd's votes are a by-product of the celebration and are to be discarded by the track chairs. If this is only for publicity - then it could be considered a complete and total waste of time to even start the voting process.
Promote the sessions. Asking people to vote and then throw their efforts and opinion away could actually be considered an insult.
For what it is worth - I personally think we should do away with it all together
I agree that having a voting system and then ignoring the votes may be insulting. You say you're in favor of nixing voting altogether -- would you mind explaining your rationale? Cheers, Florian
On 18/05/16 11:10, Florian Haas wrote:
On 05/18/2016 12:07 PM, Maish Saidel-Keesing wrote:
On 18/05/16 09:14, Florian Haas wrote:
On 05/18/2016 12:24 AM, Stefano Maffulli wrote:
On 05/17/2016 12:40 PM, Claire Massey wrote:
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case). I have always considered the public voting a celebration of the success of the summit and nothing else. It's a ritual for the OpenStack community: twice a year we ask speakers to propose their talks and we celebrate all of the submissions. In this celebration, people from anywhere (the crowd) give their opinions...
The rituals and the celebrations contribute to define communities, online and not.
I've argued at length that the crowd's votes are a by-product of the celebration and are to be discarded by the track chairs. If this is only for publicity - then it could be considered a complete and total waste of time to even start the voting process.
Promote the sessions. Asking people to vote and then throw their efforts and opinion away could actually be considered an insult.
For what it is worth - I personally think we should do away with it all together
I agree that having a voting system and then ignoring the votes may be insulting. You say you're in favor of nixing voting altogether -- would you mind explaining your rationale?
Cheers, Florian
Let me clarify my statement. In the current format - where the votes are more or less ignored by the track owners and the they are there for 'eye candy' - then I think there is absolutely no benefit in using the voting process. As far as I understand - that is the situation today. The votes have no official weight or influence on the sessions that will be selected. Track chairs have If we can come to an agreed format that the votes DO have say - but NOT the only or main criteria - then it is worthwhile.
From the selection process [1]
Track Chairs will receive a slate of presentations to review and they will determine the final schedule. Community votes are meant to **help inform the decision, but are not the only guide**. Track chairs are expected to exercise judgment in their area of expertise and help ensure diversity. Real-world user stories and in-the-trenches experiences are favored over sales pitches. This is so ambiguous - and completely unclear [1] https://www.openstack.org/summit/austin-2016/categories/selection-process -- Best Regards, Maish Saidel-Keesing
On 05/18/2016 12:54 PM, Maish Saidel-Keesing wrote:
Let me clarify my statement.
In the current format - where the votes are more or less ignored by the track owners and the they are there for 'eye candy' - then I think there is absolutely no benefit in using the voting process.
As far as I understand - that is the situation today. The votes have no official weight or influence on the sessions that will be selected. Track chairs have
If we can come to an agreed format that the votes DO have say - but NOT the only or main criteria - then it is worthwhile.
I see. I proposed an alternative (and in my humble opinion, advantageous) voting scheme upthread — would you mind sharing your opinion on that? Cheers, Florian
On 05/18/2016 03:14 AM, Florian Haas wrote:
Which means that track chairs would have to completely guess what the audience might find interesting.
The crowd doesn't know this either. On 05/18/2016 05:54 AM, Maish Saidel-Keesing wrote:
If we can come to an agreed format that the votes DO have say - but NOT the only or main criteria - then it is worthwhile.
We don't need to ask for votes, since those are not very useful: large and well organized corporations will always get more votes because clicks are too cheap. Instead of votes, we may ask for written comments instead: I'd read those if I were a track chair and undecided about a proposal. I'm not attached to votes per se: what I think we should keep is - the drums beating signaling the incoming summit: these bring visibility - the openness: the fact that I can browse proposed talks is a valuable thing as a speaker and one that sets apart OpenStack from other conferences /stef
On Wed, May 18, 2016 at 3:09 PM, Stefano Maffulli <stefano@openstack.org> wrote:
On 05/18/2016 03:14 AM, Florian Haas wrote:
Which means that track chairs would have to completely guess what the audience might find interesting.
The crowd doesn't know this either.
Are you saying the audience doesn't know what the audience finds interesting? Just asking for clarification here. Cheers, Florian
On 05/18/2016 08:28 AM, Florian Haas wrote:
On Wed, May 18, 2016 at 3:09 PM, Stefano Maffulli <stefano@openstack.org> wrote:
On 05/18/2016 03:14 AM, Florian Haas wrote:
Which means that track chairs would have to completely guess what the audience might find interesting.
The crowd doesn't know this either.
Are you saying the audience doesn't know what the audience finds interesting?
Just asking for clarification here.
As a previous track chair, I find the vote tallies "interesting" - they are more data to inform my decisions. Sometimes they match my assessment of the talk and speaker. However, it often tells me more about the popularity of the speaker (or size of their company) than whether anyone would find the talk interesting at the summit. I am aware that some employers only send technical contributors to the conference if they are speaking; because of this, for some people, voting becomes a means to enable your friends/coworkers to attend the summit. While that is a choice employers are entitled to make for their travel budgets regardless of my opinions of that practice, popularity is clearly not the metric we should be using to gauge talk quality. So, I agree with Stef: the vote results do not actually indicate what the audience finds interesting, because enough people are not voting based on what they would find interesting. --devananda
On Wed, May 18, 2016 at 7:52 PM, Devananda van der Veen <devananda.vdv@gmail.com> wrote:
On 05/18/2016 08:28 AM, Florian Haas wrote:
On Wed, May 18, 2016 at 3:09 PM, Stefano Maffulli <stefano@openstack.org> wrote:
On 05/18/2016 03:14 AM, Florian Haas wrote:
Which means that track chairs would have to completely guess what the audience might find interesting.
The crowd doesn't know this either.
Are you saying the audience doesn't know what the audience finds interesting?
Just asking for clarification here.
As a previous track chair, I find the vote tallies "interesting" - they are more data to inform my decisions. Sometimes they match my assessment of the talk and speaker. However, it often tells me more about the popularity of the speaker (or size of their company) than whether anyone would find the talk interesting at the summit.
I am aware that some employers only send technical contributors to the conference if they are speaking; because of this, for some people, voting becomes a means to enable your friends/coworkers to attend the summit. While that is a choice employers are entitled to make for their travel budgets regardless of my opinions of that practice, popularity is clearly not the metric we should be using to gauge talk quality.
So, I agree with Stef: the vote results do not actually indicate what the audience finds interesting, because enough people are not voting based on what they would find interesting.
OK, that's a very fair point. Thanks. It's a very different angle compared with "the crowd doesn't *know*" (emphasis mine), though. Cheers, Florian
On 5/18/2016 9:09 AM, Stefano Maffulli wrote:
Instead of votes, we may ask for written comments instead: I'd read those if I were a track chair and undecided about a proposal.
I'm not attached to votes per se: what I think we should keep is
- the drums beating signaling the incoming summit: these bring visibility - the openness: the fact that I can browse proposed talks is a valuable thing as a speaker and one that sets apart OpenStack from other conferences
I agree with Stef on all of this. I think we should have the ability for people to leave comments on a proposal, which then also necessitates having all of the talks visible at some point. Yes, there will likely be some spam management to be handled, but I think it's a small price to pay. We may even find that comments are helpful in shaping the talks and improving them prior to the summit. ---- Nick
Excerpts from Nick Chase's message of 2016-05-18 11:57:54 -0400:
On 5/18/2016 9:09 AM, Stefano Maffulli wrote:
Instead of votes, we may ask for written comments instead: I'd read those if I were a track chair and undecided about a proposal.
I'm not attached to votes per se: what I think we should keep is
- the drums beating signaling the incoming summit: these bring visibility - the openness: the fact that I can browse proposed talks is a valuable thing as a speaker and one that sets apart OpenStack from other conferences
I agree with Stef on all of this. I think we should have the ability for people to leave comments on a proposal, which then also necessitates having all of the talks visible at some point. Yes, there will likely be some spam management to be handled, but I think it's a small price to pay.
Given the amount of spam we deal with in our other tools, I think you're underestimating the cost. See the -dev and -infra mailing lists for the discussions of locking down (or shutting down) the wiki, for example. A web form with a text box is only slightly more complicated to game than a voting link. The alternate voting system proposed is interesting, but looks a bit complex to implement (especially given the rules for introducing negative feedback based on apparent attempts to game the vote). It may take us a while to build a system to support it, so I think we want a solution we can put in place more quickly, even if we do decide to try it out. Other conferences I'm involved with have a small program committee for each track. We did that for the Upstream Development track this last time around (maybe other tracks also have multiple chairs, I don't know). Having a group of informed people selecting talks based on the quality of the proposal and the subject matter included produced a track with good feedback from attendees. It seems like that should be able to work for other tracks, too, as long as we have a good balance in the chairs. Doug
We may even find that comments are helpful in shaping the talks and improving them prior to the summit.
---- Nick
On 5/18/2016 1:11 PM, Doug Hellmann wrote:
Given the amount of spam we deal with in our other tools, I think you're underestimating the cost. See the -dev and -infra mailing lists for the discussions of locking down (or shutting down) the wiki, for example.
Remember, though, that the wiki has been around for a long time; this is a tool that will exist for just a week or so, and it takes spammers time to find it. And a Capcha can help there as well.
A web form with a text box is only slightly more complicated to game than a voting link.
Not necessarily; the number of comments is irrelevant. The only comments that really will affect anything are those that are substantive with regards to the content of the proposal, and that's not something that you can automate. In other words we can specify that we're looking for more than just "This is great" or "I would go to this".
Other conferences I'm involved with have a small program committee for each track. We did that for the Upstream Development track this last time around (maybe other tracks also have multiple chairs, I don't know). Having a group of informed people selecting talks based on the quality of the proposal and the subject matter included produced a track with good feedback from attendees. It seems like that should be able to work for other tracks, too, as long as we have a good balance in the chairs.
They were all like that. We wrote about another suggestion here: https://www.mirantis.com/blog/fixing-openstack-summit-submission-process/ (Look after "... and here's mine".) I know there there some plusses and minuses in the interviews we did about it, but here it is: Borodaenko suggests a more radical change. “I think we should go more in the direction used by the scientific community and more mature open source communities such as the linux kernel.” The process, he explained, works like this: 1. All submissions are made privately; they cannot be disclosed until after the selection process is over, so there’s no campaigning, and no biasing of the judges. 2. The Peer Review panel is made up of a much larger number of people, and it’s known who they are, but not who reviewed what. So instead of 3 people reviewing all 300 submissions for a single track, you might have 20 people for each track, each of whom review a set of randomly selected submissions. So in this case, if each of those submissions was reviewed by 3 judges, that’s only 45 per person, rather than 300. 3. Judges are randomly assigned proposals, which have all identifying information stripped out. The system will know not to give a judge a proposal from his/her own company. 4. Judges score each proposal on content (is it an interesting topic?), fit for the conference (should we cover this topic at the Summit?), and presentation (does it look like it’s been well thought out and will be presented well?). These scores are used to determine which presentations get in. 5. Proposal authors get back the scores, and the explanations. In an ideal world, authors have a chance to appeal and resubmit with improvements based on the comments to be rescored as in steps 3 and 4, but even if there’s not enough time for that, authors will have a better idea of why they did or didn’t get in, and can use that feedback to create better submissions for next time. 6. Scores determine which proposals get in, potentially with a final step where a set of publicly known individuals reviews the top scorers to make sure that we don’t wind up with 100 sessions on the same topic, but still, the scores should be the final arbiters between whether one proposal or the other is accepted. “So in the end,” he explained, “it’s about the content of the proposal, and not who you work for or who knows you or doesn’t know you. Scientific conferences have been doing it this way for many years, and it works very well.” ---- Nick
On Wed, May 18, 2016 at 7:29 PM, Nick Chase <nchase@mirantis.com> wrote:
We wrote about another suggestion here: https://www.mirantis.com/blog/fixing-openstack-summit-submission-process/ (Look after "... and here's mine".) I know there there some plusses and minuses in the interviews we did about it, but here it is:
Borodaenko suggests a more radical change. “I think we should go more in the direction used by the scientific community and more mature open source communities such as the linux kernel.” The process, he explained, works like this:
All submissions are made privately; they cannot be disclosed until after the selection process is over, so there’s no campaigning, and no biasing of the judges. The Peer Review panel is made up of a much larger number of people, and it’s known who they are, but not who reviewed what. So instead of 3 people reviewing all 300 submissions for a single track, you might have 20 people for each track, each of whom review a set of randomly selected submissions. So in this case, if each of those submissions was reviewed by 3 judges, that’s only 45 per person, rather than 300. Judges are randomly assigned proposals, which have all identifying information stripped out. The system will know not to give a judge a proposal from his/her own company. Judges score each proposal on content (is it an interesting topic?), fit for the conference (should we cover this topic at the Summit?), and presentation (does it look like it’s been well thought out and will be presented well?). These scores are used to determine which presentations get in. Proposal authors get back the scores, and the explanations. In an ideal world, authors have a chance to appeal and resubmit with improvements based on the comments to be rescored as in steps 3 and 4, but even if there’s not enough time for that, authors will have a better idea of why they did or didn’t get in, and can use that feedback to create better submissions for next time. Scores determine which proposals get in, potentially with a final step where a set of publicly known individuals reviews the top scorers to make sure that we don’t wind up with 100 sessions on the same topic, but still, the scores should be the final arbiters between whether one proposal or the other is accepted.
“So in the end,” he explained, “it’s about the content of the proposal, and not who you work for or who knows you or doesn’t know you. Scientific conferences have been doing it this way for many years, and it works very well.”
This has some characteristics that are congruent with my own proposal (random selection of a subset being offered to reviewers being one of them), which I obviously agree with. Others, not so much. - While I love the idea of giving detailed feedback and possibly even an option to refine and resubmit talks, even for "just" 45 talks that is a daunting task that would likely keep a reviewer occupied for a week, if the feedback is expected to be any good. - Anonymizing submissions is a double-edged sword. Yes, of course it removes all bias related to an individual speaker's person or company affiliation. However, when you talk to attendees of technical conferences, most people will much prefer to listen to a knowledgeable person who is also a good presenter, than to *the* best expert in their field who can't present worth a damn. Judging the quality of the oral presentation from the written submission is impossible (not least because there is no way of telling whether the submission was written by the presenter), and past speaker performance is a good indicator of future speaker performance. Anonymizing submissions means the latter is ignored, and the former will be attempted. If we determine the benefits to outweigh the downsides that's fine, but this downside should be considered. Then again, at the risk of repeating myself, why not take this to its logical conclusion and make the review "panel" identical to the group of submitters, in other words, with submitting a talk comes the responsibility to review others? This also has the added benefit that rather than divvying up talks with every talk being reviewed by just one (naturally flawed) reviewer, you get multiple eyeballs on each and every talk. Cheers, Florian
On 05/18/2016 10:29 AM, Nick Chase wrote:
On 5/18/2016 1:11 PM, Doug Hellmann wrote:
Given the amount of spam we deal with in our other tools, I think you're underestimating the cost. See the -dev and -infra mailing lists for the discussions of locking down (or shutting down) the wiki, for example.
Remember, though, that the wiki has been around for a long time; this is a tool that will exist for just a week or so, and it takes spammers time to find it. And a Capcha can help there as well.
A web form with a text box is only slightly more complicated to game than a voting link.
Not necessarily; the number of comments is irrelevant. The only comments that really will affect anything are those that are substantive with regards to the content of the proposal, and that's not something that you can automate. In other words we can specify that we're looking for more than just "This is great" or "I would go to this".
Other conferences I'm involved with have a small program committee for each track. We did that for the Upstream Development track this last time around (maybe other tracks also have multiple chairs, I don't know). Having a group of informed people selecting talks based on the quality of the proposal and the subject matter included produced a track with good feedback from attendees. It seems like that should be able to work for other tracks, too, as long as we have a good balance in the chairs.
They were all like that.
We wrote about another suggestion here: https://www.mirantis.com/blog/fixing-openstack-summit-submission-process/ (Look after "... and here's mine".) I know there there some plusses and minuses in the interviews we did about it, but here it is:
Borodaenko suggests a more radical change. “I think we should go more in the direction used by the scientific community and more mature open source communities such as the linux kernel.”
Thanks for the summary and reposting here, Nick. I think this direction is good. Not that you're suggesting it, but just to be clear, I don't think we should simply copy what works for other communities. After all, those communities function differently than OpenStack in certain ways and with good reasons, so what works for them may be different than what works for us.
The process, he explained, works like this:
1. All submissions are made privately; they cannot be disclosed until after the selection process is over, so there’s no campaigning, and no biasing of the judges.
+1
2. The Peer Review panel is made up of a much larger number of people, and it’s known who they are, but not who reviewed what. So instead of 3 people reviewing all 300 submissions for a single track, you might have 20 people for each track, each of whom review a set of randomly selected submissions. So in this case, if each of those submissions was reviewed by 3 judges, that’s only 45 per person, rather than 300.
-1 As a former track chair, I value the collaboration with the other track chairs during the review process. Also, I would continue the practice of grouping track chairs with subject matter relevant to them. Don't ask me to review presentations on storage systems, for example, because I won't know what's relevant to that audience -- I'll probably find it all equally interesting and evaluate only based on how well the proposal is written.
3. Judges are randomly assigned proposals, which have all identifying information stripped out.
-1 (see above) Also, -1 because I believe that speaker quality is important, as is the proximity of the speaker to the subject matter they're presenting. It is not possible to assess either of these from an anonymized abstract. Rather, I trust the Foundation to be selecting track chairs who are conscientious, aware of who the known (good and bad) actors are within a given subject matter, and will perform this duty respectfully and impartially.
The system will know not to give a judge a proposal from his/her own company. 4. Judges score each proposal on content (is it an interesting topic?), fit for the conference (should we cover this topic at the Summit?), and presentation (does it look like it’s been well thought out and will be presented well?). These scores are used to determine which presentations get in.
Those first two criteria are fine, but the latter requires that the judges see the talk material (not just the abstract) ahead of time, and I do not want to require that. It would mean that I would never be able to give a talk at the summit -- there's no way I'm writing the slides 3 months in advance, and if I did, you won't want to attend my talk 'cause it would already be on slideshare/youtube/whatever. This applies to many of the speakers whose talks I want to attend, too. While I do think there is value in normalizing the criteria that track chairs use, and making that criteria visible to presenters, I feel like the Foundation has done a good job with this so far.
5. Proposal authors get back the scores, and the explanations. In an ideal world, authors have a chance to appeal and resubmit with improvements based on the comments to be rescored as in steps 3 and 4, but even if there’s not enough time for that, authors will have a better idea of why they did or didn’t get in, and can use that feedback to create better submissions for next time.
A very big +1 to this
6. Scores determine which proposals get in, potentially with a final step where a set of publicly known individuals reviews the top scorers to make sure that we don’t wind up with 100 sessions on the same topic, but still, the scores should be the final arbiters between whether one proposal or the other is accepted.
“So in the end,” he explained, “it’s about the content of the proposal, and not who you work for or who knows you or doesn’t know you. Scientific conferences have been doing it this way for many years, and it works very well.”
Regards, Devananda
On 5/18/2016 2:41 PM, Devananda van der Veen wrote:
On 05/18/2016 10:29 AM, Nick Chase wrote:
We wrote about another suggestion here: https://www.mirantis.com/blog/fixing-openstack-summit-submission-process/ (Look after "... and here's mine".) I know there there some plusses and minuses in the interviews we did about it, but here it is:
Borodaenko suggests a more radical change. “I think we should go more in the direction used by the scientific community and more mature open source communities such as the linux kernel.” Thanks for the summary and reposting here, Nick. I think this direction is good.
Not that you're suggesting it, but just to be clear, I don't think we should simply copy what works for other communities. After all, those communities function differently than OpenStack in certain ways and with good reasons, so what works for them may be different than what works for us.
The process, he explained, works like this:
1. All submissions are made privately; they cannot be disclosed until after the selection process is over, so there’s no campaigning, and no biasing of the judges. +1
2. The Peer Review panel is made up of a much larger number of people, and it’s known who they are, but not who reviewed what. So instead of 3 people reviewing all 300 submissions for a single track, you might have 20 people for each track, each of whom review a set of randomly selected submissions. So in this case, if each of those submissions was reviewed by 3 judges, that’s only 45 per person, rather than 300. -1
As a former track chair, I value the collaboration with the other track chairs during the review process.
Also, I would continue the practice of grouping track chairs with subject matter relevant to them. Don't ask me to review presentations on storage systems, for example, because I won't know what's relevant to that audience -- I'll probably find it all equally interesting and evaluate only based on how well the proposal is written.
Actually, since we published that, I have been a track chair as well, and I agree with you on that.
3. Judges are randomly assigned proposals, which have all identifying information stripped out. -1 (see above)
Also, -1 because I believe that speaker quality is important, as is the proximity of the speaker to the subject matter they're presenting. It is not possible to assess either of these from an anonymized abstract.
Rather, I trust the Foundation to be selecting track chairs who are conscientious, aware of who the known (good and bad) actors are within a given subject matter, and will perform this duty respectfully and impartially.
Agreed.
The system will know not to give a judge a
proposal from his/her own company. 4. Judges score each proposal on content (is it an interesting topic?), fit for the conference (should we cover this topic at the Summit?), and presentation (does it look like it’s been well thought out and will be presented well?). These scores are used to determine which presentations get in.
Those first two criteria are fine, but the latter requires that the judges see the talk material (not just the abstract) ahead of time, and I do not want to require that. It would mean that I would never be able to give a talk at the summit -- there's no way I'm writing the slides 3 months in advance, and if I did, you won't want to attend my talk 'cause it would already be on slideshare/youtube/whatever. This applies to many of the speakers whose talks I want to attend, too.
While I do think there is value in normalizing the criteria that track chairs use, and making that criteria visible to presenters, I feel like the Foundation has done a good job with this so far.
5. Proposal authors get back the scores, and the explanations. In an ideal world, authors have a chance to appeal and resubmit with improvements based on the comments to be rescored as in steps 3 and 4, but even if there’s not enough time for that, authors will have a better idea of why they did or didn’t get in, and can use that feedback to create better submissions for next time. A very big +1 to this
6. Scores determine which proposals get in, potentially with a final step where a set of publicly known individuals reviews the top scorers to make sure that we don’t wind up with 100 sessions on the same topic, but still, the scores should be the final arbiters between whether one proposal or the other is accepted.
“So in the end,” he explained, “it’s about the content of the proposal, and not who you work for or who knows you or doesn’t know you. Scientific conferences have been doing it this way for many years, and it works very well.”
On 2016-05-18 11:54:06 +0100 (+0100), Maish Saidel-Keesing wrote: [...]
From the selection process [1]
Track Chairs will receive a slate of presentations to review and they will determine the final schedule. Community votes are meant to **help inform the decision, but are not the only guide**. Track chairs are expected to exercise judgment in their area of expertise and help ensure diversity. Real-world user stories and in-the-trenches experiences are favored over sales pitches.
This is so ambiguous - and completely unclear [...]
As a chair emeritus from several past summits, I feel like ambiguity is the only way this can work. There is data provided to the chairs, which they can choose to use in helping them make their collective decisions as to which talks are selected. Providing chairs with selection rules undermines any faith in their ability to do the job, and at that point you might as well get rid of track chairs entirely. -- Jeremy Stanley
Hi, Personally, I have always found voting to be exclusionary in its nature - it is a popularity contest where those with the broadest reach get more and better votes - and tacky ("vote for my talk!", or worse, "vote for my employer's talk proposals!" tweets are uncouth at best, actively damaging to community identity at worst). Certainly, voting can help eliminate some options - out of laziness, we have considered only the top 30 talks out of 60 proposals for 8 talk slots in a past conference, and 5 of the talks were voted in the top 8. But in general, I do not see a lot of alignment between what makes the best content and what gets the most/best votes. Also, as a presenter, I have never felt comfortable in the "pimp my talk" zone - and I'm pretty extroverted. I can only imagine that having to "sell" your proposal to the community is even more uncomfortable for others - especially those new to our community - so, as I say above, I see the practice as exclusionary and intimidating. I'm all in favour of dropping it, as I have said for the past 3 summits. Thanks, Dave. On 05/17/2016 06:24 PM, Stefano Maffulli wrote:
On 05/17/2016 12:40 PM, Claire Massey wrote:
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case).
I have always considered the public voting a celebration of the success of the summit and nothing else. It's a ritual for the OpenStack community: twice a year we ask speakers to propose their talks and we celebrate all of the submissions. In this celebration, people from anywhere (the crowd) give their opinions...
The rituals and the celebrations contribute to define communities, online and not.
I've argued at length that the crowd's votes are a by-product of the celebration and are to be discarded by the track chairs. The crowd has no wisdom in this context, it can and will be easily manipulated (think what happened when Pilates run a popularity contest asking the crowd to vote). Track chairs should be trusted instead, therefore they need to be picked carefully (like they've been).
[...]Our thinking is that by removing voting from the process, we will:
- Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle
These considerations by Foundation staff are very practical and it's hard to counter these. Time is not compressible and it's quite clear that development efforts are already stretching the team thin.
My vote is to keep the celebrations pre-summit but not at the expense of dev team's sanity. The executives at the foundation are the best judges for this.
- Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes
Twitter *is* fatigue by another name, honestly... this is a thin objection, can be fixed with little investment from the Foundation staff.
- Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting
This can be easily fixed by recommending the track chairs to ignore the votes (even hide the results from track chair tool) and communicate clearly that the crowd is not 'voting' on anything, just participating in a ritual. It's a small fix to documentation and recommendations to chairs.
/stef
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
-- Dave Neary - NFV/SDN Community Strategy Open Source and Standards, Red Hat - http://community.redhat.com Ph: +1-978-399-2182 / Cell: +1-978-799-3338
Hi, On Wed, May 18, 2016 at 6:26 PM, Dave Neary <dneary@redhat.com> wrote:
Hi,
Personally, I have always found voting to be exclusionary in its nature - it is a popularity contest where those with the broadest reach get more and better votes - and tacky ("vote for my talk!", or worse, "vote for my employer's talk proposals!" tweets are uncouth at best, actively damaging to community identity at worst).
Certainly, voting can help eliminate some options - out of laziness, we have considered only the top 30 talks out of 60 proposals for 8 talk slots in a past conference, and 5 of the talks were voted in the top 8. But in general, I do not see a lot of alignment between what makes the best content and what gets the most/best votes. Also, as a presenter, I have never felt comfortable in the "pimp my talk" zone - and I'm pretty extroverted. I can only imagine that having to "sell" your proposal to the community is even more uncomfortable for others - especially those new to our community - so, as I say above, I see the practice as exclusionary and intimidating.
I'm all in favour of dropping it, as I have said for the past 3 summits.
+1 to Dave
Thanks, Dave.
On 05/17/2016 06:24 PM, Stefano Maffulli wrote:
On 05/17/2016 12:40 PM, Claire Massey wrote:
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case).
I have always considered the public voting a celebration of the success of the summit and nothing else. It's a ritual for the OpenStack community: twice a year we ask speakers to propose their talks and we celebrate all of the submissions. In this celebration, people from anywhere (the crowd) give their opinions...
The rituals and the celebrations contribute to define communities, online and not.
I've argued at length that the crowd's votes are a by-product of the celebration and are to be discarded by the track chairs. The crowd has no wisdom in this context, it can and will be easily manipulated (think what happened when Pilates run a popularity contest asking the crowd to vote). Track chairs should be trusted instead, therefore they need to be picked carefully (like they've been).
[...]Our thinking is that by removing voting from the process, we will:
- Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle
These considerations by Foundation staff are very practical and it's hard to counter these. Time is not compressible and it's quite clear that development efforts are already stretching the team thin.
My vote is to keep the celebrations pre-summit but not at the expense of dev team's sanity. The executives at the foundation are the best judges for this.
- Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes
Twitter *is* fatigue by another name, honestly... this is a thin objection, can be fixed with little investment from the Foundation staff.
- Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting
This can be easily fixed by recommending the track chairs to ignore the votes (even hide the results from track chair tool) and communicate clearly that the crowd is not 'voting' on anything, just participating in a ritual. It's a small fix to documentation and recommendations to chairs.
/stef
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
-- Dave Neary - NFV/SDN Community Strategy Open Source and Standards, Red Hat - http://community.redhat.com Ph: +1-978-399-2182 / Cell: +1-978-799-3338
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Best, -- Atul Jha www.atuljha.com @ koolhead17 <https://twitter.com/koolhead17>
On Wed, May 18, 2016 at 2:56 PM, Dave Neary <dneary@redhat.com> wrote:
Hi,
Personally, I have always found voting to be exclusionary in its nature - it is a popularity contest where those with the broadest reach get more and better votes - and tacky ("vote for my talk!", or worse, "vote for my employer's talk proposals!" tweets are uncouth at best, actively damaging to community identity at worst).
Certainly, voting can help eliminate some options - out of laziness, we have considered only the top 30 talks out of 60 proposals for 8 talk slots in a past conference, and 5 of the talks were voted in the top 8. But in general, I do not see a lot of alignment between what makes the best content and what gets the most/best votes. Also, as a presenter, I have never felt comfortable in the "pimp my talk" zone - and I'm pretty extroverted. I can only imagine that having to "sell" your proposal to the community is even more uncomfortable for others - especially those new to our community - so, as I say above, I see the practice as exclusionary and intimidating.
I don't disagree, but I do maintain that if we move this to a peer ranking scheme, where only those how submit talks get to review other submissions, the exclusion/intimidation aspect would likely vanish. If people are no longer offered the whole slew of talks, but only a small random subset thereof, we get better review coverage and the task is a lot less daunting than it is now. In fact, with a randomized subset peer-review scheme like that, we could even drop track chairs, which would completely remove any popularity contest effect from track chair selection as well, and would completely nix the risk of track chair lobbying. You submit a talk, you have a say in the overall program. Simple as that. Kind of like you submit patches, you have a say in the direction your project is taking. Cheers, Florian
You submit a talk, you have a say in the overall program. Simple as that. Kind of like you submit patches, you have a say in the direction your project is taking.
The challenge there my friend is it incentivizes the community to submit talks solely for the purpose of gaining influence within the selection process. Kind of like submitting patches for spelling/punctuation in the OpenStack documentation solely for the purpose gaining voting rights during TC elections and ATC status -- hence free Summit pass. No easy answers for sure. I'm not opinionated strongly either way but would support testing the idea of eliminating voting in Barcelona to see how it goes. I'm guessing that if it does not go well or have unexpected social consequence, the Foundation will have ample opportunity to course-correct. //adam *Adam Lawson* AQORN, Inc. 427 North Tatnall Street Ste. 58461 Wilmington, Delaware 19801-2230 Toll-free: (844) 4-AQORN-NOW ext. 101 International: +1 302-387-4660 Direct: +1 916-246-2072 On Wed, May 18, 2016 at 8:25 AM, Florian Haas <florian@hastexo.com> wrote:
On Wed, May 18, 2016 at 2:56 PM, Dave Neary <dneary@redhat.com> wrote:
Hi,
Personally, I have always found voting to be exclusionary in its nature - it is a popularity contest where those with the broadest reach get more and better votes - and tacky ("vote for my talk!", or worse, "vote for my employer's talk proposals!" tweets are uncouth at best, actively damaging to community identity at worst).
Certainly, voting can help eliminate some options - out of laziness, we have considered only the top 30 talks out of 60 proposals for 8 talk slots in a past conference, and 5 of the talks were voted in the top 8. But in general, I do not see a lot of alignment between what makes the best content and what gets the most/best votes. Also, as a presenter, I have never felt comfortable in the "pimp my talk" zone - and I'm pretty extroverted. I can only imagine that having to "sell" your proposal to the community is even more uncomfortable for others - especially those new to our community - so, as I say above, I see the practice as exclusionary and intimidating.
I don't disagree, but I do maintain that if we move this to a peer ranking scheme, where only those how submit talks get to review other submissions, the exclusion/intimidation aspect would likely vanish. If people are no longer offered the whole slew of talks, but only a small random subset thereof, we get better review coverage and the task is a lot less daunting than it is now.
In fact, with a randomized subset peer-review scheme like that, we could even drop track chairs, which would completely remove any popularity contest effect from track chair selection as well, and would completely nix the risk of track chair lobbying.
You submit a talk, you have a say in the overall program. Simple as that. Kind of like you submit patches, you have a say in the direction your project is taking.
Cheers, Florian
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On 05/18/2016 05:45 PM, Adam Lawson wrote:
You submit a talk, you have a say in the overall program. Simple as that. Kind of like you submit patches, you have a say in the direction your project is taking.
The challenge there my friend is it incentivizes the community to submit talks solely for the purpose of gaining influence within the selection process. Kind of like submitting patches for spelling/punctuation in the OpenStack documentation solely for the purpose gaining voting rights during TC elections and ATC status -- hence free Summit pass.
Fair point, which I addressed in my original message. Relevant quote below:
When I first floated this idea on the track chair mailing list a few months ago, Duncan Thomas made this point:
[Duncan]
I think limiting votes only to people who submit talks would lead to people/companies submitting poor talks just to get a vote (gaming the system).
To which I then replied:
[Florian]
That's a fair point. However, reviewers could separately flag proposals that don't meet certain quality criteria. (*Some* formal criteria could even be checked by computers, not humans.) And there could be a rule that if, say, the majority of a talk's (anonymous) reviewers flag foul play, all the proposer's proposals *and* all and the proposer's votes would be invalidated. I think that would be a fairly strong deterrent. And in order to deter abuse of *that* system, the event of a proposer being thus sin-binned should probably be reviewed by a panel of some description.
No easy answers for sure. I'm not opinionated strongly either way but would support testing the idea of eliminating voting in Barcelona to see how it goes. I'm guessing that if it does not go well or have unexpected social consequence, the Foundation will have ample opportunity to course-correct.
One technical point on that, "testing" would imply that there is data from Austin that could be compared to data from Barcelona, to judge the relative quality of the two Summits. I'm not sure if we even have sufficient data (attendee feedback) from Austin and Tokyo to be able to determine whether quality improves in Barcelona. If others think voting is so awful right now that rather than having another summit with the current system and then switching to another, it is a better idea to ditch voting now and possibly re-adopt it later, then I'll be happy to accept that. I, for one, am not fond of letting go of community involvement, for concern of never getting it back. Cheers, Florian
I don't know if the voice of somebody very new to the OpenStack community has any value, but as somebody who is indeed new I wanted to put in my support for the ending of voting for speaking sessions. The amount of sheer spam I see for the couple days or a week (honestly not sure) asking people to vote for this session or that session is frustrating. I've had to mute or unfollow a number of social media accounts in the past because of the bombardment. I don't know what the better way is, but there's got to be one. On Wed, May 18, 2016 at 8:58 AM, Florian Haas <florian@hastexo.com> wrote:
On 05/18/2016 05:45 PM, Adam Lawson wrote:
You submit a talk, you have a say in the overall program. Simple as that. Kind of like you submit patches, you have a say in the
direction
your project is taking.
The challenge there my friend is it incentivizes the community to submit talks solely for the purpose of gaining influence within the selection process. Kind of like submitting patches for spelling/punctuation in the OpenStack documentation solely for the purpose gaining voting rights during TC elections and ATC status -- hence free Summit pass.
Fair point, which I addressed in my original message. Relevant quote below:
When I first floated this idea on the track chair mailing list a few months ago, Duncan Thomas made this point:
[Duncan]
I think limiting votes only to people who submit talks would lead to people/companies submitting poor talks just to get a vote (gaming the system).
To which I then replied:
[Florian]
That's a fair point. However, reviewers could separately flag proposals that don't meet certain quality criteria. (*Some* formal criteria could even be checked by computers, not humans.) And there could be a rule that if, say, the majority of a talk's (anonymous) reviewers flag foul play, all the proposer's proposals *and* all and the proposer's votes would be invalidated. I think that would be a fairly strong deterrent. And in order to deter abuse of *that* system, the event of a proposer being thus sin-binned should probably be reviewed by a panel of some description.
No easy answers for sure. I'm not opinionated strongly either way but would support testing the idea of eliminating voting in Barcelona to see how it goes. I'm guessing that if it does not go well or have unexpected social consequence, the Foundation will have ample opportunity to course-correct.
One technical point on that, "testing" would imply that there is data from Austin that could be compared to data from Barcelona, to judge the relative quality of the two Summits. I'm not sure if we even have sufficient data (attendee feedback) from Austin and Tokyo to be able to determine whether quality improves in Barcelona.
If others think voting is so awful right now that rather than having another summit with the current system and then switching to another, it is a better idea to ditch voting now and possibly re-adopt it later, then I'll be happy to accept that.
I, for one, am not fond of letting go of community involvement, for concern of never getting it back.
Cheers, Florian
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
-- Chris Jones, JNCIE-ENT #272 SDN Engineer www.sdnessentials.com 858-888-0373 (cell) E-Mail: chris@sdnessentials.com
On 05/18/2016 06:12 PM, Chris Jones wrote:
I don't know if the voice of somebody very new to the OpenStack community has any value, but as somebody who is indeed new I wanted to put in my support for the ending of voting for speaking sessions. The amount of sheer spam I see for the couple days or a week (honestly not sure) asking people to vote for this session or that session is frustrating. I've had to mute or unfollow a number of social media accounts in the past because of the bombardment.
I don't know what the better way is, but there's got to be one.
I have been arguing in favor of one that would completely eliminate what you're rightfully complaining about. Cheers, Florian
We do look at the voting popularity but I agree that this is not used as a major factor in our decision due to the "data noise". Should we consider setting a rule for voting? For example, people from the same company is not eligible to vote for the sessions which are "solely presented" by their company (i.e. All presenter from the same company). However, he/she is eligible to vote for that session if that session is a joint-presentation between different companies. -----Original Message----- From: Florian Haas [mailto:florian@hastexo.com] Sent: Wednesday, May 18, 2016 9:40 AM To: Chris Jones <chris@sdnessentials.com> Cc: community@lists.openstack.org Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit On 05/18/2016 06:12 PM, Chris Jones wrote:
I don't know if the voice of somebody very new to the OpenStack community has any value, but as somebody who is indeed new I wanted to put in my support for the ending of voting for speaking sessions. The amount of sheer spam I see for the couple days or a week (honestly not sure) asking people to vote for this session or that session is frustrating. I've had to mute or unfollow a number of social media accounts in the past because of the bombardment.
I don't know what the better way is, but there's got to be one.
I have been arguing in favor of one that would completely eliminate what you're rightfully complaining about. Cheers, Florian _______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
I, for one, am not fond of letting go of community involvement, for concern of never getting it back.
Cheers, Florian
I think that's a fair point, and another reason to enable comments instead of votes. People can not only feel involved but they will be making a more substantial contribution to the process. Granted, this may not solve the "spam" issue, though, as people will still be soliciting comments. However it will be more clear that getting comments doesn't mean getting in, so there's less incentive. ---- Nick
On Wed, May 18, 2016 at 6:52 PM, Nick Chase <nchase@mirantis.com> wrote:
I, for one, am not fond of letting go of community involvement, for concern of never getting it back.
Cheers, Florian
I think that's a fair point, and another reason to enable comments instead of votes. People can not only feel involved but they will be making a more substantial contribution to the process.
Granted, this may not solve the "spam" issue, though, as people will still be soliciting comments. However it will be more clear that getting comments doesn't mean getting in, so there's less incentive.
---- Nick
So we currently have a fairly massive volume of submissions every Summit. I'm trying to put myself in the shoes of J. Random Summit Attendee here, right after the CfP closes. What would be my motivation for commenting on a talk that I don't even know for sure will make it into the schedule? Content suggestions like "could you please talk about X as well" or "could you spend some time on Y" are great *after* the talk is confirmed. But while it's still being considered, the logical thing is to say something like "I'd love to see this" or "I'd not attend this" — which is exactly what we have with voting, now. Thoughts? Cheers, Florian
On 05/18/2016 11:24 AM, Florian Haas wrote:
On Wed, May 18, 2016 at 6:52 PM, Nick Chase <nchase@mirantis.com> wrote:
I, for one, am not fond of letting go of community involvement, for concern of never getting it back.
Cheers, Florian
I think that's a fair point, and another reason to enable comments instead of votes. People can not only feel involved but they will be making a more substantial contribution to the process.
Granted, this may not solve the "spam" issue, though, as people will still be soliciting comments. However it will be more clear that getting comments doesn't mean getting in, so there's less incentive.
---- Nick
So we currently have a fairly massive volume of submissions every Summit. I'm trying to put myself in the shoes of J. Random Summit Attendee here, right after the CfP closes. What would be my motivation for commenting on a talk that I don't even know for sure will make it into the schedule? Content suggestions like "could you please talk about X as well" or "could you spend some time on Y" are great *after* the talk is confirmed. But while it's still being considered, the logical thing is to say something like "I'd love to see this" or "I'd not attend this" — which is exactly what we have with voting, now.
Thoughts?
Cheers, Florian
Given the massive volume of proposals, and the time pressure we all feel, I do not expect J. Random Summit Attendee to read / vote on / review all of the proposals. At best, they might look at all the proposals to the project(s) they are a contributor to, vote on them, and wonder if the track chairs will feel the same. But the situation we're in, I think, is that folks only vote on the talks that surface within their social network. I question whether that kind of limited exposure can allow for valuable feedback to be given to track chairs, even if we change the quality of the feedback from a "rate between 0 and 3" to a "fill in this text box". So, I think I agree with you on this, Florian. --devananda
Ya know, there's actually one fairly simple way we can avoid the "vote for me" spam: Don't have static URLs for the talks. I know that sounds a little crazy, but basically what I'm saying is that you make it so that you can go to the website and choose a category, but then the actual individual talks are linked dynamically so there's no way to send out a "Vote for this <link>" tweet. Instead you'd have to say, "Please vote for 'My talk' in the 'My track' category at <main link>" and I think few people are going to do that. Just a thought. ---- Nick
On Wed, May 18, 2016 at 12:50 PM, Nick Chase <nchase@mirantis.com> wrote:
I know that sounds a little crazy, but basically what I'm saying is that you make it so that you can go to the website and choose a category, but then the actual individual talks are linked dynamically so there's no way to send out a "Vote for this <link>" tweet. Instead you'd have to say, "Please vote for 'My talk' in the 'My track' category at <main link>" and I think few people are going to do that.
Just a thought.
Brilliant.
This is an intriguing idea IMHO On May 18, 2016 1:51:17 PM Nick Chase <nchase@mirantis.com> wrote:
Ya know, there's actually one fairly simple way we can avoid the "vote for me" spam:
Don't have static URLs for the talks.
I know that sounds a little crazy, but basically what I'm saying is that you make it so that you can go to the website and choose a category, but then the actual individual talks are linked dynamically so there's no way to send out a "Vote for this <link>" tweet. Instead you'd have to say, "Please vote for 'My talk' in the 'My track' category at <main link>" and I think few people are going to do that.
Just a thought.
---- Nick
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
How about a good old fashioned barrel draw?!?! Sure we might get a few randoms, but they'd make for some entertaining interludes between everything else. Cheers Tristan
-----Original Message----- From: Mark Collier [mailto:mark@openstack.org] Sent: Thursday, 19 May 2016 12:10 PM To: Nick Chase <nchase@mirantis.com>; Devananda van der Veen <devananda.vdv@gmail.com>; community@lists.openstack.org Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
This is an intriguing idea IMHO
On May 18, 2016 1:51:17 PM Nick Chase <nchase@mirantis.com> wrote:
Ya know, there's actually one fairly simple way we can avoid the "vote for me" spam:
Don't have static URLs for the talks.
I know that sounds a little crazy, but basically what I'm saying is that you make it so that you can go to the website and choose a category, but then the actual individual talks are linked dynamically so there's no way to send out a "Vote for this <link>" tweet. Instead you'd have to say, "Please vote for 'My talk' in the 'My track' category at <main link>" and I think few people are going to do that.
Just a thought.
---- Nick
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
I was going to say, "Tristan, I'm serious" but actually a small track of randomly chosen talks could be fun. :) But that's pretty far off topic. ----- Nick On 5/18/2016 10:16 PM, Tristan Goode wrote:
How about a good old fashioned barrel draw?!?!
Sure we might get a few randoms, but they'd make for some entertaining interludes between everything else.
Cheers Tristan
-----Original Message----- From: Mark Collier [mailto:mark@openstack.org] Sent: Thursday, 19 May 2016 12:10 PM To: Nick Chase <nchase@mirantis.com>; Devananda van der Veen <devananda.vdv@gmail.com>; community@lists.openstack.org Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
This is an intriguing idea IMHO
On May 18, 2016 1:51:17 PM Nick Chase <nchase@mirantis.com> wrote:
Ya know, there's actually one fairly simple way we can avoid the "vote for me" spam:
Don't have static URLs for the talks.
I know that sounds a little crazy, but basically what I'm saying is that you make it so that you can go to the website and choose a category, but then the actual individual talks are linked dynamically so there's no way to send out a "Vote for this <link>" tweet. Instead you'd have to say, "Please vote for 'My talk' in the 'My track' category at <main link>" and I think few people are going to do that.
Just a thought.
---- Nick
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
The OpenStack "Grab Bag" track. Gary Kevorkian MARKETING COMMUNICATIONS MANAGER - EVENTS Marketing and Communications gkevorki@cisco.com Tel: +3237912058 Cisco.com <http://www.cisco.com> Think before you print.This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. On 5/18/16, 9:52 PM, "Nick Chase" <nchase@mirantis.com> wrote:
I was going to say, "Tristan, I'm serious" but actually a small track of randomly chosen talks could be fun. :)
But that's pretty far off topic.
----- Nick
On 5/18/2016 10:16 PM, Tristan Goode wrote:
How about a good old fashioned barrel draw?!?!
Sure we might get a few randoms, but they'd make for some entertaining interludes between everything else.
Cheers Tristan
-----Original Message----- From: Mark Collier [mailto:mark@openstack.org] Sent: Thursday, 19 May 2016 12:10 PM To: Nick Chase <nchase@mirantis.com>; Devananda van der Veen <devananda.vdv@gmail.com>; community@lists.openstack.org Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
This is an intriguing idea IMHO
On May 18, 2016 1:51:17 PM Nick Chase <nchase@mirantis.com> wrote:
Ya know, there's actually one fairly simple way we can avoid the "vote for me" spam:
Don't have static URLs for the talks.
I know that sounds a little crazy, but basically what I'm saying is that you make it so that you can go to the website and choose a category, but then the actual individual talks are linked dynamically so there's no way to send out a "Vote for this <link>" tweet. Instead you'd have to say, "Please vote for 'My talk' in the 'My track' category at <main link>" and I think few people are going to do that.
Just a thought.
---- Nick
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On Thu, May 19, 2016 at 4:52 AM, Nick Chase <nchase@mirantis.com> wrote:
I was going to say, "Tristan, I'm serious" but actually a small track of randomly chosen talks could be fun. :)
Given the fact that a few proposals I've previously reviewed were indeed complete rubbish (they seemed like someone had submitted a talk just to appease someone who told them to), a purely random selection would be of doubtful merit. :) Cheers, Florian
A reasonable quantity of talks that have been through the selection processes for each summit have been complete rubbish, so, your argument is invalid.
-----Original Message----- From: Florian Haas [mailto:florian@hastexo.com] Sent: Thursday, 19 May 2016 7:24 PM To: Nick Chase <nchase@mirantis.com> Cc: Tristan Goode <tristan@aptira.com>; community <community@lists.openstack.org> Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
On Thu, May 19, 2016 at 4:52 AM, Nick Chase <nchase@mirantis.com> wrote:
I was going to say, "Tristan, I'm serious" but actually a small track of randomly chosen talks could be fun. :)
Given the fact that a few proposals I've previously reviewed were indeed complete rubbish (they seemed like someone had submitted a talk just to appease someone who told them to), a purely random selection would be of doubtful merit. :)
Cheers, Florian
On Thu, May 19, 2016 at 3:16 PM, Tristan Goode <tristan@aptira.com> wrote:
A reasonable quantity of talks that have been through the selection processes for each summit have been complete rubbish, so, your argument is invalid.
I had no idea you were actually being serious about total randomization. Florian
I wasn’t serious to begin with, but faced with the two options of either: The Foundation dropping the voting and using the (same old) "cheer squad" to select talks.. Or Your proposal of only letting speakers pick talks.. It increasingly looks as valid selection method as any. Both the above methods, by dropping the audience's ability to have their say in any way at all say fuck the audience. Maybe voting for the track chairs is a way to give the audience some say. The current method of the foundation picking the cheer squad denies any critique. I was once a track chair, but haven't been for a couple of years because I got dropped from the cheer squad possibly for not agreeing that everything the foundation and the TC do is beyond question. But at least I get to vote for talks and take great pleasure in voting against the usual blowhards that the cheer squad nominate _every_single_summit_.
-----Original Message----- From: Florian Haas [mailto:florian@hastexo.com] Sent: Thursday, 19 May 2016 11:39 PM To: Tristan Goode <tristan@aptira.com> Cc: community <community@lists.openstack.org> Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
On Thu, May 19, 2016 at 3:16 PM, Tristan Goode <tristan@aptira.com> wrote:
A reasonable quantity of talks that have been through the selection processes for each summit have been complete rubbish, so, your argument is invalid.
I had no idea you were actually being serious about total randomization.
Florian
On Thu, May 19, 2016 at 4:15 PM, Tristan Goode <tristan@aptira.com> wrote:
I wasn’t serious to begin with, but faced with the two options of either:
The Foundation dropping the voting and using the (same old) "cheer squad" to select talks.. Or Your proposal of only letting speakers pick talks..
It increasingly looks as valid selection method as any. Both the above methods, by dropping the audience's ability to have their say in any way at all say fuck the audience.
Maybe voting for the track chairs is a way to give the audience some say. The current method of the foundation picking the cheer squad denies any critique.
Problem with that, just like we're now having a popularity contest with talk voting, we would then have a popularity contest with talk chair voting.
I was once a track chair, but haven't been for a couple of years because I got dropped from the cheer squad possibly for not agreeing that everything the foundation and the TC do is beyond question. But at least I get to vote for talks and take great pleasure in voting against the usual blowhards that the cheer squad nominate _every_single_summit_.
Quite frankly, I don't have a good idea for naming track chairs, and your grudge highlights exactly what the problem is with track chair appointment. I also don't have a good idea for talk voting that involves the entire audience and doesn't make it a popularity contest. That's why I proposed one where the group of reviewers is larger than the current hand-picked track chair group, and where the votes are more meaningful than with what we have now. But here's another idea that does away with track chairs, with voting talks into the Summit, and with favoritism altogether: - Immediately after the CfP ends, all abstracts go public and are in a review period for, say, two weeks. - Anyone can comment on any abstract providing input and questions. - The only possible audience *vote* that exists is pass/fail. Meaning, would this be suitable for the Summit or not (the idea is that the default is "pass", and that you would not light-heartedly give a fail, particularly if your judgment is public). - Once the review period is over, based on the number of fails given overall, you can work out what's the cutoff to be eliminated from the race. - From all talks not thus eliminated, there's a randomized draw. What does this mean? - Everyone gets a chance at voicing their opinions. - Speakers can get audience feedback before they even start their talk, which gives us better talks. - Everyone has the same fair chance at getting a talk slot. - We get speaker churn; new speakers are more likely to get a shot than in the current system which is biased in favor of repeat performers. - No-one needs to be disappointed because their talk doesn't get accepted. It's just the roll of the dice. - The only way to mathematically increase your chances of getting a slot is to submit more talks, which is easily solved by capping the total number of talks you can submit. Which we already do. Thoughts? Cheers, Florian
So essentially a barrel draw, but the entrants into the barrel get a bit of a prior brush over and your name gets put against anything you brush off. It's innovative. I like it.
-----Original Message----- From: Florian Haas [mailto:florian@hastexo.com] Sent: Friday, 20 May 2016 12:55 AM To: Tristan Goode <tristan@aptira.com> Cc: community <community@lists.openstack.org> Subject: Re: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit
On Thu, May 19, 2016 at 4:15 PM, Tristan Goode <tristan@aptira.com> wrote:
I wasn’t serious to begin with, but faced with the two options of either:
The Foundation dropping the voting and using the (same old) "cheer squad" to select talks.. Or Your proposal of only letting speakers pick talks..
It increasingly looks as valid selection method as any. Both the above methods, by dropping the audience's ability to have their say in any way at all say fuck the audience.
Maybe voting for the track chairs is a way to give the audience some say. The current method of the foundation picking the cheer squad denies any critique.
Problem with that, just like we're now having a popularity contest with talk voting, we would then have a popularity contest with talk chair voting.
I was once a track chair, but haven't been for a couple of years because I got dropped from the cheer squad possibly for not agreeing that everything the foundation and the TC do is beyond question. But at least I get to vote for talks and take great pleasure in voting against the usual blowhards that the cheer squad nominate _every_single_summit_.
Quite frankly, I don't have a good idea for naming track chairs, and your grudge highlights exactly what the problem is with track chair appointment.
I also don't have a good idea for talk voting that involves the entire audience and doesn't make it a popularity contest. That's why I proposed one where the group of reviewers is larger than the current hand-picked track chair group, and where the votes are more meaningful than with what we have now.
But here's another idea that does away with track chairs, with voting talks into the Summit, and with favoritism altogether:
- Immediately after the CfP ends, all abstracts go public and are in a review period for, say, two weeks. - Anyone can comment on any abstract providing input and questions. - The only possible audience *vote* that exists is pass/fail. Meaning, would this be suitable for the Summit or not (the idea is that the default is "pass", and that you would not light-heartedly give a fail, particularly if your judgment is public). - Once the review period is over, based on the number of fails given overall, you can work out what's the cutoff to be eliminated from the race. - From all talks not thus eliminated, there's a randomized draw.
What does this mean? - Everyone gets a chance at voicing their opinions. - Speakers can get audience feedback before they even start their talk, which gives us better talks. - Everyone has the same fair chance at getting a talk slot. - We get speaker churn; new speakers are more likely to get a shot than in the current system which is biased in favor of repeat performers. - No-one needs to be disappointed because their talk doesn't get accepted. It's just the roll of the dice. - The only way to mathematically increase your chances of getting a slot is to submit more talks, which is easily solved by capping the total number of talks you can submit. Which we already do.
Thoughts?
Cheers, Florian
On Thu, May 19, 2016 at 5:04 PM, Tristan Goode <tristan@aptira.com> wrote:
So essentially a barrel draw, but the entrants into the barrel get a bit of a prior brush over and your name gets put against anything you brush off.
Yes.
It's innovative. I like it.
What do others think? Cheers, Florian
On 19/05/16 15:55, Florian Haas wrote:
On Thu, May 19, 2016 at 4:15 PM, Tristan Goode <tristan@aptira.com> wrote:
I wasn’t serious to begin with, but faced with the two options of either:
The Foundation dropping the voting and using the (same old) "cheer squad" to select talks.. Or Your proposal of only letting speakers pick talks..
It increasingly looks as valid selection method as any. Both the above methods, by dropping the audience's ability to have their say in any way at all say fuck the audience.
Maybe voting for the track chairs is a way to give the audience some say. The current method of the foundation picking the cheer squad denies any critique. Problem with that, just like we're now having a popularity contest with talk voting, we would then have a popularity contest with talk chair voting.
I was once a track chair, but haven't been for a couple of years because I got dropped from the cheer squad possibly for not agreeing that everything the foundation and the TC do is beyond question. But at least I get to vote for talks and take great pleasure in voting against the usual blowhards that the cheer squad nominate _every_single_summit_. Quite frankly, I don't have a good idea for naming track chairs, and your grudge highlights exactly what the problem is with track chair appointment.
I also don't have a good idea for talk voting that involves the entire audience and doesn't make it a popularity contest. That's why I proposed one where the group of reviewers is larger than the current hand-picked track chair group, and where the votes are more meaningful than with what we have now.
But here's another idea that does away with track chairs, with voting talks into the Summit, and with favoritism altogether:
- Immediately after the CfP ends, all abstracts go public and are in a review period for, say, two weeks. - Anyone can comment on any abstract providing input and questions. - The only possible audience *vote* that exists is pass/fail. Meaning, would this be suitable for the Summit or not (the idea is that the default is "pass", and that you would not light-heartedly give a fail, particularly if your judgment is public). - Once the review period is over, based on the number of fails given overall, you can work out what's the cutoff to be eliminated from the race. - From all talks not thus eliminated, there's a randomized draw.
What does this mean? - Everyone gets a chance at voicing their opinions. - Speakers can get audience feedback before they even start their talk, which gives us better talks. - Everyone has the same fair chance at getting a talk slot. - We get speaker churn; new speakers are more likely to get a shot than in the current system which is biased in favor of repeat performers. - No-one needs to be disappointed because their talk doesn't get accepted. It's just the roll of the dice. - The only way to mathematically increase your chances of getting a slot is to submit more talks, which is easily solved by capping the total number of talks you can submit. Which we already do.
Thoughts?
Cheers, Florian
I for one - think this approach could work. Of course though there will be those who traditionally have presented at each and every summit up until now that might not get chosen... But hey - it is all in the best interests of the community !! (It will be interesting to see the reactions on this) -- Best Regards, Maish Saidel-Keesing
Call me a cynic and this may be an unfair statement but those people who present at every summit will continue to be chosen, only now at random. Just my unsubstantiated feelings on that subject. I've stopped raging against the machine long ago. ;) //adam On May 19, 2016 8:37 AM, "Maish Saidel-Keesing" <maishsk@maishsk.com> wrote:
On 19/05/16 15:55, Florian Haas wrote:
On Thu, May 19, 2016 at 4:15 PM, Tristan Goode <tristan@aptira.com> <tristan@aptira.com> wrote:
I wasn’t serious to begin with, but faced with the two options of either:
The Foundation dropping the voting and using the (same old) "cheer squad" to select talks.. Or Your proposal of only letting speakers pick talks..
It increasingly looks as valid selection method as any. Both the above methods, by dropping the audience's ability to have their say in any way at all say fuck the audience.
Maybe voting for the track chairs is a way to give the audience some say. The current method of the foundation picking the cheer squad denies any critique.
Problem with that, just like we're now having a popularity contest with talk voting, we would then have a popularity contest with talk chair voting.
I was once a track chair, but haven't been for a couple of years because I got dropped from the cheer squad possibly for not agreeing that everything the foundation and the TC do is beyond question. But at least I get to vote for talks and take great pleasure in voting against the usual blowhards that the cheer squad nominate _every_single_summit_.
Quite frankly, I don't have a good idea for naming track chairs, and your grudge highlights exactly what the problem is with track chair appointment.
I also don't have a good idea for talk voting that involves the entire audience and doesn't make it a popularity contest. That's why I proposed one where the group of reviewers is larger than the current hand-picked track chair group, and where the votes are more meaningful than with what we have now.
But here's another idea that does away with track chairs, with voting talks into the Summit, and with favoritism altogether:
- Immediately after the CfP ends, all abstracts go public and are in a review period for, say, two weeks. - Anyone can comment on any abstract providing input and questions. - The only possible audience *vote* that exists is pass/fail. Meaning, would this be suitable for the Summit or not (the idea is that the default is "pass", and that you would not light-heartedly give a fail, particularly if your judgment is public). - Once the review period is over, based on the number of fails given overall, you can work out what's the cutoff to be eliminated from the race. - From all talks not thus eliminated, there's a randomized draw.
What does this mean? - Everyone gets a chance at voicing their opinions. - Speakers can get audience feedback before they even start their talk, which gives us better talks. - Everyone has the same fair chance at getting a talk slot. - We get speaker churn; new speakers are more likely to get a shot than in the current system which is biased in favor of repeat performers. - No-one needs to be disappointed because their talk doesn't get accepted. It's just the roll of the dice. - The only way to mathematically increase your chances of getting a slot is to submit more talks, which is easily solved by capping the total number of talks you can submit. Which we already do.
Thoughts?
Cheers, Florian
I for one - think this approach could work.
Of course though there will be those who traditionally have presented at each and every summit up until now that might not get chosen... But hey - it is all in the best interests of the community !! (It will be interesting to see the reactions on this) -- Best Regards, Maish Saidel-Keesing
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On Thu, May 19, 2016 at 6:28 PM, Adam Lawson <alawson@aqorn.com> wrote:
Call me a cynic and this may be an unfair statement but those people who present at every summit will continue to be chosen, only now at random. Just my unsubstantiated feelings on that subject.
I've stopped raging against the machine long ago. ;)
So I'm seeing a disconnect here: we have some folks arguing any talk selection process that is self-policing implies a lack of trust in good faith and thus does more harm than good (Doug, Dave). We have others indicating that from personal experience, good faith and impartiality on everyone's part is already something we shouldn't assume (Tristan, Adam). I realize I'm not quoting you directly, but forgive me for paraphrasing/condensing/sharpening. So to summarize, it looks like there are four options currently being discussed: (1) Keep everything exactly as it has been (this implies rejecting Claire's original proposal to drop voting). (2) Drop voting, put everything in the track chairs' hands. (3) Use some form of random talk selection, in combination with submission pre-filtering and public comments. (4) Use an approach where speakers vote on a random subset of talks and rank them, also with public comments. Out of those, (4) uses a self-policing review scheme, and (3) promises to curb abuse and favoritism by effectively leaving things mostly to chance. (3) and (4) can ultimately do without track chairs. Just to get a reference gauge, would those following this discussion respond with a ranking in order of preference? Mine (from most preferred to least preferred) is 4, 3, 1, 2. Cheers, Florian
Hi Florian, With all respect, the most important voice(s) in this debate are also notably absent - we have not heard anything from any foundation staff since the start of the thread, and I suggest we take a pause to let them weigh in, as (in the end) whatever happens will be implemented by them. Thanks, Dave. On 05/19/2016 04:40 PM, Florian Haas wrote:
On Thu, May 19, 2016 at 6:28 PM, Adam Lawson <alawson@aqorn.com> wrote:
Call me a cynic and this may be an unfair statement but those people who present at every summit will continue to be chosen, only now at random. Just my unsubstantiated feelings on that subject.
I've stopped raging against the machine long ago. ;)
So I'm seeing a disconnect here: we have some folks arguing any talk selection process that is self-policing implies a lack of trust in good faith and thus does more harm than good (Doug, Dave). We have others indicating that from personal experience, good faith and impartiality on everyone's part is already something we shouldn't assume (Tristan, Adam). I realize I'm not quoting you directly, but forgive me for paraphrasing/condensing/sharpening.
So to summarize, it looks like there are four options currently being discussed:
(1) Keep everything exactly as it has been (this implies rejecting Claire's original proposal to drop voting). (2) Drop voting, put everything in the track chairs' hands. (3) Use some form of random talk selection, in combination with submission pre-filtering and public comments. (4) Use an approach where speakers vote on a random subset of talks and rank them, also with public comments.
Out of those, (4) uses a self-policing review scheme, and (3) promises to curb abuse and favoritism by effectively leaving things mostly to chance. (3) and (4) can ultimately do without track chairs.
Just to get a reference gauge, would those following this discussion respond with a ranking in order of preference?
Mine (from most preferred to least preferred) is 4, 3, 1, 2.
Cheers, Florian
-- Dave Neary - NFV/SDN Community Strategy Open Source and Standards, Red Hat - http://community.redhat.com Ph: +1-978-399-2182 / Cell: +1-978-799-3338
On 2016-05-19 17:15:57 -0400 (-0400), Dave Neary wrote:
With all respect, the most important voice(s) in this debate are also notably absent - we have not heard anything from any foundation staff since the start of the thread, and I suggest we take a pause to let them weigh in, as (in the end) whatever happens will be implemented by them.
To be fair, I'm on the foundation staff and have simply been replying with my community-member/track-chair-emeritus hat on; however I'm also not one of the event coordinators and wouldn't attempt to speak on their behalf (my opinions do not necessarily reflect those of the summit crew). -- Jeremy Stanley
On 2016-05-19 22:40:58 +0200 (+0200), Florian Haas wrote: [...]
So to summarize, it looks like there are four options currently being discussed:
(1) Keep everything exactly as it has been (this implies rejecting Claire's original proposal to drop voting). (2) Drop voting, put everything in the track chairs' hands. [...]
The second is a slight mischaracterization. Even with the status quo, everything is already entirely in the hands of the track chairs. The difference between 1 and 2 is whether or not we keep an (arguably noisy beyond help) existing community feedback channel for the track chairs to consider in making their decisions. -- Jeremy Stanley
On Thu, May 19, 2016 at 11:16 PM, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2016-05-19 22:40:58 +0200 (+0200), Florian Haas wrote: [...]
So to summarize, it looks like there are four options currently being discussed:
(1) Keep everything exactly as it has been (this implies rejecting Claire's original proposal to drop voting). (2) Drop voting, put everything in the track chairs' hands. [...]
The second is a slight mischaracterization. Even with the status quo, everything is already entirely in the hands of the track chairs.
With the help of the community, who do provide input. By putting "everything" in the track chairs' hands I meant not only the final decision, but reviewing talks, ranking and selecting them completely among themselves, without community input — which what dropping voting alone would be, if everything else remained unchanged. Cheers, Florian
On 5/19/2016 4:40 PM, Florian Haas wrote:
So to summarize, it looks like there are four options currently being discussed:
(1) Keep everything exactly as it has been (this implies rejecting Claire's original proposal to drop voting). (2) Drop voting, put everything in the track chairs' hands. (3) Use some form of random talk selection, in combination with submission pre-filtering and public comments. (4) Use an approach where speakers vote on a random subset of talks and rank them, also with public comments. You forgot:
(5) Keep everything as-is but change the "voting" to "commenting" (6) Keep voting but prevent social spam by removing static URLs (7) Change the voting structure ... (8) etc... In other words, there are a LOT of options being discussed here, to be honest. It's not as simple as just "should we vote or not". In fact, we've left out the simplest version: (9) Keep everything as-is but make it more clear that the votes are just a guideline I'm not sure the best way to proceed here from a problem-solving standpoint, to be honest. I'd suggest that we had people come up with concrete proposals for up/down votes, but that's kind of what we've been doing here and the options have expanded rather than contracting. :) ---- Nick
Thanks Nick and Florian. Yes, it has been a very dynamic conversation. :) Our Summit team here at the Foundation thinks traditions are an important part of our community culture, but we hate to do things (like voting) just because that’s what we’ve always done. In making the proposal to end voting, I think we’ve challenged our thinking around it and come up with some really good ideas. I am now personally leaning toward keeping voting this round with some tweaks and better communication around the purpose, with the option to reevaluate after Barcelona. If possible from a development standpoint, we would support the following ideas in this thread: - enabling comments to provide track chairs more feedback ahead of their decision and speakers more input as they prepare their talk - removing static/linkable URLs for promotion to reduce the noise and gaming - clarifying the purpose of voting as a piece of input for track chairs in the speaker selection process We're on the fence about displaying each voter a limited, random subset of talks, because some folks only care about specific tracks or topics, and some folks have diligently rated every single talk in the past. We think it would be very difficult and counter to the purpose of community voting if we tried to limit the pool of voters to speakers or conference attendees (the latter simply because a very small percentage are actually registered at the point in time we start voting). Additionally, we would support publishing the names of track chairs earlier in the process with clear guidance not to contact or badger them about your talk. On a related note, as we continue to produce more and more content across OpenStack events we've opened a position for a content manager role to help with this workload.
On May 19, 2016, at 5:26 PM, Nick Chase <nchase@mirantis.com> wrote:
On 5/19/2016 4:40 PM, Florian Haas wrote:
So to summarize, it looks like there are four options currently being discussed:
(1) Keep everything exactly as it has been (this implies rejecting Claire's original proposal to drop voting). (2) Drop voting, put everything in the track chairs' hands. (3) Use some form of random talk selection, in combination with submission pre-filtering and public comments. (4) Use an approach where speakers vote on a random subset of talks and rank them, also with public comments. You forgot:
(5) Keep everything as-is but change the "voting" to "commenting" (6) Keep voting but prevent social spam by removing static URLs (7) Change the voting structure ... (8) etc...
In other words, there are a LOT of options being discussed here, to be honest. It's not as simple as just "should we vote or not". In fact, we've left out the simplest version:
(9) Keep everything as-is but make it more clear that the votes are just a guideline
I'm not sure the best way to proceed here from a problem-solving standpoint, to be honest. I'd suggest that we had people come up with concrete proposals for up/down votes, but that's kind of what we've been doing here and the options have expanded rather than contracting. :)
---- Nick
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
I think the dynamic links and removing static links (of voting is preserved) is awesome. I apparently missed that. Huge+1 from me on that idea. //adam On May 19, 2016 4:37 PM, "Claire Massey" <claire@openstack.org> wrote:
Thanks Nick and Florian. Yes, it has been a very dynamic conversation. :)
Our Summit team here at the Foundation thinks traditions are an important part of our community culture, but we hate to do things (like voting) just because that’s what we’ve always done. In making the proposal to end voting, I think we’ve challenged our thinking around it and come up with some really good ideas. I am now personally leaning toward keeping voting this round with some tweaks and better communication around the purpose, with the option to reevaluate after Barcelona.
If possible from a development standpoint, we would support the following ideas in this thread: - enabling comments to provide track chairs more feedback ahead of their decision and speakers more input as they prepare their talk - removing static/linkable URLs for promotion to reduce the noise and gaming - clarifying the purpose of voting as a piece of input for track chairs in the speaker selection process
We're on the fence about displaying each voter a limited, random subset of talks, because some folks only care about specific tracks or topics, and some folks have diligently rated every single talk in the past. We think it would be very difficult and counter to the purpose of community voting if we tried to limit the pool of voters to speakers or conference attendees (the latter simply because a very small percentage are actually registered at the point in time we start voting).
Additionally, we would support publishing the names of track chairs earlier in the process with clear guidance not to contact or badger them about your talk.
On a related note, as we continue to produce more and more content across OpenStack events we've opened a position for a content manager role to help with this workload.
On May 19, 2016, at 5:26 PM, Nick Chase <nchase@mirantis.com> wrote:
On 5/19/2016 4:40 PM, Florian Haas wrote:
So to summarize, it looks like there are four options currently being discussed:
(1) Keep everything exactly as it has been (this implies rejecting Claire's original proposal to drop voting). (2) Drop voting, put everything in the track chairs' hands. (3) Use some form of random talk selection, in combination with submission pre-filtering and public comments. (4) Use an approach where speakers vote on a random subset of talks and rank them, also with public comments.
You forgot:
(5) Keep everything as-is but change the "voting" to "commenting" (6) Keep voting but prevent social spam by removing static URLs (7) Change the voting structure ... (8) etc...
In other words, there are a LOT of options being discussed here, to be honest. It's not as simple as just "should we vote or not". In fact, we've left out the simplest version:
(9) Keep everything as-is but make it more clear that the votes are just a guideline
I'm not sure the best way to proceed here from a problem-solving standpoint, to be honest. I'd suggest that we had people come up with concrete proposals for up/down votes, but that's kind of what we've been doing here and the options have expanded rather than contracting. :)
---- Nick
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
On Fri, May 20, 2016 at 2:51 AM, Adam Lawson <alawson@aqorn.com> wrote:
I think the dynamic links and removing static links (of voting is preserved) is awesome. I apparently missed that.
Huge+1 from me on that idea.
I think this is a good idea as well, with one minor detail though: as a community reviewer, I'd like to be able to give a talk just a cursory review, not immediately make a decision on how to grade or what to comment, and come back to it later. With static links that's easy: I can simply compile my own bookmark list. With dynamic/random links, the system would have to provide an alternative. So, rather than inventing a separate bookmark system, maybe make links static per-user? Meaning every logged-in user gets a different link for the same talk, but for that user the link doesn't change? That still keeps me from sharing the link to my own talk (or more precisely, make it useless — another user would simply get a 404 if they tried to open it), but allows me to compile my own bookmarks. Just an idea. Cheers, Florian
On Fri, May 20, 2016 at 1:36 AM, Florian Haas <florian@hastexo.com> wrote:
On Fri, May 20, 2016 at 2:51 AM, Adam Lawson <alawson@aqorn.com> wrote:
I think the dynamic links and removing static links (of voting is preserved) is awesome. I apparently missed that.
Huge+1 from me on that idea.
I think this is a good idea as well, with one minor detail though: as a community reviewer, I'd like to be able to give a talk just a cursory review, not immediately make a decision on how to grade or what to comment, and come back to it later. With static links that's easy: I can simply compile my own bookmark list. With dynamic/random links, the system would have to provide an alternative.
So, rather than inventing a separate bookmark system, maybe make links static per-user? Meaning every logged-in user gets a different link for the same talk, but for that user the link doesn't change? That still keeps me from sharing the link to my own talk (or more precisely, make it useless — another user would simply get a 404 if they tried to open it), but allows me to compile my own bookmarks.
Just an idea.
I want to give this idea a huge +1, it's a really good idea. I'm strongly in favor of removing the incentive to share direct links to talks (even though I'm guilty of tweeting "vote for my talk!" more than once). Having individualized links like this would be very helpful and sounds to me like it would be an elegant solution to at least part of the problem we are trying to solve. -Christopher
On Fri, May 20, 2016 at 1:35 AM, Claire Massey <claire@openstack.org> wrote:
Thanks Nick and Florian. Yes, it has been a very dynamic conversation. :)
Our Summit team here at the Foundation thinks traditions are an important part of our community culture, but we hate to do things (like voting) just because that’s what we’ve always done. In making the proposal to end voting, I think we’ve challenged our thinking around it and come up with some really good ideas. I am now personally leaning toward keeping voting this round with some tweaks and better communication around the purpose, with the option to reevaluate after Barcelona.
If possible from a development standpoint, we would support the following ideas in this thread: - enabling comments to provide track chairs more feedback ahead of their decision and speakers more input as they prepare their talk - removing static/linkable URLs for promotion to reduce the noise and gaming - clarifying the purpose of voting as a piece of input for track chairs in the speaker selection process
Right. As an interim solution for Barcelona, all this sounds prudent. I would also add actively encouraging community reviewers to provide free-form comments on suggested talks, rather than "only" giving their vote.
We're on the fence about displaying each voter a limited, random subset of talks, because some folks only care about specific tracks or topics, and some folks have diligently rated every single talk in the past. We think it would be very difficult and counter to the purpose of community voting if we tried to limit the pool of voters to speakers or conference attendees (the latter simply because a very small percentage are actually registered at the point in time we start voting).
Fair enough, as far as I am concerned.
Additionally, we would support publishing the names of track chairs earlier in the process with clear guidance not to contact or badger them about your talk.
Yes, that would be very helpful. Cheers, Florian
On Wed, May 18, 2016 at 12:42 PM, Devananda van der Veen < devananda.vdv@gmail.com> wrote:
But the situation we're in, I think, is that folks only vote on the talks that surface within their social network.
In prior summits, I voted on nearly every proposal within the tracks I was proposing and interested in. It just became impossible to do that (partly due to the tools created and partly due to the volume) this go round. So, while I did vote on a good number of "random" talks, I usually started with one I was alerted to by Soc Media or personal relationships (including company relationships). I don't think there is a way currently to enforce this though (maybe by only considering votes when someone has voted on 10+ sessions in a given track, then give weight to their votes... otoh,you wouldn't want to heavily advertise this criteria or folks would "game" around it.
On 19/05/16 04:24, Florian Haas wrote:
On Wed, May 18, 2016 at 6:52 PM, Nick Chase <nchase@mirantis.com> wrote:
I, for one, am not fond of letting go of community involvement, for concern of never getting it back.
Cheers, Florian
I think that's a fair point, and another reason to enable comments instead of votes. People can not only feel involved but they will be making a more substantial contribution to the process.
Granted, this may not solve the "spam" issue, though, as people will still be soliciting comments. However it will be more clear that getting comments doesn't mean getting in, so there's less incentive.
---- Nick
So we currently have a fairly massive volume of submissions every Summit. I'm trying to put myself in the shoes of J. Random Summit Attendee here, right after the CfP closes. What would be my motivation for commenting on a talk that I don't even know for sure will make it into the schedule? Content suggestions like "could you please talk about X as well" or "could you spend some time on Y" are great *after* the talk is confirmed. But while it's still being considered, the logical thing is to say something like "I'd love to see this" or "I'd not attend this" — which is exactly what we have with voting, now.
Thoughts?
Regarding the massive quantity of submissions: I rely on people sending out lists of talk to vote on, because the sheer volume of talks to go through and vote on is so huge, and the signal:noise (talks I care about vs talks I have no interest in) is so vanishingly small as to make it not worth my time. I have too many code reviews waiting to go wading through talk voting for a whole afternoon. This is why the 'vote on a subset' idea really strikes me as valuable. If I could do it in fifteen minutes, then I totally would. L -- Lana Brindley Technical Writer Rackspace Cloud Builders Australia http://lanabrindley.com
On 2016-05-18 17:58:41 +0200 (+0200), Florian Haas wrote: [...]
I, for one, am not fond of letting go of community involvement, for concern of never getting it back.
Given that track chairs are volunteers from the community, frequently rotated out, I don't see how getting rid of voting means getting rid of community involvement. Other aspects of OpenStack don't (or shouldn't) operate on mob rule; the people who step forward to do the work are the ones who make the decisions. That said, I wouldn't mind seeing us have some platform to allow members of the general community to provide constructive feedback to prospective speakers _prior_ to starting the chairing process, allowing them the opportunity to refine their abstracts based on audience interest. Like code review, but for talk proposals. (Also, a mechanism to provide feedback to speakers _after_ they've given their presentations would be nifty.) -- Jeremy Stanley
Maybe we can involve the community more in the selection of track chairs so there is still an ability to participate but then leave it to track chairs to pick the best sessions. I also noticed we had a huge difference in attendance across sessions, so i would think we would want to use data and past session success (or not) to influence our decisions, which people voting would not have access to. On Tue, May 17, 2016 at 3:40 PM, Claire Massey <claire@openstack.org> wrote:
Hi everyone,
For the past few Summits, we've received mixed feedback about having the community vote on proposed sessions as part of the call for speakers process. Historically, after the call for speakers closes, we publish all submitted sessions for community voting before the track chairs review them. The track chairs then choose how much weight to put on the voting resuls, if any, because they make the ultimate decision about which sessions are selected. More info on Track Chairs can be found at the bottom of this email.
With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case).
We would like to propose removing voting from the selection process for the October 2016 Barcelona Summit, but want to get your input before making a final decision. Our thinking is that by removing voting from the process, we will:
- Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle - Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes - Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting
We initially started the voting process for good reasons and we do think there's value, but we're reaching a point where the costs are starting to outweigh the benefits. We'd like to get your input before we open the call for speakers in early June for the Barcelona Summit.
Thanks, Claire
Track Chair Info Track Chairs are subject matter experts who review submissions to their particular track, for example "storage" or "cloud applications." There are typically 3-4 chairs per track who review and collaboratively decide which presentations are ultimately accepted for inclusion on the final agenda. The Foundation strives to recruit Track Chairs from a diverse set of companies, regions, roles in the community (i.e., contributing developers, users and business leaders) and areas of expertise. Information on how to nominate yourself or someone else to serve as a track chair for the Barcelona Summit will be published when the call for speakers goes live in early June. For reference, here's information on the Track Chairs from the Austin Summit: https://etherpad.openstack.org/p/Austin_Summit_Track_Chairs and https://www.openstack.org/summit/austin-2016/categories/selection-process.
_______________________________________________ Community mailing list Community@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/community
-- Margaret Dawson Global Product Marketing Infrastructure Business Group Red Hat @seattledawson Cel: 206-355-6872
On Wed, May 18, 2016 at 3:15 PM, Margaret Dawson <mdawson@redhat.com> wrote:
Maybe we can involve the community more in the selection of track chairs so there is still an ability to participate but then leave it to track chairs to pick the best sessions.
I'm afraid this would mean that we're shifting the talk popularity contest to a track chair popularity contest. This doesn't strike me as particularly beneficial. Cheers, Florian
Claire, Unfortunately, I have been able to follow the whole discussion but I would like to give my opinion. Austin summit was my first time as track chair and I have to admit I was so happy and very excited about it. My idea as a chair was to follow this process to make a fair selection of sessions: - Vote Ave >= 2.5 - Description: All questions should have been addressed properly and should be "understandable" - Innovation: I am looking for sessions that are new and very informative of the change in the networking side of openStack - Sessions showing vendor-specific details: These are my less favorites. I am personally trying to avoid them at least the technology is open-sourced and we can test it without paying for it. Somebody replied to my suggestion the following and I quote: “I have actually somewhat ignored the vote, as this can be gamified.” And somebody else mentioned: “Me too, the votes are essentially meaningless. See the discussions last month about this.” I do respect those opinions and makes wonder if more chairs have the similar thinking. Then it is very clear that the voting is not needed anymore. Thanks, Edgar From: Claire Massey <claire@openstack.org> Date: Tuesday, May 17, 2016 at 12:40 PM To: "Community@lists.openstack.org" <Community@lists.openstack.org> Subject: [openstack-community] Proposal: remove voting on speaking proposals for Barcelona Summit Hi everyone, For the past few Summits, we've received mixed feedback about having the community vote on proposed sessions as part of the call for speakers process. Historically, after the call for speakers closes, we publish all submitted sessions for community voting before the track chairs review them. The track chairs then choose how much weight to put on the voting resuls, if any, because they make the ultimate decision about which sessions are selected. More info on Track Chairs can be found at the bottom of this email. With the growing number of speaking submissions (we had 1,300 for Austin), some community members have expressed concerns about social media channels and email getting spammed during the week of voting. We also think many community members are unclear as to how much the votes weigh on the final decision. For example, some think that if someone campaigns for votes or asks their colleagues to vote, the session will likely be accepted (which may not be the case). We would like to propose removing voting from the selection process for the October 2016 Barcelona Summit, but want to get your input before making a final decision. Our thinking is that by removing voting from the process, we will: - Save valuable time during the overall Summit programming process, which should allow us to publish the final agenda and notify speakers sooner - Allow our development teams more time to focus on improving the mobile app and web schedule developed during the last Summit cycle - Reduce the spam and noise around voting, so we don't cause Twitter fatigue before we're promoting the final agenda and key themes - Level the playing field for speakers from startups, new community members, etc. who may not have an established network in the community for voting We initially started the voting process for good reasons and we do think there's value, but we're reaching a point where the costs are starting to outweigh the benefits. We'd like to get your input before we open the call for speakers in early June for the Barcelona Summit. Thanks, Claire Track Chair Info Track Chairs are subject matter experts who review submissions to their particular track, for example "storage" or "cloud applications." There are typically 3-4 chairs per track who review and collaboratively decide which presentations are ultimately accepted for inclusion on the final agenda. The Foundation strives to recruit Track Chairs from a diverse set of companies, regions, roles in the community (i.e., contributing developers, users and business leaders) and areas of expertise. Information on how to nominate yourself or someone else to serve as a track chair for the Barcelona Summit will be published when the call for speakers goes live in early June. For reference, here's information on the Track Chairs from the Austin Summit: https://etherpad.openstack.org/p/Austin_Summit_Track_Chairs<https://urldefense.proofpoint.com/v2/url?u=https-3A__etherpad.openstack.org_p_Austin-5FSummit-5FTrack-5FChairs&d=CwMFAg&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=gsB4ibRydRCSG-J30gqIPF8EwVOgklYLFsfv9TEQfyY&s=bn5mkoo-L7DXHlzsZu2sOVNdyAmKBLufAlIYjcBI0Ck&e=> and https://www.openstack.org/summit/austin-2016/categories/selection-process<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.openstack.org_summit_austin-2D2016_categories_selection-2Dprocess&d=CwMFAg&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ&m=gsB4ibRydRCSG-J30gqIPF8EwVOgklYLFsfv9TEQfyY&s=IlwFWHkfk5go_Rr-f4eav90gHDEkisGaJWieZvtFcaw&e=>.
participants (24)
-
Adam Lawson
-
atul jha
-
Chris Jones
-
Christopher Aedo
-
Claire Massey
-
Dave Neary
-
David Medberry
-
Devananda van der Veen
-
Doug Hellmann
-
Edgar Magana
-
Florian Haas
-
Gary Kevorkian (gkevorki)
-
Jeremy Stanley
-
Lana Brindley
-
Maish Saidel-Keesing
-
Margaret Dawson
-
Mark Collier
-
Nick Chase
-
Richard Raseley
-
Roland Chan
-
Stefano Maffulli
-
Sun, Yih Leong
-
Sylvain Bauza
-
Tristan Goode