emsejournal
MSR - RR dates and timeline
Tentative timeline, based on other MSR tracks:
- Pre-registration plans due: February 6, 2020
- Author notification: March 2, 2020
- 1 page summary plan / camera-ready: March 16, 2020
- Reports registered with OSF registry: Mar 31, 2020
The period between initial plan and decision is only 4 weeks, which is pretty short if the registration is complex.
Since this is new, I think we should start earlier. Can we do it at the beginning of January, because we also include a "rebuttal" phase? And I also think that reviewers might be having more questions for us, so more time for them might also be necessary.
How about
- Jan 10 plans due
- Jan 31 initial reviews
- Feb 14 rebuttals/clarifications due
- Mar 2 final decision notifications
- (rest as above)
I think the revised timeline sounds good to give you more time in the middle. Did any of you already try out OSF?
We have been looking into the format. I think for now we will have a custom registration template, as the OSF one is pretty weighty. Then if accepted we ask the authors to put the registration on OSF.
Sounds good. Is there a backup plan if your custom template doesn't go through? Or perhaps I misunderstood: do you need some form of OSF approval to create a custom template?
hmm, good question. @feigensp is working on the CFP/template. I think our model was the authors would then upload the registration using the "IPA" approach in the screenshot (but I will check with OSF).
So from that snapshot, it seems this is supported by default by the platform and doesn't need any approval from their part. I guess you just need to figure out the questions you want the authors to answer. So I guess it's all OK :-)
Ok, in that case I’ll close this issue and you can use these dates for the MSR website - let me know if you want a different format or via email.
| Date | Milestone |
|---|---|
| January 10, 2020 | study protocols and plans due |
| January 31, 2020 | initial protocol reviews |
| February 14,2020 | rebuttals/clarifications due |
| March 2, 2020 | In Principle Acceptance (IPA) decision notifications |
| March 16, 2020 | 1 page summary plan / camera-ready |
| March 31, 2020 | Reports registered with OSF registry |
Just a clarification... why is it in principle acceptance? Aren't you simply accepting the registered report, not actually the study yet? Or is this the typical term used for registered reports?
That’s the typical term used. In principle it can be accepted if the protocol is followed.
On Sep 19, 2019, at 15:18, snadi [email protected] wrote:
Just a clarification... why is it in principle acceptance? Aren't you simply accepting the registered report, not actually the study yet? Or is this the typical term used for registered reports?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
One question (I think from Laurie Williams) when I mentioned pre-registration/RRs in my ESEM19 keynote:
"How are changes to be handled that come from comments on the pre-reg report when it is presented at MSR?"
I guess comments come either from reviewers during Stage 1 review but can also come from audience when presented at MSR. The former are handled in rebuttal/revision process pre MSR accept? While changes that happens between that and Stage 2 review must be explicitly explained and clarified in the paper submitted for Stage 2 review?
That’s an excellent question. One solution would be open review but I believe @snadi looked at this and it didn’t meet the needs.
The problem is how we check that a study actually responded to the MSR comments (and which comments, sorry, ‘questions’ are worth responding to).
If the protocol is deviated from after IAP I think it can still be published, but would need more fulsome review.
On Sep 20, 2019, at 11:45, Robert Feldt [email protected] wrote:
One question (I think from Laurie Williams) when I mentioned pre-registration/RRs in my ESEM19 keynote:
"How are changes to be handled that come from comments on the pre-reg report when it is presented at MSR?"
I guess comments come either from reviewers during Stage 1 review but can also come from audience when presented at MSR. The former are handled in rebuttal/revision process pre MSR accept? While changes that happens between that and Stage 2 review must be explicitly explained and clarified in the paper submitted for Stage 2 review?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Would it be feasible to use (some of) the same reviewers for the versions submitted to MSR and the versions submitted to EMSE? And making comments made during the presentation at MSR available to the EMSE reviewers?
I was sort of thinking the first part would be true .. that you would review it twice. But maybe that’s not necessary.
As for making comments available, I think this would be better served with a more deliberate mechanism like openreview.net or a github repo.
On Sep 21, 2019, at 19:39, ctreude [email protected] wrote:
Would it be feasible to use (some of) the same reviewers for the versions submitted to MSR and the versions submitted to EMSE? And making comments made during the presentation at MSR available to the EMSE reviewers?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
@neilernst one option for sharing reviews without using a specific system is to just make it a requirement that authors submit the reviews with their submission to the special issue. Alternatively, if you are worried they might tamper with them (Although I would assume everyone will be honest), you (as editors of the special issue?) can just export reviews with easychair and make it available to reviewers yourselves.
oh good idea. the only catch is I'm not sure we own the reviews, but we could ask MSR reviewers to check a box when they submit right?
- [ ] I agree the PC chairs may share these reviews with Phase 2 Reviewers
Yup, you need the permission of the reviewers and that checkbox works. Better yet, when inviting MSR reviewers, make it clear that if they accept the invite, they accept the fact that their reviews will be shared with the next phase reviewers. Otherwise, you may get in awkward situations where they decide not to check the checkbox, then some papers will have the reviews while others won’t. In the CFP, make sure it’s also clear for authors that the reviews will be passed on.