FreeOurKnowledge
Campaign proposal: Open Code Pledge
- Campaign co-creator/s: Manuel Illanes @rmib200
- Rationale: Many areas of Neuroscience are critically dependent on computational tools for things like analysis of large volumes of data, tasks, visualization, etc. Right now, we can say that sharing code is the exception when new research is published. Only certain journals require papers to include a statement on whether programs are available. We belive that all jounals should adopt policies that strongly encourage or mandate the sharing of code. Sharing all code and data needed to allow others to reproduce our work should be considered a regular practice. At least the key parts when this is not possible or practical.
- Action: (1) Whenever you publish an article, also publish the code you used for the analysis and implementation via an open, freely available platform (e.g., Github, OSF, papers with code). (2) Generate a DOI for the code and link to this DOI in the article. (Optional)
* Make sure that at least one or more figures from the manuscript are reproducible.
* If you think the work is too computationally intensive, one small "toy" example could be used to demostrate it.
* Also, you can upload a config file when possible.
We can demonstrate, leading by example, the obvious benefits of adopting these practices like more transparency, reliability and faster adoption of new methods. This will improve the future work produced on Neuroscience, even if the code doesn't fully work in 20 years. Because it will still be more precise and complete than text in a paper.
Open-source code can be used for promotion, reference letters, and mentions. Take into account that right now, the limitation is cultural rather than technical so let's take a part in changing that. Let's free our knowledge.
If you can provide feedback on anything that could be done differently to benefit the field or get more people motivated to do anything regarding science, please don't doubt to comment. For Example, we could start by listing all the possible best practices for this.
- Eligibility criteria: All fields (but we will target neuroscience researchers initially).
- Optional anonymity: Yes.
- Threshold: ~~100~~ 500 (calculated separately for each field)
- Pledge duration: ~~18 months.~~ Lifetime
Great idea! Thanks for proposing this campaign! Some initial thoughts:
- Action: publish ALL your code. ALWAYS.
I think we could make this more specific, e.g. ask people to publish code in an open (indexed?) repository (e.g., OSF, Github). Also, I wonder if will scare people off by asking them to publish ALL of their code (as great as this would be!). What do you think about instead asking people to publish all code related to any papers/preprints they publish?
- Eligibility criteria: Mainly for researchers on BCI and Neuroscience but could apply for all fields.
In previous campaigns we've included a drop-down box for people to indicate their field. This way the campaign can be open to all researchers, but pledges only activate when there is a critical mass of support in your field. But I'm also happy to target a campaign specifically at Neuroscience researchers, if you prefer. This is all experimentation after all, and we are finding out what works as we go :)
- Pledge duration: ? Not sure. Should be permanent.
Totally agree that permanent would be best, but wonder if we might get more people to sign if we only ask for a fixed period (e.g. 1 or 2 years)? I can imagine some people thinking "Sure, but what if I change labs and my new PI doesn't support open code?". Again, it's just about lowering barriers to signing, and we can of course follow up with another (bigger) campaign in a years time in which we ask for a longer/permanent duration.
That's a great idea!
I'm thinking, a good way to achieve this goal is to create a GitHub repo for this purpose. Let me elaborate on what I mean.
I'm thinking if a template github repo is made with github pages web integration. We can ask researchers to fork that repository and add their code on top of that. This way a template for data and code sharing comes along with the open code proposal/pledge.
This way, ideally after a while, there will be quite a few forks of the repositories that have a similar format and are all linked to a separate paper.
This may require a bit of development to prepare the template (it should contain a proposed directory structure, and a landing page which loads through github pages and is linked to the published article).
Hi and welcome to the forum @sina-mansour :) that's a cool idea! You mentioned using a similar format -- did you have something in mind? I'm aware of standards for neuroimaging data (e.g., BIDS), but not for neuroimaging code.
One thing I'm wary of is making these campaigns too complex, because various people have advised me to keep campaigns as simple as possible to reduce barriers to entry. Basically, every requirement you add gives people an opportunity to misunderstand/pause for thought/disagree, and can ultimately mean they don't take the pledge in the first place. E.g. for this idea, I can imagine some people might be happy uploading their code to OSF, but as soon as you start talking about 'forking' and 'Github pages' they will just tune out and we lose them. But I wonder if this could work as a suggestion (e.g. linking to a template as your suggest), rather than a requirement? That way people would still be free to use other methods, if they prefer, and we don't lose them to confusion.
Thanks for all the feedback, I edited the details of the original a little. I guess the main goal is to encourage people who is about to publish their research to also add the code they wrote for greater chance of replicability. Report the version of the libraries, OS, docker containers, etc. Still looking for more feedback on this.
I support this idea of publishing the code. To keep this idea simple, I think if the researchers/authors put a link in the article paper to take you to their code either in GitHub or the institutional website or personal page. It will also help new PhD students in the Neuroscience field to learn and practice to reproduce their work.
Hi @I-Khormi, thanks for showing your support (helps us know which campaigns will have traction when we launch!)
I think if the researchers/authors put a link in the article paper to take you to their code either in GitHub or the institutional website or personal page.
Absolutely, best practices is for authors to link to their code in the paper. I wonder if this should also be part of the pledge, i.e. to provide a link in their paper?
The cool part about some repositories (e.g., OSF) is that you can also get a DOI for your data, so if people were to upload their data AND code, they could get a DOI, which they link to in their paper, and gives people a chance to cite their data/code
@CooperSmout great!! And OSF asks you for anything on particular to let you get a DOI? A link for the paper would be a great idea.
@I-Khormi yes! Authors can put a link of the repo on the same paper to facilitate access to the code.
Great campaign! My thought is whether it would be better to frame the message positively, like "publish all research with code". Framing it as "stop doing" definitely catches attention, but it may be easier for people to put their names with a positive message (since we generally do not want to be told to stop something 😄). Just a thought!
@nsunami Great idea! Yes, people will definetly will be more likely to put their names with a more positive tone. 😄 Thanks!
OSF asks you for anything on particular to let you get a DOI?
The project just needs to be public I believe
better to frame the message positively, like "publish all research with code"
Great suggestion @nsunami, totally agree! Maybe we should edit the campaign proposal template to prompt positive framing in future proposals. For this campaign title, what do people think about "Open code pledge"? This would fit with the other live campaign titles, which are all quite snappy and just focus on the targetted behaviour. Then the subtitle can provide more details, e.g. "Pledge to share code underlying every article you publish, along with X of your peers"?
Great, the official name will change to Open Code Pledge then. We need to make sure people feel encouraged to sign so this positive tone is key. Maybe a video could be done to encourage people to share the idea with peers and every time someone publishes a paper with code they can get a retweet from other people involved in this campaign.
The title sounds great!
... Maybe a video could be done to encourage people to share the idea with peers and every time someone publishes a paper with code they can get a retweet from other people involved in this campaign.
I like the idea of incentivizing the signers in the end (like a Twitter bot to retweet open code papers of those who signed). Perhaps we cannot do this for anonymous signees. But I really like the general idea of getting a reward and celebrating the activation of the pledge.
Great, the official name will change to Open Code Pledge then.
Done :)
Maybe a video could be done to encourage people to share the idea with peers and every time someone publishes a paper with code they can get a retweet from other people involved in this campaign.
These are both great ideas. A video would be great, and now that we have a little bit of funding we can actually pay for someone to make one. If you find any good animators please let me know!
Would also be great if people can retweet. IMO the trouble is motivating action! There's two different timelines here: (1) while collecting pledges, and (2) after pledges become active and people start publishing their papers/code. I think the latter will be easier to get tweets/retweets, because it's a celebration of their paper and we can use the mailing list to ask fellow pledgers to retweet (note @nsunami at this point anonymity isn't a problem, because all names get published when the pledges go live).
The more immediate concern is getting people to promote the campaign before pledges activate, as there's less activity and reason to get excited. Point in case, the Preregistration Pledge had an initial flurry of activity but has since stalled at ~70 pledges (in Psychology), and getting these final few pledges is proving diffficult as I've exhausted most of the communication channels I can think of (if you can think of any extra, especially in Psychology, please add them to the Communication Strategy document).
I just attended an unconference on "Why don't we share data and code?" lead by @dylangomes at the SORTEE conference, where we discussed this proposal and a potential Open Data campaign (which has now been proposed here -- please help develop it!!). Just thought I'd summarise the main points relevant to this campaign (credit: unconference attendees listed in this document; which also contains more discussion points)
- 100 pledges seems like a reasonable target for the field of ecol/evol biol (we should get at least 13 from the unconference)
- would be good to include advice on campaign page about how to upload code
- best to give multiple options for where to upload code, as people have different preferences. Main options raised were: Dryad, Zenodo, Github (though concern over deleteability of accounts), OSF
- some participants preferred the idea of a combined data+code pledge, others preferred to keep pledges separate because not always possible to share data alongside code. I think a neat solution to this could be to keep the pledges separate, but provide a checkbox in each pledge so that people can easily pledge to both campaigns at the same time if they wish. To achieve this, we would just need to release both campaigns at the same time.
- would help to highlight open code practice if we were to write a paper outlining the reasons for/against, ways to upload, response to criticisms etc. If we do this, we could also link to this campaign to demonstrate community support for Open Data.
- useful to include links showing scarcity of data, e.g. this paper for ecol/evol biol
- blog post by @luispedro on open code as a collective action problem
- open research policy
More details in the unconference document for anyone interested (e.g. list of reasons people might not upload data/code)
@CooperSmout I recently got up to date with the documents. I was very busy last weeks due to the Neuromatch. (Which by the way was great, I hope to work on Neuroscience the rest of my life (: ). I think that the reasons listed on why not to share code on the blog and the doc are important but not really an issue. They were rather expected actually. The benefits of this campaign is that it is completely democratic and will set a starting point for more change with the collaboration of many. After all, we are here to help, not to shame. I was thinking that maybe we should work with the first few people interested in joining the campaign, so in that way we make sure the code is excellent (at least to a point) and we can use it as examples on how to start publishing the code more independently with less insecurities, knowing there's a community behind. The benefits will outweigh the cons rapidly. Better programs, better libraries, more room for meta-analysis and more incentive for students to learn how to code. I don't know any animators :( but it shouldn't be so hard to find. If we have a script or a rough idea on how the video should look like I'm pretty sure we can find someone. I'm pretty happy for the funding! What should be the next steps for this campaign?
I was thinking that maybe we should work with the first few people interested in joining the campaign, so in that way we make sure the code is excellent (at least to a point) and we can use it as examples on how to start publishing the code more independently with less insecurities
Absolutely, would be great to have good examples that others can follow in their own practice.
I think this idea could gel nicely with an idea that @malvikasharan of the Turing Way project recently suggested to me. Her idea was to try and find an 'open science champion' to represent each FOK campaign, who we can highlight as an exemplar of the practice in question. Our campaign could bring them recognition for their open science efforts (which are all too often ignored) and in return we could leverage their expertise to (1) teach others how to do it, (2) provide examples to follow (as per your suggestion @rmib200), and (3) answer questions from the community. An effective way to do this could be to organise a workshop, in which our champion teaches the participants how to do the practice, shows some good examples, and answers any questions that participants might have. I was also thinking that we could potentially coincide the workshop with the campaign launch, so that after our champion has given her presentation, we could introduce the campaign, invite feedback, finalise the campaign design and launch it right then and there. Hopefully many of the workshop participants would be willing to sign, and promote the campaign more widely, but at the very least it would be good advertising for the campaign and an opportunity to collect critical feedback (plus highlight our champion's awesome work (: )
What does everyone think of this idea? Would you join us at a workshop @rmib200 @sina-mansour @nsunami @I-Khormi?
Some considerations
- it would be great to select/identify champions from diverse backgrounds, to counteract the usual biases in academia
- maybe we should select a different champion for each research field? E.g. one champion from neuroscience, another from biology, etc., who can teach about best practices within their field? This would mean either hosting a separate workshop for each research field (which would be a lot of work)... or perhaps we could use breakout rooms to allocate people into different rooms within the one workshop?
- Lenny Teytleman (Biologist) springs to mind as a potential champion for this particular campaign, since he founded protocols.io and might be interested in promoting open code. I had contact with someone from Protocols a while back, so could follow up on this if we go with this idea
Ps. @malvikasharan also suggested we could rename this campaign 'Citeable code', to make the benefits of sharing code (with a DOI) explicitly obvious
Hi all, I just received this email from the OHBM organisers -- looks like we'll get to publish a short piece on this campaign (and Project FOK) in OHBM Aperture :) I'm planning to start a Latex/Overleaf document to draft the piece, so let me know if you'd like to help write it and I'll grant you access.
I'm also wondering if this could be a good opportunity to launch the campaign, e.g. by linking to the live campaign in the published article. If so, the article is due 10 Sept, so we would have 1 month to finalise the design and post the campaign to the website.
"As we mentioned during the opening and closing sessions of the event, we are joining forces with the OHBM Aperture journal to create a citable research object out of all the projects from the Brainhack. As project leaders, we wanted to ask you first to confirm that you would be happy to contribute to this paper and then to send us a short submission summarizing your project. The submission should include all the co-authors (everyone who contributed to the project) and the related affiliations, clearly indicate the title of the project, and also describe:
- the motivation for your project;
- what you have found/achieved during the hackathon;
- the implication of your results and/or future steps for the project.
Each submission can have 1 figure and up to 10 citations. The project summary should not be longer than 400 words (without counting title, citations, or authors/affiliations). If possible, we would ask you to send all your submissions by Friday the 10th of September. If you think that this time frame does not work for you, let us know. We look forward to hearing from you and making this Brainhack project paper happen!"
I apologize for the radio silence on this Cooper. If it is helpful to you, I would be happy to contribute to writing or helping others with code sharing during a workshop. If not, that is okay too. So glad to see all these ideas and work on these issues!
Dylan Gomes
On Wed, Aug 4, 2021 at 6:31 AM Cooper Smout @.***> wrote:
Hi all, I just received this email from the OHBM organisers -- looks like we'll get to publish a short piece on this campaign (and Project FOK) in OHBM Aperture :) I'm planning to start a Latex/Overleaf document to draft the piece, so let me know if you'd like to help write it and I'll grant you access.
I'm also wondering if this could be a good opportunity to launch the campaign, e.g. by linking to the live campaign in the published article. If so, the article is due 10 Sept, so we would have 1 month to finalise the design and post the campaign to the website.
"As we mentioned during the opening and closing sessions of the event, we are joining forces with the OHBM Aperture journal to create a citable research object out of all the projects from the Brainhack. As project leaders, we wanted to ask you first to confirm that you would be happy to contribute to this paper and then to send us a short submission summarizing your project. The submission should include all the co-authors (everyone who contributed to the project) and the related affiliations, clearly indicate the title of the project, and also describe:
- the motivation for your project;
- what you have found/achieved during the hackathon;
- the implication of your results and/or future steps for the project.
Each submission can have 1 figure and up to 10 citations. The project summary should not be longer than 400 words (without counting title, citations, or authors/affiliations). If possible, we would ask you to send all your submissions by Friday the 10th of September. If you think that this time frame does not work for you, let us know. We look forward to hearing from you and making this Brainhack project paper happen!"
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FreeOurKnowledge/website/issues/30#issuecomment-892661095, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHYRLDO3GLVNKEJFCO77RB3T3E6LHANCNFSM462AVTIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
-- Cheers, Dylan Gomes
Hi Cooper, I am happy to join your team and help with this document.
Best regards,
Ibrahim
Hi Cooper,
Please accept my apologies for this belated email.
I think the idea of having 'open science champions' sounds really interesting, it could serve as a stage to promote open science practices. I was also thinking of another possible way to promote such practices too. I'm thinking we could create a gallery of scientific work that can resemble and showcase the open science practices being promoted (with a short note explaining the work and links to the openly shared content). This way, the article will get more publicity, and in return, someone who's thinking of implementing open science practices could have plenty of examples to look into. This gallery could be selected and updated to include all the diverse methodologies used in research (different types of studies, programming languages, data sources, etc.).
I'd also be happy to contribute to the writing of the manuscript, and/or help to organize workshops (i.e. using technologies like git and zenodo to make code openly available and citable) to promote the campaign.
Kind regards, Sina
On Thu, 5 Aug 2021 at 10:50, Ibrahim Khormi @.***> wrote:
Hi Cooper, I am happy to join your team and help with this document.
Best regards,
Ibrahim
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FreeOurKnowledge/website/issues/30#issuecomment-893077999, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD235MVIZW3FUFGHN7DOSVDT3HN6DANCNFSM462AVTIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
Awesome thanks all. I'll set up a Latex document in the next couple of weeks and link it to you when it's ready.
I think the idea of having 'open science champions' sounds really interesting, it could serve as a stage to promote open science practices. I was also thinking of another possible way to promote such practices too. I'm thinking we could create a gallery of scientific work that can resemble and showcase the open science practices being promoted (with a short note explaining the work and links to the openly shared content). This way, the article will get more publicity, and in return, someone who's thinking of implementing open science practices could have plenty of examples to look into.
I like this idea, particularly if we tie it into the campaign to celebrate our pledging community. In the preregistration pledge campaign we're planning to link to people's preregistrations once they complete them, to increase visibility and celebrate those people who complete their pledge. So we could do something similar here, linking to people's code (and relevant papers) once they share their code. Does that fit with your vision?
I'd also be happy to contribute to the writing of the manuscript, and/or help to organize workshops (i.e. using technologies like git and zenodo to make code openly available and citable) to promote the campaign.
Thanks for the offer! Will certainly be needing some help on the workshop, if we go that direction. I was also thinking we could reach out to OHBM and see if they can support a workshop, either with resources/expertise.
By the way, I reposted the champion/workshop idea here to open it up for discussion, since it's a general idea that could relate to other campaigns as well (also, curious to test out this new 'Discussions' feature in Github).
I also posted a new idea, about publishing pledges in an indexed journal, with a view to increasing visiibility and creating citable documents reflecting people's pledges. Would love to hear if you have any thoughts on that idea, as it's something that we could potentially do for this campaign.
All these ideas are incredible. @CooperSmout What should be the focus of the paper? Best practices? Future directions? Maybe a global scope of democratic science practice possibilities? I'm very excited that this is getting attention!. Regarding the champions, when shoud we start sending the invitations? can we make a database with possible names and availabilities? And when the workshop could take place? This year? Also, I never used latex, but I'm ready to learn!.
Ps. @malvikasharan also suggested we could rename this campaign 'Citeable code', to make the benefits of sharing code (with a DOI) explicitly obvious
Yeah, I see the potential of making the title more explicit. But I think that just 'Citeable code' is not very clear. Maybe something like 'Citeable code 4 science' or 'Citeable code 4 neuroscience' something like that. Does anyone has any ideas maybe? please feel free to share.
One option is
Open code: reproducible and citable
You can add science if you want:
Open code in science: reproducible and citable
OR
something like
Open science: reproducible research and citable code (the alliteration here is nice)
I think the word reproducible is more important to include, in my opinion. Being citable is a nice incentive, but the *goal *is to have reproducible work - is it not?
Cheers, Dylan Gomes
On Mon, Aug 9, 2021 at 7:44 AM Manuel Illanes @.***> wrote:
Ps. @malvikasharan https://github.com/malvikasharan also suggested we could rename this campaign 'Citeable code', to make the benefits of sharing code (with a DOI) explicitly obvious
Yeah, I see the potential of making the title more explicit. But just 'Citeable code' is not very clear. Maybe something like 'Citeable code 4 science' or 'Citeable code 4 neuroscience' something like that. Does anyone has any ideas maybe? please feel free to share.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FreeOurKnowledge/website/issues/30#issuecomment-895283787, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHYRLDIAU3BNUYCL3XAIYFTT37STDANCNFSM462AVTIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .
Hi all, sorry for the delay on this, I've been swamped with other parts of the project. But I've now drafted a first version of the Aperture submission, which you can see here (I ended up just using Google Docs because it was easier and my Zotero references were already set up). I'm hesitant to share an edit link here so please contact me via email if you would like edit rights.
I was also wondering if those of you who are keen to join the paper would consider becoming 'ambassadors' for the campaign? This is something I've been wanting to implement since the beginning of FOK but haven't had the time or volunteers to do so. The main idea here would be to increase visibility on the campaign, using small but repeatable actions (e.g. retweeting each others tweets about the campaign). I'm not anticipating a big time commitment, perhaps 15 mins/week -- the main thing is to just keep the campaign fresh in people's minds (and also our own minds, so that we remember to mention it where relevant).
Since this is another generic idea that can apply to all campaigns, I've started a new thread here to discuss the idea. Would appreciate any comments or suggestions you might have!