codalab-competitions
codalab-competitions copied to clipboard
codalab
Suggestions for V2/V1.5 Wishlist
I am referencing here old suggestions that we can revisit for the design of V2:
- Hall of fame: https://github.com/codalab/codalab-competitions/issues/1196
- Calendar view of events (Chagle feature?) https://github.com/codalab/codalab-competitions/issues/1887
- Enhanced user profile info: https://github.com/codalab/codalab-competitions/issues/511
- Better admin features: https://github.com/codalab/codalab-competitions/issues/1025, https://github.com/codalab/codalab-competitions/issues/1617
- Execution wrapper: https://github.com/codalab/codalab-competitions/issues/991, related to https://github.com/codalab/codalab-competitions/issues/1913
- Dataset submission: https://github.com/codalab/codalab-competitions/issues/1310
- Multiple tracks: https://github.com/codalab/codalab-competitions/issues/1201
- Improve user panel: https://github.com/codalab/codalab-competitions/issues/1245
- Improve leaderboard: https://github.com/codalab/codalab-competitions/issues/1341; change leaderboard from editor: https://github.com/codalab/codalab-competitions/issues/1555; mode to show multiple submissions by the same user/team https://github.com/codalab/codalab-competitions/issues/1614
- re-think coopetitions https://github.com/codalab/codalab-competitions/issues/1618
- create evaluation with dependencies #1948
- automatic tweeting to get presence on twitter #1192
- nicely display "time remaining" in hours/seconds underneath competition end date
- More meaningful error messages https://github.com/codalab/codalab-competitions/issues/1626
- Better email notifications https://github.com/codalab/codalab-competitions/issues/1637
- Better sorting of competitions into student/research/trial/ etc. avoid accumulating dummy competitions https://github.com/codalab/codalab-competitions/issues/1641
- Better exception handling: https://github.com/codalab/codalab-competitions/issues/1647
- Pre-registering groups of people: https://github.com/codalab/codalab-competitions/issues/1649
- Elastic compute workers: https://github.com/codalab/codalab-competitions/issues/1606, https://github.com/codalab/codalab-competitions/issues/1889
- Allow NOT using cloud storage: https://github.com/codalab/codalab-competitions/issues/1720
- Work on task program and ingestion program: https://github.com/codalab/codalab-competitions/issues/1906, https://github.com/codalab/codalab-competitions/issues/1913
- Clean up the notion of team: https://github.com/codalab/codalab-competitions/issues/1648
- Completely close accounts: https://github.com/codalab/codalab-competitions/issues/2006, also opt-out of notifications https://github.com/codalab/codalab-competitions/issues/2005
- Changes to ingestion program https://github.com/codalab/codalab-competitions/issues/1987
- Points system/gamification #909
- Submit via worksheet
- Ability to "manage submissions" -- Instead of phase-to-phase migrations, create a nice screen for handling running submissions. You can move them between phases manually. You can lock phases down, open them, etc. easily.
- Image optimization after any image has been uploaded (resize properly, tinypng, compression, etc.)
- Add a "slug" section when you edit competitions. The "slug" will be the competition name in the url.
https://codalab.org/c/<your slug>will redirect to your competition
This is a list of bugs or enhancements with lower priority:
- [x] Detailed results not properly displayed: https://github.com/codalab/codalab-competitions/issues/1741
- Add search bar when logged in: https://github.com/codalab/codalab-competitions/issues/1740
- Make it possible to download multiple submissions: https://github.com/codalab/codalab-competitions/issues/1632
- Fix static files in competition dumps: https://github.com/codalab/codalab-competitions/issues/1897
- Leaderboard does not behave when phases are added from the editors: https://github.com/codalab/codalab-competitions/issues/1693
- Show ranks in separate fields in CSV files: https://github.com/codalab/codalab-competitions/issues/1672
- Improve mail message capabilities (include HTML, attachments, etc.) https://github.com/codalab/codalab-competitions/issues/1654
- Improve HTML editor https://github.com/codalab/codalab-competitions/issues/1635
Adding a couple of features from #2115
- Filtering based on a very simple, generic tag. Ex: NLP, CV, BioMedical etc
- Sorting wrt End Date.
- Filter for open/closed competitions.
- Make it so scoring is not delayed by other submissions, scoring tasks should be higher priority
-
Compute worker chunked upload -- replace
put_blob - Make deleting a submission also cancel the Celery task
- Add a "display name" profile field
- Similar to organizer deleting submission cancels it, we should have a "cancel submission" button that regular user can run
- Create a bundle checker
- migrate users to new competition
- Add a simple button "Request Chahub Access"
Submissions limit
- Less than 1 maximum_submissions_per_day: #2434
- Set a maximum number of submissions per team? #2751
Adding more information in a string type column in the leaderboard:
- Show method details on leaderboard: #2378
- How to add string column in leaderboard table: #2404
- Can I add text type column for leaderboard: #2406
Registrations and un-registrations
- Spam messages in Forum: #2627
- Hall of fame opt in + explanation: #2349
- [Suggestion] Registration Timestamp: #2203
GUI
- [x] Add "health/simple_status" page to "Admin monitoring links"
Baselines
- [ ] Being able to to mark some submissions as baselines in the leaderboard (highlight)
- [x] Let the organizers put several baseline scores in the leaderboard without creating new accounts https://groups.google.com/g/codalab-competitions/c/gFtfFoaBlRs
Example ticket #3155
Documentation
- Improve documentation about competition Docker image and how to define the evaluation environment https://groups.google.com/g/codalab-competitions/c/-9nkgisJfAQ