Fedora improvements

[decimalator]

Descriptive Summary

Our single replica for od2-prod/fcrepo can handle a decent amount of load, but not enough to handle large exports. Reindexing, exporting and any operations that require a large number of Fedora requests are taking a really long to respond. Storage speed is a big part of the issue, but there are still improvements that we can make to help Fedora scale.

• Worker node tuning
• Fedora container tuning
• Fedora Clustering via Infinispan / JGroups
• Storage improvements
• Better logging

Storage improvements are in progress, but are awaiting delivery of our new storage hardware, estimated mid-April.

Worker node tuning

• [ ] Investigate additional node level tuning options

• update remaining nodes for a consistent set of options across the cluster

• sysctl

• Docker

• kubelet

Fedora container tuning

• [ ] Investigate additional container level tuning options

Fedora Clustering

• [ ] Investigate Tomcat clustering for HA and load balancing and see if it's stable enough for production.

https://wiki.lyrasis.org/display/FEDORA46/Deploying+a+Fedora+Cluster

Fedora Configuration Tuning

• [ ] Investigate using PostgreSQL as a backend in place of MySQL for more stable performance with larger repositories

https://wiki.lyrasis.org/display/FEDORA47/MySQL%2C+PostgreSQL%2C+and+LevelDB+Performance

Storage Improvements

OSULP LIT has new storage hardware ordered that we're hoping to have delivered and setup by mid-April. Once Fedora content storage has been migrated to the new storage we should see a lot of improvements in response times. OSU Infrastructure's capacity storage is pretty slow compared to contemporary storage.

Better output from Fedora

Troubleshooting Fedora issues is pretty painful at the moment with our current level of logging.

• [ ] Logging improvements
• [ ] Investigate adding JMX metrics

https://wiki.lyrasis.org/display/FEDORA47/Metrics+Reporting https://wiki.lyrasis.org/display/FEDORA47/Setup+a+Graphite+instance