elsa icon indicating copy to clipboard operation
elsa copied to clipboard

QA Spider

Open encukou opened this issue 8 years ago • 8 comments

Make a spider tool that will go through all pages of a site and check stuff. It would run as part of CI and reject bad PRs. This might make more sense as a separate project, not part of elsa, but tracker currently this seems like the best place to track the idea. Please coordinate here before starting work; a few people are exploring the idea.

Ideas for checks:

  • [ ] Weed out 404s/500s in internal links
  • [ ] Check external links
  • [ ] Check orphaned pages (frozen, but no links to them)
  • [ ] Validate HTML
  • [ ] Verify slashes at end of URLs
  • [ ] Measure code coverage
  • [ ] Lint URL structure (e.g. removing last URL segment results in valid URL)

encukou avatar Mar 16 '17 15:03 encukou

I currently use this hack for naucse.python.cz: https://gist.github.com/encukou/6f0038a3e104dfbed8a08031de03ab0b

encukou avatar Mar 16 '17 15:03 encukou

Note that LinkChecker and similar tools exist, but all seem to target already deployed sites.

encukou avatar Mar 16 '17 16:03 encukou

Already deployed sites - not a problem, we can freeze --serve and run it on localhost, can't we?

hroncok avatar Mar 16 '17 16:03 hroncok

Indeed (it won't find orphaned pages, but I guess that's not that important)

encukou avatar Mar 16 '17 16:03 encukou

Check orphaned pages (frozen, but no links to them)

I think Flask-Frozen does this out of the box, sort of. It follows links and if there is a route which isn't linked, you need to write a generator for it. So it should be enough if you don't write any generators 😄

honzajavorek avatar Aug 10 '18 11:08 honzajavorek

Also, many of the points are provided by Sphinx extensions. I'm wondering whether we're not reimplementing a wheel here.

honzajavorek avatar Aug 10 '18 11:08 honzajavorek

Sphinx is nice, but I don't think it's really useful for sites that run on Elsa – I wouldn't want to rewrite pyladies.cz, fedoralovespython.org to ReST... Something generic (or at least specific to Flask) would be more appropriate.

encukou avatar Aug 10 '18 11:08 encukou

Makes sense. I thought more about Naucse.

On Fri, 10 Aug 2018 at 13:55, Petr Viktorin [email protected] wrote:

Sphinx is nice, but I don't think it's really useful for sites that run on Elsa – I wouldn't want to rewrite pyladies.cz, fedoralovespython.org to ReST... Something generic (or at least specific to Flask) would be more appropriate.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pyvec/elsa/issues/33#issuecomment-412060665, or mute the thread https://github.com/notifications/unsubscribe-auth/AARTMXXD8g6T8fzwK2bpYWUw-pqXr9Hdks5uPXSXgaJpZM4MfgnD .

honzajavorek avatar Aug 10 '18 17:08 honzajavorek