rich metadata proposal in VOTable

[gilleslandais]

Dear astroquery team! in the VO context, we promote a rich VOTable called Mango (see https://github.com/ivoa/dm-usecases/wiki/mango). It is not yet a standard, but we are very interested to have astroquery feedbacks :-) .

The idea consists to add an annotation block in VOTable to describe celestial sources who are in the table. The serialization is based on the Mango DM (not a standard yet) and highlights measures (like position, photometry, parallaxes or other quantities) - The XML annotation uses "VOdmlInstance" which seems difficult at first glance but which enables to serialize complex models. Among the new capabilities, we could provide the photometry and their filter assignment. In the VizieR prototype, columns describing the same quantity are grouped in a measure (for instance a velocity measure which groups the velocity column and columns like a quality flag; an error, etc.) -

more details are available here : https://github.com/ivoa/dm-usecases/wiki/mango example : http://viz-beta.u-strasbg.fr/viz-bin/Mango?-out.max=10&-source=II/305/archive

• An other development concerns the Provenance of the VizieR catalogue. Today provenance information in VOTable is very poor, so we developed a Provenance output dedicated for users (e.g: authors, article, filter assignment..). The output is still in test and can evolve - it is built on the IVOA ProvDM.

example: https://cdsarc.unistra.fr/viz-bin/provenance?cat=II/305&filter=true&out=yaml https://cdsarc.unistra.fr/viz-bin/provenance?cat=II/305&filter=true&

[keflavich]

I think the first thing to check is if astropy.table can read these new VOTables and how they present to users. Maybe try adding a quick demo here?

[bsipocz]

I agree with Adam that this seems more like a question for astropy core, as we don't do any parsing here but rely on the astropy.io.votable module from core (or if something more complicated on top of that, on pyvo). So I would also cc @tomdonaldson here as the votable maintainer in astropy, as well as a pyvo maintainer.

[tomdonaldson]

@gilleslandais Where this sort of functionality belongs is an interesting question.

• As @bsipocz said, the fundamental VOTable parsing (and writing) happens in astropy core (astropy.io.votable).

• The pyvo package depends on astropy and implements some value-added features for querying using various standards which includes some semantics and assumptions about the VOTable results based on the standard used in the query. It also has a more robust representation of the VOTable itself which might be amenable to Mango metadata.

• Then astroquery depends on pyvo for its TAP and DataLink features.

It's clear that anything that is a VO standard should either be in astropy.io.votable or pyvo. Which of those is the best place is still an interesting question and will likely depend on the feature, and how fundamental it is to VOTable I/O.

For features that are specific to a data provider and will likely never be standard, then astroquery may be the most sensible place for the feature.

For features that are in between, i.e., working towards becoming a standard, then it's less clear, but again will depend on the feature. For example, with Mango, even with a desire to keep the functionality modularly separate from astropy.io.votable, that module will likely still need updates to accept the new metadata. That said, pyvo seems like a better sandbox for progressing a standard than astropy where possible.

[bsipocz]

As per the last comment, I moved this issue upstream from astroquery here to pyvo, but happy to move even further up in astropy if you think it's better placed there.

[lmichel]

The VOTable annotation syntax (MIVOT) entered last month in the recommendation step. This sequence is not completed but we can bet that no major changed will occur. In parallel with the Gilles's work on Vizier I would like to inform the community about the ongoing work on the model feature implementation in AStropy/PyVO.

There are some design notes here resulting from discussion we have had on that topic.

• I exercised the model annotation processing on my own forks of both astropy and pyvo.
• I also put a little implementation note on the Pyvo wiki

Notice the this code would require serious clean up before to be merged