IRAF tasks of interest to be replaced

[pllim]

This was originally sent to STScI as a help call (INC0131510).

I list below the IRAF tasks that I would most like to see 'carbon copy' replacements for in Python.

1. splot
2. ccdproc - especially interactive bias overscan fitting and trimming
3. longslit.identify, reidentify, fitcoords, transform, standard, sensfunc
4. imcopy with the ability to select image sections and extensions
5. implot

cc @crawfordsm

p.s. Feel free to crossref with other issues as appropriate.

[jehturner]

Changed longlist -> longslit :slightly_smiling_face:.

My 2c: At Gemini, @chris-simpson has been making significant progress towards 3, which we also need, but not as carbon copies -- we need to do better than IRAF in some respects and don't think cloning it is the way to go. Our early work is currently more tied to our pipeline system than I would like, but the algorithm underlying the line matching is fairly general. You can see our in-progress pipeline "primitives" at https://github.com/GeminiDRSoftware/DRAGONS/blob/spectroscopy/geminidr/core/primitives_spect.py. I think Chris is planning to give a talk on wavelength calibration at ADASS. Our pipeline also does bias/overscan, but there's already Python ccdproc for that stuff(?). Our support for plotting & interaction is currently very minimal & primitive. It's certainly true that as IRAF disappears, better Python support for basic interactive tasks like display, plotting, manipulating files, checking stats and so on is going to become more important (as opposed to replacing tools for use in DR pipelines), but that's more general than specreduce. I think 1 may belong in https://github.com/astropy/specutils?

[crawfordsm]

For reference, identify is almost fully reproduce by: https://github.com/crawfordsm/specidentify

This is an interactive PyQT GUI that is generalized for identifying spectral lines and can be used with any system. It also has some of the functionality of reidentify but uses different algorithms.

However, this does return a different type of object than in identify. It is a dictionary of astropy.models that can be used for transforming the data or creating a new WCS for the data.

[crawfordsm]

General routines that Gemini has developed and want to move upstream would be very welcome here to make the available to a wider audience and reduce redundancy. Please consider/encourage them to be migrated to here!

[crawfordsm]

And for completeness:

1. splot would be covered by https://github.com/spacetelescope/specviz
2. ccdproc is covered by https://github.com/astropy/ccdproc
3. This functionality still needs to be implemented along with other functionality in apsum
4. This functionality is in astropy.nddata including CCDData: https://stak-notebooks.readthedocs.io/en/latest/images.imutil.html#imcopy
5. implot: Glue handles some of this work along with jupyter widgets, but there is a good one to one conversion at this time

[kakirastern]

I am starting work on porting the tasks standard, sensfunc, and calibrate from the Gemini PyRAF noao package's onedspec directory to specreduce for flux calibration, as per a Slack conversation with @crawfordsm moments ago.

[jehturner]

Thanks. These tasks actually come with IRAF and are not part of the Gemini IRAF package per se (though it does make use of them). The Gemini package does have a look-up table for an extra standard (EG131, in gmos/calib).

[pllim]

Someone asked about mkmultispec today. Should I add it to the list here?

http://www.stsci.edu/itt/review/dhb_2011/Intro/intro_ch37.html#1927923

mkmultispec: This task writes wavelength information into the header of a flux file while preserving all the original information. It is therefore a better choice than resample for most applications, and we describe it in more detail below.

[stscicrawford]

Mkmultispec probably should be part of specutils -- as it is involved with input/output and the format of spectra

[kakirastern]

Found this recently: https://pydis.readthedocs.io/stepbystep.html#flux-calibration

Could be used as a reference for the code needed in specreduce, though in reality to really generalize I figured more work might be needed to achieve that goal.

[kakirastern]

Hi @pllim I have finally submitted a PR with a draft of some flux calibration utilities for specreduce in #45, and I was in touch with someone else (James Davenport of pydis) working on a similar task as well who will probably fine tune this work further with another PR to replace it. So the standard and sensfunc will likely be taken care of during the Spectroscopy Sprint to take place between the coming week during April 6-10, 2020.

[pllim]

Someone asked about continuum IRAF task today. Does it have a replacement yet?

[jehturner]

I've never really used it much, but it seems like it's basically fit1d, except that it can understand the IRAF WCS and rescale spectra with different wavelength solutions? We do have an implementation of fit1d (which may change a bit) but currently it depends on an implementation of spline fitting in DRAGONS because modeling doesn't have one (which I understand is a more difficult problem to solve). https://github.com/GeminiDRSoftware/DRAGONS/blob/sky_sub/gempy/library/fitting.py This requires #10687.

[pllim]

You mean astropy/astropy#10687 ? Thanks, @jehturner ! I'll pass this info on to the inquirer.

[jehturner]

I think everything needed is in AstroPy 4.2 now. We've just refactored that fit_1D code (same link) into a class that now allows evaluation on user-specified co-ordinates and exposes the rejected pixel mask, which should be closer to its final form.

That should also simplify use cases like the model set fitting example in the latest astropy documentation?