3. What are Recipes, Recipe Libraries, and Primitives

A DRAGONS recipe is a set of instructions, called primitives, that processes data in a certain way. Depending on the recipe, the input data can be in any state of processing. Most often though, people will want to process Gemini raw data into a master calibration or a processed calibrated output.

A recipe library is a collection of recipes. The recipes in a library have one thing in common: they all apply to the same type of data. When DRAGONS search for a matching recipe for some input data, it searches for a recipe library. In each library, one recipe is set as the default recipe. To use the others, the user needs to specify the name of the non-default recipes.

A primitive is the function that does something to the data. The name of the primitive normally gives you a good idea of what its purpose is.

3.1. Recipes

This is what a recipe looks like:

def reduce(p):
   p.prepare()
   p.addDQ()
   p.removeFirstFrame()
   p.ADUToElectrons()
   p.addVAR(read_noise=True, poisson_noise=True)
   p.nonlinearityCorrect()
   p.darkCorrect()
   p.flatCorrect()
   p.separateSky()
   p.associateSky(stream='sky')
   p.skyCorrect(instream='sky', mask_objects=False, outstream='skysub')
   p.detectSources(stream='skysub')
   p.transferAttribute(stream='sky', source='skysub', attribute='OBJMASK')
   p.clearStream(stream='skysub')
   p.associateSky()
   p.skyCorrect(mask_objects=True)
   p.detectSources()
   p.adjustWCSToReference()
   p.resampleToCommonFrame()
   p.stackFrames()
   p.writeOutputs()

A recipe is a Python function that calls “primitives” sequentially. The user does not need to know Python to understand more or less what is being done to the data when the recipe is run. From the recipe above, we can read that the data will be corrected for dark current, for flat field effects, the sky background will be subtracted and the frames will be stacked. The p argument is the primitive set that matches the input.

3.2. Recipe libraries

Recipes are stored in recipe libraries, or in Python language, a module. A recipe library can have one or more recipes in them. One recipe is identified as the default recipe.

When DRAGONS searches for a recipe, it is actually searching for a matching recipe library. Once found, it will run the default recipe, unless instructed otherwise by the user.

The recipe libraries are associated with the input files by matching astrodata tags (Astrodata User Manual). The tags are qualifiers like “NIRI”, “IMAGE”, “FLAT”. The tags of the first file in the list of inputs are used. Each recipe library is assigned a set of tags that defines which type of data this library is for.

3.3. Primitives and primitive sets

A primitive is a data reduction step involving a transformation of the data or providing a service. By convention, the primitives are named to convey the scientific meaning of the transformation. For example biasCorrect will remove the bias signal from the input data.

A primitive is always a member of a primitive set. It is the primitive set that gets matched to the data by the Recipe System, not the individual primitives.

Technically, a primitive is a method of a primitive class. A primitive class gets associated with the input dataset by matching the astrodata tags. Once associated, all the primitives in that class, locally defined or inherited, are available to reduce that dataset. We refer to that collection of primitives as a “primitive set”.

3.4. Primitive input parameters

Also attached to a primitive set are the input parameters for each primitives and the defaults appropriate for that primitive set, that is for that type of data.

For the same generic primitive, the default values for input parameters for NIRI data can be different from the defaults applicable to GMOS data. Even the set of available input parameters can be different, though we try to keep things as uniform as possible.

3.5. Exploring recipes and primitives

3.5.1. showrecipes

showrecipes allows the user to see which recipe library gets picked up by the system and which recipe is run by default, which others are available.

The syntax is:

showrecipes fitsfile_name.fits

Let’s look at a couple examples. From the niriimg_tutorial/playdata/example1/ directory:

Recipe not provided, default recipe (makeProcessedFlat) will be used.
Input file: /Users/klabrie/data/tutorials/niriimg_tutorial/playdata/example1/N20160102S0373.fits
Input tags: ['GCALFLAT', 'NORTH', 'AT_ZENITH', 'NON_SIDEREAL', 'AZEL_TARGET', 'RAW', 'IMAGE', 'GCAL_IR_ON', 'NIRI', 'GEMINI', 'UNPREPARED', 'LAMPON', 'CAL', 'FLAT']
Input mode: sq
Input recipe: makeProcessedFlat
Matched recipe: geminidr.niri.recipes.sq.recipes_FLAT_IMAGE::makeProcessedFlat
Recipe location: /Users/klabrie/condaenvs/public3.10_3.1.0/lib/python3.10/site-packages/geminidr/niri/recipes/sq/recipes_FLAT_IMAGE.py
Recipe tags: {'FLAT', 'IMAGE', 'CAL', 'NIRI'}
Primitives used:
   p.prepare()
   p.addDQ()
   p.addVAR(read_noise=True)
   p.nonlinearityCorrect()
   p.ADUToElectrons()
   p.addVAR(poisson_noise=True)
   p.makeLampFlat()
   p.normalizeFlat()
   p.thresholdFlatfield()
   p.storeProcessedFlat()

This contains: the name of the recipe, the location of the recipe library, the :astrodata: tags of the input file, those assigned to the recipe library, and the recipe itself.

A library can have more than one recipe, to see them all use the --all flag:

showrecipes N20160102S0270.fits --all

Input file: /Users/klabrie/data/tutorials/niriimg_tutorial/playdata/example1/N20160102S0270.fits
Input tags: {'NIRI', 'NORTH', 'GEMINI', 'UNPREPARED', 'IMAGE', 'RAW', 'SIDEREAL'}
Recipes available for the input file:
   geminidr.niri.recipes.sq.recipes_IMAGE::alignAndStack
   geminidr.niri.recipes.sq.recipes_IMAGE::makeSkyFlat
   geminidr.niri.recipes.sq.recipes_IMAGE::reduce
   geminidr.niri.recipes.qa.recipes_IMAGE::makeSkyFlat
   geminidr.niri.recipes.qa.recipes_IMAGE::reduce
   geminidr.niri.recipes.sq.recipes_IMAGE::alignAndStack
   geminidr.niri.recipes.sq.recipes_IMAGE::makeSkyFlat
   geminidr.niri.recipes.sq.recipes_IMAGE::reduce

The recipe library for science quality has three recipes: alignAndStack, makeSkyFlat, and reduce.

Note

Regarding the “sq” and “qa” in the paths. DRAGONS has the concept of “reduction mode”. Right now, there are two modes: the science quality mode, “sq”, and the quality assessment mode, “qa”. You can safely ignore the “qa” mode, it is used exclusively at the observatory, at night, to help with the assessment of the sky conditions and the resulting quality of the data. Everything defaults to “sq”.

The last three “sq” recipes are really the “ql” recipes. This a known bug (circa April 2023). The NIRI quicklook recipes are identical to the science recipes and are just “Python imported” from the science module, and that import trips the current implementation of showrecipes.

To see what a specific recipe looks like, not just the default recipe, use the -r flag:

showrecipes N20160102S0270.fits -r makeSkyFlat

Input file: /Users/klabrie/data/tutorials/niriimg_tutorial/playdata/example1/N20160102S0270.fits
Input tags: ['UNPREPARED', 'NORTH', 'NIRI', 'SIDEREAL', 'RAW', 'IMAGE', 'GEMINI']
Input mode: sq
Input recipe: makeSkyFlat
Matched recipe: geminidr.niri.recipes.sq.recipes_IMAGE::makeSkyFlat
Recipe location: /Users/klabrie/condaenvs/gemini3_2.1.x_20200925/lib/python3.6/site-packages/dragons-2.1.1-py3.6-macosx-10.7-x86_64.egg/geminidr/niri/recipes/sq/recipes_IMAGE.py
Recipe tags: {'IMAGE', 'NIRI'}
Primitives used:
   p.prepare()
   p.addDQ()
   p.ADUToElectrons()
   p.addVAR(read_noise=True, poisson_noise=True)
   p.nonlinearityCorrect()
   p.darkCorrect()
   p.stackFrames(operation='median', scale=True, outstream='fastsky')
   p.normalizeFlat(stream='fastsky')
   p.thresholdFlatfield(stream='fastsky')
   p.flatCorrect(flat=p.streams['fastsky'][0], outstream='flattened')
   p.detectSources(stream='flattened')
   p.dilateObjectMask(dilation=10, stream='flattened')
   p.addObjectMaskToDQ(stream='flattened')
   p.writeOutputs(stream='flattened')
   p.transferAttribute(source='flattened', attribute='mask')
   p.stackFrames(operation='mean', scale=True, reject_method="minmax", nlow=0, nhigh=1)
   p.normalizeFlat()
   p.thresholdFlatfield()
   p.storeProcessedFlat(force=True)

3.5.2. showpars

Primitive input parameters can be customized. To see the available input parameters, their default values, and range of allowed values, use showpars. The syntax is:

showpars filename.fits primitive_name

A dataset is required here because the parameters and their defaults do change depending on the type of data. Defaults between instruments can be different because the data has different characteristics. Also, the primitive might be doing slightly different things (eg. optical vs near-IR).

Let’s say that we want to reduce NIRI science data, an example of which is file N20160102S0270.fits but we want to customize the sky correction, the dark correction, and the rejection method during the final stack. Here’s how one would proceed to check for primitive names and parameters names.

showrecipes N20160102S0270.fits

...
   p.prepare()
   p.addDQ()
   p.removeFirstFrame()
   p.ADUToElectrons()
   p.addVAR(read_noise=True, poisson_noise=True)
   p.nonlinearityCorrect()
   p.darkCorrect()
   p.flatCorrect()
   p.separateSky()
   p.associateSky(stream='sky')
   p.skyCorrect(instream='sky', mask_objects=False, outstream='skysub')
   p.detectSources(stream='skysub')
   p.transferAttribute(stream='sky', source='skysub', attribute='OBJMASK')
   p.clearStream(stream='skysub')
   p.associateSky()
   p.skyCorrect(mask_objects=True)
   p.detectSources()
   p.adjustWCSToReference()
   p.resampleToCommonFrame()
   p.stackFrames()
   p.writeOutputs()

Here I spot the primtives: skyCorrect, darkCorrect, and stackFrames.

If I wanted to turn off the dark correction…

showpars N20160102S0270.fits darkCorrect

Dataset tagged as {'NORTH', 'IMAGE', 'NIRI', 'UNPREPARED', 'SIDEREAL', 'GEMINI', 'RAW'}
Settable parameters on 'darkCorrect':
========================================
 Name                       Current setting

do_cal               'procmode'           Calibration requirement
Allowed values:
    procmode        Use the default rules set by the processingmode.
    force   Require a calibration regardless of theprocessing mode.
    skip    Skip this correction, no calibration required.
    None    Field is optional

suffix               '_darkCorrected'     Filename suffix
dark                 None                 Dark frame

I would set do_cal to “skip”.

If I wanted to change the operation done when combining sky frames to a mean…

showpars N20160102S0270.fits skyCorrect

...
operation            'median'             Averaging operation
Allowed values:
    mean    arithmetic mean
    wtmean  variance-weighted mean
    median  median
    lmedian low-median
...

I would reset operation to mean.

If I wanted to change the rejection method when doing the final stack…

showpars N20160102S0270.fits stackFrames

...
reject_method        'sigclip'            Pixel rejection method
Allowed values:
    none    no rejection
    minmax  reject highest and lowest pixels
    sigclip reject pixels based on scatter
    varclip reject pixels based on variance array
...

I would pick one of the allowed value for reject_method.