7. Local calibration manager

The local calibration manager is the tool that will allow DRAGONS to find the most appropriate processed calibration for the data being reduced.

The local calibration manager contains all the same rules as the Gemini Observatory Archive, but works locally, with your files, without internet, and alongside DRAGONS to make the calibration association as seamless as possible. As we have seen in the demo earlier, we create the master calibrations, we add them to the database, and when DRAGONS (reduce) needs one, it gets matched and retrieved automatically.

The system uses a lightweight sqlite database that will not store the data file, it will store only information about the file, like its location on disk, and key astrodata tags and descriptors.

Warning

If you move or delete a processed calibration on disk after having entered it in the database, the calibration manager will no longer be able to find it. The database does not contain the file, just information about the file.

See the caldb documentation for more information, including information about the programming interface.

7.1. Configuration and initialization

Warning

The information provided here applies to DRAGONS version 3.1 (and newer). The configuration and features of the calibration manager in the previous versions, 3.0 and older, are substantially different. Please refer to a previous version of this workhop if you are using DRAGONS 3.0 or older,

7.1.1. Configuration

The behavior and configuration of the local calibration manager are controlled in the file ~/.dragons/dragonsrc. When you first install DRAGONS, you will have to create that directory and that file.

If you ran the demo earlier, the dragonsrc file should look like this:

[calibs]
databases = <where_the_data_package_is>/niriimg_tutorial/playground/cal_manager.db get

The value of databases can be any existing path on your machine. The calibration database will be created, and then expected to be in that directory. The database name can be anything, here we name it cal_manager.db for continuity with pre-v3.1 DRAGONS.

We recommend setting up a new database for each observing project. It helps keep things clean and avoids calibrations from other programs being unexpectedly picked up. (If they were picked up, they would be a match, but it might get confusing if you are expecting another calibration to be picked up.)

Here is an example of a dragonsrc with several databases entries. Note that at all times, only one databases directive is active, the others are commented out.

[calibs]
databases = /Users/klabrie/data/tutorials/niriimg_tutorial/playground/cal_manager.db get
#databases = /Users/klabrie/data/tutorials/gmosimg_tutorial/playground/cal_manager.db get

#databases = /Users/klabrie/data/workspace/GMOS540feature/cal_manager.db get

7.1.2. Initialization

Once your dragonsrc is set up, you can create a new database with

caldb init

If the database already exists, that command will refuse to work. This ensures that you will not wipe the database accidentally. The init is required only once. If you already have the database created, maybe even populated, you can switch back to it simply by ensuring databases points to it.

Exercise - Caldb 1

Create a new database in the niri_tutorial directory. Keep the playground entry that should already be there, but deactivate it.

Confirm with ls that a new cal_manager.db has been created. You can also use caldb config to confirm that the new database is active.

[Solution]

7.2. Usage

The usage is fairly basic. One can list the content, add content, and remove content. DRAGONS will retrieve content from the database. If the “store” flag is set, DRAGONS will also automatically add content to the database.

Warning

In DRAGONS version 3.0 and older, DRAGONS cannot add content automatically, the user must add the processed calibrations to the database with an explicit caldb add.

To verify which database is currently active, use the config option:

caldb config

To see the content of the database:

caldb list

To add information about a processed calibration to the database:

caldb add name_of_file.fits

And finally, to delete information about one file from the database:

caldb remove name_of_file.fits

remove will only remove the entry in the database, it will not remove the file on disk.

Exercise - Caldb 2

Do Exercise - Caldb 1 first.

  1. Add the flat field from the demo to the new calibration manager created in the first exercise.

  2. Show the content of that database is indeed just that file.

  3. Reactivate the original database (the one in playground that we used for the demo) and list its content. Both the dark and the flat should now be listed.

[Solution]