Azspec processed Hawk data has spectral spikes

[benj]

Azspec-processed level 1b hawk data does not appear to be correct - it contains large anomalous spikes in some bands, seemingly correlated to very low data values. Eg from Nigg Bay line 3:

Compared to Caligeo spectrum from same pixel:

Could be underflowing, but would expect a much higher value for the spikes (~65k) in this case. Might still be underflowing and then changed by maths in azspec though.

Note extended from ticket #106, but separate to Hawk noise issue

[benj]

Hawk spectrum from azspec-processed level 1b file over water (8px average)

[benj]

Hawk spectrum from caligeo-processed level 1b file over water (8px average), same place as azspec example

[benj]

Hawk level 1b spectra over land from azspec and Caligeo

[benj]

Azspec output over land (ie values largely not close to 0) is much closer to Caligeo output, but still not the same. Note below magnitude differences, exaggerated absorption feature at ~1600nm, slightly different shape of leftmost peak and lower starting low-end value (note may be due to spectral shifting, see #107).

[benj]

Hawk raw data over water. White line is single pixel, red line is 8px average

[benj]

Hawk level 1b spectrum processed by azspec over water - single pixel

[benj]

Checked relationship between raw data bad pixels and spikes in azspec-processed data. Of the two graphs below (top is raw data, white line being single pixel spectrum and red being 8-pixel average, bottom being azspec-processed level 1b single pixel for the same pixel), it can be seen that the azspec processed data has spikes that do not correspond to bad pixels even within the 8-pixel range (red line).

Raw data:

Azspec-processed:

[mggr]

This problem has been passed to Bill for investigation (first formal contact 6/March), investigation begun on 10th. Follow up contact on 11th and 12th.

[mggr]

Spikes vs raw, vertical profile

[mggr]

Bill responded yesterday with some initial thoughts, particularly on bad pixels. I did a little more investigation on the exact locations of some of these spikes.

The following image has the following important points to note:

• very bright speckles in the azspec image are the hawk "spikes"
  • note these are not exactly in lines, as with the obvious bad pixels
• the two plots on the right are vertical profiles (image-Y coord is the plot x axis) for a single line where speckles manifest, top plot is the raw data and bottom is the azspec calibrated output (spikes are obvious)
• note that the spikes correspond with raw values that are below the measured dark value (actually align with the pixel before-in-raw/after-in-processing the low value, but this may be something to do with the processing algorithm or the way I measured the Y offset between raw and azspec data)

Minor other comments for reproducing:

• azspec inserts blank padding lines for missing frames, hence there is an offset of approximately 18 in Y coordinates (measured by hand in the region of interest by finding a small bright pixel present in raw and azspec and computing the different in coords)
• band 166, line 30 in azspec, line 290 (320-30, due to the flip) was the one examined
• dark value was measured by hand (couldn't get azspec to report it with -da). Used all the dark frames but the first one, which was obviously non-dark and averaged them to get 5519.77777 (~5520).

  actual values averaged:
  (5520+5536+5531+5516+5509+5512+5509+5519+5520+5513+5539+5522+5523+5509+5521+5522+5520+5519+5533+5517+5520+5528+5529+5521+5512+5518+5522+5510+5519+5512+5504+5530+5520+5514+5523+5520)/36
  

An interpretation is that the bright spikes occur when a pixel goes below the dark value.

[mggr]

Bill also found the underflow bug above and has supplied an updated azspec to test. The bad pixel issue is still outstanding (note other ticket for this)

[mggr]

Visual examination indicates spikes are fixed.

(left to right: old-azspec, caligeo, new-azspec)

[benj]

Old and new azspec processed data over water

[benj]

New azspec and Caligeo processed data over water

[benj]

New azspec and caligeo over land

[benj]

New Azspec from Bill seems to have fixed the spike problem - Azspec over water no longer shows spikes (see below), old and new azspec match exactly except for the spikes over both land and water. Marking bug fixed.

[mggr]

Work order number was 200803-1

[mggr]

Work orders renumbered to make it obvious when one is missing. This work order is now wo2008001.