Opened 9 years ago

Closed 6 years ago

#589 closed flight processing (fixed)

EU15/18, flight day 174, Toulouse

Reported by: lah Owned by:
Priority: immediate Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by dac)

Data location: /users/rsg/arsf/arsf_data/2015/flight_data/france/EUFAR15_18-2015_174_Toulouse

Data arrived from ARSF via SATA disk on 09/07/2015.

Scientific objective: Hyperspectral measurements over agro-forestry areas for landscape assessment of agricultural health, ecophysiology, and satellite product validation.

Priority: Unknown

PI: Jean-Louis Roujean
EUFAR Project ID: AHSPECT

Sensors:

  • Fenix (requested)
  • Leica LIDAR (not requested)
  • Leica FW LIDAR (not requested but flown anyway)
  • Owl (not requested but flown anyway)

Change History (54)

comment:1 Changed 9 years ago by stgo

  • Description modified (diff)

comment:2 Changed 9 years ago by gej

Navigation Processing complete

Basestation:
Lat: 43 38 13.01189
Long: 1 21 58.49814
Ell. Height: 199.663

Position Seperation: Slight spike to -0.06 towards end of flight line, rest below -0.04
Position Accuracy: Lat and long all within +-0.035 Height within +-0.05

comment:3 Changed 9 years ago by lah

Unpacking
Removed corrupt webcam images:

  • 150623_095016_000.jpg
  • 150623_123907_000.jpg
  • 150623_124117_000.jpg

comment:4 Changed 9 years ago by asm

Lidar Processing
Started.

comment:5 Changed 9 years ago by gej

Hyperspectral Processing
Started.

comment:6 Changed 9 years ago by dac

Sent notification email to PI with links to KML

comment:7 Changed 9 years ago by asm

Lidar Processing

Found pitch and roll consistent with all lines

Roll -0.0010109
Pitch -0.0022489

comment:8 Changed 9 years ago by asm

Lidar Processing

Classification started.

comment:9 Changed 9 years ago by asm

Lidar Processing

Classification for discrete lidar finished.

Last edited 9 years ago by asm (previous) (diff)

comment:10 Changed 9 years ago by asm

Lidar Processing

Classification for lidar full wave form started.

comment:11 Changed 9 years ago by asm

Lidar Processing
Two flightlines are bigger than 50gbs. Process is killed due to that, stgo is splitting the lines on the spare machine.

comment:12 Changed 9 years ago by gej

Hyperspectral Processing
found SCT values

Long flightlines, some had a varying SCT value, have currently gone for best throughout line, but if it looks like they may be too incorrect will change

Flightline FENIX
1 23.45
2 3.2
3 2.90
4 2.25
5 3.675
6 2.75
7 2.5
8 1.45
Version 0, edited 9 years ago by gej (next)

comment:13 Changed 9 years ago by asm

Lidar Processing
Classification for full wave form finished. Elevation difference measured, delivery created including Readme file. Ready for Delivery Check'.

Last edited 9 years ago by asm (previous) (diff)

comment:14 Changed 9 years ago by gej

Lidar Delivery Check

Read Me
Need to add N/S to UTM projection,

Data qaulity remarks needs reworkding. instead of:
"This might have as a result that the elevation difference on overlapping areas could be not precise",
try:
"This may have resulted in the elevation difference on overlapping areas being less precise"

Navigation
There are only 6 trj files, but 8 flightlines. Not sure if this is due to the flightlines being split or not, if it is should maybe make a comment about it somewhere.

Screenshots
Don't think anything can be done, but maybe make a comment in readme that the screenshots aren't of the best quality due to the length of the flightlines.

comment:15 Changed 9 years ago by dac

  • Description modified (diff)

comment:16 Changed 9 years ago by asm

Lidar Processing

All comments has been changed to trj files and Readme.

Please, ready for re-check.

comment:17 Changed 9 years ago by gej

Lidar Delivery Check

All looks good now that changes have been made.
All trj files are now there and comments made in readme on screenshots, and previous comment reworded.

Will be delivered with Fenix Hyperspectral data

comment:18 Changed 8 years ago by gej

Lidar delivered to Jean-Louis Roujean via FTP on 12 October 2015

comment:19 Changed 8 years ago by gej

Fenix Processing
Started creating mapped using files with new calibration

comment:20 Changed 8 years ago by dac

Fenix

Hyperspectral lines are very large producing mapped files 0.5 - 1.5 TB. Emailed PI to check preferred method for splitting data.

comment:21 Changed 8 years ago by dac

Fenix

PI happy to receive data split by scanline.

comment:22 Changed 8 years ago by dac

Fenix

Started testing splitting by scanline by adding cal_args_extra = -lines 0 100 to config file.

comment:23 Changed 8 years ago by gej

Fenix Processing

Lines have now been split, using cal_args_extra. Split line functionality has been added to generate_apl_config.py for future projects.

There was a problem with this flight where the SCT values varied through the flight by about -0.4 for every 10000 scan lines.
This has been fixed by increasing the fps by 0.08 so I am now finding the new scts

comment:24 Changed 8 years ago by gej

Fenix Processing

Found new SCT values, SCT values are consistent through sections so have only put full line values.

FlightlineFENIX
127.5
24.3
33.34
42.58
53.92
63.11
74.28
81.85

Number of sections per line:

FlightlineNumber Sections
111
23
34
49
51
61
75
81
total35

Files now being mapped with all bands.

comment:25 Changed 8 years ago by gej

Fenix Processing

Have split line 6 as final filesize was at 200GB.
Has been split into 3 sections. SCT hasn't changed.

comment:26 Changed 8 years ago by gej

Fenix processing

Mapped files processed, creating delivery.

comment:27 Changed 8 years ago by stgo

Fenix processing
Have taken over from gej for this week, some files have been included in the delivery creation which resulted in bad naming of all files. Irritating but fixable, if I can't identify the correct ones (should be easy, they will be the largest files) I will reprocess for simplicity's sake.

comment:28 Changed 8 years ago by stgo

Fenix processing

Have resubmitted to the grid, will recreate delivery once complete.

comment:29 Changed 8 years ago by stgo

All lines still had their dark frames included, have updated end lines to remove these. Also fixed the sct value for line 1 and will be recreating delivery this afternoon.

comment:30 Changed 8 years ago by stgo

Fenix processing

Created readme and placed in the delivery folder. A pre release version of the data has already been provided to the PI via ftp however the size of the delivery means it is not very feasible for them to download it on their own connection. Mapped files are already zipped.

Needs final DC before delivery.

comment:31 Changed 8 years ago by dac

Fenix Delivery Check

  • Created ASCII view vectors and converted to Windows line endings.
  • Added header to the table at the top of page 3
  • Compared spectra to Py6S simulations - look good

Ready to deliver.

Need to remove old deliveries from processing/deliveries.

comment:32 Changed 8 years ago by dac

Archiving

Have started uploading delivered LiDAR data to NEODC - large delivery so will take a while.

comment:33 Changed 8 years ago by stgo

Fenix delivery

Fenix data sent on hard drive 10/12/2015

comment:34 Changed 8 years ago by dac

Archiving

Started archiving

comment:35 Changed 8 years ago by dac

Archiving

Finished upload to NEODC - Notified Wendy.

comment:36 Changed 8 years ago by dac

Delivery

Notification from PI: Received hard drive with Fenix data (22/12/2015)

comment:37 Changed 8 years ago by dac

LiDAR'

Started reprocessing with full range of scan angles.

comment:38 Changed 8 years ago by lah

OWL

found sct for sample flight line 2 = 3.93

Created delivery, ready for checking.

comment:39 Changed 8 years ago by asm

Owl Delivery Check

-check_apl_cmd only created the masked files but I tested the instructions written on readme and everthing looks good.
-proj_tidy do not recognise the owl delivery folder and list everything as unrecognised files. Looks fine however.
-There is only one flightline so both screenshots were the same. Removed mosaic.
-Tried to run py6S with the igm created from apl commands for a given pixel x,y (did not work for ndvi pixel finder). Did not work, error: Unexpected radiance units. May be worth implementing this on the future.
-Spectra looks OK for mapped file on Tuiview.
-As dac pointed out on other project: XML metadata files list Carl as the pilot and Tom as the instrument operator (different to logsheet). This is the same as the Fenix data so not worth blocking the project for but will need fixing in templates.
No major problems found. Owl should be ready to deliver.

comment:41 Changed 8 years ago by dac

  • Resolution set to fixed
  • Status changed from new to closed

Archiving

Raw data also available from NEODC (http://browse.ceda.ac.uk/browse/neodc/arsf/raw_data/2015/EUFAR15_18-2015_174_Toulouse_raw).

Leaving ticket open for Owl processing (outside EUFAR)

comment:42 Changed 8 years ago by dac

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:43 Changed 8 years ago by lah

Owl

Uploaded sample delivery to NEODC.

comment:44 Changed 7 years ago by lah

Owl Processing

Found scts with a combination of wobble minimisation and alignment to fenix. There isn't much overlap between flight lines in this flight. The same sct is good for each split of an entire line, though there is some increased misalignment with the fenix, which does appear to wobble in these regions.

FlightlineOWL
12.67
23.89
32.44
43.14
52.60
62.96
72.51

Now mapping full band files.

comment:45 Changed 7 years ago by lah

Owl Processing

Created delivery. Scripts deleted all level 1b split lines so recreating. Some of these appear to have been renamed as the full files but the number of lines between these and the mask files are inconsistent so reprocessing level 1 files.

comment:46 Changed 7 years ago by lah

Owl Delivery

Created readme from hyp_config. Manually edited owl sections and removed under/overflow table as there were none. Added in missing lines (files) to delivery content table.

Manually added Owl details to logsheet (FPS & IT).

comment:47 Changed 7 years ago by lah

Owl Delivery

Copied in new level 1b files and renamed. Ready to delivery check.

comment:48 Changed 7 years ago by asm

Owl Delivery Check

Started.
*No xml files for flightline 4.
*Screenshots are currently being generated and will recheck later.
*Project tidy does not like owl names but everything seems fine.
*Everything else looks fine, including checks for apl comands

Once xml for flightline 4 has been created and the screenshorts sorted out, should be ready to deliver.

comment:49 Changed 7 years ago by lah

Owl

Recreated xml files for line 4. These were missing because line 4 had no black body files so used line 5 calibration file. Manually added in software info.

New screenshots have finished. Have renamed these and removed old ones.

comment:50 Changed 7 years ago by asm

Owl Delivery Check

Everything looks fine now. Have removed the tmp_cmd_check and will zip mapped files. Once done, project will be ready to be delivered.

comment:51 Changed 7 years ago by asm

Owl Delivery Check

Mapped files have been zipped. Checked logfiles and everything looks correct. Will mark as Ready to be delivered.

comment:52 Changed 7 years ago by asm

OWL Delivered

By postal mail including a hard drive. Dispatched on 2nd December 2016 to PI. A notification email has been sent as well.

comment:53 Changed 7 years ago by lah

Sent hard disk to Philippe Crebassol with Owl data (and 274) on 24/01/17.

comment:54 Changed 6 years ago by dac

  • Resolution set to fixed
  • Status changed from reopened to closed

Archiving

Owl data available from CEDA. Closing ticket.

Note: See TracTickets for help on using tickets.