Version 294 (modified by asm, 9 years ago) (diff)

--

File structure and naming

This pages denotes the current project structure, and naming conventions.
The structure for 2011 is located here.
The structure for previous years is located here.

A single site/logical group of flight lines/sortie should be contained in a single project directory. If there are multiple sites, significantly different times of overflight or large time gaps, each should have its own directory.

The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.

There should be:

  • No spaces, brackets or other Unix-upsetting characters in filenames
  • Execute permission removed from anything non-executable

The structure under the project directory should look like:

Top Level Dirs 2nd Level Dirs 3rd Level Dirs 4th Level Dirs 5th Level Dirs 6th Level Dirs Example File Formats Comments
admin Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc)
GB08_02-2009_078a-Delamere.(doc|txt|pdf) Logsheet
GB08_02-2009_078a-Delamere.fpd Flight planning document
delivery_readmes Read_Me-20100809.(aux|log|tex) Intermediate files used to create the ReadMe(s)
(cam|hyp|lidar)_genreadme.cfg Configuration files generated by generate_readme_config.py
calibration Typically a symlink to the appropriate calibration directory
camera Camera-type sensors
rcd RCD digital camera
090803-100306ImageEvents1-processed.csv Processed events file
logs
camera_delivery.log
090803-100306ImageEvents(1|2).csv
090803-145404PhotoId(1|2).csv
090803-100306SensorLog.csv
090803-145404SensorStat(1|2).csv
raw_images 16093417100001G(3|4).raw
webcam 100715_124101_000.jpg Webcam image
20100715_105036_evt.csv Webcam event file
hyperspectral
eagle Raw Eagle data files
dataCube.raw
VNIR078a-09-1.hdr ENVI compatible header
VNIR078a-09-1.log Log files
VNIR078a-09-1.nav Navigation synchronisation data
VNIR078a-09-1.raw Uncalibrated image data in .bil format
hawk Raw Hawk data files
dataCube.raw
SWIR078a-09-1.hdr ENVI compatible header
SWIR078a-09-1.log Log files
SWIR078a-09-1.nav Navigation synchronisation data
SWIR078a-09-1.raw Uncalibrated image data in .bil format
orignal_headers (optional) SWIR078a-09-1.hdr-orig If the header files have been altered for any reason, the originals should be in here
lidar als50
aux_files Reg file used, any logs produced, intensity mosaics, boresight values
lidar_2009078a.mos ENVI mosaic of lidar lines intensity
lidar_2009078a.mos.hdr ENVI mosaic header
2009078a.reg Contains ALSPP settings
lidar_2009078a.tif Lidar intensity image
line01_2009078a.tif Individual flightline intensity image
logs Contains the flight/ops log files
100411_084840_FlightLineLog.csv
20091123212501_LogData.mdb
091005_LogFile.txt
090812_DLLogFile.txt
20091123212501_LogDataSysDesc.xml
RawLaser Contains the raw laser data
090601_111411 Contains raw laser for one contiguous recording
LDR090601_111411_0000000001.scn Raw LIDAR scan files
RawWFD Raw waveform data
100706_094620
WFD100706_094620_0000000001A.LWV
logfiles EU11_06_2011-098_eagle_-3.o2810538 Logs from hyperspectral script runs on gridnode (one log per flightline)
posatt Position/attitude data
basestation Basestation data
ppp_result Contain the output from the PPP processing of basestation
oxfr154g30.09n
oxfr154g30.09o
oxfr154g30.epp
oxfr154g30.gpb POSGNSS friendly file format
oxfr154g30.sta Text file with info about basestation
ipas_honeywell Data from ipas using the honeywell imu
*.gpj IPAS Pro Project File
*.itp IPAS TC Project File
extract
*.epp
*.EVT
*_extractLog.txt
*.fbv
*.rbv
*.gpb Grafnav friendly file format
*.GPS
*.IMU
*.RNV
*.RTG
*_RTMsg.txt
*.sta
*.SUP
*.tm
gps
*.cfg Grafnav project file
*.fml
*.fss
*.fsl PPP log file
*.fwd
*.his
*.lat Grafnav final output which is used by IPAS (text version)
*.lat.bin Grafnav final output which is used by IPAS
*.rev
*.rml
*.rsl PPP log file
*.rss
proc Processed ipas nav data
20090319_142319.sol Final output file
sbet_20090319_from_sol.out Applanix sbet created from sol file
20090319_142319.ers
20090319_142319.sup
2009078a_procLog.txt
20090319_142319.iqc IPAS TC quality control file
raw 20090319_111251.001 Raw GPS/IMU data
ipas20 Data from ipas using the old applanix imu
extract
gps
proc Processed ipas nav data
raw 20120323_103627.dat.000 Raw GPS/IMU data
processing
als50
las Contains the finalised processed LAS files
LDR091026_060148_1.LAS LAS file
pplog_20100226_122027.txt Log output from ALSPP
stats.txt Info from ALSPP about the LAS files
las-classified Contains the classified LAS files for delivery
LDR091026_060148_1.LAS LAS file
delivery Contains copies of data delivered to end users
EUFAR11_06-098-hyperspectral-20110621 Delivery structures described below Delivery folder
hyperspectral 0532010.cfg Config file
dem
flightlines
georeferencing
igm
mapped e078a013b_n_sct-0.03_p_gpt0.0.tif Geo-rectified images
level 1b
line information
navigation
interpolated
post_processed
real_time
rcd
proc_images 16093417100001G(3|4).tif
thumbnails 16093417100001G(3|4).jpg
tabi Raw TABI data

The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:

  • s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
  • ddd is the Julian day of the data acquisition
  • f distinguishes between multiple flights on the same day (“a” being the first flight)
  • nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)

Delivered data

As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support). The lev1/ and lev3/ directories will typically contain the latest processed version.

Delivered data is stored processing/delivery/PROJECTCODE-JDAY-SENSOR-YYYYMMDD/...., where YYYYMMDD is the date the delivery was created, PROJECTCODE is the project code, JDAY is the day of year of the flight (+ optional sortie ID) and SENSOR is {camera,hyperspectral,lidar}. The directory structure of a delivery looks like:

Hyperspectral deliveries

Top level Dir 2nd level Dir Example Files Comments
Read_Me-20100202.pdf Instructions for use and description of dataset, plus any problems encountered.
dem A dem to help the user with processing to lev3. Should be ASTER.
GB08_19-2010_254b-ASTER.dem ASTER binary dem
GB08_19-2010_254b-ASTER.hdr Header file for DEM
doc data quality report and other documentation
hyperspectral_data_quality_report2012.pdf data quality report
ISO19115-stylesheet.xsl XSLT style sheet for the XML files in line_information and project_information directories
ISO19115-css.css CSS style sheet for the XML files in line_information and project_information directories
flightlines
fodis (only for Eagle deliveries and not 2013) Data for the FODIS instrument
e078a021b_FODIS.bil Do not include in 2013 projects
e078a021b_FODIS.bil.hdr Do not include in 2013 projects
level1b Level 1 image data
(e|h)078a021b.bil ENVI bil file of level 1 data
(e|h)078a021b.bil.hdr
(e|h)078a021b_mask.bil A bil mask, generated for each project
(e|h)078a021b_mask.bil.hdr
(h|f)078a021b_mask-badpixelmethod.bil A bil mask, generated during the yearly calibration (hawk and fenix only)
(h|f)078a021b_mask-badpixelmethod.bil.hdr
line_information
(e|h)221021b.xml metadata for each line
ISO19115-stylesheet.xsl XSLT style sheet for the XML files in line_information and project_information directories
ISO19115-css.css CSS style sheet for the XML files in line_information and project_information directories
mapped
e078a023b_mapped_(utm*|osng).bil Mapped level 3 bil files (these have had the mask applied)
e078a023b_mapped_(utm*|osng).bil.hdr
e078a023b_mapped_(utm*|osng)_bil.zip Zipped mapped files (this is how they are delivered)
unzipped_filesize.csv Details of the filesizes before zipping
navigation Navigation data for each line
e078a021b_nav_post_processed.bil Envi bil file with navigation data
e078a021b_nav_post_processed.bil.hdr Header file for navigation
e078a021b_nav_post_processed_quality.bil Bil file indicating quality of navigation file
e078a021b_nav_post_processed_quality.bil.hdr Header file for quality file
logsheet FW10_01-2010_187-Dennys_wood.pdf Copy of pdf version of logsheet
misc (optional) Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc)
osgb02
sphsep15lx.grd
project_information
BB11_01-2011_244a-project.xml Meta data for the project as a whole
ISO19115-stylesheet.xsl XSLT style sheet for the XML files in line_information and project_information directories
ISO19115-css.css CSS style sheet for the XML files in line_information and project_information directories
screenshots Images of the level 3 output we created when testing the data
(e|h)078a013b.jpg Individual line screenshot
(eagle|hawk)_mosaic.jpg Screenshot of all level 3 data
(eagle|hawk_mosaic_with_vectors.jpg Screenshot of all level 3 data with overlaid vectors (UK only)
sensor_FOV_vectors
(eagle|hawk)_fov_fullccd_vectors.bil
(eagle|hawk)_fov_fullccd_vectors.hdr
(eagle|hawk)_viewvector_list.txt

The image filenames in the lev1 directory are structured as sdddfnnLL.hdf, where:

  • s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
  • ddd is the Julian day of the data acquisition
  • f distinguishes between multiple flights on the same day ("a" being the first flight)
  • nn is the flight line number
  • LL is the processing level
    • 1b = ungeocorrected data
    • 3a = geocorrected without a DEM
    • 3b = geocorrected with a DEM

Lidar deliveries

Dirs Sub Dir Level 1 Sub Dir Level 2 Example Files Comments
Read_Me-20100202.pdf Instructions for use and description of dataset, plus any problems encountered
bin Latest versions of the pt_cloud_filter for linux and windows
dem DEMs created from the LiDAR
GB08_19-2010_254b-lidar_ASTER-wgs84_utm.dem LiDAR dem patched with ASTER data (do not use NEXTMAP data - license restrictions). DEM is in same co-ordinate system as processed LAS files.
GB08_19-2010_254b-lidar_ASTER-wgs84_utm.hdr
GB08_19-2010_254b-lidar_ASTER-wgs84_latlong.dem Above LiDAR dem converted to wgs84_latlong (compatible with apl)
GB08_19-2010_254b-lidar_ASTER-wgs84_latlong.hdr
README_DEM_MAY_NOT_BE_ACCURATE.txt Warning to end user that there are potential flaws with the provided DEMs
doc lidar_quality_report_2011.pdf Lidar quality report
full_waveform.pdf FW Lidar Data review (full waveform deliveries only)
flightlines Classified lidar point clouds
(IF DISCRETE)
ascii LDR-EU11_03-2011-102-10.txt ASCII point clouds
(las1.0|las1.2) LDR-EU11_03-2011-102-10.LAS LAS1.0 point clouds
(IF FULL WAVEFORM PRESENT)
discrete_laser
ascii LDR-EU11_03-2011-102-10.txt ASCII point clouds
(las1.0|las1.2) LDR-EU11_03-2011-102-10.LAS LAS1.0 point clouds
fw_laser
las1.3 LDR-FW-GB13_10-2014-156-01.LAS LAS1.3 point clouds
fw_extractions (FW only) Full waveform area extractions
FW10_02-2010_101_extractions.jpg
FW10_02-2010_101_extractions.txt
area1
waveform_42437_649348_only.txt ?
logsheet GB08_02-2009_078a-Delamere.pdf Flight logsheet, typed up or created by generate_logsheet.py
navigation
FW10_02-2010_101.sol
trj_files (FW only)
FW10_02-2010-101-01.trj
screenshots Images of the dem and intensity mosaic
GB08_19-2010_254b-intensity.jpg
GB08_19-2010_254b-intensity_vectors.jpg UK only
GB08_19-2010_254b-dem.jpg
LDR-GB12_05-2014-075-22.jpg Screenshot for each flightline in the flight.

Photographic deliveries

Dirs Example Files Comments
GB08_02-2009_078a.kml Google earth file containing eagle/hawk/photo positions
Read_Me-20100202.pdf Instructions for use and description of dataset, plus any problems encountered
doc Useful documents
camera_data_quality_report2012.pdf Camera data quality report
RCD105_SN21_Calibration_Certificate.pdf Camera Calibration Certificate
eventfile RCD105-GB08_02-2009078a-ImageEvents.csv CSV file containing pos/att info per photograph event (may be missing if camera crashed)
photographs RCD105-GB08_02-2009078a-00001.tif Tagged tif files of each photograph in area of interest
thumbnails RCD105-GB08_02-2009078a-00001.jpg JPEG thumbnails of each photograph in area of interest