View source for UAS Data Management Procedures



==Field Computer Operating Procedure==

1.	Create 'flight' folder for the data collection run 

a.	Execute the python program create_flight_directory_v03.py from a command window.

usage: create_flight_directory.py [-h] [-l LOG] [-f FLIGHT]

optional arguments:
-h, --help            show this help message and exit
-l LOG, --log LOG     The full path to apm log file.
-f FLIGHT, --flight   The full path to the directory where 				    the flight folder is to be created.

Example Command Line Invocation MAC OS X: ./create_flight_directory_v03.py -l /Users/mlucas/Desktop/"2015-01-19 09-08-54.log" -f /Users/mlucas/Desktop/

Example Command Line Invocation Windows: create_flight_directory_v03.py -l C:\Users\mlucas\Desktop\"2015-01-19 09-08-54.log" -f C:\Users\mlucas\Desktop\


Note that since the log file has a space in it, it needs to be enclosed in quotes. Alternatively, you could rename the log files to replace the space with a hyphen '-' to avoid having to use quotes.

Note also that all folder paths have to have '/' on the end to make sure that they reference the correct folder.

Example Folder Name: uas_20150119_091005_020150119


2.	Store all data for a flight in the 'flight' folder:

a.	1 operator log file e.g. DroneSettingsLog_2015-01-19_09-05-18.txt

b.	1 autopilot log file e.g. 2015-01-19-09-08-54.log

c.	1 telemetry log file e.g. 2015-01-19-09-03-03.tlog

d.	All image files e.g. *.CR2. N.B. There is no need to put these in a separate sub-folder. Just copy all images to the same level as the other log files. 


3.	tar the 'flight' folder. (If on Windows, 7zip will do this.)

a.	filename e.g. uas_20150119_091005_020150119.tar


4.	Compute MD5 checksum for tar file. If on Windows the Microsoft utility fciv.exe will do this. 


5.	Store checksum for the tar file in text file that contains the same 'flight ID' as the tarfile with '_md5.txt' appended.

       Example using fciv utility:

fciv  uas_20150119_091005_020150119.tar > uas_20150119_091005_020150119_md5.txt 

This will output the md5 checksum to a text file. Edit the text file to remove all text except for the line with the checksum:

427b87f05c72a0d7aea0cb2d8582aa2b uas_20150119_091005_020150119.tar


6.	Transmit data to server 'uas_staging' folder: /homes/jpoland/images/staging/uas_staging

a.	Copy the tar file and the md5.txt file to the server together.

b.	Ideally via sftp if not too large. Otherwise, store a copy on external hard drive and return hard drive when all runs for the experiment have been completed.

c.	N.B. Keep a backup copy of data for each run on the field laptop!
 

==Server Operating Procedure==


7.	Navigate to the 'incoming' folder where new run data is staged.

8.	Compute checksum for the 'flight ' data tar file and verify that it matches the value in the checksum file.

9.	Untar the 'flight' data folder.

10.	Post-process the flight data:
a.	Rename each image file to a unique file name per image
b.	Generate a geo-referenced metadata file for the full set of images in the run (one row per image)
i.	Include a checksum for each image in the metadata
ii.	N.B. The program to create the metadata could load the database table directly without having to create an intermediate metadata file for import.

11.	Perform a sanity check on geo-referenced image file to make sure that it does not contain any spurious data 
a.	N.B. The software for doing this is TBD. It needs to be able to read the position of every image and generate a visual plot of the flight path.

12.	Import geo-referenced image metadata file into database 'uas_images' (new) table.
a.	N.B. If database is loaded by geo-referencing program directly this step is not required.

13.	 Cleanup the run folder after verifying successful completion of post-processing steps
a.	Delete the geo-referenced metadata file

14.	Tar the 'flight' data file folder containing the renamed image files and original log files.

15.	Compute checksum for image folder tar file.

16.	 Store a record for the run into database 'uas_run' (new) table that contains at least the flight ID, checksum and notes columns. 

17.	Move the  'flight' folder tar file containing the images and log files to production directory.

18.	Backup the flight data to NAS.