Difference between revisions of "Poland Lab Wiki"

From Poland Lab Wiki
Jump to: navigation, search
(Added UAS Data Management Operating Procedure - Initial Draft)
(Database)
 
(114 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 
<strong>Welcome to the Poland Lab Wiki!</strong>
 
<strong>Welcome to the Poland Lab Wiki!</strong>
  
== [Protocols] ==
+
== General Lab Information ==
  
 +
[[Lab Procedures]]
  
 +
[[Reading List]]
  
== Standard Operating Procedures ==
+
[[Travel Instructions]]
===UAS Data Management Operating Procedure===
+
  
====Field Computer Operating Procedure====
+
== Projects==
  
1. Create 'flight' folder for the data collection run
+
[[NSF BREAD]]
  
a. Execute the python program create_flight_directory_v03.py from a command window.
+
== Genotyping Procedures ==
  
usage: create_flight_directory.py [-h] [-l LOG] [-f FLIGHT]
+
Lab protocols
  
optional arguments:
+
[[Media:GBS_NGS_Service_Order_Quebec.pdf|NGS Ordering]]
-h, --help            show this help message and exit
+
-l LOG, --log LOG    The full path to apm log file.
+
-f FLIGHT, --flight  The full path to the directory where     the flight folder is to be created.
+
  
Example Command Line Invocation MAC OS X: ./create_flight_directory_v03.py -l /Users/mlucas/Desktop/"2015-01-19 09-08-54.log" -f /Users/mlucas/Desktop/
+
[[GBS Data Management]]
  
Example Command Line Invocation Windows: create_flight_directory_v03.py -l C:\Users\mlucas\Desktop\"2015-01-19 09-08-54.log" -f C:\Users\mlucas\Desktop\
+
== High-Throughput Phenotyping Procedures ==
  
 +
[[UAS Phenotyping and Aerial Image Acquisition Procedures]]
  
Note that since the log file has a space in it, it needs to be enclosed in quotes. Alternatively, you could rename the log files to replace the space with a hyphen '-' to avoid having to use quotes.
+
[[UAS Phenotyping and Aerial Image Acquisition Procedure for DJI]]
  
Note also that all folder paths have to have '/' on the end to make sure that they reference the correct folder.
+
[[Media:UAV_Data_Upload_SOP_v1.3.pdf|UAS Data Management Procedures]]
  
Example Folder Name: uas_20150119_091005_020150119
+
[[UAS HTP Data Analysis]]
  
 +
[[Media:CAMERAS_README.txt|Cameras README]]
  
2. Store all data for a flight in the 'flight' folder:
+
== Research Methods ==
  
a. 1 operator log file e.g. DroneSettingsLog_2015-01-19_09-05-18.txt
+
[[Genomic Selection]]
  
b. 1 autopilot log file e.g. 2015-01-19-09-08-54.log
+
[[Meta-GWAS]]
  
c. 1 telemetry log file e.g. 2015-01-19-09-03-03.tlog
+
== Database ==
  
d. All image files e.g. *.CR2. N.B. There is no need to put these in a separate sub-folder. Just copy all images to the same level as the other log files.
+
[[Connecting to the Database]]
  
3. tar the 'flight' folder. (If on Windows, 7zip will do this.)
+
[[Media:Wheatgenetics_Quick_Start_Guide_ML_20171016.pdf|Wheatgenetics Database Quick Start Guide]]
  
a. filename e.g. uas_20150119_091005_020150119.tar
+
[[Media:DatabaseQueryFromR.pdf|Connecting to the Database with R]]
  
4. Compute MD5 checksum for tar file. If on Windows the Microsoft utility fciv.exe will do this.
+
[[Genotyping Database]]
  
5. Store checksum for the tar file in text file that contains the same 'flight ID' as the tarfile with '_md5.txt' appended.
+
[[Land-based High-Throughput Phenotyping Database]]
  
      Example using fciv utility:
+
[[UAV High-Throughput Phenotyping Database]]
  
fciv  uas_20150119_091005_020150119.tar > uas_20150119_091005_020150119_md5.txt
+
[[Manual Phenotyping Database]]
  
This will output the md5 checksum to a text file. Edit the text file to remove all text except for the line with the checksum:
+
[[Seed and Plant Database]]
  
427b87f05c72a0d7aea0cb2d8582aa2b uas_20150119_091005_020150119.tar
+
[[CIMMYT Wheat Breeding Database]]
  
 +
[[Media:CIMMYT_Automation_File_Interfaces.pdf|CIMMYT Database Import File Interface Specifications]]
  
6. Transmit data to server 'uas_staging' folder: /homes/jpoland/images/staging/uas_staging
+
== Scripts ==
  
a. Copy the tar file and the md5.txt file to the server together.
+
[[Tassel 5 GBS v2 Pipeline sample script for calling SNPs using reference genome]]
  
b. Ideally via sftp if not too large. Otherwise, store a copy on external hard drive and return hard drive when all runs for the experiment have been completed.
+
== Equipment ==
  
c. N.B. Keep a backup copy of data for each run on the field laptop!
+
The links below provide access to the user manuals for the Toshiba Stake Printer and the Zebra Label Printer.
+
  
====Server Operating Procedure====
+
If you encounter problems while using the printers please refer to the Troubleshooting section in the relevant user manual.
  
7. Navigate to the 'incoming' folder where new run data is staged.
+
[[Media:B-SX5T-R_Owners-Manual_EN.pdf|Toshiba Stake Printer User Manual]]
  
8. Compute checksum for the 'flight ' data tar file and verify that it matches the value in the checksum file.
+
[[Media:zt200series_userguide.pdf|Zebra Printer User Manual]]
  
9. Untar the 'flight' data folder.
+
== Guides ==
  
10. Post-process the flight data:
+
[[ How to Submit GBS Data to the NCBI Sequence Read Archive (SRA) ]]
a. Rename each image file to a unique file name per image
+
b. Generate a geo-referenced metadata file for the full set of images in the run (one row per image)
+
i. Include a checksum for each image in the metadata
+
ii. N.B. The program to create the metadata could load the database table directly without having to create an intermediate metadata file for import.
+
 
+
11. Perform a sanity check on geo-referenced image file to make sure that it does not contain any spurious data
+
a. N.B. The software for doing this is TBD. It needs to be able to read the position of every image and generate a visual plot of the flight path.
+
 
+
12. Import geo-referenced image metadata file into database 'uas_images' (new) table.
+
a. N.B. If database is loaded by geo-referencing program directly this step is not required.
+
 
+
13. Cleanup the run folder after verifying successful completion of post-processing steps
+
a. Delete the geo-referenced metadata file
+
 
+
14. Tar the 'flight' data file folder containing the renamed image files and original log files.
+
 
+
15. Compute checksum for image folder tar file.
+
 
+
16. Store a record for the run into database 'uas_run' (new) table that contains at least the flight ID, checksum and notes columns.
+
 
+
17. Move the  'flight' folder tar file containing the images and log files to production directory.
+
 
+
18. Backup the flight data to NAS.
+

Latest revision as of 20:04, 13 May 2020

Welcome to the Poland Lab Wiki!

General Lab Information

Lab Procedures

Reading List

Travel Instructions

Projects

NSF BREAD

Genotyping Procedures

Lab protocols

NGS Ordering

GBS Data Management

High-Throughput Phenotyping Procedures

UAS Phenotyping and Aerial Image Acquisition Procedures

UAS Phenotyping and Aerial Image Acquisition Procedure for DJI

UAS Data Management Procedures

UAS HTP Data Analysis

Cameras README

Research Methods

Genomic Selection

Meta-GWAS

Database

Connecting to the Database

Wheatgenetics Database Quick Start Guide

Connecting to the Database with R

Genotyping Database

Land-based High-Throughput Phenotyping Database

UAV High-Throughput Phenotyping Database

Manual Phenotyping Database

Seed and Plant Database

CIMMYT Wheat Breeding Database

CIMMYT Database Import File Interface Specifications

Scripts

Tassel 5 GBS v2 Pipeline sample script for calling SNPs using reference genome

Equipment

The links below provide access to the user manuals for the Toshiba Stake Printer and the Zebra Label Printer.

If you encounter problems while using the printers please refer to the Troubleshooting section in the relevant user manual.

Toshiba Stake Printer User Manual

Zebra Printer User Manual

Guides

How to Submit GBS Data to the NCBI Sequence Read Archive (SRA)