Predictive drivers: Data collection and mining
Tang
July 24, 2017
Raw data file example
The data was downloaded from https://www.nodc.noaa.gov/OC5/WOD/pr_wod.html using the WOD select tool to grab data (latitude 18.5 - 32.5, longitude -97.8 - -80.0) using all defaults (selecting all variables from all data sources/years for the entire GOM). Data were downloaded in standard .csv format. We chose observed level data, and applied the most recently published XBT correction (Cheng et al. 2014).
## [1] "#--------------------------------------------------------------------------------,"
## [2] "CAST ,, 13332338,WOD Unique Cast Number,WOD code,"
## [3] "NODC Cruise ID ,,99-0 ,,,"
## [4] "Originators Station ID ,, ,,,alpha,"
## [5] "Originators Cruise ID ,,GLNE ,,,"## [1] "VARIABLES ,Depth ,F,O,Temperatur ,F,O,Salinity ,F,O,,"
## [2] "UNITS ,m , , ,degrees C ,, , ,PSS ,, , ,,"
## [3] "Prof-Flag , ,0, , ,0, , ,0, ,,"
## [4] " 1, 0.00,0, , 24.167,0, , 36.520,0, ,"
## [5] " 2, 33.50,0, , 24.167,0, , 36.570,0, ,"
## [6] " 3, 42.70,0, , 23.889,0, , 36.600,0, ,"
## [7] " 4, 48.80,0, , 23.222,0, , 36.610,0, ,"And, we have four huge tables, each containing more than 20,000 tables.
Step 1, extract date and location information
We want to create a table in the following format:
To do that, based on the structure of available data, we first extract the location, date and cast data using str_match. We want cast, because this is the unique ID we can use later to join the environmental data.
Using this process we obtain data in the following format
Step 2, extract all tables with environmental data
Insepecting the raw data tables, we found that the rows we need always started with VARIABLES and ended with END OF VARIABLE SECTION. Thus, we use these two strings and str_detect to find the beginning and ending row index for each table we want to read. Then we either used loops or tapply to extract all tables. Finally, we also assigned the cast ID for each table.
We extracted tables from CTD, XBT, and PFL casts using parallel processing.
For OSD specifically, we used a different process due to many rows missing environmental data.
We obtained one table for each cast with all oceanographic measurements and cast IDs:
Then, we combined the four tables into one big table with all oceanographic measurements from all different methods.
Step 3, locate ocean depth for each cast id and merge with result from step 1
We needed oceanographic data from different depth levels. For example, we needed water column averages for temperature and salinity, and dissolved oxygen from bottom water depth. We find the water depth for each cast ID using get.depth from marmap, and save the ocean bottom depth in the file from step 1.
Then, we can obtain a table with all cast ID and bottom ocean depth.
Step 4, merge data from step 2 with data from step 3
Now, we combine the location, time and bottom depth data with the environment table by cast ID using merge with all = TURE. Finally, we arrived at one big table with all information
## X.1 X Cast lat lon year month day temp sal Depth
## 1 1 1 812002 31.392 -80.887 1981 7 28 27.50000 36.27000 10
## 2 2 2 812026 31.687 -80.345 1981 7 29 25.71250 36.33250 24
## 3 3 74 869140 28.217 -92.067 1987 7 17 23.69900 35.58560 69
## 4 4 75 873218 28.033 -92.033 1988 7 21 25.48167 36.27700 50
## 5 5 76 873232 28.717 -92.100 1988 7 21 27.57483 34.74050 31
## 6 6 77 873233 28.850 -92.100 1988 7 21 29.15600 31.21200 1
## 7 7 78 876071 28.738 -94.985 1991 6 15 27.38720 35.08960 19
## 8 8 79 876072 28.418 -95.002 1991 6 15 27.20667 35.39100 35
## 9 9 80 876073 28.258 -94.993 1991 6 15 27.25600 35.25100 28
## 10 10 81 876074 28.068 -94.990 1991 6 15 25.91686 35.38529 58
## Oxygen
## 1 5.84
## 2 6.21
## 3 3.05
## 4 5.03
## 5 4.53
## 6 4.88
## 7 4.25
## 8 4.28
## 9 4.60
## 10 4.45
Step 5, data manipulation
We needed to assign each cast id one unique row of oceanographic data, including water column averaged temperature and salinity, surface chlorophyl, and bottom water DO.
The final data table looked like this:
And contained 10 casts.
Step 6, create interactive map
Now, we create an interactive map with layers : oil spill incidents, temprature, salinity, DO, and Chol for future prediction of hot spots.
## OGR data source with driver: ESRI Shapefile
## Source: "cb_2016_us_state_20m/cb_2016_us_state_20m.shp", layer: "cb_2016_us_state_20m"
## with 52 features
## It has 9 fields
## Integer64 fields read as doubles: ALAND AWATER