Deploying to gh-pages from @ carlylovas/species.shifts@42b2563 🚀

Author: carlylovas

articles/data_cleaning_functions.html

@@ -74,6 +74,8 @@ to the area between -60 and -80 degrees west and 20 degrees north.
 The data that is returned includes all that is shown above, and
 `sub_trip_id` as the unique trip identifier.
 
+------------------------------------------------------------------------
+
 ### Fisheries Observer
 
 `observer <- pull_observer(proj_path = my_path)`
@@ -87,7 +89,7 @@ The data that is returned includes all that is shown above, and
 \*Trip ID information removed from demo to maintain confidentiality.
 
 **How it works:** The fisheries observer information was received from
-the \*\*Northeast Fisheries Science Center\* as a Microsoft Excel
+the **Northeast Fisheries Science Center** as a Microsoft Excel
 Workbook. The data spans 1989 to 2024 and contains trip, haul and catch
 data from that time frame. The haul data is broken up into two separate
 sheets within the excel workbook. Catch data is stored within individual
@@ -107,6 +109,8 @@ capture the proper recorded coordinates to each trip. This list is then
 joined back to the combined catch-haul data set to ensure accurate
 coordinates based on trip type.
 
+------------------------------------------------------------------------
+
 ### NEFSC Bottom Trawl
 
 `nefsc <- pull_nefsc(proj_path = my_path)`
@@ -117,8 +121,8 @@ coordinates based on trip type.
 | 1.97003e+12 | 28 | thorny skate | 1970 | 3 | 0 | Spring | 41.58333 | -69.46667 | 3/12/70 8:45 | 0.900 |
 | 1.97003e+12 | 73 | atlantic cod | 1970 | 3 | 0 | Spring | 41.58333 | -69.46667 | 3/12/70 8:45 | 8.910 |
 
-**How it works:** The trawl data is received from the Northeast
-Fisheries Science Center as a .Rdata file pulled from an SQL database.
+**How it works:** The trawl data is received from the **Northeast
+Fisheries Science Center** as a .Rdata file pulled from an SQL database.
 `survdat` is extracting from the larger list of data and is the basis of
 the data used here. `survdat` is combined with a previously built
 species list to add the common names of survey species.
@@ -136,6 +140,8 @@ Strata and species not regularly sampled are then filtered out from the
 data, and the data subset to begin at 1970, when the trawl began to run
 consistently.
 
+------------------------------------------------------------------------
+
 ### GARFO Federal Permits
 
 `permits <- pull_permits(proj_path = my_path)`
@@ -146,7 +152,36 @@ consistently.
 | 1996 | 227830 | 127133 | GLOUCESTER | MA | lobster_1 | lobster | commercial | 42.61536 | -70.66246 | Massachusetts | New England |
 | 1996 | 227831 | 231461 | RYE | NH | lobster_1 | lobster | commercial | 43.01202 | -70.77194 | New Hampshire | New England |
 
-**How it works:**
+**How it works:** Federal permits data is received as a series of
+individual Excel spreadsheets, one for each year. This function is
+designed to read in multiple Excel files from a single source folder and
+combine them into one data frame.
+
+Permits are grouped by species, in that each species is represented by a
+single column and each permits type represented by a number, letter, or
+some combination of both. If an individual holds multiple permits
+types/endorsements, a single entry is made and those values are
+separated by a comma within a single cell. As such, this function breaks
+up each column into the respective number of permits types, and renamed
+with the target species name and permit type (ex: `black_sea_bass_1` and
+`black_sea_bass_2`). The all permit type columns are then pivoted, so
+that each permit application number has a corresponding permit type and
+value `1` for that permit.
+
+From there, the names of target species are cleaned and added as a
+separate column, for later species-level grouping. An additional column
+is added based on whether a permit type is categorized as `commercial`,
+`for-hire/charter` or `recreational`. Because we are interested in also
+accessing species-level trends, permits that cover multiple species
+(`multispecies` and `squid/mackerel/butterfish`) have been parsed out so
+that each specific permit type corresponds with it’s target species.
+
+Lastly, permits are geocoded using `tidygeocoder` to their reported
+principal port and grouped to their respective management council
+region. Roughly 1% of entries are lost due to misspelled principal
+ports.
+
+------------------------------------------------------------------------
 
 ### GARFO Dealer-reported landings
 
@@ -158,6 +193,15 @@ consistently.
 | 1996 | ADDISON | ME | lobster, american | 355359 | 355359 | 1178910 | NA | american lobster | 44.61882 | -67.74445 | Maine | New England |
 | 1996 | ADDISON | ME | scallop, sea | 9704 | 80837 | 66560 | NA | sea scallop | 44.61882 | -67.74445 | Maine | New England |
 
+**How it works:** Confidential landings data was received along with the
+vessel trip reports as an Excel spreadsheet. The first 9 lines contained
+metadata as provided by **GARFO**. As such, this function uses `readxl`
+to read in the file and requires a `skip` argument to skip the first few
+rows that contain metadata. `SwimmeR` is used to clean the species names
+and `tidygeocoder` to geocode the ports associated with landings.
+
+------------------------------------------------------------------------
+
 ### Marine Recreational Information Program
 
 #### Directed trips
@@ -170,6 +214,28 @@ consistently.
 | 2018 | 1 | 10 | 5 | 3 | 5 | striped bass | 169377.67 | 40817.095 | 24.1 | DE | Shore | Inland |
 | 2018 | 1 | 10 | 5 | 7 | 1 | striped bass | 44944.57 | 38431.208 | 85.5 | DE | Private & Rental | State Territorial Seas (Ocean\<=3 mi excluding Inland) |
 
+**How it works:** The function to pull in and clean the MRIP directed
+trips is arguably the most complex of the data cleaning functions. The
+data is hosted online and is downloadable as a ZIP file for individual
+years. This function pulls in the zipped files, unzips them and saves
+them locally in intermediate files, which are required for the trip
+estimation code. Once the data has been pulled in and saved to the
+intermediary files, the trip estimates are calculated using [Gary
+Nelson](https://carlylovas.github.io/species.shifts/articles/gary.nelson@state.ma.us)’s
+[MRIP directed trip estimate R
+template](https://www.fisheries.noaa.gov/recreational-fishing-data/recreational-fishing-data-downloads).
+This code estimates the number of directed trips by species, wave,
+domain, trip type and year.
+
+For this analyses, we used all waves, primary target trip type, all
+states along the east coast, and calculate estimates for Atlantic
+croaker, Atlantic mackerel, black sea bass, blueline tilefish, bluefish,
+gray triggerfish, king mackerel, Spanish mackerel, striped bass, summer
+flounder, scup, spiny dogfish, goosefish (monkfish) and tilefish (golden
+tilefish).
+
+------------------------------------------------------------------------
+
 #### Catch estimates
 
 `mrip_catch <- pull_mrip_catch()`
@@ -180,6 +246,14 @@ consistently.
 | Final | 2024 | Atlantic Coast | Mid Atlantic | Delaware | shark, dogfish, spiny | Squalus acanthias | 0 | 0.0 | 0 | 0.0 | 35430 | 6611 | 64249 | 35430 | 6611 | 64249 | 41.5 | spiny dogfish |
 | Final | 2024 | Atlantic Coast | Mid Atlantic | Delaware | bass, black sea | Centropristis striata | 105763 | 28.8 | 130995 | 28.1 | 1008441 | 290955 | 1725926 | 1114203 | 395720 | 1832686 | 32.9 | black sea bass |
 
+**How it works:** Catch estimates are publically available on the [ACCSP
+Data
+Warehouse](https://safis.accsp.org/accsp_prod/f?p=1490:1:1685966511435:::::)
+and downloaded as a CSV File. This function reads in the CSV and cleans
+the species names to be uniform with other data sets.
+
+------------------------------------------------------------------------
+
 ## Plotting
 
 Once the data has been pulled and saved to the local environment, each

articles/data_cleaning_functions.md

@@ -7,7 +7,7 @@ articles:
   plotting_functions: plotting_functions.html
   scup_demo: scup_demo.html
   summer_flounder_demo: summer_flounder_demo.html
-last_built: 2026-06-18T20:35Z
+last_built: 2026-06-22T17:44Z
 urls:
   reference: https://carlylovas.github.io/species.shifts/reference
   article: https://carlylovas.github.io/species.shifts/articles