Annual Commercial Catch and Effort Data Load Process
Commercial catch and effort data are loaded into the Data Warehouse by the Data Team twice a year. The preliminary load of the previous year’s data occurs from February through April. These data are considered preliminary due to known late reporting issues, but are still a good measure of landings. During the final load of the previous year’s data, from August through September, updates are made to previously submitted data sets and late reports are added. Once this final load process is complete, the previous year’s data are a robust and complete measure of landings.
A. Identify Data Sets & Consolidate Participant Files: The data load process begins each year when approximately 30 data sets are identified and individual participant files (e.g., dealers, fishermen, vessels) are submitted to ACCSP staff by the program partners. Staff then load the participant files for each data source and merge those data to remove duplicate entries and maintain a history of the individual permits.
B. Load Initial Data: Next, program partners submit trip-level data sets including dealer landings, as well as harvester and fishermen catch records. Before data submission, program partners are responsible for the first round of quality assurance and control measures. Each partner has its own unique set of measures. These submitted trip-level data sets include references to the previously loaded participant files (see Identify Data Sets & Consolidate Participant Files). At this point, the Data Team completes an initial load of the data sets. If the participant information is not linked, the landings are entered as “unknown.” During this step, data sets are considered initial because there may be records that indicate dealer and fishermen information for the same fish being landed. However, this information is merged in the next step.
C. Merge Dealer and Fishermen Data & Load Secondary Data: Due to variations in reporting requirements along the coast, in some cases the most accurate accounting for landings is a combination of dealer and fisherman reports. Therefore, the Data Team has written software (customized to program partner requests) to merge records for five states that have both fishermen and dealer data reporting requirements. During this step, the Data Team completes a secondary load of all of the merged data. Also, there is a second round of quality assurance and control measures, as ACCSP staff validates poundage, price, and conversion factors.
D. Views are Refreshed: Once the process is complete, there is a refresh of the Data Warehouse, including the preliminary data sets from the previous year, conducted in April and then again in September incorporating final data sets from the previous year. At either point in the process, confidential and non-confidential account holders can view the previous year’s consolidated data sets in the Data Warehouse. Both the preliminary and final data sets are submitted to NOAA Fisheries for inclusion within the fisheries databases of the Northeast Fisheries Science Center, Southeast Fisheries Science Center, and Headquarters. The data are also incorporated in the annual FUS document. Once preliminary or final data sets are uploaded to the Data Warehouse, quality assurance and control queries can be run by members of the Commercial Technical Committee. If it turns out that partner data sets from any year need to be updated, those new data sets are integrated into the Data Warehouse after merges. If significant changes are incorporated to data sets, a refresh to the Data Warehouse will occur and partners are notified so that they can update the data in their systems.
Annual Recreational Catch and Effort Data Load Process
The data set for the recreational catch and effort data is the information gathered from MRIP. Recreational catch and effort data are loaded into the Data Warehouse by the Data Team one week after they are released by MRIP. MRIP releases data 45 days after each collection period, known as a “wave” is complete. Each wave is a two month interval (i.e., January and February = wave 1, March and April = wave 2, May and June = wave 3, July and August = wave 4, September and October = wave 5, November and December = wave 6). While the Data Warehouse stores the same information as MRIP, ACCSP provides added value to that information with time-saving tools (such as exporting pivot tables and the ability to email workbooks), attractive reports tailored to user needs, and the ability to create a query with a uniquely fine-tuned level of detail for Atlantic coast fisheries.
The ACCSP is a cooperative state-federal program that designs, implements, and conducts marine fisheries statistics data collection programs and integrates those data into a single data management system that will meet the needs of fishery managers, scientists, and fishermen