Catalogue
All data entering the sytem is recording in the csiem_data_catalogue.xlsx spreadsheet that can be found the data-lake directory in the cseim-data GitHub repository. A summary overview diagram is shown below.
A full outline of the latest CSIEM data catalogue is included below, subdivided into priorities.
- High Priority: Data not uploaded into the data-lake folder structure, and is required to drive the model
- Medium Priority: Data not uploaded into the data-lake folder structure, and is required to validate the model
- Low Priority: Data not uploaded into the data-lake folder structure, and is required to add context to other data sources
- Import Reqired: Data that has been uploaded into the lake, but not ingested into the warehouse
- Import Finalised: Data that has been ingested into the warehouse, and may require review.
| Agency / Organisation | Agency ID | Program | Program Code | Data Classification | Import Status |
|---|---|---|---|---|---|
| Fremantle Ports Authority | FPA | FPA WQ Sampling | FPAWQ | WQ Grab | Action Required |
| Fremantle Ports Authority | FPA | FPA ADCP | FPAADCP | HYDRO Sensor | Action Required |
| Fremantle Ports Authority | FPA | Fremantle Bulk Terminal | FBT-WQ | WQ Grab | Action Required |
| Fremantle Ports Authority | FPA | Fremantle Bulk Jetty | FBJ-WQ | WQ Grab | Action Required |
| Water Corporation | WC | Perth Long-Term Ocean Outlet Monitoring | PLOOM | WQ Grab | Action Required |
| Water Corporation | WC | Sepia Depression Ocean Outlet Landline | SDOOL | WQ Grab | Action Required |
| Water Corporation | WC | Sepia Depression Ocean Outlet Landline | SDOOL-FLOW | HYDRO Flow | Action Required |
| Water Corporation | WC | Perth Seawater Desalination Plant | PSDP-Profiling | WQ Sensor | Action Required |
| Water Corporation | WC | Perth Seawater Desalination Plant | PSDP-SCADA | WQ Sensor | Action Required |
| Water Corporation | WC | Perth Seawater Desalination Plant | PSDP-WQ | WQ Grab | Action Required |
| Water Corporation | WC | Perth Seawater Desalination Plant | PSDP-BRINE | HYDRO Sensor | Action Required |
| Agency / Organisation | Agency ID | Program | Program Code | Data Classification | Import Status |
|---|---|---|---|---|---|
| Department of Primary Industries and Regional Development | DPRID | Automatic Weather Stations | AWS | MET General | Action Required |
| BP Refinery | BPR | BP Refinery Kwinana | BPR-WQ | WQ Grab | Action Required |
| CSBP Limited | CSBP | CSBP Limited | CSBP-WQ | WQ Grab | Action Required |
| Water Corporation | WC | Perth Coastal Waters 1995 | Action Required | ||
| Department of Biodiversity, Conservation and Attractions | DBCA | Cockburn Sound Env Study (1970) | DEC-CSES | Action Required | |
| Department of Water and Environmental Regulation | DWER | Souther Metropolitan Coastal Waters (1990) | EPA-SMCW | Action Required |
| Agency / Organisation | Agency ID | Program | Program Code | Data Classification | Import Status |
|---|---|---|---|---|---|
| Bureau of Meteorology | BOM | Australian Baseline Sea Level Monitoring Project | IDO | HYDRO Level | Action Required |
| Bureau of Meteorology | BOM | Australia Rainfall and River Conditions | IDW | HYDRO Flow | Action Required |
| Bureau of Meteorology | BOM | Australian Groundwater Data | NGIS | GW Level | Action Required |
| University of Western Australia | UWA | CWR studies | CWR | WQ Sensor; HYDRO Sensor | Action Required |
| University of Western Australia | UWA | SCEVO | AED | WQ Grab;WQ Sensor;WQ Profile;HYDRO Flow | Action Required |
| Agency / Organisation | Agency ID | Program | Program Code | Data Classification | Import Status |
|---|---|---|---|---|---|
| Department of Water and Environmental Regulation | DWER | Cockburn Sound Moorings | CSMOORING | WQ Sensor | Data Lake |
| Tronox Kwinana Pigment Plant | TKPP | Tronox Kwinana Pigment Plant | TKPP-WQ | SED Grab | Data Lake |
| Western Australian Marine Science Institution | WAMSI | WAMSI Westport Marine Science Program | WWMSP3 | WQ Grab | Data Lake |
| Western Australian Marine Science Institution | WAMSI | WAMSI Westport Marine Science Program | WWMSP3.1 | WQ Sensor | Data Lake |
| University of Western Australia | UWA | CWR CTD Profiling | CWRCTD | WQ Sensor | Data Lake |
| University of Western Australia | UWA | GK-PAR | GK-PAR | WQ Sensor | Data Lake |
| Agency / Organisation | Agency ID | Program | Program Code | Data Classification | Import Status |
|---|---|---|---|---|---|
| Bureau of Meteorology | BOM | Weather | IDY | MET General | Data Warehouse |
| Integrated Marine Observing System | IMOS | AMNM Mooring | AMNMMOORING | HYDRO Sensor | Data Warehouse |
| Integrated Marine Observing System | IMOS | BGC | BGC | WQ Grab | Data Warehouse |
| Integrated Marine Observing System | IMOS | ANMN Burst/Profiles | AMNMProfile | WQ Profile | Data Warehouse |
| Department of Water and Environmental Regulation | DWER | Swan Canning Estuary Monitoring | SWANEST | WQ Grab | Data Warehouse |
| Department of Water and Environmental Regulation | DWER | Swan Catchment Sampling | SWANCATCH | WQ GRAB; HYDRO Flow | Data Warehouse |
| Department of Water and Environmental Regulation | DWER | Canning River Estuary Sampling | CANEST | WQ Grab | Data Warehouse |
| Department of Water and Environmental Regulation | DWER | Cockburn Sound Marine Water Quality Monitoring | CSMWQ | WQ Grab | Data Warehouse |
| Department of Transport | DOT | Coastal Data | TIDE | HYDRO Level | Data Warehouse |
| Cockburn Sound Management Council | CSMC | Cockburn Sound Marine Water Quality Monitoring | CSMWQ-MAFRL | WQ Grab | Data Warehouse |
| Western Australian Marine Science Institution | WAMSI | WAMSI Westport Marine Science Program | WWMSP5.1 | WQ Sensor; HYDRO Sensor; HYDRO Level | Data Warehouse |
Aquatic EcoDynamics
