Skip to content

Commit 0293a8d

Browse files
christopherpickeringchristopherpickering
committed
Release 2021.06.1
1 parent f09c6ec commit 0293a8d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+1893
-3682
lines changed

‎.gitlab-ci.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ pages:
7777
- public
7878
only:
7979
- master
80+
- dev
8081

8182
deploy_test:
8283
image: alpine:latest

‎Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626

2727
FROM microsoft/dotnet:2.2-sdk-alpine3.8 AS build
2828
WORKDIR /app
29-
COPY Data_Governance_WebApp.sln .
29+
COPY web.sln .
3030
COPY ["./web/web.csproj", "./web/"]
3131
RUN ls "./web/"
3232
RUN dotnet restore
@@ -96,7 +96,7 @@ RUN apt-get update || true && \
9696
WORKDIR /app
9797

9898
# copy and execute database creation and seed scriptsnot
99-
COPY ["./web/atkas-creation_script.sql", "create.sql"]
99+
COPY ["./web/atlas-creation_script.sql", "create.sql"]
100100
COPY ["./web/atlas-demo-seed_script.sql", "seed.sql"]
101101

102102
# sql server takes about 30 seconds to start.

‎docs/atlas/deploy.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ Deploy to IIS
6969

7070
Deploying with Visual Studio is the preferred method. After opening the ``web.sln`` file -
7171

72+
7273
- First update ``web/appsettings.json`` with the correct settings for your database and organization.
7374
- In Visual Studio's menu, click **Build** then **Publish Web**
7475
- Create a new publish profile.

‎docs/atlas/etl.rst‎

Lines changed: 0 additions & 52 deletions
This file was deleted.

‎docs/atlas/etl/deploy.rst‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
..
2+
Atlas of Information Management
3+
Copyright (C) 2020 Riverside Healthcare, Kankakee, IL
4+
5+
This program is free software: you can redistribute it and/or modify
6+
it under the terms of the GNU General Public License as published by
7+
the Free Software Foundation, either version 3 of the License, or
8+
(at your option) any later version.
9+
10+
This program is distributed in the hope that it will be useful,
11+
but WITHOUT ANY WARRANTY; without even the implied warranty of
12+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13+
GNU General Public License for more details.
14+
15+
You should have received a copy of the GNU General Public License
16+
along with this program. If not, see <https://www.gnu.org/licenses/>.
17+
18+
**********
19+
Deploy
20+
**********
21+
22+
23+
Once the ETL performs to your satisfaction in Visual Studio it can be deployed to an SSRS server and scheduled as a job.
24+
25+
- Open ETL solution (ETL.sln) in Visual Studio
26+
27+
- Highlight all the packages you want to deploy and right click on the ETL > deploy.
28+
29+
.. list-table::
30+
31+
* - .. figure:: ../../images/deploy/packages.png
32+
:alt: visual studio deploy
33+
34+
- Choose SSIS as the deploy target
35+
36+
.. list-table::
37+
38+
* - .. figure:: ../../images/deploy/target.png
39+
:alt: visual studio deploy
40+
41+
- Enter the destination server name, and path the ETL location. You can create the folder and project.
42+
43+
.. list-table::
44+
45+
* - .. figure:: ../../images/deploy/destination.png
46+
:alt: destination
47+
* - .. figure:: ../../images/deploy/folder.png
48+
:alt: folder name
49+
* - .. figure:: ../../images/deploy/project_name.png
50+
:alt: project name
51+
52+
- Click "Deploy"
53+
54+
.. list-table::
55+
56+
* - .. figure:: ../../images/deploy/deploy_button.png
57+
:alt: deploy button
58+
59+
- Verify "Success"
60+
61+
.. list-table::
62+
63+
* - .. figure:: ../../images/deploy/results.png
64+
:alt: deploy success

‎docs/atlas/etl/index.rst‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
..
2+
Atlas of Information Management
3+
Copyright (C) 2020 Riverside Healthcare, Kankakee, IL
4+
5+
This program is free software: you can redistribute it and/or modify
6+
it under the terms of the GNU General Public License as published by
7+
the Free Software Foundation, either version 3 of the License, or
8+
(at your option) any later version.
9+
10+
This program is distributed in the hope that it will be useful,
11+
but WITHOUT ANY WARRANTY; without even the implied warranty of
12+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13+
GNU General Public License for more details.
14+
15+
You should have received a copy of the GNU General Public License
16+
along with this program. If not, see <https://www.gnu.org/licenses/>.
17+
18+
ETL
19+
===
20+
21+
.. toctree::
22+
:maxdepth: 1
23+
:titlesonly:
24+
:hidden:
25+
26+
install.rst
27+
ldap.rst
28+
deploy.rst
29+
schedule.rst
30+
31+
.. toctree::
32+
:caption: Releases
33+
:hidden:
34+
35+
updates/index
36+
37+
38+
The Atlas ETL (designed for use with the `Atlas web application <https://github.com/Riverside-Healthcare/Atlas>`_), is used to collect report metadata from a variety of sources and merge it into a common database.
39+
40+
There are two primary ETLs -
41+
42+
- Report Metadata (split into 8 sections)
43+
- Report Run Data
44+
45+
The ETL's are both SSIS packages which are scheduled to run as jobs on a Microsft SQL Server. The metadata ETl can be scheduled to run at a fairly frequent interval - every 15 minutes - hour, while the run data ETL can be run daily.
46+
47+
The Clarity ETL can be added here: `Clarity ETL <https://datahandbook.epic.com/Reports/Details/9000648>`_
48+
49+
Requirements
50+
------------
51+
52+
- `Active Directory Explorer <https://docs.microsoft.com/en-us/sysinternals/downloads/adexplorer>`_, or other access to Active Directory
53+
54+
55+
Steps to Run
56+
------------
57+
1. Run database create scripts (LDAP, Data Governance, DG Staging). Set Datagov user credentials in database.
58+
2. If you want to add demo data you can run the Data Governance database seeding script
59+
3. Configure and run python LDAP script. Modify script and settings.py to match your LDAP build. Schedule in SQL Agent Jobs
60+
4. Configure and run Atlas ETL's (main ETL and run data)
61+
62+
- Delete SSRS sections if not used
63+
- Update Clarity server and credentials
64+
- Update Database connection strings
65+
- Schedule ETL's in SQL Agent Jobs
66+
67+
5. Populate Atlas/appsettings.json & appsettings.Development.json
68+
6. Run website locally in Visual Studio
69+
7. Update publish settings & publish
70+
71+
72+
Credits
73+
-------
74+
75+
76+
Atlas ETL was created by the Riverside Healthcare Analytics team -
77+
78+
* Paula Courville
79+
* `Darrel Drake <https://www.linkedin.com/in/darrel-drake-57562529>`_
80+
* `Dee Anna Hillebrand <https://github.com/DHillebrand2016>`_
81+
* `Scott Manley <https://github.com/Scott-Manley>`_
82+
* `Madeline Matz <mailto:[email protected]>`_
83+
* `Christopher Pickering <https://github.com/christopherpickering>`_
84+
* `Sean Pickering <https://github.com/sean-pickering>`_
85+
* `Dan Ryan <https://github.com/danryan1011>`_
86+
* `Richard Schissler <https://github.com/schiss152>`_
87+
* `Eric Shultz <https://github.com/eshultz>`_

‎docs/atlas/etl/install.rst‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
..
2+
Atlas of Information Management
3+
Copyright (C) 2020 Riverside Healthcare, Kankakee, IL
4+
5+
This program is free software: you can redistribute it and/or modify
6+
it under the terms of the GNU General Public License as published by
7+
the Free Software Foundation, either version 3 of the License, or
8+
(at your option) any later version.
9+
10+
This program is distributed in the hope that it will be useful,
11+
but WITHOUT ANY WARRANTY; without even the implied warranty of
12+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13+
GNU General Public License for more details.
14+
15+
You should have received a copy of the GNU General Public License
16+
along with this program. If not, see <https://www.gnu.org/licenses/>.
17+
18+
**************
19+
How to Install
20+
**************
21+
22+
23+
################
24+
Create Databases
25+
################
26+
27+
There are two databases used -
28+
29+
- ``atlas-staging`` used to prepare raw data and insert it into several staging tables.
30+
- ``atlas`` is the destination database used by the webapp.
31+
32+
33+
The databases can be created by running the two database creation scripts:
34+
35+
- ``atlas_staging-creation_script.sql``
36+
- ``atlas-creation_script.sql`` (kept in the `Atlas repo <https://github.com/Riverside-Healthcare/Atlas/blob/master/web/atlas-creation_script.sql>`_.)
37+
38+
39+
#####################
40+
Create Database Users
41+
#####################
42+
43+
There are two user accounts needed to make the ETL's work.
44+
45+
1. A domain account with read access to all source databases, and write access to the staging database.
46+
2. A local SSRS account with full access to the two databases.
47+
48+
SSRS User
49+
=========
50+
51+
The SSRS user is created at the server level. Once created, they can be added to the databases.
52+
53+
The account should have reader, writer, admin and owner memberships to the two databases.
54+
55+
.. list-table::
56+
57+
* - .. figure:: ../../images/install/add_user.png
58+
:alt: add user
59+
* - .. figure:: ../../images/install/user.png
60+
:alt: create user
61+
* - .. figure:: ../../images/install/user_cred.png
62+
:alt: user credentials
63+
* - .. figure:: ../../images/install/user_membership.png
64+
:alt: user membership
65+
66+
67+
##################
68+
Set ETL Parameters
69+
##################
70+
71+
The ETL connection are kept in the **Connection Manager** and **Parameters**. Update the placeholder connection to valid credentials.
72+
73+
The default ETL uses sources of
74+
75+
- Tableau (see `tableau install guide <https://github.com/Riverside-Healthcare/Tableau-Metadata-Exporter>`_.)
76+
- Crystal (see `crystal install guide <https://github.com/Riverside-Healthcare/Sqlize-Crystal-Reports>`_.)
77+
- Two SSRS Report Servers
78+
- Clarity
79+
- LDAP Users (see :doc:`LDAP install guide <./ldap>`.)
80+
81+
If any of these are not needed, delete them from the ETL.
82+
83+
.. note:: The same steps can be followed to the Run Data ETL.

‎docs/atlas/etl/ldap.rst‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
..
2+
Atlas of Information Management
3+
Copyright (C) 2020 Riverside Healthcare, Kankakee, IL
4+
5+
This program is free software: you can redistribute it and/or modify
6+
it under the terms of the GNU General Public License as published by
7+
the Free Software Foundation, either version 3 of the License, or
8+
(at your option) any later version.
9+
10+
This program is distributed in the hope that it will be useful,
11+
but WITHOUT ANY WARRANTY; without even the implied warranty of
12+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13+
GNU General Public License for more details.
14+
15+
You should have received a copy of the GNU General Public License
16+
along with this program. If not, see <https://www.gnu.org/licenses/>.
17+
18+
***********
19+
Set Up LDAP
20+
***********
21+
22+
23+
It is presumed that most user profile information will be coming from LDAP. There are `LDAP scripts <https://github.com/Riverside-Healthcare/Atlas/tree/master/LDAP>`_ provided to scrape the necessary data from LDAP server(s) into a separate database.
24+
25+
This can be setup to run as a scheduled job with `Extract Management <https://github.com/Riverside-Healthcare/extract_management>`_, or another tool, to provide data to the Atlas ETL.

0 commit comments

Comments
 (0)