1
1
# nodestream-plugin-github
2
2
3
3
# Overview
4
- This plugin provides a way to scrape github data from the REST api and ingest
5
- them as extractors in nodestream pipelines.
6
4
5
+ This plugin provides a way to scrape GitHub data from the REST api and ingest
6
+ them as extractors in nodestream pipelines.
7
7
8
8
# Setup Neo4j
9
+
9
10
1 . Download and install Neo4j: https://neo4j.com/docs/desktop-manual/current/installation/download-installation/
10
- 1 . Create and start database (version 5.7.0: https://neo4j.com/docs/desktop-manual/current/operations/create-dbms/
11
- 1 . Install APOC: https://neo4j.com/docs/apoc/5/installation/
11
+ 2 . Create and start database (version 5.7.0: https://neo4j.com/docs/desktop-manual/current/operations/create-dbms/
12
+ 3 . Install APOC: https://neo4j.com/docs/apoc/5/installation/
12
13
13
- # Create github credentials
14
- 1 . Create and github access codes:
https://docs.github.com/en/[email protected] /apps/creating-github-apps/authenticating-with-a-github-app/generating-a-user-access-token-for-a-github-app
15
- NOTE: These values will be used in your ` .env `
14
+ # Create GitHub credentials
15
+
16
+ 1 . Create and GitHub access
17
+ codes:
https://docs.github.com/en/[email protected] /apps/creating-github-apps/authenticating-with-a-github-app/generating-a-user-access-token-for-a-github-app
18
+ NOTE: These values will be used in your ` .env `
16
19
17
20
# Install and run the app
18
- 1 . Install python3: https://www.python.org/downloads/
19
- 1 . Install poetry: https://python-poetry.org/docs/#installation
20
- 1 . Install nodestream: https://nodestream-proj.github.io/nodestream/0.5/docs/tutorial/
21
- 1 . Generate a new nodestream project
22
- 1 . Add ` nodestream-github ` to your project dependencies in your nodestream projects pyproject.toml file.
23
- 1 . Install necessary dependencies: ` poetry install `
24
- 1 . In ` nodestream.yaml ` add the following:
21
+
22
+ 1 . Install python3: https://www.python.org/downloads/
23
+ 2 . Install poetry: https://python-poetry.org/docs/#installation
24
+ 3 . Install nodestream: https://nodestream-proj.github.io/nodestream/0.5/docs/tutorial/
25
+ 4 . Generate a new nodestream project
26
+ 5 . Add ` nodestream-github ` to your project dependencies in your nodestream projects pyproject.toml file.
27
+ 6 . Install necessary dependencies: ` poetry install `
28
+ 7 . In ` nodestream.yaml ` add the following:
29
+
25
30
``` yaml
26
31
plugins :
27
32
- name : github
@@ -34,25 +39,33 @@ plugins:
34
39
all_public : True
35
40
rate_limit_per_minute : 225
36
41
targets :
37
- - my-db :
42
+ - my-db :
38
43
pipelines :
39
- - name : github_repos
40
- - name : github_teams
44
+ - name : github_repos
45
+ - name : github_teams
41
46
targets :
42
- database : neo4j
43
- uri : bolt://localhost:7687
44
- username : neo4j
45
- password : neo4j123
47
+ database : neo4j
48
+ uri : bolt://localhost:7687
49
+ username : neo4j
50
+ password : neo4j123
46
51
` ` `
52
+
47
53
1. Set environment variables in your terminal session for: ` GITHUB_ACCESS_TOKEN`.
48
- 1 . Verify nodestream has loaded the pipelines : ` poetry run nodestream show`
49
- 1 . Use nodestream to run the pipelines : ` poetry run nodestream run <pipeline-name> --target my-db`
54
+ 2 . Verify nodestream has loaded the pipelines : ` poetry run nodestream show`
55
+ 3 . Use nodestream to run the pipelines : ` poetry run nodestream run <pipeline-name> --target my-db`
50
56
51
57
# Using make
58
+
52
59
1. Install make (ie. `brew install make`)
53
- 1 . Run `make run`
60
+ 2 . Run `make run`
54
61
62
+ # Contributing
63
+
64
+ When contributing, make sure to sign your commits. To find out more about how to do this, refer to
65
+ this [GitHub documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits).
55
66
56
67
# Authors
68
+
57
69
* Jon Bristow
58
70
* Zach Probst
71
+ * Rohith Reddy
0 commit comments