Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: nodestream-proj/nodestream
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.12.5a14
Choose a base ref
...
head repository: nodestream-proj/nodestream
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Loading
Showing with 13,661 additions and 13,133 deletions.
  1. +150 −0 .all-contributorsrc
  2. +3 −3 .github/workflows/ci.yaml
  3. +2 −2 .github/workflows/release.yaml
  4. +1 −1 .gitignore
  5. +2 −10 Makefile
  6. +121 −27 README.md
  7. +14 −0 SECURITY.md
  8. +0 −59 docs/contributing.md
  9. +0 −84 docs/docs/guides/creating-your-own-argument-resolver.md
  10. +0 −88 docs/docs/guides/creating-your-own-audits.md
  11. +0 −124 docs/docs/guides/creating-your-own-commands.md
  12. +0 −225 docs/docs/guides/creating-your-own-database-connector.md
  13. +0 −30 docs/docs/guides/creating-your-own-extractor.md
  14. +0 −75 docs/docs/guides/creating-your-own-file-format-handler.md
  15. +0 −119 docs/docs/guides/creating-your-own-interpretation.md
  16. +0 −77 docs/docs/guides/creating-your-own-normalizer.md
  17. +0 −65 docs/docs/guides/creating-your-own-transformer.md
  18. +0 −131 docs/docs/guides/creating-your-own-value-provider.md
  19. +0 −122 docs/docs/guides/customizing-the-queue-extractor.md
  20. +0 −122 docs/docs/guides/customizing-the-stream-extractor.md
  21. +0 −46 docs/docs/guides/implementing-pipeline-testing.md
  22. +0 −66 docs/docs/guides/implementing-time-to-live.md
  23. +0 −53 docs/docs/guides/ingestion-hooks.md
  24. +0 −166 docs/docs/guides/project-plugins.md
  25. +0 −45 docs/docs/reference/argument-resovlers.md
  26. +0 −368 docs/docs/reference/extractors.md
  27. +0 −44 docs/docs/reference/file-formats.md
  28. +0 −37 docs/docs/reference/graph-database-writer.md
  29. +0 −365 docs/docs/reference/interpretations.md
  30. +0 −205 docs/docs/reference/interpreter.md
  31. +0 −14 docs/docs/reference/normalizers.md
  32. +0 −63 docs/docs/reference/transfomers.md
  33. +0 −224 docs/docs/reference/value-providers.md
  34. +0 −406 docs/docs/tutorial.md
  35. +0 −20 docs/gen_ref_pages.py
  36. BIN docs/img/NodeSteamLogo_Small_Dark.png
  37. BIN docs/img/NodeSteamLogo_Small_Light.png
  38. BIN docs/img/demo.gif
  39. +0 −42 docs/index.md
  40. +0 −163 docs/technology.md
  41. +0 −20 docs/vhs/demo.tape
  42. +0 −8 docs_overrides/main.html
  43. +0 −64 mkdocs.yml
  44. +2 −0 nodestream/cli/commands/__init__.py
  45. +33 −21 nodestream/cli/commands/copy.py
  46. +15 −4 nodestream/cli/commands/run.py
  47. +12 −2 nodestream/cli/commands/run_migrations.py
  48. +41 −0 nodestream/cli/commands/shared_options.py
  49. +31 −0 nodestream/cli/commands/squash_migrations.py
  50. +4 −0 nodestream/cli/operations/__init__.py
  51. +33 −0 nodestream/cli/operations/generate_squashed_migration.py
  52. +4 −3 nodestream/cli/operations/initialize_logger.py
  53. +43 −0 nodestream/cli/operations/intitialize_metrics.py
  54. +4 −1 nodestream/cli/operations/run_copy.py
  55. +26 −6 nodestream/cli/operations/run_pipeline.py
  56. +11 −6 nodestream/databases/debounced_ingest_strategy.py
  57. +6 −1 nodestream/databases/ingest_strategy.py
  58. +9 −1 nodestream/databases/null.py
  59. +4 −0 nodestream/databases/query_executor.py
  60. +8 −10 nodestream/databases/query_executor_with_statistics.py
  61. +5 −0 nodestream/databases/writer.py
  62. +1 −1 nodestream/interpreting/interpretation_passes.py
  63. +2 −1 nodestream/interpreting/interpretations/__init__.py
  64. +134 −0 nodestream/interpreting/interpretations/conditions.py
  65. +17 −1 nodestream/interpreting/interpretations/interpretation.py
  66. +18 −0 nodestream/interpreting/interpretations/property_mapping.py
  67. +14 −0 nodestream/interpreting/interpretations/relationship_interpretation.py
  68. +24 −3 nodestream/interpreting/interpretations/source_node_interpretation.py
  69. +297 −0 nodestream/metrics.py
  70. +6 −3 nodestream/model/graph_objects.py
  71. +0 −1 nodestream/pipeline/__init__.py
  72. +169 −0 nodestream/pipeline/channel.py
  73. +2 −3 nodestream/pipeline/extractors/__init__.py
  74. +3 −3 nodestream/pipeline/extractors/credential_utils.py
  75. +44 −10 nodestream/pipeline/extractors/extractor.py
  76. +202 −49 nodestream/pipeline/extractors/files.py
  77. +15 −1 nodestream/pipeline/extractors/iterable.py
  78. +1 −2 nodestream/pipeline/extractors/stores/aws/__init__.py
  79. +27 −6 nodestream/pipeline/extractors/stores/aws/athena_extractor.py
  80. +36 −130 nodestream/pipeline/extractors/stores/aws/dynamodb_extractor.py
  81. +0 −118 nodestream/pipeline/extractors/stores/aws/s3_extractor.py
  82. +9 −12 nodestream/pipeline/extractors/ttls.py
  83. +211 −10 nodestream/pipeline/filters.py
  84. +0 −37 nodestream/pipeline/meta.py
  85. +4 −2 nodestream/pipeline/normalizers/__init__.py
  86. +7 −1 nodestream/pipeline/normalizers/normalizer.py
  87. +8 −0 nodestream/pipeline/normalizers/uppercase_strings.py
  88. +340 −0 nodestream/pipeline/object_storage.py
  89. +169 −245 nodestream/pipeline/pipeline.py
  90. +19 −2 nodestream/pipeline/pipeline_file_loader.py
  91. +16 −4 nodestream/pipeline/progress_reporter.py
  92. +140 −20 nodestream/pipeline/step.py
  93. +77 −91 nodestream/pipeline/transformers/transformer.py
  94. +2 −0 nodestream/pipeline/value_providers/__init__.py
  95. +55 −0 nodestream/pipeline/value_providers/cast_value_provider.py
  96. +51 −8 nodestream/pipeline/value_providers/jmespath_value_provider.py
  97. +1 −3 nodestream/pipeline/value_providers/normalizer_value_provider.py
  98. +11 −14 nodestream/pipeline/writers.py
  99. +3 −3 nodestream/project/pipeline_scope.py
  100. +14 −1 nodestream/project/project.py
  101. +84 −0 nodestream/project/storage.py
  102. +121 −12 nodestream/schema/migrations/migrations.py
  103. +246 −0 nodestream/schema/migrations/operations.py
  104. +21 −1 nodestream/schema/migrations/project_migrations.py
  105. +1 −1 nodestream/schema/migrations/state_providers.py
  106. +5 −0 nodestream/schema/printers/graphql_schema_printer.py
  107. +7 −0 nodestream/schema/state.py
  108. +6 −0 playground_project/nodestream.yaml
  109. +1 −0 playground_project/test.yaml
  110. +1 −1 playground_project/test2.yaml
  111. +1,395 −1,295 poetry.lock
  112. +30 −25 pyproject.toml
  113. +0 −72 scripts/release.sh
  114. +35 −0 tests/data/test_snapshot_handling_during_errors.py
  115. +1 −0 tests/integration/fixtures/pipelines/airports.yaml
  116. +1 −0 tests/integration/fixtures/pipelines/dns.yaml
  117. +1 −0 tests/integration/fixtures/pipelines/fifa_2021_player_data.yaml
  118. +1 −0 tests/integration/fixtures/pipelines/multiple_passes.yaml
  119. +1 −0 tests/integration/fixtures/pipelines/people.yaml
  120. +1 −0 tests/integration/fixtures/pipelines/source_eager.yaml
  121. +1 −0 tests/integration/fixtures/pipelines/source_match_only.yaml
  122. +5,772 −5,772 tests/integration/snapshots/interpretation_snapshot_airports.yaml.json
  123. +400 −400 tests/integration/snapshots/interpretation_snapshot_fifa_2021_player_data.yaml.json
  124. +0 −163 tests/integration/test_pipeline_error_propagation.py
  125. +11 −4 tests/integration/test_pipeline_flush_handling.py
  126. +4 −2 tests/unit/cli/commands/test_copy.py
  127. +1 −1 tests/unit/cli/commands/test_run_command.py
  128. +14 −2 tests/unit/cli/commands/test_run_migrations.py
  129. +17 −0 tests/unit/cli/commands/test_squash_migrations.py
  130. +22 −0 tests/unit/cli/operations/test_generate_squashed_migration.py
  131. +3 −2 tests/unit/cli/operations/test_initialize_logger.py
  132. +68 −0 tests/unit/cli/operations/test_initialize_metrics.py
  133. +20 −3 tests/unit/cli/operations/test_run_pipeline.py
  134. +7 −0 tests/unit/databases/test_debounced_ingest_strategy.py
  135. +28 −17 tests/unit/databases/test_query_executor_with_statistics.py
  136. +22 −0 tests/unit/databases/test_writer.py
  137. +35 −0 tests/unit/interpreting/interpretations/test_interpretation.py
  138. +2 −0 tests/unit/interpreting/interpretations/test_relationship_interpretation.py
  139. +22 −1 tests/unit/interpreting/interpretations/test_source_node_interpretation.py
  140. +139 −0 tests/unit/interpreting/test_conditions.py
  141. +3 −9 tests/unit/interpreting/test_interpreter.py
  142. +6 −6 tests/unit/pipeline/extractors/queues/test_sqs.py
  143. +186 −0 tests/unit/pipeline/extractors/stores/aws/test_athena_extractor.py
  144. +29 −0 tests/unit/pipeline/extractors/stores/aws/test_credential_utils.py
  145. +157 −179 tests/unit/pipeline/extractors/stores/aws/test_dynamo_extractor.py
  146. +0 −294 tests/unit/pipeline/extractors/stores/aws/test_s3_extractor.py
  147. +464 −7 tests/unit/pipeline/extractors/test_files.py
  148. +19 −0 tests/unit/pipeline/extractors/test_iterable_extractor.py
  149. +1 −1 tests/unit/pipeline/normalizers/test_lowercase_strings.py
  150. +1 −1 tests/unit/pipeline/normalizers/test_remove_trailing_dots.py
  151. +1 −1 tests/unit/pipeline/normalizers/test_trim_whitespace.py
  152. +21 −0 tests/unit/pipeline/normalizers/test_uppercase_strings.py
  153. +69 −0 tests/unit/pipeline/test_channel.py
  154. +142 −6 tests/unit/pipeline/test_filters.py
  155. +0 −21 tests/unit/pipeline/test_meta.py
  156. +244 −0 tests/unit/pipeline/test_object_storage.py
  157. +95 −109 tests/unit/pipeline/test_pipeline.py
  158. +1 −1 tests/unit/pipeline/test_pipeline_progress_reporter.py
  159. +104 −0 tests/unit/pipeline/test_step.py
  160. +6 −7 tests/unit/pipeline/test_writers.py
  161. +1 −5 tests/unit/pipeline/transformers/test_expand_json_field.py
  162. +49 −75 tests/unit/pipeline/transformers/test_transformer.py
  163. +2 −2 tests/unit/pipeline/transformers/test_value_projection.py
  164. +62 −0 tests/unit/pipeline/value_providers/test_cast_value_provider.py
  165. +8 −9 tests/unit/pipeline/value_providers/test_jmespath_value_provider.py
  166. +7 −0 tests/unit/project/fixtures/simple_project.yaml
  167. +11 −0 tests/unit/project/test_project.py
  168. +90 −0 tests/unit/project/test_storage.py
  169. +100 −3 tests/unit/schema/migrations/test_migrations.py
  170. +310 −0 tests/unit/schema/migrations/test_operations.py
  171. +12 −0 tests/unit/schema/migrations/test_project_migrations.py
  172. +71 −0 tests/unit/test_extractor.py
  173. +109 −0 tests/unit/test_metrics.py
150 changes: 150 additions & 0 deletions .all-contributorsrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
{
"projectName": "nodestream",
"projectOwner": "nodestream-proj",
"repoType": "github",
"repoHost": "https://github.com",
"files": [
"README.md"
],
"imageSize": 100,
"commit": true,
"commitConvention": "none",
"contributors": [
{
"login": "zprobst",
"name": "Zach Probst",
"avatar_url": "https://avatars.githubusercontent.com/u/22159145?v=4",
"profile": "https://github.com/zprobst",
"contributions": [
"code",
"review",
"maintenance"
]
},
{
"login": "ccloes",
"name": "Chad Cloes",
"avatar_url": "https://avatars.githubusercontent.com/u/1000538?v=4",
"profile": "https://github.com/ccloes",
"contributions": [
"code",
"review",
"maintenance"
]
},
{
"login": "angelosantos4",
"name": "asantos4",
"avatar_url": "https://avatars.githubusercontent.com/u/142852840?v=4",
"profile": "https://github.com/angelosantos4",
"contributions": [
"code",
"review",
"maintenance"
]
},
{
"login": "grantleehoffman",
"name": "Grant Hoffman",
"avatar_url": "https://avatars.githubusercontent.com/u/603848?v=4",
"profile": "https://github.com/grantleehoffman",
"contributions": [
"code",
"review"
]
},
{
"login": "khneal",
"name": "khneal",
"avatar_url": "https://avatars.githubusercontent.com/u/40273388?v=4",
"profile": "https://github.com/khneal",
"contributions": [
"code"
]
},
{
"login": "orozen",
"name": "orozen",
"avatar_url": "https://avatars.githubusercontent.com/u/62594754?v=4",
"profile": "https://github.com/orozen",
"contributions": [
"code"
]
},
{
"login": "sophiadt",
"name": "Sophia Don Tranho",
"avatar_url": "https://avatars.githubusercontent.com/u/66497192?v=4",
"profile": "https://sites.google.com/view/ai4allrobotics",
"contributions": [
"code"
]
},
{
"login": "bechbd",
"name": "bechbd",
"avatar_url": "https://avatars.githubusercontent.com/u/6898505?v=4",
"profile": "https://github.com/bechbd",
"contributions": [
"code"
]
},
{
"login": "yasonk",
"name": "yasonk",
"avatar_url": "https://avatars.githubusercontent.com/u/6750414?v=4",
"profile": "https://github.com/yasonk",
"contributions": [
"code",
"review"
]
},
{
"login": "stuartio",
"name": "Stuart Macleod",
"avatar_url": "https://avatars.githubusercontent.com/u/22449467?v=4",
"profile": "https://github.com/stuartio",
"contributions": [
"code"
]
},
{
"login": "Cole-Greer",
"name": "Cole Greer",
"avatar_url": "https://avatars.githubusercontent.com/u/112986082?v=4",
"profile": "https://github.com/Cole-Greer",
"contributions": [
"code"
]
},
{
"login": "Aclucas1",
"name": "Austin Lucas",
"avatar_url": "https://avatars.githubusercontent.com/u/21301692?v=4",
"profile": "https://github.com/Aclucas1",
"contributions": [
"code"
]
},
{
"login": "rreddy2",
"name": "rreddy2",
"avatar_url": "https://avatars.githubusercontent.com/u/18334818?v=4",
"profile": "https://github.com/rreddy2",
"contributions": [
"code"
]
},
{
"login": "jbristow",
"name": "Jon Bristow",
"avatar_url": "https://avatars.githubusercontent.com/u/314260?v=4",
"profile": "http://jondotcomdotorg.net/",
"contributions": [
"code"
]
}
],
"contributorsPerLine": 7,
"linkToUsage": true
}
6 changes: 3 additions & 3 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ jobs:
fail-fast: false
matrix:
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- name: Install poetry
@@ -27,12 +27,12 @@ jobs:
pip install --upgrade pip
pip install --upgrade setuptools
poetry env use ${{ matrix.python-version }}
poetry install
poetry install --all-extras
- name: Run Tests
run: poetry run pytest --cov nodestream --cov-report term --cov-report xml -m "not e2e"
- name: Run Lints
run: |
poetry run ruff nodestream tests
poetry run ruff check nodestream tests
poetry run black nodestream tests --check
poetry run isort nodestream tests --check-only
- name: Upload coverage reports to Codecov
4 changes: 2 additions & 2 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
@@ -63,14 +63,14 @@ jobs:
# This is so we can sign the package and upload it to the GitHub release.
# this is being done as a seperate job so that we can minimize the permissions needed for the publish job.
- name: Store the Built Package
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: python-package-distribution
path: dist/

# We are then going to sign the package using Github's Attest Build Provenance action.
# This action will sign the package and upload the signature to the GitHub release.
# This is to ensure that the package is verified and trusted by the user.
- uses: actions/attest-build-provenance@v1
- uses: actions/attest-build-provenance@v2
with:
subject-path: 'dist/*'
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -160,4 +160,4 @@ cython_debug/
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
.idea/
12 changes: 2 additions & 10 deletions Makefile
Original file line number Diff line number Diff line change
@@ -23,7 +23,7 @@ clean-test:
clean: clean-pyc clean-test

venv: poetry.lock
poetry install
poetry install --all-extras

.PHONY: format
format: venv
@@ -33,7 +33,7 @@ format: venv
.PHONY: lint
lint: venv
poetry run black nodestream tests --check
poetry run ruff nodestream tests
poetry run ruff check nodestream tests

.PHONY: test-unit
test-unit: venv
@@ -43,14 +43,6 @@ test-unit: venv
test-e2e: venv
poetry run pytest -m "e2e"

.PHONY: docs-build
docs-build: venv
poetry run mkdocs build

.PHONY: docs-server
docs-server: venv
poetry run mkdocs serve

.PHONY: snapshot
snapshot: venv
poetry run pytest --snapshot-update
Loading