Skip to content

Commit 2f40625

Browse files
andrewelambBryanFaublejaymedinathomasyu888linglp
authored
V24.11.1 (#1543)
* add new tests * add unit tests * ran black * Update schematic/models/validate_attribute.py Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> * added tests * Update README.md * Update README.md * add unit tests * run black * Update README.md * temp commit * remove old tests * [FDS-2386] Synapse entity tracking and code concurrency updates (#1505) * [FDS-2386] Synapse entity tracking and code concurrency updates * ran black * Update CODEOWNERS * updated data model type rules to include error param * fix validate type attribute to use msg level param * added error handling * run black * create Node class * sat up Node class so that nodes with no displayName fields cause an error on creation * ran black * ran mypy * added new configs for CLI tests * added new manifests for testing CLI commands * automate manual CLI tests * ran black * Update CODEOWNERS * Update scan_repo.yml * Update .github/CODEOWNERS * Update .github/workflows/scan_repo.yml * Attach additional telemetry data to OTEL traces (#1519) * Attach additional telemetry data to OTEL traces * feat: added tracing for cross manifest validation and file name validation (#1509) * add tracing for GX validation * temp commit * Updating contribution doc to expect squash and merge (#1534) * [FDS-2491] Integration tests for Schematic API Test plan (#1512) Integration tests for Schematic API Test plan * [FDS-2500] Add Integration Tests for: Manifest Validation (#1516) * Add Integration Tests for: Manifest Validation * [FDS-2449] Lock `sphinx` version and update `poetry.lock` (#1530) Also install `typing-extensions` in the build * manual test files now being saved in manifests folder * manual test files now being saved in manifests folder * remove lines to delete json files that were under git control * ran black * add try finally blocks to remove created files * ran black * add lines to remove created json files * Update file annotation store process to require filename be present in order to annotate file * add lines to remove created json files * Revert "Update file annotation store process to require filename be present in order to annotate file" This reverts commit f57c718. * Don't attempt to annotate the table * add code in finally blocks to reset config to default values, when tests change them * complete submit manifest command test * ran black * add test for bug case * update test for table tidyness * remove unused import * remove etag column if already present when building temp file view * catch all exceptions to switch to sequential mode * update test for updated data * Revert "update test for updated data" This reverts commit 255e3c0. * Revert "catch all exceptions to switch to sequential mode" This reverts commit 68b0b24. * catch ValueErrors as well * Updates for integration test failures (#1537) * Updates for integration test failures, Config file reset and scope changes * add todos for removing config resets * [FDS-2525] Authenticated export of telemetry data (#1527) * Authenticated export of telemetry data, updating to HTTP otel library * temp reduce tests * restore tests * uncomment tests * redid how files are deleted, manual tests values are set * ran black * [SCHEMATIC-157] Make some dependencies required to avoid `schematic CLI` commands from potentially erroring when doing a pip install (#1540) * Make otel flash non-optional * Add dependencies as non-optional * Update pyproject.toml --------- Co-authored-by: BryanFauble <17128019+BryanFauble@users.noreply.github.com> Co-authored-by: Jenny V Medina <jenny.medina@sagebase.org> Co-authored-by: Thomas Yu <thomas.yu@sagebase.org> Co-authored-by: Lingling <55448354+linglp@users.noreply.github.com> Co-authored-by: GiaJordan <gianna.jordan@sagebase.org>
1 parent d68f624 commit 2f40625

File tree

67 files changed

+9925
-2822
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+9925
-2822
lines changed

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
# Global owners of whole repo
2-
* @andrewelamb @GiaJordan @linglp
2+
* @Sage-Bionetworks/schematic-developers

‎.github/workflows/scan_repo.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ jobs:
1717
uses: actions/checkout@v4
1818

1919
- name: Run Trivy vulnerability scanner in repo mode
20-
uses: aquasecurity/trivy-action@master
20+
uses: aquasecurity/trivy-action@0.28.0
2121
with:
2222
# the scan targets the file system.
2323
scan-type: 'fs'

‎.github/workflows/test.yml‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,16 @@ jobs:
134134
SERVICE_ACCOUNT_CREDS: ${{ secrets.SERVICE_ACCOUNT_CREDS }}
135135
run: >
136136
poetry run pytest --durations=0 --cov-append --cov-report=term --cov-report=html:htmlcov --cov-report=xml:coverage.xml --cov=schematic/
137-
-m "not (rule_benchmark)" --reruns 4 -n 8 --ignore=tests/unit
137+
-m "not (rule_benchmark or single_process_execution)" --reruns 4 -n 8 --ignore=tests/unit
138+
139+
- name: Run integration tests single process
140+
if: ${{ contains(fromJSON('["3.10"]'), matrix.python-version) }}
141+
env:
142+
SYNAPSE_ACCESS_TOKEN: ${{ secrets.SYNAPSE_ACCESS_TOKEN }}
143+
SERVICE_ACCOUNT_CREDS: ${{ secrets.SERVICE_ACCOUNT_CREDS }}
144+
run: >
145+
poetry run pytest --durations=0 --cov-append --cov-report=term --cov-report=html:htmlcov --cov-report=xml:coverage.xml --cov=schematic/
146+
-m "single_process_execution" --reruns 4 --ignore=tests/unit
138147
139148
140149
- name: Upload pytest test results

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,7 @@ clean.sh
164164
# Intermediate files
165165
data/json_schema_logs/json_schema_log.json
166166
great_expectations/checkpoints/manifest_checkpoint.yml
167-
great_expectations/expectations/Manifest_test_suite.json
167+
great_expectations/expectations/Manifest_test_suite*.json
168168

169169
tests/data/example.MockComponent.schema.json
170170
tests/data/mock_manifests/Invalid_Test_Manifest_censored.csv
@@ -177,6 +177,7 @@ tests/data/schema.gpickle
177177
# Created during testting
178178
Example*
179179
manifests/*
180+
https:*
180181

181182
# schematic config file
182183
config.yml

‎.readthedocs.yml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,15 @@ version: 2
77

88
# Set the version of Python and other tools you might need
99
build:
10-
os: ubuntu-20.04
10+
os: ubuntu-22.04
1111
tools:
1212
python: "3.9"
1313
jobs:
1414
post_install:
1515
- pip install poetry==1.3.0
1616
- poetry config virtualenvs.create false
17-
- poetry install --with doc
17+
- poetry install --all-extras
18+
- pip install typing-extensions==4.12.2
1819
#Poetry will install my dependencies into the virtualenv created by readthedocs if I set virtualenvs.create=false
1920
# You can also specify other tool versions:
2021
# nodejs: "16"

‎CONTRIBUTION.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ For new features, bugs, enhancements:
4949
#### 4. Pull Request and Review
5050
* Create a PR from `develop-<feature/fix-name>` into the develop branch of the upstream repo
5151
* Request a code review on the PR
52-
* Once code is approved merge in the develop branch. We suggest creating a merge commit for a cleaner commit history on the `develop` branch.
52+
* Once code is approved merge in the develop branch. The **"Squash and merge"** strategy should be used for a cleaner commit history on the `develop` branch. The description of the squash commit should include enough information to understand the context of the changes that were made.
5353
* Once the actions pass on the main branch, delete the `develop-<feature/fix-name>` branch
5454

5555
### Updating readthedocs documentation

‎README.md‎

Lines changed: 125 additions & 55 deletions
Large diffs are not rendered by default.

‎env.example‎

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,37 @@ SERVER_DOMAIN=localhost
55
USE_LISTEN_PORT=81
66
SERVICE_ACCOUNT_CREDS='Provide service account creds'
77

8-
# Integration testing variables (Optional)
8+
## Opentelemetry configuration variables (Optional)
9+
## Only otlp is supported
910
# TRACING_EXPORT_FORMAT=otlp
1011
# LOGGING_EXPORT_FORMAT=otlp
11-
# TRACING_SERVICE_NAME=unique-name-testing
12-
# LOGGING_SERVICE_NAME=unique-name-testing
13-
# LOGGING_INSTANCE_NAME=unique-name-testing
12+
# TRACING_SERVICE_NAME=schematic-api
13+
# LOGGING_SERVICE_NAME=schematic-api
14+
## Other examples: dev, staging, prod
15+
# DEPLOYMENT_ENVIRONMENT=local
16+
# OTEL_EXPORTER_OTLP_ENDPOINT=https://..../telemetry
17+
## Opentelemetry authentication
18+
# TELEMETRY_EXPORTER_CLIENT_ID=...
19+
# TELEMETRY_EXPORTER_CLIENT_SECRET-...
20+
# TELEMETRY_EXPORTER_CLIENT_TOKEN_ENDPOINT=...
21+
# TELEMETRY_EXPORTER_CLIENT_AUDIENCE=...
22+
## Alternative Opentelemetry authentication: Sets a static Authorization header to use for all requests. Used when developing locally
23+
# OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer ey...
24+
25+
26+
# Used during integration test run to determine if files will be output for manual
27+
# inspection. These tests cannot fully finish all validation via code. All of these
28+
# tests will be marked by pytest "manual_verification_required"
29+
# More information: https://sagebionetworks.jira.com/wiki/spaces/SCHEM/pages/3055779846/Schematic+API+test+plan
30+
MANUAL_TEST_VERIFICATION=false
31+
32+
# Used to determine if a local flask instance is created during integration testing. If
33+
# this is true schematic tests will use a schematic API server running outside of the
34+
# context of the integration test. The url used is defined below.
35+
USE_DEPLOYED_SCHEMATIC_API_SERVER=false
36+
37+
# The URL used to execute integration tests for schematic API. Defaults to localhost.
38+
# dev: https://schematic-dev.api.sagebionetworks.org
39+
# staging: https://schematic-staging.api.sagebionetworks.org
40+
# prod: https://schematic.api.sagebionetworks.org
41+
SCHEMATIC_API_SERVER_URL=http://localhost:3001

‎main.py‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,4 @@
11
import os
2-
import connexion
3-
from schematic import CONFIG
4-
from flask_cors import CORS
52
from schematic_api.api import app
63

74

0 commit comments

Comments
 (0)