v24.11.1

NOTE: there is a bug with this release in the CLI

 schematic --help
Traceback (most recent call last):
  File "/Users/tyu/anaconda3/envs/schematic-testing/bin/schematic", line 5, in <module>
    from schematic.__main__ import main
  File "/Users/tyu/anaconda3/envs/schematic-testing/lib/python3.9/site-packages/schematic/__init__.py", line 23, in <module>
    from schematic_api.api.security_controller import info_from_bearer_auth
ModuleNotFoundError: No module named 'schematic_api'

What's Changed

FEATURES

• [FDS-2386] Synapse entity tracking and code concurrency updates by @BryanFauble in #1505
• Attach additional telemetry data to OTEL traces by @BryanFauble in #1519
• feat: added tracing for cross manifest validation and file name validation by @linglp in #1509
• [SCHEMATIC-126] Update file annotation store process to require filename be present i… by @BryanFauble in #1535

BUG FIXES

• [FDS-2506] Fix type Validation by @andrewelamb in #1526
• [SCHEMATIC-1] BugFix: manifest submission when data previously annotated without manifest upload by @GiaJordan in #1538

TECH DEBT

• Update CODEOWNERS by @andrewelamb in #1525
• Update CODEOWNERS and pin down trivy version by @thomasyu888 in #1531
• Fds 2333 validate attribute unit tests by @andrewelamb in #1517
• [FDS-2347] Add unit tests to DataModelValidator class by @andrewelamb in #1523
• [FDS-2466] Update references to schema location in config.yml by @jaymedina in #1522
• Updating contribution doc to expect squash and merge by @BryanFauble in #1534
• [FDS-2491] Integration tests for Schematic API Test plan by @BryanFauble in #1512
• [FDS-2500] Add Integration Tests for: Manifest Validation by @jaymedina in #1516
• [FDS-2449] Lock sphinx version and update poetry.lock by @jaymedina in #1530
• Update for integration test failures by @BryanFauble in #1537
• [FDS-2525] Authenticated export of telemetry data by @BryanFauble in #1527
• [FDS-2238, FDS-2236] Automate CLI test plan by @andrewelamb in #1528
• [SCHEMATIC-157] Make some dependencies required to avoid schematic CLI commands from potentially erroring when doing a pip install by @thomasyu888 in #1540

Full Changelog: v24.10.2...v24.11.1

v24.11.1-rc

What's Changed

FEATURES

• [FDS-2386] Synapse entity tracking and code concurrency updates by @BryanFauble in #1505
• Attach additional telemetry data to OTEL traces by @BryanFauble in #1519
• feat: added tracing for cross manifest validation and file name validation by @linglp in #1509
• [SCHEMATIC-126] Update file annotation store process to require filename be present i… by @BryanFauble in #1535

BUG FIXES

• [FDS-2506] Fix type Validation by @andrewelamb in #1526
• [SCHEMATIC-1] BugFix: manifest submission when data previously annotated without manifest upload by @GiaJordan in #1538

TECH DEBT

• Update CODEOWNERS by @andrewelamb in #1525
• Update CODEOWNERS and pin down trivy version by @thomasyu888 in #1531
• Fds 2333 validate attribute unit tests by @andrewelamb in #1517
• [FDS-2347] Add unit tests to DataModelValidator class by @andrewelamb in #1523
• [FDS-2466] Update references to schema location in config.yml by @jaymedina in #1522
• Updating contribution doc to expect squash and merge by @BryanFauble in #1534
• [FDS-2491] Integration tests for Schematic API Test plan by @BryanFauble in #1512
• [FDS-2500] Add Integration Tests for: Manifest Validation by @jaymedina in #1516
• [FDS-2449] Lock sphinx version and update poetry.lock by @jaymedina in #1530
• Update for integration test failures by @BryanFauble in #1537
• [FDS-2525] Authenticated export of telemetry data by @BryanFauble in #1527
• [FDS-2238, FDS-2236] Automate CLI test plan by @andrewelamb in #1528
• [SCHEMATIC-157] Make some dependencies required to avoid schematic CLI commands from potentially erroring when doing a pip install by @thomasyu888 in #1540

Full Changelog: v24.10.2...v24.11.1-rc

v24.10.2

What's Changed

FEATURES

• feat: set annotations in an async way when submitting a manifest -- new changes added by @linglp in #1452
• Feature: filenameExists Validation Rule by @GiaJordan in #1456
• feat: added mermaid diagrams for some functions in ManifestGenerator by @linglp in #1463

BUG FIXES

• fix: updated schematic installation instructions by @linglp in #1462
• fix: check existing great expectation suite before adding a new one by @linglp in #1458
• [FDS-2294] Prevent including project name twice in walked path by @BryanFauble in #1474
• Bugfix: async tests skipped during regular testing runs by @GiaJordan in #1479
• fix: updated test after updating to synapse python client 4.4.0 and updated test after test resources get updated on synapse by @linglp in #1471
• fix: added unit test for get_node_validation_rules function by @linglp in #1483
• fix: update artifact to v4 by @linglp in #1500
• [FDS-2415] Fixes pytest CI Python Versions by @BWMac in #1502
• [FDS-2415] Fixes pdoc Workflow by @BWMac in #1503
• fix: call get_node_validation_rules with appropriate parameters when storing annotations by @GiaJordan in #1478
• Fixes failing pdoc deployment by @BWMac in #1504
• update how annotations are accessed by @GiaJordan in #1485
• Fix: remove caching poetry dependencies in CI workfows by @linglp in #1507

TECH DEBT

• feat/technical debt: added trivy code scan to schematic repo by @linglp in #1448
• Enable google credentials tests by @GiaJordan in #1459
• update synapse login functionality by @GiaJordan in #1465
• update python client to 4.4.0 by @andrewelamb in #1468
• [FDS-2127] Update URL validation to use requests.options to verify connectivity by @BryanFauble in #1472
• [FDS-2294] Run schematic_api tests by @BryanFauble in #1476
• add csv data model to unit tests by @lakikowolfe in #1464
• Revert "add csv data model to unit tests" by @andrewelamb in #1488
• Added unit tests for validate attribute class by @andrewelamb in #1486
• Update existing file paths in manifests at generation to conform to new convention by @GiaJordan in #1467
• [FDS-2373] Update jwt verification by @BryanFauble in #1493
• [FDS-2373] Remove the need to do version checking on Synapse creation by @BryanFauble in #1495
• Change how dataset to validate against is specified for filenameExists rule by @GiaJordan in #1491
• [FDS-305] Prevent shared snypase resources during integration tests by @BryanFauble in #1482
• [FDS-2383] Update to run integration tests on default runner with python 3.10 by @BryanFauble in #1498
• FDS-2403 Add Codeowners file by @andrewelamb in #1501
• [FDS-1725] Missing entityId handling testing by @BWMac in #1496
• [FDS-2502] Walk back up directory tree to get location in project by @BryanFauble in #1518
• [FDS-2497] Wrap google API execute calls with a 5 attempt retry by @BryanFauble in #1513
• [FDS-2218] Update the README by @jaymedina in #1510

---

• Schematic v24.10.1 by @lakikowolfe in #1514
• v24.10.2 by @lakikowolfe in #1520

New Contributors

• @jaymedina made their first contribution in #1510

Full Changelog: v24.7.2...v24.10.2

v24.7.2

What's Changed

Bug Fixes:

• fix: used environment variable to control tracing which allows tracing to be off during pytest run and normal API run by @linglp in #1455
• fix: install patch version of synapse python client by @linglp in #1457
• v24.7.2 release by @lakikowolfe in #1461

Full Changelog: v24.7.1...v24.7.2

v24.7.1

Features

• Automatically save the JSON Validation schema by @mialy-defelice in #1415
• feat: surfaced schematic exceptions in flask APIs by @linglp in #1438
• feat: added tracing to schematic code base for manifest generate and manifest submission by @linglp in #1430

Bug Fixes

• Bugfix: FDS-1686 test manifest being changed by @andrewelamb in #1412
• BugFix: Manifest version increment by @GiaJordan in #1435
• BugFix: Files no longer added to records based manifests by @GiaJordan in #1447

Technical Debt

• Develop FDS 243 mypy schema module by @andrewelamb in #1416
• added pydantic to toml file by @andrewelamb in #1419
• update pandas and numpy by @andrewelamb in #1421
• FDS-511 removed typing extensions lock by @andrewelamb in #1432
• technical debt: Remove service_acct_creds_synapse_id from schematic config by @linglp in #1436
• Add maintainers and version info to AWS Docker image by @GiaJordan in #1437
• technical debt: run black for schematic api and test and modify pre-commit by @linglp in #1442
• technical debt: added unit test for changeFileMetaData function by @linglp in #1444
• fix: added black compatibility to isort by @linglp in #1453
• technical debt: added isort to pre-commit hook in schematic repo by @linglp in #1441

Full Changelog: v24.5.1...v24.7.1

v24.5.1

Bug Fix

• BugFix: JSON Schema Validation Issues by @mialy-defelice in #1420
• technical debt: remove release.md from repo by @linglp in #1426
• Bug Fix: Fix failing test to conform to data model updates by @mialy-defelice in #1428

Full Changelog: v24.4.1...v24.5.1

v24.4.1

Features

• Allow non-entry if Required=False, while still enforcing validation rules if value is entered by @mialy-defelice in #1390
• Add Manifest download to the CLI by @mialy-defelice in #736
• Cross Manifest Validation: Allow first manifest to submit, if no target manifest to validate against. by @mialy-defelice in #1398
• Reorganize validation message level determination as a precursor to single rule disjunction by @GiaJordan in #1366
• Match None Manifest Validation Rule FDS-1766 by @mialy-defelice in #1387
• Add functionality to set requirements based per component by @mialy-defelice in #1401
• Updated python client to ^4.1.0 by @andrewelamb in #1403

Bug fixes

• Updated workflows to make sure uwsgi gets installed when publishing an image by @linglp in #1377
• Updated workflow to install dependencies and root project together by @linglp in #1380
• Schematic v24.2.1 AWS deployment fix by @linglp in #1378
• Schematic v24.2.1 AWS deployment fix - 2 by @linglp in #1382
• Fix wrong function name reference in function get_viz_component_attributes_explorer by @linglp in #1384
• Added installation of uWSGI in docker file by @linglp in #1381
• Avoid repeatedly calling generate_data_model_graph by @linglp in #1374
• Fix check_synapse_cache_size function; allow file size to be float by @linglp in #1389
• Added help text for file-annotations-upload parameter by @linglp in #1406
• Remove schematic init from CLI by @linglp in #1400
• Added project scope parameter in test_api.py by @linglp in #1409
• JSONLD dictionary parsing error by @mialy-defelice in #1407
• Convert nans in manifest to empty strings just prior to JSON Schema Validation by @mialy-defelice in #1402

Technical debt

• Mypy visualization module by @andrewelamb in #1373

• Mypy utils files by @andrewelamb in #1375

• Pylint schema utils by @andrewelamb in #1371

• Add pre commit hook for black by @GiaJordan in #1379

• Remove makrupsafe from toml file by @andrewelamb in #1383

• Remove temp data model references from test_api by @mialy-defelice in #1386

• Develop mypy schema utils by @andrewelamb in #1392

• Add synapse cache as a parameter to SynapseStorage by @andrewelamb in #1394

• Added file-annotations-upload parameter when submitting a manifest by @linglp in #1395

• Develop pylint schema module by @andrewelamb in #1393

• Turn pyopenssl package optional by @linglp in #1413

• Remove uneeded parameter from submit_metadata_manifest by @andrewelamb in #1410

• Schematic v24.4.1 by @lakikowolfe in #1417

New Contributors

• @lakikowolfe made their first contribution in #1417

Full Changelog: v24.2.1...v24.4.1

v24.2.1

What's Changed

Features:

• Retain formatting of Display name for annotations and table column names by @mialy-defelice in #1330
• Data Model Labels, display_label or class_label options by @mialy-defelice in #1356
• Component Based Rule Settings by @mialy-defelice in #1368

Bug Fixes

• feature-configuration-fixes by @andrewelamb in #1357
• Fixed an issue where empty rows would not be removed from manifests, and add test for load_df util function by @GiaJordan in #1346
• Increase schematic API time out by @linglp in #1358
• Deleted old version of readthedocs yaml file by @linglp in #1365

Technical Debt

• Updated poetry version in readthedocs.yml by @linglp in #1364
• Update pandas and schematic_db dependencies by @GiaJordan in #1347
• Run rule combinations tests at every push instead of only selectively by @GiaJordan in #1363
• Linted visualization module by @andrewelamb in #1359
• Move remained logic in routes.py related to manifest generator to library by @linglp in #1351
• Update parallel_map to parallel_applymap by @mialy-defelice in #1372
• Schematic v24.2.1 by @GiaJordan in #1376
• Make api packages optional by @andrewelamb in #1362
• Linted utils module by @andrewelamb in #1361
• Ran black on all modules by @andrewelamb in #1355
• Update synapse client minimum version to 3.2.0 by @GiaJordan in #1345

Full Changelog: v24.1.1...v24.2.1

Schematic Release (v24.1.1)

What's Changed

We have added a Data Model Validator that is currently run during the convert step to check that some key assumptions about the data model are fulfilled, and that is converted properly to the graph network. We have plans to expand the validator in the future and allow it to be run directly.
Users can now supply the CSV data model itself as the data model, instead of the JSONLD.
Breaking Changes: Users will be expected to regenerate their JSON-LD using the new refactor release. Previously generated JSON-LDs will not currently work with the refactored schematic.
Other Changes:

• Schema Refactor: Unit Tests for TestDataModelGraph by @mialy-defelice in #1309
• Tests for DataModelRelationships class by @andrewelamb in #1293
• Schema Refactor: Develop unit tests DataModelJsonSchema by @mialy-defelice in #1307
• Schema Refactor: Unit Tests for DataModelJSONLDParser by @mialy-defelice in #1308
• Refactor Schemas: Unit tests for DataModelJSONLD FDS-1064 by @mialy-defelice in #1313
• Refactor Schemas: Response to review comments to original PR by @mialy-defelice in #1315
• Schema Refactor: Unit Tests for DataModelParser by @mialy-defelice in #1304
• Refactor Schemas: unit tests FDS-1061 TestDataModelNodes by @mialy-defelice in #1310
• Schema Refactor: Tests for Data Model CSV Parser by @mialy-defelice in #1305
• Schema Refactor: Create tests for DataModelEdges by @GiaJordan in #1306
• Changes to address JSONLD processing by @mialy-defelice in #1321
• Revert changes related to workflow by @linglp in #1328
• Refactor Schemas: Initial Release BugFix, incorrect name causing issues with Manifest generation -- FDS-1442 by @mialy-defelice in #1338
• Revert "Refactor Schemas: Initial Release BugFix, incorrect name causing issues with Manifest generation -- FDS-1442" by @mialy-defelice in #1343
• Re-add change to change parentOf to subClassOf by @mialy-defelice in #1344
• Refactor Schemas: JSONLD Data Model Parsing, change from using labels to display names, merge develop by @mialy-defelice in #1348
• Refactor Schemas - Latest with Fixes. by @mialy-defelice in #1350
• Schematic Schemas Refactor: DM Parser by @mialy-defelice in #1126
• Schematic release v24.1.1 by @linglp in #1354
  Bug Fixes:
• Bug fix: Added missing import statements by @linglp in #1327
• Bug Fix: Fix issue with trailing comma creating an empty node by @mialy-defelice in #1329
• Bug fix: Fixed tests in test_api.py by @linglp in #1352

Full Changelog: v23.12.1...v24.1.1

Schematic release (v23.12.1)

What's Changed

Bug fixes

• Update dockerfile poetry versions FDS-1325 by @GiaJordan in #1325
• Updated poetry version when building schematic docker image for AWS deployment by @linglp in #1326
• Fixed typing that API uses by @andrewelamb in #1332
• Added fit=true parameter when setting a dataframe by @linglp in #1318
• Resolved issues with cross-manifest validation that occured when a manifest was only a single row by @GiaJordan in #1337
• Resolved an issue where a visualization endpoint intermittently returned the wrong information by @GiaJordan in #1336
• Make access token optional to fix CLI manifest submission by @mialy-defelice in #1340
• Remove great_expectations/expectations/Manifest_test_suite.json before running validation tests by @linglp in #1342

Features

• Reduce execution time for Test Schematic workflow by @GiaJordan in #1320

Others

• Move methods of creating multiple manifests from API to manifest generator by @linglp in #1333
• Update deprecated GX v0.15.x functions by @GiaJordan in #1335
• Update manifest unit test for new manifest behavior by @GiaJordan in #1334
• Update pyproject.toml notation by @GiaJordan in #1331

Full Changelog: v23.11.1...v23.12.1