-
Notifications
You must be signed in to change notification settings - Fork 2
[IBCDPE-1110] Add initial DUO templates + Python code to apply DUO bindings. #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
thomasyu888
wants to merge
18
commits into
Sage-Bionetworks:main
Choose a base branch
from
thomasyu888:main
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
93fef33
Restructure repo
thomasyu888 2d705fa
Add code
thomasyu888 8ab8988
Revise
thomasyu888 68577bc
Add contributing
thomasyu888 db1a625
Add else
thomasyu888 360acdb
Update templates/elite_study1.json
thomasyu888 68f7c10
Add changes
thomasyu888 9a2e8d5
Merge branch 'main' of github.com:thomasyu888/governanceDUO
thomasyu888 22edc5d
ADd python code
thomasyu888 9727610
Update code to be runnable
thomasyu888 ffc7dda
null
thomasyu888 af15854
Remove else
thomasyu888 6aa4126
Add elite long json example
thomasyu888 41b1af7
Update
thomasyu888 7d3dc22
Add example
thomasyu888 ed3a749
Add to readme
thomasyu888 d0fa4ef
Add ARs
thomasyu888 ec0ec8e
Don't use staging
thomasyu888 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Contributing | ||
|
|
||
| To add to this repository, please follow these steps: | ||
|
|
||
| 1. Determine a folder or project with an access requirement need | ||
| 2. Create an AR with `subjectsDefinedByAnnotations` checked | ||
| 3. Determine combination of DUO codes and add the template to the template folder here | ||
| 4. Add to the config.yml with the synapse id of a folder or project linked to the template you just added in step 3. | ||
| 5. TODO: the code will execute and bind the schema specificed in step 3 to the project specified in step 4. |
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,26 @@ | ||
| The Data Use Ontology (DUO) provides a helpful framework for gating access to data managed by Sage Bionetworks on the Synapse platform. | ||
|
|
||
| [DUO was developed by members of the Global Alliance for Genomic Health (GA4GH)](https://github.com/EBISPOT/DUO/blob/master/README.md): "DUO allows [users] to semantically tag datasets with restriction about their usage, making them discoverable automatically based on the authorization level of users, or intended usage". | ||
|
|
||
| At Sage, we extended DUO modifiers for our use cases and incorporated [derived annotations](https://sagebionetworks.jira.com/wiki/spaces/PLFM/pages/2597617665/API+Changes+to+support+Extension+of+Data+Access+Management+to+Users+outside+of+Sage+ACT) as a way of scaling governance support on projects by assigning access requirements (ARs)* to entities based on its DUO annotation. | ||
|
|
||
| _*ARs are applied in the form of a clickwrap (i.e., the user must agree to terms) and/or a managed access requirement (i.e., the user must provide evidence). Managed ARs may require evidence in the form of **Authentication** (e.g., training certification, profile validation, two-factor authorization) and/or **Authorization** (e.g., intended data use (IDU) statement, data use certificate (DUC), ethics approval letter from an institutional review board (IRB) or independent ethics committee (IEC))._ | ||
|
|
||
|
|
||
| Learn more about [metadata structures](metadata_structure.md) | ||
|
|
||
| ## PoC | ||
|
|
||
| The `add_duo_schema_bindings.py` is a proof-of-concept for how to add DUO schema bindings to Synapse projects. The script reads a configuration file (`config.yml`) that specifies the Synapse project ID and the DUO schema ID to bind to that project. The script then binds the specified DUO schema to the specified Synapse project or folder. | ||
|
|
||
| What's expected is the annotation key "duoCodes" and the codes can be found here: https://github.com/mc2-center/data-models/blob/main/modules/shared/duo.csv. | ||
|
|
||
|
|
||
| # Resources | ||
| - [EBISPOT DUO](https://github.com/EBISPOT/DUO/blob/master/README.md) | ||
| - [Extension of Data Access Management](https://sagebionetworks.jira.com/wiki/spaces/PLFM/pages/2597617665/API+Changes+to+support+Extension+of+Data+Access+Management+to+Users+outside+of+Sage+ACT) | ||
|
|
||
|
|
||
| # Publications | ||
| - https://doi.org/10.1016/j.xgen.2023.100381 | ||
| - https://www.semantic-web-journal.net/system/files/swj3583.pdf |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| import asyncio | ||
| from synapseclient.models import Project, File | ||
| import synapseclient | ||
| import json | ||
| import os | ||
|
|
||
| import yaml | ||
|
|
||
|
|
||
| def main(): | ||
| """Invoke adding DUO schema bindings""" | ||
| syn = synapseclient.Synapse() | ||
| # syn.setEndpoints(**synapseclient.client.STAGING_ENDPOINTS) | ||
| syn.login() | ||
| with open("config.yml", "r") as f: | ||
| config = yaml.safe_load(f) | ||
|
|
||
| for configuration in config: | ||
| syn.get_available_services() | ||
| js = syn.service("json_schema") | ||
| my_org = js.JsonSchemaOrganization("governance.schemas") | ||
| my_org.create() | ||
| with open(configuration['filepath'], 'r') as f: | ||
| example_schema = json.load(f) | ||
| # Create a new JSON schema version for an existing organization | ||
| schema_name = os.path.basename(configuration['filepath']).replace(".json", '').replace("_", '.') | ||
| # TODO: need to figure out only pushing new schemas | ||
| print(example_schema) | ||
| print(schema_name) | ||
| new_version1 = my_org.create_json_schema(example_schema, schema_name) | ||
| print(new_version1.uri) | ||
| synapse_id = configuration['synapse_id'] | ||
| request_body = { | ||
| "entityId": synapse_id, | ||
| "schema$id": new_version1.uri, | ||
| "enableDerivedAnnotations": True | ||
| } | ||
| syn.restPUT( | ||
| f"/entity/{synapse_id}/schema/binding", body=json.dumps(request_body) | ||
| ) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # - filepath: templates/elite_study1.json | ||
| # synapse_id: syn59199129 | ||
| # - filepath: templates/elite_long_life.json | ||
| # synapse_id: syn234324324 | ||
| - filepath: templates/genie.json | ||
| synapse_id: syn63886516 |
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,58 @@ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "$schema": "http://json-schema.org/draft-07/schema", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "title": "DUO schema", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "$id": "some.project-main-1.3", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "description": "This schema defines how DUO should be used with Some Project.", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "allOf": [ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "if": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "properties": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "duoCodes": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "type": "array", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "items": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "type": "string" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "contains": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "const": "IRB" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "then": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "properties": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "_accessRequirementIds": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "type": "array", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "contains": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "const": 9606538 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Create a fresh set of ARs specific to that project |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "if": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "properties": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "duoCodes": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "type": "array", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "items": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "type": "string" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "contains": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "const": "HMB" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "then": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "properties": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "_accessRequirementIds": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "type": "array", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "contains": { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "const": 9606537 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add another script to select the DUO codes, and to auto generate the json schema template + ARs