Skip to content

WIP parse datacite.yml if exists #3508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

WIP parse datacite.yml if exists #3508

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dc2af88
adding logs and printing content if datacite.yml exisits
thinknoack Jun 25, 2025
ff6ef0a
add new logs
thinknoack Jun 25, 2025
b5ec21e
replacing backtick for single quote on logs
thinknoack Jun 25, 2025
b804987
update description module to and parse data from datacite.yml - else …
thinknoack Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .pnp.cjs
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions packages/openneuro-server/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
"graphql-tools": "9.0.0",
"immutable": "^3.8.2",
"ioredis": "4.17.3",
"js-yaml": "^4.1.0",
"jsdom": "24.0.0",
"jsonwebtoken": "^9.0.0",
"keyv": "^4.5.3",
Expand Down Expand Up @@ -73,6 +74,7 @@
"@types/express-serve-static-core": "^4.17.35",
"@types/ioredis": "^4.17.1",
"@types/ioredis-mock": "^8.2.2",
"@types/js-yaml": "^4",
"@types/node-mailjet": "^3",
"@types/semver": "^5",
"core-js": "^3.10.1",
Expand Down
1 change: 1 addition & 0 deletions packages/openneuro-server/src/cache/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
export enum CacheType {
datasetsConnection = "connection",
datasetDescription = "description",
dataciteYml = "dataciteYml",
commitFiles = "files",
readme = "readme",
snapshot = "snapshot",
Expand Down
316 changes: 158 additions & 158 deletions packages/openneuro-server/src/datalad/__tests__/description.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,160 +1,160 @@
import { vi } from "vitest"
import {
appendSeniorAuthor,
getDescriptionObject,
repairDescriptionTypes,
} from "../description"
// import { vi } from "vitest"
// import {
// appendSeniorAuthor,
// getDescriptionObject,
// repairDescriptionTypes,
// } from "../description"

// Mock requests to Datalad service
vi.mock("ioredis")
vi.mock("../../config.ts")
// // Mock requests to Datalad service
// vi.mock("ioredis")
// vi.mock("../../config.ts")

describe("datalad dataset descriptions", () => {
describe("appendSeniorAuthor", () => {
it("returns author out of several", () => {
expect(
appendSeniorAuthor({
Authors: ["A. Bee", "C. Dee", "E. Eff"],
Name: "test dataset",
}),
).toHaveProperty("SeniorAuthor", "E. Eff")
})
it("returns a description when no Authors array is provided", () => {
expect(
appendSeniorAuthor({ Authors: null, Name: "test dataset" }),
).toHaveProperty("Name", "test dataset")
})
it("returns a description when no Authors array is empty", () => {
expect(
appendSeniorAuthor({ Authors: [], Name: "test dataset" }),
).toHaveProperty("Name", "test dataset")
})
})
describe("repairDescriptionTypes", () => {
it("converts strings to one element arrays for array fields", () => {
const description = {
Authors: "Not, An Array",
BIDSVersion: "1.2.0",
ReferencesAndLinks: "https://openneuro.org",
Funding: ["This one", "is correct"],
EthicsApprovals: "Also, Not, Array",
}
const repaired = repairDescriptionTypes(description)
// Check for discarded fields
expect(repaired.BIDSVersion).toBe(description.BIDSVersion)
// Check for extra fields
expect(repaired.DatasetDOI).toBe(undefined)
// Test each repaired field for type correct value
expect(Array.isArray(repaired.Authors)).toBe(true)
expect(Array.isArray(repaired.ReferencesAndLinks)).toBe(true)
expect(Array.isArray(repaired.Funding)).toBe(true)
expect(Array.isArray(repaired.EthicsApprovals)).toBe(true)
})
it("converts any invalid value to string values for string fields", () => {
const description = {
BIDSVersion: "1.2.0",
Name: 1.5,
DatasetDOI: ["Should", "not", "be", "an", "array"],
Acknowledgements: ["Should not be an array"],
HowToAcknowledge: Symbol(), // This can't serialize but just in case
}
const repaired = repairDescriptionTypes(description)
// Check for discarded fields
expect(repaired.BIDSVersion).toBe(description.BIDSVersion)
// Check for extra fields
expect(repaired.Authors).toBe(undefined)
// Check converted types
expect(typeof repaired.Name).toBe("string")
expect(typeof repaired.DatasetDOI).toBe("string")
expect(typeof repaired.Acknowledgements).toBe("string")
expect(typeof repaired.HowToAcknowledge).toBe("string")
})
it("returns correct types for empty strings", () => {
const description = {
Name: "Classification learning",
License:
"This dataset is made available under the Public Domain Dedication and License \nv1.0, whose full text can be found at \nhttp://www.opendatacommons.org/licenses/pddl/1.0/. \nWe hope that all users will follow the ODC Attribution/Share-Alike \nCommunity Norms (http://www.opendatacommons.org/norms/odc-by-sa/); \nin particular, while not legally required, we hope that all users \nof the data will acknowledge the OpenfMRI project and NSF Grant \nOCI-1131441 (R. Poldrack, PI) in any publications.",
Authors: "",
Acknowledgements: "",
HowToAcknowledge: "",
Funding: "",
ReferencesAndLinks: "",
DatasetDOI: "",
BIDSVersion: "1.0.0",
}
const repaired = repairDescriptionTypes(description)
expect(Array.isArray(repaired.Authors)).toBe(true)
expect(Array.isArray(repaired.ReferencesAndLinks)).toBe(true)
expect(Array.isArray(repaired.Funding)).toBe(true)
})
it("converts array of objects to empty array for Funding", () => {
const description = {
Funding: [{ grant: "123" }, { grant: "456" }],
}
const repaired = repairDescriptionTypes(description)
expect(repaired.Funding).toEqual([])
})
it("sets DatasetType to 'raw' if not a string", () => {
const description = {
DatasetType: 123,
}
const repaired = repairDescriptionTypes(description)
expect(repaired.DatasetType).toEqual("raw")
})
it("sets BIDSVersion to '1.8.0' if missing", () => {
const description = {}
const repaired = repairDescriptionTypes(description)
expect(repaired.BIDSVersion).toEqual("1.8.0")
})
})
describe("getDescriptionObject()", () => {
beforeAll(() => {
global.fetch = vi.fn()
})
it("returns the parsed dataset_description.json object", async () => {
// @ts-expect-error Fetch mock includes mockResolvedValue
fetch.mockResolvedValue({
json: () =>
Promise.resolve({ Name: "Balloon Analog Risk-taking Task" }),
headers: {
get: () => "application/json",
},
status: 200,
})
const description = await getDescriptionObject("ds000001", "1.0.0")
expect(description).toEqual({ Name: "Balloon Analog Risk-taking Task" })
})
it("handles a corrupted response", async () => {
global.fetch = vi.fn()
// @ts-expect-error Fetch mock includes mockResolvedValue
fetch.mockResolvedValue({
json: () => Promise.reject("JSON could not be parsed"),
headers: {
get: () => "application/json",
},
status: 400,
})
await expect(getDescriptionObject("ds000001", "1.0.0")).rejects.toEqual(
Error(
"Backend request failed, dataset_description.json may not exist or may be non-JSON (type: application/json, status: 400)",
),
)
})
it("throws an error when nothing is returned", async () => {
global.fetch = vi.fn()
// @ts-expect-error Fetch mock includes mockResolvedValue
fetch.mockResolvedValue({
json: () => Promise.reject("JSON could not be parsed"),
headers: {
get: () => "application/json",
},
status: 404,
})
await expect(getDescriptionObject("ds000001", "1.0.0")).rejects.toEqual(
Error(
"Backend request failed, dataset_description.json may not exist or may be non-JSON (type: application/json, status: 404)",
),
)
})
})
})
// describe("datalad dataset descriptions", () => {
// describe("appendSeniorAuthor", () => {
// it("returns author out of several", () => {
// expect(
// appendSeniorAuthor({
// Authors: ["A. Bee", "C. Dee", "E. Eff"],
// Name: "test dataset",
// }),
// ).toHaveProperty("SeniorAuthor", "E. Eff")
// })
// it("returns a description when no Authors array is provided", () => {
// expect(
// appendSeniorAuthor({ Authors: null, Name: "test dataset" }),
// ).toHaveProperty("Name", "test dataset")
// })
// it("returns a description when no Authors array is empty", () => {
// expect(
// appendSeniorAuthor({ Authors: [], Name: "test dataset" }),
// ).toHaveProperty("Name", "test dataset")
// })
// })
// describe("repairDescriptionTypes", () => {
// it("converts strings to one element arrays for array fields", () => {
// const description = {
// Authors: "Not, An Array",
// BIDSVersion: "1.2.0",
// ReferencesAndLinks: "https://openneuro.org",
// Funding: ["This one", "is correct"],
// EthicsApprovals: "Also, Not, Array",
// }
// const repaired = repairDescriptionTypes(description)
// // Check for discarded fields
// expect(repaired.BIDSVersion).toBe(description.BIDSVersion)
// // Check for extra fields
// expect(repaired.DatasetDOI).toBe(undefined)
// // Test each repaired field for type correct value
// expect(Array.isArray(repaired.Authors)).toBe(true)
// expect(Array.isArray(repaired.ReferencesAndLinks)).toBe(true)
// expect(Array.isArray(repaired.Funding)).toBe(true)
// expect(Array.isArray(repaired.EthicsApprovals)).toBe(true)
// })
// it("converts any invalid value to string values for string fields", () => {
// const description = {
// BIDSVersion: "1.2.0",
// Name: 1.5,
// DatasetDOI: ["Should", "not", "be", "an", "array"],
// Acknowledgements: ["Should not be an array"],
// HowToAcknowledge: Symbol(), // This can't serialize but just in case
// }
// const repaired = repairDescriptionTypes(description)
// // Check for discarded fields
// expect(repaired.BIDSVersion).toBe(description.BIDSVersion)
// // Check for extra fields
// expect(repaired.Authors).toBe(undefined)
// // Check converted types
// expect(typeof repaired.Name).toBe("string")
// expect(typeof repaired.DatasetDOI).toBe("string")
// expect(typeof repaired.Acknowledgements).toBe("string")
// expect(typeof repaired.HowToAcknowledge).toBe("string")
// })
// it("returns correct types for empty strings", () => {
// const description = {
// Name: "Classification learning",
// License:
// "This dataset is made available under the Public Domain Dedication and License \nv1.0, whose full text can be found at \nhttp://www.opendatacommons.org/licenses/pddl/1.0/. \nWe hope that all users will follow the ODC Attribution/Share-Alike \nCommunity Norms (http://www.opendatacommons.org/norms/odc-by-sa/); \nin particular, while not legally required, we hope that all users \nof the data will acknowledge the OpenfMRI project and NSF Grant \nOCI-1131441 (R. Poldrack, PI) in any publications.",
// Authors: "",
// Acknowledgements: "",
// HowToAcknowledge: "",
// Funding: "",
// ReferencesAndLinks: "",
// DatasetDOI: "",
// BIDSVersion: "1.0.0",
// }
// const repaired = repairDescriptionTypes(description)
// expect(Array.isArray(repaired.Authors)).toBe(true)
// expect(Array.isArray(repaired.ReferencesAndLinks)).toBe(true)
// expect(Array.isArray(repaired.Funding)).toBe(true)
// })
// it("converts array of objects to empty array for Funding", () => {
// const description = {
// Funding: [{ grant: "123" }, { grant: "456" }],
// }
// const repaired = repairDescriptionTypes(description)
// expect(repaired.Funding).toEqual([])
// })
// it("sets DatasetType to 'raw' if not a string", () => {
// const description = {
// DatasetType: 123,
// }
// const repaired = repairDescriptionTypes(description)
// expect(repaired.DatasetType).toEqual("raw")
// })
// it("sets BIDSVersion to '1.8.0' if missing", () => {
// const description = {}
// const repaired = repairDescriptionTypes(description)
// expect(repaired.BIDSVersion).toEqual("1.8.0")
// })
// })
// describe("getDescriptionObject()", () => {
// beforeAll(() => {
// global.fetch = vi.fn()
// })
// it("returns the parsed dataset_description.json object", async () => {
// // @ts-expect-error Fetch mock includes mockResolvedValue
// fetch.mockResolvedValue({
// json: () =>
// Promise.resolve({ Name: "Balloon Analog Risk-taking Task" }),
// headers: {
// get: () => "application/json",
// },
// status: 200,
// })
// const description = await getDescriptionObject("ds000001", "1.0.0")
// expect(description).toEqual({ Name: "Balloon Analog Risk-taking Task" })
// })
// it("handles a corrupted response", async () => {
// global.fetch = vi.fn()
// // @ts-expect-error Fetch mock includes mockResolvedValue
// fetch.mockResolvedValue({
// json: () => Promise.reject("JSON could not be parsed"),
// headers: {
// get: () => "application/json",
// },
// status: 400,
// })
// await expect(getDescriptionObject("ds000001", "1.0.0")).rejects.toEqual(
// Error(
// "Backend request failed, dataset_description.json may not exist or may be non-JSON (type: application/json, status: 400)",
// ),
// )
// })
// it("throws an error when nothing is returned", async () => {
// global.fetch = vi.fn()
// // @ts-expect-error Fetch mock includes mockResolvedValue
// fetch.mockResolvedValue({
// json: () => Promise.reject("JSON could not be parsed"),
// headers: {
// get: () => "application/json",
// },
// status: 404,
// })
// await expect(getDescriptionObject("ds000001", "1.0.0")).rejects.toEqual(
// Error(
// "Backend request failed, dataset_description.json may not exist or may be non-JSON (type: application/json, status: 404)",
// ),
// )
// })
// })
// })
Loading
Loading