Skip to content

fix: migrate affected datasets where size numberOfFiles packedSize and numberOfFilesArchived were overwritten #1854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
martin-trajanovski merged 3 commits into from
May 22, 2025
Merged

fix: migrate affected datasets where size numberOfFiles packedSize and numberOfFilesArchived were overwritten #1854

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a545c97
fix: migrate affected datasets where size numberOfFiles packedSize an…
martin-trajanovski Apr 24, 2025
1e24dae
fix the filter in last query
martin-trajanovski Apr 24, 2025
43b1165
Merge branch 'master' into SWAP-4629-fix-affected-dataset-with-size-0
martin-trajanovski May 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 156 additions & 0 deletions migrations/20250423114619-populate-overwritten-datasets-size.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
module.exports = {
async up(db, client) {
await db
.collection("Dataset")
.find({ size: 0 })
.forEach(async (dataset) => {
const origDatablocks = await db
.collection("OrigDatablock")
.find({
datasetId: dataset._id,
})
.toArray();
const origDatablockSizeSum = origDatablocks.reduce(
(acc, origDatablock) => acc + origDatablock.size,
0,
);

const datablocks = await db
.collection("Datablock")
.find({
datasetId: dataset._id,
})
.toArray();

const datablockSizeSum = datablocks.reduce(
(acc, datablock) => acc + datablock.size,
0,
);

const datasetSizeSum = origDatablockSizeSum + datablockSizeSum;
Copy link
Member

@nitrosx nitrosx Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure that this is the way that it is calculated?
I think that the dataset total size is the sum of the size of the files listed in the origDatablock

Copy link
Collaborator Author

@martin-trajanovski martin-trajanovski Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you see this line for example on the creation of origDatablock: https://github.com/SciCatProject/scicat-backend-next/blob/master/src/datasets/datasets.controller.ts#L1945
Or this line in the update: https://github.com/SciCatProject/scicat-backend-next/blob/master/src/datasets/datasets.controller.ts#L2101
We can see that the size it is used and not the sum of the files listed.

Copy link
Member

@nitrosx nitrosx May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. You are correct.
Let's resolve this conversation, and bring the discussion to the collaborators


console.log(
`Updating Dataset (Id: ${dataset._id}) with size =>${datasetSizeSum}<=`,
);
await db.collection("Dataset").updateOne(
{
_id: dataset._id,
},
{
$set: {
size: datasetSizeSum,
},
},
);
});

await db
.collection("Dataset")
.find({ numberOfFiles: 0 })
.forEach(async (dataset) => {
const origDatablocks = await db
.collection("OrigDatablock")
.find({
datasetId: dataset._id,
})
.toArray();
const origDatablockNumberOfFilesSum = origDatablocks.reduce(
(acc, origDatablock) => acc + origDatablock.dataFileList.length,
0,
);

const datablocks = await db
.collection("Datablock")
.find({
datasetId: dataset._id,
})
.toArray();

const datablockNumberOfFilesSum = datablocks.reduce(
(acc, datablock) => acc + datablock.dataFileList.length,
0,
);

const datasetNumberOfFilesSum =
origDatablockNumberOfFilesSum + datablockNumberOfFilesSum;

console.log(
`Updating Dataset (Id: ${dataset._id}) with numberOfFiles =>${datasetNumberOfFilesSum}<=`,
);
await db.collection("Dataset").updateOne(
{
_id: dataset._id,
},
{
$set: {
numberOfFiles: datasetNumberOfFilesSum,
},
},
);
});

await db
.collection("Dataset")
.find({ packedSize: 0 })
.forEach(async (dataset) => {
const datablocks = await db
.collection("Datablock")
.find({
datasetId: dataset._id,
})
.toArray();

const datablockPackedSizeSum = datablocks.reduce(
(acc, datablock) => acc + datablock.packedSize,
0,
);

console.log(
`Updating Dataset (Id: ${dataset._id}) with packedSize =>${datablockPackedSizeSum}<=`,
);
await db.collection("Dataset").updateOne(
{
_id: dataset._id,
},
{
$set: {
packedSize: datablockPackedSizeSum,
},
},
);
});

await db
.collection("Dataset")
.find({ numberOfFilesArchived: 0 })
.forEach(async (dataset) => {
const datablocks = await db
.collection("Datablock")
.find({
datasetId: dataset._id,
})
.toArray();

const datablockNumberOfFilesArchivedSum = datablocks.reduce(
(acc, datablock) => acc + datablock.dataFileList.length,
0,
);

console.log(
`Updating Dataset (Id: ${dataset._id}) with numberOfFilesArchived =>${datablockNumberOfFilesArchivedSum}<=`,
);
await db.collection("Dataset").updateOne(
{
_id: dataset._id,
},
{
$set: {
numberOfFilesArchived: datablockNumberOfFilesArchivedSum,
},
},
);
});
},

async down(db, client) {},
};