Skip to content

Allow monitor to track removed files #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 84 commits into
base: remove-command
Choose a base branch
from
Open

Allow monitor to track removed files #91

wants to merge 84 commits into from

Conversation

y-popov
Copy link
Contributor

@y-popov y-popov commented Mar 14, 2025

rk1274 and others added 30 commits February 18, 2025 11:51
@rk1274
Add implentation to remove files and dirs from a set
200ae79
@rk1274
Add implementation to remove metadata and files from iRODS
2d73f4d
@rk1274
Add test for removing hardlink files
ec3ce10
@rk1274
Add implementation for removing hardlinks and dirs from iRODS
45ab258
@rk1274
Add validation to check if files/dirs are in the set
f3c6283
@rk1274
Refactor removeFilesFromIRODS and removeDirsFromIRODS functions
b04ee4e
@rk1274
Fix wrong implementation about deleting hardlink inodes file
e8bfdf2
@rk1274
Add implementation to remove requester from iRODS metadata
b879294
@rk1274
Add extra validation for different user`s sets with the same name WIP
fc55102
@rk1274
Refactor handleSetsAndRequesters
3e418f2
@rk1274
Fix bug when removing a failed file that was successful in a differen…
043d554 
…t set
@rk1274
Update set counts after removal WIP
26fc4be
@y-popov @rk1274
Refactor SetNewCounts to recalculate file counts
ace746b
@rk1274
Combine removeFiles and removeDirs endpoints and add recently removed…
73708fe 
… to status
@rk1274
Implement removeQueue WIP
48dc79d
@rk1274
Finish implentation for removeQueue WIP
25131d3
@rk1274
Finish removeQueue implementation
973a34d
@rk1274
Update status to have a Removal status line when removing
53aa3bb
@rk1274
Handle removal retries
cd1bae3
@rk1274
Refactor removeQueue implementation
18c0cb8
@rk1274
Add handling for when an item is removed from iRODS but fails to be r…
812e3a0 
…emoved from DB and delint
@rk1274
Delint and fix tests
6818231
@rk1274
Refactor classification of a path to use db instead of file system
2b954c6
@rk1274
Modify code for pull request
621627a
@rk1274
Small PR fixes
4b87f5b
@rk1274
Change implementation to carry out iRODS commands through a handler o…
596919f 
…n the server
@rk1274
Store removeReq in db incase server shutdown WIP
540463d
@rk1274
Allow removal to recover if server goes down
345cfbb
@rk1274
Small PR fixes, and optimisation of GetFilesInDir
9ae8992
@rk1274
Move removal functionality from put pkg to remove pkg
7d3b389
@y-popov
Copy link
Contributor Author

y-popov commented Mar 17, 2025

I might have looked through too quickly, but didn't notice where the removals from iRODS get scheduled, nor the tests for that.

We schedule removals whenever the monitor is triggered, just before doing discovery.
We have tests to see that after the monitor is triggered you no longer see the files in a database, but we now added tests to see that files are being removed from iRODS as well.

@rk1274 rk1274 requested a review from sb10 March 18, 2025 08:59
sb10
sb10 requested changes Mar 18, 2025
View reviewed changes
rk1274 added 24 commits March 18, 2025 16:24
@rk1274
Add further tests for discovery coordinator; fix PR comments
9647a51
@rk1274
Fix race condition
5032098
@rk1274
Add logging and ignore unexpected errors when removing dirs
4b37676
@rk1274
Reset removal values when readding a set
606e376
@rk1274
Add check for if a sub bucket exists before trying to remove it
7af56ca
@rk1274
Add check in removeEntry for if the bucket exists before using it
9f43aa9
@rk1274
remove discovered folders from the database during removal of a base …
9a337fb 
…folder
@rk1274
Don't count duplicate files when removing
01bc5a3
@rk1274
Handle removals for files that failed to upload
863c9ab
@rk1274
Do not allow removals of pending files
ae084e9
@rk1274
Remove from failed bucket during removal
e626234
@rk1274
Make removal work on sets created with old code
eb33804
@rk1274
Finalise issues with sets made with old code and finish issues with i…
a785d17 
…ncomplete sets
@rk1274
Delint
a3289b7
@rk1274
Add DirRemovalError struct
8c81f7e
@rk1274
Fix UpdateRemoveRequest for directories
cd136e2
@rk1274
Add DirNotEmptyError struct
e0077c0
@rk1274
Reset sizeRemoved when readding a set
2d5376f
@rk1274
Add functionality to monitor to also track removed files
0a2330c
@rk1274
Implement two-way-sync for directories and improve test
a15341f
@rk1274
Add new flag --monitor-removals to add cmd
b055cfe
@rk1274
Add tests specific to remote removal and update docs
1490e96
@rk1274
Update docs
14723cd
@rk1274
remove unused makeSetComplete
b057b8a
@rk1274 rk1274 force-pushed the remove-command branch 2 times, most recently from 0b06b10 to 01967db Compare April 9, 2025 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rk1274 rk1274 rk1274 left review comments

@sb10 sb10 sb10 requested changes

@mjkw31 mjkw31 mjkw31 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@y-popov @sb10 @mjkw31 @rk1274