Skip to content

artefactual-sdps/preprocessing-sfa

BranchesTags

Repository files navigation

preprocessing-sfa

preprocessing-sfa provides two Enduro child workflows for SFA SIPs: a preprocessing child workflow and an AIS poststorage child workflow. Despite the project name, the worker binary starts two independent Temporal workers, one for each child workflow.

Configuration

The worker binary starts a preprocessing Temporal worker and an AIS poststorage Temporal worker. They need to share the filesystem with Enduro's a3m or Archivematica workers, connect to the same Temporal server, and be related to Enduro with the correct namespace, task queue and workflow names.

Worker configuration

An example configuration for the worker binary:

debug = false
verbosity = 0

sharedPath = "/home/preprocessing/shared"
checkDuplicates = false

[persistence]
dsn = "user:password@tcp(mysql.enduro-sdps:3306)/preprocessing_sfa"
driver = "mysql"
migrate = true

[temporal]
address = "temporal-frontend.enduro-sdps:7233"
namespace = "default"
taskQueue = "preprocessing"
workflowName = "preprocessing"

[worker]
maxConcurrentSessions = 1

[bagit]
checksumAlgorithm = "md5"

[apis]
enabled = true
url = "http://apis-mock.enduro-sdps:8080"
timeout = "10s"
pollInterval = "1s"
token = "mock-token"

[apis.oidc]
enabled = false
providerURL = "http://keycloak:7470/realms/artefactual"
tokenURL = ""
clientID = "enduro-s2s"
clientSecret = "uSh7f2r4j2U5wA9d7mJ3xP6nQ8cT1vL0"
scopes = ""
audience = ""
tokenExpiryLeeway = "30s"
retryMaxAttempts = 3
retryInitialInterval = "500ms"
retryMaxInterval = "2s"
retryBackoffCoefficient = 2.0

[ais]
workingDir = "/tmp"

[ais.temporal]
address = "temporal-frontend.enduro-sdps:7233"
namespace = "default"
taskQueue = "ais"
workflowName = "ais"

[ais.worker]
maxConcurrentSessions = 1

[ais.amss]
url = "http://ambox.enduro-sdps:64081"
user = "test"
key = "test"

[ais.bucket]
endpoint = "http://minio.enduro-sdps:9000"
pathStyle = true
accessKey = "minio"
secretKey = "minio123"
region = "us-west-1"
bucket = "ais"

[fileFormat]
allowlistPath = "/home/preprocessing/.config/allowed_file_formats.csv"

[filevalidate.verapdf]
path = "/opt/verapdf/verapdf"

Enduro

The child workflow sections for Enduro's configuration:

[[childWorkflows]]
type = "preprocessing"
namespace = "default"
taskQueue = "preprocessing"
workflowName = "preprocessing"
extract = true
sharedPath = "/home/enduro/preprocessing"

[[childWorkflows]]
type = "poststorage"
namespace = "default"
taskQueue = "ais"
workflowName = "ais"

Local environment

This project provides two child workflows for the Enduro development environment. The supported development workflow is to run tilt up from the Enduro repository and load this repository through Enduro's CHILD_WORKFLOW_PATHS mechanism.

Bring up the Enduro environment by following the Enduro development manual.

Set up

The specific requirements for preprocessing-sfa are:

  • clone this repository as a sibling of the Enduro repository
  • configure CHILD_WORKFLOW_PATHS=../preprocessing-sfa
  • configure MOUNT_PREPROCESSING_VOLUME=true
  • run tilt up from the Enduro repository

All other development workflow details, including .tilt.env, live updates, starting, stopping, and clearing the environment, are documented in Enduro. This repository can also provide local overrides through its own .tilt.env file, including settings such as TRIGGER_MODE_AUTO.

Requirements for development

While we run the services inside a Kubernetes cluster we recomend installing Go and other tools locally to ease the development process.

Makefile

The Makefile provides developer utility scripts via command line make tasks. Running make with no arguments (or make help) prints the help message. Dependencies are downloaded automatically.

Debug mode

The debug mode produces more output, including the commands executed. E.g.:

$ make env DBG_MAKEFILE=1
Makefile:10: ***** starting Makefile for goal(s) "env"
Makefile:11: ***** Fri 10 Nov 2023 11:16:16 AM CET
go env
GO111MODULE=''
GOARCH='amd64'
...

Available activities

Most of the activities documented below belong to the preprocessing child workflow.

Calculate SIP checksum

Part 1 of a 2-part activity around duplicate checking - see also:

Generates and stores a checksum for the entire SIP, so it can be used to check for duplicates

Steps

  • Generate a SHA256 checksum for the incoming package
  • Read SIP name
  • Store SIP name and checksum in the persistence layer (sips table)

Success critera

  • A SHA256 checksum is successfully generate for the SIP
  • the SIP name and generated checksum are stored in the persistence layer

Check for duplicate SIP

Part 2 of a 2-part activity around duplicate checking - see also:

Determines if an identical SIP has previously been ingested

Steps

  • Use the generated checksum from part 1 to search for an existing match in the sips database table
  • If an existing match is found, return a content error for a duplicateSIP and terminate the workflow
  • Else, continue to next activity

Success critera

  • The activity is able to read the generated checksum and the sips database table
  • No matching checksum is found the SIPs database table

Unbag SIP

Extracts the contents of the bag.

Only runs if the SIP is a BagIt bag. If the SIP is not a bag, this activity will not run.

Steps

  • Check if SIP is a bag
  • If yes, extract the contents of the bag for additional ingest processing
  • Else, skip

Success critera

  • Bag is successfully extracted

Identify SIP structure

Determines the SIP type by analyzing the name and distinguishing features of the package, based on eCH-0160 requirements and other internal policies.

Package types include:

  • BornDigitalSIP
  • DigitizedSIP
  • BornDigitalAIP
  • DigitizedAIP

Steps

  • Base type is BornDigitalSIP; assume this is the SIP type unless other conditions are met
  • Check if the package contains a Prozess_Digitalisierung_PREMIS.xml file
    • If yes, it is a Digitized package - either DigitizedSIP or DigitizedAIP
  • Check if the package contains an additional directory
    • If yes, it is a migration AIP - either BornDigitalAIP or DigitizedAIP
  • Compare check results and determine package type

Success criteria

  • Package is successfully identified as one of the 4 supported types

Validate SIP structure

Ensures that the SIP directory structure conforms to eCH-0160 specifications, that no empty directories are included, and that there are no disallowed characters used in file and directory names.

Note: Character restrictions for file and directory names are based on some of the requirements of the tools used by Archivematica during preservation processing - at present, the file name cleanup steps in Archivematica cannot be modified or disabled without forking. To ensure that SFA package metadata matches the content, this validation check ensures that no disallowed characters are included in file or directory names that might be automatically changed once received by Archivematica.

Steps

  • Read SIP type from previous activity
  • Check for presence of content and header directories
  • Check all file and directory names for invalid characters
  • Check for empty directories

Success critera

  • Files and directories only contain valid characters
    • A-Z, a-z, 0-9, or -_.()
  • SIPs contain content and header directories
    • If content type is an AIP, it also contains an additional directory
  • No empty directories are found

Validate SIP name

Ensure that submitted SIPs use the required naming convention for the identified package type.

Steps

  • Read SIP type from previous activity
  • Use regular expression to validate SIP name based on identified type

Success critera

  • SIP follows expected naming convention for package type:
    • BornDigitalSIP: SIP_[YYYYMMDD]_[delivering office]_[reference]
    • DigitizedSIP: SIP_[YYYYMMDD]_Vecteur_[delivering office]_[reference]

Verify SIP manifest

Checks if all files and directories listed in the metadata manifest match those found in the SIP, and that no extra files or directories are found.

Steps

  • Load SIP metadata manifest into memory
  • Parse the manifest contents and return a list of files and directories
  • Parse the SIP and return a list of files and directories
  • Compare lists
  • Return a list of any missing files found in the manifest but not the SIP
  • Return a list of unexpected files found in the SIP but not the manifest

Success critera

  • There is a matching file or directory for every entry found in the metadata.xml (or UpdatedAreldaMetadata.xml) manifest
  • No unexpected files that are not listed in the manifest are found

Verify SIP checksums

Confirms that the checksums included in the metadata manifest match those calculated during validation.

Steps

  • Check if a given file exists in the manifest
  • If yes, calculate a checksum - else skip
  • Compare calculated checksum to manifest checksum

Success critera

  • A checksum calculated using the same algorithm as the one used in the metadata file returns the same value as the one included in the metadata manifest for each file listed

Validate SIP files

Ensures that files included in the SIP are well-formed and match their format specifications.

Steps

  • For PDF/As, use VeraPDF to validate against the PDF/A specification
  • Note: additional format validation checks will be added in the future

Success critera

  • All files pass validation

Validate logical metadata

Ensures that a logical metadata file is included for AIPs being migrated from DIR and validates the file against a PREMIS schema file

Note : this activity uses some custom workflow code and a locally stored copy of the PREMIS schema to run the general temporal activity xmlvalidate.

Steps

  • Read package type from memory
  • If package type is bornDigitalAIP or DigitizedAIP, check for XML file in additional directory
  • If found, validate the XML file against a locally stored copy of the PREMIS schema; fail ingest if any errors are returned

Success critera

  • Logical metadata file is found in the additional directory of the package
  • Logical metadata file validates against PREMIS 3.x schema

Create premis.xml

Generates a PREMIS XML file that captures ingest preservation actions performed by Enduro as PREMIS events for inclusion in the resulting AIP METS file.

NOTE: This activity is broken up into 3 different activity files in /internal/activites:

  • add_premis_agent.go
  • add_premis_event.go
  • add_premisobjects.go

The XML output is then assembled via /internal/premis/premis.go.

Steps

  • Review event details for all successful tasks
  • Create premis.xml file in a new metadata directory
  • Write PREMIS objects to file
  • Write PREMIS events to file
  • Write PREMIS agents to file

Success critera

  • A premis.xml file is successfully generated with ingest events

Restructure SIP

Reorganizes SIP directory structure into a Preservation Information Package (PIP) that the preservation engine (Archivematica) can process.

Steps

  • Check if metadata directory exists, else create a new metadata directory
  • Move the Prozess_Digitalisierung_PREMIS.xml file to the metadata directory
  • For AIPs, move the UpdatedAreldaMetatdata.xml and logical metadata files to the metadata directory
  • Create an objects directory, and in that directory create a sub-directory with the SIP name
  • Delete xsd directory and its contents from header directory
  • Move content directory into the new objects directory
  • Create a new header directory in objects
  • Move the metadata.xml file into the new header directory
  • Delete original top-level directories

Success critera

  • XSD files are removed
  • Restructured package now has objects and metadata directories immediately inside parent container
  • All content for preservation is within the objects directory
  • Enduro-generated PREMIS file is in the metadata directory
  • For Digitized packages, Prozess_Digitalisierung_PREMIS.xml file is in the metadata directory

Create identifiers.json

Extract original UUIDs from the SIP metadata file and add them to an identifiers.json file added to the metadata directory of the package for parsing by the preservation engine

Steps

  • Parse SIP metadata file
  • Extract persistent identifiers and write to memory
  • Convert manifest file paths to the restructured PIP file paths
  • Exclude any files in the manifest that aren't found in the PIP
  • Using extracted identifiers, generate an identifiers.json file that conforms to Archivematica's expectations
  • Move generated file to package metadata directory

Success critera

  • An identifiers.json file is added to the metadata directory of the package
  • UUIDs present in the original SIP metadata are maintained and used by the preservation engine during preservation processing

Other activities

The preprocessing child workflow that invokes the activities listed above (see the preprocessing.go file) also uses a number of other more general Enduro temporal activites, including:

  • archiveextract
  • bagcreate
  • bagvalidate
  • ffvalidate
  • xmlvalidate

The AIS poststorage child workflow uses one custom workflow activity maintained in this repository:

Create AIS metadata bundle

Extracts all relevant metadata from the SIP and resulting AIP and delivers it to the AIS for synchronization.

Steps
  • Generate a new XML document that combines the contents of the two source files (the SIP metadata.xml or UpdatedAreldaMetadata.xml file, and the AIP METS file)
  • ZIP the generated file and deposit it in an ais MinIO bucket
Success criteria
  • Metadata bundle is successfully generated and deposited
  • AIS is able to receive and ingest the metadata

About

preprocessing-sfa is an Enduro preprocessing workflow for SFA SIPs

Resources

Readme
Activity
Custom properties

Stars

1 star

Watchers

8 watching

Forks

1 fork
Report repository

Releases

17 tags

Packages

 
 
 

Contributors

Languages