Fcar chute

.yardopts

@@ -11,6 +11,7 @@ doc/file_registry_entry.md
 doc/cli.md
 doc/common_patterns_tips_tricks.md
 doc/cookbook.md
+doc/fcar_chute.md
 doc/fcar_recipes.md
 doc/iterative_cleanup.md
 doc/kiba_extend_concepts.md

CHANGELOG.adoc

@@ -61,6 +61,8 @@ https://github.com/lyrasis/kiba-extend/pull/252[PR#252]
 https://github.com/lyrasis/kiba-extend/pull/255[PR#255]
 * :recursive_nuke option to `:pre_job_task_action` setting
 https://github.com/lyrasis/kiba-extend/pull/258[PR#258]
+* [FCAR chute functionality](https://lyrasis.github.io/kiba-extend/file.fcar_chute.html)
+https://github.com/lyrasis/kiba-extend/pull/259[PR#259]
 
 === Bugfixes
 

README.md

@@ -2,21 +2,44 @@
 
 Kiba is a [Data processing & ETL framework for Ruby](https://github.com/thbar/kiba).
 
-kiba-extend is a suite of Kiba extensions useful in transforming and reshaping data. It includes the following:
+kiba-extend is a suite of Kiba extensions useful in transforming and reshaping data.
+It has been developed in the context of performing complex custom data migration projects.
+These projects tend to have the following features, which have informed the funtionality included in kiba-extend:
 
-- An extensive library of abstract, reusable transformations
-- Some custom source and destination types
-- File/job registry support for use in migration projects. This handles repetitive aspects of configuring source, lookup, and destination files, as well as ensures dependency jobs are called to create files created for a given job. Files/jobs may be tagged and run from a project application via Rake tasks
+- Even when the source system is the same for 5 clients, data entry practice and use of modules/fields will vary, often wildly.
+- They can take a long time because they frequently involve (a) a significant level of client instruction on the function of the target system, so they can understand the impact of different data mapping decisions; (b) clients who must make migration mapping decisions by committee; and/or (c) a significant amount of work to render the data functional for the target system.
+- The source system often does not allow the client to get a view of the data that would be required to make migration decisions for that data. It also usually does not provide any way for the client to do any data cleanup or categorization needed to prepare for a migration, other than record-by-record editing.
+- Active use of the source system for critical work, which cannot be put on hold for the entire time it takes us to work with the client to develop their migration. This means we are often developing the migration
+- Most of our clients are not data experts or incredibly technically savvy. They tend to be most comfortable reviewing data in a tabular data format (CSV or Excel file).
+
+It includes the following:
+
+- An extensive library of abstract, reusable data transformations that can be used to create custom transformation jobs
+- Some custom [source](https://lyrasis.github.io/kiba-extend/Kiba/Extend/Sources.html) and [destination](https://lyrasis.github.io/kiba-extend/Kiba/Extend/Destinations.html) types
+- File/job registry support for use in migration projects. This handles repetitive aspects of configuring source, lookup, and destination files, as well as ensures dependency jobs are called to create files created for a given job. Files/jobs may be tagged and run from a project application via Thor tasks
 - Job templating and decoration. No need to repeat the same source/destination setup, requirements running, pre-processing, post-processing, and initial/final transforms over and over again in your ETL code.
+- Support for [iterative cleanup processes](https://lyrasis.github.io/kiba-extend/file.iterative_cleanup.html)
 
 Some current possibilities with job templating/decoration:
 
-- You can turn on "show me!" when you run a job via Rake task, without doing anything in your code.
-- You can similarly turn on "tell me" from the command line, which will have your computer say something when a job is complete---useful for long running jobs.
-- There is a TestingJob that can be used to set up automated tests for sequences of transforms (i.e. job definition xforms/segments)
+- You can turn on "show me!" when you run a job via Thor task, without doing anything in your code. This causes the output to be emitted to STDOUT.
+- You can similarly turn on "tell me" from the command line, which will have your computer say something when a job is complete---useful for long running jobs. **This is currently really annoying for jobs with dependencies, as all dependency job completions will also be announced.**
 
 **The transformations and source/destination types may be used completely independently of the registry/job templating.** The registry and job templating functionality are highly dependent on one another.
 
+
+One powerful way of using kiba-extend is to create an "abstract" ETL project.
+An abstract project handles the general logic of transforming data from a specific source system into the format required by a given target system.
+For example, if you frequently need to migrate data from OldSystem to NewSystem, you may create an abstract OldSystem kiba-extend project that can handle the general structure of data out of OldSystem and its transformation:what the source data files are, hardcoded enum values that need to be replaced in the data, what preprocessing needs to be done, how to merge data from lookup tables into the records using the lookups, and remapping the data into the "shape" you need it to be in for NewSystem.
+
+All the specifics that may change per specific instance of such a project are defined as configuration settings in the abstract project.
+For instance one OldSystem user may only want to migrate records with `active=true` values to NewSystem, while another may wish to also migrate all records regardless of `active` status.
+
+You would create a new kiba-extend project for each of these clients.
+These client projects would have your abstract project as a dependency.
+This is where you would set the per-project configuration settings you defined in the abstract project.
+You can also define client-specific jobs and transforms here as needed.
+
 On the to-do list:
 
 - Wiki documentation for how to use the registry and job templating. In the meantime the best place to get an understanding of this is [kiba-extend-project](https://github.com/lyrasis/kiba-extend-project).
@@ -30,20 +53,14 @@ On the to-do list:
 I'm working to develop this more fully. If there is no documentation for a given transformation here, please refer to the relevant `spec` file for that transformation to see exactly what it does.
 
 ### Specs
-To get a full overview of available transformations and what they do, run `rake spec` from the repo base directory. This will give you the names of all the transformations in `kiba-extend` and brief descriptions of what they do.
+To get a full overview of available transformations and what they do, run `bundle exec rspec` from the repo base directory. This will give you the names of all the transformations in `kiba-extend` and brief descriptions of what they do.
 
 For more clarity about exactly what each transformation does, if it is not described in the documentation yet, check the actual test files in `/spec/kiba/extend/transforms`, which include sample input rows, transformation calls, and the resulting output
 
 ## Example project applications
 
 [kiba-extend-project](https://github.com/lyrasis/kiba-extend-project) is a Github template repository for starting a new ETL project using `kiba-extend`. It is heavily commented in an attempt to explain how things work.
 
-[kiba-tms](https://github.com/lyrasis/kiba-tms/) is a publicly available project not for a specific client. It uses `kiba-extend` to handle most of the data transformations required for a TMS->CollectionSpace migration. It makes heavy use of `dry-configurable` settings and probably ill-advised metaprogramming to account for the fact that every client uses TMS differently and thus basically everything needs to be configurable. [Private, client-specific repos for individual TMS->CollectionSpace migration clients](https://cs.github.com/?scope=org%3Alyrasis&scopeName=lyrasis&q=gem+%27kiba-tms%27) that require `kiba-tms` are set up to define client-specific migration configs, transforms, and jobs.
-
-[mimsy-to-cspace](https://github.com/lyrasis/mimsy-to-cspace) is a publicly available example of `kiba-extend` usage. It was completed before the registry/job templating functions were added, so it only shows how transformations get used. (And it is a good example of how repetitive the code gets without templating)
-
-LYRASIS staff with permissions to private repos can find a number of other project examples using `kiba-extend` [in our organizationrepo list](https://cs.github.com/?scope=org%3Alyrasis&scopeName=lyrasis&q=gem+%27kiba-extend%27)
-
 ## Contributing
 
 Please see [Contributing to `kiba-extend`](https://lyrasis.github.io/kiba-extend/file.contributing.html) for contributor guidelines.

doc/fcar_chute.md

@@ -0,0 +1,192 @@
+<!--
+# @markup markdown
+# @title FCAR chute
+-->
+
+* TOC
+{:toc}
+
+## About FCAR chute {#about}
+
+The FCAR "chute" is an option sequence of FCAR processes that you can configure for your project.
+
+Implementing a chute is particularly useful in the following situations:
+
+- An abstract project where individual client projects will not need to use all the FCAR processes
+- One-off client projects with a lot of FCAR processes, that you might not implement in the final expected order.
+
+### Benefits of setting up an FCAR  chute {#benefits}
+
+One main benefit of the chute is that it gives you a `Kiba::Extend::Fcar.final_merged` method that you can use as the Job definition module `source` for the first job that is going to be based on the results of your FCAR phase. You don't need to change the `source` in this job as you add FCAR processes or change their sequence.
+
+The other main benefit of the chute is that you don't need to hard-code the `source` in the Job definition modules that create the `base_job` output for each FCAR process. Instead you can do something like:
+
+~~~ ruby
+# frozen_string_literal: true
+
+module Project
+  module Jobs
+    module FcarPrep
+      module NameCatPlus
+        module_function
+
+        def job
+          Kiba::Extend::Jobs::Job.new(
+            files: {
+              source:
+              Kiba::Extend::Fcar.previous_merged(Project::NameCatPlus),
+              destination: :fcar_prep__name_cat_plus
+            },
+            transformer: xforms
+          )
+        end
+      end
+    end
+  end
+end
+~~~
+
+For instance, if our project has:
+
+~~~ ruby
+Kiba::Extend::Fcar.config.chute = [
+  ItemCount,
+  NameSplit,
+  NameCatPlus,
+  MiscFields
+]
+~~~
+
+Then, the merged results of the `NameSplit` FCAR process will be used as the source of `Project::Jobs::FcarPrep::NameCatPlus`.
+
+If there are no files recorded in `Project::NameSplit.config.provided_worksheets`, then the `NameSplit` FCAR is still in your chute, but not considered active for your project.
+
+In this case, the merged results of the `ItemCount` FCAR process will be used as the source of `Project::Jobs::FcarPrep::NameCatPlus`.
+
+If `ItemCount` also isn't activated for your project, then the output of the job given as `Kiba::Extend::Fcar.base_source` will be used as the source of `Project::Jobs::FcarPrep::NameCatPlus`.
+
+If you realize that some of the miscellanous fields contain name values that need to be split and categorized, then you can just edit the `chute` setting:
+
+~~~ ruby
+Kiba::Extend::Fcar.config.chute = [
+  ItemCount,
+  MiscFields,
+  NameSplit,
+  NameCatPlus
+]
+~~~
+
+### Limitations of FCAR chute {#limitations}
+
+The Fcar chute feature isn't really that intelligent. It will totally let you skip `NameSplit` without doing anything special, if your name fields are all single-value. But it will also let you do `NameCatPlus` without doing `NameSplit` if you do have messily-delimited multi-value name fields.
+
+It is up to you to document the requirements of your chute sequence and make sure you stick to them when implementing projects. One way you can do this is to define your `chute` as a Hash with comments as the values:
+
+~~~ ruby
+Kiba::Extend::Fcar.config.chute = {
+  "ItemCount" => "standalone; order doesn't really matter",
+  "MiscFields" => "potential dependency of name sequence, if any fields are categorized as containing names",
+  "NameSplit" => "name sequence; optional first step if multivalue name fields present",
+  "NameCatPlus" => "name sequence"
+}
+~~~
+
+Now, if you do `thor fcar chute` you can see all of this info.
+
+`thor fcar processes` will show you only the names of the active FCAR processes for your project.
+
+## How to implement an FCAR chute {#implement}
+
+### In base config of project: Configure `base_source` and `chute` {#baseconfig}
+
+Set the `base_source` and `chute` settings for your project. This should be done in your main or base config for the project, not any individual project-specific config files. Example:
+
+~~~ ruby
+Kiba::Extend::Fcar.config.base_source = :inv_sum__combined
+Kiba::Extend::Fcar.config.chute = {
+  "Itemandboxcount" => "standalone; I'm sure there's a reason "\
+    "this is first but I can't remember what it is",
+  "AgencyMuseumNameCleanup" => "sequence; legal-control",
+  "CollLevelLegalControl" => "sequence; optional step; legal-control",
+  "SiteProjectMapping" => "standalone; client specific, client1",
+  "SiteProjectMapping2" => "standalone; client specific, client2",
+  "NameSplit" => "sequence; name; DEPENDS ON legal-control "\
+    "sequence completion",
+  "NameCatPlus" => "sequence; name",
+  "County" => "standalone",
+  "SiteSplit" => "sequence; site",
+  "Site" => "sequence; site"
+}
+Kiba::Extend::Fcar.config.pending_processes = []
+~~~
+
+### In the config files for each FCAR process: define `merge_job` {#mergejob}
+
+This one is important!
+
+This is the job key of the job that produces the full, final version of the affected data with the FCAR process configured in this file merged back in.
+
+This can be defined as a method, or a dry-configurable setting:
+
+~~~ ruby
+def merge_job = :name_cat_plus__merged
+
+# OR
+
+setting :merge_job, reader: true, default: :name_cat_plus__merged
+~~~
+
+### Dynamically refer to FCAR output in other places {#refer}
+
+To get the job key to be used as a source in the next process in the chute:
+
+~~~ ruby
+Kiba::Extend::Fcar.previous_merged(MyProject::County)
+~~~
+
+Given the example chute above, this will return the `merge_job` job key of the previous completed process in the chute, or, if there are no completed processes prior to the given one, the value of `Kiba::Extend::Fcar.base_source`.
+
+
+To get the final output of the entire chute:
+
+~~~ ruby
+Kiba::Extend::Fcar.final_merged
+~~~
+
+## Using the FCAR chute {#use}
+
+### thor commands {#thor}
+
+`thor fcar chute` displays the entire available chute, with any comments in Hash values.
+
+`thor fcar processes` displays the active FCAR processes for your project, in chute-order.
+
+### Activating an FCAR process for your project {#activate}
+
+An FCAR process in a project's chute is considered active if at least one file is registered in the FCAR process' `provided_worksheets` setting.
+
+But if you haven't produced any worksheets yet, and you try running `thor run job itemandboxcount__worksheet`, you will get an error like:
+
+~~~ bash
+JOB FAILED: Error handling source file dependency for
+itemandboxcount__base_job_cleaned: Cannot find Kiba::Csu::Itemandboxcount in
+configured iterative cleanup chute. If there are not yet any files associated
+with the cleanup, you need to add it to Kiba::Extend::Fcar.pending_processes in
+the project config
+~~~
+
+You need to add the following to your project's main/base config:
+
+### Pending an FCAR process to temporarily activate it {#pend}
+
+~~~ ruby
+Kiba::Extend::Fcar.pending_processes << MyProject::Itemandboxcount
+~~~
+
+Now you can generate the worksheet. Once the worksheet is finalized and registered in your main/base config as shown below, you can remove the line adding this process to `pending_processes`.
+
+~~~ ruby
+MyProject::Itemandboxcount.config.provided_worksheets = [
+  "itemandboxcount_worksheet0.csv"
+]
+~~~

doc/fcar_recipes.md

@@ -3,24 +3,16 @@
 # @title FCAR recipes
 -->
 
-# FCAR (facilitated cleanup and remapping) recipes
-
 This page documents common patterns of client FCAR using kiba-extend's iterative cleanup functionality.
 
 Each recipe has three components:
 
- - Prep/setup job - the structure of the data required as input for the FCAR process, and any transforms that exist to streamline achieving this structure
- - FCAR configuration - A commented version of the configuration Module to include in your project to activate this FCAR
- - Merge job - patterns for merging the FCAR back into the rest of your project
-
-**Table of contents**
+ 1. Prep/setup job - the structure of the data required as input for the FCAR process, and any transforms that exist to streamline achieving this structure
+ 2. FCAR configuration - A commented version of the configuration Module to include in your project to activate this FCAR
+ 3. Merge job - patterns for merging the FCAR back into the rest of your project
 
-- [Review and correction of programmatic value splitting](#split)
-   * [Prep/setup job](#splitprep)
-      + [Normalization job example](#splitprepnorm)
-      + [Prep example](#splitprepprep)
-   * [FCAR configuration](#splitconfig)
-   * [Merge job](#splitmerge)
+* TOC
+{:toc}
 
 ## Review and correction of programmatic value splitting {#split}
 
@@ -124,7 +116,7 @@ module Project
             }
 
             # Set up the splitters you need here
-            transform Fcar::SplitPrep,
+            transform StandardFcar::SplitPrep,
               orig: :normloc,
               splitters: {
                 / *; */ => :semicolon,
@@ -189,7 +181,7 @@ module Project
         Kiba.job_segment do
           mod = bind.receiver
 
-          transform Kiba::Extend::Transforms::Fcar::Helpers::SumCollatedOccurrences,
+          transform Kiba::Extend::Transforms::StandardFcar::Helpers::SumCollatedOccurrences,
             field: :occurrences,
             delim: mod.collation_delim
         end

lib/kiba/extend/command/fcar/chute.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Kiba
+  module Extend
+    module Command
+      module Fcar
+        class Chute
+          def self.call
+            Kiba::Extend::Fcar.chute
+              .map do |mod, comment|
+                formatted_comment = if comment.empty?
+                  nil
+                else
+                  "  #{comment}"
+                end
+                [mod, formatted_comment].compact
+                  .join("\n")
+              end.join("\n")
+          end
+        end
+      end
+    end
+  end
+end

lib/kiba/extend/command/jobs/tagged_and.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Kiba
+  module Extend
+    module Command
+      module Jobs
+        class TaggedAnd
+          def self.call(tags)
+            result = Kiba::Extend::Registry::RegistryEntrySelector.new
+              .tagged_all(tags)
+            return [] if result.empty?
+
+            result
+          end
+        end
+      end
+    end
+  end
+end

lib/kiba/extend/command/jobs/tagged_or.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Kiba
+  module Extend
+    module Command
+      module Jobs
+        class TaggedOr
+          def self.call(tags)
+            result = Kiba::Extend::Registry::RegistryEntrySelector.new
+              .tagged_any(tags)
+            return [] if result.empty?
+
+            result
+          end
+        end
+      end
+    end
+  end
+end