Skip to content

Commit 99ff443

Browse files
committed
docs: Add project typology
1 parent f68f77e commit 99ff443

2 files changed

Lines changed: 42 additions & 14 deletions

File tree

README.md

Lines changed: 1 addition & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -27,18 +27,7 @@ Some current possibilities with job templating/decoration:
2727

2828
**The transformations and source/destination types may be used completely independently of the registry/job templating.** The registry and job templating functionality are highly dependent on one another.
2929

30-
31-
One powerful way of using kiba-extend is to create an "abstract" ETL project.
32-
An abstract project handles the general logic of transforming data from a specific source system into the format required by a given target system.
33-
For example, if you frequently need to migrate data from OldSystem to NewSystem, you may create an abstract OldSystem kiba-extend project that can handle the general structure of data out of OldSystem and its transformation:what the source data files are, hardcoded enum values that need to be replaced in the data, what preprocessing needs to be done, how to merge data from lookup tables into the records using the lookups, and remapping the data into the "shape" you need it to be in for NewSystem.
34-
35-
All the specifics that may change per specific instance of such a project are defined as configuration settings in the abstract project.
36-
For instance one OldSystem user may only want to migrate records with `active=true` values to NewSystem, while another may wish to also migrate all records regardless of `active` status.
37-
38-
You would create a new kiba-extend project for each of these clients.
39-
These client projects would have your abstract project as a dependency.
40-
This is where you would set the per-project configuration settings you defined in the abstract project.
41-
You can also define client-specific jobs and transforms here as needed.
30+
See [the typology of `kiba-extend` projects](https://lyrasis.github.io/kiba-extend/file.kiba_extend_concepts.html#projecttypes) for a description of how you can build configurable parent projects with child projects having specific configurations.
4231

4332
On the to-do list:
4433

doc/kiba_extend_concepts.md

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,50 @@
88

99
## Glossary {#glossary}
1010

11-
Full job key
12-
: A Ruby Symbol built from `FileRegistry` namespace (if used) and registry entry name, separated by the value of `Kiba::Extend.registry_namespace_separator` (defaults to `__` (two underscores)). Examples: With namespace: `:namespace__entry_name`; No namespace: `:unnamespaced_entry_name`.
1311
File registry key
1412
: Synonym for `Full job key`.
1513

14+
Full job key
15+
: A Ruby Symbol built from `FileRegistry` namespace (if used) and registry entry name, separated by the value of `Kiba::Extend.registry_namespace_separator` (defaults to `__` (two underscores)). Examples: With namespace: `:namespace__entry_name`; No namespace: `:unnamespaced_entry_name`.
16+
17+
Job definition module
18+
: Ruby Module in your project that defines: (1) source, destination, and optional lookups for the job; and (2) the data transformations for the job. See [Kiba::Extend::Jobs](https://lyrasis.github.io/kiba-extend/Kiba/Extend/Jobs.html).
19+
20+
## Typology of kiba-extend projects {#projecttypes}
21+
22+
### Oneoff/direct
23+
24+
This project uses kiba-extend directly and is self-contained. [kiba-extend-project](https://github.com/lyrasis/kiba-extend-project) is an example of a oneoff/direct project.
25+
26+
### Parent
27+
28+
One powerful way of using kiba-extend is to create an "abstract" parent ETL project.
29+
30+
A parent project handles the general logic of transforming data from a specific source system into the format required by a given target system.
31+
For example, if you frequently need to migrate data from OldSystem to NewSystem, you may create a parent OldSystem kiba-extend project that can handle the general structure of data out of OldSystem and its transformation, such as:
32+
33+
- what the source data files are;
34+
- hardcoded enum values that need to be replaced in the data;
35+
- what standard data preprocessing needs to be done;
36+
- how to merge data from lookup tables; and
37+
- remapping the data into the "shape" you need it to be in for NewSystem.
38+
39+
All the specifics that may change per specific instance of such a project are defined as configuration settings in the parent project.
40+
For instance one OldSystem user may only want to migrate records with `active=true` values to NewSystem, while another may wish to also migrate all records regardless of `active` status.
41+
42+
In this case, you might define a `migrate_inactive` setting in the parent project.
43+
44+
### Child
45+
46+
A project for a specific client or data set, based on a Parent project
47+
48+
You set the client-specific values for the configuration settings you defined in the parent project (such as `migrate_inactive`) for each client in the child projects.
49+
50+
Client-specific jobs and transforms can also be defined in child projects as needed.
51+
52+
Theoretically, you can also have grandchild projects and even deeper projects.
53+
However, those would get pretty difficult to understand and manage.
54+
1655
## Assumptions and concepts used in libraries used by `kiba-extend` {#other-libraries}
1756

1857
### Folder structure, file names, module/class name constants defined in files {#zeitwerk}

0 commit comments

Comments
 (0)