Skip to content

Commit e0a4625

Browse files
omri374claude
omri374
and
claude
committed
docs: fix notebook list, move data-generation doc, link to Presidio
- README: add the blank line before the "Using notebooks" list so it renders as a list (Python-Markdown/Zensical needs it), not a run-on paragraph. - Move the data-generator README into docs/data_generation.md (added to the nav) and turn the in-package README into a pointer to it; update the main README links to the new location. - Add an external nav link to the Presidio docs from the Presidio-Research docs. - zensical_build.py: keep the README's docs/*.md links on-site (strip the leading docs/) instead of absolutising them to GitHub. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 06d2030 commit e0a4625

5 files changed

Lines changed: 102 additions & 84 deletions

File tree

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ It also includes a fake data generator that creates synthetic sentences based on
1414

1515

1616
### Using notebooks
17-
The easiest way to get started is by reviewing the notebooks.
17+
The easiest way to get started is by reviewing the notebooks.
18+
1819
- [Notebook 1](notebooks/1_Generate_data.ipynb): Shows how to use the PII data generator.
1920
- [Notebook 2](notebooks/2_PII_EDA.ipynb): Shows a simple analysis of the PII dataset.
2021
- [Notebook 3](notebooks/3_Split_by_pattern_number.ipynb): Provides tools to split the dataset into train/test/validation sets while avoiding leakage due to the same pattern appearing in multiple folds (only applicable for synthetically generated data).
@@ -67,13 +68,13 @@ Note that some dependencies (such as Flair and Stanza) are no longer supported.
6768

6869
## 1. Data generation
6970

70-
See [Data Generator README](presidio_evaluator/data_generator/README.md) for more details.
71+
See the [Data Generation docs](docs/data_generation.md) for more details.
7172

7273
The data generation process takes a file with templates, e.g. `My name is {{name}}`.
7374
Then, it creates new synthetic sentences by sampling templates and PII values.
7475
Furthermore, it tokenizes the data, creates tags (either IO/BIO/BILUO) and spans for the newly created samples.
7576

76-
- For information on data generation/augmentation, see the data generator [README](presidio_evaluator/data_generator/README.md).
77+
- For information on data generation/augmentation, see the [Data Generation docs](docs/data_generation.md).
7778
- For an example for running the generation process, see [this notebook](notebooks/1_Generate_data.ipynb).
7879
- For an understanding of the underlying fake PII data used, see this [exploratory data analysis notebook](notebooks/2_PII_EDA.ipynb).
7980

‎docs/data_generation.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Data Generation
2+
3+
The `PresidioSentenceFaker` generates sentences from templates (e.g. `my name is {{person}}`) where the placeholders
4+
are replaced with fake PII entities, along with metadata about the spans (the start and end of each entity) for model training and evaluation.
5+
6+
## Scenarios
7+
8+
There are two main scenarios for using the `PresidioSentenceFaker`:
9+
10+
1. Create a fake dataset for evaluation or training purposes, given a list of predefined templates
11+
(uses [this file](https://github.com/data-privacy-stack/presidio-research/blob/main/presidio_evaluator/data_generator/raw_data/templates.txt) by default)
12+
2. Augment an existing labeled dataset with additional fake values.
13+
14+
In both scenarios the process is similar. In scenario 2, the existing dataset is first translated into templates,
15+
and then scenario 1 is applied.
16+
17+
## Process
18+
19+
This generator heavily relies on the [Faker package](https://www.github.com/joke2k/faker) with a few differences:
20+
21+
1. `PresidioSentenceFaker` returns not only fake text, but also the spans in which fake entities appear in the text.
22+
2. `Faker` samples each value independently.
23+
In many cases, we would want to keep the semantic dependency between two values.
24+
For example, for the template `My name is {{name}} and my email is {{email}}`,
25+
we would prefer a result which has the name within the email address,
26+
such as `My name is Mike and my email is mike1243@gmail.com`.
27+
For this functionality, a new `RecordGenerator` (based on Faker's `Generator` class) is implemented.
28+
It accepts a dictionary / pandas DataFrame, and favors returning objects from the same record (if possible).
29+
30+
## Example
31+
32+
For a full example, see the [Generate Data notebook](notebooks/1_Generate_data.ipynb).
33+
34+
`PresidioSentenceFaker` provides a high-level interface for using the full power of the `presidio_evaluator`
35+
package. Its results use the presidio PII entities, not the `Faker` entities.
36+
It is loaded by default with template strings, and the additional Presidio Entity Providers.
37+
38+
```python
39+
from presidio_evaluator.data_generator import PresidioSentenceFaker
40+
41+
record_generator = PresidioSentenceFaker(locale='en', lower_case_ratio=0.05)
42+
fake_records = record_generator.generate_new_fake_sentences(1500)
43+
44+
# Print the spans of the first sample
45+
print(fake_records[0].fake)
46+
print(fake_records[0].spans)
47+
```
48+
49+
The process at a high level is the following:
50+
51+
1. Translate a NER dataset (e.g. CONLL or OntoNotes) into a list of
52+
templates: `My name is John` -> `My name is [PERSON]`
53+
2. Construct a `PresidioSentenceFaker` instance by:
54+
- Choosing your appropriate locale, e.g. `en_US`
55+
- Choosing the lower case ratio
56+
- Passing in your list of templates (or default to those provided)
57+
- Optionally extend with provided templates accessible via `from presidio_evaluator.data_generator import presidio_templates_file_path`
58+
- Passing in any custom entity providers (or default to those provided)
59+
- Optionally extend with inbuilt presidio entity providers accessible via `from presidio_evaluator.data_generator import presidio_additional_entity_providers`
60+
- Adding a mapping from the output provider entity type to a Presidio recognized entity type where appropriate
61+
- e.g. For a `TownProvider` which outputs entity type of `town`, execute `PresidioSentenceFaker.ENTITY_TYPE_MAPPING['town'] = 'GPE'`)
62+
- Passing in a DataFrame representing your underlying PII records (or default to those provided)
63+
- Optionally extend with inbuilt presidio entity providers accessible via `from presidio_evaluator.data_generator.faker_extensions.datasets import load_fake_person_df`
64+
- Adding any additional aliases required by your dataset by adding to `PresidioSentenceFaker.PROVIDER_ALIASES`
65+
- e.g. if the entity providers support "name" but your dataset templates contain "person", you can add this alias
66+
with `PresidioSentenceFaker.PROVIDER_ALIASES['name'] = 'person'`)
67+
3. Generate sentences
68+
4. Split the generated dataset into train/test/validation while making sure
69+
that samples from the same template would only appear in one set
70+
5. Adapt datasets for the various models (Spacy, Flair, CRF, sklearn)
71+
6. Train models
72+
7. Evaluate using one of the [evaluation notebooks](https://github.com/data-privacy-stack/presidio-research/tree/main/notebooks/models)
73+
74+
Notes:
75+
76+
- For steps 5, 6, 7 see the [home page](index.md).
77+
78+
79+
*Copyright notice:*
80+
81+
Fake Name Generator identities by the Fake Name Generator are licensed under a
82+
Creative Commons Attribution-Share Alike 3.0 United States License.
83+
Fake Name Generator and the Fake Name Generator logo
84+
are trademarks of Corban Works, LLC.

‎mkdocs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ nav:
2020
- Evaluate Presidio Analyzer: notebooks/4_Evaluate_Presidio_Analyzer.ipynb
2121
- Evaluate a custom Analyzer: notebooks/5_Evaluate_Custom_Presidio_Analyzer.ipynb
2222
- Interactive entity mapping: notebooks/6_Interactive_Entity_Mapping.ipynb
23+
- Data generation: data_generation.md
2324
- Concepts:
2425
- Evaluation: evaluation.md
2526
- Token evaluation: token_evaluation.md
@@ -29,6 +30,7 @@ nav:
2930
- Why canonical entity mapping: why_canonical_entity_mapping.md
3031
- Mapping scenarios: mapping_scenarios.md
3132
- Migration guide: migration-guide.md
33+
- Presidio ↗: https://presidio.dataprivacystack.org
3234

3335
theme:
3436
name: material

‎presidio_evaluator/data_generator/README.md‎

Lines changed: 5 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -1,84 +1,9 @@
11
# Data Generation
22

3-
The `PresidioSentenceFaker` generates sentences from templates (e.g. `my name is {{person}}`) where the placeholders
4-
are replaced with fake PII entities, along with metadata about the spans (the start and end of each entity) for model training and evaluation.
3+
📖 The data generation documentation now lives in the Presidio-Research docs:
54

6-
## Scenarios
5+
- **Online:** <https://presidio-research.dataprivacystack.org/data_generation/>
6+
- **Source:** [`docs/data_generation.md`](../../docs/data_generation.md)
77

8-
There are two main scenarios for using the `PresidioSentenceFaker`:
9-
10-
1. Create a fake dataset for evaluation or training purposes, given a list of predefined templates
11-
(uses [this file](raw_data/templates.txt) by default)
12-
2. Augment an existing labeled dataset with additional fake values.
13-
14-
In both scenarios the process is similar. In scenario 2, the existing dataset is first translated into templates,
15-
and then scenario 1 is applied.
16-
17-
## Process
18-
19-
This generator heavily relies on the [Faker package](https://www.github.com/joke2k/faker) with a few differences:
20-
21-
1. `PresidioSentenceFaker` returns not only fake text, but also the spans in which fake entities appear in the text.
22-
2. `Faker` samples each value independently.
23-
In many cases, we would want to keep the semantic dependency between two values.
24-
For example, for the template `My name is {{name}} and my email is {{email}}`,
25-
we would prefer a result which has the name within the email address,
26-
such as `My name is Mike and my email is mike1243@gmail.com`.
27-
For this functionality, a new `RecordGenerator` (based on Faker's `Generator` class) is implemented.
28-
It accepts a dictionary / pandas DataFrame, and favors returning objects from the same record (if possible).
29-
30-
## Example
31-
32-
For a full example, see the [Generate Data Notebook](../../notebooks/1_Generate_data.ipynb).
33-
34-
`PresidioSentenceFaker` provides a high-level interface for using the full power of the `presidio_evaluator`
35-
package. Its results use the presidio PII entities, not the `Faker` entities.
36-
It is loaded by default with template strings, and the additional Presidio Entity Providers.
37-
38-
```python
39-
from presidio_evaluator.data_generator import PresidioSentenceFaker
40-
41-
record_generator = PresidioSentenceFaker(locale='en', lower_case_ratio=0.05)
42-
fake_records = record_generator.generate_new_fake_sentences(1500)
43-
44-
# Print the spans of the first sample
45-
print(fake_records[0].fake)
46-
print(fake_records[0].spans)
47-
```
48-
49-
The process at a high level is the following:
50-
51-
1. Translate a NER dataset (e.g. CONLL or OntoNotes) into a list of
52-
templates: `My name is John` -> `My name is [PERSON]`
53-
2. Construct a `PresidioSentenceFaker` instance by:
54-
- Choosing your appropriate locale, e.g. `en_US`
55-
- Choosing the lower case ratio
56-
- Passing in your list of templates (or default to those provided)
57-
- Optionally extend with provided templates accessible via `from presidio_evaluator.data_generator import presidio_templates_file_path`
58-
- Passing in any custom entity providers (or default to those provided)
59-
- Optionally extend with inbuilt presidio entity providers accessible via `from presidio_evaluator.data_generator import presidio_additional_entity_providers`
60-
- Adding a mapping from the output provider entity type to a Presidio recognized entity type where appropriate
61-
- e.g. For a `TownProvider` which outputs entity type of `town`, execute `PresidioSentenceFaker.ENTITY_TYPE_MAPPING['town'] = 'GPE'`)
62-
- Passing in a DataFrame representing your underlying PII records (or default to those provided)
63-
- Optionally extend with inbuilt presidio entity providers accessible via `from presidio_evaluator.data_generator.faker_extensions.datasets import load_fake_person_df`
64-
- Adding any additional aliases required by your dataset by adding to `PresidioSentenceFaker.PROVIDER_ALIASES`
65-
- e.g. if the entity providers support "name" but your dataset templates contain "person", you can add this alias
66-
with `PresidioSentenceFaker.PROVIDER_ALIASES['name'] = 'person'`)
67-
3. Generate sentences
68-
4. Split the generated dataset into train/test/validation while making sure
69-
that samples from the same template would only appear in one set
70-
5. Adapt datasets for the various models (Spacy, Flair, CRF, sklearn)
71-
6. Train models
72-
7. Evaluate using one of the [evaluation notebooks](../../notebooks/models)
73-
74-
Notes:
75-
76-
- For steps 5, 6, 7 see the main [README](../../README.md).
77-
78-
79-
*Copyright notice:*
80-
81-
Fake Name Generator identities by the Fake Name Generator are licensed under a
82-
Creative Commons Attribution-Share Alike 3.0 United States License.
83-
Fake Name Generator and the Fake Name Generator logo
84-
are trademarks of Corban Works, LLC.
8+
It covers the `PresidioSentenceFaker` scenarios, the generation process, and a
9+
full end-to-end example.

‎scripts/zensical_build.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,9 +155,15 @@ def repl(match: re.Match) -> str:
155155
if url.startswith(("http://", "https://", "//", "mailto:", "#")):
156156
return match.group(0)
157157
path, _, frag = url.partition("#")
158+
clean_path = path.lstrip("./")
158159
# Notebook links become on-site pages; leave them for the .ipynb pass.
159-
if path.lstrip("./") in notebook_rels:
160+
if clean_path in notebook_rels:
160161
return match.group(0)
162+
# Links into docs/ resolve to on-site pages: the docs tree is the site
163+
# root, so drop the leading ``docs/`` and keep the link relative.
164+
if clean_path.startswith("docs/") and clean_path.endswith(".md"):
165+
rel = clean_path[len("docs/") :]
166+
return f"{prefix}{rel}" + (f"#{frag}" if frag else "")
161167
is_image = prefix.startswith("!") or prefix.startswith(("src=",))
162168
base = GH_RAW if is_image else GH_BLOB
163169
clean = path.lstrip("./")

0 commit comments

Comments
 (0)