Refactor Jinja environment variable handling by MACKAT05 · Pull Request #353 · Snowflake-Labs/schemachange

MACKAT05 · 2025-08-31T04:38:41Z

This pull request introduces a major refactor and enhancement to how local data files are injected into Jinja templates, replacing the legacy JinjaEnvVar approach with the new LocalDataInjection class. It adds new functions for loading CSV, JSON, and YAML files directly into templates, updates documentation, and provides comprehensive demos comparing the new and legacy methods. The changes are backward compatible and include deprecation notices for legacy features.

Local Data Injection Enhancements

Refactored the legacy JinjaEnvVar class into the new LocalDataInjection class, which now provides the env_var() function and introduces from_csv(), from_json(), and from_yaml() functions for loading local data files in Jinja templates. This enables direct access to structured data from CSV, JSON, and YAML files within templates. (CHANGELOG.md [1] README.md [2] [3]
Updated the Jinja template processor and documentation to use LocalDataInjection and its new functions, including detailed usage examples and a dedicated documentation file. (README.md [1] [2] docs/LocalDataInjection.md [3]

Demo and Documentation

Added a comprehensive demo (demo/citibike_demo_jinja_data_injection) showcasing both the new LocalDataInjection approach and the legacy method, including SQL scripts and configuration files that demonstrate loading and processing data from external files. (demo/citibike_demo_jinja_data_injection/1_setup/A__setup.sql [1] 2_test/V1.1.0__initial_database_objects.sql [2] 2_test/V1.1.0__initial_database_objects_legacy.sql [3] and related files)

Deprecation and Backward Compatibility

Deprecated the JinjaEnvVar class in favor of LocalDataInjection, while maintaining backward compatibility for existing templates using env_var(). (CHANGELOG.md CHANGELOG.mdR6-R22)

Configuration Files for Demo

Added example configuration files (file_formats.json, stages.yaml) and their legacy Jinja equivalents to support the demo and illustrate the benefits of loading data from external sources. (file_formats.json [1] stages.yaml [2] file_formats_legacy.j2 [3]

These changes make it much easier and more maintainable to inject local structured data into Jinja templates, improving both developer experience and template flexibility.

MACKAT05 · 2025-09-01T07:13:43Z

I have yet to try to deploy the new example... i have run the render command and inspected the results... the main focus was to show the macro pattern that would inflate structured data instead of hard coded SQL and that the structured data was loading without throwing fits about not being able to find files.

sahil-walia · 2025-09-13T19:53:15Z

README.md


+##### from_csv, from_json, from_yaml
+
+These functions provide access to local data files for use in Jinja templates. For detailed documentation and examples, see [LocalDataInjection.md](docs/LocalDataInjection.md).


The reference to LocalDataInjection.md is invalid. We can remove .md if the file is still valid.

i missed this one. I used AI tooling to help create examples ( without posting sensitive customer/ client data) it unfortunately was very aggressive about littering the repository with .md files and not using the existing markdown file entries.

I think this should at this point be revised to:
For examples, see citibike_demo_jinja_data_injection.

How are you finding this feature branch otherwise?

I have not completed my review. I generally understand what you are doing.

Curious - Was there a scenario you faced that lead to developing this solution? Does it relate to an open PR?

sahil-walia

Thank you so much for contributing.

sahil-walia · 2025-09-14T20:08:07Z

README.md


+##### from_csv, from_json, from_yaml
+
+These functions provide access to local data files for use in Jinja templates. For detailed documentation and examples, see [LocalDataInjection.md](docs/LocalDataInjection.md).


I have not completed my review. I generally understand what you are doing.

Curious - Was there a scenario you faced that lead to developing this solution? Does it relate to an open PR?

sahil-walia · 2025-09-14T20:11:29Z

schemachange/JinjaEnvVar.py

Should we not rename this file from schemachange/JinjaEnvVar.py to JinjaTemplateDataProvider and code from your localdatainjection.py since we are extending the capability?

I think we should not name the file with Injection as it is generally perceived with negative connotation.

I have plans to add additional modules that invoke snowflake connections and connections to other databases. to allow database introspection. I'm not opposed to renaming them. however with that in mind i also had thought it best to keep them as seperate modules to allow the default option to not load them to employ some level surface reduction from the security stand point hence local data injection and remote data injection.

sahil-walia · 2025-09-14T20:12:09Z

tests/test_LocalDataInjection.py

+        expected = "John is 30 years old\nJane is 25 years old\n"
+        assert result == expected
+    finally:
+        os.unlink(csv_path)


nit: add one blank line at the end

sahil-walia · 2025-09-14T20:12:47Z

schemachange/LocalDataInjection.py

+        with open(yaml_path, encoding=encoding) as yamlfile:
+            data = yaml.safe_load(yamlfile)
+
+        return data


nit: add a blank line at the end.

sahil-walia · 2025-09-14T20:14:09Z

schemachange/LocalDataInjection.py

+        return data
+
+    @staticmethod
+    def from_json(


nit: formatting inconsistencies .

Suggested change

def from_json(

def from_json(

file_path: str,

encoding: str = "utf-8"

) -> dict[str, Any] | list[Any]:

sahil-walia · 2025-09-14T20:15:30Z

schemachange/LocalDataInjection.py

+        return data
+
+    @staticmethod
+    def from_yaml(


nit: formatting for consistency similar to from_csv

Suggested change

def from_yaml(

def from_yaml(

file_path: str,

encoding: str = "utf-8"

) -> dict[str, Any] | list[Any]:

sahil-walia · 2025-09-14T20:18:52Z

schemachange/LocalDataInjection.py

+        csv_path = Path(file_path)
+        if not csv_path.exists():
+            raise FileNotFoundError(f"CSV file not found: {file_path}")
+


validation for input delimiter. .

if not isinstance(delimiter, str) or len(delimiter) != 1: raise ValueError("Delimiter must be a single character")

One more to check file is actually a csv

if not csv_path.suffix.lower() == '.csv': raise ValueError(f"File {file_path} is not a CSV file")

what about tsv? i added some additional functionality to check if the file looks like the expected csv and to throw warnings instead of adding a requirement that a file be suffixed with .csv

sfc-gh-tmathew · 2026-02-09T17:45:43Z

Hi @MACKAT05,

Thank you for this comprehensive enhancement to Jinja data injection! The LocalDataInjection class with from_csv(), from_json(), and from_yaml() functions is a great addition that will improve template flexibility.

We've just released 4.3.0, and this PR is targeted for 4.4.0.

Action needed: This PR has merge conflicts with the master branch. Could you please rebase against master to resolve them?

Once rebased, we'll proceed with final review and merge. Looking forward to including this in 4.4.0!

…Injection class. Added functions to load data from CSV, JSON, and YAML files directly into Jinja templates.

…ks added as guardrails

MACKAT05 · 2026-02-19T23:48:18Z

rebased and split commits. i added a couple of updates along the lines of the provided commentary.

sahil-walia reviewed Sep 13, 2025

View reviewed changes

sahil-walia reviewed Sep 14, 2025

View reviewed changes

sfc-gh-tmathew added Under Review This is being discussed without planned changes community-contribution Submitted by community target: 4.3.0 Planned for 4.3.0 release labels Nov 18, 2025

sfc-gh-tmathew added target: 4.4.0 Planned for 4.4.0 release and removed target: 4.3.0 Planned for 4.3.0 release labels Feb 9, 2026

sfc-gh-tmathew mentioned this pull request Feb 9, 2026

Add support for Jinja environment options in schemachange configuration. #227

Open

MACKAT05 added 5 commits February 15, 2026 22:30

Refactor Jinja environment variable handling by introducing LocalData…

a402e03

…Injection class. Added functions to load data from CSV, JSON, and YAML files directly into Jinja templates.

additional csv options and default file size limit set and other chec…

3fa259f

…ks added as guardrails

updated tests and adjusted formatting lints etc...

2d3a315

add Example deployment showing utility of new functionality

c073ba8

add root level documentation

89c741c

MACKAT05 force-pushed the enhance-data-injection branch from 8085e72 to 89c741c Compare February 18, 2026 21:54


		##### from_csv, from_json, from_yaml

		These functions provide access to local data files for use in Jinja templates. For detailed documentation and examples, see [LocalDataInjection.md](docs/LocalDataInjection.md).

-    def from_json(
+    def from_json(
+        file_path: str,
+        encoding: str = "utf-8"
+    ) -> dict[str, Any] | list[Any]:

Conversation

MACKAT05 commented Aug 31, 2025

Uh oh!

MACKAT05 commented Sep 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sahil-walia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfc-gh-tmathew commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MACKAT05 commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sfc-gh-tmathew commented Feb 9, 2026 •

edited

Loading