Create a Sqlite connector utility

[austinweisgrau]

Should follow #1242

Works with query(), copy(), and the DB sync utilities

[shaunagm]

@austinweisgrau just to clarify, are this and #1243 meant to be tested/reviewed/merged after #1242 is merged? Is that why the tests are failing? Or is that unrelated?

[austinweisgrau]

Yes, the tests in this PR use the refactored code in #1242 so this should only be merged after #1242 is approved and merged.

The tests are failing because I haven't figured out how to get sqlite properly installed and accessible in the github actions test environment, so that's a work in progress. (Tests should be passing on a local setup, it's a github actions specific issue. I think.)

[austinweisgrau]

Although it might be a windows issue specifically since that's the tests that are failing, so that's another avenue to look into

[bmos]

So this is to connect to a local SQL instance and SQL is not installed within the GitHub Actions environment?

I'd suggest an actions step of installing SQL via apt on Linux and winget on Windows. Not. Sure about macOS.

[bmos]

That being said, you might not actually need SQL installed to test the connector properly. There might be a Pytest SQL plugin or something which acts like an SQL instance.

You might want to look at the MySQL connector to see how they're doing it there.

[austinweisgrau]

The test actually is appropriately failing, I realized. Even though sqlite3 comes with python as part of the stdlib, on windows machines and probably other misc architectures, the sqlite3 cli utility isn't available by default.

One of the methods in the connector attempts to use the shell utility through a subprocess, but that won't work if the shell utility isn't installed and on path.

I'm going to modify that behavior to check if the shell utility is available first, and use a fallback behavior if it is not.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
parsons/databases
db_sync.py
table.py
parsons/databases/sqlite
__init__.py
sqlite.py					48-49, 152, 157, 159, 161, 173, 271, 273, 293, 306-309
parsons/etl
etl.py
Project Total

This report was generated by python-coverage-comment-action

[austinweisgrau]

yay

[austinweisgrau]

cool security check @bmos , very helpful

[shaunagm]

This looks good to me although it might make sense to have someone who uses the database connectors more often (like @IanRFerguson or @sharinetmc or @KasiaHinkson) take a quick look to.

The only thing missing from my perspective is docs. The database utilities/connectors are documented in this file: https://github.com/move-coop/parsons/blob/main/docs/databases.rst

You can see what this looks like from the reader's perspective here:
https://move-coop.github.io/parsons/html/stable/databases.html

[austinweisgrau]

This looks good to me although it might make sense to have someone who uses the database connectors more often (like @IanRFerguson or @sharinetmc or @KasiaHinkson) take a quick look to.

The only thing missing from my perspective is docs. The database utilities/connectors are documented in this file: https://github.com/move-coop/parsons/blob/main/docs/databases.rst

You can see what this looks like from the reader's perspective here: https://move-coop.github.io/parsons/html/stable/databases.html

Done!

[shaunagm]

LGTM!