Skip to content

Commit a851fc5

Browse files
fivetran-sahilkhirwalfivetran-sahilkhirwal
authored
feature(connector_sdk): Adding github workflows (#3)
* adding github workflows * minor changes * minor changes * minor changes
1 parent 88117da commit a851fc5

14 files changed

Lines changed: 1333 additions & 0 deletions

‎.flake8‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[flake8]
2+
max-line-length = 99
3+
show-source = true
4+
statistics = false
5+
doctests = true
6+
select = C,E,F,W,B,B950
7+
extend-ignore = E203, E501, B008
8+
exclude = .git,__pycache__,.mypy_cache,.pytest_cache
9+
per-file-ignores =
10+
tests/*:D102,D104,F401

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: Bug Report
2+
description: File a bug report.
3+
title: "[Bug]: "
4+
labels: ["bug", "triage"]
5+
6+
body:
7+
- type: markdown
8+
attributes:
9+
value: |
10+
Thanks for taking the time to fill out this bug report!
11+
- type: input
12+
id: version
13+
attributes:
14+
label: Connector SDK version
15+
description: What version of Connector SDK are you using? You can check the version with 'pip show fivetran-connector-sdk'
16+
placeholder: ex. 1.4.0
17+
validations:
18+
required: false
19+
- type: textarea
20+
id: what-happened
21+
attributes:
22+
label: What happened?
23+
description: Also tell us, what did you expect to happen?
24+
placeholder: Tell us what you see!
25+
value: "A bug happened!"
26+
validations:
27+
required: true
28+
- type: textarea
29+
id: logs
30+
attributes:
31+
label: Relevant log output
32+
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
33+
render: shell
34+
- type: checkboxes
35+
id: terms
36+
attributes:
37+
label: Code of Conduct
38+
description: By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/fivetran/fivetran-csdk-connectors/tree/main/CODE_OF_CONDUCT.md)
39+
options:
40+
- label: I agree to follow this project's Code of Conduct
41+
required: true

‎.github/ISSUE_TEMPLATE/request.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Request
2+
description: Suggest an idea for a Connector SDK project.
3+
title: "[Request]: "
4+
labels: ["bug", "enhancement", "documentation", "question", "duplicate", "invalid", "wontfix"]
5+
6+
body:
7+
- type: markdown
8+
attributes:
9+
value: |
10+
Thanks for taking the time to fill out this Connector SDK request!
11+
- type: input
12+
id: version
13+
attributes:
14+
label: Connector SDK version
15+
description: What version of Connector SDK are you using? You can check the version with 'pip show fivetran-connector-sdk'
16+
placeholder: ex. 1.4.0
17+
validations:
18+
required: false
19+
- type: textarea
20+
id: what-happened
21+
attributes:
22+
label: Is your request related to a problem? Please describe...
23+
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
24+
placeholder: Tell us what you need!
25+
value: "Feature x would be great to help with y."
26+
validations:
27+
required: true
28+
- type: textarea
29+
id: solution
30+
attributes:
31+
label: Describe the solution you'd like
32+
description: A clear and concise description of what you want to happen.
33+
- type: textarea
34+
id: info
35+
attributes:
36+
label: Additional context
37+
description: Add any other context or screenshots about the request here.
38+
- type: checkboxes
39+
id: terms
40+
attributes:
41+
label: Code of Conduct
42+
description: By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/fivetran/fivetran-csdk-connectors/tree/main/CODE_OF_CONDUCT.md)
43+
options:
44+
- label: I agree to follow this project's Code of Conduct
45+
required: true

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
### Jira ticket
2+
Closes `<ADD TICKET LINK HERE, EACH PR MUST BE LINKED TO A JIRA TICKET>`
3+
4+
### Description of Change
5+
`<MENTION A SHORT DESCRIPTION OF YOUR CHANGES HERE>`
6+
7+
### Testing
8+
`<MENTION ABOUT YOUR TESTING DETAILS HERE, ATTACH SCREENSHOTS IF NEEDED (WITHOUT PII)>`
9+
10+
### Checklist
11+
Some tips and links to help validate your PR:
12+
13+
- [ ] Tested the connector with `fivetran debug` command.
14+
- [ ] Added/Updated example-specific README.md file, see [the README template](https://github.com/fivetran/fivetran-csdk-connectors/tree/main/_template_connector/README_template.md) for the required structure and guidelines.
15+
- [ ] Followed Python Coding Standards, [refer here](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/PYTHON_CODING_STANDARDS.md)

‎.github/copilot-instructions.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Fivetran Connector SDK
2+
A public collection of Python examples, templates, and guides for building custom connectors using the Fivetran Connector SDK. It includes ready-to-run example connectors, best-practice patterns, and quickstart examples to help users understand the usage of Fivetran Connector SDK.
3+
4+
## Role
5+
You are an AI code reviewer specialized in Python development for the Fivetran Connector SDK. Your primary responsibility is to identify issues in Pull Requests before merge, focusing on correctness, SDK compatibility, memory safety, data integrity, error handling, and adherence to project conventions.
6+
7+
## Objectives
8+
- Detect SDK v2+ breaking changes and deprecated patterns
9+
- Identify memory safety violations and unbounded data loading
10+
- Verify proper state management and checkpointing logic
11+
- Ensure data integrity through validation of sync logic, primary keys, and type consistency
12+
- Validate error handling, retry logic, and exception management
13+
- Enforce required docstrings, comments, and documentation standards
14+
- Check code quality metrics (complexity, naming, constants)
15+
- Verify configuration validation and logging practices
16+
17+
## Review scope
18+
This file provides high-level guidance for reviewing connector Pull Requests. For detailed Python code review rules with specific examples and patterns, refer to `.github/instructions/python-review.instructions.md`, `.github/instructions/readme-markdown.instructions.md`, and `.github/instructions/configuration-review.instructions.md`.
19+
20+
## Repository context
21+
22+
### Structure
23+
- `_template_connector/` - Canonical template: connector.py, configuration.json, requirements.txt, README_template.md
24+
- `.github/instructions/` - Detailed review instructions for Python, JSON, and Markdown files
25+
- `<connector_name>/` (top-level folders) - Examples for Fivetran Connector SDK
26+
27+
### Critical SDK v2+ breaking changes
28+
As of SDK v2.0.0 (August 2025), yield is NO LONGER USED:
29+
- DEPRECATED: `yield op.upsert(table, data)`, `yield op.checkpoint(state)`
30+
- REQUIRED: `op.upsert(table, data)`, `op.checkpoint(state)` (direct calls, no yield)
31+
- All operations (`upsert`, `update`, `delete`, `checkpoint`) are now synchronous
32+
- Backward compatible: old v1 connectors still work, but new code must not use `yield`
33+
34+
### Runtime and tooling
35+
- Python versions: 3.10-3.12 (3.13 experimental support)
36+
- Pre-installed packages: `fivetran_connector_sdk` (latest), `requests` (latest) - NEVER declare in requirements.txt
37+
- Linting: `flake8` with `.flake8` config at repo root (PEP 8 compliance)
38+
- Formatting: `black` via pre-commit hooks (run `.github/scripts/setup-hooks.sh`)
39+
- Naming conventions: `snake_case` for functions/variables, `PascalCase` for classes, `UPPER_SNAKE_CASE` for constants
40+
41+
## How to validate PRs: connector structure
42+
43+
When a PR adds/modifies a connector, verify:
44+
45+
### Required files (BLOCKER if missing)
46+
- connector.py:
47+
- Must import: `from fivetran_connector_sdk import Connector, Operations as op, Logging as log`
48+
- Must define: `update(configuration: dict, state: dict)` function
49+
- Should define: `schema(configuration: dict)` function
50+
- Must initialize: `connector = Connector(update=update, schema=schema)` at module level
51+
- NO yield: `op.upsert(table, data)` not `yield op.upsert(table, data)`
52+
- MUST always have `validate_configuration()` function defined
53+
- First log statement in update method: `log.warning("Example: <CATEGORY> : <EXAMPLE_NAME>")`
54+
55+
- configuration.json (if connector needs configuration):
56+
- All values must use placeholder format: `"api_key": "<YOUR_API_KEY>"`
57+
- NO real secrets, credentials, or personal data
58+
- Keys must be descriptive (no abbreviations): `database_url` not `db_url`
59+
- Must match all fields referenced in connector.py
60+
61+
- README.md:
62+
- Must follow README_template.md structure
63+
- Must have single H1 heading: `# <Source Name> Connector Example`
64+
- Required sections: Connector overview, Requirements, Getting started, Features, Data handling, Error handling, Tables created, Additional considerations
65+
- See `.github/instructions/readme-markdown.instructions.md` for detailed rules
66+
67+
- requirements.txt (only if external dependencies needed):
68+
- Explicit versions: `pandas==2.0.3` not `pandas`
69+
- NEVER include `fivetran_connector_sdk` or `requests` (provided by runtime)
70+
- Prefer minimal dependencies; avoid heavyweight libraries for simple tasks
71+
72+
## Additional review resources
73+
For detailed rules, reference:
74+
- Python code: `.github/instructions/python-review.instructions.md`
75+
- JSON config: `.github/instructions/configuration-review.instructions.md`
76+
- README files: `.github/instructions/readme-markdown.instructions.md`
77+
- Coding standards: `PYTHON_CODING_STANDARDS.md`
78+
79+
## When to search vs. trust this guide
80+
Default to these instructions. Only search repo/docs if:
81+
- PR introduces new SDK features not mentioned here
82+
- Code contradicts this guidance (e.g., new Python version support, new operations)
83+
- Inconsistency found (cite the newest official docs at https://fivetran.com/docs/connectors/connector-sdk)

‎.github/git_hooks/pre-commit‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
#!/bin/bash
2+
# Pre-commit hook to auto-format staged Python files using black.
3+
4+
echo "Running black auto-formatter for staged Python files"
5+
6+
# Ensure black is installed
7+
if ! command -v black &> /dev/null; then
8+
echo "'black' is not installed. Attempting to install..."
9+
if command -v pip3 &> /dev/null; then
10+
pip3 install --user black
11+
export PATH="$HOME/.local/bin:$PATH"
12+
elif command -v pip &> /dev/null; then
13+
pip install --user black
14+
export PATH="$HOME/.local/bin:$PATH"
15+
else
16+
echo "pip not found. Please install pip and rerun."
17+
echo "Skipping formatting!"
18+
fi
19+
20+
# Check again after install
21+
if ! command -v black &> /dev/null; then
22+
echo "Failed to install 'black'. Aborting formatting but allowing commit!"
23+
exit 0 # Github Actions will prevent merging of PRs if its not properly formatted, we want to continue with the commit.
24+
fi
25+
fi
26+
27+
# Get list of staged Python files
28+
PY_FILES=$(git diff --cached --name-only --diff-filter=ACMR | grep -E '\.py$')
29+
if [ -z "$PY_FILES" ]; then
30+
echo "No Python files to format."
31+
exit 0
32+
fi
33+
34+
# Run black on each staged file and re-add to the index
35+
echo "$PY_FILES" | while read -r file; do
36+
if [ -f "$file" ]; then
37+
black --line-length=99 "$file"
38+
git add "$file"
39+
echo "Formatted: $file"
40+
fi
41+
done
42+
43+
echo "Finished formatting Python files."

‎.github/instructions/configuration-review.instructions.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
applyTo: "**/*.json"
3+
---
4+
# Your role
5+
You are the code reviewer for pull requests in this repo. Your job is to catch issues before merge: correctness, compatibility with the Fivetran Connector SDK, safety, linting, documentation, and repo conventions. Prefer actionable, specific review comments. When material issues are present, Request changes with a clear checklist. Use this instruction set as ground truth. Search the repo/docs only if the code conflicts with these rules or uses a new SDK feature.
6+
7+
# Review guidelines for configuration.json files
8+
When a PR changes or adds a connector/example/template that includes a `configuration.json`, perform these comprehensive checks:
9+
10+
## JSON Structure and Syntax (BLOCKER if violated)
11+
- **Valid JSON**: File must parse correctly as valid JSON (no trailing commas, proper quotes, balanced braces)
12+
- **Root object**: Configuration must be a JSON object `{}`, not an array or primitive
13+
- **Template compliance**: Follow the structure in [template configuration.json](https://github.com/fivetran/fivetran-csdk-connectors/blob/main/_template_connector/configuration.json)
14+
15+
## Key Naming Conventions (REQUEST_CHANGES if violated)
16+
- **Descriptive names**: Keys must clearly describe their purpose (e.g., `api_key`, `database_url`, `max_retries`)
17+
- **Snake case preferred**: Use lowercase with underscores: `api_key`, `rate_limit_per_hour`
18+
- **Uppercase acceptable**: For constants/env-style: `API_KEY`, `DATABASE_URL`, `MAX_RETRIES`
19+
- **Be consistent**: Use one convention throughout the file
20+
- **NO abbreviations**: Avoid `cfg`, `db_conn`, `usr` - use full words `configuration`, `database_connection`, `user`
21+
22+
## Value Format and Placeholders (BLOCKER if violated)
23+
- **Placeholder format**: All values **must** use angle bracket format: `<DESCRIPTION_HERE>`
24+
- GOOD: `"api_key": "<YOUR_API_KEY>"`
25+
- BAD: `"api_key": "your_api_key"` (no brackets)
26+
- BAD: `"api_key": ""` (empty string)
27+
- BAD: `"api_key": "abc123xyz"` (real value)
28+
- **Descriptive placeholders**: Placeholder text should describe what the user needs to provide
29+
- GOOD: `<YOUR_HARNESS_API_TOKEN>`
30+
- GOOD: `<YOUR_CLICKHOUSE_SERVER_HOSTNAME>`
31+
- BAD: `<VALUE>` (too generic)
32+
- BAD: `<STRING>` (describes type, not purpose)
33+
34+
## Security and Secrets (BLOCKER if violated)
35+
- **NO real secrets**: Configuration must not contain:
36+
- Real API keys, tokens, passwords
37+
- Real URLs with credentials embedded
38+
- Real email addresses or personal information
39+
- Real database connection strings
40+
- Real IP addresses or internal hostnames
41+
- **Scan for patterns**: Check for:
42+
- Long alphanumeric strings that look like keys
43+
- URLs with `username:password@host` format
44+
- Email addresses (unless clearly example.com)
45+
- Base64-encoded content
46+
- JWT tokens (starting with `eyJ`)
47+
48+
## Configuration Completeness (REQUEST_CHANGES if violated)
49+
- **All required fields**: Every field required by `connector.py` must be present
50+
- **NO extra fields**: Remove fields that are not used in the connector code
51+
- **Match README**: Configuration in `configuration.json` must exactly match the table in README
52+
- Same field names
53+
- Same descriptions
54+
- Same required/optional indicators
55+
- **Validation**: If connector has configuration validation, ensure all validated fields are in config file
56+
57+
## Examples of Good vs Bad Configurations
58+
59+
### GOOD Example
60+
```json
61+
{
62+
"api_token": "<YOUR_HARNESS_API_TOKEN>",
63+
"account_id": "<YOUR_HARNESS_ACCOUNT_ID>",
64+
"verify_ssl": "<TRUE_OR_FALSE_DEFAULT_TRUE>"
65+
}
66+
```
67+
68+
### BAD Example
69+
```json
70+
{
71+
"token": "abcd1234xyz",
72+
"acc": "",
73+
"url": "https://app.harness.io",
74+
"ssl": "yes",
75+
"unused_field": "<SOME_VALUE>"
76+
}
77+
```
78+
**Issues**: Real token, abbreviations, empty value, non-standard boolean, no placeholders, unused field

0 commit comments

Comments
 (0)