|
| 1 | +--- |
| 2 | +name: How to make sure too many cooks don’t spoil the broth |
| 3 | +excerpt: A custom approach to do multi-package semantic versioning in a Python monorepo |
| 4 | +author: Aatman Vaidya, Priyash Shah |
| 5 | +project: Feluda |
| 6 | +date: 2025-04-01 |
| 7 | +tags: devlog |
| 8 | +cover: |
| 9 | +--- |
| 10 | + |
| 11 | +In Feluda, functionalities are abstracted into something we call **operators**, which are built keeping in mind the need to process data in various modalities (text, audio, video, images, hybrid) and various languages. You can think of operators as plugins that you can mix and match to perform different analysis on your data. This abstraction also allows others to write operators for their unique use case and use them within Feluda. |
| 12 | +Feluda acts like a conductor in an orchestra where it brings together all the operators to let them do their function. For example, if you wish to cluster large amounts of videos, then operators will help you convert a video to an embedding, and then use embeddings to cluster them. While we encourage the community to build and use their own operators, we also provide some core operators out of the box with Feluda, which are maintained within the same Feluda repository. |
| 13 | + |
| 14 | +### **The Challenge** |
| 15 | + |
| 16 | +In a monorepo with multiple Python packages we wanted a dynamic system that could automatically maintain and update package versions. As manually tracking changes and version numbers is tedious and not manageable as number of packages increase, we needed a solution that could: |
| 17 | + |
| 18 | +* Automatically detect which packages have changed |
| 19 | +* Determine the appropriate version bump based on commit messages (follow [semantic versioning](https://semver.org/) rules) |
| 20 | +* Update package versions in pyproject.toml files |
| 21 | +* Create corresponding Git tags |
| 22 | +* Handle all of this while maintaining semantic versioning principles |
| 23 | + |
| 24 | +Here is a demo folder structure we wanted to cater too |
| 25 | +``` |
| 26 | +.(root) |
| 27 | +├── package1/ |
| 28 | +│ ├── package1.py |
| 29 | +│ ├── test_package1.py |
| 30 | +│ ├── pyproject.toml |
| 31 | +│ └── README.md |
| 32 | +├── package2/ |
| 33 | +│ ├── package2.py |
| 34 | +│ ├── test_package2.py |
| 35 | +│ ├── pyproject.toml |
| 36 | +│ └── README.md |
| 37 | +└── package3/ |
| 38 | + ├── package3.py |
| 39 | + ├── test_package3.py |
| 40 | + ├── pyproject.toml |
| 41 | + └── README.md |
| 42 | +``` |
| 43 | + |
| 44 | +### **The Solution** |
| 45 | + |
| 46 | +We wrote a [custom script](https://github.com/tattle-made/feluda/blob/main/scripts/semantic_release_workflow.py) ([github action](https://github.com/tattle-made/feluda/blob/main/.github/workflows/merge-main.yml)) that analyzes commit messages between two **git commits** and automatically handles version management. Here's how it works: |
| 47 | + |
| 48 | +Package Discovery |
| 49 | +The script starts by discovering all Python packages located in our monorepo. It looks for packages in two locations: |
| 50 | + |
| 51 | +- The root package (named "feluda" in our case) |
| 52 | +- Any package inside the **operators** directory (as each operator is a folder inside it) |
| 53 | + |
| 54 | +Each package must have a valid pyproject.toml file containing the current version number. |
| 55 | + |
| 56 | +Analyzing Changes |
| 57 | +For each package, the script: |
| 58 | + |
| 59 | +* Retrieves all commits affecting that package between two specified Git commits range. |
| 60 | +* Analyzes each commit message using [conventional commit format](https://www.conventionalcommits.org/en/v1.0.0/). |
| 61 | +* Determines the highest priority version bump needed |
| 62 | + |
| 63 | +The commit analysis follows these rules: |
| 64 | + |
| 65 | +- Commits containing **"breaking change"** trigger a **major** version bump |
| 66 | +- Commits starting with **"feat:"** trigger a **minor** version bump |
| 67 | +- Other conventional commits **(fix:, chore:, etc.)** trigger a **patch** bump |
| 68 | + |
| 69 | +Version Management |
| 70 | +Once the script determines the necessary version bump, it: |
| 71 | + |
| 72 | +* Checks if a Git tag already exists for the new version |
| 73 | +* Updates the version in pyproject.toml if needed |
| 74 | +* Creates a new Git tag using the format specified in pyproject.toml |
| 75 | + |
| 76 | +The script also includes comprehensive error handling to ensure reliability, it validates repository paths and package structures, handles missing or malformed pyproject.toml files, manages git command failures gracefully and provides clear error messages for troubleshooting. |
| 77 | + |
| 78 | +#### Here is a detailed flowchart of this approach \- [https://github.com/tattle-made/feluda/wiki/Release-Management\#semantic-versioning](https://github.com/tattle-made/feluda/wiki/Release-Management#semantic-versioning) |
| 79 | + |
| 80 | +### **Best Practices and Benefits of this Approach** |
| 81 | + |
| 82 | +To make the most of this automation: |
| 83 | + |
| 84 | +1. Always use conventional commit messages: |
| 85 | + |
| 86 | +``` |
| 87 | +feat: add new feature |
| 88 | +fix: resolve bug |
| 89 | +chore: update dependencies |
| 90 | +``` |
| 91 | + |
| 92 | +2. Include proper tag format in **pyproject.toml**: |
| 93 | + |
| 94 | +``` |
| 95 | +[tool.semantic_release.branches.main] |
| 96 | +tag_format = "{name}-{version}" |
| 97 | +``` |
| 98 | + |
| 99 | +3. Run the script as part of your CI/CD pipeline between a range of commits: |
| 100 | +``` |
| 101 | +python semantic_release_workflow.py <previous_commit> <current_commit> |
| 102 | +``` |
| 103 | + |
| 104 | +This automated approach offers several advantages: |
| 105 | + |
| 106 | +1. **Consistency**: Version numbers always reflect the nature of changes made |
| 107 | +2. **Efficiency**: Eliminates manual version management reducing human error in version numbering. |
| 108 | +3. **Traceability**: Version changes are directly linked to commit history |
| 109 | + |
| 110 | +### **Link to the [Custom Python Script](https://github.com/tattle-made/feluda/blob/main/scripts/semantic_release_workflow.py)** |
| 111 | +### **Link to the [Github Action](https://github.com/tattle-made/feluda/blob/main/.github/workflows/merge-main.yml)** |
0 commit comments