Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,25 @@ We don't use the package versions (in pyproject.toml files), no need to update t

All the contributions should go through a pull request. The pull requests must be "squashed" (ie: one commit per pull request).

## Update common dependencies

1. Go to `libs/libcommon`

2. Update dependencies in `pyproject.toml`.

3. Run a CLI on all projects that depend on libcommon:

```bash
>>> poetry run cli_all_projects
Multi CLI for projects: ...
```

4. Then you can update all dependencies using

```bash
>>> poetry lock
```

## GitHub Actions

You can use [act](https://github.com/nektos/act) to test the GitHub Actions (see [.github/workflows/](.github/workflows/)) locally. It reduces the retroaction loop when working on the GitHub Actions, avoid polluting the branches with empty pushes only meant to trigger the CI, and allows to only run specific actions.
Expand Down
6 changes: 3 additions & 3 deletions front/admin_ui/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions jobs/cache_maintenance/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions jobs/mongodb_migration/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions libs/libapi/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions libs/libcommon/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion libs/libcommon/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ python = "3.12.11"
anyio = ">=3.4.0,<5"
appdirs = "^1.4.4"
cryptography = "^43.0.1"
datasets = {git="https://github.com/huggingface/datasets.git", rev="f15c59970fef04e277a93b9c262ccdc7bd8038a4", extras = ["vision"]}
datasets = {git="https://github.com/huggingface/datasets.git", rev="6db5f805a444a767726d58d5dd1cfaa25fc852c0", extras = ["vision"]}
duckdb = "^1.2.2"
environs = "^14.3.0"
filelock = "^3.18.0"
Expand Down Expand Up @@ -73,6 +73,9 @@ validators = "^0.22.0"
build-backend = "poetry.core.masonry.api"
requires = ["poetry-core>=1.0.0"]

[tool.poetry.scripts]
cli_all_projects = 'libcommon.scripts:cli_all_projects'

[tool.pytest.ini_options]
filterwarnings = ["ignore::DeprecationWarning"]
markers = [
Expand Down
20 changes: 20 additions & 0 deletions libs/libcommon/src/libcommon/scripts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import subprocess # nosec
from pathlib import Path


def cli_all_projects() -> None:
all_projects_dir = Path(__file__).resolve().parents[-5]
projects = [
p
for d in all_projects_dir.glob("*")
if not d.name.startswith(".") and d.is_dir()
for p in list(d.glob("*")) + [d]
if (p / "pyproject.toml").exists()
]
projects.sort(key=lambda p: (p.name != "libcommon", "libs" not in p.parts, p))

print(f"Multi CLI for projects: {', '.join(str(project) for project in projects)}")
while (command := input(">>> ")) not in ["exit", "exit()"]:
for project in projects:
print(f"cd {project}")
subprocess.run(command.split(" "), cwd=project) # nosec
6 changes: 3 additions & 3 deletions services/admin/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions services/api/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions services/rows/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions services/search/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions services/sse-api/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions services/webhook/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions services/worker/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions services/worker/tests/fixtures/hub.py
Original file line number Diff line number Diff line change
Expand Up @@ -566,9 +566,9 @@ def create_dataset_info_response_for_audio(dataset: str, config: str) -> Any:
"config_name": config,
"dataset_name": dataset_name,
"version": {"version_str": "0.0.0", "major": 0, "minor": 0, "patch": 0},
"splits": {"train": {"name": "train", "num_bytes": 15448, "num_examples": 1, "dataset_name": None}},
"splits": {"train": {"name": "train", "num_bytes": 15414, "num_examples": 1, "dataset_name": None}},
"download_size": AUDIO_PARQUET_SIZE,
"dataset_size": 15448,
"dataset_size": 15414,
}


Expand Down Expand Up @@ -648,7 +648,7 @@ def create_parquet_and_info_response(

CSV_PARQUET_SIZE = 1730
PARTIAL_CSV_PARQUET_SIZE = 5806
AUDIO_PARQUET_SIZE = 16546
AUDIO_PARQUET_SIZE = 16477
BIG_PARQUET_FILE = 18418

DATA_cols = {
Expand Down
Loading