Skip to content

Commit 7ae37da

Browse files
author
Adriano Sanges
committed
Refactor real estate ETL project structure and remove dlt-specific components
- Remove real-estate-etl-dlt directory and its Scrapy/dlt-specific implementation - Update README.md to reflect current project structure - Add README.md to real-estate-etl directory with project details - Simplify project configuration by removing unnecessary dlt-related files
1 parent 0e64ec0 commit 7ae37da

File tree

18 files changed

+15
-2012
lines changed

18 files changed

+15
-2012
lines changed

README.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
This repository contains data pipelines for various datasets.
22

3-
## Real Estate ETL DLT
3+
## Real Estate ETL
44

55
This pipeline is used to extract data from the real estate website Immobiliare.it.
66

7-
The pipeline is defined in the `real-estate-etl-dlt` directory.
7+
The pipeline is defined in the `real-estate-etl` directory.
8+
9+
It uses the BeatifulSoup library to scrape and DuckDB to store the data.
810

9-
It uses the `dlt` library to define the pipeline and the `scrapy` library to scrape the data.

real-estate-etl-dlt/.dlt/.sources

Lines changed: 0 additions & 40 deletions
This file was deleted.

real-estate-etl-dlt/.dlt/config.toml

Lines changed: 0 additions & 10 deletions
This file was deleted.

real-estate-etl-dlt/.gitignore

Lines changed: 0 additions & 10 deletions
This file was deleted.

real-estate-etl-dlt/.python-version

Lines changed: 0 additions & 1 deletion
This file was deleted.

real-estate-etl-dlt/README.md

Whitespace-only changes.

real-estate-etl-dlt/pyproject.toml

Lines changed: 0 additions & 14 deletions
This file was deleted.

real-estate-etl-dlt/scraping/README.md

Lines changed: 0 additions & 112 deletions
This file was deleted.

real-estate-etl-dlt/scraping/__init__.py

Lines changed: 0 additions & 71 deletions
This file was deleted.
-130 KB
Binary file not shown.

0 commit comments

Comments
 (0)