Skip to content

Commit ddeb2db

Browse files
azvoleffazvoleff
committed
Edit readme
1 parent 4294a8d commit ddeb2db

File tree

1 file changed

+171
-170
lines changed

1 file changed

+171
-170
lines changed

‎README.md‎

Lines changed: 171 additions & 170 deletions
Original file line numberDiff line numberDiff line change
@@ -6,189 +6,35 @@
66

77
This project belongs to the trends.earth project by Conservation International.
88

9-
This repository implements the Core Platform of the trends.earth Environment for running Google Earth Engine scripts.
9+
This repository implements the environment used for executing jobs run via the trends.earth API.
1010

1111
## Project Structure
1212

1313
```
1414
trends.earth-Environment/
15-
├── gefcore/ # Core Python package
16-
│ ├── __init__.py # Package initialization and main entry point
17-
│ ├── api.py # API client for trends.earth-API
18-
│ ├── loggers.py # Custom logging handlers
19-
│ └── runner.py # Script execution engine
20-
├── scripts/ # Utility scripts
15+
├── gefcore/ # Core Python package
16+
│ ├── __init__.py # Package initialization and main entry point
17+
│ ├── api.py # API client for trends.earth-API
18+
│ ├── loggers.py # Custom logging handlers
19+
│ └── runner.py # Script execution engine
20+
├── scripts/ # Utility scripts
2121
│ └── dependency_manager.py # Security and dependency management
22-
├── tests/ # Test suite
23-
├── .github/workflows/ # CI/CD workflows
24-
├── Dockerfile # Container build configuration
25-
├── main.py # Application entry point
26-
├── entrypoint.sh # Docker container entrypoint
27-
├── requirements.txt # Production dependencies
28-
├── requirements-dev.txt # Development dependencies
29-
├── pyproject.toml # Project configuration and tools
30-
└── run_tests.sh # Test execution script
22+
├── tests/ # Test suite
23+
├── .github/workflows/ # CI/CD workflows
24+
├── Dockerfile # Container build configuration
25+
├── main.py # Application entry point
26+
├── entrypoint.sh # Docker container entrypoint
27+
├── requirements.txt # Production dependencies
28+
├── requirements-dev.txt # Development dependencies
29+
├── pyproject.toml # Project configuration and tools
30+
└── run_tests.sh # Test execution script
3131
```
3232

3333
## Related Projects
3434

3535
- [trends.earth API](https://github.com/ConservationInternational/trends.earth-API) - Backend API service
3636
- [trends.earth CLI](https://github.com/ConservationInternational/trends.earth-CLI) - Command-line interface
3737

38-
## Development Setup
39-
40-
### Prerequisites
41-
42-
- Python 3.10+
43-
- Docker (for containerized development)
44-
- Google Earth Engine service account (for production use)
45-
46-
### Local Development
47-
48-
1. **Clone the repository:**
49-
```bash
50-
git clone https://github.com/ConservationInternational/trends.earth-Environment.git
51-
cd trends.earth-Environment
52-
```
53-
54-
2. **Create and activate virtual environment:**
55-
```bash
56-
python -m venv .venv
57-
source .venv/bin/activate # On Windows: .venv\Scripts\activate
58-
```
59-
60-
3. **Install dependencies:**
61-
```bash
62-
pip install -r requirements.txt
63-
pip install -r requirements-dev.txt
64-
```
65-
66-
4. **Run tests:**
67-
```bash
68-
./run_tests.sh
69-
# Or directly with pytest
70-
pytest tests/ -v
71-
```
72-
73-
### Docker Development
74-
75-
1. **Build the container:**
76-
```bash
77-
docker build -t trends-earth-env .
78-
```
79-
80-
2. **Run the container:**
81-
```bash
82-
docker run --rm \
83-
-e ENV=development \
84-
-e TESTING=false \
85-
trends-earth-env
86-
```
87-
88-
## Environment Variables
89-
90-
Required environment variables for production use:
91-
92-
- `ENV` - Environment mode (`production`, `staging`, `development`, `test`)
93-
- `TESTING` - Set to `true` for test environments
94-
- `ROLLBAR_SCRIPT_TOKEN` - Rollbar error reporting token
95-
- `GOOGLE_PROJECT_ID` - Google Cloud Project ID
96-
- `GEE_ENDPOINT` - Google Earth Engine API endpoint
97-
- `API_URL` - trends.earth API base URL
98-
- `API_USER` - API authentication username
99-
- `API_PASSWORD` - API authentication password
100-
- `EXECUTION_ID` - Unique execution identifier
101-
- `PARAMS_S3_BUCKET` - S3 bucket for parameters
102-
- `PARAMS_S3_PREFIX` - S3 key prefix for parameters
103-
104-
## Testing
105-
106-
The project includes comprehensive testing with pytest:
107-
108-
```bash
109-
# Run all tests
110-
./run_tests.sh
111-
112-
# Run specific test files
113-
pytest tests/test_api.py -v
114-
115-
# Run with coverage
116-
pytest tests/ --cov=gefcore --cov-report=html
117-
118-
# Run only fast tests (exclude slow integration tests)
119-
pytest tests/ -m "not slow"
120-
```
121-
122-
## Code Quality
123-
124-
Code quality is maintained using:
125-
126-
- **Ruff** - Fast Python linter and formatter
127-
- **mypy** - Static type checking
128-
- **pytest** - Testing framework with 92%+ coverage requirement
129-
130-
Run quality checks:
131-
132-
```bash
133-
# Linting and formatting
134-
ruff check gefcore/ tests/
135-
ruff format gefcore/ tests/
136-
137-
# Type checking
138-
mypy gefcore/ --ignore-missing-imports
139-
```
140-
141-
## Security
142-
143-
### Security Tools
144-
145-
Use the dependency manager script for security checks:
146-
147-
```bash
148-
# Check for vulnerabilities
149-
python scripts/dependency_manager.py --check-vulns
150-
151-
# Check for outdated packages
152-
python scripts/dependency_manager.py --check-outdated
153-
154-
# Run comprehensive security audit
155-
python scripts/dependency_manager.py --audit
156-
157-
# Run all security checks
158-
python scripts/dependency_manager.py --all
159-
```
160-
161-
### Manual Security Scanning
162-
163-
```bash
164-
# Install security tools
165-
pip install safety bandit[toml]
166-
167-
# Check dependencies for vulnerabilities
168-
safety scan -r requirements.txt
169-
170-
# Scan code for security issues
171-
bandit -r gefcore/
172-
```
173-
174-
## Architecture
175-
176-
### Core Modules
177-
178-
- **`gefcore.api`** - HTTP client for communicating with trends.earth-API, handles authentication, retries, and error handling
179-
- **`gefcore.runner`** - Main script execution engine that initializes Google Earth Engine and executes user scripts
180-
- **`gefcore.loggers`** - Custom logging handlers that send logs to the API and handle different environment configurations
181-
- **`scripts.dependency_manager`** - Security and dependency management utilities
182-
183-
### Execution Flow
184-
185-
1. Container starts via `main.py` → `gefcore.__init__.py`
186-
2. Environment validation and logger setup
187-
3. Google Earth Engine authentication and initialization
188-
4. Script parameter retrieval from S3
189-
5. User script execution with monitoring and logging
190-
6. Results and status reporting back to API
191-
19238
## Developing Custom Scripts
19339

19440
This environment package serves as the execution platform for custom geospatial analysis scripts developed using the [trends.earth CLI](https://github.com/ConservationInternational/trends.earth-CLI). Here's how to create and deploy new scripts:
@@ -368,6 +214,161 @@ def run(params, logger):
368214
- **Parameter validation** - Validate input parameters before processing
369215
- **Modular code** - Split complex logic into separate functions/modules
370216

217+
## Development Setup
218+
219+
### Prerequisites
220+
221+
- Python 3.10+
222+
- Docker (for containerized development)
223+
- Google Earth Engine service account (for production use)
224+
225+
### Local Development
226+
227+
1. **Clone the repository:**
228+
```bash
229+
git clone https://github.com/ConservationInternational/trends.earth-Environment.git
230+
cd trends.earth-Environment
231+
```
232+
233+
2. **Create and activate virtual environment:**
234+
```bash
235+
python -m venv .venv
236+
source .venv/bin/activate # On Windows: .venv\Scripts\activate
237+
```
238+
239+
3. **Install dependencies:**
240+
```bash
241+
pip install -r requirements.txt
242+
pip install -r requirements-dev.txt
243+
```
244+
245+
4. **Run tests:**
246+
```bash
247+
./run_tests.sh
248+
# Or directly with pytest
249+
pytest tests/ -v
250+
```
251+
252+
### Docker Development
253+
254+
1. **Build the container:**
255+
```bash
256+
docker build -t trends-earth-env .
257+
```
258+
259+
2. **Run the container:**
260+
```bash
261+
docker run --rm \
262+
-e ENV=development \
263+
-e TESTING=false \
264+
trends-earth-env
265+
```
266+
267+
## Environment Variables
268+
269+
Required environment variables for production use:
270+
271+
- `ENV` - Environment mode (`production`, `staging`, `development`, `test`)
272+
- `TESTING` - Set to `true` for test environments
273+
- `ROLLBAR_SCRIPT_TOKEN` - Rollbar error reporting token
274+
- `GOOGLE_PROJECT_ID` - Google Cloud Project ID
275+
- `GEE_ENDPOINT` - Google Earth Engine API endpoint
276+
- `API_URL` - trends.earth API base URL
277+
- `API_USER` - API authentication username
278+
- `API_PASSWORD` - API authentication password
279+
- `EXECUTION_ID` - Unique execution identifier
280+
- `PARAMS_S3_BUCKET` - S3 bucket for parameters
281+
- `PARAMS_S3_PREFIX` - S3 key prefix for parameters
282+
283+
## Testing
284+
285+
The project includes comprehensive testing with pytest:
286+
287+
```bash
288+
# Run all tests
289+
./run_tests.sh
290+
291+
# Run specific test files
292+
pytest tests/test_api.py -v
293+
294+
# Run with coverage
295+
pytest tests/ --cov=gefcore --cov-report=html
296+
297+
# Run only fast tests (exclude slow integration tests)
298+
pytest tests/ -m "not slow"
299+
```
300+
301+
## Code Quality
302+
303+
Code quality is maintained using:
304+
305+
- **Ruff** - Fast Python linter and formatter
306+
- **mypy** - Static type checking
307+
- **pytest** - Testing framework with 92%+ coverage requirement
308+
309+
Run quality checks:
310+
311+
```bash
312+
# Linting and formatting
313+
ruff check gefcore/ tests/
314+
ruff format gefcore/ tests/
315+
316+
# Type checking
317+
mypy gefcore/ --ignore-missing-imports
318+
```
319+
320+
## Security
321+
322+
### Security Tools
323+
324+
Use the dependency manager script for security checks:
325+
326+
```bash
327+
# Check for vulnerabilities
328+
python scripts/dependency_manager.py --check-vulns
329+
330+
# Check for outdated packages
331+
python scripts/dependency_manager.py --check-outdated
332+
333+
# Run comprehensive security audit
334+
python scripts/dependency_manager.py --audit
335+
336+
# Run all security checks
337+
python scripts/dependency_manager.py --all
338+
```
339+
340+
### Manual Security Scanning
341+
342+
```bash
343+
# Install security tools
344+
pip install safety bandit[toml]
345+
346+
# Check dependencies for vulnerabilities
347+
safety scan -r requirements.txt
348+
349+
# Scan code for security issues
350+
bandit -r gefcore/
351+
```
352+
353+
## Architecture
354+
355+
### Core Modules
356+
357+
- **`gefcore.api`** - HTTP client for communicating with trends.earth-API, handles authentication, retries, and error handling
358+
- **`gefcore.runner`** - Main script execution engine that initializes Google Earth Engine and executes user scripts
359+
- **`gefcore.loggers`** - Custom logging handlers that send logs to the API and handle different environment configurations
360+
- **`scripts.dependency_manager`** - Security and dependency management utilities
361+
362+
### Execution Flow
363+
364+
1. Container starts via `main.py` → `gefcore.__init__.py`
365+
2. Environment validation and logger setup
366+
3. Google Earth Engine authentication and initialization
367+
4. Script parameter retrieval from S3
368+
5. User script execution with monitoring and logging
369+
6. Results and status reporting back to API
370+
371+
371372
## Container security scan (requires Docker)
372373
docker run --rm -v $(pwd):/workspace aquasec/trivy fs /workspace
373374
```

0 commit comments

Comments
 (0)