Skip to content

USAGE.md

Latest commit

 

History

History
145 lines (116 loc) · 3.41 KB

USAGE.md

File metadata and controls

145 lines (116 loc) · 3.41 KB
Raw

Usage Guide

Available Commands

The run.sh script provides convenient commands for common operations:

Building and Managing

./run.sh build    # Build the Docker image
./run.sh up       # Start container in background
./run.sh down     # Stop container
./run.sh logs     # View container logs

Interactive Sessions

./run.sh jupyter  # Start Jupyter Lab (recommended)
./run.sh python   # Start Python REPL
./run.sh bash     # Start bash shell
./run.sh claude   # Start Claude Code CLI

Running Code

./run.sh run script.py    # Run a Python script from code/ directory
./run.sh install pandas   # Install additional packages

Working with Data

Directory Structure

docker-setup/
├── data/           # Your datasets go here
├── notebooks/      # Jupyter notebooks
├── code/          # Python scripts
├── Dockerfile     # Container configuration
└── run.sh         # Helper script

Loading Data

Put your datasets in the data/ directory. They'll be available at /workspace/data/ inside the container.

import pandas as pd

# Load a dataset
df = pd.read_csv('/workspace/data/your_dataset.csv')

# For large files, use chunks
for chunk in pd.read_csv('/workspace/data/large_file.csv', chunksize=10000):
    process(chunk)

Saving Work

  • Jupyter notebooks are automatically saved to notebooks/
  • Python scripts go in code/
  • Processed data can be saved to data/

Performance Optimization

Memory Management

# Check memory usage
import psutil
print(f"Memory: {psutil.virtual_memory().percent}%")

# Optimize data types
df['id'] = df['id'].astype('int32')
df['category'] = df['category'].astype('category')

# Clear variables
del large_dataframe
import gc; gc.collect()

Large Dataset Handling

# Use chunking for large files
def process_large_file(filename):
    for chunk in pd.read_csv(filename, chunksize=10000):
        # Process each chunk
        result = chunk.groupby('column').sum()
        # Save or accumulate results
        yield result

# Use efficient file formats
df.to_parquet('data.parquet')  # Faster than CSV
df.to_feather('data.feather')  # Very fast for temporary storage

Common Workflows

Data Analysis Pipeline

  1. Load data: ./run.sh jupyter
  2. Explore in Jupyter notebook
  3. Create processing script in code/
  4. Run script: ./run.sh run process_data.py
  5. Visualize results back in Jupyter

Machine Learning Workflow

  1. Data preprocessing in Jupyter
  2. Model training script in code/
  3. Evaluation and visualization in Jupyter
  4. Save models to data/models/

NLP Pipeline

import spacy

# Load pre-installed model
nlp = spacy.load("en_core_web_sm")

# Process text
doc = nlp("Your text here")
for token in doc:
    print(token.text, token.pos_, token.lemma_)

Tips and Tricks

Jupyter Lab Extensions

Install extensions inside the container:

./run.sh bash
pip install jupyterlab-git
jupyter labextension install @jupyterlab/git

Persistent Package Installation

To permanently add packages, edit the Dockerfile and rebuild:

RUN pip install --no-cache-dir \
    your-new-package

Environment Variables

Set environment variables in docker-compose.yml:

environment:
  - CUSTOM_VAR=value

Custom Startup Scripts

Create a startup script in code/startup.py and run it automatically by modifying the Dockerfile CMD.