Skip to content

Python Dependency Management Hell: From Chaos to Control

Python dependency management has become increasingly complex in 2024-2025, with developers juggling multiple tools (pip, conda, poetry, pipenv) and facing constant conflicts. This cookbook provides practical solutions to the most common dependency nightmares.

The Current State of Python Dependencies (2024-2025)

The Core Problem

Python's ecosystem is suffering from "dependency hell" - a perfect storm of: - Tool fragmentation (pip vs conda vs poetry vs pipenv vs uv) - Version conflicts and incompatible dependencies - Security vulnerabilities in the supply chain - Performance issues with traditional package managers - Corporate environments with strict security policies

Why Traditional Solutions Are Breaking Down

# The old way - slow and fragile
pip install -r requirements.txt
# ❌ No lock file by default
# ❌ Slow dependency resolution
# ❌ No built-in security scanning
# ❌ Virtual environment management is manual

# Poetry attempt - better but still problematic
poetry install
# ❌ Still slow for large projects
# ❌ Complex corporate proxy configurations
# ❌ Lock file conflicts in team environments

1. The New Champion: UV Package Manager

What is UV?

UV is a Rust-based Python package installer that's 10-100x faster than pip and aims to replace the entire Python toolchain.

Installation and Basic Setup

# Install UV
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or via pip (ironically)
pip install uv

# Verify installation
uv --version

# Enable shell completion
echo 'eval "$(uv generate-shell-completion bash)"' >> ~/.bashrc
source ~/.bashrc

Project Initialization with UV

# Initialize a new project
uv init my-fastapi-project
cd my-fastapi-project

# Project structure created:
# ├── .gitignore
# ├── .python-version
# ├── hello.py
# ├── pyproject.toml
# ├── README.md
# └── uv.lock

# Add dependencies (replaces pip install)
uv add fastapi uvicorn sqlalchemy

# Add development dependencies
uv add --dev pytest black ruff mypy

# Install dependencies
uv sync

# Run commands in the environment
uv run python hello.py
uv run pytest

Advanced UV Usage

# Python version management (replaces pyenv)
uv python list  # List available Python versions
uv python install 3.12  # Install Python 3.12
uv python use 3.12  # Use Python 3.12 for current project

# Virtual environment management
uv venv  # Create virtual environment
uv venv --python 3.11  # Create with specific Python version

# Lock file management
uv lock  # Generate/update lock file
uv sync  # Install exact versions from lock file

# Tool management (replaces pipx)
uvx black .  # Run black without installing globally
uvx --from 'mypy[reports]' mypy  # Run with extras

# Speed comparison
time pip install numpy pandas  # ~30 seconds
time uv add numpy pandas      # ~3 seconds (with cache)

2. Solving Virtual Environment Chaos

The Problem: Environment Collision

# Common disaster scenario
cd project1
source venv/bin/activate
pip install django==3.2

cd ../project2
# Forgot to deactivate/create new env
pip install django==4.2  # 💥 Conflicts!

Solution: Automated Environment Management

# .envrc file (using direnv)
layout python-uv

# Install direnv
brew install direnv  # macOS
sudo apt install direnv  # Ubuntu

# Add to shell config
echo 'eval "$(direnv hook bash)"' >> ~/.bashrc

# Allow the .envrc file
direnv allow

# Now environments activate automatically when you cd into project

Docker Development Environment

# Dockerfile.dev
FROM python:3.12-slim

# Install UV
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY pyproject.toml uv.lock ./

# Install dependencies
RUN uv sync --frozen

# Copy application code
COPY . .

# Run application
CMD ["uv", "run", "python", "main.py"]

# docker-compose.dev.yml
version: '3.8'
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - .:/app
      - uv-cache:/root/.cache/uv
    environment:
      - PYTHONPATH=/app
    ports:
      - "8000:8000"

volumes:
  uv-cache:

3. Corporate Environment Solutions

Problem: Corporate Proxies and Private Repositories

# Traditional pip configuration nightmare
mkdir -p ~/.pip
cat > ~/.pip/pip.conf << EOF
[global]
proxy = http://proxy.company.com:8080
trusted-host = pypi.company.com
index-url = https://pypi.company.com/simple/
extra-index-url = https://pypi.org/simple/
EOF

UV Corporate Configuration

# pyproject.toml
[tool.uv]
index-url = "https://pypi.company.com/simple/"
extra-index-url = ["https://pypi.org/simple/"]
trusted-host = ["pypi.company.com"]

[tool.uv.pip]
proxy = "http://proxy.company.com:8080"

# Environment variables for CI/CD
export UV_INDEX_URL="https://pypi.company.com/simple/"
export UV_EXTRA_INDEX_URL="https://pypi.org/simple/"
export UV_TRUSTED_HOST="pypi.company.com"

4. Security Best Practices

Dependency Scanning

# Install security scanner
uv add --dev safety bandit pip-audit

# Scan for vulnerabilities
uv run safety check
uv run pip-audit
uv run bandit -r src/

Lockfile Verification

# scripts/verify_deps.py
import hashlib
import json
from pathlib import Path

def verify_lockfile():
    """Verify that uv.lock hasn't been tampered with"""
    lock_file = Path("uv.lock")
    if not lock_file.exists():
        raise FileNotFoundError("uv.lock not found")

    # Check if lock file matches dependencies
    import subprocess
    result = subprocess.run(
        ["uv", "lock", "--dry-run"], 
        capture_output=True, 
        text=True
    )

    if result.returncode != 0:
        raise RuntimeError("Lock file verification failed")

    print("✅ Dependencies verified")

if __name__ == "__main__":
    verify_lockfile()

Supply Chain Protection

# Pin exact versions in CI
uv export --frozen > requirements-prod.txt

# Use hash verification
uv add "requests==2.31.0" --hash "sha256:..."

5. Team Collaboration Solutions

Preventing Lock File Conflicts

# scripts/sync_team.py
"""Team synchronization script"""
import subprocess
import sys
from pathlib import Path

def check_python_version():
    """Ensure team uses same Python version"""
    python_version_file = Path(".python-version")
    if not python_version_file.exists():
        print("❌ .python-version file missing")
        return False

    expected_version = python_version_file.read_text().strip()
    current_version = f"{sys.version_info.major}.{sys.version_info.minor}"

    if not current_version.startswith(expected_version.split('.')[0]):
        print(f"❌ Python version mismatch: expected {expected_version}, got {current_version}")
        return False

    print(f"✅ Python version {current_version} matches requirements")
    return True

def sync_dependencies():
    """Sync dependencies from lock file"""
    try:
        subprocess.run(["uv", "sync", "--frozen"], check=True)
        print("✅ Dependencies synced successfully")
        return True
    except subprocess.CalledProcessError:
        print("❌ Failed to sync dependencies")
        return False

def main():
    if not check_python_version():
        sys.exit(1)

    if not sync_dependencies():
        sys.exit(1)

    print("🎉 Team environment ready!")

if __name__ == "__main__":
    main()

Pre-commit Hooks for Dependency Management

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: uv-lock-check
        name: Check uv.lock is up to date
        entry: uv
        args: [lock, --locked]
        language: system
        files: ^(pyproject\.toml|uv\.lock)$

      - id: dependency-scan
        name: Scan dependencies for vulnerabilities
        entry: uv
        args: [run, safety, check]
        language: system
        pass_filenames: false

6. Performance Optimization

Cache Management

# Check cache size
uv cache info

# Clean cache (for debugging)
uv cache clean

# Preload common dependencies for team
uv cache prune  # Remove unused cache entries

CI/CD Optimization

# .github/workflows/test.yml
name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4

    - name: Install UV
      uses: astral-sh/setup-uv@v1

    - name: Set up Python
      run: uv python install

    - name: Cache UV dependencies
      uses: actions/cache@v3
      with:
        path: ~/.cache/uv
        key: uv-${{ runner.os }}-${{ hashFiles('uv.lock') }}

    - name: Install dependencies
      run: uv sync --frozen

    - name: Run tests
      run: uv run pytest

7. Migration Strategies

From pip to UV

# 1. Backup current setup
pip freeze > requirements-backup.txt

# 2. Initialize UV project
uv init --no-readme .

# 3. Import existing requirements
uv add $(cat requirements.txt | tr '\n' ' ')

# 4. Generate lock file
uv lock

# 5. Test installation
uv sync --frozen
uv run python -c "import sys; print(sys.path)"

From Poetry to UV

# migration_script.py
import toml
import subprocess

def migrate_poetry_to_uv():
    """Migrate from Poetry to UV"""

    # Read poetry configuration
    with open("pyproject.toml", "r") as f:
        poetry_config = toml.load(f)

    deps = poetry_config.get("tool", {}).get("poetry", {}).get("dependencies", {})
    dev_deps = poetry_config.get("tool", {}).get("poetry", {}).get("dev-dependencies", {})

    # Initialize UV project
    subprocess.run(["uv", "init", "--no-readme", "."], check=True)

    # Add regular dependencies
    for name, version in deps.items():
        if name != "python":
            if isinstance(version, str):
                subprocess.run(["uv", "add", f"{name}{version}"], check=True)
            else:
                subprocess.run(["uv", "add", name], check=True)

    # Add dev dependencies
    for name, version in dev_deps.items():
        if isinstance(version, str):
            subprocess.run(["uv", "add", "--dev", f"{name}{version}"], check=True)
        else:
            subprocess.run(["uv", "add", "--dev", name], check=True)

    print("✅ Migration completed!")
    print("🔍 Please review pyproject.toml and uv.lock")
    print("🧪 Test with: uv run python -m pytest")

if __name__ == "__main__":
    migrate_poetry_to_uv()

8. Troubleshooting Common Issues

Issue: Slow Installation on Corporate Networks

# Symptoms: UV takes forever to install packages
# Solution: Configure corporate settings

export UV_HTTP_TIMEOUT=300
export UV_RETRIES=3
export UV_INDEX_URL="https://your-corporate-pypi.com/simple/"

# Alternative: Use offline mode
uv sync --offline  # Use only cached packages

Issue: Lock File Conflicts in Git

# Symptoms: Constant uv.lock merge conflicts
# Solution: Automated resolution

# .gitattributes
uv.lock merge=union

# Or use automation
cat > .git/hooks/pre-commit << 'EOF'
#!/bin/bash
if git diff --cached --name-only | grep -q "pyproject.toml"; then
    uv lock
    git add uv.lock
fi
EOF
chmod +x .git/hooks/pre-commit

Issue: Platform-Specific Dependencies

# pyproject.toml
[tool.uv.dependency-groups]
windows = [
    "pywin32; platform_system=='Windows'",
]
linux = [
    "uvloop; platform_system=='Linux'",
]
macos = [
    "uvloop; platform_system=='Darwin'",
]

# Install platform-specific dependencies
uv sync --group $(python -c "import platform; print(platform.system().lower())")

9. Advanced Patterns

Monorepo Dependency Management

project/
├── services/
│   ├── api/
│   │   ├── pyproject.toml
│   │   └── uv.lock
│   └── worker/
│       ├── pyproject.toml
│       └── uv.lock
├── shared/
│   └── pyproject.toml
└── pyproject.toml  # Root project

# Root pyproject.toml
[tool.uv.workspace]
members = [
    "services/api",
    "services/worker",
    "shared"
]

Development vs Production Dependencies

# Development installation
uv sync --extra dev --extra test

# Production installation
uv sync --frozen --no-dev

# Minimal production image
uv export --no-dev --format requirements-txt > requirements-prod.txt

10. Best Practices Summary

DO's ✅

  1. Always use lock files: uv.lock ensures reproducible builds
  2. Pin Python versions: Use .python-version files
  3. Separate dev dependencies: Use --dev flag for development tools
  4. Regular security scans: Integrate safety and pip-audit
  5. Cache optimization: Use UV's built-in caching effectively
  6. Team synchronization: Use pre-commit hooks and sync scripts

DON'Ts ❌

  1. Never commit virtual environments: Use .gitignore properly
  2. Don't mix package managers: Pick UV and stick with it
  3. Avoid global installs: Use uvx for tools instead
  4. Don't ignore security alerts: Act on vulnerability reports
  5. Never skip lock file updates: Keep dependencies current
  6. Don't hardcode paths: Use relative paths and environment variables

Emergency Recovery Procedures

# Nuclear option: Complete environment reset
rm -rf .venv uv.lock
uv venv
uv sync

# Diagnose dependency conflicts
uv tree  # Show dependency tree
uv pip list --outdated  # Check for updates

# Restore from backup
git checkout HEAD~1 -- uv.lock
uv sync --frozen

Python dependency management doesn't have to be painful. With UV and these practices, you can achieve fast, secure, and reliable dependency management that scales with your team and project complexity.