# Valstorm Web Intelligence & Scraper (valstorm-web-scraper)

> High-performance web scraping, domain crawling, RSS monitoring, and visual diff microservice with Python 3.14 native Zstandard compression.

- **GitHub Repository:** https://github.com/Valstorm-Corp/valstorm-web-scraper
- **License:** PolyForm Noncommercial License 1.0.0 (https://polyformproject.org/licenses/noncommercial/1.0.0)
- **Installation:** `git clone https://github.com/Valstorm-Corp/valstorm-web-scraper.git && cd valstorm-web-scraper && uv sync`
- **Language/Stack:** Python 3.14 (FastAPI, HTTPX, Playwright, Zstd PEP 784)
- **Web Documentation:** https://www.valstorm.com/free-tools/valstorm-web-scraper

## Key Capabilities
- **3-Tier Waterfall Pipeline:** Attempts sub-second async HTTP/2 first, escalates automatically to stealth headless Chromium for client-rendered SPAs, and falls back to Firecrawl only when anti-bot challenges require it.
- **Native Python 3.14 Zstandard (PEP 784) Archival:** Compresses cleaned Markdown and HTML snapshots using Python 3.14 native zstd with Valkey single-flight deduplication locks.
- **Competitor & Documentation Change Diffing (/v1/diff):** Compares live webpages against permanently archived snapshots to detect pricing changes, feature updates, or policy modifications.
- **Direct VFS Knowledge Vault Ingestion:** Automatically writes scraped Markdown documents directly into Valstorm Knowledge Vaults via valstorm-client with ServiceKeyAuth.

## Quickstart
```bash
# 1. Clone and install dependencies with uv
git clone https://github.com/Valstorm-Corp/valstorm-web-scraper.git
cd valstorm-web-scraper && uv sync

# 2. Launch the FastAPI scraper service on port 8661
uv run uvicorn app.main:app --host 0.0.0.0 --port 8661

# 3. Scrape a URL through the 3-tier waterfall and ingest into a VFS Vault
curl -X POST http://localhost:8661/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://stripe.com/pricing", "ingest_to_vfs": true}'
```