Valstorm Logo
VALSTORM
Back to All Free Tools
Media & Web Microservices PolyForm Noncommercial License 1.0.0

Valstorm Web Intelligence & Scraper (valstorm-web-scraper)

High-performance web scraping, domain crawling, RSS monitoring, and visual diff microservice with Python 3.14 native Zstandard compression.

Convert any public webpage, competitor pricing page, or documentation site into clean, token-optimized Markdown. Uses a 3-tier waterfall pipeline (Fast HTTP/2 -> Stealth Playwright Chromium -> Firecrawl Fallback), SSRF IP guards, Valkey single-flight locks, and Python 3.14 native zstd compression (70–85% token reduction).

View Source on GitHub (valstorm-web-scraper) Run Managed on Valstorm Cloud ($25/mo)
3-Tier Waterfall (Fast HTTP/2 -> Stealth Playwright -> Cloud Fallback)
Native Python 3.14 Zstandard Compression (PEP 784) with 70–85% Token Savings
Built-In SSRF Protection Blocking Loopback, Private & Cloud Metadata IPs
Live Web Content Diffing (/v1/diff), Domain Crawling (/v1/crawl) & RSS Polling
Section 01 • Installation & Quickstart

1. Start the Scraper Microservice & Convert URLs to Clean Markdown 

# 1. Clone and install dependencies with uv
git clone https://github.com/Valstorm-Corp/valstorm-web-scraper.git
cd valstorm-web-scraper && uv sync

# 2. Launch the FastAPI scraper service on port 8661
uv run uvicorn app.main:app --host 0.0.0.0 --port 8661

# 3. Scrape a URL through the 3-tier waterfall and ingest into a VFS Vault
curl -X POST http://localhost:8661/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://stripe.com/pricing", "ingest_to_vfs": true}'
Section 02 • Architecture & Data Flow

2. How valstorm-web-scraper Works Under the Hood 

  🌐 Target URL / Domain / RSS Feed
          │
          ▼  (SSRF Guard: Blocks 127.0.0.0/8, 10.0.0.0/8, 169.254.169.254)
  ┌──────────────────────────────────────────────────────────┐
  │ 3-TIER WATERFALL EXTRACTION PIPELINE                     │
  │  Tier 1: Fast Async HTTP/2 (httpx + Brotli/Gzip)         │
  │     ↓ (If JS challenge or SPA detected)                  │
  │  Tier 2: Stealth Headless Playwright (Canvas spoofing)   │
  │     ↓ (If Cloudflare Turnstile / Akamai block)           │
  │  Tier 3: Firecrawl Cloud API Fallback                    │
  └──────────────────────────────────────────────────────────┘
          │
          ▼
  🗜️ Python 3.14 Native Zstandard (PEP 784) + Valkey L1 Cache
     • 70–85% Token Reduction + Direct VFS Knowledge Vault Sink
Section 03 • Engineering Capabilities

3. Core Capabilities & Technical Specifications

3-Tier Waterfall Pipeline

Attempts sub-second async HTTP/2 first, escalates automatically to stealth headless Chromium for client-rendered SPAs, and falls back to Firecrawl only when anti-bot challenges require it.

Native Python 3.14 Zstandard (PEP 784) Archival

Compresses cleaned Markdown and HTML snapshots using Python 3.14 native zstd with Valkey single-flight deduplication locks.

Competitor & Documentation Change Diffing (/v1/diff)

Compares live webpages against permanently archived snapshots to detect pricing changes, feature updates, or policy modifications.

Direct VFS Knowledge Vault Ingestion

Automatically writes scraped Markdown documents directly into Valstorm Knowledge Vaults via valstorm-client with ServiceKeyAuth.

Section 04 • Dual-License Model

4. Open-Source & Commercial Licensing

Personal, Academic & Noncommercial Use

PolyForm Noncommercial License 1.0.0

Free to clone, inspect, modify, and run for personal projects, educational study, and noncommercial research under the PolyForm Noncommercial License 1.0.0.

Commercial & Enterprise Use

Valstorm Commercial License

Commercial use in business operations is included automatically with any active Valstorm BOS workspace ($25/mo in usage credits with $0/seat tax) or available via enterprise BYOC licensing.

5. Frequently Asked Questions

How does valstorm-web-scraper prevent Server-Side Request Forgery (SSRF)?

Before any socket connection opens, the SSRF guard resolves DNS records and blocks all loopback, private network (RFC 1918), link-local, and cloud metadata IP ranges (such as 169.254.169.254).

Explore Other Valstorm Open-Source Packages

View All 7 Tools →
valstorm-agent

Enterprise-grade autonomous AI execution runtime, ReAct loop, and parallel subagent swarm engine for developer workstations, CI pipelines, and remote team automation.

valstorm-cli

Command-line interface for Valstorm developers to authenticate via OAuth2 PKCE, pull and push object schemas as code, and manage local workspaces.

valstorm-mcp

Stateless Streamable HTTP, SSE, and stdio Model Context Protocol (MCP) server exposing 16 granularly scoped Valstorm VFS, SQL, and CRM tools.

valstorm-video-worker

Dedicated asynchronous microservice for multi-preset FFmpeg video transcoding, Deepgram/Whisper speech-to-text captions, and AI chapter generation.

valstorm-client

Official typed Python 3.13+ SDK with 1:1 PlatformContext parity across records, SQL/GraphQL queries, VFS Knowledge Vaults, schemas, and sandboxes.

valstorm-sanitizer

Pure Python standard-library secret redaction engine that strips API keys, JWTs, database passwords, and environment secrets before logs or LLM prompts egress.