Advanced Web Scraping Portfolio

A collection of production-grade Scrapy spiders demonstrating expertise in bypassing sophisticated anti-bot protection mechanisms for e-commerce data extraction.

🎯 Project Overview

This repository showcases three advanced web scraping implementations that overcome modern security challenges:

1. Esselunga Web Spider - Google reCAPTCHA Enterprise v3 bypass
2. Esselunga Mobile App Spider - SSL Certificate Pinning circumvention
3. Intermarche Spider - DataDome protection evasion

Each spider represents real-world solutions to complex anti-scraping challenges commonly faced in enterprise-scale data collection.

---

📁 Repository Structure

├── google_recaptcha_enterprise_protected_website_and_ssl_pinning_protected_mobile_app/
│   ├── esselunga_captcha_spider.py          # Web spider with reCAPTCHA bypass
│   ├── esselunga_app_prices_scrapy_spider.py # Mobile app SSL pinning bypass
│   └── utils/
│       ├── esselunga_service.py             # CAPTCHA solving & API helpers
│       ├── esselunga_client.py              # Custom SSL context factory
│       ├── esselunga_mobile_service.py      # Mobile app authentication
│       └── esselunga_types.py               # Type definitions
│
├── datadom_protected_ecom_chain/
│   ├── intermarche_spider.py                # DataDome bypass implementation
│   ├── intermarche_stores_update.py         # Store database management
│   └── utils/
│       ├── intermarche_service.py           # Category parsing & validation
│       ├── intermarche_constants.py         # Headers & configuration
│       └── intermarche_types.py             # Type definitions
│
└── CLAUDE.md                                 # Technical documentation

---

🛡️ Protection Mechanisms & Solutions

1️⃣ Esselunga Web Spider - Google reCAPTCHA Enterprise v3

Challenge: Esselunga's e-commerce platform is protected by Google reCAPTCHA Enterprise v3, which analyzes user behavior patterns and blocks automated requests with high confidence scores.

Solution Highlights:

• CAPTCHA Solving Integration: Implemented CapSolver API integration for automated token generation
• Token Lifecycle Management: Handles anchor and reload tokens with proper refresh cycles
• Session Persistence: Maintains XSRF-ECOM-TOKEN across multi-step authentication flows
• Proxy Rotation: Uses residential proxies to distribute requests and avoid IP-based detection
• Cookie Orchestration: Manages complex cookie requirements (JSESSIONID, GUEST_TROLLEY, XSRF-ECOM-TOKEN)

Technical Implementation:

# CAPTCHA solving with retry mechanism (max 20 attempts)
captcha_token = get_esselunga_captcha_token(
    captcha_service=self.capsolve_service,
    proxy=proxy,
    logger=self.logger,
    cookies=cookies,
)

# Multi-step authentication flow
1. Visit store endpoint with captcha token
2. Extract XSRF token from response cookies
3. Fetch product categories with authenticated session
4. Paginate through products maintaining session state

Key Features:

• Handles both "Shopping at Home" and "Order & Collect" store types
• Product variant support (parent-child relationships)
• Detailed product information extraction (EAN, ingredients, allergens, origin)
• Intelligent EAN-based deduplication to skip already-scraped products

---

2️⃣ Esselunga Mobile App Spider - SSL Certificate Pinning Bypass

Challenge: Esselunga's mobile app implements SSL certificate pinning, rejecting any HTTPS connection that doesn't present the exact expected certificate. Standard proxy/debugging tools are blocked at the TLS layer.

Solution Highlights:

• Certificate Extraction: Reverse-engineered the app to extract pinned public/private key pairs
• Custom SSL Context: Implemented CustomClientContextFactory that presents the expected certificates
• TLS Fingerprinting: Mimics mobile app's TLS handshake using curl_cffi with Android Chrome impersonation
• Header Fingerprinting: Replicates exact mobile app headers including device identifiers

Technical Implementation:

# Custom SSL context factory configuration
"DOWNLOADER_CLIENTCONTEXTFACTORY": "GdoChainPrices.spiders.italy.esselunga.utils.esselunga_client.CustomClientContextFactory"

# Mobile app header replication
common_headers = {
    "user-agent": "okhttp/3.14.9",
    "x-app-build": "409025",
    "x-app-version": "4.0.9",
    "x-client-type": "EsselungaApp20",
    "x-device-model": "SM-G991B",  # Samsung Galaxy S21
    "x-installation-identifier": "9a4293dc-a443-452b-a2ab-28fc3d649156",
    # ... additional fingerprinting headers
}

Cart-Based Scraping Strategy:

1. QR code authentication to obtain store session
2. Extract store nickname from onboarding response
3. Iteratively add products to cart by EAN
4. Parse cart responses for pricing data
5. Remove products to prevent cart overflow
6. Track progress: products_processed, products_with_data, products_without_data

Key Achievements:

• Successfully bypassed SSL pinning without requiring a rooted device
• Achieved stable scraping of 1000+ products per session
• Maintained session integrity across sequential requests

---

3️⃣ Intermarche Spider - DataDome Protection Evasion

Challenge: Intermarche implements DataDome's enterprise anti-bot solution, which uses advanced behavioral analysis, device fingerprinting, and rate limiting (≈30 requests per IP per 8 minutes).

Solution Highlights:

• Dynamic Cookie Generation: Builds store-specific cookies from metadata (city, store type, chain ID)
• Request Timing Optimization: Configured concurrent requests and delays to stay under detection threshold
• Header Consistency: Maintains consistent header patterns across request types
• Smart Rate Limiting: Implements CONCURRENT_REQUESTS: 3 with DOWNLOAD_DELAY: 0 for optimal throughput

Technical Implementation:

# Dynamic cookie construction
def get_store_static_cookie(self) -> dict[str, str]:
    itm_pdv = {
        "ref": self.chain_store_identifier,
        "isEcommerce": True,
        "name": f"{self.store_type} {city}".title(),
        "city": city
    }
    novaParams = {"pdvRef": self.chain_store_identifier}

    # URL-encode with specific format to match browser behavior
    return {
        "itm_pdv": json.dumps(itm_pdv).replace('"', '%22').replace(',', '%2C'),
        "novaParams": json.dumps(novaParams).replace('"', '%22').replace(',', '%2C')
    }

Multi-Step Flow:

1. Store Selection: POST to cart synchronization endpoint with customerDateTime
2. Category Discovery: Fetch hierarchical categories (maxDepth=3) and extract leaf nodes
3. Pagination Strategy: First page determines total products, subsequent pages scraped in parallel
4. Product Processing: Complex price/promotion logic including variable weight handling

Advanced Features:

• Promotion Detection: Handles 5+ promotion types (immediate discount, multi-buy, quantity discount, long-term pricing)
• Variable Weight Products: Special logic for "en vrac" items with per-kg/per-liter pricing
• Category Statistics: Optional stats tracking with total_match validation
• Packaging Parser: Regex-based extraction of weight/UOM from multilingual descriptions

Price & Promotion Intelligence:

# Multi-buy promotion example: "3x2" (buy 2 get 1 free)
if promo_type == "offeredDiscount":
    Q = first_promo.get("quantity")  # 2
    V = first_promo.get("value")     # 1
    promo_description = f"{Q + V}x{Q}"  # "3x2"

# Percentage discount on Nth item: "-30% on 2nd item"
if promo_type == "quantityDiscount":
    factor = 1 - ((discount_percentage / 100) / Q)
    promo_description = f"{Q}x({Q}*{factor:.3f})"

---

🚀 Running the Spiders

Prerequisites

pip install scrapy curl-cffi peewee

Environment Variables:

export CAPSOLVER_API_KEY="your_capsolver_key"
export SCRAPOXY_PROXY_MULTICOUNTRY_URL="http://your_proxy:port"
export SCRAPOXY_PROXY_FRANCE_URL="http://your_france_proxy:port"

Execution Examples

Esselunga Web (reCAPTCHA):

scrapy crawl esselunga -a store_id=6428 -a category_limit=5 -O esselunga_output.json

Esselunga Mobile App (SSL Pinning):

scrapy crawl esselunga_app_prices_scrapy \
  -a qr_code_data="https://app.services.esselunga.it/prestospesa/onBoarding.html?token=YOUR_TOKEN" \
  -O mobile_prices.json

Intermarche (DataDome):

scrapy crawl intermarche -a store_id=6869 -a category_limit=10 -O intermarche_output.json

Additional Options:

• pages_limit: Limit pagination per category
• target_category: Scrape specific category only (Intermarche)
• step: Pagination step size for Esselunga (default: 15)

---

📊 Sample Output Structure

{
  "product_id": "123456",
  "ean": "8001234567890",
  "name": "Organic Tomatoes",
  "brand": "BioBrand",
  "base_price": 299,
  "promo_price": 249,
  "promo_name": "Immediate Discount",
  "promo_description": "-16%",
  "ending_date": "2024-12-31",
  "weight": 500.0,
  "uom": "g",
  "variable_weight": false,
  "images": ["https://example.com/image1.jpg"],
  "ingredients": "Tomatoes, Salt",
  "allergen": "May contain traces of celery",
  "origin": "Italy",
  "stock": 150,
  "chain_cat": {
    "cat1": "Fresh Produce",
    "cat2": "Vegetables",
    "cat3": "Tomatoes"
  },
  "require_fidelity": false,
  "store_id": 6428,
  "chain_id": 1
}

---

🔧 Technical Highlights

Architecture Patterns

1. Service Layer Separation

• Clean separation between spider logic and business logic
• Reusable service modules for API interactions, parsing, and validation
• Type-safe interfaces using TypedDict definitions

2. Session Management

• Custom CurlCffiSession wrapper with automatic retry logic
• Browser impersonation using curl_cffi (Chrome 136)
• Proxy rotation with error handling

3. Error Handling & Validation

• Product-level validation with automatic total_products adjustment
• Price validation (converts to cents, validates positive values)
• Duplicate detection using sets
• Detailed logging for debugging

4. Concurrency Control

• Configured per-spider based on target's rate limits
• Retry mechanisms with exponential backoff
• Proxy middleware for distributed requests

Code Quality

• Type Hints: Comprehensive typing using TypedDict for complex data structures
• Logging: Structured logging with progress tracking and color-coded messages
• Configuration: Externalized settings via custom_settings and environment variables
• Documentation: Inline comments explaining complex logic (promotion detection, packaging parsing)

---

🎓 Key Learnings & Challenges

CAPTCHA Bypass

• Understanding reCAPTCHA Enterprise's multi-layered verification
• Implementing token refresh cycles to maintain session validity
• Balancing solve time vs. request throughput

SSL Pinning

• Reverse engineering mobile apps to extract cryptographic materials
• TLS fingerprint matching at the protocol level
• Maintaining compatibility with server-side certificate rotation

DataDome Evasion

• Identifying behavioral patterns that trigger blocks
• Optimizing request timing to appear human-like
• Dynamic cookie generation based on store context

General Scraping

• Handling multilingual content (French/Italian)
• Complex promotion logic with multiple edge cases
• Variable weight product pricing calculations
• Category tree traversal and leaf node extraction

---

📝 Notes

Ethical Considerations: These spiders were developed for legitimate price comparison and market research purposes. The techniques demonstrated are for educational and professional portfolio purposes. Always ensure compliance with applicable laws and website terms of service.

Production Considerations:

• Implement proper proxy rotation pools for scale
• Add database persistence layer (current implementation uses Peewee ORM)
• Set up monitoring and alerting for protection mechanism changes
• Consider CAPTCHA solving cost optimization strategies

---

📫 Contact

This portfolio demonstrates advanced capabilities in:

• Anti-bot bypass techniques
• Mobile app reverse engineering
• Large-scale data extraction
• Production-grade code architecture

For questions or collaboration opportunities, please reach out through your preferred channel.

---

🔒 License

This code is provided for portfolio demonstration purposes. Commercial use requires appropriate licensing and legal compliance.