Scrapling 🕷️ @Scrapling_dev

🕷️ Scrapling tip: spider callbacks are async GENERATORS — `async def` + `yield`, never `return`. `return item` closes the generator early; the item never reaches your results. Even a callback yielding ONE item must use yield, not return. ⚡

🕷️ Scrapling tip: wait=<ms> on browser fetchers — a FIXED sleep AFTER load_dom + network_idle + wait_selector all pass, right before the page closes StealthyFetcher.fetch(url, wait=2000) Catches late-firing analytics or lazy content past every other wait ⚡

🕷️ Scrapling tip: load_dom=True is the DEFAULT on browser fetchers — waits for DOMContentLoaded before returning Flip load_dom=False when the HTML shell has what you need (SSR, JSON-LD, meta tags) — response returns the moment navigation commits ⚡

🕷️ Scrapling tip: LinkExtractor rewrites every URL it extracts — canonicalize=True is the DEFAULT Sorts query params + normalizes the path so /a?b=1&a=2 collapses to /a?a=2&b=1 in dedup Flip canonicalize=False when raw URLs matter (signed URLs, tokens) ⚡

🕷️ Scrapling tip: `download_delay = 1.0` on your Spider — fixed wait BEFORE each request (default 0.0s). Global across domains, so pair with `concurrent_requests_per_domain` for per-host politeness. When robots_txt_obey is on, Crawl-delay is MAX'd on top ⚡

🕷️ Scrapling tip: every Fetcher (static, browser & sessions) accepts `retries=` and `retry_delay=` — built-in retry loop, default 3 attempts, 1s apart. Fetcher.get(url, retries=5, retry_delay=2) Skip the try/except-then-sleep boilerplate for flaky endpoints ⚡

Scrapling v0.4.10 is out 🎉 Use Scrapling's parsing API inside your existing Scrapy spiders now - one decorator on your callback and the response becomes a Scrapling Response. Plus custom browser support for the MCP server and a batch of fixes 👇 github.com/D4Vinci/Scrapl…

🕷️ Scrapling tip: adaptive fingerprints are bucketed PER DOMAIN A selector saved on web.archive.org WON'T relocate on the live site. adaptive_domain="site.com" on Fetcher.configure() buckets both under one key — survive URL changes AND redesigns 🎯

🕷️ Scrapling tip: network_idle=True — the SPA-safe wait Default load_dom fires at DOMContentLoaded, BEFORE XHRs settle. network_idle waits for 500ms of zero network traffic before returning. Chain it with wait_selector for bulletproof SPA scrapes ⚡

🕷️ Scrapling tip: pair ProxyRotator with a browser session and Chromium can't set a proxy per tab — Scrapling auto-forks a FRESH context per proxy (one tab each) and tears both down when the fetch ends. No shared tab pool across rotations ⚡

🕷️ Scrapling tip: adaptive scraping isn't only for CSS/XPath. The manual save/retrieve/relocate API fingerprints ANY element you find — by text, regex, similarity — then re-locates it after a site redesign 🎯

🕷️ Scrapling tip: override start_requests() to seed your spider with ANY Request — POST a login first, set custom headers, even pre-assign priority/sid. start_urls = [...] is just the default. Yield Request objects from start_requests() for full control ⚡

🕷️ Scrapling tip: extra_flags=[...] passes raw Chromium CLI flags to the browser at launch on StealthyFetcher/DynamicFetcher The escape hatch for browser tuning Scrapling doesn't surface as a named flag (--lang, --disable-features, GPU flags) ⚡

🕷️ Scrapling tip: stats.proxies — every proxy used in a crawl, auto-collected After a ProxyRotator run, audit which proxies actually touched the crawl — straight off the final CrawlStats dump. result.stats.proxies # ['http://p1:8080','http://p2:8080'] ⚡

🕷️ Scrapling tip: find_by_text() defaults to case-INSENSITIVE matching — surprising default for a Python API page.find_by_text("the", partial=True) → also matches "The Boys", "The Requiem Red" Flip case_sensitive=True for exact-case matches ⚡

🕷️ Scrapling tip: LinkExtractor reads <a href> + <area href> by default — but tags AND attrs are both tunable Pull URLs from <link rel=alternate>, lazy-loaded <img data-src>, <button data-href>, or any (tag, attr) pair you need ⚡

🕷️ Scrapling tip: additional_args on StealthyFetcher/DynamicFetcher forwards ANY Playwright context kwarg — geolocation, permissions, viewport, color_scheme. Higher priority than Scrapling's own settings. Pair with locale + timezone_id to spoof GPS too ⚡

🕷️ Scrapling tip: `allowed_domains` on Spider silently drops off-domain links — but the count lives in `stats.offsite_requests_count`, so you can see how leaky a crawl was. Subdomains match for free (api.example.com → example.com) ⚡

🕷️ Scrapling tip: LinkExtractor(restrict_css=...) scopes link extraction to a region of the DOM links = LinkExtractor( allow=r"/posts/", restrict_css="main", ) links.extract(response) Skip nav/footer/sidebar links entirely — no chaining ⚡

🕷️ Scrapling tip: every Selector has .urljoin() built in — resolves a relative href to an absolute URL using the page's own URL as the base. No urllib.parse import needed: page.urljoin(el.attrib['href']) # → absolute URL ⚡