尧图网页设计 
Crawl4AI 内容选择实战指南用 CrawlerRunConfig 精准控制 CSS 选取、内容过滤与 Shadow DOM 扁平化【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai在 Crawl4AI 中CrawlerRunConfig是控制“爬什么、留什么、扔什么”的核心配置对象。本篇围绕官方文档 content-selection.md 展开系统讲解四类能力用css_selector/target_elements圈定内容区域用一组exclude_*参数过滤标签、链接、域名和媒体用process_iframes与flatten_shadow_dom处理内嵌内容与 Web Components以及用 CSS 或 LLM 抽取策略把过滤后的 HTML 变成结构化 JSON。读完本文你可以复制文中示例直接运行并对照 CrawlerRunConfig 源码 理解每个参数的默认值与底层实现。1. 基于 CSS 的内容选取Crawl4AI 提供两种选取方式css_selector把整页裁剪到某个区域和更灵活的target_elements只让 Markdown 生成与结构化抽取聚焦指定元素同时保留整页上下文。1.1 使用css_selector将抓取结果限制在页面某个区域最直接的方式就是设置css_selector只有匹配该选择器的元素会保留在result.cleaned_html中import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( # e.g., first 30 items from Hacker News css_selector.athing:nth-child(-n30) ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://news.ycombinator.com/newest, configconfig ) print(Partial HTML length:, len(result.cleaned_html)) if __name__ __main__: asyncio.run(main())从源码看css_selector在 CrawlerRunConfig 中的默认值是None即处理整页它会在内容清洗与 Markdown 生成之前的 HTML 预处理阶段生效作用范围覆盖所有抽取过程。1.2 使用target_elementstarget_elements接受一个 CSS 选择器列表允许同时指定多个目标元素。它的语义与css_selector有本质区别import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( # Target article body and sidebar, but not other content target_elements[article.main-content, aside.sidebar] ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.com/blog-post, configconfig ) print(Markdown focused on target elements) print(Links from entire page still available:, len(result.links.get(internal, []))) if __name__ __main__: asyncio.run(main())关键差异设置target_elements后Markdown 生成与结构化数据抽取只聚焦这些元素但链接、图片、表格等仍从整页提取。这让你在控制 Markdown 内容的同时保留完整的链接分析与媒体采集上下文。这个区别在 配置类文档字符串 中有明确描述css_selector会“把初始 raw HTML 收缩到选定元素”而target_elements只影响抽取与 Markdown 生成不裁剪原始 HTML。源码中该参数默认归一化为空列表self.target_elements target_elements or []async_configs.py空列表即表示处理整页。2. 内容过滤与排除2.1 参数总览CrawlerRunConfig提供一整组过滤参数可按需组合config CrawlerRunConfig( # Content thresholds word_count_threshold10, # Minimum words per block # Tag exclusions excluded_tags[form, header, footer, nav], # Link filtering exclude_external_linksTrue, exclude_social_media_linksTrue, # Block entire domains exclude_domains[adtrackers.com, spammynews.org], exclude_social_media_domains[facebook.com, twitter.com], # Media filtering exclude_external_imagesTrue )各参数的作用与当前仓库中的默认值如下参数默认值作用word_count_thresholdMIN_WORD_THRESHOLD忽略字数低于阈值的文本块跳过过短的导航/免责声明excluded_tags[]整体移除指定标签form、header、footer等exclude_external_linksFalse剥离外部链接并可能从result.links中移除exclude_social_media_linksFalse移除指向已知社媒域名的链接exclude_domains[]自定义域名黑名单命中即从链接中剔除exclude_social_media_domainsSOCIAL_MEDIA_DOMAINS社媒域名列表可覆盖或扩充exclude_external_imagesFalse丢弃非本站域名含子域名托管的图片关于word_count_threshold的默认值需要注意源码 async_configs.py 中其取值为MIN_WORD_THRESHOLD而该常量在当前仓库 config.py 中定义为1即默认几乎不过滤短块如需过滤噪音块应显式设置如 10、20。exclude_social_media_domains默认取自 config.py 中的 SOCIAL_MEDIA_DOMAINS与文档列出的清单一致[ facebook.com, twitter.com, x.com, linkedin.com, instagram.com, pinterest.com, tiktok.com, snapchat.com, reddit.com, ]链接过滤的底层实现在 LXMLWebScrapingStrategy._process_element 中每个a href会经normalize_url归一化后判定内外部当exclude_external_links为 True 或链接域名命中exclude_domains时代码直接执行link.getparent().remove(link)即从 HTML 树中删除该节点——这就是“外部链接会被剥离”的实现来源。2.2 组合使用示例import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode async def main(): config CrawlerRunConfig( css_selectormain.content, word_count_threshold10, excluded_tags[nav, footer], exclude_external_linksTrue, exclude_social_media_linksTrue, exclude_domains[ads.com, spammytrackers.net], exclude_external_imagesTrue, cache_modeCacheMode.BYPASS ) async with AsyncWebCrawler() as crawler: result await crawler.arun(urlhttps://news.ycombinator.com, configconfig) print(Cleaned HTML length:, len(result.cleaned_html)) if __name__ __main__: asyncio.run(main())提示如果这些参数过滤掉了过多内容请相应调低阈值或关闭对应开关。另外从源码默认值看cache_mode的默认即为CacheMode.BYPASSasync_configs.py示例中显式写出是为了演示意图清晰。3. 处理 Iframe部分站点把内容嵌在iframe中。若希望将其内联进最终输出设置process_iframesTrue同时可配合remove_overlay_elements与remove_consent_popups清理弹窗config CrawlerRunConfig( # Merge iframe content into the final output process_iframesTrue, remove_overlay_elementsTrue, # Remove GDPR/cookie consent popups (OneTrust, Cookiebot, etc.) remove_consent_popupsTrue )完整用法import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( process_iframesTrue, remove_overlay_elementsTrue ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.org/iframe-demo, configconfig ) print(Iframe-merged length:, len(result.cleaned_html)) if __name__ __main__: asyncio.run(main())从 CrawlerRunConfig 文档字符串 可以看到这三个开关的默认值均为False且remove_consent_popups专门针对 IAB TCF/CMP 标准可识别 OneTrust、Cookiebot、TrustArc、Quantcast、Didomi 等主流 CMP 提供商的弹层。4. 扁平化 Shadow DOM使用 Web Components 构建的站点Stencil、Lit、Shoelace、Angular Elements 等会把内容渲染在 Shadow DOM 中——一个被封装的子树常规的页面序列化如 Playwright 的page.content()完全看不到它。设置flatten_shadow_domTrue后Crawl4AI 会遍历所有 shadow tree、解析slot投影最终产出一份扁平化的单一 HTML 文档config Crawl4AI_CrawlerRunConfig_placeholderconfig CrawlerRunConfig( # Flatten shadow DOM into the main document flatten_shadow_domTrue, # Give web components time to hydrate wait_untilload, delay_before_return_html3.0, )完整示例——抓取一个规格参数藏在 shadow root 里的产品页import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( flatten_shadow_domTrue, wait_untilload, delay_before_return_html3.0, ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011, configconfig, ) # Without flatten_shadow_dom: ~1 KB of markdown (breadcrumbs only) # With flatten_shadow_dom: ~33 KB (full product specs, downloads, etc.) print(len(result.markdown.raw_markdown)) if __name__ __main__: asyncio.run(main())实现细节当flatten_shadow_domTrue时AsyncWebCrawlerStrategy 会加载并注入 flatten_shadow_dom.js 初始化脚本该脚本会 patchElement.prototype.attachShadow强制打开mode: closed的 shadow root使闭包组件同样可访问。提示Web 组件需要 JavaScript 运行完成才会渲染内容即hydration。因此建议wait_untilload并配合 2–5 秒的delay_before_return_html确保组件完成水合后再执行扁平化。注意wait_until的默认值是domcontentloadedasync_configs.py若沿用默认值可能赶在组件渲染之前抓取。仓库中提供了完整可运行的示例 shadow_dom_crawling.py可配合 tests/general/test_flatten_shadow_dom.py 查看对应测试覆盖。5. 结构化抽取示例内容选择可以进一步与高级抽取策略组合CSS 或 LLM 抽取策略都运行在经过过滤/裁剪后的 HTML 上。5.1 基于模式的 JsonCssExtractionStrategyimport asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai import JsonCssExtractionStrategy async def main(): # Minimal schema for repeated items schema { name: News Items, baseSelector: tr.athing, fields: [ {name: title, selector: span.titleline a, type: text}, { name: link, selector: span.titleline a, type: attribute, attribute: href } ] } config CrawlerRunConfig( # Content filtering excluded_tags[form, header], exclude_domains[adsite.com], # CSS selection or entire page css_selectortable.itemlist, # No caching for demonstration cache_modeCacheMode.BYPASS, # Extraction strategy extraction_strategyJsonCssExtractionStrategy(schema) ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://news.ycombinator.com/newest, configconfig ) data json.loads(result.extracted_content) print(Sample extracted item:, data[:1]) # Show first item if __name__ __main__: asyncio.run(main())5.2 基于 LLM 的抽取import asyncio import json from pydantic import BaseModel, Field from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig from crawl4ai import LLMExtractionStrategy class ArticleData(BaseModel): headline: str summary: str async def main(): llm_strategy LLMExtractionStrategy( llm_config LLMConfig(provideropenai/gpt-4,api_tokensk-YOUR_API_KEY) schemaArticleData.schema(), extraction_typeschema, instructionExtract headline and a short summary from the content. ) config CrawlerRunConfig( exclude_external_linksTrue, word_count_threshold20, extraction_strategyllm_strategy ) async with AsyncWebCrawler() as crawler: result await crawler.arun(urlhttps://news.ycombinator.com, configconfig) article json.loads(result.extracted_content) print(article) if __name__ __main__: asyncio.run(main())在这个流程中爬虫依次执行过滤外部链接exclude_external_linksTrue忽略超短文本块word_count_threshold20把最终 HTML 交给 LLM 策略做 AI 驱动的解析。注意CrawlerRunConfig构造时会校验extraction_strategy必须是ExtractionStrategy的实例否则抛出ValueErrorasync_configs.py自定义策略需先继承该基类。6. 综合示例选取 过滤 抽取三合一下面的函数把CSS 选取、排除逻辑和模式化抽取统一起来演示如何精细调整最终数据import asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai import JsonCssExtractionStrategy async def extract_main_articles(url: str): schema { name: ArticleBlock, baseSelector: div.article-block, fields: [ {name: headline, selector: h2, type: text}, {name: summary, selector: .summary, type: text}, { name: metadata, type: nested, fields: [ {name: author, selector: .author, type: text}, {name: date, selector: .date, type: text} ] } ] } config CrawlerRunConfig( # Keep only #main-content css_selector#main-content, # Filtering word_count_threshold10, excluded_tags[nav, footer], exclude_external_linksTrue, exclude_domains[somebadsite.com], exclude_external_imagesTrue, # Extraction extraction_strategyJsonCssExtractionStrategy(schema), cache_modeCacheMode.BYPASS ) async with AsyncWebCrawler() as crawler: result await crawler.arun(urlurl, configconfig) if not result.success: print(fError: {result.error_message}) return None return json.loads(result.extracted_content) async def main(): articles await extract_main_articles(https://news.ycombinator.com/newest) if articles: print(Extracted Articles:, articles[:2]) # Show first 2 if __name__ __main__: asyncio.run(main())为什么有效用#main-content做 CSS 域限定多个exclude_参数移除无关域名、外部图片等JsonCssExtractionStrategy解析重复的文章块。7. 抓取模式LXMLWebScrapingStrategy 与自定义策略Crawl4AI 默认使用基于 LXML 的LXMLWebScrapingStrategy作为 HTML 内容处理策略对大型 HTML 文档有出色性能表现from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LXMLWebScrapingStrategy async def main(): # Default configuration already uses LXMLWebScrapingStrategy config CrawlerRunConfig() # Or explicitly specify it if desired config_explicit CrawlerRunConfig( scraping_strategyLXMLWebScrapingStrategy() ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.com, configconfig )从 CrawlerRunConfig 初始化代码 可确认默认值self.scraping_strategy scraping_strategy or LXMLWebScrapingStrategy()即不传参时自动采用 LXML 策略。向后兼容方面content_scraping_strategy.py 末尾 显式声明WebScrapingStrategy LXMLWebScrapingStrategy旧代码无需任何修改即可继续工作。你也可以继承ContentScrapingStrategy编写自定义策略必须返回ScrapingResult对象from crawl4ai import ContentScrapingStrategy, ScrapingResult, MediaItem, Media, Link, Links class CustomScrapingStrategy(ContentScrapingStrategy): def scrap(self, url: str, html: str, **kwargs) - ScrapingResult: # Implement your custom scraping logic here return ScrapingResult( cleaned_htmlhtml.../html, # Cleaned HTML content successTrue, # Whether scraping was successful mediaMedia( images[ # List of images found MediaItem( srchttps://example.com/image.jpg, altImage description, descSurrounding text, score1, typeimage, group_id1, formatjpg, width800 ) ], videos[], # List of videos (same structure as images) audios[] # List of audio files (same structure as images) ), linksLinks( internal[ # List of internal links Link( hrefhttps://example.com/page, textLink text, titleLink title, base_domainexample.com ) ], external[] # List of external links (same structure) ), metadata{ # Additional metadata title: Page Title, description: Page description } ) async def ascrap(self, url: str, html: str, **kwargs) - ScrapingResult: # For simple cases, you can use the sync version return await asyncio.to_thread(self.scrap, url, html, **kwargs)性能考量LXML 策略在处理大型 HTML尤其 100KB时性能优异官方描述其相比 BeautifulSoup 方案最快可达 10–20 倍同时具备内存占用低、对规整 HTML 处理稳健、表格检测与提取能力强等优势。8. 组合 CSS 选取方法的进阶玩法css_selector与target_elements可以组合使用实现细粒度控制import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode async def main(): # Target specific content but preserve page context config CrawlerRunConfig( # Focus markdown on main content and sidebar target_elements[#main-content, .sidebar], # Global filters applied to entire page excluded_tags[nav, footer, header], exclude_external_linksTrue, # Use basic content thresholds word_count_threshold15, cache_modeCacheMode.BYPASS ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.com/article, configconfig ) print(fContent focuses on specific elements, but all links still analyzed) print(fInternal links: {len(result.links.get(internal, []))}) print(fExternal links: {len(result.links.get(external, []))}) if __name__ __main__: asyncio.run(main())这种组合兼得两者之长Markdown 生成与内容抽取聚焦你关心的元素链接、图片等页面数据仍保留整页上下文内容过滤参数全局生效。9. 关键参数速查与结论把target_elements / css_selector 域限定、内容过滤参数与高级抽取策略混合使用你可以精确决定保留哪些数据。CrawlerRunConfig中用于内容选择的关键参数汇总如下target_elements— CSS 选择器数组聚焦 Markdown 生成与数据抽取同时为链接和媒体保留整页上下文css_selector— 基础域限定对全部抽取流程生效word_count_threshold— 跳过短块注意当前仓库默认MIN_WORD_THRESHOLD 1需显式调高才有过滤效果excluded_tags— 整体移除指定 HTML 标签exclude_external_links、exclude_social_media_links、exclude_domains— 过滤不需要的链接或域名exclude_external_images— 移除外部来源图片process_iframes— 按需内联 iframe 内容flatten_shadow_dom— 将 Shadow DOM 扁平化进主文档覆盖 Web Components 场景。再叠加 CSS、LLM 等结构化抽取策略即可构建出从 raw/cleaned HTML 到精密 JSON 的完整数据管线。更多参数细节可查阅 Configuration Reference。【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Crawl4AI 内容选择实战指南：用 CrawlerRunConfig 精准控制 CSS 选取、内容过滤与 Shadow DOM 扁平化
Crawl4AI 内容选择实战指南用 CrawlerRunConfig 精准控制 CSS 选取、内容过滤与 Shadow DOM 扁平化【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai在 Crawl4AI 中CrawlerRunConfig是控制“爬什么、留什么、扔什么”的核心配置对象。本篇围绕官方文档 content-selection.md 展开系统讲解四类能力用css_selector/target_elements圈定内容区域用一组exclude_*参数过滤标签、链接、域名和媒体用process_iframes与flatten_shadow_dom处理内嵌内容与 Web Components以及用 CSS 或 LLM 抽取策略把过滤后的 HTML 变成结构化 JSON。读完本文你可以复制文中示例直接运行并对照 CrawlerRunConfig 源码 理解每个参数的默认值与底层实现。1. 基于 CSS 的内容选取Crawl4AI 提供两种选取方式css_selector把整页裁剪到某个区域和更灵活的target_elements只让 Markdown 生成与结构化抽取聚焦指定元素同时保留整页上下文。1.1 使用css_selector将抓取结果限制在页面某个区域最直接的方式就是设置css_selector只有匹配该选择器的元素会保留在result.cleaned_html中import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( # e.g., first 30 items from Hacker News css_selector.athing:nth-child(-n30) ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://news.ycombinator.com/newest, configconfig ) print(Partial HTML length:, len(result.cleaned_html)) if __name__ __main__: asyncio.run(main())从源码看css_selector在 CrawlerRunConfig 中的默认值是None即处理整页它会在内容清洗与 Markdown 生成之前的 HTML 预处理阶段生效作用范围覆盖所有抽取过程。1.2 使用target_elementstarget_elements接受一个 CSS 选择器列表允许同时指定多个目标元素。它的语义与css_selector有本质区别import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( # Target article body and sidebar, but not other content target_elements[article.main-content, aside.sidebar] ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.com/blog-post, configconfig ) print(Markdown focused on target elements) print(Links from entire page still available:, len(result.links.get(internal, []))) if __name__ __main__: asyncio.run(main())关键差异设置target_elements后Markdown 生成与结构化数据抽取只聚焦这些元素但链接、图片、表格等仍从整页提取。这让你在控制 Markdown 内容的同时保留完整的链接分析与媒体采集上下文。这个区别在 配置类文档字符串 中有明确描述css_selector会“把初始 raw HTML 收缩到选定元素”而target_elements只影响抽取与 Markdown 生成不裁剪原始 HTML。源码中该参数默认归一化为空列表self.target_elements target_elements or []async_configs.py空列表即表示处理整页。2. 内容过滤与排除2.1 参数总览CrawlerRunConfig提供一整组过滤参数可按需组合config CrawlerRunConfig( # Content thresholds word_count_threshold10, # Minimum words per block # Tag exclusions excluded_tags[form, header, footer, nav], # Link filtering exclude_external_linksTrue, exclude_social_media_linksTrue, # Block entire domains exclude_domains[adtrackers.com, spammynews.org], exclude_social_media_domains[facebook.com, twitter.com], # Media filtering exclude_external_imagesTrue )各参数的作用与当前仓库中的默认值如下参数默认值作用word_count_thresholdMIN_WORD_THRESHOLD忽略字数低于阈值的文本块跳过过短的导航/免责声明excluded_tags[]整体移除指定标签form、header、footer等exclude_external_linksFalse剥离外部链接并可能从result.links中移除exclude_social_media_linksFalse移除指向已知社媒域名的链接exclude_domains[]自定义域名黑名单命中即从链接中剔除exclude_social_media_domainsSOCIAL_MEDIA_DOMAINS社媒域名列表可覆盖或扩充exclude_external_imagesFalse丢弃非本站域名含子域名托管的图片关于word_count_threshold的默认值需要注意源码 async_configs.py 中其取值为MIN_WORD_THRESHOLD而该常量在当前仓库 config.py 中定义为1即默认几乎不过滤短块如需过滤噪音块应显式设置如 10、20。exclude_social_media_domains默认取自 config.py 中的 SOCIAL_MEDIA_DOMAINS与文档列出的清单一致[ facebook.com, twitter.com, x.com, linkedin.com, instagram.com, pinterest.com, tiktok.com, snapchat.com, reddit.com, ]链接过滤的底层实现在 LXMLWebScrapingStrategy._process_element 中每个a href会经normalize_url归一化后判定内外部当exclude_external_links为 True 或链接域名命中exclude_domains时代码直接执行link.getparent().remove(link)即从 HTML 树中删除该节点——这就是“外部链接会被剥离”的实现来源。2.2 组合使用示例import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode async def main(): config CrawlerRunConfig( css_selectormain.content, word_count_threshold10, excluded_tags[nav, footer], exclude_external_linksTrue, exclude_social_media_linksTrue, exclude_domains[ads.com, spammytrackers.net], exclude_external_imagesTrue, cache_modeCacheMode.BYPASS ) async with AsyncWebCrawler() as crawler: result await crawler.arun(urlhttps://news.ycombinator.com, configconfig) print(Cleaned HTML length:, len(result.cleaned_html)) if __name__ __main__: asyncio.run(main())提示如果这些参数过滤掉了过多内容请相应调低阈值或关闭对应开关。另外从源码默认值看cache_mode的默认即为CacheMode.BYPASSasync_configs.py示例中显式写出是为了演示意图清晰。3. 处理 Iframe部分站点把内容嵌在iframe中。若希望将其内联进最终输出设置process_iframesTrue同时可配合remove_overlay_elements与remove_consent_popups清理弹窗config CrawlerRunConfig( # Merge iframe content into the final output process_iframesTrue, remove_overlay_elementsTrue, # Remove GDPR/cookie consent popups (OneTrust, Cookiebot, etc.) remove_consent_popupsTrue )完整用法import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( process_iframesTrue, remove_overlay_elementsTrue ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.org/iframe-demo, configconfig ) print(Iframe-merged length:, len(result.cleaned_html)) if __name__ __main__: asyncio.run(main())从 CrawlerRunConfig 文档字符串 可以看到这三个开关的默认值均为False且remove_consent_popups专门针对 IAB TCF/CMP 标准可识别 OneTrust、Cookiebot、TrustArc、Quantcast、Didomi 等主流 CMP 提供商的弹层。4. 扁平化 Shadow DOM使用 Web Components 构建的站点Stencil、Lit、Shoelace、Angular Elements 等会把内容渲染在 Shadow DOM 中——一个被封装的子树常规的页面序列化如 Playwright 的page.content()完全看不到它。设置flatten_shadow_domTrue后Crawl4AI 会遍历所有 shadow tree、解析slot投影最终产出一份扁平化的单一 HTML 文档config Crawl4AI_CrawlerRunConfig_placeholderconfig CrawlerRunConfig( # Flatten shadow DOM into the main document flatten_shadow_domTrue, # Give web components time to hydrate wait_untilload, delay_before_return_html3.0, )完整示例——抓取一个规格参数藏在 shadow root 里的产品页import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main(): config CrawlerRunConfig( flatten_shadow_domTrue, wait_untilload, delay_before_return_html3.0, ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011, configconfig, ) # Without flatten_shadow_dom: ~1 KB of markdown (breadcrumbs only) # With flatten_shadow_dom: ~33 KB (full product specs, downloads, etc.) print(len(result.markdown.raw_markdown)) if __name__ __main__: asyncio.run(main())实现细节当flatten_shadow_domTrue时AsyncWebCrawlerStrategy 会加载并注入 flatten_shadow_dom.js 初始化脚本该脚本会 patchElement.prototype.attachShadow强制打开mode: closed的 shadow root使闭包组件同样可访问。提示Web 组件需要 JavaScript 运行完成才会渲染内容即hydration。因此建议wait_untilload并配合 2–5 秒的delay_before_return_html确保组件完成水合后再执行扁平化。注意wait_until的默认值是domcontentloadedasync_configs.py若沿用默认值可能赶在组件渲染之前抓取。仓库中提供了完整可运行的示例 shadow_dom_crawling.py可配合 tests/general/test_flatten_shadow_dom.py 查看对应测试覆盖。5. 结构化抽取示例内容选择可以进一步与高级抽取策略组合CSS 或 LLM 抽取策略都运行在经过过滤/裁剪后的 HTML 上。5.1 基于模式的 JsonCssExtractionStrategyimport asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai import JsonCssExtractionStrategy async def main(): # Minimal schema for repeated items schema { name: News Items, baseSelector: tr.athing, fields: [ {name: title, selector: span.titleline a, type: text}, { name: link, selector: span.titleline a, type: attribute, attribute: href } ] } config CrawlerRunConfig( # Content filtering excluded_tags[form, header], exclude_domains[adsite.com], # CSS selection or entire page css_selectortable.itemlist, # No caching for demonstration cache_modeCacheMode.BYPASS, # Extraction strategy extraction_strategyJsonCssExtractionStrategy(schema) ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://news.ycombinator.com/newest, configconfig ) data json.loads(result.extracted_content) print(Sample extracted item:, data[:1]) # Show first item if __name__ __main__: asyncio.run(main())5.2 基于 LLM 的抽取import asyncio import json from pydantic import BaseModel, Field from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig from crawl4ai import LLMExtractionStrategy class ArticleData(BaseModel): headline: str summary: str async def main(): llm_strategy LLMExtractionStrategy( llm_config LLMConfig(provideropenai/gpt-4,api_tokensk-YOUR_API_KEY) schemaArticleData.schema(), extraction_typeschema, instructionExtract headline and a short summary from the content. ) config CrawlerRunConfig( exclude_external_linksTrue, word_count_threshold20, extraction_strategyllm_strategy ) async with AsyncWebCrawler() as crawler: result await crawler.arun(urlhttps://news.ycombinator.com, configconfig) article json.loads(result.extracted_content) print(article) if __name__ __main__: asyncio.run(main())在这个流程中爬虫依次执行过滤外部链接exclude_external_linksTrue忽略超短文本块word_count_threshold20把最终 HTML 交给 LLM 策略做 AI 驱动的解析。注意CrawlerRunConfig构造时会校验extraction_strategy必须是ExtractionStrategy的实例否则抛出ValueErrorasync_configs.py自定义策略需先继承该基类。6. 综合示例选取 过滤 抽取三合一下面的函数把CSS 选取、排除逻辑和模式化抽取统一起来演示如何精细调整最终数据import asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai import JsonCssExtractionStrategy async def extract_main_articles(url: str): schema { name: ArticleBlock, baseSelector: div.article-block, fields: [ {name: headline, selector: h2, type: text}, {name: summary, selector: .summary, type: text}, { name: metadata, type: nested, fields: [ {name: author, selector: .author, type: text}, {name: date, selector: .date, type: text} ] } ] } config CrawlerRunConfig( # Keep only #main-content css_selector#main-content, # Filtering word_count_threshold10, excluded_tags[nav, footer], exclude_external_linksTrue, exclude_domains[somebadsite.com], exclude_external_imagesTrue, # Extraction extraction_strategyJsonCssExtractionStrategy(schema), cache_modeCacheMode.BYPASS ) async with AsyncWebCrawler() as crawler: result await crawler.arun(urlurl, configconfig) if not result.success: print(fError: {result.error_message}) return None return json.loads(result.extracted_content) async def main(): articles await extract_main_articles(https://news.ycombinator.com/newest) if articles: print(Extracted Articles:, articles[:2]) # Show first 2 if __name__ __main__: asyncio.run(main())为什么有效用#main-content做 CSS 域限定多个exclude_参数移除无关域名、外部图片等JsonCssExtractionStrategy解析重复的文章块。7. 抓取模式LXMLWebScrapingStrategy 与自定义策略Crawl4AI 默认使用基于 LXML 的LXMLWebScrapingStrategy作为 HTML 内容处理策略对大型 HTML 文档有出色性能表现from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LXMLWebScrapingStrategy async def main(): # Default configuration already uses LXMLWebScrapingStrategy config CrawlerRunConfig() # Or explicitly specify it if desired config_explicit CrawlerRunConfig( scraping_strategyLXMLWebScrapingStrategy() ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.com, configconfig )从 CrawlerRunConfig 初始化代码 可确认默认值self.scraping_strategy scraping_strategy or LXMLWebScrapingStrategy()即不传参时自动采用 LXML 策略。向后兼容方面content_scraping_strategy.py 末尾 显式声明WebScrapingStrategy LXMLWebScrapingStrategy旧代码无需任何修改即可继续工作。你也可以继承ContentScrapingStrategy编写自定义策略必须返回ScrapingResult对象from crawl4ai import ContentScrapingStrategy, ScrapingResult, MediaItem, Media, Link, Links class CustomScrapingStrategy(ContentScrapingStrategy): def scrap(self, url: str, html: str, **kwargs) - ScrapingResult: # Implement your custom scraping logic here return ScrapingResult( cleaned_htmlhtml.../html, # Cleaned HTML content successTrue, # Whether scraping was successful mediaMedia( images[ # List of images found MediaItem( srchttps://example.com/image.jpg, altImage description, descSurrounding text, score1, typeimage, group_id1, formatjpg, width800 ) ], videos[], # List of videos (same structure as images) audios[] # List of audio files (same structure as images) ), linksLinks( internal[ # List of internal links Link( hrefhttps://example.com/page, textLink text, titleLink title, base_domainexample.com ) ], external[] # List of external links (same structure) ), metadata{ # Additional metadata title: Page Title, description: Page description } ) async def ascrap(self, url: str, html: str, **kwargs) - ScrapingResult: # For simple cases, you can use the sync version return await asyncio.to_thread(self.scrap, url, html, **kwargs)性能考量LXML 策略在处理大型 HTML尤其 100KB时性能优异官方描述其相比 BeautifulSoup 方案最快可达 10–20 倍同时具备内存占用低、对规整 HTML 处理稳健、表格检测与提取能力强等优势。8. 组合 CSS 选取方法的进阶玩法css_selector与target_elements可以组合使用实现细粒度控制import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode async def main(): # Target specific content but preserve page context config CrawlerRunConfig( # Focus markdown on main content and sidebar target_elements[#main-content, .sidebar], # Global filters applied to entire page excluded_tags[nav, footer, header], exclude_external_linksTrue, # Use basic content thresholds word_count_threshold15, cache_modeCacheMode.BYPASS ) async with AsyncWebCrawler() as crawler: result await crawler.arun( urlhttps://example.com/article, configconfig ) print(fContent focuses on specific elements, but all links still analyzed) print(fInternal links: {len(result.links.get(internal, []))}) print(fExternal links: {len(result.links.get(external, []))}) if __name__ __main__: asyncio.run(main())这种组合兼得两者之长Markdown 生成与内容抽取聚焦你关心的元素链接、图片等页面数据仍保留整页上下文内容过滤参数全局生效。9. 关键参数速查与结论把target_elements / css_selector 域限定、内容过滤参数与高级抽取策略混合使用你可以精确决定保留哪些数据。CrawlerRunConfig中用于内容选择的关键参数汇总如下target_elements— CSS 选择器数组聚焦 Markdown 生成与数据抽取同时为链接和媒体保留整页上下文css_selector— 基础域限定对全部抽取流程生效word_count_threshold— 跳过短块注意当前仓库默认MIN_WORD_THRESHOLD 1需显式调高才有过滤效果excluded_tags— 整体移除指定 HTML 标签exclude_external_links、exclude_social_media_links、exclude_domains— 过滤不需要的链接或域名exclude_external_images— 移除外部来源图片process_iframes— 按需内联 iframe 内容flatten_shadow_dom— 将 Shadow DOM 扁平化进主文档覆盖 Web Components 场景。再叠加 CSS、LLM 等结构化抽取策略即可构建出从 raw/cleaned HTML 到精密 JSON 的完整数据管线。更多参数细节可查阅 Configuration Reference。【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考