Bypassing the Boilerplate: The Semantic HTML Rule for AI Crawlers

What Readability.js does to your page

Mozilla's Readability.js — the algorithm powering Firefox Reader View, Pocket, and a significant portion of AI content ingestion tools — processes a web page by attempting to identify the main article content and remove everything else. It is aggressive, opinionated, and stateless.

Readability.js works by scoring candidate content blocks based on link density (high link density = navigation = remove), tag semantics (article, main = keep; div with generic classes = evaluate), text length (short text blocks = likely UI = remove), and parent container context. A content block inside a clearly semantic container like <article> is preserved with high confidence. A content block inside a <div class="content-wrapper"> is evaluated probabilistically — and frequently discarded.

Jina.ai's reader — the tool used by many AI pipelines including some Perplexity integrations — uses a similar approach. The result: AI ingestion pipelines often silently fail to index content that is not wrapped in semantic HTML, with no error and no visibility for the content author.

The div soup problem

Modern JavaScript-heavy web applications frequently produce deeply nested <div> structures with utility class names: <div class="flex flex-col max-w-3xl mx-auto px-4">. These class names tell a human developer about layout but tell Readability.js nothing about content meaning. The algorithm cannot distinguish between a div.flex.flex-col wrapping a blog article and a div.flex.flex-col wrapping a sidebar advertisement.

Next.js, React, and Vue applications built with Tailwind CSS or similar utility frameworks are particularly susceptible. The framework encourages div-based layouts with utility class names — a pattern that produces excellent visual results but terrible AI parser readability.

The semantic container hierarchy

The correct semantic HTML hierarchy for AI-parseable content has three levels. At the page level: <main> wraps the primary content area of the page. Every page should have exactly one <main> element. At the content level: <article> wraps a self-contained piece of content (a blog post, a product page, a guide). At the section level: <section> wraps a thematic grouping within the article.

This hierarchy gives Readability.js a clear signal: everything inside <main><article> is primary content. Strip everything outside.

Semantic HTML parser survival rates

<div className="grid grid-cols-3 gap-2 text-slate-500 font-semibold"> Container Parser outcome Citation impact

• → <main><article>
• →Always preserved
• →Full indexing
• → <main> alone
• →Usually preserved
• →Good indexing
• → <article> alone
• →Usually preserved
• →Good indexing
• → <section>
• →Context-dependent
• →Partial indexing
• →
• →Probabilistic
• →Unreliable
• → <div class='flex'>
• →Likely stripped
• →Minimal indexing

Tags that survive AI parsers

These HTML elements are consistently preserved by Readability.js, Jina.ai reader, and equivalent parsers: <main>, <article>, <section> (within article), <h1> through <h4>, <p>, <ul>/<ol>/<li>, <table> with semantic markup, <figure>/<figcaption>, <blockquote>, and <script type="application/ld+json"> (JSON-LD Schema).

Tags that get stripped

These elements are consistently stripped by AI parsers: <nav>, <header>, <footer>, <aside>, <form>, <button>, <input>, <script> (non-JSON-LD), <style>, <canvas>, <iframe>, and <div> elements classified as navigation, advertisements, or boilerplate by density heuristics.

Implementation checklist

How to audit your semantic HTML

The fastest audit method: pass your page URL through Firefox Reader View (Ctrl+Alt+R). Whatever Firefox Reader View shows is approximately what Readability.js sends to the vector database. If Reader View strips your content, Perplexity and ChatGPT likely will too.

Alternatively, run your URL through Jina.ai's reader (r.jina.ai/[your-url]) to see exactly what the plain-text extraction produces. Any content not present in that output is not being indexed for AI retrieval.

The 512-Token Rule How chunking works after your HTML survives the parser. Visual RAG Why canvas and JS charts are invisible to RAG pipelines.