# Adaptive Content Delivery for AI Crawlers

RankGen.ai (Rank GenAI) is an independent Rank AI research lab. It tests adaptive content delivery with accessible HTML for people and concise WARP Markdown for clients explicitly requesting it. Both representations share canonical answers and linked evidence. Results are experimental; an increase in AI citations has not been demonstrated.

Canonical page: https://rankgen.ai/
Publisher: RankGen.ai
Status: experimental
Updated: 2026-10-19

## [a1] What is adaptive content delivery for AI crawlers?

Adaptive content delivery is a server-side technique that offers different representations of the same substantive information according to an HTTP request. On this site, normal visitors receive an accessible HTML page; clients explicitly requesting text/markdown can receive a compact text representation.

### [a1-f1] Are there two independent versions of the facts?

No. RankGen keeps one publisher-authored answer and evidence source. Its HTML and Markdown views are generated from the same content rather than maintained as competing pages.

Evidence [a1-f1-e1]: [HTTP Semantics RFC 9110: content negotiation](https://www.rfc-editor.org/rfc/rfc9110.html#section-12)

## [a2] How does the server decide which representation to return?

The request-time router checks the explicit HTTP Accept preference. An accepted text/markdown request receives Markdown; ordinary browser requests receive HTML. Vary: Accept signals that representations differ. The experiment disables page caching at the WordPress layer until end-to-end cache behavior is verified.

### [a2-f1] Does naming Googlebot or another crawler automatically trigger a different message?

No. User-agent names are easily spoofed, and this demonstration does not send different claims to search crawlers. The meaningful format switch is initiated by explicit content negotiation, not by hiding information from humans.

Evidence [a2-f1-e1]: [RFC 9110: Accept and Vary headers](https://www.rfc-editor.org/rfc/rfc9110.html#section-12.5)

## [a3] What does WARP add beyond automatic HTML-to-Markdown conversion?

WARP, the experimental Web Answer Retrieval Protocol, organizes publisher answers into stable A nodes, related fan-out F nodes and linked evidence E nodes. The proposed advantage is selective retrieval of the smallest useful answer and supporting evidence rather than a wholesale conversion of every visible page element.

### [a3-f1] Do AI search platforms currently understand WARP as a standard?

There is no demonstrated platform-level adoption. WARP is an experiment, not an industry standard or a ranking signal. RankGen compares it against normal HTML and generic Markdown and publishes only measured outcomes.

Evidence [a3-f1-e1]: [WARP specification and limitations](https://ericstrate.com/warp/)

## [a4] Is serving HTML to people and Markdown to an agent considered cloaking?

Different formats are not inherently deceptive. The critical safeguard is keeping material facts, qualifications and supporting evidence equivalent and accessible. This experiment does not promise crawler-only benefits or withhold material information from a normal browser.

### [a4-f1] Why keep the Google-search experience crawlable as HTML?

Google recommends server-side or static rendering over bot-specific dynamic rendering workarounds. A well-structured HTML page is a safe baseline for search discovery while explicit machine-readable alternatives remain optional.

Evidence [a4-f1-e1]: [Google Search Central: dynamic rendering guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering)

## [a5] Does the Markdown version save tokens or increase AI citations?

A shorter retrieval payload can reduce transmitted bytes, but actual language-model token savings depend on what a retriever consumes. Higher citation rates are unproven. RankGen treats transfer size, tokenizer-specific input length, latency, answer fidelity and citation outcomes as separate measurements.

### [a5-f1] What makes this different from Markdown for Agents?

Cloudflare offers HTML-to-Markdown conversion on request. RankGen is testing publisher-authored answer-and-evidence packaging and selective retrieval as a separate, measurable improvement hypothesis.

Evidence [a5-f1-e1]: [Cloudflare: Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)

Original HTML: https://rankgen.ai/
Test instructions: https://rankgen.ai/#try-it