# Research report #1 — methodology freeze (v1.1)

**Working title:** Who AI Search Cites When Buyers Ask for AEO Help
**Subtitle:** A source-level analysis across seven commercial questions, four AI engines and repeated runs.
**Frozen:** 2026-07-16 (per DECISIONS.md 2026-07-16 and the Codex harmonisation handoff). v1.1 amendments (denominators, locale honesty, URL normalisation, failure policy) applied 2026-07-16 BEFORE any collection, per Codex review. Changes after collection begins require a new methodology version (v1.2, v2.0) declared in the report.
**Status:** FROZEN before analysis. Nothing below may be adjusted after data collection begins, except by versioned amendment.

## 1. Research question

When a tech-literate founder asks an AI answer engine a commercial question about AEO / AI-search visibility, **what kinds of sources do the engines cite?** Secondary question: **how stable are those citations across repeated runs?**

No causal claims. The report describes what the engines cited in this sample, in this window. It does not claim why, and it does not claim that producing a given source type causes citation.

## 2. Datasets

### Primary dataset (the headline analysis)
- **Queries:** the locked 7-query Case Study Zero buyer set, verbatim, `querySetVersion 1` (drafts/citation-engine-dogfood-scan-spec.md). No additions, no rewording.
- **Engines (4):** Perplexity, ChatGPT, Claude, Gemini, each via its grounded-search path, fresh context, no history.
- **Runs:** 3 per query per engine.
- **Expected observations:** 7 × 4 × 3 = 84 answer observations.
- **Locale / conditions:** English, no personalisation, collected via the existing scan harness (DataForSEO backbone + the dogfood scan protocol). Country pinning: GB is pinned where the DataForSEO endpoint supports it (Perplexity, ChatGPT, Claude); **Gemini does not support country pinning and is recorded as unpinned**. The report never describes all four engines as UK-localised.
- **Collection window:** a single declared window of at most 72 hours. The window's actual dates are recorded in the dataset and stated in the report.
- **Storage:** drained to the `observations` table (Supabase project citation-engine) under a fresh `audit_id`, same schema as existing drains: `query_id, query_text, engine, run, observed_at, response_text, citations (jsonb), error`.

### Exploratory dataset (clearly separated, never pooled)
- The existing drained audit `AUD-20260604-002`: 600 observations, 50 prospect-category queries × 4 engines × 3 runs, collected 2026-06-04.
- Different query protocol and category, so it is **not** pooled with the primary dataset. It appears only in a clearly-labelled "does the pattern hold elsewhere?" exploratory section, with its own denominators, anonymised category framing, and no prospect names.

## 3. Units and denominators (fixed definitions)

The report always names which unit a percentage is over. Four units, defined once:

1. **Answer observation** — one engine response to one query on one run. Primary N = 84 (minus any errored observations, which are counted and disclosed).
2. **Citation appearance** — one cited source inside one answer observation. An answer citing 5 sources contributes 5 appearances. This is the default denominator for the source-type distribution.
3. **Unique URL** — de-duplicated exact URLs across the dataset.
4. **Unique domain** — de-duplicated registrable domains (eTLD+1).

**Denominator roles (v1.1):** source types are classified at page level, so one domain can legitimately contain both VEN and RSR pages. Therefore:
- The **source-type distribution** is reported over **citation appearances** and over **unique normalised URLs**, side by side, so one frequently-repeated URL cannot silently distort the picture. The most-repeated URLs are listed in their own table.
- **Unique domains** are used only for **concentration** (how few domains capture the appearances) and **stability** analysis — never as a mutually exclusive source-type denominator.

**Case Study Zero metrics stay separate:** 0/7 (queries citing patrickrobinson.consulting) and 0/28 (engine-query slots, as on the live site tracker) are tracking metrics for the dogfood programme. They are reported as context only and are NOT denominators in this study.

## 4. Source-type taxonomy (frozen before classification)

Every citation appearance is classified into exactly one of:

| Code | Type | Definition |
|------|------|------------|
| VEN | Vendor page | The cited domain sells the product/service category the query is about (marketing pages, vendor blogs, vendor docs) |
| CMP | Third-party comparison | Independent listicle, "best X" roundup, comparison table, review site, directory |
| UGC | User-generated | Reddit, Quora, Stack Exchange, forums, community wikis |
| NEWS | News / trade press | Journalistic outlets, industry press, newsletters with editorial identity |
| EDU | Educational / reference | Explainer or definition content from a party not selling in the category: encyclopaedias, courses, glossaries, standards bodies, agency-authored explainers where the agency is outside the query category |
| RSR | Research / data | First-party studies, benchmarks, datasets, academic or pre-print work |
| SOC | Social / video | X posts, LinkedIn posts, YouTube videos, podcasts |
| OTH | Other | Anything not classifiable above |

**Ambiguity rules (applied in this order, frozen now):**
1. Classify by the **page**, not the domain (a vendor's independent-methodology benchmark page = RSR, not VEN).
2. If a page both sells and compares (a vendor's "us vs competitors" page), it is VEN. Independence is required for CMP.
3. An agency/consultancy explainer counts as VEN when the agency sells services in the query's category (AEO/SEO/marketing services for this query set), EDU otherwise.
4. Aggregators of others' reviews (G2, Capterra) are CMP.
5. If still ambiguous after rules 1-4, classify OTH and log the reason. OTH above 10% of appearances triggers a taxonomy note in limitations.
6. Every classification is recorded in the dataset with the rule that decided it; a second classification pass on a 20% random sample checks consistency, and the agreement rate is published.

## 4b. URL extraction and normalisation (frozen before collection)

The scan harness retains the raw engine responses alongside the derived `allCitedDomains`; the CSV's URL-level evidence is extracted from the raw responses at drain time.

1. **Unit rule:** one citation appearance per **normalised URL per answer observation**. The same normalised URL cited twice inside one answer counts once; cited in two different answers counts twice.
2. **Normalisation:** lowercase scheme+host; strip fragments (`#…`); strip known tracking parameters (`utm_*`, `gclid`, `fbclid`, `ref`, `source`); keep other query parameters; treat `http`/`https` and trailing-slash variants as the same URL; strip `www.` for identity purposes while preserving the original string in a `raw_url` column.
3. **Redirects:** Gemini's `vertexaisearch.cloud.google.com/grounding-api-redirect/…` URLs are resolved selectively by the harness; when resolution fails, the observation keeps the original opaque URL and is flagged `unresolved=true`. Unresolved URLs are excluded from source-type classification (they cannot be read) but counted and disclosed as their own line in the results. No other redirect chasing beyond the harness's existing behaviour.
4. **Drain path:** each appearance lands in the `citations` jsonb of the `observations` row as `{raw_url, normalised_url, domain, position, unresolved}`; the classification pass later adds `{source_type, deciding_rule}`.

## 4c. Failure policy (frozen before collection)

- The harness's existing automatic retries stand (transient errors retried with 2s/8s backoff; terminal API codes never retried).
- Terminal errors are disclosed **per engine** in the report, not silently dropped.
- **Completeness rule:** the window is complete only if every engine-query pair has at least 2 successful runs of 3 AND total successful observations ≥ 80 of 84. Below either threshold, the missing cells are re-run inside the declared 72h window; if they still fail, the window is declared incomplete and collection is re-declared (new window, same protocol) rather than quietly reducing N.

## 5. Stability section (secondary analysis, same data)

From the primary dataset's 3 runs per engine-query:
- **Jaccard overlap** of the cited unique-domain sets between runs of the same engine-query pair.
- Share of engine-query pairs where the citation set is identical across all 3 runs / overlapping / fully disjoint.
- Count of domains that appear in all 3 runs ("stable citations") vs exactly 1 run ("one-off citations").
- Framed explicitly as a small-scale, first-party echo of the IQRush / St. Gallen volatility findings, with sample size stated in the same sentence. Three runs cannot establish stability; they can only show observed churn.

## 6. Deliverables (unchanged from the sprint plan)

Durable report page on patrickrobinson.consulting, downloadable CSV (one row per citation appearance: query_id, engine, run, observed_at, url, domain, source_type, deciding_rule), Medium adaptation with verified canonical, one longer YouTube explainer, first post-publication rescan on a declared date.

## 7. Limitations (stated prominently in the report)

- 7 queries in one commercial niche; 84 answer observations; a single collection window. The sample describes this niche at this moment, nothing broader.
- Engines are non-deterministic and personalise; results are clean-context, GB-pinned where supported (Gemini unpinned) and may differ elsewhere.
- Source-type distribution is descriptive. No causal claim about what earns citations.
- The classifier is a human-plus-model process with a published second-pass agreement rate, not an oracle.
- The exploratory dataset uses a different query protocol and is never pooled with the primary.

## 8. What would falsify the useful version of this report

If citation appearances spread roughly evenly across source types with no concentration, the "engines favour particular source shapes" framing fails and the report says so. The report publishes whatever the distribution shows.

---

## 9. Deviation log (appended after collection, never silently edited above)

**2026-07-16 — raw-evidence storage.** Section 1 declared that the 84 answer observations would be drained to the `observations` table (Supabase project citation-engine) under a fresh audit id. In practice, extraction ran directly against the 84 raw API response files on disk, and the Supabase drain was completed after extraction rather than before it (audit id `RES-20260716-001`, 84 rows, 651 citation appearances, drained 16 July 2026). The raw response files are also preserved in the site repository under `scripts/citation-scan/dfs-logs/research-01-primary/`. No reported number is affected; the deviation is in where the evidence was stored during analysis, not in what was measured.

---
Licence note: the compilation, source-type classifications, deciding rules and analysis are © Patrick Robinson and licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The licence covers Patrick Robinson's original work only: it does not extend to DataForSEO's service data (collected via the DataForSEO API with written permission to distribute, subject to the [DataForSEO Terms of Service](https://dataforseo.com/terms-of-service)) or to any third-party page titles or content quoted within the dataset.