Skip to content
T1KTheOneKit

t1k:datastore-cost

FieldValue
Modulet1k-devops
Version1.4.1
Effortlow
Tools—

Keywords: aggregate, cloudflare, d1, D1 pricing, d1-cost, datastore design, datastore-cost, event pipeline, hot counter, index review, ingestion, object storage, pre-deploy cost, r2, raw event, rollup, rows-read, rows-written, scan cost, workers

How to invoke

Section titled “How to invoke”
/t1k:datastore-cost
store design, schema review, ingestion pipeline

Datastore Cost Discipline — Workers / D1 / R2

Section titled “Datastore Cost Discipline — Workers / D1 / R2”

Prevent the $430/month class of surprise: writing raw rows to a row-billed managed datastore (D1), one scan away from the next surprise on the read side. Decides what a row costs before it is ever written.

Apply when: architecting an ingestion/data path, reviewing a D1 schema or a query that scans, or pre-deploy cost-checking any Worker that touches D1/R2.

Pre-flight — announce before you write or scan

Section titled “Pre-flight — announce before you write or scan”

Mandatory, before executing any command that writes to or scans a managed datastore (D1 INSERT/UPDATE/DELETE/batch, wrangler d1 execute, a migration, a bulk R2 put/list): state the estimated row/operation count and its dollar cost at the current published rate, then act. Unknown volume is itself the warning — say “volume unknown, could be unbounded” rather than staying silent.

Rates to do the arithmetic inline (retrieved 2026-08-29 — full table + source URLs in the Pricing reference under Pre-deploy cost check below):

OperationIncludedOverage
D1 rows written50M/mo$1.00 / million rows
D1 rows READ — scanned, not returned25B/mo$0.001 / million rows
R2 Class A (write/list)—$4.50 / million requests
R2 Class B (read)—$0.36 / million requests

Reads are not exempt. D1 bills rows the query planner scans, not rows it returns, so an unindexed WHERE is a cost event too — estimate the table’s row count, not the expected result-set size. If the volume can’t be bounded, smoke a small N first and surface the measured per-row cost before scaling (modules/t1k-base/rules/preview-first-batch.md).

Mandatory Cost Report — every change, every time

Section titled “Mandatory Cost Report — every change, every time”

Standing instruction (user, 2026-08-31): report the cost impact of every change you make. The Pre-flight announcement covers a command you are about to run. This covers a change you are about to ship — a schema edit, a new query, a sync cadence, an index, a mirrored table. Emit it unprompted; never wait to be asked.

Four elements, all required:

  1. Before/after rows + dollars. Rows written and rows scanned per run, before vs after, priced at the rates above. Cannot estimate? Say “volume unknown, potentially unbounded” — that is the report, not a reason to skip it.
  2. Monthly run-rate projection. Extrapolate per-run to $/day and $/month. A per-tick figure is not a cost signal on its own: a 448.8K rows/hour floor reads as trivial per tick and is ~$323/month (measured 2026-08-31). Always project before judging a number small.
  3. Read-scan cost, not just writes. Rows the planner scans for any new or changed query — unindexed WHERE, COUNT(*) on a growing table, a wide date range over a large fact table. Per Read-scan review; the number to report is the table’s row count, not the result-set size.
  4. Post-deploy verification. After any cost-affecting deploy, measure actual rows/hour against what you predicted and report the delta. Never claim a cost fix worked without this measurement — a daily total cannot tell “the fix landed” apart from “yesterday had spikes”. Method: references/measuring-d1-cost.md.

A partial replace still owes elements 2 and 4. The skip-gate in Recurring full-replace mirror/sync jobs removes writes it proves unnecessary, but leaves a floor wherever the source genuinely changed each tick; that floor is the residual to project and then measure, not to assume away. Closing it further is step 3 of that section (upsert genuine deltas) — a separate change, owing its own report.

Decision Tree

Section titled “Decision Tree”
IntentPath
”I’m about to write/scan a managed datastore”Pre-flight — mandatory, do this first
”I changed something — what does it cost?”Mandatory Cost Report — all four elements, unprompted
”Where should this data live?”Choose the right store
”Design the ingestion pipeline”Aggregate-then-write pipeline
”I have a scheduled mirror/sync/refresh job”Recurring full-replace mirror/sync jobs
”Is this query/schema going to cost me?”Read-scan review
”Before I deploy, estimate the bill”Pre-deploy cost check

Choose the right store

Section titled “Choose the right store”
  • Raw events / payload / blobs (analytics, logs, fails) → object store (R2-class) or compute at query time. If a row is not reduced, it does not belong in a row-billed store.
  • Rollup / aggregate / KPI / config → D1-class row store.
  • Hot counter (~user count, DAU, fails) → maintained counter column/row, never COUNT(*) a growing table on a hot path.
  • Ephemeral → compute locally; do not persist at all.

Aggregate-then-write pipeline

Section titled “Aggregate-then-write pipeline”

Do NOT write one row per event. Insert late; insert reduced.

  1. Buffer → batch → aggregate in the Worker (in-memory, or KV if state must survive).
  2. Roll up to fixed windows (5m / 1h / 1d) with a stable bucket key.
  3. Upsert the bucket row (INSERT … ON CONFLICT DO UPDATE).
  4. Stream the raw event to object storage keyed by id/timestamp (R2 puts, cheap), for backfill and audit.
  5. Write only the aggregate count / sum / distinct-set size / last-seen. If the raw row must exist to answer a future question, it lives in R2 — not D1.

Recurring full-replace mirror/sync jobs

Section titled “Recurring full-replace mirror/sync jobs”

Aggregating the input is not enough if the write is a full-refresh cron: a periodic job that strips and rebuilds an entire partition on every tick, whether or not the source moved. This is its own cost anti-pattern — the data is already aggregated (a Postgres → D1 mirror of rollups), so “aggregate-then-write” lets it through. The cost is not wrong granularity; it is rewriting an unchanged partition on every tick. Because DELETE and INSERT both bill as rows written, a job re-inserting ~1.2K rows to replace ~287K deletes reached 149.47M rows/day (~$4,434/mo) within a week of going live.

Gate the replace step behind a cheap content comparison before paying the DELETE+INSERT cost:

  1. Compare, don’t assume. Before replacing a partition, read a cheap fingerprint of what the target already holds — count plus summed measures, or a watermark (max timestamp). If the source fingerprint equals the target’s, skip.
  2. Keep the comparison indexed. A covering index makes the pre-check a handful of rows read against ~287K rows written avoided — reads bill ~1000× cheaper than writes.
  3. Use an upsert for genuine deltas, not an unconditional wipe-and-reload — write only the rows that actually changed.

Why this is safe by construction: the comparison can only remove writes it proves unnecessary — worst case is zero savings, never a regression. It also correctly skips partitions the target has never held (a new partition generation), which is correct behaviour, not a gap. Measured outcome: 100% skip on every already-synced partition across two production ticks, ~87% total reduction, $4,434/mo → ~$550/mo. Treat “full-refresh cron” as its own anti-pattern alongside raw-event ingestion, not a variant of it.

Read-scan review

Section titled “Read-scan review”

Row-billed stores charge rows scanned, not returned. An unindexed WHERE over an N-row table bills N rows read per query.

  • Index every filtered column — the D1 index write is offset, in nearly all cases, by the scan it avoids.
  • LIMIT every scan; never SELECT * over wide tables. Reading 1 wide row is 1 row scanned, but a fan-out SELECT scans the whole table.
  • Never COUNT(*) a growing table on a hot path. Maintain a counter row/column and read that.
  • Check the query meta: every D1 response carries rows_read and rows_written (many SDKs expose meta.rows_read). Total up your daily row-read spend during load tests.

Pre-deploy cost check

Section titled “Pre-deploy cost check”

Estimate before you deploy. A Worker that writes a raw row per event on a paid plan with something like 50M write rows included is a bill waiting to be discovered.

  1. Estimate daily event volume × rows written per event. This is the write spend.
  2. Estimate rows scanned by the hottest query × queries/day. An unindexed WHERE on your largest table is this number. Keep it under the included read budget, then pad.
  3. Verify the running query’s actual cost in staging by reading meta.rows_read.
  4. Raise the estimate by 3× for a freshly-shipped ingestion path. Fat-fingered loops and bot traffic are what the included budget does not survive.
  5. Re-check quarter-over-quarter — a growing table makes every unindexed scan linearly more expensive. Indexes are the fix, not a “we’ll see” attitude.

Pricing reference (retrieved 2026-08-29):

Gotchas

Section titled “Gotchas”
  • Write ops include INSERT, UPDATE, and DELETE — each counts toward rows written.
  • An index adds one written row per write when the indexed column changes — but it nearly always pays for itself in scans avoided.
  • Row size does not affect the count: a 1 KB row and a 100 KB row both count as one.

Worked example

Section titled “Worked example”

A Worker ingesting 6M events/day wrote 6M raw rows/day to D1 → 180M rows/mo → ~$130/mo in writes alone. Rework: aggregate to 5-minute buckets (~8.6K bucket rows/day → ~260K rows/mo → well inside the included 50M), raw payloads streamed to R2. Month-end R2 cost for ~180M small objects: roughly $25–60 (Class A puts) + a few cents storage. D1 read spend near zero because every lookup is by an indexed bucket key under LIMIT.