How 170 MB cost me $350 in 813 million S3 requests
I had wanted to see the analytics for this blog for quite some time. Turning on CloudFront access logs from the very start was a conscious decision: I wanted analytics, but I did not want to reach for Google Analytics - too much CSP complexity, the privacy concerns, the cookie policy, all of it. Logging to my own S3 bucket sidesteps every one of those, and I knew I would one day build something around the data. In practice that "something" was me running an ad-hoc Athena query from time to time to see how many hits /blog/* got in a given month - which works, but is not exactly user friendly. I had already built two dashboards (one for DMARC reports and one for email delivery), so I figured a third would be no trouble.
Adding it meant backfilling about a year of CloudFront access logs. The whole dataset is around 170 MB. Backfilling it cost me roughly $350. Done right it would have cost about $3. The data was tiny, the mistake was tiny, and the bill was not - and the reason is a single architectural decision I did not think twice about.
This article was put together by an LLM agent, not written by me word for word. The idea, the opinions and all the technical content are mine - I just did not have the time to write it up, so I asked an agent to assemble it from my earlier posts. The agent did not come up with any of this. As with every generated article on this blog, I have proofread and edited it myself before publishing.
The setup
The dashboard is built on my own @beesolve/analytics-processor and @beesolve/analytics-dashboard (more on exactly what it does at the end). The processor reads CloudFront access logs with Athena, rolls them up into hourly, daily and monthly aggregates, and stores those in DynamoDB so the dashboard can read them cheaply and efficiently.
CloudFront has two access-log formats. The old one (let's call it v1) writes gzipped tab-delimited text files. The new one (v2) writes Parquet. I had a year of v1 logs sitting in S3 from before, so the plan was: switch the distribution over to v2 going forward, and backfill the v1 history into the dashboard so the charts were not empty.
This was the first time I had ever run this stack end to end against real data. It fought back. Over one session with an LLM agent we tripped over seven separate bugs:
- log delivery not being supported in my region,
- a bucket that wanted to replace itself,
- a Lambda concurrency quota of 10,
- a Glue table whose partition layout did not match where the files actually lived,
- the distribution ID that had to be baked into the table location rather than left as a partition,
- missing S3 permissions on the Athena results bucket,
- and Parquet columns that came back as the wrong type.
Normal first-run stuff. We fixed them one by one, which meant re-running the backfill. A lot. Hold on to the re-running - I went in blaming it for the whole bill, and it turned out to be the smaller part of the story.
Eventually it all worked. The backfill completed, the dashboard rendered, the numbers were right. I was happy - until the budget alert arrived the next day.
The symptom
I have an AWS Budgets budget alert set at a $30 threshold. The whole organisation's bill usually comes in around $20 a month, so the threshold is a comfortable ceiling that never fires. On the 14th - half way through the month - it fired, projecting $365. Seeing a bill that high on the 14th, roughly 10x the usual, I was genuinely alarmed, and the obvious first question was: what for?
Cost Explorer, grouped by service for that one day, answered it:
$354.99 Amazon Simple Storage Service
$ 6.71 Amazon Athena
$ 1.62 Amazon DynamoDB
$ 1.06 CodeBuild
$ 0.73 AWS CodePipeline
$ 0.50 AmazonCloudWatch
Seeing the cost sit in S3 was surprising - I had assumed (and the agent agreed) it would be Athena, which bills $5 per terabyte scanned, and we had run a lot of queries. But Athena was only $6.71. The entire fire was S3, at $354.99, and drilling into the S3 usage type told the real story in one line:
$349.87 EUC1-Requests-Tier2 qty = 813,641,138
Eight hundred and thirteen million GET requests. On 170 MB of data.
Why it happens
The agent generated this part of the stack, I read the code, and it made sense to me - one Athena query per hour, write the rollup, move on. It looked completely reasonable. What neither of us did was ask the one question that actually mattered: how is Athena going to physically read these files, and what does that cost when I run the query 8,760 times? The layout of the logs in S3 was never designed for the way Athena would query it. I did not notice, the agent did not flag it, and Athena did exactly what it was told - inefficiently, happily, for $350.
So I asked the agent to explain, in detail, why a backfill over 170 MB generated 813 million GET requests. Here is the mechanism it walked me through - and this time I checked it against the actual numbers:
There are two multipliers. First, S3 request cost scales with the number of objects, not the number of bytes. CloudFront v1 logging writes a new little gzip file every few minutes, around the clock - about 206 files a day, averaging ~2 KB each, ~93,000 for the year. Reading 170 MB spread across 93,000 objects costs 93,000 GETs, not one.
Second, and this is the part the architecture got wrong: the Glue table over those v1 logs has no partitions at all.
dateis a plain string column, not a partition key. But the processor backfills one hour at a time, one Athena query per hour. Because nothing in the table is a partition column, theWHEREprunes nothing - the predicate is applied after the data is read. So to produce one hour of rollup, Athena opens every file in the whole table and throws away all but the hour it wanted. Then it does it again for the next hour. Gzip is not splittable, so each file costs several range GETs on top.
The query the processor runs per hour looks like this - note that nothing in it can be a partition:
WHERE "date" = DATE '2026-01-15' AND substr(timestamp, 1, 2) = '14'
One year of hourly rollups is 8,760 queries, each one a full scan of all ~93,000 files. The arithmetic falls straight out:
813,641,138 GETs / ~93,000 objects ~= ~8,700 GETs per object
~8,700 reads of each tiny file - almost exactly the ~8,760 hourly queries in a year, each one scanning every object. In other words, the measured bill is roughly a single full pass over the data: one year of hourly, full-table-scanning queries is already enough to read each file ~8,700 times. The data never grew. My request count exploded.
And then there was the re-running, which I had assumed was the whole story going in. Because of the seven bugs, I did not do one clean pass. I did a concurrency-4 pass, threw it away, did a concurrency-10 pass from scratch, then a concurrency-14 pass, then a slow gap-fill pass, plus a pile of ad-hoc validation queries and a couple of Glue table rebuilds while debugging. Some of those resumed and skipped already-done hours, but the from-scratch restarts each re-triggered the full-table-scan-per-hour storm. The restarts piled more on top - but they were piling onto an architecture that already cost ~$350 the first time through. The restart discipline was never the bug. The missing partitions were.
What stings is that none of this is exotic. It is the same lesson I have written about with DynamoDB: when you learn how it works and model your access patterns up front, you get a serverless database that costs almost nothing and scales forever. When you reach for it as if it were a SQL database you never bothered to study, you get bad performance and a surprising bill, and you walk away saying "DynamoDB is a bad service" - which is wrong, you just used it for something it was not built for. Athena is the same. Pointed at a layout you have thought about, it is a near-free way to query logs. Pointed at 93,000 tiny unpartitioned gzips, it will faithfully bill you for 813 million reads. The tool was fine. I did not understand how it would behave at scale, and that is the whole mistake.
The fix is embarrassingly simple
The data is 170 MB. The entire problem is that it lives in ~93,000 two-kilobyte files behind a table with no partitions, so every hourly query scans all of them. So: stop doing that.
The fix is to compact the files first. Read all those tiny gzips once and rewrite them as a handful of larger Parquet files, partitioned by hour. In Athena that is a single CREATE TABLE AS SELECT that reads the 93,000 files exactly once (the one unavoidable read, a few cents of GETs, about $3 worst case) and writes the compacted, hour-partitioned table. After that, every hourly backfill query opens only the files in its one hour-partition instead of the whole table. The full-table scan is gone, and so is the tiny-files tax.
And here is the bit I like. CloudFront v2 logs are already Parquet, already hour-partitioned. So when I compact v1, I am not inventing a new format - I am converting v1 into the exact shape v2 already has. I had the agent write the CTAS to rename the v1 columns to their v2 equivalents and derive the hour partitions from the old date and time columns on the way out:
SELECT
"time" AS timestamp,
"sc-status" AS sc_status,
"cs-uri-stem" AS cs_uri_stem,
"cs(User-Agent)" AS cs_user_agent,
-- ...the rest of the v1 columns, renamed to v2 names
substr(CAST("date" AS varchar), 1, 4) AS year,
substr(CAST("date" AS varchar), 6, 2) AS month,
substr(CAST("date" AS varchar), 9, 2) AS day,
substr("time", 1, 2) AS hour
FROM cf_logs_v1_backfill
The target table is Parquet, partitioned_by = ARRAY['year', 'month', 'day', 'hour'] - so date does not carry through as a column, it becomes the partition path instead. Once the old data looks like v2 data, the processor does not need to know v1 ever existed. The whole v1 code path - a second set of column mappings, a separate WHERE clause for the unpartitioned predicate, the branches that told the two formats apart - all of it becomes dead code. The fix for the cost mistake also deletes a pile of branching from the package. The cheaper architecture is also the simpler one.
Even the fix had its own gotcha. I wanted the compacted table to write exactly one Parquet file per hour-partition, which Athena will happily do if you declare the table bucketed_by with bucket_count = 1. Except then it refuses the write: INSERT is not supported for bucketed tables. And I needed INSERT, because a full year is 8,760 hour-partitions and Athena (Trino under the hood) caps you at 100 open writers per statement - so the compaction has to run as a seed CREATE TABLE AS SELECT plus a batch of INSERTs covering a few days each. Bucketing forbids INSERT; the partition cap forces INSERT. You cannot have both. So the compacted table ends up with a few files per hour rather than one. That is completely fine for the cost win - a single-hour query reads one hour-partition either way, a handful of files instead of 93,000 - it just was not the clean one-file-per-hour I first reached for.
What it should have cost
Here is the same 170 MB dataset under the different approaches. The pricing is eu-central-1, S3 Tier2 GETs at $0.0004 per 1,000.
| Approach | Cost |
|---|---|
| Unpartitioned, full-table scan per hour, re-run many times (what I did) | ~$350 |
| Unpartitioned, one clean pass, no restarts | still tens to hundreds of dollars |
| Hour-partitioned, one pass | ~$0.11 |
| Compact once to Parquet, then query the compact table | a few cents one-time, then ~nothing |
Two things jump out. First, the partition scheme is the whole lever: an unpartitioned table full-scans the entire year on every one of the ~8,760 hourly queries, so it stays in tens-to-hundreds of dollars no matter how cleanly you run it, while hour-partitioning the same data drops it to cents. Second, the compaction turns the ongoing cost into a rounding error, which matters because I am going to be querying this data again.
And to be clear about the steady state: going forward, only the hourly Lambda runs, and it queries the hour-partitioned v2 table where each hour is a handful of files. On a normal day S3 costs me about half a cent. This was never going to be an ongoing bill. It was a one-time self-inflicted spike, paid entirely in request count on a trivial amount of data.
What the dashboard actually is
Since the whole reason I went down this hole was to get analytics for my blog, it is worth a word on what I ended up with - because it is public and you can run the same thing.
It is two packages. @beesolve/analytics-processor is the backend: an hourly Lambda runs a single Athena query over the last closed hour of CloudFront logs and writes totals and per-dimension breakdowns into DynamoDB with atomic counters, then daily and monthly rollup Lambdas sum the finer-grain items without ever touching Athena again. @beesolve/analytics-dashboard is the frontend: a SvelteKit app on Lambda behind CloudFront that reads that table read-only, with email-code sign-in and no passwords.
What you get on screen is an overview you can switch between hourly, daily, monthly and yearly grain, stat cards for hits, successes, errors and bytes, and breakdowns by HTTP status, country, device and top paths. All of it is driven by your own CloudFront access logs - no third-party analytics script, no cookie banner, nothing shipped to the client's browser. The traffic data never leaves your account.
There is one extra piece that exists specifically because of how real blog traffic looks. The hourly path breakdown only keeps the top 100 paths by hits per hour and collapses everything else into a single __other__ bucket. Bots hammer a lot of junk URLs, so genuine articles I actually care about kept getting pushed out of the top 100 and losing their identity in __other__. To get around that there is a separate sitemap job: it reads the site's sitemap.xml, and a cron counts hits per day only for that known set of URLs, straight from Athena into its own per-day, per-URL records. So instead of "whatever the bots hit most this hour", I get a clean per-article view of the pages I publish - which article trended over time, which were read most this month, year over year.
Both are CDK constructs, so wiring it up is a handful of lines: turn on standard logging v2 on your distribution, point the processor at the resulting Glue table, and hand the processor to the dashboard construct. The READMEs have the full snippet. This post is the story of the one step that is not in the happy path - getting the pre-existing v1 history in without setting fire to your bill.
Takeaway
If an S3-backed job costs wildly more than the size of the data can explain, stop looking at bytes and go count your requests. The trap here was not big data. It was lots of tiny files behind a table with no partitions at all, so a per-hour query had to read every one of them, over and over. Small data, small mistake, big number.
But the deeper lesson is not really about S3 - it is about Athena, and about understanding your tools before you run them at scale. Before you point Athena at a pile of objects in S3, you have to know two things together: the physical layout of those objects (how many, how big, how they are compressed) and how Athena will turn your query into reads against that layout. You cannot range-read inside a gzip file. Without partition columns, a WHERE prunes nothing and every query scans everything. Many tiny files cost far more than a few big ones for the exact same bytes. None of this is hidden - it is just not the part you look at when the code reads fine and the numbers come out right. I pointed an unpartitioned, per-hour query at 93,000 two-kilobyte files and never did the multiplication. The engine did exactly what I asked; I just asked for 813 million GET requests without realising it.
The compaction step - read the old logs once, rewrite them as hour-partitioned Parquet, and drop the legacy code path entirely - is going into @beesolve/analytics-processor as a one-shot command, so the next person with a pile of v1 logs converts them to v2 first and never gets near the expensive path. The package is public. Learn from my bill, not your own.