Project · Stand up the lab and prove it matches the reference

project

hard

module project

Ship something real. Submit your work when you're done.

Brief

Bring up both databases — your own or the compose file — load the dataset at a scale you choose, and produce the evidence that the lab is sound. The deliverable is not 'it ran'; it is a verification report a reader could use to confirm your environment will produce the numbers the rest of the course expects, plus a committed baseline to measure everything else against.

Deliverables

Your `.env.example` (credentials removed) and a note stating which option you chose — compose, managed, or an existing install — and why.
The output of the connection check showing both engines reachable, with versions.
A verification report covering all four checks: row counts against your scale, orders-per-user coefficient of variation, physical size against cache size, and confirmation that no non-primary indexes exist.
A committed baseline file with median timings and plans for the five reference queries, annotated with your scale and machine.
A short troubleshooting note recording any failure you hit and how you resolved it, in the format of the seven in task 8.

How we grade it

Row counts match the declared scale exactly in both engines, and the scale is stated in the report.
The coefficient of variation for orders per user is above 2, demonstrating the skew survived the load.
Total data size is demonstrably larger than the configured cache, with both numbers shown — this is what makes the I/O lessons work.
Every plan in the baseline is a sequential scan or COLLSCAN, proving the lab starts with no indexes.
The baseline reports a median over repeated warmed runs, not a single timing, and records scale and machine alongside.

Hints

Do the connection check before the seed, not after. A load against a half-working connection produces a partially populated dataset that passes a casual glance and then makes every measurement in the course quietly wrong. If you are on a free managed tier, start with SCALE=small and confirm the whole pipeline works end to end before deciding whether the default will fit — the storage limits on those tiers are lower than people expect.

Expected output

$ node check.js postgres ok PostgreSQL 16.4 on aarch64-unknown-linux-gnu mongodb ok MongoDB 7.0.14 $ SCALE=default node seed.js postgres users 200000/200000 postgres orders 2000000/2000000 postgres events 5000000/5000000 postgres ANALYZE done mongodb users 200000/200000 mongodb orders 2000000/2000000 mongodb events 5000000/5000000 seeded in 174s $ mongosh "$MONGO_URL" --quiet verify.js users 200000 / 200000 orders 2000000 / 2000000 events 5000000 / 5000000 orders/user cv 4.18 data 612MB uncompressed, 288MB on disk, cache 256MB lab verified — every later measurement will be comparable $ ./baseline.sh > baseline-20260913.txt ## postgres orders by user+status median 1.402s plan: Seq Scan on orders (actual time=204.1..1402.1 rows=23 loops=1) Filter: ((user_id = 8812) AND (status = 'refunded'::text)) Rows Removed by Filter: 1999977 Buffers: shared hit=32210 scale=default · MacBook Pro M2, 16GB · docker compose

Stretch goals

Load all three scales into separate databases and record how the baseline timings change, then check whether the ratios stay constant even though the absolute numbers do not.

Point the same scripts at a managed Postgres and a managed MongoDB, and document exactly which settings you could not apply and what you substituted.

Add a CI job that stands the lab up from scratch and fails if the verification report reports any problem — proving the lab is reproducible by someone other than you.

Extend the generator with a fourth scale sized to your own production database, and compare its skew against the synthetic power law.

Write a teardown that measures how long a full rebuild takes at each scale, so you know the real cost of experimenting freely.