Production Readiness Checklist: Avoid 3AM Outages with These

The Production Readiness Checklist

In the last article, I walked through finding and fixing the slow parts of a SaaS before launch: N+1 queries, missing indexes, unbounded payloads, the usual suspects. That work makes the product fast. It says nothing about whether the product is actually ready to run unattended, at 3am, when something inevitably goes wrong.

This is the closing article of Module 4 in the Full Stack SaaS Masterclass, and it's meant as a checklist you go through before flipping the switch on a real launch, not a tutorial you follow once and forget. Every item on it is something I've seen skipped on a first release, and every skip eventually became an incident.

"Production ready" doesn't mean gold-plated. It means the gaps that turn a minor bug into a multi-hour outage have been closed, and the ones that remain are gaps you've consciously accepted rather than ones nobody thought about. That distinction is the whole point of a checklist: it forces the conscious part.

Configuration and Secrets Are Locked Down

The most common first-launch mistake isn't a code bug. It's a config that quietly behaved differently than everyone assumed. A .env file with production credentials committed by accident. A CORS policy left wide open because it was easier during development. A database connection pool sized for a laptop, not for concurrent traffic.

Before launch, walk through every environment variable your app reads and confirm three things: it has a real value in production, that value came from a secrets manager rather than a plaintext file in the repo, and there's a documented fallback (or a hard failure) if it's missing.

import { z } from &#39;zod&#39;;

const envSchema = z.object({
  NODE_ENV: z.enum([&#39;development&#39;, &#39;staging&#39;, &#39;production&#39;]),
  DATABASE_URL: z.string().url(),
  REDIS_URL: z.string().url(),
  JWT_SECRET: z.string().min(32),
  CORS_ORIGIN: z.string().url(),
  SENTRY_DSN: z.string().url().optional(),
});

export type Env = z.infer;

export function validateEnv(config: Record): Env {
  const result = envSchema.safeParse(config);
  if (!result.success) {
    console.error(&#39;Invalid environment configuration:&#39;, result.error.format());
    process.exit(1);
  }
  return result.data;
}

Wiring this into NestJS's ConfigModule at bootstrap means a missing or malformed secret fails the deploy. It doesn't fail silently at 2pm on a Tuesday when a customer hits the one code path that touches it. The tradeoff is a slightly stricter local dev setup, since every new environment variable now needs a schema entry. That's a cost worth paying; a schema that drifts from reality is a bug waiting to be discovered by a user, not by CI.

The pitfall I see most often here is treating staging and production as identical except for the URL. They rarely are. Different rate limits on a third-party API, different database sizes, different feature flags. Document the differences explicitly rather than assuming staging behavior is a preview of production behavior.

Health, Readiness, and Graceful Shutdown

Your orchestrator, whether it's ECS, Kubernetes, or a plain load balancer, needs a reliable way to know if an instance can take traffic. Without it, a deploy can route requests to a container that's still booting, or keep sending traffic to one that's stuck.

The three things worth having are a liveness check (is the process alive), a readiness check (can it actually serve requests, meaning the database and cache connections are up), and a graceful shutdown handler that stops accepting new work before the process exits.

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  app.enableShutdownHooks();
  const server = await app.listen(process.env.PORT ?? 3000);
  process.on(&#39;SIGTERM&#39;, async () =&gt; {
    console.log(&#39;SIGTERM received, draining connections&#39;);
    await app.close();
    server.close(() =&gt; process.exit(0));
  });
}
bootstrap();

Without this, a rolling deploy or an autoscaler scaling down can kill an instance mid-request, and the client gets a connection reset instead of a response. The fix costs almost nothing to implement. It removes an entire class of intermittent, hard-to-reproduce errors that show up as noise in your error tracker rather than as a clear signal.

Backups Exist and, More Importantly, Restores Are Tested

Every team says they have backups. Far fewer have actually restored from one. A backup you've never restored is a hypothesis, not a safety net. Automated snapshots from a managed Postgres provider are a reasonable starting point, but they don't protect you from a bad migration that corrupts data quietly over a week, or a schema change that makes an old backup incompatible with current application code.

The practice worth adopting before launch is a quarterly restore drill: take a recent backup, restore it to a scratch environment, and run your application's smoke tests against it. This surfaces two things people usually get wrong: a restore that silently drops a table due to permission issues, and a restore time that's much longer than anyone assumed, which matters a great deal when you're calculating an acceptable recovery time objective.

pg_restore \
  --dbname=&#34;$SCRATCH_DATABASE_URL&#34; \
  --clean \
  --if-exists \
  --no-owner \
  --jobs=4 \
  ./backups/latest.dump

The tradeoff is time. A quarterly drill takes an afternoon and produces nothing customer-visible, which makes it easy to deprioritize against feature work. I'd argue that's exactly why it needs to be scheduled rather than left to "whenever there's time," because that time never quite arrives on its own.

Observability That Answers "What Broke" in Minutes, Not Hours

By the time you're running in production, you should already have metrics, structured logs, and error tracking wired up from earlier articles in this module. The readiness question here is narrower: when something breaks, can an on-call engineer go from alert to root cause without pulling in three other people?

That means alerts are tied to symptoms a customer would notice (elevated error rate, rising latency, a queue backing up) rather than to infrastructure internals nobody can act on at 3am. Logs need to carry a request ID that ties a customer-facing error back to the exact backend trace that produced it. And the on-call rotation needs real access to the dashboards and log queries they'd require, tested before the first real incident rather than during it.

import { Injectable, NestMiddleware } from &#39;@nestjs/common&#39;;
import { randomUUID } from &#39;crypto&#39;;
import { Request, Response, NextFunction } from &#39;express&#39;;

@Injectable()
export class RequestIdMiddleware implements NestMiddleware {
  use(req: Request, res: Response, next: NextFunction) {
    const requestId = (req.headers[&#39;x-request-id&#39;] as string) ?? randomUUID();
    req.headers[&#39;x-request-id&#39;] = requestId;
    res.setHeader(&#39;x-request-id&#39;, requestId);
    next();
  }
}

A pitfall worth naming directly: alert fatigue. A team that gets paged for every 500 error, including expected ones like a client submitting invalid input, learns to ignore alerts within a month. Reserve pages for things that need a human right now, and route everything else to a dashboard someone checks during business hours.

A Rollback Path That Doesn't Require a Hero

Launch day confidence usually comes from "we tested it in staging," which is necessary but not sufficient. The question a production readiness review should force is: if this release breaks something for real users an hour after deploy, what's the actual sequence of commands to get back to the last known good state, and has anyone run that sequence before?

For a stateless application, rolling back the deployed image is usually fast if your CI/CD pipeline tags releases by commit SHA and your orchestrator supports a one-command rollback. The harder case is a release that shipped alongside a database migration. If the migration is backward compatible, meaning the old application version still works against the new schema, a rollback is safe. If it isn't, you're stuck rolling forward with a fix instead, which takes longer and is riskier under pressure.

-- Backward compatible: old code ignores the new nullable column
ALTER TABLE organizations ADD COLUMN plan_tier TEXT;

-- Not backward compatible: old code will fail on the NOT NULL constraint
ALTER TABLE organizations ADD COLUMN plan_tier TEXT NOT NULL;

The practical rule is to ship schema changes in two steps: add the column nullable, deploy, backfill, then add the constraint in a later deploy once the old code path is gone. It's more deploys for the same feature, and that's the cost. What you get in exchange is a rollback option that actually works, which matters far more the one time you need it than it costs on every other release.

Security Basics That Are Easy to Defer and Expensive to Skip

None of these are exotic. Rate limiting on authentication and password reset endpoints, so a credential-stuffing attempt doesn't take down your database. Dependency scanning in CI, so a known vulnerability doesn't sit in production for months because nobody was watching. HTTPS enforced everywhere, including internal service-to-service calls if they cross a network boundary you don't fully control. Least-privilege IAM roles, so a compromised container doesn't have write access to every S3 bucket in the account.

# GitHub Actions: dependency vulnerability scan on every PR
name: security-audit
on: [pull_request]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: &#39;20&#39;
      - run: npm ci
      - run: npm audit --audit-level=high

The tradeoff with npm audit --audit-level=high gating a PR is the occasional false positive: a vulnerability in a transitive dependency that doesn't actually apply to how you use the package. That's a legitimate annoyance, and the fix is an explicit, reviewed suppression list, not disabling the check.

Next Steps

Print this checklist. Go through each item for your service. If you can't check something off, you've found a risk you need to address before launch. The goal isn't perfection — it's closing the gaps that turn a minor bug into a multi-hour outage.

Production Readiness Checklist: Avoid 3AM Outages with These 7 Steps

The Production Readiness Checklist

Configuration and Secrets Are Locked Down

Health, Readiness, and Graceful Shutdown

Backups Exist and, More Importantly, Restores Are Tested

Observability That Answers "What Broke" in Minutes, Not Hours

A Rollback Path That Doesn't Require a Hero

Security Basics That Are Easy to Defer and Expensive to Skip

Next Steps

Editor's Take

Key Takeaways

Why It Matters

Get the weekly digest

Stop Using Date.now() for Latency: Use performance.now()

Why Google Translate Crashes React Apps (And the Fix)