Production Is Not a Place to Discover Things
Production should be where the system runs, not where we first discover how it behaves.
DevOps
Tip
Oct 4, 2026 · 2 min read
Production Is Not a Place to Discover Things
Production is where software runs. It shouldn't be where we discover how it works.
A lot of production incidents don't happen because the code is fundamentally broken. They happen because something important was never actually tested, measured, or considered before deployment.
A migration that worked locally. A background worker that silently stopped. An API with no meaningful timeout. A deployment with no rollback path. A log that tells you something failed, but not why.
None of these necessarily look dangerous during development. They become dangerous when the system has real users, real data, and real consequences.
Over time, I've started to think about production differently.
Before deploying something, I don't just ask: Does it work?
I also ask: What happens when it doesn't?
What happens if the database is slow? What happens if an external service doesn't respond? What happens if the worker crashes halfway through a task? What happens if the same request arrives twice? What happens if the deployment needs to be rolled back? What happens if nobody is watching?
These questions usually lead to less glamorous engineering work: timeouts, retries, idempotency, health checks, structured logs, monitoring, migrations, backups, and rollback strategies.
But that's often what separates software that works from software that can actually be trusted.
Production will always surprise us. No amount of testing can eliminate every unknown.
The goal isn't to predict everything.
The goal is to make the unknowns smaller, failures understandable, and recovery possible.
Production should be where the system runs — not where we first learn how the system behaves.
If we're going to discover something important, it's better to discover it before production.
Related Notes
The API Worked. The Architecture Didn’t.
A practical look at why a working API does not always mean a healthy architecture, covering coupling, synchronous work, database bottlenecks, caching, microservices, failure modes, and the importance of clear boundaries.
When Redis Is Not the Solution
Redis is fast and extremely useful, but that doesn't make it the right solution for every backend problem. A practical look at when Redis helps with caching, rate limiting, OTPs, locks, queues, sessions, and temporary data—and when it simply adds unnecessary infrastructure and complexity.
Django Transactions: What atomic() Actually Protects
Learn how Django transactions work, what transaction.atomic() actually protects, and how to handle race conditions, row locking, database constraints, nested transactions, and post-commit side effects with select_for_update() and on_commit().
The Hidden Cost of Django REST Framework Serializers
Learn how Django REST Framework serializers can cause N+1 queries, slow API responses, and unnecessary database work—and how to optimize them with select_related, prefetch_related, annotations, and better serializer design.
Building a Django Package — Part 4: Publishing django-healthkit to PyPI
In this final part, we prepare django-healthkit for release, build and validate the package, test it on TestPyPI, publish it to PyPI, and create a Git tag and GitHub release.
Building a Django Package — Part 3: Health Check Manager and Endpoint
In Part 3, we connect the database and cache health checks, build the health check manager, expose a Django health endpoint, measure check latency, and return a structured health status.
Building a Django Package — Part 2: Database and Cache Health Checks
In this part, we implement the first health checks for django-healthkit, covering database connectivity and Django cache functionality with simple, independent, and testable checks.
Building a Django Package — Part 1: Setting Up django-healthkit
In this part, we build the initial structure of django-healthkit, configure the package with pyproject.toml, and prepare it for development.
Building a Secure Webhook Receiver for Server-to-Server Communication
How I built a secure FastAPI webhook receiver using RSA signatures to enable authenticated server-to-server communication, proxy requests, and connect applications across different network environments.
Why I Switched to Conventional Git Commit Messages
Why I switched from inconsistent Git commit messages to a Conventional Commits style, with practical examples for cleaner and more maintainable Git history.
Building django-healthkit
How a simple health endpoint for Bidar turned into django-healthkit, a lightweight Django health-check package.
Designing a Gold Jewelry E-Commerce Database with Django
Learn how to design a scalable Django database for a gold jewelry e-commerce platform by modeling products, attributes, and purchasable product variants using real-world domain-driven design principles.
How to Fix Common Next.js 16 Build Errors (Proxy, Dynamic Rendering & Revalidation)
Learn how to fix common Next.js 16 build errors, understand Proxy, Dynamic Rendering, and Revalidation, and improve your application's SEO and performance.
Why I Built cyber-ui: A Minimal Design System for Developers
How my projects, from backend systems to monitoring tools, led me to create a minimal and personal UI foundation.