Platform-Engineering

How small differences turn into big problems. Pipeline drift rarely starts intentionally. It starts with a small change. A special case. A temporary workaround. Then it sticks. Drift Is Hard to See Two pipelines look similar. They are not. Different flags. Different versions. Different checks. No one notices until behavior diverges. You have a standard pipeline template. Version 1.2. Most teams use it. A few are still on 1.1. One team forked it months ago and never updated. Another team copy-pasted the template and made “minor adjustments.” ...

For a long time, this Azure Function felt like a success story. It was small. It was event-driven. It scaled automatically. On paper, it was exactly the kind of workload serverless is built for. Eventually, we moved it out of serverless anyway. Not because it was broken. Because it stopped being the right fit. The Function That Kept Growing The function started simple. It processed inbound data, did some validation, and pushed results downstream. Execution time was short. Volume was low. Failures were rare. ...

Human approval does not scale the way you think. Manual gates sound responsible. Someone reviews. Someone approves. Nothing risky slips through. In practice, they fail quietly. Approvals Become Rituals Over time, manual approvals turn into habits. Approve because: the build looks normal nothing failed this is routine people are waiting Approval loses meaning. It becomes a checkbox. The first few times you approve a production deployment, you take it seriously. You check the changeset. You review test results. You verify that everything looks correct. You ask questions if something seems off. ...

Consistency beats flexibility at scale. Early on, we let teams build their own pipelines. It felt empowering. It felt flexible. It felt fast. It did not scale. Flexibility Creates Variance When every team builds pipelines independently: conventions diverge checks differ deployments behave differently failure modes multiply That variance is invisible at first. It becomes painful later. Team A used Azure Pipelines. Team B used GitHub Actions. Team C used Jenkins because they inherited a project that already had it. All three were valid choices. ...

Complexity always finds a way in. Our CI/CD pipeline started with a few scripts. Build the code. Run the tests. Deploy the artifact. It worked well. For a while. Then requirements accumulated. Quietly. Complexity Did Not Arrive All at Once No one decided to make the pipeline complicated. It happened incrementally: add a security scan support another environment introduce feature flags handle hotfixes add approvals support rollback Each change made sense in isolation. ...

Systems are often designed for correctness and performance. Supportability comes later. If at all. That ordering is backwards. Supportability Is a Feature If a system cannot be understood under pressure, it is incomplete. Supportable systems have: clear boundaries predictable behavior obvious ownership simple failure modes visible state These are design choices. Clear boundaries mean you know where one service ends and another begins. You can reason about dependencies. You can isolate failures. You can answer “is this my problem or someone else’s” quickly. ...

On call is often treated like a staffing problem. Who is rotating. How often pages fire. Which alerts wake people up. Those details matter, but they are not the root cause. On call quality is largely a product of architecture. Architecture Decides Who Gets Woken Up Every architectural decision carries operational weight. Synchronous dependencies increase blast radius. Tight coupling turns small failures into outages. Hidden retries create noisy cascades. Poor isolation spreads pain across services. ...

Where Key Vault belongs and where it does not. Secrets often get treated like infrastructure. They are stored with infra. Managed by infra. Reviewed with infra. That is usually a mistake. Why Secrets Feel Like Infrastructure Secrets feel permanent. They feel critical. They feel risky. So they end up bundled with infrastructure decisions. But secrets change more often than infrastructure. They also belong closer to applications. Infrastructure teams often manage Key Vault because it lives in Azure alongside virtual networks, storage accounts, and databases. It gets deployed with Terraform or Bicep. It has firewall rules and access policies. It looks and feels like infrastructure. ...

Why permission models rot over time. Azure RBAC feels simple at first. Assign a role. Pick a scope. Move on. The problems show up later. RBAC Accumulates History Permissions tend to grow, not shrink. Temporary access becomes permanent. Emergency grants never get revisited. Roles pile up across scopes. Over time, no one remembers why access exists. They only remember that removing it feels risky. I have audited Azure subscriptions where people had role assignments from three jobs ago. Former contractors still had Contributor access years after their contracts ended. Service principals created for one-off migrations still had Owner access to production. ...

Practical IAM, not zero trust theater. Access control often swings between two extremes. Everything open. Everything locked down. Neither works. Why Overly Restrictive IAM Fails When access is too hard to get: engineers work around it secrets get shared permissions creep quietly reviews become rubber stamps Security that blocks work does not create safety. It creates shadow systems. I have seen this pattern repeat across multiple teams. Access requests take days or weeks to get approved. The approval process requires three levels of sign-off, none of which understand the technical need. Engineers get frustrated and find workarounds. ...