As verbose as cloud formation is, I really like it. You can represent the entire state of your architecture in a single JSON file: VPCs, EC2 instances, Elasicache clusters, security groups, IAM roles, everything. Updates to the cloud formation template are atomic and can be rolled back, saves you the trouble of writing a long runbook to install your system, and your template can be kept in version control.
You want to talk weirdly over-engineered, check out SWF
We break our cloud formation stacks up and reference the outputs of those stacks in consuming stacks. You have a VPC, IAM, IAM policy, security group, network acl, and then your application stacks. So, if you have an application that references a security group from it's respective stack and you want to change those rules, you just update the resource in the security group stack and the application stack never needs to get updated. (edit: by separating stacks I mean that they live in different templates, not just different stacks in one file)
I don't quite see the advantage. If you update a stack it will only do the minimal required changes. So if you have an EC2 instance and a security group defined in one stack and you update the security group the EC2 instance won't be affected, it's not like it would be shut down and an identical one would be spun up.
Late reply, but mainly shared resources like security groups (like a specific security group for NTP or HTTP/S), IAM roles, DB's, Kinesis streams, etc. The other reason being the limit on template length. You can't spec out a 3-tier VPC with network ACLs and security groups in a single template.
SWF is awesome, but it can feel very abstract for most use-cases. In light of this, one of my good colleagues wrote a library called Super Simple Worfklow. It provides the level of simplicity that one would have expected from SWF to begin with. We opened sourced it at my company, check it out[1]. We use this library for production services.
(Full disclosure: I work at Bazaarvoice, and my colleague does, too!)
The user interface of SWF looks like it is the most neglected thing in AWS. Faar too much whitespace and crappy layout makes navigating it a real PITA.
The Step Functions UI on the other hand is really rather intuitive.
Nice! SWF is sadly an under-utilized service outside of Amazon - great to see more accessible abstractions being built on top of it. We wrote a simple abstraction around SWF to make is easier to work with:
