Other than ec2, what AWS services were you using and how did you migrate those? At minimum, I'd guess you were using elb for load balancing, sqs for queues and elastic cache for redis.
We are still a loyal customer for some of their services. For example, we still use S3 for off-site backups and Route53 is still our primary DNS provider.
For load balancing we have moved to a Route53 (health checks and round-robin) + a group of nginx+haproxy+lua-based frontend boxes.
Everything else was either built in-house or used open-source components and wasn't really tied to EC2 infrastructure.
When I setup an AWS account for pet projects the very first thing I do is create a billing alarm that alerts me the second my bill goes above $0. This way if something is accidentally running without my knowledge, ill get an alert (email / sms) and can quickly take action.
For working with AWS apis, securing access keys, and granting least privilege, the first step to take is to make completely separate AWS accounts per environment. Make a Bigco-Prod, Bigco-Test, Bigco-Dev AWS accounts. Developers only access the Development account, so even if they connect with a highly privileged key / role and a bug wipes out your services, only the Dev account is effected and production keeps on humming. AWS makes this extremely easy to do, they offer consolidated billing to pay one bill for multiple accounts. Takes no more than 30 min to set this all up and I would recommend this for even a company with just 1 developer.
The difficult part of separate AWS accounts is that connecting between them is tricky (Security Groups, VPC, etc.). Not impossible, but a pain to get right. The spectrum between dev and prod isn't so clear-cut for a business with petabytes of data: you often want to develop systems against production data (read-only, of course), and having two copies of the data can be prohibitively expensive. Of course, with the appropriate access policies you can share the data safely, but then what about that IRC server that was non-production but then wound up part of the deployment flow and became mission-critical?
I'm still not sure why it has taken so long for IAM to support this kind of configuration, using VPC as the isolation point.
Our use case is running packer[0], which requires a large IAM profile[1], without needing to create a separate account for AMI creation. It would be awesome to issue credentials that limited Security Group creation/deletion, Instance Creation/Termination, etc to a single, VPC.
I wish AWS made it possible to move reserved instances between accounts. We have reserved instances for our test environment that are in the same account as our production instances, and I'd need to sell them from one account at a loss and repurchase them in the other account.
That is definitely a very good practice.
Still it does not eliminate the need for something like Hologram, since you still need to take care of your keys (both for security reasons) and to make sure your dev env is as close as possible to your production and ...
Didn't mean to imply no need for Hologram or not to keep dev as close as possible to production. However, I do think you should be doing this before looking into a solution like hologram. I've seen many startups that run all their environments on a single account and eventually something bad happens knocking production offline. If you were to use hologram, but still only a single AWS account you could still just as easily kill production.
Absolutely, I've seen similar things. Even after explaining the pitfalls some people are reluctant to do it. Partly because people don't want to embrace the aws ("cloud", the new way of doing things) completely, they still want to think of everything as if it is their own machines in their own data server only amazon plays the role of the ops guys.
I've only briefly read over the documentation, but this service seems to not follow deployment best practices that aws and others such as netflix have been talking about for years. Specifically the pattern of pre-baking an ami with your current version of the app you are deploying and any other needed software completely installed on the ami and then having an autoscale group be able to boot that ami up in a few seconds and start working. This greatly helps with scaling up, doing rolling upgrades and also very easy rollbacks.
The CodeDeploy service seems to operate by you manually launching base ec2 instance with a code deploy agent and then this agent will checkout your git code on the live instance, run any provisioning steps and then if things break somehow rollback all that work, still on the live instance.
I'm sure this is still a big improvement to companies who are manually sshing into servers and running deployments by hand, but as someone who pre-bakes ami's and does rolling upgrades with autoscaling groups this service seems like a step backwards.
I've been working on the CodeDeploy Integration here at Codeship and have been working with the service for a bit (as a preface on my thoughts)
While Immutable Infrastructure is also in our opinion (and I've written about this extensively) the way to go in the future updating systems in place is still the primary way to deploy systems and will be for a while. By providing a centralized systems to upload new released and manage the deployment (how many instances get the new deployment in which timeframe) you can take away some of the security problems of opening up ports for access and potential deployment errors where the SSH connection dies.
Especially when deploying into a large infrastructure connecting into each instance for update becomes painful. That's where an agent based services like CodeDeploy is really powerful and removes the single point of failure that is the machine/network that you deploy from.
With ElasticBeanstalk, Opsworks and Cloudformation they now really start to surround all the deployment workflows.
Definitely a great service that will in my opinion become very important to many many teams. You can also read more about our specific integration in our blog: http://blog.codeship.com/aws-codedeploy-codeship/
In place update is useful in the success case - agreed.
In the failure case however, even with a fleet of only 20 instances, a rolling update that has issues after the 10th instance puts you in a world of pain.
Have you written anywhere how you guys deal with operational monitoring (eg. Boundary, New Relic, etc.) when you're spinning up brand new instances all of the time?
We use librato for monitoring our build server infrastructure and mostly only look at max/min values for metrics that could mean trouble. Generally we're able to separate data of different instances by their instance id so we could look into them individually.
We use NewRelic for our Rails application on Heroku and pump Heroku data into Librato as well (we love data and metrics)
And of course you can always send me an email to flo@codeship.com with questions.
I have used Stackdriver (http://www.stackdriver.com/) before and it works good. It can get a bit pricey. They got bought by Google few months ago -- something to watch out for. I really had a good exp with their product.
Baking everything into AMIs is the "right" way to go, but if AWS had a supported, hosted git server inside their network to push into, I'd rather have the speed of deploying from git, and only bake AMIs when necessary for system upgrades.
I have the same opinion after reading over the product page and glancing at the documentation.
At my org, we currently use a very similar method of deployment that AWS CodeDeploy seems to provide, except that we wrote it in house and use some python fabric scripts to do it. It sounds like AWS CodeDeploy would help us to "AWSify" this method of deployment and also have some nice benefits of health checks/rollbacks, UI based management, and a deployment history log. However, we would essentially still be maintaining/writing bash/python scripts to do the heavy lifting.
However, we are in the process of moving to a method of deployment which uses SaltStack to prebake an AMI with everything that is required to run in production and then use Netflix's Asgard to manage the deployment of the prebuilt AMIs. We are very excited about this method of managing deployments to take advantage of AutoScaling groups and using a well defined/tested stack of tools.
Interested to hear thoughts from others using a deployment approach that they think would benefit from AWS CodeDeploy.
> The CodeDeploy service seems to operate by you manually launching base ec2 instance with a code deploy agent and then this agent will checkout your git code on the live instance, run any provisioning steps and then if things break somehow rollback all that work, still on the live instance.
Deploy and rollback doesn't happen on live instances. (Group of) Instances are taken off service for rolling deployment and you can configure what percentage of your fleet should be deployed to at once.
I have a very similar setup as the one mentioned in the article where we create a full baked ami, then create a new autoscaling group which attaches itself with a load balancer.
The reason you want a fully created ami is to greatly cut down on the time it takes to accept new traffic. Lets say you get a spike in traffic, if you have a fully baked ami your autoscaling group can get an alert of the increase in traffic, launch a new instance and that instance will start handling traffic around 60 or so seconds. If you booted a generic ami and then provisioned the box you could easily be looking at 10's of minutes before it could handle traffic depending on the complexity of your app. This could easily lead towards long periods of down time.
DevOps here! Very much this. You can use Jenkins with Packer Builder and a configuration management tool of your choice to build an AMI for each code deployment (AMI generation costs nothing except for the space of the AMI). Need to scale 100 instances in <5 minutes? EASY.
What you don't want to do is attempt to scale, and you can't reach Github, or you overwhelm your configuration management tool, etc. Your AMIs should be built so they need almost nothing from external resources before being ready to serve requests.
Totally agree. In practice, we've found that running bootstrap scripts on fresh instances is a major pain point. When an instance is spun up as a result of a load spike, you need it up ASAP so waiting for bootstrapping to finish sucks. It's really a tradeoff between flexibility (configuration in code) vs spin up speed (configuration baked in AMI).
I've been using DynamoDB since it was released. I wrote a nodejs driver[1] and a nodejs data mapper for it[2], so I have a decent bit of experience with it. Browsing the DocumentDB docs the two services seem to be very different. DynamoDB is really just a key value store with some very nice properties, but also a lot of tradeoffs. One such tradeoff is querying data is very limited in DynamoDB. In Dynamo you can only query data by its primary hash key and optional range key. These keys you must specify upfront when you create your table and cannot be changed afterwards.
DocumentDB seems much more similar to mongodb and appears to have a very flexible query ability. In my opinion, one of the best features with DynamoDB is you can tune the number of reads/writes each individual table requires. This lets you scale up and down your database and greatly helps keep costs down. This is a feature that only a hosted database service can offer. I haven't yet read any pricing info on DocumentDB, but hopefully they offer a similar feature as this is really where a hosted database service can shine.
I recently created a login flow for a RESTful API. The solution I went with was instead of thinking of it as a high level activity such as login, what I was really doing was creating auth tokens to then be used in the Authorization header in subsequent requests.
I created a /tokens endpoint, where I POST the auth credentials and in return I get back a newly generated auth token. In my opinion this is a nice RESTFul solution.
I bake ami using both packer and ansible and then do blue/green deployments using asgard from netflix. Asgard streamlines the whole process of creating a new autoscaling group, ramp up new instances and then cut of live traffic to the older instances. They also expose a simple http api to fully automate deploys.
I have been doing continuous deployments baking ami's with asgard for a couple of years now and have had great results.
I can only speak for NYC, but its very hot. A couple of months ago my startup failed and I was looking for a job. I was only interested in early stage startups and wanted to work on every part of the tech stack (frontend, backend, devops....). In the span of 2 weeks I met with over 2 dozen startups and received a number of great offers.
About 50% were through my personal network, the rest came from various online sources. Angellist worked really well for me, I highly recommend trying them if you're looking for early stage startups. Linkedin was overloaded with recruiters, I got little use from it.
Thank you! When my Marine Corps contract expires in about ten months I'm going to be looking and both the advice to try AngelList and the confidence that it will be alright will be a huge help.
