Move production documentation from Playbook to Guides

README.md

@@ -37,6 +37,7 @@ programming in style.
 - [Security](/security/)
 - [Web](/web/)
 - [Web Performance](/web-performance/)
+- [Production](/production)
 
 ### Languages
 

production/README.md

@@ -0,0 +1,23 @@
+# Production
+
+We live in a magical modern era where many problems have already been solved for
+us. We focus on the core product as much as possible and outsource operations as
+much as possible to external services.
+
+This saves time and money. We can get started using those services in minutes,
+and pay a service tens or hundreds of dollars per month instead of paying
+developers thousands or tens of thousands.
+
+We often create a Google spreadsheet for our clients, listing the monthly cost,
+description, and credentials of each of the products' external services. It
+includes line items like GitHub, Heroku, SendGrid, New Relic, and Airbrake.
+
+* [Production Checklist](production-checklist.md)
+* [Domain Names and DNS](domain-names-and-dns.md)
+* [SSL Certificates](ssl-certificates.md)
+* [Hosting](hosting.md)
+* [Performance Monitoring](performance-monitoring.md)
+* [Log Collection](log-collection.md)
+* [Error Tracking](error-tracking.md)
+* [Transactional Email](transactional-email.md)
+* [Payment Processing](payment-processing.md)

production/domain-names-and-dns.md

@@ -0,0 +1,21 @@
+---
+title: Domain Names and DNS
+summary: >
+    We use Porkbun to buy domain names, and Cloudflare to host DNS.
+---
+
+Use [Instant Domain Search][instant-domain-search] to see what's available.
+
+Use [Porkbun][porkbun] to buy and maintain domain names.
+
+Once your domain has been purchased, point its name servers to
+[Cloudflare][cloudflare]. We like Cloudflare because it supports `CNAME
+Flattening`, which is required in order for a root domain to work with Heroku.
+
+Follow the [Custom Domains][custom-domains] tutorial to set up root and
+subdomains on Heroku.
+
+[cloudflare]: https://www.cloudflare.com
+[custom-domains]: https://devcenter.heroku.com/articles/custom-domains
+[instant-domain-search]: https://instantdomainsearch.com
+[porkbun]: https://porkbun.com

production/error-tracking.md

@@ -0,0 +1,20 @@
+---
+title: Error Tracking
+summary: >
+  Despite our best efforts, sometimes things go wrong in production. Error
+  tracking services notify us and give us the context to fix the bugs quickly.
+---
+
+We use error tracking software to capture exceptions in production.
+
+We try to use tools that support all of the languages and platforms we use
+to build applications, and those which provide integrations that make it easy to
+stay on top of errors. Such as Sentry's [Slack integration] for real-time
+alerts and [Trello integration] so we can prioritize bugs in the same place
+as features.
+
+We like and use [Sentry].
+
+[sentry]: https://sentry.io
+[slack integration]: https://sentry.io/integrations/slack/
+[trello integration]: https://sentry.io/integrations/trello/

production/hosting.md

@@ -0,0 +1,106 @@
+---
+
+title: Hosting
+summary: >
+  We use services like Heroku, Fastly, and Amazon S3 to scale our products from
+  prototype to production without hiring an operations team. We use Flightdeck
+  for high security and enterprise infrastructure.
+
+---
+
+## Our defaults
+
+We use [Heroku][]. It's a Platform as a Service (PaaS) built on Amazon's cloud
+infrastructure. It is simple to use when our app is just a toy and is built to
+scale up for high concurrency or high sustained load.
+
+They act as our outsourced operations team, helping us deliver working
+software, instead of dealing with already solved problems which we would have
+to worry about with a bare-bones hosting provider.
+
+Heroku uses conventions to make decisions for us that are unnecessary for us to
+make. Some things like web servers and app servers are solved problems and can
+be standardized across applications, regardless of the framework that they're
+written in. Heroku has additional features like review apps (which helps in
+code review by being able to test behaviour as it is used), pipelines (which
+makes handling review, staging and production apps easier) and integrations
+with other third-party services.
+
+The cloud promises lower operating costs, especially at the beginning when
+capacity can be lower. Forget about sunk costs of expensive servers.
+
+The cloud and the services it enables will empower our clients' businesses to
+start and operate in a manner that has never been possible before without
+significant upfront investment.
+
+If we offer file uploads for features like user avatars, we upload them to
+[Amazon S3][].
+
+We also serve our images, CSS, and JavaScript assets from a CDN such as
+[Fastly][] or [Cloudflare][].
+
+Beyond Heroku, we also support and promote the use of Infrastructure as Code
+(IaC), especially with [Terraform][] for [AWS][] and [Kubernetes][] deployments.
+
+## Higher security needs
+
+The common [Heroku][] platform will not suffice when a strong level of security
+or auditability is required, such as complying with SOC2 or NIST standards.
+Typically, financial and healthcare scenarios involve this need, as do public
+companies, enterprises, and those that do business with a federal or national
+government.
+
+Heroku does offer an [enterprise level][heroku enterprise]. This requires
+arranging a negotiated contract with their parent company, Salesforce. Rates are
+substantially higher than the common Heroku platform and have to be renewed
+annually and needed capacity predicted. This does however offer the same easy to
+deploy features as before. The additional cost can be offset by a lower need for
+in-house Site Reliability Engineering (SRE) staff, though it is recommended to have someone available that can
+periodically manage configuraton and monitor server loads.
+
+A more sophisticated and adaptable solution is to consider Amazon Web Services
+([AWS][]). We have created [Flightdeck][] to help manage high-security,
+high-availability SOC2-compliant infrastructure. Flightdeck allows us to deploy
+and manage our applications on Kubernetes. [Kubernetes][] is an open-source
+container orchestration system that automates the deployment, scaling, and
+management of containerized applications.
+
+We strongly recommend that when involving a full AWS-based solution, the organization has [hired us to do this](https://thoughtbot.com/services/platform-engineering), has its own SRE or infrastructure management staff, or has hired another service provider.
+
+These cannot be managed well casually and require period
+updates and tuning to perform reliability and keep costs contained. We are able
+to provide training for using Flightdeck, but any AWS-based solution at this
+level will not be install-and-forget.
+
+If a high level of security or auditability are not crucial, then the cost and
+complexity introduced by either [Heroku Enterprise][] or Flightdeck is likely
+not worth it. For instance, merely wanting to add [Amazon S3][] or
+[AWS Cloudfront][] (CDN) will not be worth adopting a full AWS-based solution given
+the complexity and cost introduced.
+
+## Technologies
+
+We are well-versed in the technologies we use and when to involve them. We don't
+expect our clients to be experts in these technologies, but we do expect them to
+trust us to make the right decisions.
+
+We use the following technologies and services:
+
+- Heroku
+- Amazon S3
+- Fastly
+- Flightdeck
+- Docker
+- Kubernetes
+- Terraform
+
+[amazon s3]: http://aws.amazon.com/s3/
+[aws]: https://aws.amazon.com/
+[cloudflare]: https://www.cloudflare.com
+[aws cloudfront]: https://aws.amazon.com/cloudfront/
+[fastly]: http://www.fastly.com/
+[flightdeck]: https://github.com/thoughtbot/flightdeck
+[heroku]: http://heroku.com
+[heroku enterprise]: https://www.heroku.com/enterprise
+[kubernetes]: https://kubernetes.io/
+[terraform]: https://www.terraform.io/

production/log-collection.md

@@ -0,0 +1,28 @@
+---
+title: Log Collection
+summary: >
+  Helpful information often ends up in our applications' log files, but they're
+  discarded by default, so its important to set up a collection service.
+---
+
+Most applications write useful debugging information to logs. On Heroku, these
+go to standard output by default and are eventually discarded.
+
+We typically use [Logentries] to accept logs from Heroku and other sources. Once
+sent to Logentries, you can search previous logs and set up alerts for errors
+outside the Rails stack, such as out of memory errors.
+
+If we're adding Logentries to a client project, the best solution is to have
+the client set up their own Logentries account and add the relevant thoughtbot
+members. If the client doesn't want to set it up or there's too much red tape,
+we can create a new project on the thoughtbot Logentries account and then
+delete it once our engagement has ended.
+
+We don't use the Heroku Logentries addon, as it will automatically create a
+number of alerts and send them to every admin on our Heroku organization.
+
+To set up Logentries for a Heroku application, follow the
+[instructions for setting up a syslog drain][logentries-setup-instructions].
+
+[Logentries]: https://logentries.com
+[logentries-setup-instructions]: https://logentries.com/doc/heroku/#syslog_drain

production/payment-processing.md

@@ -0,0 +1,19 @@
+---
+title: Payment Processing
+summary: >
+  We use services like Stripe and Plaid to easily and securely enable
+  payment features in our products.
+---
+
+For collecting payments from users via credit or debit card, we use [Stripe].
+It is a payment gateway and merchant account. We also use it for recurring
+billing.
+
+Charges for Stripe will vary depending on usage. Successful credit card charges
+are 2.9% + 30 cents. There are no setup fees, monthly fees, or card storage
+fees.
+
+For sending money to users' bank accounts via ACH, we use [Plaid].
+
+[Plaid]: https://plaid.com/
+[Stripe]: http://stripe.com/

production/performance-monitoring.md

@@ -0,0 +1,34 @@
+---
+title: Performance Monitoring
+summary: >
+  We love debugging performance problems in production, but only when the right
+  monitoring tools are in place to help guide our efforts.
+---
+
+We use [Sentry] or [Skylight] to monitor performance of production applications.
+
+Debugging performance might be the best part of a developer's job. There's a
+clear, numeric problem. When we fix it, that number improves. We can say
+things like "We made this 175% better."
+
+There's many established techniques for fixing performance problems. A number
+of them come "for free" with Rails + Heroku:
+
+* Amazon server clusters
+* gzipping
+* [Asset pipeline]
+* [SQL query caching]
+
+A number of them require developer thought:
+
+* Database indexing
+* Eager loading
+* HTTP caching
+
+Page caching is the heaviest handed technique we have, but if we can cache an
+entire page and push it into a CDN, that will be the fastest option.
+
+[Asset pipeline]: http://guides.rubyonrails.org/asset_pipeline.html
+[Sentry]: http://www.sentry.io
+[Skylight]: https://www.skylight.io
+[SQL query caching]: https://tbot.io/sql-caching

production/production-checklist.md

@@ -0,0 +1,51 @@
+---
+title: Production Checklist
+summary: >
+  Running through a short checklist when setting up a production environment or
+  preparing for a launch ensures we don't miss an important step.
+---
+
+We have found that a short checklist is valuable when setting up a new
+production environment or preparing for a launch:
+
+- Are we on the [latest Heroku stack][heroku-stacks]?
+- Are we using a concurrent web server?
+  See [how to deploy with Puma][deploy-rails-with-puma].
+- Are long-running processes such as email delivery being run in background
+  jobs?
+  See [how to set up background jobs][background-job-setup].
+- Are there redundant (at least two) web and background processes running?
+- Are we using SSL?
+  See "[SSL Certificates]" section below.
+- Are API requests being made via a separate subdomain (`api.example.com`)? Even
+  if the same app, this gives us architectural flexibility in the future.
+- Is the [latest Ruby][latest-ruby] defined in the `Gemfile`?
+  See [how to set it up][specify-bundler-ruby].
+- Is [config stored in environment variables][12-factor-environment]?
+- Are deploys done manually at a scheduled time when teammates are fresh and
+  available if something goes wrong?
+- Do deploys follow [a well-documented script][deploy-script]?
+- Are we sending logs to a remote logging service?
+  See "[Log Collection]" section below.
+- Are we using a Heroku "Standard" database or higher?
+  See [Heroku production databases][heroku-production-databases].
+- Are we backing up our production database?
+  See [Heroku PGBackups][heroku-postgres-backups].
+- Are we monitoring performance and uptime?
+  See "[Performance Monitoring]" section below.
+- Are we tracking errors?
+  See "[Error Tracking]" section below.
+
+[12-factor-environment]: http://12factor.net/config
+[error tracking]: ./error-tracking.md
+[log collection]: ./log-collection.md
+[performance monitoring]: ./performance-monitoring.md
+[ssl certificates]: ./ssl-certificates.md
+[background-job-setup]: https://tbot.io/background-jobs
+[deploy-rails-with-puma]: https://tbot.io/deploy-rails-puma
+[deploy-script]: https://github.com/thoughtbot/guides/blob/main/rails/how-to/deploy_a_rails_app_to_heroku.md
+[heroku-postgres-backups]: https://tbot.io/heroku-pgbackup
+[heroku-production-databases]: https://tbot.io/production-db
+[heroku-stacks]: https://devcenter.heroku.com/articles/stack
+[latest-ruby]: https://www.ruby-lang.org/en/downloads/
+[specify-bundler-ruby]: https://bundler.io/guides/gemfile_ruby.html

production/ssl-certificates.md

@@ -0,0 +1,23 @@
+---
+title: SSL Certificates
+summary: >
+  Working with SSL and DNS requires coordination and attention, but Let's
+  Encrypt and Heroku make the process painless.
+---
+
+We want to ensure that our user's data is encrypted during transit and that the
+data they may provide is sent securely. [The HTTPS-Only Standard][] provides a
+great, and much more detailed, description of the reasoning behind this.
+
+This has become much easier with [Let's Encrypt], which provides a free to use,
+automatic and secure certificate authority. It's integrated with Heroku, which
+allows us to use their [Automated Certificate Management][heroku] feature.
+
+When we need wildcard certificates (e.g.: when we want to use the same
+certificate across `www.`, `staging.`, etc.), or those with advanced features,
+we use [dnsimple].
+
+[The HTTPS-Only Standard]: https://https.cio.gov/everything/
+[Let's Encrypt]: https://letsencrypt.org
+[heroku]: https://devcenter.heroku.com/articles/automated-certificate-management
+[dnsimple]: https://dnsimple.com/ssl-certificate

production/transactional-email.md

@@ -0,0 +1,22 @@
+---
+title: Transactional Email
+summary: >
+  We rely on email experts like SendGrid to reliably deliver emails to our
+  users without ending up in their junk folder.
+---
+
+We use [SendGrid] to have our application deliver email to users, known as
+[transactional email].
+
+Examples of transactional email are:
+
+* Confirmations
+* Follow ups after the first 3 days of use
+* Free trial is expiring
+* Message another user in the system
+
+We use SendGrid directly, not via the Heroku add-on, in order to avoid being
+lumped under the same IP group as others on Heroku (who might be misbehaving).
+
+[SendGrid]: http://sendgrid.com
+[transactional email]: https://tbot.io/transactional-email