Step 2 · Deploy and Verify

Goal

Execute the deployment, run smoke tests and health checks, confirm monitoring shows a healthy system, and trigger a rollback immediately if anything is wrong.

Instructions

You are in workflow step 2 of the deploy-cycle. The deployment itself should be the least exciting part of the cycle. Follow the sequence, watch the signals, and do not hesitate to roll back.

Tasks to Perform

1. Execute the Deployment

# --- Kubernetes (rolling update) ---
kubectl set image deployment/myapp \
  myapp=registry.example.com/myapp:$(git rev-parse --short HEAD)

# Watch rollout
kubectl rollout status deployment/myapp --timeout=5m

# --- Docker Compose ---
docker-compose pull myapp
docker-compose up -d --no-deps myapp
docker-compose ps myapp

# --- Laravel / PHP (Deployer) ---
dep deploy production

# --- Heroku ---
heroku container:release web --app myapp-prod

# --- Run database migrations (only after new code is up if backwards-compat) ---
php artisan migrate --force          # Laravel
alembic upgrade head                 # Python
./manage.py migrate                  # Django

2. Remove Maintenance Mode

# If maintenance mode was enabled in Step 1, bring the site back up now
php artisan up

# Verify the site is up immediately
curl -o /dev/null -sw "%{http_code}" https://yourapp.com/
# Expected: 200 (not 503)

3. Smoke Tests

Run the smoke test URLs from the runbook. At minimum:

BASE="https://yourapp.com"
TOKEN="your-test-token"

# Health check
http_status=$(curl -so /dev/null -w "%{http_code}" ${BASE}/api/health)
[[ $http_status == "200" ]] && echo "✅ health" || echo "❌ health: $http_status"

# Public page
http_status=$(curl -so /dev/null -w "%{http_code}" ${BASE}/)
[[ $http_status == "200" ]] && echo "✅ homepage" || echo "❌ homepage: $http_status"

# Authenticated endpoint
http_status=$(curl -so /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer ${TOKEN}" \
  ${BASE}/api/v1/me)
[[ $http_status == "200" ]] && echo "✅ auth endpoint" || echo "❌ auth endpoint: $http_status"

# Critical business path (adjust to your app)
http_status=$(curl -so /dev/null -w "%{http_code}" ${BASE}/api/v1/products)
[[ $http_status == "200" ]] && echo "✅ products" || echo "❌ products: $http_status"

All smoke tests must pass before declaring the deploy successful.

4. Check Error Rate and Latency

# Watch error rate in the 5 min after deploy (Datadog / Grafana / CloudWatch)
# Key signals to watch:
# • HTTP 5xx error rate — should be < 0.1% and not rising
# • p99 latency — should not spike > 2× baseline
# • Apdex score — should not drop significantly

# If using CloudWatch
aws cloudwatch get-metric-statistics \
  --namespace AWS/ApplicationELB \
  --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=arn:aws:... \
  --start-time $(date -u -d "10 minutes ago" +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 60 --statistics Sum

# Check application logs for exceptions
kubectl logs deployment/myapp --since=5m | grep -E "ERROR|Exception|FATAL" | tail -20
# or
docker-compose logs --tail=100 myapp | grep -E "ERROR|Exception|FATAL"

5. Verify Migrations Applied

# Confirm migrations ran successfully in production
php artisan migrate:status --env=production | grep -v "Ran"   # should show no pending
alembic current   # should show head

# Spot-check data integrity if migrations were destructive
# e.g. count rows, check a critical record

6. Rollback Procedure (execute if any check fails)

# --- Kubernetes ---
kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp --timeout=3m

# --- Docker Compose ---
docker-compose pull myapp --tag PREVIOUS_SHA
docker-compose up -d --no-deps myapp

# --- Database migration rollback ---
php artisan migrate:rollback --step=1 --force
alembic downgrade -1

# After rollback: re-run smoke tests to confirm recovery

Rollback decision criteria — roll back if any of the following:

  • Any smoke test returns non-200
  • Error rate > 1% sustained for 2 min post-deploy
  • P99 latency > 3× baseline for 2 min
  • Any FATAL error in logs

7. Update Deployment Record

# Update CHANGELOG or release notes
# Mark the deploy as successful / rolled-back in your tracking tool

# If successful: share a brief success notice in team chat
#   "✅ vX.Y.Z deployed to production at HH:MM. All smoke tests passing."

# If rolled back: notify team, open a post-mortem ticket
#   "⚠️ vX.Y.Z deployment rolled back. Post-mortem: [link]"

Exit Criteria

  • [ ] Application deployed without errors
  • [ ] Maintenance mode lifted (if used)
  • [ ] All smoke tests return correct HTTP status codes
  • [ ] Error rate within normal range for ≥ 5 min post-deploy
  • [ ] No new FATAL/ERROR log lines from the deployed code
  • [ ] DB migrations applied and verified
  • [ ] Team notified of deployment outcome

Cycle Complete

The deploy-cycle is finished. If this was a major release, trigger the release-cycle to update changelogs and cut a GitHub Release.