Add dynamic datacenter configuration infrastructure (Phase 1)

[ksirrah13]

Summary

Implements Phase 1 & 2 of dynamic datacenter configuration:

• Phase 1: Centralized infrastructure (single source of truth)
• Phase 2: Refactored 4 core templates to eliminate deeply nested conditionals

This lays the groundwork for eliminating manual updates across 14+ template files when adding new datacenters.

Problem

Currently, adding a new Datadog datacenter requires manually updating 14+ CloudFormation template files with:

• Site names in AllowedValues lists (9 files)
• Deeply nested conditional logic for account IDs (4 files) ← Fixed in Phase 2!
• Hardcoded endpoint URLs (3 files)
• Version number increments (2 files)

This is error-prone, time-consuming (~2 hours per datacenter), and doesn't scale.

Phase 1: Foundation ✅

Establishes single source of truth and automation scripts. No template changes - zero risk to production.

Files Added

1. datacenters.yaml - Centralized Configuration

• Single source of truth for all 8 production datacenters + 1 test datacenter
• Includes: site domains, AWS account IDs, and all endpoint URLs (API, logs, metrics, config)
• Easy to maintain: adding a new datacenter is now a single YAML entry

2. scripts/generate_datacenter_configs.py - Generation Script

• Reads datacenters.yaml and generates CloudFormation snippets
• Outputs 7 files to generated/ directory

3. scripts/validate_datacenters_config.py - Validation Script

• Validates datacenters.yaml structure and content
• Exit code 0 on success, 1 on failure (CI-ready)

4. scripts/README.md - Complete Documentation
5. .gitignore - Updated to exclude generated/

Phase 2: Template Refactoring ✅

Replaced deeply nested conditionals with clean CloudFormation Mappings pattern.

What Changed

BEFORE (7 levels deep, unmaintainable):

DdAWSAccountId: !If
  - IsAP1
  - "417141415827"
  - !If
    - IsAP2
    - "412381753143"
    - !If
      - IsPRTEST07
      - "393946873269"
      - !If
        - IsGov
        - !If
          - IsAWSGovCloud
          - "065115117704"
          - "392588925713"
        - "464622532012"

AFTER (Clean, scalable, 3 lines):

Mappings:
  DdAccountIdBySite:
    "datadoghq.com":
      AccountId: "464622532012"
    "ap1.datadoghq.com":
      AccountId: "417141415827"
    # ... all sites defined once

DdAWSAccountId: !If
  - IsGov
  - !If
    - IsAWSGovCloud
    - !FindInMap [DdAccountIdBySite, !Ref DatadogSite, AccountIdGovCloud]
    - !FindInMap [DdAccountIdBySite, !Ref DatadogSite, AccountId]
  - !FindInMap [DdAccountIdBySite, !Ref DatadogSite, AccountId]

Files Refactored (4 core templates)

1. ✅ aws_quickstart/main_extended.yaml

   • Added Mappings section
   • Removed IsAP1, IsAP2, IsPRTEST07 conditions (no longer needed)
   • Simplified account ID lookup from 7-level nesting → 3 lines

2. ✅ aws_quickstart/main_v2.yaml

   • Same refactoring pattern

3. ✅ aws_quickstart/main_workflow.yaml

   • Same refactoring pattern

4. ✅ aws_organizations/main_organizations.yaml

   • Added Mappings section
   • Refactored AssumeRolePolicyDocument to use !Sub with !FindInMap

Benefits of Phase 2

✅ Readability: From 7-level nesting → flat, scannable Mappings
✅ Maintainability: Add new datacenter = add one Mappings entry
✅ Scalability: Works with unlimited datacenters
✅ Consistency: Same account ID source used everywhere
✅ Less error-prone: No deeply nested !If chains to maintain

Testing Phase 2

• ✅ YAML syntax validated (CloudFormation custom tags expected)
• ✅ All 4 templates successfully refactored
• ✅ Mappings match existing account IDs exactly
• ✅ GovCloud special case preserved (AccountIdGovCloud)
• ✅ Backward compatible (same outputs, different structure)

Current Datacenters Included

• datadoghq.com (US) - Account: 464622532012
• datadoghq.eu (EU) - Account: 464622532012
• us3.datadoghq.com (US3) - Account: 464622532012
• us5.datadoghq.com (US5) - Account: 464622532012
• ap1.datadoghq.com (AP1) - Account: 417141415827
• ap2.datadoghq.com (AP2) - Account: 412381753143
• prtest07.datadoghq.com (PRTEST07) - Account: 393946873269
• ddog-gov.com (GovCloud) - Accounts: 392588925713 / 065115117704
• datad0g.com (Internal/Test) - Account: 464622532012

Impact

Phase 1: Zero risk - infrastructure only, no template changes
Phase 2: Low-medium risk - refactored templates produce same results with better structure

Future Phases (Not in This PR)

• Phase 3: Integrate generation into release.sh build scripts
• Phase 4: Complete migration of remaining endpoint lists to Mappings

Benefits After Full Implementation

• Time savings: ~2 hours → ~10 minutes per new datacenter
• Error reduction: 14+ manual file updates → 1 YAML entry
• Code quality: Eliminated deeply nested conditionals
• Maintainability: Single source of truth for all datacenter config
• Scalability: Infrastructure ready for unlimited datacenters

Requirements

• Python 3.6+
• PyYAML (pip install pyyaml)

How to Test Locally

Phase 1 - Generate configs:

# Validate configuration
python scripts/validate_datacenters_config.py

# Generate CloudFormation snippets
python scripts/generate_datacenter_configs.py

# Review generated files
ls -la generated/

Phase 2 - Deploy templates:

# Deploy any of the refactored templates to test AWS account
aws cloudformation deploy \
  --template-file aws_quickstart/main_v2.yaml \
  --stack-name test-datadog-integration \
  --parameter-overrides DatadogSite=datadoghq.com \
  --capabilities CAPABILITY_IAM

# Verify IAM role trust policy uses correct account ID

Migration Path for Adding New Datacenter

BEFORE this PR (manual - error prone):

1. Update 4 files with new IsXXX conditions
2. Update 4 files with nested !If chains (+3 lines per file)
3. Update other 10 files with various configs
4. High risk of missing a file or typo in account ID

AFTER this PR (partially automated):

1. Add entry to datacenters.yaml
2. For now, manually add to Mappings in 4 templates
3. Future: Generation script will auto-update all templates

FUTURE (fully automated - Phase 3+4):

1. Add entry to datacenters.yaml
2. Run generation script
3. Done! ✨

Related

• Implements the plan documented in /Users/kyle.harris/.claude/plans/nifty-twirling-hamster.md
• Follows up on PR Add support for prtest07.datadoghq.com datacenter #267 (added prtest07.datadoghq.com manually)
• Based on analysis of PRs [SEC-21739] add(agentless): add ap2 to agentless CF template #210, add kinesis logs fed endpoint to CF template for aws account level logs #219, [AWSCORE-398] Update templates to support AP2 #221, [aws-ints] Add US3 CSP intake endpoint to streams template #231 (historical datacenter additions)