Add mTLS app-to-app routing support (RFC draft)#535
Draft
Add mTLS app-to-app routing support (RFC draft)#535
Conversation
cf-foundation-community-automation
bot
moved this to Inbox
in Application Runtime Platform Working Group
- Add MtlsDomainConfig struct with domain-specific CA pool and forwarding modes - Add MtlsDomains field and mtlsDomainMap for fast domain lookups - Implement GetMtlsDomainConfig() for exact and wildcard domain matching - Add processMtlsDomains() to validate and build CA pools per domain - Support wildcard domains like *.apps.internal This enables GoRouter to enforce different mTLS policies per domain.
- Add getTLSConfigForClient() callback to dynamically select TLS config - For mTLS domains: require and verify client certs with domain CA pool - For regular domains: use base TLS configuration - Use SNI to determine which domain configuration to apply This allows GoRouter to enforce client certificate validation only on designated mTLS domains while leaving other domains unchanged.
- Add config parameter to NewClientCert() and clientCert struct - Check if request is for an mTLS domain using GetMtlsDomainConfig() - Use domain-specific ForwardedClientCert mode when applicable - Update all call sites and tests to pass config This allows different XFCC header handling policies per domain, supporting both legacy and mTLS-secured routes simultaneously.
- Add router.mtls_domains property to gorouter job spec - Implement ERB template validation and processing logic - Validate required fields (domain, ca_certs) and optional forwarded_client_cert - Support both wildcard (*.apps.internal) and exact domain matching This completes Phase 1a, enabling operators to configure per-domain mTLS policies via BOSH deployment manifests.
- Add AllowedSources struct to subscriber.go with app_guids list - Add AllowedSources field to RegistryMessage for NATS route messages - Add AllowedSourceAppGUIDs to EndpointOpts and Endpoint structs - Update Endpoint.Equal() to compare allowed source GUIDs - Add helper function getAllowedSourceAppGUIDs() This enables route registrations to specify which source apps are authorized to access endpoints on mTLS domains. Work in progress for Phase 1b authorization enforcement.
Ensures the allowed source app GUIDs are properly propagated from EndpointOpts to the Endpoint instance.
- Create handlers/identity.go with XFCC header parsing logic - Add CallerIdentity struct containing app GUID - Add CallerIdentity field to RequestInfo - Extract app GUID from certificate OU field (format: app:<guid>) - Parse X-Forwarded-Client-Cert header with PEM certificate The identity handler extracts the calling application's identity from the client certificate, enabling authorization checks in downstream handlers.
- Create handlers/mtls_authorization.go to enforce authorization - Check if request is on an mTLS domain using config.IsMtlsDomain() - Verify endpoint has AllowedSourceAppGUIDs configured - Match caller identity app GUID against allowed sources list - Return 403 Forbidden if caller not in allowed list - Return 401 Unauthorized if no caller identity present - Skip authorization for non-mTLS domains This handler ensures only explicitly authorized apps can communicate on mTLS-secured domains, completing the authorization enforcement layer for Phase 1b.
- Add NewIdentity() handler after NewClientCert() - Add NewMtlsAuthorization() handler after NewIdentity() - Handlers execute in order: ClientCert -> Identity -> Authorization Handler chain ensures: 1. XFCC header is processed and validated (ClientCert) 2. Caller identity is extracted from certificate (Identity) 3. Authorization is enforced on mTLS domains (MtlsAuthorization) This completes the request processing pipeline for mTLS app-to-app authorization in Phase 1b.
- Add AllowedSourceAppGUIDs field to RouteSchema and Route structs - Add AllowedSources field to NATS Message struct - Implement mapAllowedSources() helper to convert config to message format - Include allowed_sources in route registration messages sent via NATS This enables route-registrar to communicate authorization policies to GoRouter when registering routes on mTLS domains. Route registrations can now specify which source apps are allowed to access the endpoint, completing the authorization data flow from app configuration through to GoRouter enforcement.
Change all example domain names from *.apps.internal to *.apps.mtls.internal to match the RFC specification. The .mtls. segment differentiates mTLS-authenticated routing from direct container-to-container networking. Changes: - Update test files to use *.apps.mtls.internal - Update BOSH job spec example to use *.apps.mtls.internal - All 312 handler tests passing
Implement complete RFC-compliant authorization supporting apps, spaces, organizations, and 'any authenticated app' policies per the app-to-app mTLS routing RFC specification. Changes: - Expand AllowedSources struct with Apps/Spaces/Orgs/Any fields - Update Endpoint and EndpointOpts to use expanded AllowedSources - Enhance CallerIdentity to extract space and org GUIDs from cert OUs - Implement multi-level authorization in mtls_authorization handler - Update route-registrar to support expanded AllowedSources - Add 12 new tests for space/org/any authorization scenarios Authorization logic: - If Any=true, allow any authenticated app (mutually exclusive) - If Any=false, check Apps list, Spaces list, then Orgs list - Allow if caller matches any level (app, space, or org) - Default-deny if no AllowedSources or empty lists All 324 handler tests passing
Add end-to-end integration tests covering: - mTLS domain configuration and client certificate requirements - App-level, space-level, and org-level authorization - Multi-level authorization with OR logic - 'Any authenticated app' authorization (any=true) - Default-deny behavior for mTLS domains - X-Forwarded-Client-Cert header forwarding Test helpers added: - CreateInstanceIdentityCert() generates certificates with app/space/org GUIDs in OUs - registerWithAllowedSources() registers routes with authorization policies Integration tests require: - NATS server (set NATS_SERVER_BINARY env var) - Full GoRouter runtime environment - Can be run with: ginkgo --focus='App-to-App mTLS' integration/ Complements 324 passing unit tests with full end-to-end scenarios.
- Restore locket/lock package required by routing-api - Restore cactus/go-statsd-client package - Remove routing-api from vendor (it's a local submodule, not a vendored dep) - Remove routing-api from go.mod (local packages are resolved directly) This fixes the BOSH release build that was broken by unintended vendor changes in the 'Fix domain names to match RFC specification' commit.
The authorization handler was checking reqInfo.RouteEndpoint which is nil at handler execution time. Changed to use reqInfo.RoutePool which is set by the Lookup handler earlier in the chain. Added AllowedSources() and ApplicationId() methods to EndpointPool to support the authorization check at the pool level.
CAPI stores allowed_sources inside the route options JSON field, while route-registrar uses top-level allowed_sources. Updated GoRouter to check both locations with top-level taking precedence. - Added AllowedSources field to RegistryMessageOpts struct - Added getEffectiveAllowedSources() method to check both locations - Exported MakeEndpoint method for testing - Added unit tests for nested allowed_sources parsing
The clientcert.go sanitize() function produces XFCC headers with raw base64 certificate data (no PEM markers). The identity handler previously only supported Envoy-style Cert="<PEM>" format. Now it supports both formats: 1. GoRouter format: raw base64 without PEM markers 2. Envoy format: Cert="<PEM>" for compatibility This fixes the 'no-caller-identity' error in mTLS authorization.
Rename the AllowedSources struct and related fields/methods to MtlsAllowedSources throughout the codebase for better clarity and consistency with the feature naming. Changes include: - gorouter/route/pool.go: Rename struct and methods - gorouter/mbus/subscriber.go: Rename struct, update JSON tags - gorouter/handlers/mtls_authorization.go: Update variable names - route-registrar/config/config.go: Rename struct, update JSON/YAML tags - route-registrar/messagebus/messagebus.go: Rename helper functions All tests updated to use new naming convention.
Add xfcc_format configuration option for mTLS domains: - 'raw' (default): Full base64-encoded certificate (~1.5KB) - 'envoy': Compact 'Hash=<sha256>;Subject="<DN>"' format (~250 bytes) The Envoy format is ~6x smaller, reducing header overhead for high-volume app-to-app communication. Identity extraction in the identity handler now supports both formats, parsing the Subject DN from either raw certificates or Envoy format to extract app/space/org GUIDs.
Set RouteEndpoint on RequestInfo before returning 401/403 responses so that access logs are emitted to the target app's log stream. This allows operators to see denied requests in 'cf logs <app>' for the backend app, which is essential for debugging authorization issues in mTLS app-to-app communication.
RFC-0027 requires options values to be only strings, numbers, or booleans - not nested objects/arrays. Updated: - RegistryMessageOpts: Use flat fields (mtls_allowed_apps, mtls_allowed_spaces, mtls_allowed_orgs, mtls_allow_any) with comma-separated GUIDs instead of nested MtlsAllowedSources struct - parseCommaSeparatedGUIDs(): New helper to split comma-separated GUID strings into slices - getEffectiveMtlsAllowedSources(): Parse flat options from Options struct while maintaining top-level MtlsAllowedSources precedence for route-registrar compatibility - Tests: Updated to verify flat options parsing
Replace MtlsAllowedSources model with AccessScope/AccessRules selectors, add per-connection TLS state tracking via ConnContext, implement two-layer RFC authorization handler (SNI/Host 421 check + scope/rules enforcement), emit mTLS fields in RTR access logs, and rename router.mtls_domains to router.domains in BOSH config.
rkoster
force-pushed
the
feature/app-to-app-mtls-routing
branch
from
46b4007 to
23b4a0c
Compare
…tion BREAKING CHANGE: Replace pre-selection authorization with post-selection enforcement for strict org/space scope and access rules checking. This implementation follows the Cloud Foundry RFC for App-to-App mTLS Routing (lines 475-517) which mandates post-selection authorization to ensure proper scope boundary enforcement. Changes: - Add composable PostSelectionHandler interface for middleware pipeline - Implement post-selection scope checking (org/space boundaries) - Implement post-selection access rules evaluation (cf:app:, cf:space:, etc.) - Separate pre-selection checks (SNI, route lookup, identity extraction) - Return 403 immediately on authorization failure (non-retriable) - Add MtlsAuthError type with Rule/Reason/HTTPStatus fields - Deprecate old pre-selection authorization handlers - Add :post-selection suffix to MtlsRule values for observability Test Coverage: - 14 new tests for scope authorization - 17 new tests for access rules authorization - 13 new tests for post-selection pipeline infrastructure - 4 new integration tests for shared route scenarios - All 393 tests passing (349 existing + 44 new) RFC Compliance: - Intermittent 403s expected for shared routes across scope boundaries - Error messages include 'caller org X does not match selected backend org Y' - Strict enforcement prevents unauthorized cross-scope access Migration: - handlers/mtls_authorization.go is deprecated with migration notes - route/pool.go EndpointOrgIDs/SpaceIDs methods deprecated - No feature flag - this is a breaking security improvement
Author
Latest Update: RFC-Compliant Post-Selection AuthorizationImplemented breaking change to replace pre-selection authorization with strict post-selection enforcement per RFC lines 475-517. Key Changes (commit cbf0695)Architecture:
Implementation:
Test Coverage:
RFC Compliance✅ Intermittent 403s - Expected for shared routes across scope boundaries (RFC-compliant) Breaking ChangeDeprecated:
Integration Test ResultsAll integration tests compile successfully. Shared route scenarios validate:
Ready for full integration test run and review. |
Rename MtlsAuthError to AuthError to prepare for future authentication methods beyond mTLS, such as SPIFFE JWT tokens. This makes the error type generic and reusable across different authentication mechanisms. Changes: - Rename handlers/mtls_auth_error.go to handlers/auth_error.go - Rename MtlsAuthError struct to AuthError - Rename NewMtlsAuthError to NewAuthError - Rename NewMtlsAuthErrorWithStatus to NewAuthErrorWithStatus - Update error message from "mTLS authorization denied" to "authorization denied" - Update all references in handlers and proxy round tripper - Update test files with new type names No functional changes - this is purely a refactoring for better naming and future extensibility.
Author
Refactoring: AuthError for Future ExtensibilityCommit: 4ff64b9 Renamed Changes
Benefits
No functional changes - pure refactoring for extensibility. |
Extract three shared helper functions from the deprecated mtls_authorization.go handler into a dedicated mtls_helpers.go file: - domainMatches(): checks if hostname matches domain pattern (wildcards) - setRouteEndpointForAccessLog(): sets endpoint for access logs pre-selection - evaluateAccessRules(): evaluates RFC access rules (cf:app:, cf:space:, etc.) These helpers are used by both the deprecated pre-selection handler and the new RFC-compliant post-selection handlers (MtlsScopeAuth, MtlsAccessRulesAuth). This improves code organization by separating reusable utilities from the deprecated handler implementation, making it easier to eventually remove the deprecated code while preserving the shared logic. No functional changes - purely a refactoring for better code structure.
Remove the old pre-selection mtls_authorization handler and its tests. This handler was part of the initial iteration and has been superseded by: - MtlsPreAuth: pre-selection checks (SNI, route lookup, identity) - MtlsScopeAuth: post-selection scope enforcement (RFC-compliant) - MtlsAccessRulesAuth: post-selection access rules evaluation The shared helper functions (domainMatches, setRouteEndpointForAccessLog, evaluateAccessRules) have been preserved in mtls_helpers.go. This cleanup removes ~350 lines of deprecated code and 33 obsolete tests, leaving only the production implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Phase 1 (1a + 1b) of the App-to-App mTLS Routing RFC.
Note: This PR is a draft because the RFC for App-to-App mTLS Routing has not been approved yet.
Phase 1a: mTLS Infrastructure
GetConfigForClientcallbacksanitize_setmode) for mTLS domainsraw(base64 cert) orenvoy(compact hash+subject)router.mtls_domainsPhase 1b: Authorization
mtls_allowed_apps,mtls_allowed_spaces,mtls_allowed_orgs(comma-separated GUIDs),mtls_allow_any(boolean)Testing
Key Files Changed
GoRouter:
src/code.cloudfoundry.org/gorouter/config/config.go- MtlsDomainConfig structsrc/code.cloudfoundry.org/gorouter/router/router.go- GetConfigForClient callbacksrc/code.cloudfoundry.org/gorouter/handlers/clientcert.go- Domain-aware XFCCsrc/code.cloudfoundry.org/gorouter/handlers/identity.go- XFCC parsingsrc/code.cloudfoundry.org/gorouter/handlers/mtls_authorization.go- Authorization handlersrc/code.cloudfoundry.org/gorouter/mbus/subscriber.go- Route message parsingsrc/code.cloudfoundry.org/gorouter/route/pool.go- AllowedSources storagesrc/code.cloudfoundry.org/gorouter/proxy/proxy.go- Handler wiringRoute Registrar:
src/code.cloudfoundry.org/route-registrar/config/config.go- AllowedSources in Optionssrc/code.cloudfoundry.org/route-registrar/messagebus/messagebus.go- NATS message formatBOSH:
jobs/gorouter/spec- router.mtls_domains propertyjobs/gorouter/templates/gorouter.yml.erb- Template configurationConfiguration Example
Related PRs