diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 3b136fc49..a34202ace 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -452,6 +452,8 @@ **** xref:networking:configure-privatelink-in-cloud-ui.adoc[Configure PrivateLink in the Cloud Console] **** xref:networking:aws-privatelink.adoc[Configure PrivateLink with the Cloud API] **** xref:networking:byoc/aws/transit-gateway.adoc[Add a Transit Gateway] +**** xref:networking:byoc/aws/nat-free-egress.adoc[Configure Centralized Egress] +**** xref:networking:byoc/aws/aws-hub-egress.adoc[Create an AWS Hub for Centralized Egress] *** xref:networking:byoc/azure/index.adoc[Azure] **** xref:networking:azure-private-link-in-ui.adoc[] **** xref:networking:azure-private-link.adoc[] diff --git a/modules/get-started/pages/cluster-types/byoc/aws/create-byoc-cluster-aws.adoc b/modules/get-started/pages/cluster-types/byoc/aws/create-byoc-cluster-aws.adoc index 38911c0b6..adf9ab19e 100644 --- a/modules/get-started/pages/cluster-types/byoc/aws/create-byoc-cluster-aws.adoc +++ b/modules/get-started/pages/cluster-types/byoc/aws/create-byoc-cluster-aws.adoc @@ -43,6 +43,11 @@ Optionally, click *Advanced settings* to specify up to five key-value custom tag ** Clusters with private networking include a setting for API Gateway network access. Public access exposes endpoints for Redpanda Console, the Data Plane API, and the MCP Server API, but they remain protected by your authentication and authorization controls. Private access restricts endpoint access to your VPC only. + NOTE: After the cluster is created, you can change the API Gateway access on the Dataplane settings page. If you change from public to private access, users without VPN access to the Redpanda VPC will lose access to these services. ++ +[TIP] +==== +To route all cluster egress through your own AWS Transit Gateway and hub VPC instead of a per-VPC NAT Gateway, set the *Transit Gateway ID* field on this page. The field is only available on clusters with a private connection type, and is only visible if centralized egress is enabled for your organization. This option is in beta. See xref:networking:byoc/aws/nat-free-egress.adoc[]. +==== . Click *Next*. . On the Deploy page, follow the steps to log in to Redpanda Cloud and deploy the agent. + diff --git a/modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc b/modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc index ebb3bc88e..32df78da8 100644 --- a/modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc +++ b/modules/get-started/pages/cluster-types/byoc/aws/vpc-byo-aws.adoc @@ -111,7 +111,7 @@ module "redpanda_byovpc" { [NOTE] ==== -* To send telemetry back to the Redpanda control plane, the cluster needs outbound internet access. You can provide this through at least one public subnet, or through network peering or a transit gateway to another VPC that routes traffic through a public subnet. The example configuration includes multiple public subnets to allow for future scaling. +* To send telemetry back to the Redpanda control plane, the cluster needs outbound internet access. You can provide this through at least one public subnet, or through network peering or a transit gateway to another VPC that routes traffic through a public subnet. The example configuration includes multiple public subnets to allow for future scaling. Standard BYOC clusters can also route egress through a customer-owned hub VPC and Transit Gateway, eliminating the per-VPC NAT Gateway entirely. See xref:networking:byoc/aws/nat-free-egress.adoc[]. * The example creates an Internet Gateway and an associated Route Table rule that routes traffic into the VPC, which allows the Redpanda control plane to access the cluster. To disable creation of the Internet Gateway, either remove the configuration and value for `create_internet_gateway` or set `"create_internet_gateway": false`. * When using a pre-existing VPC, at least one public subnet must already exist in that VPC. Setting `public_subnet_cidrs = []` only prevents the module from creating new ones. ==== diff --git a/modules/get-started/pages/whats-new-cloud.adoc b/modules/get-started/pages/whats-new-cloud.adoc index c6197e14f..689ef9f75 100644 --- a/modules/get-started/pages/whats-new-cloud.adoc +++ b/modules/get-started/pages/whats-new-cloud.adoc @@ -8,6 +8,10 @@ This page lists new features added to Redpanda Cloud. == May 2026 +=== Centralized egress for BYOC on AWS (beta) + +You can route all BYOC cluster egress through your own AWS Transit Gateway and hub VPC instead of a per-VPC NAT Gateway, so outbound traffic exits through your centralized inspection point. This is useful for regulated environments that prohibit per-VPC NAT Gateways and for consolidating egress behind a single, predictable public IP for outbound allowlisting. Centralized egress is in beta and is enabled per organization. Contact your account team for access. See xref:networking:byoc/aws/nat-free-egress.adoc[Configure Centralized Egress with AWS Transit Gateway]. + === Schema Registry Authorization enabled by default Schema Registry Authorization is now enabled automatically on all new BYOC and Dedicated clusters. The xref:reference:properties/cluster-properties.adoc#schema_registry_enable_authorization[`schema_registry_enable_authorization`] cluster property is set to `true` at provisioning, and the predefined Admin, Writer, and Reader roles include Schema Registry permissions for the `subject` and `registry` ACL resource types. You can use ACLs and RBAC roles to grant fine-grained access to schemas and subjects without any additional setup. See xref:manage:schema-reg/schema-reg-authorization.adoc[Schema Registry Authorization] and xref:security:authorization/rbac/rbac.adoc#predefined-roles[Predefined roles]. diff --git a/modules/manage/partials/controlplane-api.adoc b/modules/manage/partials/controlplane-api.adoc index 200a11924..b0cab27b5 100644 --- a/modules/manage/partials/controlplane-api.adoc +++ b/modules/manage/partials/controlplane-api.adoc @@ -125,11 +125,40 @@ curl -d \ "region": "us-west1" } }' -H "Content-Type: application/json" \ --H "Authorization: Bearer " -X POST https://api.redpanda.com/v1/networks +-H "Authorization: Bearer " -X POST https://api.redpanda.com/v1/networks +---- +// The AWS BYOC network example adds `egress_spec.aws.transit_gateway_id` +// to route all cluster egress through a customer-owned Transit Gateway. +// It is gated behind `:show-preview-api:` while the field is in preview. +// To enable, set the attribute in the playbook or in this page header. +ifdef::show-preview-api[] + +To route all cluster egress through your own AWS Transit Gateway and hub VPC instead of a per-VPC NAT Gateway, set `egress_spec.aws.transit_gateway_id` on an AWS BYOC network. Centralized egress is in beta. The Transit Gateway ID is immutable after the network is created. Before calling this endpoint, provision the hub VPC and Transit Gateway and share the Transit Gateway with the Redpanda cluster account. See xref:networking:byoc/aws/aws-hub-egress.adoc[Create an AWS Hub for Centralized Egress] and xref:networking:byoc/aws/nat-free-egress.adoc[Configure Centralized Egress with AWS Transit Gateway]. + +[,bash] ---- +curl -d \ +'{ + "network": { + "cidr_block": "10.10.0.0/20", + "cloud_provider": "CLOUD_PROVIDER_AWS", + "cluster_type": "TYPE_BYOC", + "name": "", + "resource_group_id": "", + "region": "us-east-2", + "egress_spec": { + "aws": { + "transit_gateway_id": "tgw-0b629c5b4fb6e364b" + } + } + } +}' -H "Content-Type: application/json" \ +-H "Authorization: Bearer " -X POST https://api.redpanda.com/v1/networks +---- +endif::[] endif::[] -This endpoint returns a <>. +This endpoint returns a <>. === Create a new cluster diff --git a/modules/networking/pages/byoc/aws/aws-hub-egress.adoc b/modules/networking/pages/byoc/aws/aws-hub-egress.adoc new file mode 100644 index 000000000..f5edf88bb --- /dev/null +++ b/modules/networking/pages/byoc/aws/aws-hub-egress.adoc @@ -0,0 +1,573 @@ += Create an AWS Hub for Centralized Egress +:description: Provision a hub VPC, Transit Gateway, and AWS RAM share so one or more BYOC clusters can route their internet egress through a single, predictable public IP. +:page-beta: true +:page-topic-type: how-to +:personas: platform_admin +:learning-objective-1: Provision a hub VPC with a NAT Gateway and Transit Gateway on AWS +:learning-objective-2: Share the Transit Gateway with the Redpanda cluster account using AWS RAM +:learning-objective-3: Collect the Transit Gateway ID and NAT Gateway public IP needed for BYOC cluster configuration + +Use this guide to provision the customer-side infrastructure required for centralized egress on a Redpanda Bring Your Own Cloud (BYOC) cluster on AWS. After completing the procedure, you have a hub VPC with a NAT Gateway, a Transit Gateway shared with the Redpanda cluster account, and the IDs needed to configure your BYOC network. You can then follow xref:networking:byoc/aws/nat-free-egress.adoc[Configure Centralized Egress with AWS Transit Gateway] to enable centralized egress at cluster creation. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +== What this sets up + +A hub VPC with a single NAT Gateway provides centralized internet egress for one or more spoke VPCs. All outbound traffic from spokes is routed through the hub and exits from one public IP, which you can use for outbound IP allowlisting on external services. + +---- +Spoke VPC --> Transit Gateway --> Hub Private Subnet --> NAT Gateway --> Internet Gateway --> Internet + ^ + | + Return traffic via Transit Gateway ---+ +---- + +Traffic path: + +. The spoke private subnet sends `0.0.0.0/0` to the Transit Gateway. +. The Transit Gateway default route forwards `0.0.0.0/0` to the hub VPC attachment. +. The hub private subnet routes `0.0.0.0/0` to the NAT Gateway. +. The NAT Gateway exits through the Internet Gateway. +. Return traffic enters through the Internet Gateway. The hub public route table sends it back to the Transit Gateway, which delivers it to the correct spoke. + +[#automated-alternative] +== Automated alternative + +If you prefer to provision this infrastructure with Terraform, the configuration used to produce the setup in this guide is published in the Redpanda Cloud Examples repository: https://github.com/redpanda-data/cloud-examples/tree/main/aws-hub-egress[redpanda-data/cloud-examples — aws-hub-egress^] + +The rest of this guide covers the equivalent steps using the AWS Console or AWS CLI. + +== Prerequisites + +* AWS CLI configured with credentials for the hub account. +* Permissions on the hub account for VPC, EC2, Transit Gateway, AWS RAM, and IAM operations. +* The Redpanda spoke VPC CIDR must not overlap your hub VPC CIDR. +* If the Redpanda cluster runs in a different AWS account than your hub, note the Redpanda cluster account ID for the AWS RAM share. + +[WARNING] +==== +Hub and spoke VPC CIDRs must not overlap. AWS allows Transit Gateway attachments with overlapping CIDRs, but the hub VPC's local route always wins over the spoke CIDR return route, which silently blackholes reply traffic. Plan your CIDRs before you start. For background, see xref:networking:cidr-ranges.adoc#what-are-cidrs[What are CIDRs?] +==== + +== Default values + +The procedure uses the values in the following table. Replace any value that does not match your environment. + +[cols="2,2,3"] +|=== +| Parameter | Default | Notes + +| Hub VPC CIDR | `100.64.0.0/16` | CGNAT range, unlikely to conflict with typical workload CIDRs +| Public subnet 1 | `100.64.0.0/24` | AZ index 0 +| Public subnet 2 | `100.64.1.0/24` | AZ index 1 +| Private subnet 1 | `100.64.2.0/24` | AZ index 0 +| Private subnet 2 | `100.64.3.0/24` | AZ index 1 +| Spoke CIDR | `10.0.0.0/16` | Replace with your actual Redpanda spoke VPC CIDR +| Region | `us-east-2` | Replace with your target region +|=== + +== Create the hub VPC + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Your VPCs* > *Create VPC*. +. Set the name to `hub-vpc`. +. Set the IPv4 CIDR to `100.64.0.0/16`. +. Enable DNS resolution and DNS hostnames. +. Click *Create VPC*. + +CLI:: ++ +[,bash] +---- +VPC_ID=$(aws ec2 create-vpc \ + --cidr-block 100.64.0.0/16 \ + --region us-east-2 \ + --query 'Vpc.VpcId' --output text) + +aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-support +aws ec2 modify-vpc-attribute --vpc-id $VPC_ID --enable-dns-hostnames + +aws ec2 create-tags --resources $VPC_ID \ + --tags Key=Name,Value=hub-vpc \ + --region us-east-2 + +echo "VPC ID: $VPC_ID" +---- +====== + +== Get availability zones + +The public and private subnets each require two availability zones. + +[,bash] +---- +AZ1=$(aws ec2 describe-availability-zones \ + --region us-east-2 \ + --filters Name=opt-in-status,Values=opt-in-not-required \ + --query 'AvailabilityZones[0].ZoneName' --output text) + +AZ2=$(aws ec2 describe-availability-zones \ + --region us-east-2 \ + --filters Name=opt-in-status,Values=opt-in-not-required \ + --query 'AvailabilityZones[1].ZoneName' --output text) + +echo "AZ1: $AZ1, AZ2: $AZ2" +---- + +== Create public subnets + +Leave auto-assign public IPv4 disabled. The hub VPC never assigns public IPs to instances directly. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Subnets* > *Create Subnet*. +. Select the `hub-vpc` VPC. +. Create two public subnets: +.. Name `hub-public-0`, in the first AZ, with CIDR `100.64.0.0/24`. +.. Name `hub-public-1`, in the second AZ, with CIDR `100.64.1.0/24`. +. Leave *Auto-assign public IPv4* disabled. + +CLI:: ++ +[,bash] +---- +PUB_SUBNET_0=$(aws ec2 create-subnet \ + --vpc-id $VPC_ID \ + --cidr-block 100.64.0.0/24 \ + --availability-zone $AZ1 \ + --region us-east-2 \ + --query 'Subnet.SubnetId' --output text) + +PUB_SUBNET_1=$(aws ec2 create-subnet \ + --vpc-id $VPC_ID \ + --cidr-block 100.64.1.0/24 \ + --availability-zone $AZ2 \ + --region us-east-2 \ + --query 'Subnet.SubnetId' --output text) + +aws ec2 create-tags --resources $PUB_SUBNET_0 \ + --tags Key=Name,Value=hub-public-0 --region us-east-2 +aws ec2 create-tags --resources $PUB_SUBNET_1 \ + --tags Key=Name,Value=hub-public-1 --region us-east-2 + +echo "Public subnets: $PUB_SUBNET_0, $PUB_SUBNET_1" +---- +====== + +== Create private subnets + +The private subnets are where the Transit Gateway lands traffic delivered from the Redpanda spoke. They have no Internet Gateway route, so workloads in them never get public IPs, and outbound traffic exits the hub through the NAT Gateway. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Subnets* > *Create Subnet*. +. Select the `hub-vpc` VPC. +. Create two private subnets: +.. Name `hub-private-0`, in the first AZ, with CIDR `100.64.2.0/24`. +.. Name `hub-private-1`, in the second AZ, with CIDR `100.64.3.0/24`. + +CLI:: ++ +[,bash] +---- +PRIV_SUBNET_0=$(aws ec2 create-subnet \ + --vpc-id $VPC_ID \ + --cidr-block 100.64.2.0/24 \ + --availability-zone $AZ1 \ + --region us-east-2 \ + --query 'Subnet.SubnetId' --output text) + +PRIV_SUBNET_1=$(aws ec2 create-subnet \ + --vpc-id $VPC_ID \ + --cidr-block 100.64.3.0/24 \ + --availability-zone $AZ2 \ + --region us-east-2 \ + --query 'Subnet.SubnetId' --output text) + +aws ec2 create-tags --resources $PRIV_SUBNET_0 \ + --tags Key=Name,Value=hub-private-0 --region us-east-2 +aws ec2 create-tags --resources $PRIV_SUBNET_1 \ + --tags Key=Name,Value=hub-private-1 --region us-east-2 + +echo "Private subnets: $PRIV_SUBNET_0, $PRIV_SUBNET_1" +---- +====== + +== Create the Internet Gateway + +The Internet Gateway gives the hub VPC its outbound path to the internet. The NAT Gateway depends on it for source-NAT to a public IP, and reply traffic from the internet enters the hub through it. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Internet Gateways* > *Create internet gateway*. +. Set the name to `hub-igw`. +. Click *Create*. +. Choose *Actions* > *Attach to VPC* and select `hub-vpc`. + +CLI:: ++ +[,bash] +---- +IGW_ID=$(aws ec2 create-internet-gateway \ + --region us-east-2 \ + --query 'InternetGateway.InternetGatewayId' --output text) + +aws ec2 attach-internet-gateway \ + --internet-gateway-id $IGW_ID \ + --vpc-id $VPC_ID \ + --region us-east-2 + +aws ec2 create-tags --resources $IGW_ID \ + --tags Key=Name,Value=hub-igw --region us-east-2 + +echo "IGW ID: $IGW_ID" +---- +====== + +== Create the NAT Gateway + +The NAT Gateway goes in the first public subnet only. It requires an Elastic IP. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Elastic IPs* > *Allocate Elastic IP address* > *Allocate*. +. Choose *NAT Gateways* > *Create NAT gateway*. +. Set the name to `hub-nat`. +. Select subnet `hub-public-0`. +. Set *Connectivity type* to *Public*. +. Select the Elastic IP you just allocated. +. Click *Create NAT Gateway*. The NAT Gateway takes a few minutes to become available. + +CLI:: ++ +[,bash] +---- +EIP_ALLOC=$(aws ec2 allocate-address \ + --domain vpc \ + --region us-east-2 \ + --query 'AllocationId' --output text) + +NAT_GW_ID=$(aws ec2 create-nat-gateway \ + --subnet-id $PUB_SUBNET_0 \ + --allocation-id $EIP_ALLOC \ + --region us-east-2 \ + --query 'NatGateway.NatGatewayId' --output text) + +aws ec2 create-tags --resources $NAT_GW_ID \ + --tags Key=Name,Value=hub-nat --region us-east-2 + +aws ec2 wait nat-gateway-available \ + --nat-gateway-ids $NAT_GW_ID \ + --region us-east-2 + +echo "NAT Gateway ID: $NAT_GW_ID" +---- +====== + +== Create the Transit Gateway + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Transit Gateways* > *Create transit gateway*. +. Set the name to `hub-tgw`. +. Enable *Default route table association*, *Default route table propagation*, and *Auto accept shared attachments*. +. Click *Create transit gateway*. The Transit Gateway takes a few minutes to become available. + +CLI:: ++ +[,bash] +---- +TGW_ID=$(aws ec2 create-transit-gateway \ + --description "Hub egress transit gateway" \ + --options "DefaultRouteTableAssociation=enable,DefaultRouteTablePropagation=enable,AutoAcceptSharedAttachments=enable" \ + --region us-east-2 \ + --query 'TransitGateway.TransitGatewayId' --output text) + +aws ec2 create-tags --resources $TGW_ID \ + --tags Key=Name,Value=hub-tgw --region us-east-2 + +aws ec2 wait transit-gateway-available \ + --transit-gateway-ids $TGW_ID \ + --region us-east-2 + +HUB_ACCOUNT=$(aws sts get-caller-identity --query Account --output text) +TGW_ARN="arn:aws:ec2:us-east-2:${HUB_ACCOUNT}:transit-gateway/$TGW_ID" + +echo "Transit Gateway ID: $TGW_ID" +echo "Transit Gateway ARN: $TGW_ARN" +---- +====== + +== Attach the hub VPC to the Transit Gateway + +The hub VPC attaches using its private subnets. This is where the Transit Gateway delivers inbound traffic into the hub. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Transit Gateway Attachments* > *Create transit gateway attachment*. +. Select the `hub-tgw` Transit Gateway. +. Set *Attachment type* to *VPC*. +. Select the `hub-vpc` VPC. +. Select both private subnets (`hub-private-0` and `hub-private-1`). +. Enable *Associate with the default association route table* and *Propagate to the default propagation route table*. +. Click *Create*. The attachment takes a few minutes to become available. + +CLI:: ++ +[,bash] +---- +TGW_ATTACH_ID=$(aws ec2 create-transit-gateway-vpc-attachment \ + --transit-gateway-id $TGW_ID \ + --vpc-id $VPC_ID \ + --subnet-ids $PRIV_SUBNET_0 $PRIV_SUBNET_1 \ + --options "ApplianceModeSupport=disable,DnsSupport=enable,Ipv6Support=disable" \ + --region us-east-2 \ + --query 'TransitGatewayVpcAttachment.TransitGatewayAttachmentId' --output text) + +aws ec2 create-tags --resources $TGW_ATTACH_ID \ + --tags Key=Name,Value=hub-tgw-attachment --region us-east-2 + +aws ec2 wait transit-gateway-attachment-available \ + --transit-gateway-attachment-ids $TGW_ATTACH_ID \ + --region us-east-2 + +echo "Transit Gateway Attachment ID: $TGW_ATTACH_ID" +---- +====== + +== Add a default route to the Transit Gateway route table + +This static route forwards all `0.0.0.0/0` traffic to the hub VPC attachment, so spoke traffic reaches the NAT Gateway. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Transit Gateway Route Tables*. +. Select the default route table for `hub-tgw`. +. Choose *Routes* > *Create static route*. +. Set the CIDR to `0.0.0.0/0`. +. Select the hub VPC attachment. +. Click *Create static route*. + +CLI:: ++ +[,bash] +---- +TGW_RTB_ID=$(aws ec2 describe-transit-gateway-route-tables \ + --filters "Name=transit-gateway-id,Values=$TGW_ID" \ + "Name=default-association-route-table,Values=true" \ + --region us-east-2 \ + --query 'TransitGatewayRouteTables[0].TransitGatewayRouteTableId' --output text) + +aws ec2 create-transit-gateway-route \ + --destination-cidr-block 0.0.0.0/0 \ + --transit-gateway-route-table-id $TGW_RTB_ID \ + --transit-gateway-attachment-id $TGW_ATTACH_ID \ + --region us-east-2 + +echo "Default route added to Transit Gateway route table: $TGW_RTB_ID" +---- +====== + +== Create the route tables + +=== Public route table + +The public subnet route table needs two types of routes: + +* `0.0.0.0/0` to the Internet Gateway -- allows the NAT Gateway to reach the internet. +* One entry per spoke CIDR to the Transit Gateway -- sends reply packets back to the correct spoke. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Route Tables* > *Create route table*. +. Name it `hub-public-rt` and select the `hub-vpc` VPC. +. Click *Create*. +. Choose *Routes* > *Edit routes* and add the following: +.. Destination `0.0.0.0/0`, target type *Internet Gateway*, value `hub-igw`. +.. Destination `10.0.0.0/16` (your spoke CIDR), target type *Transit Gateway*, value `hub-tgw`. Repeat this entry for each additional spoke. +. Choose *Subnet Associations* > *Edit subnet associations* and select both public subnets. + +CLI:: ++ +[,bash] +---- +PUB_RTB_ID=$(aws ec2 create-route-table \ + --vpc-id $VPC_ID \ + --region us-east-2 \ + --query 'RouteTable.RouteTableId' --output text) + +aws ec2 create-tags --resources $PUB_RTB_ID \ + --tags Key=Name,Value=hub-public-rt --region us-east-2 + +aws ec2 create-route \ + --route-table-id $PUB_RTB_ID \ + --destination-cidr-block 0.0.0.0/0 \ + --gateway-id $IGW_ID \ + --region us-east-2 + +# Add one route per spoke CIDR. Repeat this block for each additional spoke. +aws ec2 create-route \ + --route-table-id $PUB_RTB_ID \ + --destination-cidr-block 10.0.0.0/16 \ + --transit-gateway-id $TGW_ID \ + --region us-east-2 + +aws ec2 associate-route-table \ + --route-table-id $PUB_RTB_ID \ + --subnet-id $PUB_SUBNET_0 \ + --region us-east-2 + +aws ec2 associate-route-table \ + --route-table-id $PUB_RTB_ID \ + --subnet-id $PUB_SUBNET_1 \ + --region us-east-2 +---- +====== + +=== Private route table + +Private subnets only need a default route to the NAT Gateway. Traffic arrives from the Transit Gateway and exits using NAT. + +[tabs] +====== +Console:: ++ +. Open the VPC console and choose *Route Tables* > *Create route table*. +. Name it `hub-private-rt` and select the `hub-vpc` VPC. +. Click *Create*. +. Choose *Routes* > *Edit routes* and add `0.0.0.0/0` with target type *NAT Gateway* and value `hub-nat`. +. Choose *Subnet Associations* > *Edit subnet associations* and select both private subnets. + +CLI:: ++ +[,bash] +---- +PRIV_RTB_ID=$(aws ec2 create-route-table \ + --vpc-id $VPC_ID \ + --region us-east-2 \ + --query 'RouteTable.RouteTableId' --output text) + +aws ec2 create-tags --resources $PRIV_RTB_ID \ + --tags Key=Name,Value=hub-private-rt --region us-east-2 + +aws ec2 create-route \ + --route-table-id $PRIV_RTB_ID \ + --destination-cidr-block 0.0.0.0/0 \ + --nat-gateway-id $NAT_GW_ID \ + --region us-east-2 + +aws ec2 associate-route-table \ + --route-table-id $PRIV_RTB_ID \ + --subnet-id $PRIV_SUBNET_0 \ + --region us-east-2 + +aws ec2 associate-route-table \ + --route-table-id $PRIV_RTB_ID \ + --subnet-id $PRIV_SUBNET_1 \ + --region us-east-2 +---- +====== + +== Share the Transit Gateway with the Redpanda account + +Skip this section if the Redpanda BYOC cluster runs in the same AWS account as your hub. + +Use AWS Resource Access Manager (RAM) to share the Transit Gateway with the Redpanda cluster account. + +[tabs] +====== +Console:: ++ +. In the hub account, open the AWS RAM console and choose *Resource shares* > *Create resource share*. +. Set the name to `hub-tgw-share`. +. Under *Resources*, select resource type *Transit Gateways* and pick `hub-tgw`. +. Under *Principals*, enter the Redpanda cluster AWS account ID. +. Click *Create resource share*. +. If the accounts are not in the same AWS Organization, switch to the BYOC AWS account, open AWS RAM, and accept the pending invitation under *Resource share invitations*. Accept the invitation before you create the Redpanda BYOC cluster. Within an AWS Organization, the share is auto-accepted. + +CLI:: ++ +In the hub account: ++ +[,bash] +---- +REDPANDA_ACCOUNT_ID="" + +RAM_SHARE_ARN=$(aws ram create-resource-share \ + --name hub-tgw-share \ + --resource-arns "$TGW_ARN" \ + --principals "$REDPANDA_ACCOUNT_ID" \ + --region us-east-2 \ + --query 'resourceShare.resourceShareArn' --output text) + +echo "RAM Share ARN: $RAM_SHARE_ARN" +---- ++ +If the accounts are not in the same AWS Organization, the spoke side must accept the invitation. Within an AWS Organization, shares are auto-accepted. +====== + +== Collect the values to provide to Redpanda + +Record these values. Provide the Transit Gateway ID when you configure centralized egress on your BYOC network. See xref:networking:byoc/aws/nat-free-egress.adoc[Configure Centralized Egress with AWS Transit Gateway]. + +[,bash] +---- +NAT_PUBLIC_IP=$(aws ec2 describe-nat-gateways \ + --nat-gateway-ids $NAT_GW_ID \ + --region us-east-2 \ + --query 'NatGateways[0].NatGatewayAddresses[0].PublicIp' --output text) + +echo "hub_vpc_id = $VPC_ID" +echo "transit_gateway_id = $TGW_ID" +echo "transit_gateway_arn = $TGW_ARN" +echo "nat_gateway_public_ip = $NAT_PUBLIC_IP" +echo "ram_resource_share_arn = $RAM_SHARE_ARN" +---- + +All outbound internet traffic from every BYOC spoke that attaches to this Transit Gateway exits from the same NAT Gateway public IP. Use that public IP for outbound IP allowlisting on external services. + +== Plan for high availability + +The procedure in this guide deploys a single NAT Gateway in `hub-public-0`, which is not AZ-resilient. For production deployments, run one NAT Gateway per availability zone and use per-AZ private route tables that direct local traffic to the AZ-local NAT Gateway. The <> also deploys a single NAT Gateway. Extend it with per-AZ NAT Gateways if you need HA egress. + +== Troubleshooting + +[cols="2,2"] +|=== +| Symptom | Likely cause + +| Spoke cannot reach the internet | Missing `0.0.0.0/0` -> Transit Gateway route in the spoke private route table. The Redpanda agent creates this route when the network is configured for centralized egress. +| Outbound traffic leaves the hub NAT Gateway but no replies arrive at the spoke | Missing spoke CIDR route in the hub public route table. +| Transit Gateway attachment is stuck in `pending` | The BYOC AWS account has not accepted the AWS RAM share invitation. In that account, open AWS RAM and accept the pending invitation. Shares are auto-accepted only within an AWS Organization. +| CIDR conflict error on attachment | The spoke CIDR overlaps the hub CIDR `100.64.0.0/16`. Choose non-overlapping CIDRs. +| Outbound traffic is intermittently dropped | Your hub firewall is not allowlisting the third-party endpoints required by Redpanda. See xref:networking:byoc/aws/nat-free-egress.adoc#internet-endpoints[Internet endpoints required from your hub]. +| All traffic appears to come from one public IP | This is the expected behavior of centralized egress. Use that IP for outbound allowlisting. +|=== + +== Next steps + +* xref:networking:byoc/aws/nat-free-egress.adoc[Configure Centralized Egress with AWS Transit Gateway] +* xref:get-started:cluster-types/byoc/aws/create-byoc-cluster-aws.adoc[Create a BYOC Cluster on AWS] diff --git a/modules/networking/pages/byoc/aws/nat-free-egress.adoc b/modules/networking/pages/byoc/aws/nat-free-egress.adoc new file mode 100644 index 000000000..9dd8b4d22 --- /dev/null +++ b/modules/networking/pages/byoc/aws/nat-free-egress.adoc @@ -0,0 +1,232 @@ += Configure Centralized Egress with AWS Transit Gateway +:description: Route all BYOC cluster egress through your own AWS Transit Gateway and hub VPC instead of a per-VPC NAT Gateway. +:page-beta: true +:page-topic-type: how-to +:personas: platform_admin +:learning-objective-1: Enable centralized egress on a new BYOC cluster using the Redpanda Cloud UI +:learning-objective-2: Identify the shared responsibility boundaries between Redpanda and your hub account +:learning-objective-3: Troubleshoot egress failures caused by misconfigured hub routes or firewall rules + +[IMPORTANT] +==== +This feature is in beta and is available only on AWS BYOC clusters with private networking. Availability is controlled per organization. Contact your account team to request access to centralized egress. +==== + +Centralized egress lets your BYOC cluster send all internet-bound traffic through your own AWS Transit Gateway and hub VPC instead of a Redpanda-managed NAT Gateway in the Redpanda VPC. When centralized egress is enabled, the Redpanda spoke VPC attaches to your Transit Gateway, all outbound traffic exits through the NAT Gateway or firewall in your hub, and no per-VPC NAT Gateway is created in the spoke. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +== When to use centralized egress + +Use centralized egress when: + +* Your security policy prohibits per-VPC NAT Gateways and requires all internet-bound traffic to traverse a centralized inspection point such as an AWS Network Firewall, NVA, or third-party firewall appliance. +* You already operate a hub-and-spoke network in AWS and want Redpanda to fit into that topology. +* You want to consolidate egress through a single, predictable public IP for outbound allowlisting on external services. +* You want to reduce NAT Gateway data-processing fees on high-throughput Redpanda workloads. + +If none of these apply, the default NAT Gateway egress used by standard BYOC clusters is simpler and requires no extra customer-managed infrastructure. + +== How it works + +Without centralized egress, the Redpanda agent creates a NAT Gateway and Internet Gateway inside the Redpanda VPC. With centralized egress enabled, the agent attaches the Redpanda VPC to your existing Transit Gateway instead, routes all `0.0.0.0/0` traffic through it, and provisions Interface VPC Endpoints in the Redpanda VPC so critical AWS API calls (ECR, STS, CloudWatch Logs) do not need to traverse your hub. + +The following diagram shows the connectivity model: + +---- ++----------------------------------+ +-----------+ +----------------------------------+ +| Redpanda VPC | | Transit | | Customer Hub VPC | +| (Redpanda account) | | Gateway | | (Customer account) | +| | | | | | +| Transit Gateway VPC Attachment +-------->+ Shared +-------->+ Transit Gateway VPC Attachment | +| 0.0.0.0/0 -> Transit Gateway | | via AWS | | 0.0.0.0/0 -> NAT GW | +| VPC Endpoints | | RAM | | NAT Gateway -> IGW | +| EKS Worker Nodes | | | | Internet Gateway | +| No NAT / No IGW | | | | | ++----------------------------------+ +-----------+ +-----------+----------------------+ + | + v + Internet + (Cloudsmith, public.ecr.aws, + NTP, Helm) +---- + +=== Outbound traffic flow + +. An EKS node in a Redpanda private subnet sends a packet destined for the internet. +. The private route table forwards `0.0.0.0/0` to the Transit Gateway VPC attachment in the Redpanda VPC. +. The Transit Gateway's default route table forwards `0.0.0.0/0` to your hub VPC attachment (static route). +. The hub private route table forwards `0.0.0.0/0` to your NAT Gateway. +. The NAT Gateway forwards traffic to your Internet Gateway and out to the internet. + +=== Return traffic flow + +. The internet response arrives at the Internet Gateway in your hub VPC. +. The hub public route table forwards the Redpanda spoke CIDR to the Transit Gateway. +. The Transit Gateway forwards traffic back to the Redpanda spoke attachment through a propagated route. +. The packet arrives at the originating EKS node. + +The Redpanda spoke CIDR route on the hub public route table is your responsibility. It is added when you provision your hub. See xref:networking:byoc/aws/aws-hub-egress.adoc[Create an AWS Hub for Centralized Egress] for the full procedure. + +[#prerequisites] +== Prerequisites + +Before you configure centralized egress, confirm all of the following: + +* Centralized egress is enabled on your Redpanda Cloud organization. Contact your account team to request access. +* Your BYOC cluster uses private networking (connection type *Private*). Centralized egress is not available on clusters with a public connection type. +* You have an existing AWS Transit Gateway in the same region as the Redpanda BYOC cluster, and you know whether it auto-accepts cross-account attachments. Check the setting on your Transit Gateway before you create the cluster: ++ +[,bash] +---- +aws ec2 describe-transit-gateways \ + --transit-gateway-ids \ + --query 'TransitGateways[0].Options.AutoAcceptSharedAttachments' +---- ++ +** If the value is `enable`, the Redpanda spoke attachment is accepted automatically when the BYOC cluster provisions. No action is required from you during cluster creation. +** If the value is `disable`, you must manually accept the spoke attachment during cluster provisioning. In the AWS account that owns the Transit Gateway, open *VPC > Transit Gateway Attachments*, select the pending attachment from the Redpanda BYOC account, and choose *Actions > Accept transit gateway attachment*. Cluster provisioning pauses until you accept the attachment. ++ +[NOTE] +==== +Enabling `AutoAcceptSharedAttachments` causes the Transit Gateway to accept attachments from any AWS account it is shared with through AWS Resource Access Manager. Share the Transit Gateway only with accounts you trust to attach VPCs. +==== +* Your hub VPC has a NAT Gateway, AWS Network Firewall, or other NVA providing internet egress. See xref:networking:byoc/aws/aws-hub-egress.adoc[Create an AWS Hub for Centralized Egress] if you need to provision a hub. +* If the hub Transit Gateway is in a different AWS account than the account where you deploy your BYOC cluster, you have created an AWS Resource Access Manager (RAM) share in the hub account and accepted the invitation in the BYOC account before creating the cluster. Within an AWS Organization, the share is auto-accepted and no manual acceptance step is needed. +* Your hub VPC CIDR and the planned Redpanda spoke VPC CIDR do not overlap. ++ +[WARNING] +==== +Terraform and AWS do not enforce CIDR uniqueness across Transit Gateway attachments, but overlapping CIDRs cause silent return-route failures. The hub VPC's local route always wins over a Redpanda-spoke-CIDR route on the public route table, blackholing reply traffic. + +Use a non-overlapping range for your hub. The `100.64.0.0/16` CGNAT range is a safe default because it rarely conflicts with typical RFC 1918 workload CIDRs. +==== +* The Transit Gateway ID, in the form `tgw-` followed by at least 8 hexadecimal characters (for example, `tgw-0b629c5b4fb6e364b`). + +== Shared responsibility model + +[cols="2,1"] +|=== +| Component | Owner + +| Redpanda spoke VPC, subnets, and EKS cluster | Redpanda +| Spoke route table entry: `0.0.0.0/0` -> Transit Gateway | Redpanda +| Interface VPC Endpoints (ECR, STS, CloudWatch Logs) in spoke | Redpanda +| Accepting the AWS RAM share invitation (cross-account) | Customer +| Transit Gateway | Customer +| Hub VPC | Customer +| Hub NAT Gateway, firewall, or NVA | Customer +| Hub route table entry: Redpanda spoke CIDR -> Transit Gateway | Customer +| Transit Gateway route tables | Customer +| AWS RAM resource share (cross-account) | Customer +| Egress path availability and HA | Customer +|=== + +Redpanda does not manage the availability of the egress path. Plan for multi-AZ NAT or redundant firewall instances in your hub if you need high availability. + +== Limitations + +Centralized egress is currently limited to setting the Transit Gateway ID at network creation. The field is immutable after the network is created. + +== Configure egress at network creation + +Centralized egress is set at network creation. You cannot change it on an existing network. Complete all hub-side setup, including your decision on Transit Gateway attachment acceptance (see <>), before you start the cluster creation flow. + +[IMPORTANT] +==== +To switch back to NAT Gateway egress, create a new network and a new cluster on it, then migrate your data. +==== + +To enable centralized egress, use the Redpanda Cloud UI: + +. Log in to https://cloud.redpanda.com[Redpanda Cloud^]. +. On the Clusters page, click *Create cluster*, then click *Create* for BYOC. +. Complete the cluster details. See xref:get-started:cluster-types/byoc/aws/create-byoc-cluster-aws.adoc[Create a BYOC Cluster on AWS] for the full create flow. +. On the *Network* page, in the *Transit Gateway ID* field, enter your Transit Gateway ID. This field is only visible if centralized egress is enabled for your organization. +. Click *Next* and complete the remaining steps. + +[NOTE] +==== +The CIDR block you enter on the *Network* page is the Redpanda spoke CIDR. Your hub public route table must include a static route for this CIDR with the Transit Gateway as the target so reply traffic from the internet routes back to the cluster. Add this route when you provision the hub. See xref:networking:byoc/aws/aws-hub-egress.adoc[Create an AWS Hub for Centralized Egress]. The Transit Gateway's own route table picks up the spoke CIDR automatically through route propagation when the spoke attaches. +==== + +// The Cloud API path for centralized egress is intentionally hidden while the +// `egress_spec` field is in PREVIEW. To re-enable the section when the field +// graduates, set the :show-preview-api: attribute (either in this page's +// header as `:show-preview-api:` or as a build-time attribute in the playbook). +ifdef::show-preview-api[] +=== Use the Cloud API + +[IMPORTANT] +==== +The `egress_spec` field is in preview. Its shape and validation rules may change. Pin a specific API version when scripting against this field, and revisit your integration when the field graduates to general availability. +==== + +Set `egress_spec.aws.transit_gateway_id` on the Network resource when you create a network. The field is required when `egress_spec` is set, and the value must match the pattern `tgw-` followed by hexadecimal characters. + +[,bash] +---- +curl -X POST https://api.redpanda.com/v1/networks \ + -H "Authorization: Bearer $REDPANDA_CLOUD_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "prod-us-east-2", + "cloud_provider": "CLOUD_PROVIDER_AWS", + "region": "us-east-2", + "cidr_block": "10.10.0.0/20", + "cluster_type": "TYPE_BYOC", + "egress_spec": { + "aws": { "transit_gateway_id": "tgw-0b629c5b4fb6e364b" } + } + }' +---- + +For the surrounding workflow (authenticating to the Cloud API, creating the resource group, creating the cluster against this network, and running `rpk cloud byoc aws apply`) see xref:manage:api/cloud-byoc-controlplane-api.adoc[Use the BYOC Control Plane API]. After the network is created, you can create the cluster against it through the UI or by calling the Cluster create endpoint with the new network ID. +endif::[] + +[#internet-endpoints] +== Internet endpoints required from your hub + +The Redpanda VPC uses Interface VPC Endpoints for ECR (`ecr.api` and `ecr.dkr`), STS, and CloudWatch Logs, so traffic to those AWS services does not leave the Redpanda VPC. Your hub VPC must still provide internet egress for the following non-AWS endpoints: + +[cols="1,1,2"] +|=== +| Endpoint | Protocol and port | Purpose + +| `docker.cloudsmith.io` | TCP/443 | Redpanda container images +| `public.ecr.aws` | TCP/443 | EKS public CNI and network policy agent images +| NTP servers | UDP/123 | Time sync +| `prometheus-community.github.io` and similar Helm repositories | TCP/443 | Monitoring stack Helm charts +|=== + +If your hub firewall enforces an allowlist, allow outbound traffic to these endpoints from the Redpanda spoke CIDR. + +== DNS resolution + +Centralized egress does not affect DNS resolution. The Redpanda VPC uses AmazonProvidedDNS at `VPC_CIDR + 2`, which resolves both internal and external hostnames as long as `0.0.0.0/0` is routable through your hub. You do not need a separate DNS path or DNS resolver endpoint in your hub. + +== Troubleshooting + +[cols="1,1"] +|=== +| Symptom | Likely cause + +| A new Transit Gateway VPC attachment from the Redpanda BYOC account appears in your hub AWS account with state `pendingAcceptance`, and cluster provisioning is stuck +| Your Transit Gateway has `AutoAcceptSharedAttachments` set to `disable`. In the AWS account that owns the Transit Gateway, open *VPC > Transit Gateway Attachments*, select the pending attachment, and choose *Actions > Accept transit gateway attachment*. To remove this manual step on future BYOC cluster provisions, set `AutoAcceptSharedAttachments=enable` on the Transit Gateway, but only if you trust every account the Transit Gateway is shared with through AWS RAM. + +| The Transit Gateway resource share does not appear in the BYOC AWS account, or cluster creation rejects the Transit Gateway ID +| The BYOC AWS account has not accepted the AWS RAM share invitation, so Redpanda cannot see the Transit Gateway. In that account, open AWS RAM and accept the pending invitation under *Resource share invitations*. Within an AWS Organization, shares are auto-accepted. + +| Cluster creation does not complete +| The cluster cannot reach the external services it needs during bootstrap. On the hub side, verify all of the following: the Transit Gateway default route table has `0.0.0.0/0` to the hub VPC attachment; the hub public route table has a static route for the Redpanda spoke CIDR to the Transit Gateway; the hub firewall allowlist includes the endpoints in <>; and the hub VPC CIDR and the Redpanda spoke CIDR do not overlap. If these all look correct, contact Redpanda Support. +|=== + +== Next steps + +* xref:networking:byoc/aws/aws-hub-egress.adoc[Create an AWS Hub for Centralized Egress] +* xref:get-started:cluster-types/byoc/aws/create-byoc-cluster-aws.adoc[Create a BYOC Cluster on AWS] +* xref:networking:byoc/aws/transit-gateway.adoc[Add a Transit Gateway] (for routing client traffic, not egress)