kOps로 AWS에 K8S 배포하기
by softPine가시다님과 함께하는 PKOS 1주차 스터디.
Kops 설치 방법(EC2 Instance)
- OS
- Amazon Linux 2
curl -LO <https://github.com/kubernetes/kops/releases/download/$>(curl -s <https://api.github.com/repos/kubernetes/kops/releases/latest> | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
mv kops-linux-amd64 /usr/local/bin/kops
Route53 도메인 설정
- Route53에서 서브도메인을 지원하기에 서브도메인을 이용하여 생성한다.
- 생성 후 나오는 NameServers의 값을 저장한다.
aws route53 create-hosted-zone --name kops.domain.co.kr --caller-reference 1
---
{
"Location": "<https://route53.amazonaws.com/2013-04-01/hostedzone/Z0801636DLEL2UTQFMB6>",
"HostedZone": {
"Id": "/hostedzone/Z0801636DLEL2UTQFMB6",
"Name": "kops.domain.co.kr.",
"CallerReference": "1",
"Config": {
"PrivateZone": false
},
"ResourceRecordSetCount": 2
},
"ChangeInfo": {
"Id": "/change/C00464822MVNRLF9UM031",
"Status": "PENDING",
"SubmittedAt": "2023-03-06T03:01:44.432000+00:00"
},
"DelegationSet": {
"NameServers": [
"ns-451.awsdns-56.com",
"ns-1671.awsdns-16.co.uk",
"ns-1239.awsdns-26.org",
"ns-536.awsdns-03.net"
]
}
}- 상위 도메인인 domain.co.kr에서 서브도메인으로 라우팅을 할 수 있도록 Record를 서브도메인 명으로 생성한다.
- aws cli로 간단하게 처리를 하기 위해 아래 JSON 파일을 생성한다.
{
"Comment": "CREATE/DELETE/UPSERT a record ",
"Changes": [{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "kops.domain.co.kr",
"Type": "NS",
"TTL": 172800,
"ResourceRecords": [
{ "Value": "ns-451.awsdns-56.com"},
{ "Value": "ns-1671.awsdns-16.co.uk"},
{ "Value": "ns-1239.awsdns-26.org"},
{ "Value": "ns-536.awsdns-03.net"}
]
}}]
}- 위와 같은 json 파일 생성 후 아래 명령어로 서브도메인을 생성한다.
aws route53 change-resource-record-sets --hosted-zone-id Z14NHEMS72ESY4 --change-batch file://Record.json
---
{
"ChangeInfo": {
"Id": "/change/C00498933LQH5QIAAL9R",
"Status": "PENDING",
"SubmittedAt": "2023-03-06T03:18:17.965000+00:00",
"Comment": "CREATE/DELETE/UPSERT a record "
}
}
- JSON 파일에 오류가 없는 한 Pending Status가 고유 ID와 함께 반환되어야 한다.
- 고유 ID를 포함한 아래 명령어로 Route53의 전파 상태를 확인한다. INSYNC는 변경 내용이 Route53 DNS 서버에 전파 되었음을 나타낸다.
aws route53 get-change --id /change/C00498933LQH5QIAAL9R
---
{
"ChangeInfo": {
"Id": "/change/C00498933LQH5QIAAL9R",
"Status": "INSYNC",
"SubmittedAt": "2023-03-06T03:18:17.965000+00:00",
"Comment": "CREATE/DELETE/UPSERT a record "
}
}
- INSYNC 상태가 보인다면 아래 명령어로 정상적으로 NameServer가 질의되는 것을 확인하면 된다.
dig ns kops.domain.co.kr
---
; <<>> DiG 9.16.1-Ubuntu <<>> ns kops.domain.co.kr
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3571
;; flags: qr rd ad; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;kops.domain.co.kr. IN NS
;; ANSWER SECTION:
kops.domain.co.kr. 0 IN NS ns-1239.awsdns-26.org.
kops.domain.co.kr. 0 IN NS ns-1671.awsdns-16.co.uk.
kops.domain.co.kr. 0 IN NS ns-451.awsdns-56.com.
kops.domain.co.kr. 0 IN NS ns-536.awsdns-03.net.
ns-1239.awsdns-26.org. 0 IN A 205.251.196.215
ns-1239.awsdns-26.org. 0 IN AAAA 2600:9000:5304:d700::1
ns-1671.awsdns-16.co.uk. 0 IN A 205.251.198.135
ns-1671.awsdns-16.co.uk. 0 IN AAAA 2600:9000:5306:8700::1
ns-451.awsdns-56.com. 0 IN A 205.251.193.195
ns-451.awsdns-56.com. 0 IN AAAA 2600:9000:5301:c300::1
ns-536.awsdns-03.net. 0 IN A 205.251.194.24
ns-536.awsdns-03.net. 0 IN AAAA 2600:9000:5302:1800::1
;; Query time: 0 msec
;; SERVER: 172.29.192.1#53(172.29.192.1)
;; WHEN: Mon Mar 06 12:33:49 KST 2023
;; MSG SIZE rcvd: 452클러스터 저장용 S3 버킷 생성
- Kops는 클러스터 설치 이후에도 클러스터를 관리할 수 있는데, 이를 위해 클러스터의 상태나 사용하는 키 정보들을 지속적으로 추적해야 하는데 이 정보를 S3에 저장할 수 있다.
- 다수의 클러스터에서 동일한 S3 버킷을 사용할 수 있고, 사용자는 이 S3 버킷을 같은 클러스터를 운영하는 운영자에게 공유할 수 있다.
- 하지만 이 S3 버킷에 접근 가능한 사람은 모든 클러스터에 관리자 권한으로 접근이 가능하게되니, 운영팀 이외에 공유되지 않도록 해야한다.
- 저는 Kops 운영 인스턴스를 AWS에 생성을 해두었기에 Instance Profile로 설정하여 사용하기로 했습니다.
S3 Bucket 생성
S3 Bucket 생성
aws s3api create-bucket --bucket cluster.kops.domain.co.kr \\
--region ap-northeast-2 \\
--object-ownership BucketOwnerEnforced \\
--create-bucket-configuration LocationConstraint=ap-northeast-2 \\
--versioning-configuration Status=EnableS3 Bucket Public Access Block 설정
aws s3api put-public-access-block --bucket cluster.kops.domain.co.kr \\
--public-access-block-configuration \\
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=trueS3 Bucket Policy 설정
- KopsS3-Bucket-Policy.json
{
"Id": "Policy1678159299540",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1678159297382",
"Action": [
"s3:*",
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::cluster.kops.domain.co.kr/*",
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:role/domain-SKR-KOPS-role"
]
}
}
]
}- Bucket Policy 설정
aws s3api put-bucket-policy --bucket cluster.kops.domain.co.kr --policy file://KopsS3-Bucket-Policy.jsonEC2 Instance Profile 설정
IAM Instance Profile Trust 파일 생성
- KopsS3-EC2-Trust.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
IAM Instance Profile Permission 파일 생성
- KopsS3-EC2-Permission.json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EventBridgeActions",
"Effect": "Allow",
"Action": [
"events:*",
"schemas:*",
"scheduler:*",
"pipes:*"
],
"Resource": "*"
},
{
"Sid": "IAMCreateServiceLinkedRoleForApiDestinations",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/AmazonEventBridgeApiDestinationsServiceRolePolicy",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": "apidestinations.events.amazonaws.com"
}
}
},
{
"Sid": "IAMCreateServiceLinkedRoleForAmazonEventBridgeSchemas",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/schemas.amazonaws.com/AWSServiceRoleForSchemas",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": "schemas.amazonaws.com"
}
}
},
{
"Sid": "SecretsManagerAccessForApiDestinations",
"Effect": "Allow",
"Action": [
"secretsmanager:CreateSecret",
"secretsmanager:UpdateSecret",
"secretsmanager:DeleteSecret",
"secretsmanager:GetSecretValue",
"secretsmanager:PutSecretValue"
],
"Resource": "arn:aws:secretsmanager:*:*:secret:events!*"
},
{
"Sid": "IAMPassRoleAccessForEventBridge",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/*",
"Condition": {
"StringLike": {
"iam:PassedToService": "events.amazonaws.com"
}
}
},
{
"Sid": "IAMPassRoleAccessForScheduler",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/*",
"Condition": {
"StringLike": {
"iam:PassedToService": "scheduler.amazonaws.com"
}
}
},
{
"Sid": "IAMPassRoleAccessForPipes",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/*",
"Condition": {
"StringLike": {
"iam:PassedToService": "pipes.amazonaws.com"
}
}
},
{
"Action": [
"sqs:*"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:AcceptVpcPeeringConnection",
"ec2:AcceptVpcEndpointConnections",
"ec2:AllocateAddress",
"ec2:AssignIpv6Addresses",
"ec2:AssignPrivateIpAddresses",
"ec2:AssociateAddress",
"ec2:AssociateDhcpOptions",
"ec2:AssociateRouteTable",
"ec2:AssociateSubnetCidrBlock",
"ec2:AssociateVpcCidrBlock",
"ec2:AttachClassicLinkVpc",
"ec2:AttachInternetGateway",
"ec2:AttachNetworkInterface",
"ec2:AttachVpnGateway",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateCarrierGateway",
"ec2:CreateCustomerGateway",
"ec2:CreateDefaultSubnet",
"ec2:CreateDefaultVpc",
"ec2:CreateDhcpOptions",
"ec2:CreateEgressOnlyInternetGateway",
"ec2:CreateFlowLogs",
"ec2:CreateInternetGateway",
"ec2:CreateLocalGatewayRouteTableVpcAssociation",
"ec2:CreateNatGateway",
"ec2:CreateNetworkAcl",
"ec2:CreateNetworkAclEntry",
"ec2:CreateNetworkInterface",
"ec2:CreateNetworkInterfacePermission",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:CreateSecurityGroup",
"ec2:CreateSubnet",
"ec2:CreateTags",
"ec2:CreateVpc",
"ec2:CreateVpcEndpoint",
"ec2:CreateVpcEndpointConnectionNotification",
"ec2:CreateVpcEndpointServiceConfiguration",
"ec2:CreateVpcPeeringConnection",
"ec2:CreateVpnConnection",
"ec2:CreateVpnConnectionRoute",
"ec2:CreateVpnGateway",
"ec2:DeleteCarrierGateway",
"ec2:DeleteCustomerGateway",
"ec2:DeleteDhcpOptions",
"ec2:DeleteEgressOnlyInternetGateway",
"ec2:DeleteFlowLogs",
"ec2:DeleteInternetGateway",
"ec2:DeleteLocalGatewayRouteTableVpcAssociation",
"ec2:DeleteNatGateway",
"ec2:DeleteNetworkAcl",
"ec2:DeleteNetworkAclEntry",
"ec2:DeleteNetworkInterface",
"ec2:DeleteNetworkInterfacePermission",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSubnet",
"ec2:DeleteTags",
"ec2:DeleteVpc",
"ec2:DeleteVpcEndpoints",
"ec2:DeleteVpcEndpointConnectionNotifications",
"ec2:DeleteVpcEndpointServiceConfigurations",
"ec2:DeleteVpcPeeringConnection",
"ec2:DeleteVpnConnection",
"ec2:DeleteVpnConnectionRoute",
"ec2:DeleteVpnGateway",
"ec2:DescribeAccountAttributes",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeCarrierGateways",
"ec2:DescribeClassicLinkInstances",
"ec2:DescribeCustomerGateways",
"ec2:DescribeDhcpOptions",
"ec2:DescribeEgressOnlyInternetGateways",
"ec2:DescribeFlowLogs",
"ec2:DescribeInstances",
"ec2:DescribeInternetGateways",
"ec2:DescribeIpv6Pools",
"ec2:DescribeLocalGatewayRouteTables",
"ec2:DescribeLocalGatewayRouteTableVpcAssociations",
"ec2:DescribeKeyPairs",
"ec2:DescribeMovingAddresses",
"ec2:DescribeNatGateways",
"ec2:DescribeNetworkAcls",
"ec2:DescribeNetworkInterfaceAttribute",
"ec2:DescribeNetworkInterfacePermissions",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribePrefixLists",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroupReferences",
"ec2:DescribeSecurityGroupRules",
"ec2:DescribeSecurityGroups",
"ec2:DescribeStaleSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcClassicLink",
"ec2:DescribeVpcClassicLinkDnsSupport",
"ec2:DescribeVpcEndpointConnectionNotifications",
"ec2:DescribeVpcEndpointConnections",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeVpcEndpointServiceConfigurations",
"ec2:DescribeVpcEndpointServicePermissions",
"ec2:DescribeVpcEndpointServices",
"ec2:DescribeVpcPeeringConnections",
"ec2:DescribeVpcs",
"ec2:DescribeVpnConnections",
"ec2:DescribeVpnGateways",
"ec2:DetachClassicLinkVpc",
"ec2:DetachInternetGateway",
"ec2:DetachNetworkInterface",
"ec2:DetachVpnGateway",
"ec2:DisableVgwRoutePropagation",
"ec2:DisableVpcClassicLink",
"ec2:DisableVpcClassicLinkDnsSupport",
"ec2:DisassociateAddress",
"ec2:DisassociateRouteTable",
"ec2:DisassociateSubnetCidrBlock",
"ec2:DisassociateVpcCidrBlock",
"ec2:EnableVgwRoutePropagation",
"ec2:EnableVpcClassicLink",
"ec2:EnableVpcClassicLinkDnsSupport",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:ModifySecurityGroupRules",
"ec2:ModifySubnetAttribute",
"ec2:ModifyVpcAttribute",
"ec2:ModifyVpcEndpoint",
"ec2:ModifyVpcEndpointConnectionNotification",
"ec2:ModifyVpcEndpointServiceConfiguration",
"ec2:ModifyVpcEndpointServicePermissions",
"ec2:ModifyVpcPeeringConnectionOptions",
"ec2:ModifyVpcTenancy",
"ec2:MoveAddressToVpc",
"ec2:RejectVpcEndpointConnections",
"ec2:RejectVpcPeeringConnection",
"ec2:ReleaseAddress",
"ec2:ReplaceNetworkAclAssociation",
"ec2:ReplaceNetworkAclEntry",
"ec2:ReplaceRoute",
"ec2:ReplaceRouteTableAssociation",
"ec2:ResetNetworkInterfaceAttribute",
"ec2:RestoreAddressToClassic",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress",
"ec2:UnassignIpv6Addresses",
"ec2:UnassignPrivateIpAddresses",
"ec2:UpdateSecurityGroupRuleDescriptionsEgress",
"ec2:UpdateSecurityGroupRuleDescriptionsIngress"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:*",
"organizations:DescribeAccount",
"organizations:DescribeOrganization",
"organizations:DescribeOrganizationalUnit",
"organizations:DescribePolicy",
"organizations:ListChildren",
"organizations:ListParents",
"organizations:ListPoliciesForTarget",
"organizations:ListRoots",
"organizations:ListPolicies",
"organizations:ListTargetsForPolicy"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:*",
"s3-object-lambda:*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"route53:*",
"route53domains:*",
"cloudfront:ListDistributions",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticbeanstalk:DescribeEnvironments",
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:GetBucketWebsite",
"ec2:DescribeVpcs",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeRegions",
"sns:ListTopics",
"sns:ListSubscriptionsByTopic",
"cloudwatch:DescribeAlarms",
"cloudwatch:GetMetricStatistics"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "apigateway:GET",
"Resource": "arn:aws:apigateway:*::/domainnames"
},
{
"Action": "ec2:*",
"Effect": "Allow",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "elasticloadbalancing:*",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "cloudwatch:*",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "autoscaling:*",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "*",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": [
"autoscaling.amazonaws.com",
"ec2scheduled.amazonaws.com",
"elasticloadbalancing.amazonaws.com",
"spot.amazonaws.com",
"spotfleet.amazonaws.com",
"transitgateway.amazonaws.com"
]
}
}
}
]
}
IAM Role 및 Instnace Profile 생성
aws iam create-role --role-name domain-SKR-KOPS-role --assume-role-policy-document file://KopsS3-EC2-Trust.json
aws iam put-role-policy --role-name domain-SKR-KOPS-role --policy-name domain-SKR-KOPS-policy --policy-document file://KopsS3-EC2-Permission.json
aws iam create-instance-profile --instance-profile-name domain-SKR-KOPS-instance-profile
aws iam add-role-to-instance-profile --instance-profile-name domain-SKR-KOPS-instance-profile --role-name domain-SKR-KOPS-roleEC2 Instance에 Instance Profile 연결
# Instance ID를 meta-data로 확인
curl http://169.254.169.254/latest/meta-data/instance-id
# InstanceID를 갖고 Role 설정
aws ec2 associate-iam-instance-profile --instance-id i-0f5816da773b2efeb --iam-instance-profile Name=domain-SKR-KOPS-instance-profileK8S Cluster 생성
kubectl 설치(EC2 Instance)
curl -LO "<https://dl.k8s.io/release/$>(curl -L -s <https://dl.k8s.io/release/stable.txt>)/bin/linux/amd64/kubectl"
curl -LO "<https://dl.k8s.io/$>(curl -L -s <https://dl.k8s.io/release/stable.txt>)/bin/linux/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256) kubectl" | sha256sum --check
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client --output=yaml
rm -rf ./kubectl*
ssh key 생성
ssh-keygen
---
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:0yGvi5rMwH77upu8l6tk3r66iD/ImnqcnOS72hlxwCo root@SKRI-Kops-01
The key's randomart image is:
+---[RSA 2048]----+
| |
| . |
| o . . |
| . . + . |
|E . . S o |
|...o o |
|.*+oo .. |
|.*O%.+o. . |
|O+O=^#Bo. |
+----[SHA256]-----+
환경 변수 설정
- cluster 이름(이전에 생성한 kops.domain.co.kr 도메인)과 s3명을 입력한다.
echo export KOPS_CLUSTER_NAME=kops.domain.co.kr >> ~/.bashrc
echo export KOPS_STATE_STORE=s3://cluster.kops.domain.co.kr >> ~/.bashrc
source ~/.bashrc클러스터 생성
kops create cluster \\
--name=${KOPS_CLUSTER_NAME} \\
--cloud=aws \\
--networking amazonvpc \\
--zones=ap-northeast-2a,ap-northeast-2c \\
--network-cidr 172.30.0.0/16 \\
--master-size t3.medium \\
--node-size t3.medium \\
--node-count=2 \\
--kubernetes-version "1.24.10" \\
--ssh-public-key ~/.ssh/id_rsa.pub -y
생성된 클러스터 확인
source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -F __start_kubectl k' >> ~/.bashrc
source ~/.bashrc
k get no
---
NAME STATUS ROLES AGE VERSION
i-025ff97f4a34dd7fa Ready node 19m v1.24.10
i-0c69a38808d05a0d7 Ready control-plane 20m v1.24.10
i-0e37ceecc596e40da Ready node 19m v1.24.10
블로그의 정보
나의 삽질저장소
softPine