Aws

My company, Klassify s.r.o., has had a domain for a long time before it was even registered (I registered klassify.com way back in early 2000 in Canada, but never actually did anything with it) - and a web page since this afternoon. For all that time klassify.com carried only my email - there was no A record at all, nothing was served, and whoever looked up the company from an invoice or a contract got a browser error and, I suppose, some doubts about whether the company exists. So here is how it got fixed in one afternoon, in the order it happened and with the reasoning. Nothing here is clever or sophisticated or the only/the best way. The point is the sequence of creation: there are two places in it where it is quite easy to break something that matters more than a web page.

Many shapes of Architecture # According to the wise people, an architecture is the shared understanding that the expert developers/system builders/maintainers have of the system design. It’s purpose is to maximize the probability that the decisions you have to make early in a project are the right ones. It is sort of a blueprint for the future system and captures parts, relations, boundaries, constraints.

The beginnings # Once upon a time - actually, in late 2017-early 2018 - there was a QA/DevOps engineer named José who liked containers, Docker and worked for a company named Thinkwrap (that soon would become Pivotree) in Valencia, Spain. José was responsible (among other things) for setting up integration and QA environments for multiple Hybris projects. Such environment, when being built in traditional way (from physical servers or VMs), normally requires quite a few pieces:

All Day DevOps 2019 - Notes # I have attended the All Day Devops - https://www.alldaydevops.com/ on 06 Nov 2019 - an event that runs for 24 hours and has multiple tracks of content organized in 4 blocks. The tracks: Keynotes Cultural Change DevSecOps SRE CI/CD Everything Cloud The complete list is here: https://www.alldaydevops.com/2019-live-schedule - the play button points to the block video.

Bunch of one-liners for AWS # .. so that I find them faster next time aws --output=json ec2 describe-instances | jq -r '.Reservations[].Instances[] | "\n" + .InstanceId + " : " + .KeyName + " => " + .PublicIpAddress + " | " + .PublicDnsName, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 describe-instances | jq '.Reservations[].Instances[] | .InstanceId + " : " + .Placement.AvailabilityZone + " => " + .PublicIpAddress' aws iam list-group-policies --group-name kops aws ec2 describe-vpcs | jq '.Vpcs[] | .VpcId + " " + .CidrBlock' aws iam list-users | jq '.Users[].UserName' aws --output=json --region=ca-central-1 ec2 describe-vpcs | jq -r '.Vpcs[] | .VpcId + " " + .CidrBlock,.Tags[] as $tt | " .. " + $tt.Key +":"+$tt.Value' aws ec2 describe-vpcs | jq -e --raw-output '.Vpcs[] | "\n" + .VpcId + " : " + .CidrBlock, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 describe-subnets | jq -e --raw-output '.Subnets[] | "\n" + .VpcId + " : " + .CidrBlock +" : " + .AvailabilityZone, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 authorize-security-group-ingress --group-id sg-bfa45bd4 --protocol tcp --port 31672 --cidr 204.101.219.210/31 How many instances of each type do I have, and in what states? # Considering buying reserved instances or thinking about migrating to a newly introduced class? aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]" What CIDRs have Ingress Access to which Ports? # # This is helpful when you need to perform a survey or audit of your system boundaries. While such a task isn’t ever “easy”, it can go more smoothly with with a summary: aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add' Which Services am I using? # aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31 --granularity MONTHLY --metrics UsageQuantity --group-by Type=DIMENSION,Key=SERVICE | jq '.ResultsByTime[].Groups[] | select(.Metrics.UsageQuantity.Amount > 0) | .Keys[0]' # How much they cost aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31 --granularity MONTHLY --metrics USAGE_QUANTITY BLENDED_COST --group-by Type=DIMENSION,Key=SERVICE | jq '[ .ResultsByTime[].Groups[] | select(.Metrics.BlendedCost.Amount > "0") | { (.Keys[0]): .Metrics.BlendedCost } ] | sort_by(.Amount) | add' # Instances running aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]" CIDR access to ports # aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add' Lambda runtimes # aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ runtime:.[0].Runtime, functions:[.[]|.FunctionName] } ]" Memory size # aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ (.[0].Runtime): [.[]|{ name: .FunctionName, timeout: .Timeout, memory: .MemorySize }] }]" Lambda Environment variables # aws lambda list-functions | jq -r '[.Functions[]|{name: .FunctionName, env: .Environment.Variables}]|.[]|select(.env|length > 0)'

One day round trip Valencia to Madrid and back. It is 350 km, but AVE (high velocity train) makes it in 1.5 hr. Take that, Via Rail !! Travelling at 300 km/h is unreal. It is actually quieter and more smooth than Canadian train at 80 - all that makes the speed believable is the velocity display and view out of the window when train goes around something closer to the rails.

Got the second certification today https://www.certmetrics.com/amazon/public/badge.aspx?i=2&t=c&d=2018-11-28&ci=AWS00261816

Fellow Canadian describes his path to Spain https://tapasforever.com/

Step 1 done. https://www.certmetrics.com/amazon/public/badge.aspx?i=1&t=c&d=2018-06-06&ci=AWS00261816&fbclid=IwAR2K8B4FyKWVstBZXCXn5GhYX9iRyB4U2q8l2ROrM_16m-rlclioalnp0O4 And now dilemma: go deeper into AWS or look around what Google and Azure have to offer ? So many clouds, so little time …

This describes creation of K8s cluster in AWS environment from scratch as done for microservices based eCommerce project. Pre-requisites # AWS account access - IAM with Admin privileges AWS CLI installed - https://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html CLI credentials (secret key + api Key) - https://docs.aws.amazon.com/cli/latest/reference/iam/index.html Kubectl and Kops local install ** see https://github.com/kubernetes/kops and https://kubernetes.io/docs/tasks/tools/install-kubectl/ Test of access # ➜ etc git:(feature/kubernetes) aws --version aws-cli/1.11.180 Python/3.6.4 Darwin/17.5.0 botocore/1.7.38 ➜ etc git:(feature/kubernetes) kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} ➜ etc git:(feature/kubernetes) kops version Version 1.9.0 ➜ etc git:(feature/kubernetes) aws --profile twfulfill-miro iam list-users | jq '.Users[].UserName' .. DELETED ... "twfulfillment.prod.miro.adamy" .... Set the environment variables # These will be used by subsequent kops / aws commands