Cloud

The beginnings # Once upon a time - actually, in late 2017-early 2018 - there was a QA/DevOps engineer named José who liked containers, Docker and worked for a company named Thinkwrap (that soon would become Pivotree) in Valencia, Spain. José was responsible (among other things) for setting up integration and QA environments for multiple Hybris projects. Such environment, when being built in traditional way (from physical servers or VMs), normally requires quite a few pieces:

Bunch of one-liners for AWS # .. so that I find them faster next time aws --output=json ec2 describe-instances | jq -r '.Reservations[].Instances[] | "\n" + .InstanceId + " : " + .KeyName + " => " + .PublicIpAddress + " | " + .PublicDnsName, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 describe-instances | jq '.Reservations[].Instances[] | .InstanceId + " : " + .Placement.AvailabilityZone + " => " + .PublicIpAddress' aws iam list-group-policies --group-name kops aws ec2 describe-vpcs | jq '.Vpcs[] | .VpcId + " " + .CidrBlock' aws iam list-users | jq '.Users[].UserName' aws --output=json --region=ca-central-1 ec2 describe-vpcs | jq -r '.Vpcs[] | .VpcId + " " + .CidrBlock,.Tags[] as $tt | " .. " + $tt.Key +":"+$tt.Value' aws ec2 describe-vpcs | jq -e --raw-output '.Vpcs[] | "\n" + .VpcId + " : " + .CidrBlock, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 describe-subnets | jq -e --raw-output '.Subnets[] | "\n" + .VpcId + " : " + .CidrBlock +" : " + .AvailabilityZone, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 authorize-security-group-ingress --group-id sg-bfa45bd4 --protocol tcp --port 31672 --cidr 204.101.219.210/31 How many instances of each type do I have, and in what states? # Considering buying reserved instances or thinking about migrating to a newly introduced class? aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]" What CIDRs have Ingress Access to which Ports? # # This is helpful when you need to perform a survey or audit of your system boundaries. While such a task isn’t ever “easy”, it can go more smoothly with with a summary: aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add' Which Services am I using? # aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31 --granularity MONTHLY --metrics UsageQuantity --group-by Type=DIMENSION,Key=SERVICE | jq '.ResultsByTime[].Groups[] | select(.Metrics.UsageQuantity.Amount > 0) | .Keys[0]' # How much they cost aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31 --granularity MONTHLY --metrics USAGE_QUANTITY BLENDED_COST --group-by Type=DIMENSION,Key=SERVICE | jq '[ .ResultsByTime[].Groups[] | select(.Metrics.BlendedCost.Amount > "0") | { (.Keys[0]): .Metrics.BlendedCost } ] | sort_by(.Amount) | add' # Instances running aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]" CIDR access to ports # aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add' Lambda runtimes # aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ runtime:.[0].Runtime, functions:[.[]|.FunctionName] } ]" Memory size # aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ (.[0].Runtime): [.[]|{ name: .FunctionName, timeout: .Timeout, memory: .MemorySize }] }]" Lambda Environment variables # aws lambda list-functions | jq -r '[.Functions[]|{name: .FunctionName, env: .Environment.Variables}]|.[]|select(.env|length > 0)'

Got the second certification today https://www.certmetrics.com/amazon/public/badge.aspx?i=2&t=c&d=2018-11-28&ci=AWS00261816

Fellow Canadian describes his path to Spain https://tapasforever.com/

Step 1 done. https://www.certmetrics.com/amazon/public/badge.aspx?i=1&t=c&d=2018-06-06&ci=AWS00261816&fbclid=IwAR2K8B4FyKWVstBZXCXn5GhYX9iRyB4U2q8l2ROrM_16m-rlclioalnp0O4 And now dilemma: go deeper into AWS or look around what Google and Azure have to offer ? So many clouds, so little time …

Install AWS CLI on Mac # The Mac comes with Python 2.7 and no pip This guide recommends upgrade to Python 3 which I want to avoid: http://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html Unsuccesfull path - keep the Python 2.7 # ➜ ~ pip --version zsh: command not found: pip ➜ ~ sudo easy_install pip Password: Searching for pip Reading https://pypi.python.org/simple/pip/ Best match: pip 9.0.1 Downloading https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9 Processing pip-9.0.1.tar.gz Writing /tmp/easy_install-QnLLJp/pip-9.0.1/setup.cfg Running pip-9.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QnLLJp/pip-9.0.1/egg-dist-tmp-03hMh7 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires' warnings.warn(msg) warning: no previously-included files found matching '.coveragerc' warning: no previously-included files found matching '.mailmap' warning: no previously-included files found matching '.travis.yml' warning: no previously-included files found matching '.landscape.yml' warning: no previously-included files found matching 'pip/_vendor/Makefile' warning: no previously-included files found matching 'tox.ini' warning: no previously-included files found matching 'dev-requirements.txt' warning: no previously-included files found matching 'appveyor.yml' no previously-included directories found matching '.github' no previously-included directories found matching '.travis' no previously-included directories found matching 'docs/_build' no previously-included directories found matching 'contrib' no previously-included directories found matching 'tasks' no previously-included directories found matching 'tests' creating /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg Extracting pip-9.0.1-py2.7.egg to /Library/Python/2.7/site-packages Adding pip 9.0.1 to easy-install.pth file Installing pip script to /usr/local/bin Installing pip2.7 script to /usr/local/bin Installing pip2 script to /usr/local/bin Installed /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg Processing dependencies for pip Finished processing dependencies for pip ➜ ~ pip --version pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7) ➜ ~ pip install awscli --upgrade --user Collecting awscli Downloading awscli-1.11.185-py2.py3-none-any.whl (1.2MB) 100% |████████████████████████████████| 1.2MB 1.1MB/s Collecting colorama<=0.3.7,>=0.2.5 (from awscli) Downloading colorama-0.3.7-py2.py3-none-any.whl Collecting s3transfer<0.2.0,>=0.1.9 (from awscli) Downloading s3transfer-0.1.11-py2.py3-none-any.whl (54kB) 100% |████████████████████████████████| 61kB 6.2MB/s Collecting botocore==1.7.43 (from awscli) Downloading botocore-1.7.43-py2.py3-none-any.whl (3.7MB) 100% |████████████████████████████████| 3.7MB 363kB/s Collecting docutils>=0.10 (from awscli) Downloading docutils-0.14-py2-none-any.whl (543kB) 100% |████████████████████████████████| 552kB 2.4MB/s Collecting rsa<=3.5.0,>=3.1.2 (from awscli) Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB) 100% |████████████████████████████████| 51kB 5.9MB/s Collecting PyYAML<=3.12,>=3.10 (from awscli) Downloading PyYAML-3.12.tar.gz (253kB) 100% |████████████████████████████████| 256kB 3.5MB/s Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.9->awscli) Downloading futures-3.1.1-py2-none-any.whl Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.7.43->awscli) Downloading jmespath-0.9.3-py2.py3-none-any.whl Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.7.43->awscli) Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB) 100% |████████████████████████████████| 194kB 4.5MB/s Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli) Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB) 100% |████████████████████████████████| 71kB 7.8MB/s Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.7.43->awscli) Downloading six-1.11.0-py2.py3-none-any.whl Installing collected packages: colorama, futures, jmespath, docutils, six, python-dateutil, botocore, s3transfer, pyasn1, rsa, PyYAML, awscli Running setup.py install for PyYAML ... done Successfully installed PyYAML-3.12 awscli-1.11.185 botocore-1.7.43 colorama-0.3.7 docutils-0.14 futures-3.1.1 jmespath-0.9.3 pyasn1-0.3.7 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.11 six-1.11.0 ➜ ~ aws --version zsh: command not found: aws ➜ ~ sw_vers ProductName: Mac OS X ProductVersion: 10.13.1 BuildVersion: 17B48 This does not work.