Cloud on Miro's World

History of Faith

Tue, 31 Mar 2020 11:24:48 +0800

The beginnings
#

Once upon a time - actually, in late 2017-early 2018 - there was a QA/DevOps engineer named José who liked containers, Docker and worked for a company named Thinkwrap (that soon would become Pivotree) in Valencia, Spain.

José was responsible (among other things) for setting up integration and QA environments for multiple Hybris projects. Such environment, when being built in traditional way (from physical servers or VMs), normally requires quite a few pieces:

AWS Inspection

Thu, 03 Oct 2019 11:22:48 +0800

Bunch of one-liners for AWS
#

.. so that I find them faster next time

aws --output=json ec2 describe-instances | jq -r '.Reservations[].Instances[] | "\n" + .InstanceId + "  :  " + .KeyName + "  =>  " + .PublicIpAddress + " | " + .PublicDnsName,  .Tags[] as $tt | "  ... " + $tt.Key +":"+ $tt.Value'

aws ec2 describe-instances | jq '.Reservations[].Instances[] | .InstanceId + "  :  " + .Placement.AvailabilityZone + "  =>  " + .PublicIpAddress'

aws iam list-group-policies --group-name kops

aws ec2 describe-vpcs | jq '.Vpcs[] | .VpcId + " " + .CidrBlock'

aws iam list-users | jq '.Users[].UserName'

aws --output=json --region=ca-central-1 ec2 describe-vpcs | jq -r '.Vpcs[] | .VpcId + " " + .CidrBlock,.Tags[] as $tt | "  .. " + $tt.Key +":"+$tt.Value'


aws ec2 describe-vpcs | jq -e --raw-output '.Vpcs[] | "\n" + .VpcId + "  :  " + .CidrBlock,  .Tags[] as $tt | "  ... " + $tt.Key +":"+ $tt.Value'

aws ec2 describe-subnets | jq -e --raw-output '.Subnets[] | "\n" + .VpcId + "  :  " + .CidrBlock +" : " + .AvailabilityZone,  .Tags[] as $tt | "  ... " + $tt.Key +":"+ $tt.Value'

aws ec2 authorize-security-group-ingress --group-id sg-bfa45bd4 --protocol tcp --port 31672 --cidr 204.101.219.210/31

How many instances of each type do I have, and in what states?
#

Considering buying reserved instances or thinking about migrating to a newly introduced class?

aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]"

What CIDRs have Ingress Access to which Ports?
#

# This is helpful when you need to perform a survey or audit of your system boundaries. While such a task isn’t ever “easy”, it can go more smoothly with with a summary:

aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add'

Which Services am I using?
#

aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31  --granularity MONTHLY --metrics UsageQuantity --group-by Type=DIMENSION,Key=SERVICE | jq '.ResultsByTime[].Groups[] | select(.Metrics.UsageQuantity.Amount > 0) | .Keys[0]'


# How much they cost
aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31  --granularity MONTHLY --metrics USAGE_QUANTITY BLENDED_COST  --group-by Type=DIMENSION,Key=SERVICE | jq '[ .ResultsByTime[].Groups[] | select(.Metrics.BlendedCost.Amount > "0") | { (.Keys[0]): .Metrics.BlendedCost } ] | sort_by(.Amount) | add'

# Instances running

aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]"

CIDR access to ports
#

aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add'

Lambda runtimes
#

aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ runtime:.[0].Runtime, functions:[.[]|.FunctionName] }
]"

Memory size
#

aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ (.[0].Runtime): [.[]|{ name: .FunctionName, timeout: .Timeout, memory: .MemorySize }] }]"

Lambda Environment variables
#

aws lambda list-functions | jq -r '[.Functions[]|{name: .FunctionName, env: .Environment.Variables}]|.[]|select(.env|length > 0)'

AWS Certified Developer - Associate

Sun, 28 Oct 2018 11:22:48 +0800

Got the second certification today

https://www.certmetrics.com/amazon/public/badge.aspx?i=2&t=c&d=2018-11-28&ci=AWS00261816

AWS Certified Solution Architect - Associate

Thu, 07 Jun 2018 11:22:48 +0800

Step 1 done.

https://www.certmetrics.com/amazon/public/badge.aspx?i=1&t=c&d=2018-06-06&ci=AWS00261816&fbclid=IwAR2K8B4FyKWVstBZXCXn5GhYX9iRyB4U2q8l2ROrM_16m-rlclioalnp0O4

And now dilemma: go deeper into AWS or look around what Google and Azure have to offer ?

So many clouds, so little time …

Blog of the day - Tapas Forever

Thu, 07 Jun 2018 11:22:48 +0800

Fellow Canadian describes his path to Spain

https://tapasforever.com/

Install AWS CLI on Mac

Tue, 14 Nov 2017 14:22:48 +0800

Install AWS CLI on Mac
#

The Mac comes with Python 2.7 and no pip

This guide recommends upgrade to Python 3 which I want to avoid: http://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html

Unsuccesfull path - keep the Python 2.7
#

➜  ~ pip --version
zsh: command not found: pip

➜  ~ sudo easy_install pip
Password:
Searching for pip
Reading https://pypi.python.org/simple/pip/
Best match: pip 9.0.1
Downloading https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9
Processing pip-9.0.1.tar.gz
Writing /tmp/easy_install-QnLLJp/pip-9.0.1/setup.cfg
Running pip-9.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QnLLJp/pip-9.0.1/egg-dist-tmp-03hMh7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching '.mailmap'
warning: no previously-included files found matching '.travis.yml'
warning: no previously-included files found matching '.landscape.yml'
warning: no previously-included files found matching 'pip/_vendor/Makefile'
warning: no previously-included files found matching 'tox.ini'
warning: no previously-included files found matching 'dev-requirements.txt'
warning: no previously-included files found matching 'appveyor.yml'
no previously-included directories found matching '.github'
no previously-included directories found matching '.travis'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'contrib'
no previously-included directories found matching 'tasks'
no previously-included directories found matching 'tests'
creating /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg
Extracting pip-9.0.1-py2.7.egg to /Library/Python/2.7/site-packages
Adding pip 9.0.1 to easy-install.pth file
Installing pip script to /usr/local/bin
Installing pip2.7 script to /usr/local/bin
Installing pip2 script to /usr/local/bin

Installed /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

➜  ~ pip --version
pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)

➜  ~ pip install awscli --upgrade --user
Collecting awscli
  Downloading awscli-1.11.185-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 1.1MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
  Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting s3transfer<0.2.0,>=0.1.9 (from awscli)
  Downloading s3transfer-0.1.11-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 6.2MB/s
Collecting botocore==1.7.43 (from awscli)
  Downloading botocore-1.7.43-py2.py3-none-any.whl (3.7MB)
    100% |████████████████████████████████| 3.7MB 363kB/s
Collecting docutils>=0.10 (from awscli)
  Downloading docutils-0.14-py2-none-any.whl (543kB)
    100% |████████████████████████████████| 552kB 2.4MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |████████████████████████████████| 51kB 5.9MB/s
Collecting PyYAML<=3.12,>=3.10 (from awscli)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% |████████████████████████████████| 256kB 3.5MB/s
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.9->awscli)
  Downloading futures-3.1.1-py2-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.7.43->awscli)
  Downloading jmespath-0.9.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.7.43->awscli)
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 4.5MB/s
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 7.8MB/s
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.7.43->awscli)
  Downloading six-1.11.0-py2.py3-none-any.whl
Installing collected packages: colorama, futures, jmespath, docutils, six, python-dateutil, botocore, s3transfer, pyasn1, rsa, PyYAML, awscli
  Running setup.py install for PyYAML ... done
Successfully installed PyYAML-3.12 awscli-1.11.185 botocore-1.7.43 colorama-0.3.7 docutils-0.14 futures-3.1.1 jmespath-0.9.3 pyasn1-0.3.7 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.11 six-1.11.0

➜  ~ aws --version
zsh: command not found: aws

➜  ~ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.13.1
BuildVersion:   17B48

This does not work.

Cloud on Miro's World

History of Faith

The beginnings #

AWS Inspection

Bunch of one-liners for AWS #

How many instances of each type do I have, and in what states? #

What CIDRs have Ingress Access to which Ports? #

Which Services am I using? #

CIDR access to ports #

Lambda runtimes #

Memory size #

Lambda Environment variables #