Devops

The beginnings # Once upon a time - actually, in late 2017-early 2018 - there was a QA/DevOps engineer named José who liked containers, Docker and worked for a company named Thinkwrap (that soon would become Pivotree) in Valencia, Spain. José was responsible (among other things) for setting up integration and QA environments for multiple Hybris projects. Such environment, when being built in traditional way (from physical servers or VMs), normally requires quite a few pieces:

Accessing Rancher cluster # I was doing update on Kubernetes cluster I had not touch in a while when I noticed weird behaviour: the output of get command was incomplete the -o wide option had no effect on the command output (IP addresses are masked) ➜ .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get nodes NAME AGE ip-172-xx-xx-xx.ca-central-1.compute.internal 613d ip-172-xx-xx-xx.ca-central-1.compute.internal 628d ip-172-xx-xx-xx.ca-central-1.compute.internal 558d and

Accessing Rancher cluster # After creation, you can access the K8s cluster running Rancher by saving the config file available from the console and using standard kubectl command kubectl --kubeconfig ./quickstart.kubeconfig get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE cattle-system cattle-cluster-agent-5c98cb979f-bbhxf 1/1 Running 0 5d23h cattle-system cattle-node-agent-dwnxk 1/1 Running 0 5d23h cattle-system kube-api-auth-d4zgq 1/1 Running 0 5d23h ingress-nginx default-http-backend-67cf578fc4-grmsw 1/1 Running 0 5d23h ingress-nginx nginx-ingress-controller-mpnmb 1/1 Running 0 5d23h kube-system canal-jw85q 2/2 Running 0 5d23h kube-system coredns-5c59fd465f-47q5z 1/1 Running 0 5d23h kube-system coredns-autoscaler-d765c8497-sm5xf 1/1 Running 0 5d23h kube-system metrics-server-64f6dffb84-bp864 1/1 Running 0 5d23h kube-system rke-coredns-addon-deploy-job-95qnk 0/1 Completed 0 5d23h kube-system rke-ingress-controller-deploy-job-brv7w 0/1 Completed 0 5d23h kube-system rke-metrics-addon-deploy-job-tvt89 0/1 Completed 0 5d23h kube-system rke-network-plugin-deploy-job-7rqcr 0/1 Completed 0 5d23h The same information (and much more) is available using rancher CLI.

This is the scariest thing I have read since spring 2018: https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5 written by @david.gilbertson. It pretty sure what he describes is actually happening, has been happening before he described it and will be happening going on - just obviously not in the named module. The ecosystem of Node modules is so vast and so unstable that considerable number of project do not do enough to catch behaviour like this.

Parallel building challenges # Unlike Github, Gitlab considers Hugo blogs first class citizens and does not impose any restrictions on repo naming. I had 3 challenges to overcome related to co-existence of GH and GL versions: I need to use same repo for both GH and GL the submodule link for public does not work on GL the site root is different - I have no custom domain forward for GitLab Using same repo # To separate the GH and GL, I have added 2 remotes to repo and special branch gitlab-pages.

How does the blog setup and publishing work # Before I forget, here is how the current configuration works. There are two repositories at play: source repo - https://github.com/miroadamy/miroadamy.com github pages repo (== GHPR) - https://github.com/miroadamy/miroadamy.github.io The blog source repo contains only source files (.md, static media etc). This repo has usual submodules under the /themes - e.g. /themes/even which is one currently used.

I have decided to consolidate all piecemeal versions of my blogpost uder one roof and at the same to do these four things technology upgrade - from Jekyll to Hugo visual refresh of the page review tagging and categorization merge hidden posts from Wikis to one place Why Hugo replaced Jekyll # The version 3 of my blog (see below for a bit of history) has been hosted on Github pages and using the default static site generators - Jekyll.

All Day DevOps 2019 - Notes # I have attended the All Day Devops - https://www.alldaydevops.com/ on 06 Nov 2019 - an event that runs for 24 hours and has multiple tracks of content organized in 4 blocks. The tracks: Keynotes Cultural Change DevSecOps SRE CI/CD Everything Cloud The complete list is here: https://www.alldaydevops.com/2019-live-schedule - the play button points to the block video.

Bunch of one-liners for AWS # .. so that I find them faster next time aws --output=json ec2 describe-instances | jq -r '.Reservations[].Instances[] | "\n" + .InstanceId + " : " + .KeyName + " => " + .PublicIpAddress + " | " + .PublicDnsName, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 describe-instances | jq '.Reservations[].Instances[] | .InstanceId + " : " + .Placement.AvailabilityZone + " => " + .PublicIpAddress' aws iam list-group-policies --group-name kops aws ec2 describe-vpcs | jq '.Vpcs[] | .VpcId + " " + .CidrBlock' aws iam list-users | jq '.Users[].UserName' aws --output=json --region=ca-central-1 ec2 describe-vpcs | jq -r '.Vpcs[] | .VpcId + " " + .CidrBlock,.Tags[] as $tt | " .. " + $tt.Key +":"+$tt.Value' aws ec2 describe-vpcs | jq -e --raw-output '.Vpcs[] | "\n" + .VpcId + " : " + .CidrBlock, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 describe-subnets | jq -e --raw-output '.Subnets[] | "\n" + .VpcId + " : " + .CidrBlock +" : " + .AvailabilityZone, .Tags[] as $tt | " ... " + $tt.Key +":"+ $tt.Value' aws ec2 authorize-security-group-ingress --group-id sg-bfa45bd4 --protocol tcp --port 31672 --cidr 204.101.219.210/31 How many instances of each type do I have, and in what states? # Considering buying reserved instances or thinking about migrating to a newly introduced class? aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]" What CIDRs have Ingress Access to which Ports? # # This is helpful when you need to perform a survey or audit of your system boundaries. While such a task isn’t ever “easy”, it can go more smoothly with with a summary: aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add' Which Services am I using? # aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31 --granularity MONTHLY --metrics UsageQuantity --group-by Type=DIMENSION,Key=SERVICE | jq '.ResultsByTime[].Groups[] | select(.Metrics.UsageQuantity.Amount > 0) | .Keys[0]' # How much they cost aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31 --granularity MONTHLY --metrics USAGE_QUANTITY BLENDED_COST --group-by Type=DIMENSION,Key=SERVICE | jq '[ .ResultsByTime[].Groups[] | select(.Metrics.BlendedCost.Amount > "0") | { (.Keys[0]): .Metrics.BlendedCost } ] | sort_by(.Amount) | add' # Instances running aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]" CIDR access to ports # aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add' Lambda runtimes # aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ runtime:.[0].Runtime, functions:[.[]|.FunctionName] } ]" Memory size # aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ (.[0].Runtime): [.[]|{ name: .FunctionName, timeout: .Timeout, memory: .MemorySize }] }]" Lambda Environment variables # aws lambda list-functions | jq -r '[.Functions[]|{name: .FunctionName, env: .Environment.Variables}]|.[]|select(.env|length > 0)'

Two small utilities I have places into ~/bin: git-attic # Source: https://github.com/maximeh/dotfiles/blob/master/common/.bin/git-attic #!/bin/sh # git-attic [-M] [PATH] - list deleted files of Git repositories # # Use -M to not show renamed files, and other git-log options as you like. git log --raw --no-renames --date=short --format="%h %cd" "$@" | awk '/^[0-9a-f]/ { commit=$1; date=$2 } /^:/ && $5 == "D" { print date, commit "^:" $6 }' | less Example # ➜ kubernetes-the-hard-way git:(master) ✗ git log --raw --no-renames --date=short --format="%h %cd" "$@" | awk '/^[0-9a-f]/ { commit=$1; date=$2 } /^:/ && $5 == "D" { print date, commit "^:" $6 }' 2019-11-20 374e8d9^:docs/using-rst/pdp-template-rst.rst 2019-11-17 7911114^:src/certificates/admin-csr.json 2019-11-17 7911114^:src/certificates/admin-key.pem 2019-11-17 7911114^:src/certificates/admin.csr 2019-11-17 7911114^:src/certificates/admin.pem 2019-11-17 7911114^:src/certificates/ca-config.json 2019-11-17 7911114^:src/certificates/ca-csr.json 2019-11-17 7911114^:src/certificates/ca-key.pem 2019-11-17 7911114^:src/certificates/ca.csr 2019-11-17 7911114^:src/certificates/ca.pem 2019-11-17 7911114^:src/certificates/copy-certificates-to-servers.sh git trail # I cannot remember where is this one from :-(