Devops

Got the second certification today https://www.certmetrics.com/amazon/public/badge.aspx?i=2&t=c&d=2018-11-28&ci=AWS00261816

Skopeo # I have stumbled upon this tool: https://github.com/containers/skopeo - command line tools for manipulation with Docker images and Docker image registries. It can work with OCI images as well as the original Docker v2 images. Installation # brew install skopeo Usage # ➜ deploy-2018-10-24 skopeo --override-os=linux inspect docker://docker.io/fedora { "Name": "docker.io/library/fedora", "Digest": "sha256:b41cd083421dd7aa46d619e958b75a026a5d5733f08f14ba6d53943d6106ea6d", "RepoTags": [ "20", "21", "22", "23", "24", "25", "26-modular", "26", "27", "28", "29", "branched", "heisenbug", "latest", "modular", "rawhide" ], "Created": "2018-09-07T19:20:02.809176076Z", "DockerVersion": "17.06.2-ce", "Labels": null, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:565884f490d9ec697e519c57d55d09e268542ef2c1340fd63262751fa308f047" ] } It works with protected repos as well - as long as creds are provided

Fellow Canadian describes his path to Spain https://tapasforever.com/

Step 1 done. https://www.certmetrics.com/amazon/public/badge.aspx?i=1&t=c&d=2018-06-06&ci=AWS00261816&fbclid=IwAR2K8B4FyKWVstBZXCXn5GhYX9iRyB4U2q8l2ROrM_16m-rlclioalnp0O4 And now dilemma: go deeper into AWS or look around what Google and Azure have to offer ? So many clouds, so little time …

This describes creation of K8s cluster in AWS environment from scratch as done for microservices based eCommerce project. Pre-requisites # AWS account access - IAM with Admin privileges AWS CLI installed - https://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html CLI credentials (secret key + api Key) - https://docs.aws.amazon.com/cli/latest/reference/iam/index.html Kubectl and Kops local install ** see https://github.com/kubernetes/kops and https://kubernetes.io/docs/tasks/tools/install-kubectl/ Test of access # ➜ etc git:(feature/kubernetes) aws --version aws-cli/1.11.180 Python/3.6.4 Darwin/17.5.0 botocore/1.7.38 ➜ etc git:(feature/kubernetes) kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} ➜ etc git:(feature/kubernetes) kops version Version 1.9.0 ➜ etc git:(feature/kubernetes) aws --profile twfulfill-miro iam list-users | jq '.Users[].UserName' .. DELETED ... "twfulfillment.prod.miro.adamy" .... Set the environment variables # These will be used by subsequent kops / aws commands

The idea / motivation # In order to run VisualVM from predefined configuration you need to point the app to the user directory. This directory takes about 20MB, but 90% of the the content is actually generated or repeated. ➜ 4devops-visualvm git:(master) du -sh tw-sl/* 244K tw-sl/config 7.3M tw-sl/modules 52K tw-sl/repository 4.0K tw-sl/update 104K tw-sl/update_tracking 11M tw-sl/var ➜ 4devops-visualvm git:(master) du -sh tw-wool/* 248K tw-wool/config 7.2M tw-wool/modules 60K tw-wool/repository 4.0K tw-wool/update 104K tw-wool/update_tracking 11M tw-wool/var ➜ 4devops-visualvm git:(master) du -sh tw-wool 19M tw-wool The only part that is specific and worthy of maintaining in the Git repo is repository

Install AWS CLI on Mac # The Mac comes with Python 2.7 and no pip This guide recommends upgrade to Python 3 which I want to avoid: http://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html Unsuccesfull path - keep the Python 2.7 # ➜ ~ pip --version zsh: command not found: pip ➜ ~ sudo easy_install pip Password: Searching for pip Reading https://pypi.python.org/simple/pip/ Best match: pip 9.0.1 Downloading https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9 Processing pip-9.0.1.tar.gz Writing /tmp/easy_install-QnLLJp/pip-9.0.1/setup.cfg Running pip-9.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QnLLJp/pip-9.0.1/egg-dist-tmp-03hMh7 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires' warnings.warn(msg) warning: no previously-included files found matching '.coveragerc' warning: no previously-included files found matching '.mailmap' warning: no previously-included files found matching '.travis.yml' warning: no previously-included files found matching '.landscape.yml' warning: no previously-included files found matching 'pip/_vendor/Makefile' warning: no previously-included files found matching 'tox.ini' warning: no previously-included files found matching 'dev-requirements.txt' warning: no previously-included files found matching 'appveyor.yml' no previously-included directories found matching '.github' no previously-included directories found matching '.travis' no previously-included directories found matching 'docs/_build' no previously-included directories found matching 'contrib' no previously-included directories found matching 'tasks' no previously-included directories found matching 'tests' creating /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg Extracting pip-9.0.1-py2.7.egg to /Library/Python/2.7/site-packages Adding pip 9.0.1 to easy-install.pth file Installing pip script to /usr/local/bin Installing pip2.7 script to /usr/local/bin Installing pip2 script to /usr/local/bin Installed /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg Processing dependencies for pip Finished processing dependencies for pip ➜ ~ pip --version pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7) ➜ ~ pip install awscli --upgrade --user Collecting awscli Downloading awscli-1.11.185-py2.py3-none-any.whl (1.2MB) 100% |████████████████████████████████| 1.2MB 1.1MB/s Collecting colorama<=0.3.7,>=0.2.5 (from awscli) Downloading colorama-0.3.7-py2.py3-none-any.whl Collecting s3transfer<0.2.0,>=0.1.9 (from awscli) Downloading s3transfer-0.1.11-py2.py3-none-any.whl (54kB) 100% |████████████████████████████████| 61kB 6.2MB/s Collecting botocore==1.7.43 (from awscli) Downloading botocore-1.7.43-py2.py3-none-any.whl (3.7MB) 100% |████████████████████████████████| 3.7MB 363kB/s Collecting docutils>=0.10 (from awscli) Downloading docutils-0.14-py2-none-any.whl (543kB) 100% |████████████████████████████████| 552kB 2.4MB/s Collecting rsa<=3.5.0,>=3.1.2 (from awscli) Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB) 100% |████████████████████████████████| 51kB 5.9MB/s Collecting PyYAML<=3.12,>=3.10 (from awscli) Downloading PyYAML-3.12.tar.gz (253kB) 100% |████████████████████████████████| 256kB 3.5MB/s Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.9->awscli) Downloading futures-3.1.1-py2-none-any.whl Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.7.43->awscli) Downloading jmespath-0.9.3-py2.py3-none-any.whl Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.7.43->awscli) Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB) 100% |████████████████████████████████| 194kB 4.5MB/s Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli) Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB) 100% |████████████████████████████████| 71kB 7.8MB/s Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.7.43->awscli) Downloading six-1.11.0-py2.py3-none-any.whl Installing collected packages: colorama, futures, jmespath, docutils, six, python-dateutil, botocore, s3transfer, pyasn1, rsa, PyYAML, awscli Running setup.py install for PyYAML ... done Successfully installed PyYAML-3.12 awscli-1.11.185 botocore-1.7.43 colorama-0.3.7 docutils-0.14 futures-3.1.1 jmespath-0.9.3 pyasn1-0.3.7 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.11 six-1.11.0 ➜ ~ aws --version zsh: command not found: aws ➜ ~ sw_vers ProductName: Mac OS X ProductVersion: 10.13.1 BuildVersion: 17B48 This does not work.

The purpose of this is to evaluate options and suggest possible approaches for handling the sensitive information in Git repositories such as database credentials, API credentials, passwords, keys, certificates etc. Joel asked this question a few weeks ago - this is part of Milos’s security awareness program. The issue # Git repos store in cleartext all credentials, that may also be including PROD credentials.

This is based on J.’s’ question in Slack: Do we have some customer where we are moving a git repo to svn in an automated fashion I just need to move the git repo into svn periodically nightly The setup (as I understand it): on client side we have a worktree that is Git repository the files needs to be copied to SVN regularly Options # Use git-svn bridge # Git client allows to “pretend” it is SVN and pull/push against remote SVN repo

Task # Transfer full Git history (all branches, tags, etc) to Bitbucket so that development can continue from there User setup # List all keys in Gitolite must have access to Gitolite admin interface cd $ADMIN_HOME/pensieve/PRJ-admin/gitolite-admin ➜ gitolite-admin git:(master) ll keydir total 144K -rw-r--r-- 1 miro 401 Jun 29 2015 alter.pub -rw-r--r-- 1 miro 397 Jun 29 2015 codereview.pub -rw-r--r-- 1 miro 400 Apr 18 2016 hybris.pub -rw-r--r-- 1 miro 400 Apr 19 2016 hybris2.pub -rw-r----- 1 miro 398 Mar 30 2016 irae.pub ... DELETED ... -rw-r--r-- 1 miro 401 Jun 29 2015 miro.pub -rw-r--r-- 1 miro 411 Jun 29 2015 tkuser.jenkins0.pub -rw-r--r-- 1 miro 411 Jun 29 2015 tkuser.jenkins2.pub These will be split to 5 usergroups