Devops on Miro's World

Catching up: Offsite Backups That Let Me Sleep

Thu, 01 Oct 2026 00:00:00 +0200

Consolidating a digital life onto one NAS has an ugly side effect: everything now burns in the same fire. In the previous posts I described how I moved decades of files, photos and documents onto a single Synology box at home. This post is about the second half of that story - getting a copy of it out of the house, encrypted, in a way that lets me sleep.

Picking the provider
#

The first lesson was that where a company keeps its servers and where it is registered are two different things. Backblaze has European data centers, but it is a Californian company, and the US CLOUD Act follows the company, not the building. For a family archive I wanted a European company in a European data center.

Catching up: Tailscale Everywhere

Tue, 29 Sep 2026 00:00:00 +0200

Seven devices, two of them NAS boxes, and a laptop that travels with me. For years, reaching any of them from outside the house was a small project every time. In July 2026 I put all of them on one private network, and the day every machine got one name that works from anywhere, half of my runbooks got shorter.

Why a mesh VPN
#

My home fleet is not big, but it is spread out: two MacBooks, an iPhone, an iPad, a Mac mini that is always on and serves as a small home hub, and two Synology NAS boxes, an old one and a new one.

How to monitor Bitbucket user and group permissions with API

Thu, 18 Feb 2021 13:00:00 +0800

How to manage users and groups in Bitbucket
#

Use case:

I want to maintain users and groups and their access to repositories in an organization in Bitbucket. To make sure I do not miss newly changed admin access, I want to use the Bitbucket API to retrieve the list of permissions on repository basis and on user/group bases.

How to mirror Github and Bitbucket repositories

Wed, 17 Feb 2021 13:00:00 +0800

Mirroring large number of GitHub or Bitbucket repositories
#

Use case:

I want to create local clone of a large number of repositories from an organization in GitHub / Bitbucket.

Assumptions
#

user has SSH key registered with the GH / BB - can clone using ssh protocol
user has successfully cloned at least one repo from command line before (the /etc/hosts contains host entries for BB and GH)
user has modern version of Python 3 installed (3.6 or better)
user has local installation of a reasonably modern Git

Steps
#

First I use the ./bb-repo-links.sh and ./gh-repo-links.sh to generate list of repositories. See these blog articles: How to list repositories in Bitbucket and How to list repositories in github

How to list Github repositories and users

Tue, 16 Feb 2021 13:00:00 +0800

How to list repositories in GitHub
#

Use Case:

I want to maintain repositories in an organization in GitHub. To make sure I do not miss newly added ones, I want to use API access to retrieve the list of repositories in the organization.

Setup
#

GitHub requires a token to access the API. This token is stored in an environment variable github_token and can be generated in the user profile section - settings - developer settings.

How to list Bitbucket repositories and users

Mon, 15 Feb 2021 13:00:00 +0800

How to list repositories in Bitbucket
#

Use case:

I want to maintain repositories in an organization in Bitbucket. To make sure I do not miss newly added ones, I want to use API access to retrieve the list of all repositories in the organization.

Setup
#

Bitbucket requires a username and password to access the API. This information is stored in the environment variables BBUSER and BBPASSWORD. The BBPASSWORD should be an access token / app password defined in the App passwords section of the user settings:

Documenting multi-repository projects

Mon, 23 Nov 2020 13:00:00 +0800

This blog post is a de-powerpointized version of a brownbag session in Pivotree. The goal was to increase awareness of tools and techniques for information / knowledge capture in the projects, specifically focusing on producing quality documentation so that everybody in project can access it and contribute to it, not only developers. We wanted to look at the tools that would allow engineers keep using their tools of trade (without impact on developer’s productivity) and at the same time make the information available without duplication of work to increase re-findability of information.

Reference Architecture as a Code

Mon, 10 Aug 2020 13:00:00 +0800

Many shapes of Architecture
#

According to the wise people, an architecture is the shared understanding that the expert developers/system builders/maintainers have of the system design. It’s purpose is to maximize the probability that the decisions you have to make early in a project are the right ones. It is sort of a blueprint for the future system and captures parts, relations, boundaries, constraints.

Accessing Terraform cloud Module registry programatically

Thu, 14 May 2020 21:22:48 +0800

First step is to get your personal token. Log on to Terraform Cloud, go to https://app.terraform.io/app/settings/tokens and generate new token.

Note: it is important to save the token to safe place - this is only time you will see it.

Export the token:

export TFE_TOKEN=<THE-TOKEN-VALUE>

The documentation for Terraform API is here

It is OK documentation but not the greatest one, however. There are two versions of the API - v1 and v2 and this is not clearly described. It could also used better examples.

Using SSH keys with multiple Bitbucket accounts

Tue, 12 May 2020 22:34:38 +0800

I have been using Bitbucket since 2012, long before my company moved from in-house hosted server running Gitolite and in-house instances of FishEye and Crucible to cloud based source control on Bitbucket Cloud. As legacy I was using my personal login with my gmail account as my Bitbucket identity.

With merge of 3 companies in 2018 and rebranding as Pivotree, we are in process of streamlining the identity management and using SSO for identification. Time has come to split my personal identity and work identity.

History of Faith

Tue, 31 Mar 2020 11:24:48 +0800

The beginnings
#

Once upon a time - actually, in late 2017-early 2018 - there was a QA/DevOps engineer named José who liked containers, Docker and worked for a company named Thinkwrap (that soon would become Pivotree) in Valencia, Spain.

José was responsible (among other things) for setting up integration and QA environments for multiple Hybris projects. Such environment, when being built in traditional way (from physical servers or VMs), normally requires quite a few pieces:

Kubectl client and server version mismatch

Sun, 19 Jan 2020 11:12:38 +0800

Accessing Rancher cluster
#

I was doing update on Kubernetes cluster I had not touch in a while when I noticed weird behaviour:

the output of get command was incomplete
the -o wide option had no effect on the command output

(IP addresses are masked)

➜  .kube git:(master) ✗ kubectl --kubeconfig dropship-dev-uat get nodes
NAME                                             AGE
ip-172-xx-xx-xx.ca-central-1.compute.internal    613d
ip-172-xx-xx-xx.ca-central-1.compute.internal    628d
ip-172-xx-xx-xx.ca-central-1.compute.internal    558d

and

Rancher CLI vs kubectl

Sat, 18 Jan 2020 11:22:48 +0800

Accessing Rancher cluster
#

After creation, you can access the K8s cluster running Rancher by saving the config file available from the console and using standard kubectl command

kubectl --kubeconfig ./quickstart.kubeconfig get pods --all-namespaces
NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
cattle-system   cattle-cluster-agent-5c98cb979f-bbhxf     1/1     Running     0          5d23h
cattle-system   cattle-node-agent-dwnxk                   1/1     Running     0          5d23h
cattle-system   kube-api-auth-d4zgq                       1/1     Running     0          5d23h
ingress-nginx   default-http-backend-67cf578fc4-grmsw     1/1     Running     0          5d23h
ingress-nginx   nginx-ingress-controller-mpnmb            1/1     Running     0          5d23h
kube-system     canal-jw85q                               2/2     Running     0          5d23h
kube-system     coredns-5c59fd465f-47q5z                  1/1     Running     0          5d23h
kube-system     coredns-autoscaler-d765c8497-sm5xf        1/1     Running     0          5d23h
kube-system     metrics-server-64f6dffb84-bp864           1/1     Running     0          5d23h
kube-system     rke-coredns-addon-deploy-job-95qnk        0/1     Completed   0          5d23h
kube-system     rke-ingress-controller-deploy-job-brv7w   0/1     Completed   0          5d23h
kube-system     rke-metrics-addon-deploy-job-tvt89        0/1     Completed   0          5d23h
kube-system     rke-network-plugin-deploy-job-7rqcr       0/1     Completed   0          5d23h

The same information (and much more) is available using rancher CLI.

On harvesting credit card numbers and passwords

Wed, 11 Dec 2019 21:22:48 +0800

This is the scariest thing I have read since spring 2018:

https://hackernoon.com/im-harvesting-credit-card-numbers-and-passwords-from-your-site-here-s-how-9a8cb347c5b5

written by @david.gilbertson.

It pretty sure what he describes is actually happening, has been happening before he described it and will be happening going on - just obviously not in the named module. The ecosystem of Node modules is so vast and so unstable that considerable number of project do not do enough to catch behaviour like this.

Building Hugo as well on GitLab pages

Sat, 23 Nov 2019 22:22:48 +0800

Parallel building challenges
#

Unlike Github, Gitlab considers Hugo blogs first class citizens and does not impose any restrictions on repo naming.

I had 3 challenges to overcome related to co-existence of GH and GL versions:

I need to use same repo for both GH and GL
the submodule link for public does not work on GL
the site root is different - I have no custom domain forward for GitLab

Using same repo
#

To separate the GH and GL, I have added 2 remotes to repo and special branch gitlab-pages.

Current Hugo setup on Github Pages

Fri, 22 Nov 2019 11:22:48 +0800

How does the blog setup and publishing work
#

Before I forget, here is how the current configuration works.

There are two repositories at play:

source repo - https://github.com/miroadamy/miroadamy.com
github pages repo (== GHPR) - https://github.com/miroadamy/miroadamy.github.io

The blog source repo contains only source files (.md, static media etc). This repo has usual submodules under the /themes - e.g. /themes/even which is one currently used.

From Jekyll to Hugo

Wed, 20 Nov 2019 23:22:48 +0800

I have decided to consolidate all piecemeal versions of my blogpost uder one roof and at the same to do these four things

technology upgrade - from Jekyll to Hugo
visual refresh of the page
review tagging and categorization
merge hidden posts from Wikis to one place

Why Hugo replaced Jekyll
#

The version 3 of my blog (see below for a bit of history) has been hosted on Github pages and using the default static site generators - Jekyll.

All Day DevOps 2019

Thu, 07 Nov 2019 11:22:48 +0800

All Day DevOps 2019 - Notes
#

I have attended the All Day Devops - https://www.alldaydevops.com/ on 06 Nov 2019 - an event that runs for 24 hours and has multiple tracks of content organized in 4 blocks.

The tracks:

Keynotes
Cultural Change
DevSecOps
SRE
CI/CD
Everything Cloud

The complete list is here: https://www.alldaydevops.com/2019-live-schedule - the play button points to the block video.

AWS Inspection

Thu, 03 Oct 2019 11:22:48 +0800

Bunch of one-liners for AWS
#

.. so that I find them faster next time

aws --output=json ec2 describe-instances | jq -r '.Reservations[].Instances[] | "\n" + .InstanceId + "  :  " + .KeyName + "  =>  " + .PublicIpAddress + " | " + .PublicDnsName,  .Tags[] as $tt | "  ... " + $tt.Key +":"+ $tt.Value'

aws ec2 describe-instances | jq '.Reservations[].Instances[] | .InstanceId + "  :  " + .Placement.AvailabilityZone + "  =>  " + .PublicIpAddress'

aws iam list-group-policies --group-name kops

aws ec2 describe-vpcs | jq '.Vpcs[] | .VpcId + " " + .CidrBlock'

aws iam list-users | jq '.Users[].UserName'

aws --output=json --region=ca-central-1 ec2 describe-vpcs | jq -r '.Vpcs[] | .VpcId + " " + .CidrBlock,.Tags[] as $tt | "  .. " + $tt.Key +":"+$tt.Value'


aws ec2 describe-vpcs | jq -e --raw-output '.Vpcs[] | "\n" + .VpcId + "  :  " + .CidrBlock,  .Tags[] as $tt | "  ... " + $tt.Key +":"+ $tt.Value'

aws ec2 describe-subnets | jq -e --raw-output '.Subnets[] | "\n" + .VpcId + "  :  " + .CidrBlock +" : " + .AvailabilityZone,  .Tags[] as $tt | "  ... " + $tt.Key +":"+ $tt.Value'

aws ec2 authorize-security-group-ingress --group-id sg-bfa45bd4 --protocol tcp --port 31672 --cidr 204.101.219.210/31

How many instances of each type do I have, and in what states?
#

Considering buying reserved instances or thinking about migrating to a newly introduced class?

aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]"

What CIDRs have Ingress Access to which Ports?
#

# This is helpful when you need to perform a survey or audit of your system boundaries. While such a task isn’t ever “easy”, it can go more smoothly with with a summary:

aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add'

Which Services am I using?
#

aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31  --granularity MONTHLY --metrics UsageQuantity --group-by Type=DIMENSION,Key=SERVICE | jq '.ResultsByTime[].Groups[] | select(.Metrics.UsageQuantity.Amount > 0) | .Keys[0]'


# How much they cost
aws ce get-cost-and-usage --time-period Start=2019-08-01,End=2019-08-31  --granularity MONTHLY --metrics USAGE_QUANTITY BLENDED_COST  --group-by Type=DIMENSION,Key=SERVICE | jq '[ .ResultsByTime[].Groups[] | select(.Metrics.BlendedCost.Amount > "0") | { (.Keys[0]): .Metrics.BlendedCost } ] | sort_by(.Amount) | add'

# Instances running

aws ec2 describe-instances | jq -r "[[.Reservations[].Instances[]|{ state: .State.Name, type: .InstanceType }]|group_by(.state)|.[]|{state: .[0].state, types: [.[].type]|[group_by(.)|.[]|{type: .[0], count: ([.[]]|length)}] }]"

CIDR access to ports
#

aws ec2 describe-security-groups | jq '[ .SecurityGroups[].IpPermissions[] as $a | { "ports": [($a.FromPort|tostring),($a.ToPort|tostring)]|unique, "cidr": $a.IpRanges[].CidrIp } ] | [group_by(.cidr)[] | { (.[0].cidr): [.[].ports|join("-")]|unique }] | add'

Lambda runtimes
#

aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ runtime:.[0].Runtime, functions:[.[]|.FunctionName] }
]"

Memory size
#

aws lambda list-functions | jq ".Functions | group_by(.Runtime)|[.[]|{ (.[0].Runtime): [.[]|{ name: .FunctionName, timeout: .Timeout, memory: .MemorySize }] }]"

Lambda Environment variables
#

aws lambda list-functions | jq -r '[.Functions[]|{name: .FunctionName, env: .Environment.Variables}]|.[]|select(.env|length > 0)'

Gitbits

Mon, 14 Jan 2019 11:22:48 +0800

Two small utilities I have places into ~/bin:

git-attic
#

Source: https://github.com/maximeh/dotfiles/blob/master/common/.bin/git-attic

#!/bin/sh
# git-attic [-M] [PATH] - list deleted files of Git repositories
#
# Use -M to not show renamed files, and other git-log options as you like.

git log --raw --no-renames --date=short --format="%h %cd" "$@" |
  awk '/^[0-9a-f]/ { commit=$1; date=$2 }
       /^:/ && $5 == "D" { print date, commit "^:" $6 }' |
  less

Example
#

➜  kubernetes-the-hard-way git:(master) ✗ git log --raw --no-renames --date=short --format="%h %cd" "$@" |
  awk '/^[0-9a-f]/ { commit=$1; date=$2 }
       /^:/ && $5 == "D" { print date, commit "^:" $6 }'
2019-11-20 374e8d9^:docs/using-rst/pdp-template-rst.rst
2019-11-17 7911114^:src/certificates/admin-csr.json
2019-11-17 7911114^:src/certificates/admin-key.pem
2019-11-17 7911114^:src/certificates/admin.csr
2019-11-17 7911114^:src/certificates/admin.pem
2019-11-17 7911114^:src/certificates/ca-config.json
2019-11-17 7911114^:src/certificates/ca-csr.json
2019-11-17 7911114^:src/certificates/ca-key.pem
2019-11-17 7911114^:src/certificates/ca.csr
2019-11-17 7911114^:src/certificates/ca.pem
2019-11-17 7911114^:src/certificates/copy-certificates-to-servers.sh

git trail
#

I cannot remember where is this one from :-(

AWS Certified Developer - Associate

Sun, 28 Oct 2018 11:22:48 +0800

Got the second certification today

https://www.certmetrics.com/amazon/public/badge.aspx?i=2&t=c&d=2018-11-28&ci=AWS00261816

Useful tool for Dockerhub

Wed, 24 Oct 2018 11:22:48 +0800

Skopeo
#

I have stumbled upon this tool: https://github.com/containers/skopeo - command line tools for manipulation with Docker images and Docker image registries.

It can work with OCI images as well as the original Docker v2 images.

Installation
#

brew install skopeo

Usage
#

➜  deploy-2018-10-24 skopeo --override-os=linux inspect docker://docker.io/fedora
{
    "Name": "docker.io/library/fedora",
    "Digest": "sha256:b41cd083421dd7aa46d619e958b75a026a5d5733f08f14ba6d53943d6106ea6d",
    "RepoTags": [
        "20",
        "21",
        "22",
        "23",
        "24",
        "25",
        "26-modular",
        "26",
        "27",
        "28",
        "29",
        "branched",
        "heisenbug",
        "latest",
        "modular",
        "rawhide"
    ],
    "Created": "2018-09-07T19:20:02.809176076Z",
    "DockerVersion": "17.06.2-ce",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:565884f490d9ec697e519c57d55d09e268542ef2c1340fd63262751fa308f047"
    ]
}

It works with protected repos as well - as long as creds are provided

AWS Certified Solution Architect - Associate

Thu, 07 Jun 2018 11:22:48 +0800

Step 1 done.

https://www.certmetrics.com/amazon/public/badge.aspx?i=1&t=c&d=2018-06-06&ci=AWS00261816&fbclid=IwAR2K8B4FyKWVstBZXCXn5GhYX9iRyB4U2q8l2ROrM_16m-rlclioalnp0O4

And now dilemma: go deeper into AWS or look around what Google and Azure have to offer ?

So many clouds, so little time …

Blog of the day - Tapas Forever

Thu, 07 Jun 2018 11:22:48 +0800

Fellow Canadian describes his path to Spain

https://tapasforever.com/

Creating Kubernetes Cluster in AWS from scratch with kops

Mon, 07 May 2018 11:22:48 +0800

This describes creation of K8s cluster in AWS environment from scratch as done for microservices based eCommerce project.

Pre-requisites
#

AWS account access - IAM with Admin privileges
AWS CLI installed - https://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html
CLI credentials (secret key + api Key) - https://docs.aws.amazon.com/cli/latest/reference/iam/index.html
Kubectl and Kops local install ** see https://github.com/kubernetes/kops and https://kubernetes.io/docs/tasks/tools/install-kubectl/

Test of access
#

➜  etc git:(feature/kubernetes) aws --version
aws-cli/1.11.180 Python/3.6.4 Darwin/17.5.0 botocore/1.7.38
 
➜  etc git:(feature/kubernetes) kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:27:55Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
 
➜  etc git:(feature/kubernetes) kops version
Version 1.9.0
 
➜  etc git:(feature/kubernetes) aws --profile twfulfill-miro iam list-users | jq '.Users[].UserName'
.. DELETED ...
"twfulfillment.prod.miro.adamy"
....

Set the environment variables
#

These will be used by subsequent kops / aws commands

VisualVM inside Docker container

Thu, 23 Nov 2017 11:22:48 +0800

The idea / motivation
#

In order to run VisualVM from predefined configuration you need to point the app to the user directory. This directory takes about 20MB, but 90% of the the content is actually generated or repeated.

➜  4devops-visualvm git:(master) du -sh tw-sl/*
244K    tw-sl/config
7.3M    tw-sl/modules
 52K    tw-sl/repository
4.0K    tw-sl/update
104K    tw-sl/update_tracking
 11M    tw-sl/var

➜  4devops-visualvm git:(master) du -sh tw-wool/*
248K    tw-wool/config
7.2M    tw-wool/modules
 60K    tw-wool/repository
4.0K    tw-wool/update
104K    tw-wool/update_tracking
 11M    tw-wool/var

➜  4devops-visualvm git:(master) du -sh tw-wool
 19M    tw-wool

The only part that is specific and worthy of maintaining in the Git repo is repository

Install AWS CLI on Mac

Tue, 14 Nov 2017 14:22:48 +0800

Install AWS CLI on Mac
#

The Mac comes with Python 2.7 and no pip

This guide recommends upgrade to Python 3 which I want to avoid: http://docs.aws.amazon.com/cli/latest/userguide/cli-install-macos.html

Unsuccesfull path - keep the Python 2.7
#

➜  ~ pip --version
zsh: command not found: pip

➜  ~ sudo easy_install pip
Password:
Searching for pip
Reading https://pypi.python.org/simple/pip/
Best match: pip 9.0.1
Downloading https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9
Processing pip-9.0.1.tar.gz
Writing /tmp/easy_install-QnLLJp/pip-9.0.1/setup.cfg
Running pip-9.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QnLLJp/pip-9.0.1/egg-dist-tmp-03hMh7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching '.mailmap'
warning: no previously-included files found matching '.travis.yml'
warning: no previously-included files found matching '.landscape.yml'
warning: no previously-included files found matching 'pip/_vendor/Makefile'
warning: no previously-included files found matching 'tox.ini'
warning: no previously-included files found matching 'dev-requirements.txt'
warning: no previously-included files found matching 'appveyor.yml'
no previously-included directories found matching '.github'
no previously-included directories found matching '.travis'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'contrib'
no previously-included directories found matching 'tasks'
no previously-included directories found matching 'tests'
creating /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg
Extracting pip-9.0.1-py2.7.egg to /Library/Python/2.7/site-packages
Adding pip 9.0.1 to easy-install.pth file
Installing pip script to /usr/local/bin
Installing pip2.7 script to /usr/local/bin
Installing pip2 script to /usr/local/bin

Installed /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

➜  ~ pip --version
pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)

➜  ~ pip install awscli --upgrade --user
Collecting awscli
  Downloading awscli-1.11.185-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 1.1MB/s
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
  Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting s3transfer<0.2.0,>=0.1.9 (from awscli)
  Downloading s3transfer-0.1.11-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 6.2MB/s
Collecting botocore==1.7.43 (from awscli)
  Downloading botocore-1.7.43-py2.py3-none-any.whl (3.7MB)
    100% |████████████████████████████████| 3.7MB 363kB/s
Collecting docutils>=0.10 (from awscli)
  Downloading docutils-0.14-py2-none-any.whl (543kB)
    100% |████████████████████████████████| 552kB 2.4MB/s
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |████████████████████████████████| 51kB 5.9MB/s
Collecting PyYAML<=3.12,>=3.10 (from awscli)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% |████████████████████████████████| 256kB 3.5MB/s
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.9->awscli)
  Downloading futures-3.1.1-py2-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.7.43->awscli)
  Downloading jmespath-0.9.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.7.43->awscli)
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 4.5MB/s
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Downloading pyasn1-0.3.7-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 7.8MB/s
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.7.43->awscli)
  Downloading six-1.11.0-py2.py3-none-any.whl
Installing collected packages: colorama, futures, jmespath, docutils, six, python-dateutil, botocore, s3transfer, pyasn1, rsa, PyYAML, awscli
  Running setup.py install for PyYAML ... done
Successfully installed PyYAML-3.12 awscli-1.11.185 botocore-1.7.43 colorama-0.3.7 docutils-0.14 futures-3.1.1 jmespath-0.9.3 pyasn1-0.3.7 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.11 six-1.11.0

➜  ~ aws --version
zsh: command not found: aws

➜  ~ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.13.1
BuildVersion:   17B48

This does not work.

Storing sensitive information in Git

Wed, 17 May 2017 11:22:48 +0800

The purpose of this is to evaluate options and suggest possible approaches for handling the sensitive information in Git repositories such as database credentials, API credentials, passwords, keys, certificates etc. Joel asked this question a few weeks ago - this is part of Milos’s security awareness program.

The issue
#

Git repos store in cleartext all credentials, that may also be including PROD credentials.

Managing code in remote SVN repo

Wed, 10 May 2017 11:22:48 +0800

This is based on J.’s’ question in Slack:

Do we have some customer where we are moving a git repo to svn in an automated fashion I just need to move the git repo into svn periodically nightly

The setup (as I understand it):

on client side we have a worktree that is Git repository
the files needs to be copied to SVN regularly

Options
#

Use git-svn bridge
#

Git client allows to “pretend” it is SVN and pull/push against remote SVN repo

Process for transferring Git repo from Gitolite to Bitbucket

Wed, 02 Nov 2016 11:22:48 +0800

Task
#

Transfer full Git history (all branches, tags, etc) to Bitbucket so that development can continue from there

User setup
#

List all keys in Gitolite
must have access to Gitolite admin interface

cd $ADMIN_HOME/pensieve/PRJ-admin/gitolite-admin
 
 
➜  gitolite-admin git:(master) ll keydir
total 144K
-rw-r--r-- 1 miro 401 Jun 29  2015 alter.pub
-rw-r--r-- 1 miro 397 Jun 29  2015 codereview.pub
-rw-r--r-- 1 miro 400 Apr 18  2016 hybris.pub
-rw-r--r-- 1 miro 400 Apr 19  2016 hybris2.pub
-rw-r----- 1 miro 398 Mar 30  2016 irae.pub
... DELETED ...
-rw-r--r-- 1 miro 401 Jun 29  2015 miro.pub
-rw-r--r-- 1 miro 411 Jun 29  2015 tkuser.jenkins0.pub
-rw-r--r-- 1 miro 411 Jun 29  2015 tkuser.jenkins2.pub

These will be split to 5 usergroups

Update all local branches in Git repo

Mon, 05 Sep 2016 11:22:48 +0800

Update all local branches in Git repo
#

Issue: The git pull updates just one current branch.

In DevOps I am following many Git repos and need to keep up to date for all local branches.

git ffwd-update command
#

Source: http://stackoverflow.com/questions/4318161/can-git-pull-all-update-all-my-local-branches

Create file named git-ffwd-update somewhere on the path (in my case ~/bin)

Learning resources for Linux and Command line

Mon, 30 May 2016 11:22:48 +0800

This quick summary to answer Megha Maiya’s question - I am adding to the Wiki in case it can be expanded later useful for somebody else:

expanded later
useful for somebody else

Free resources
#

There is HUGE amount of resources available for Linux and Unix out there. The issue is select those reasonably good

List of lists: https://github.com/learnbyexample/scripting_course/blob/master/Linux_curated_resources.md
http://ryanstutorials.net/, specially http://ryanstutorials.net/linuxtutorial/ and http://ryanstutorials.net/linuxtutorial/
http://www.tutorialspoint.com/unix/unix-getting-started.htm
(bash scripting) - https://bash.cyberciti.biz/guide/Main_Page
free book - http://unixmages.com/ufbm.pdf - very beginner friendly
list of free books: http://www.dailyartisan.com/news/open-source-e-books-for-linux/
Eli The computer guy - https://www.youtube.com/playlist?list=PLD6B6473ACF32C59D (I like this one)
YouTube channel - ultimate Linux tutorials - https://www.youtube.com/channel/UCn58SGUDUnUHRYKyNffHQrQ (from a generous Indian guy sharing his experience with the world for free. Not ideal audio quality, could use some editing - bot has lots of content)
EDX courses - https://www.edx.org/school/linuxfoundationx
http://www.tutorialspoint.com/listtutorials/linux/basic-concepts/1
Wikibooks - https://en.wikibooks.org/wiki/LPI_Linux_Certification

Hands On
#

Reading books and watching videos will have very little impact without trying things out.

Native OS-X Docker (Beta) - not quite there yet

Tue, 17 May 2016 11:22:48 +0800

I was intrigued by convenience of using Docker natively on Mac, without dealing with differences localhost != dockerhost, so I subscribed for Beta and installed it.

Good news 1
#

It can coexist with previous Docker installation (mostly) - see the note at the end. As long as you do not try to run native Docker and docker-machine at the same time, things coexist.

Starting ACC (ATG Control Centre) on OS-X

Thu, 24 Mar 2016 12:57:35 -0500

Copy the installation of ACC for respective version from a Linux VM:

The startup script in bin/startClient will fail, because it cannot determine which Java to use to run the app.

It is trying to determine Unix flavour, failing that defaults to Solaris and ends up with invalid settings.

Fix
#

Set the variable JAVA_VM to path to executable command on your system.

Git project statistic tool

Sun, 20 Mar 2016 12:57:35 -0500

Git project statistic tool
#

Python based, requires Python 2.6+

Works OOTB on Mac

Installation
#

Download TAR from Github
unzipped to /opt

➜  ll /opt/gitinspector-0.3.2
total 52K
-rw-r--r--  1 miro  999 Jul 29  2013 DESCRIPTION.txt
-rw-r--r--  1 miro  32K Jun 14  2013 LICENSE.txt
-rw-r--r--  1 miro   78 Jul  2  2013 MANIFEST.in
-rw-r--r--  1 miro  681 Jan 13  2014 README.txt
drwxr-xr-x 45 miro 1.5K Mar 20  2015 gitinspector
-rw-r--r--  1 miro 1.9K Jan 14  2014 setup.py
-rw-r--r--  1 miro  110 Jul 27  2013 stdeb.cfg

create symlink from ~/bin (or from whatever directory on $PATH)

ln -s /opt/gitinspector-0.3.2/gitinspector/gitinspector.py ~/bin/gitinspector.py
  
➜  ~  ll ~/bin | grep gitin
lrwxr-xr-x 1 miro   52 Mar 20 16:06 gitinspector.py -> /opt/gitinspector-0.3.2/gitinspector/gitinspector.py

Example
#

Look at scala sources - git@github.com:scala/scala.git

Fixing docker-machine error after upgrade

Wed, 16 Mar 2016 12:57:35 -0500

I had to upgrade docker toolbox on Yosemite to get latest docker-compose additions.

After installing the new package, I was not able to connect to docker machine:

➜  ~ docker-machine status
Stopped
➜  ~ docker-machine start default
Starting "default"...
(default) Check network to re-create if needed...
(default) Waiting for an IP...
Machine "default" was started.
Waiting for SSH to be available...
Detecting the provisioner...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
➜  ~ docker-machine env default
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": tls: DialWithDialer timed out
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

Regenerating the certificates did not solve the problem. Neither did reboot.

Accessing internal AWS servers directly from Mac

Thu, 18 Feb 2016 00:57:35 -0500

Motivation for this hack is my unwillingness to suffer the pain of using Windows UI just to access BCC, ACC or other ATG tools requiring non command-line interface.

The credit for finding out about approach is blog post by James McOrmond - VPN over SSH - who is using this neat trick from Linux for quite some time.

The software mentioned https://github.com/apenwarr/sshuttle supposedly works on OS-X, however, it has not been updated for 4-5 years and would not function for me (OS-X Yosemite).

Docker - getting started

Fri, 11 Dec 2015 11:22:48 +0800

After the reboot
#

docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
  
➜  gitolite-admin git:(master) docker-machine active
No active host found
 
 
➜  gitolite-admin git:(master) docker-machine env default
Error checking TLS connection: default is not running. Please start it in order to use the connection settings
 
 
➜  gitolite-admin git:(master) docker-machine start
Error: Expected to get one or more machine names as arguments
 
 
➜  gitolite-admin git:(master) docker-machine start default
(default) Starting VM...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
 
 
  
➜  gitolite-admin git:(master) docker-machine env default
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/miro/.docker/machine/machines/default"
export DOCKER_MACHINE_NAME="default"
# Run this command to configure your shell:
# eval "$(docker-machine env default)"
 
 
➜  gitolite-admin git:(master) eval "$(docker-machine env default)"
  
➜  gitolite-admin git:(master) docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64
Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

Useful guide
#

See https://viget.com/extend/how-to-use-docker-on-os-x-the-missing-guide

Starting ACC in VM

Wed, 15 Jul 2015 11:22:48 +0800

The ACC is installed in default location: ~/ATG/ACC11.1

This does not work OOTB. You must set 2 additional ENV properties

cd ~/ATG/ACC11.0
export ACC_HOME=`pwd`
export JAVA_VM=/usr/java/jdk1.7.0_51//bin/java
bin/startClient &

Starting from dyn/admin
#

The start in separate VM works OK.

Start in server VM brings errors message very often complaining of malformed Help URL.

The issue is documented here: http://stackoverflow.com/questions/20868244/how-to-fix-malformed-help-url-while-opening-acc-atg - for JBOSS.

How to truncate Git history

Tue, 14 Jul 2015 11:22:48 +0800

The general idea
#

#!/bin/bash
# Parameter: tag or sha1
git checkout --orphan temp $1
git commit -m "Truncated history"
git rebase --onto temp $1 master
git branch -D temp

Requires Git 1.8+

Works well for mostly linear history (aka Git used for diff-ing of the catalog feeds).

How it works
#

Starting repo (training):

How to get information about all ATG components

Sun, 12 Jul 2015 11:22:48 +0800

How to get information about all ATG Nucleus components
#

Dyn Admin has useful feature - Configuration reporter: http://localhost:8080/dyn/admin/atg/dynamo/admin/en/config-reporter-property-representation1.jhtml

There are two variations of the report: Property representation report - PRR (shorter version) and Bean representation report (BRR). The PRR does not report properties that have default values, while BRR is reporting current values of all properties. Compare the sizes

How to fix Jenkins build issues related to Git connectivity

Thu, 18 Jun 2015 11:22:48 +0800

Symptoms:
#

build fails
cannot checkout / connect to GIT

Validate this is the case
#

ssh to jenkins2
assume identity of jenkins user (VERY IMPORTANT)
go to /home/projects/workspace/PROJECTNAME directory

This directory is repository clone. Try ‘git status’ in that directory. If you see something like

How to solve 'Agent admitted failure to sign using the key'

Fri, 22 May 2015 11:22:48 +0800

After re-creating the SSH key, I got the above error from Gitolite.

Full trace:

➜  TWC  ssh -v thinkwrap@pensieve.thinkwrap.com
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to pensieve.thinkwrap.com [24.137.198.18] port 22.
debug1: Connection established.
debug1: identity file /home/thinkwrap/.ssh/identity type -1
debug1: identity file /home/thinkwrap/.ssh/identity-cert type -1
debug1: identity file /home/thinkwrap/.ssh/id_rsa type 1
debug1: identity file /home/thinkwrap/.ssh/id_rsa-cert type -1
debug1: identity file /home/thinkwrap/.ssh/id_dsa type -1
debug1: identity file /home/thinkwrap/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'pensieve.thinkwrap.com' is known and matches the RSA host key.
debug1: Found key in /home/thinkwrap/.ssh/known_hosts:2
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_500' not found
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_500' not found
debug1: Unspecified GSS failure.  Minor code may provide more information
 
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_500' not found
debug1: Next authentication method: publickey
debug1: Offering public key: /home/thinkwrap/.ssh/id_rsa
debug1: Remote: Forced command: /home/thinkwrap/gitolite/src/gitolite-shell miro.adamy+ATG11-TrainingVM
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Pty allocation disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 277
Agent admitted failure to sign using the key.
debug1: Trying private key: /home/thinkwrap/.ssh/identity
debug1: Trying private key: /home/thinkwrap/.ssh/id_dsa
debug1: Next authentication method: password
thinkwrap@pensieve.thinkwrap.com's password:

Solution
#

log-out, log on

Fixing the shutdown problem with JBoss

Sat, 11 Apr 2015 11:22:48 +0800

Observed at QA environment. During automated shutdown sequence, the exception was thrown:

$ bin/gdi-qa-com1.sh stop
JBOSS_CMD_STOP = java -classpath /opt/jboss/jboss-as/bin/shutdown.jar:/opt/jboss/jboss-as/client/jnet.jar org.jboss.Shutdown --shutdown --user=admin --password=admin -s jnp://0.0.0.0:1099
Exception in thread "main" javax.naming.CommunicationException: Could not obtain connection to any of these urls: 0.0.0.0:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]]]
    at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
    at javax.naming.InitialContext.lookup(InitialContext.java:392)
    at org.jboss.Shutdown.main(Shutdown.java:219)
Caused by: javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]]
    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335)
    at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
    ... 4 more
Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused]
    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305)
    ... 5 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:529)
    at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:97)
    at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:82)
    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301)
    ... 5 more
To watch the log: tail -f /var/log/atg/gdi-qa-com1.log

The conflict of 1099 between Endeca and ATG was resolved according Fixing port conflict on 1099 between out-of-the-box Endeca and JBOSS - the actual port was 11199

ATG Repository structure visualizer

Tue, 04 Nov 2014 11:22:48 +0800

The groovy script does parse the XML files (scraped from dyn-admin, or output of the XML combine) and produces file in the syntax of Plant UML - http://plantuml.sourceforge.net

 if (args.size() == 0) {
    println "Usage: groovy extractItems.groovy XMLFILE-IN-GSA-FORMAT.xml"
    System.exit(1)
}
def gsaFile = new XmlSlurper().parseText(new File(args[0]).text)
def desc = gsaFile.'item-descriptor'
def processItemDescriptor(myDesc) {
    def info = [:]
    def tables = []
    def props = [:]
    def descriptors = [:]
    for (table in myDesc.table) {
        String tableName = table.@name.text()
        tables += tableName
        // collect table's properties
        for (prop in table.property) {
            String name = "${prop.@name.text()}(${tableName})".replaceAll(/[ -]/, '_')
            props[name] = [:]
            if ("${prop.@'required'.text()}" != "") {
                props[name]['required'] = "${prop.@'required'.text()}"
            }
            if ("${prop.@'hidden'.text()}" != "") {
                props[name]['hidden'] = "${prop.@'hidden'.text()}"
            }
 
            if ("${prop.@'item-type'.text()}" != "") {
                props[name]['item-type'] = "${prop.@'item-type'.text()}"
                descriptors["${prop.@name.text()}"] = "${prop.@'item-type'.text().replaceAll(/[ -]/, '_')}"
            }
        }
    }
    info['tables'] = tables
    // scan for properties outside of tables
    for (prop in myDesc.property) {
        String name = "${prop.@name.text()}".replaceAll(/[ -]/, '_')
        props[name] = [:]
        if ("${prop.@'item-type'.text()}" != "") {
            props[name]['item-type'] = "${prop.@'item-type'.text()}"
        }      
        }
    info['properties'] = props
    info['descriptors'] = descriptors
    return info
}
 
println "Got ${desc.size()} item-descriptors"
 
println "@startuml"
for (idesc in desc) {
    def info = processItemDescriptor(idesc)
    def name = "${idesc.@name.text().replaceAll(/[ -]/, '_')}"
    println "\nclass  ${name} {"
    for (p in info.tables)
        println "\t"+p
    println ".."   
    for (p in info.properties)
        println "\t"+p
    println "}"
    for (p in info.descriptors?.keySet())
        println "\t ${name} *-- ${info.descriptors[p]} : ${p}"
}
println "@enduml"

The generation (requires installed dot tools)

Dealing with 'Java Runtime SE 6' required error

Tue, 14 Oct 2014 11:22:48 +0800

When trying to start DBeaver (http://dbeaver.jkiss.org) I got this

I do have Java 7 installed:

➜  Contents  java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)
 
➜  Contents  which java
java is /Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java
java is /usr/bin/java

which should be a superset. But it is not.

Solution
#

Oracle seems to screw up the PLIST capabilities in JAva7 install

Gitbits - Get committer lists

Mon, 29 Sep 2014 11:22:48 +0800

With emails (Unix only)
#

➜  dev1 git:(miro-ui) ✗ git log --raw | grep "^Author: " | sort | uniq -c
 
      9 Author: Brendan Smith <Brendan.Smith@thinkwrap.com>
     22 Author: Kai Cheng <kai.cheng@thinkwrap.com>
     97 Author: Miro Adamy <miro.adamy@thinkwrap.com>
     49 Author: Training VM <training-vm@thinkwrap.com>
     24 Author: Your Name <your.name@thinkwrap.com>

Pure git
#

➜  dev1 git:(miro-ui) ✗ git shortlog -sen
  
    97  Miro Adamy <miro.adamy@thinkwrap.com>
    49  Training VM <training-vm@thinkwrap.com>
    24  Your Name <your.name@thinkwrap.com>
    22  Kai Cheng <kai.cheng@thinkwrap.com>
     9  Brendan Smith <Brendan.Smith@thinkwrap.com>

Mailmap option to consolidate the variety of authors
#

Before (no mailmap)
#

...
 609  milos hanzel <milos.hanzel@thinkwrap.com>
 361  brant maahs <brant.maahs@thinkwrap.com>
  69  Sullivan Tseng <sullivantseng@gmail.com>
  42  sullivan tseng <sullivan.tseng@thinkwrap.com>
  17  milos.hanzel <milos.hanzel@thinkwrap.com>
   2  Miro Adamy <miro.adamy@thinkwrap.com>
...

Generating the mailman start

How to access shared folder in Linux guest in VirtualBox

Tue, 23 Sep 2014 11:22:48 +0800

Make sure the VBOx extensions are installed.

Go here for details:

Add share
#

In VirtualBox, Device add transient shared folder:

This one has made available path /Users/Shared/ATG-11 under the name ATG-11

Mount it
#

sudo mount -t vboxsf ATG-11 /media
 
 
$ ll /media/Unix/
total 19M
drwxr-xr-x. 1 root  408 Sep 12 16:02 ATG11-1
drwxr-xr-x. 1 root  272 Sep 12 16:02 ATG11-docs
-rw-rw-r--. 1 root  675 May 13 23:46 atg-platform-mac.userlibraries
-rw-rw-r--. 1 root  787 May 13 23:17 atg-platform-vm.userlibraries
-rwxr--r--. 1 root  19M Feb 15  2013 jrebel-5.1.3-nosetup.zip
-rw-r--r--. 1 root 1020 May 15 15:36 jrebel.lic
-rw-r--r--. 1 root  582 May 15 14:58 rebel.xml
drwxr-xr-x. 1 root  272 Apr  6 01:33 security
drwxr-xr-x. 1 root  306 Jun  9 22:34 servers
-rwxr--r--. 1 root  349 Dec 17  2013 users.txt
drwxr-xr-x. 1 root  340 Sep  3 16:10 xrebel

logstash - known error

Tue, 16 Sep 2014 11:22:48 +0800

I have been trying to make this work for about 1.5 hr. Looks like there is open bug - https://logstash.jira.com/browse/LOGSTASH-703

Helpful link: https://groups.google.com/forum/#!topic/logstash-users/sZM03po7HJE

What should work:

grok {
   match => ["message", "regex to parse severity"],
   match => ["message", "regex to parse server IP"],
   match => ["message", "regex to parse user"]
}

What needs to be done instead

You _should_ be able to do exactly what you listed at the bottom of your email, 
except that you'd need `break_on_match => false` so that it would parse each snippet for each message 
instead of just parsing the first one that matches. 
Unfortunately, due to a bug (https://logstash.jira.com/browse/LOGSTASH-703), 
this doesn't work when you're matching against the same field in each match expression ("message" in your case). 
I was hoping to take a stab at fixing this bug (as several others have mentioned an interest in doing), 
but got distracted and haven't done it yet. It shouldn't be terribly hard to fix, just needs some time.
 
As a work-around, the following should work, but unfortunately the way that your tag_on_failure 
will end up working will be different 
because you have multiple Grok filters that could fail independently. 
It's probably slightly less efficient to do it this way because the event has to pass 
from one filter to the next through the pipeline, 
but my guess (based on absolutely no empirical data) is that it isn't significantly slower 
because the same work would need to be done by Regex either way, 

there's just more LogStash in the mix this way.
 
 
grok {
   match => ["message", "regex to parse severity"]
 
}
grok {
   match => ["message", "regex to parse server IP"]
}
grok {
   match => ["message", "regex to parse user"]
}
 
 
 
~Greg Mefford

Recreate txt base graph generator

Tue, 09 Sep 2014 11:22:48 +0800

Graphviz
#

This time using brew

➜  DPS  brew install graphviz
==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/graphviz-2.38.0.mavericks.bottle.tar.gz
######################################################################## 100.0%
==> Pouring graphviz-2.38.0.mavericks.bottle.tar.gz
🍺  /usr/local/Cellar/graphviz/2.38.0: 469 files, 68M
➜  DPS  dot -V
dot - graphviz version 2.38.0 (20140413.2041)
➜  DPS

Schema Spy
#

From http://sourceforge.net/projects/schemaspy/files/schemaspy/SchemaSpy%205.0.0/

To /opt/diagram

Run:

diagram java -jar ./schemaSpy_5.0.0.jar -cp .:/Users/miro/lib/jdbc/mysql-connector-java-5.1.26-bin.jar -t mysql -o library -host localhost -u training -db training_dev01 -p training

Better dynadmin

Thu, 04 Sep 2014 11:22:48 +0800

Source of the fix is https://github.com/jc7447/BetterDynAdmin. This is to capture the installation process and provide backup in case the tool disappears.

The pages generated by DynAdmin are actually not coming from a JSP or JHTML pages. The HTML you see is generated by hardcoded servlets. Each Nucleus component has assigned Admin servet, which is called when DynAdmin is trying to render the page.

The improvent here are user scripts - running inside Firefox or Chrome, utilizing infrastructure provided by Greasemonkey extension (FF) or Tampermonkey (Chrome).

Tutorial on Elastic

Thu, 28 Aug 2014 11:22:48 +0800

Running elastic
#

(see Elastic search and Kibana - getting started)

Use case: building a employee directory for Megacorp

each document - type of ’employee'
everything in megacorp index

Populating
#

# Delete the `megacorp` index in case it already exists
DELETE /megacorp
# Index document 1, type "employee", in the
# "megacorp" index
PUT /megacorp/employee/1
{
    "first_name" : "John",
    "last_name" :  "Smith",
    "age" :        25,
    "about" :      "I love to go rock climbing",
    "interests": [ "sports", "music" ]
}
# Index two more documents
PUT /megacorp/employee/2
{
    "first_name" :  "Jane",
    "last_name" :   "Smith",
    "age" :         32,
    "about" :       "I like to collect rock albums",
    "interests":  [ "music" ]
}
PUT /megacorp/employee/3
{
    "first_name" :  "Douglas",
    "last_name" :   "Fir",
    "age" :         35,
    "about":        "I like to build cabinets",
    "interests":  [ "forestry" ]
}

In server

Elastic search and Kibana - getting started

Wed, 27 Aug 2014 11:22:48 +0800

#Installation:

See

curl -L -O http://download.elasticsearch.org/PATH/TO/VERSION.zip
unzip elasticsearch-$VERSION.zip
cd  elasticsearch-$VERSION

Marvel is management plugin - console about cluster. Install:

./bin/plugin -i elasticsearch/marvel/latest
# disable data collection for local cluster

echo 'marvel.agent.enabled: false' >> ./config/elasticsearch.yml

Test
#

Startup log:

➜  elasticsearch-1.3.2  bin/elasticsearch
[2014-08-27 11:17:08,325][INFO ][node                     ] [Joe Fixit] version[1.3.2], pid[59610], build[dee175d/2014-08-13T14:29:30Z]
[2014-08-27 11:17:08,325][INFO ][node                     ] [Joe Fixit] initializing ...
[2014-08-27 11:17:08,338][INFO ][plugins                  ] [Joe Fixit] loaded [marvel], sites [marvel, kopf]
[2014-08-27 11:17:10,905][INFO ][marvel.agent             ] [Joe Fixit] collecting disabled by settings
[2014-08-27 11:17:11,020][INFO ][node                     ] [Joe Fixit] initialized
[2014-08-27 11:17:11,021][INFO ][node                     ] [Joe Fixit] starting ...
[2014-08-27 11:17:11,131][INFO ][transport                ] [Joe Fixit] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.179.121:9300]}
[2014-08-27 11:17:11,148][INFO ][discovery                ] [Joe Fixit] elasticsearch/nNgRXlHARHCPutTq7dZXWg
[2014-08-27 11:17:14,160][INFO ][cluster.service          ] [Joe Fixit] new_master [Joe Fixit][nNgRXlHARHCPutTq7dZXWg][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9300]], reason: zen-disco-join (elected_as_master)
[2014-08-27 11:17:14,181][INFO ][http                     ] [Joe Fixit] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.179.121:9200]}
[2014-08-27 11:17:14,181][INFO ][node                     ] [Joe Fixit] started
[2014-08-27 11:17:14,726][INFO ][gateway                  ] [Joe Fixit] recovered [4] indices into cluster_state
[2014-08-27 11:18:00,491][INFO ][cluster.service          ] [Joe Fixit] added {[logstash-Miros-MacBook-Pro-2.local-59618-4086][Eh5YjUFxSIWIn4xK45xu0w][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-Miros-MacBook-Pro-2.local-59618-4086][Eh5YjUFxSIWIn4xK45xu0w][Miros-MacBook-Pro-2.local][inet[/192.168.179.121:9301]]{client=true, data=false}])
Installed some pluggins - marvel, kopf ...

 

➜  ~  curl 'http://localhost:9200/?pretty'
{
  "status" : 200,
  "name" : "Sea Urchin",
  "version" : {
    "number" : "1.3.2",
    "build_hash" : "dee175dbe2f254f3f26992f5d7591939aaefd12f",
    "build_timestamp" : "2014-08-13T14:29:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.9"
  },
  "tagline" : "You Know, for Search"
}

The sense console allows to avoid using curl and do it from browser: - <http://localhost:9200/_plugin/marvel/sense/ >

Finding out the class in ATG installation

Tue, 26 Aug 2014 11:22:48 +0800

Java approach
#

Use the real time search with this JAR - https://jarscan.com/

➜  scripts git:(authoring) ✗ java -jar ~/bin/jarscan.jar
=========================
JarScan
written by Geoff Yaworski
gyaworski@hotmail.com
Version 2.0
=========================
Usage: java -jar jarscan.jar [-help | /?]
                    [-dir directory name]
                    [-zip]
                    [-showProgress]
                    <-files | -class | -package>
                    <search string 1> [search string 2]
                    [search string n]
Help:
  -help or /?           Displays this message.
  -dir                  The directory to start searching
                        from default is "."
  -zip                  Also search Zip files
  -showProgress         Show a running count of files read in
  -files or -class      Search for a file or Java class
                        contained in some library.
                        i.e. HttpServlet
  -package              Search for a Java package
                        contained in some library.
                        i.e. javax.servlet.http
  search string         The file or package to
                        search for.
                        i.e. see examples above

example

How to install Chrome in Centos 6.5

Fri, 04 Apr 2014 11:22:48 +0800

How to install Chrome in Centos 6.5
#

Step 1. Add Google Yum Repository
#

Create a new yum repository using below instructions.

sudo vim /etc/yum.repos.d/google.repo

Add following content to this file

[google]
name=Google
baseurl=http://dl.google.com/linux/rpm/stable/$basearch
enabled=1
gpgcheck=1
gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub

Step 2: Install/Update Google Chrome
#

Richard Lloid provided an shell script to grab libraries from a more recent Linux distro, put them in a tree (/opt/google/chrome/lib) exclusively picked up by Google Chrome and then you can indeed run Google Chrome on CentOS 6.4 or later. Read More

Gitolite permissions setup

Wed, 02 Apr 2014 11:22:48 +0800

This documents how to enforce the Gitolite permission to make LIVE branch writeable only by team leads

How gitolite works
#

Repository permission structure
#

This is general format of repo definition

REPO NAME
   rule line
   rule line

for example

@staff          =   dilbert alice wally bob
 
repo foo
    RW+         =   dilbert     # line 1
    RW+ dev     =   alice       # line 2
    -           =   wally       # line 3
    RW  temp/   =   @staff      # line 4
    R           =   ashok       # line 5

The Rule line has format:

Difference between matching and simple Git push

Tue, 31 Dec 2013 16:41:38 -0500

I started to see this message in some of my Git environments

Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:
 
git config --global push.default matching
 
 
To squelch this message and adopt the new behavior now, use:
 
 
git config --global push.default simple

This is collection of information gathered from Stackoverflow, Git docs and various internet sources. Putting it together so that I do not have to google it again.

Installing and running 'naked' hybris platform

Fri, 17 Aug 2012 11:22:48 +0800

Installing and running “naked” hybris platform
#

Log of an exercise to install and start the Hybris without all those fancy modules

Starting point: hybris-platform-4.7.1.zip

Install point: /opt/hybris-platform/hybris-platform-4.7.1

Start (with HSQLdb)
#

~ $ cd /opt/hybris-platform/hybris-platform-4.7.1
hybris-platform-4.7.1 $ cd bin/platform/
platform $ pwd
/opt/hybris-platform/hybris-platform-4.7.1/bin/platform
 
 
platform $ . ./setantenv.sh
Setting ant home to: /opt/hybris-platform/hybris-platform-4.7.1/bin/platform/apache-ant-1.8.2
Apache Ant(TM) version 1.8.2 compiled on April 21 2011
platform $ ant clean all | tee ant-clean-all.log

=> Select develop

Gitolite Administration Guide

Mon, 06 Aug 2012 11:22:48 +0800

How to setup new client
#

You need to be pensieve admin to do this. You will have to login as service user (non-host account) and su to root.

create new Linux user on pensieve
setup gitolite:
- git clone <git://github.com/sitaramc/gitolite>
- mkdir bin; export PATH=$PATH:~/bin
- gitolite/install -ln
add own public key to gitolite to get access to gilotile-admin repo (note the naming restrictions!!)
- gitolite setup -pk miro.pub
get public key for the client account admin and add it (there can be multiple admins)
clone the gitolite-admin repo from workstation whose pub key was added
- git clone <client@pensieve.thinkwrap>.[com:gitolite-admin.git](http://comgitolite-admin.git/)
- add the key to the repo
  - cd gitolite-admin
  - cp ~/mykey.pub keydir
  - vim conf/gitolite conf
  - add the newly added account admit to repos, create new repos if required

All gitolite installs are independent so modification in one hosting user have zero impact on others.

Using Gitolite for code sharing setup

Thu, 21 Jun 2012 11:22:48 +0800

General setup of code sharing host
#

The code sharing host (Pensieve) is dedicated Linux server in DMZ, accessible from external locations as well as from internal network (dual homed) - for the build server access.

Main features
#

multiple users
each user represent a client:
- e.g. users client1

Users are separated by Unix permissions:
#

access rights to /home/USER are 0700 - no access except the dedicated user
only ThinkWrap has root access or sudo access
the access via ssh for the user is blocked - the user client1 cannot log in using ssh and get shell, only use git to pull or push changes

Inside each user, there are multiple projects.
#

These projects will have multiple repositiories and will follow the naming convention:

Configuring permanent time updates for Linux

Thu, 03 Feb 2011 11:22:48 +0800

1) make sure the DNS works
#

sudo vi /etc/resolv.conf
cat /etc/resolv.conf

nameserver 208.67.222.222
nameserver 208.67.220.220
nameserver 192.168.16.1

2) check that you have ntpd installed
#

cat /etc/ntp.conf

3) make it work in all
#

/sbin/chkconfig --list | grep ntpd
sudo /sbin/chkconfig --level 2345 ntpd on

4) Initial sync
#

sudo /usr/sbin/ntpdate pool.ntp.org

Unit testing debt

Mon, 19 May 2008 23:44:17 -0400

I had the pleasure to listen very interesting podcast where Scott Hanselman and Quetzal Bradley discussed unit testing named "Testing after Unit Tests". To increase the Google Karma of the information discussed I have decided to rehash and sum up few main ideas of this discussion.

Quetzal introduced the idea of "negative coverage" - meaning that if you have 88 % test coverage, you should not see this as something positive you have achieved, but as something missing (you miss 12 % of completion) - thus the title of this post.

Devops on Miro's World

Catching up: Offsite Backups That Let Me Sleep

Picking the provider #

Catching up: Tailscale Everywhere

Why a mesh VPN #

How to monitor Bitbucket user and group permissions with API

How to manage users and groups in Bitbucket #

How to mirror Github and Bitbucket repositories

Mirroring large number of GitHub or Bitbucket repositories #

Assumptions #

Steps #

How to list Github repositories and users

How to list repositories in GitHub #

Setup #

How to list Bitbucket repositories and users

How to list repositories in Bitbucket #

Setup #

Documenting multi-repository projects

Reference Architecture as a Code

Many shapes of Architecture #

Accessing Terraform cloud Module registry programatically

Using SSH keys with multiple Bitbucket accounts

History of Faith

The beginnings #

Kubectl client and server version mismatch

Accessing Rancher cluster #

Rancher CLI vs kubectl

Accessing Rancher cluster #

On harvesting credit card numbers and passwords

Building Hugo as well on GitLab pages

Parallel building challenges #

Using same repo #

Current Hugo setup on Github Pages

How does the blog setup and publishing work #

From Jekyll to Hugo

Why Hugo replaced Jekyll #

All Day DevOps 2019

All Day DevOps 2019 - Notes #

AWS Inspection

Bunch of one-liners for AWS #

How many instances of each type do I have, and in what states? #

What CIDRs have Ingress Access to which Ports? #

Which Services am I using? #

CIDR access to ports #

Lambda runtimes #

Memory size #

Lambda Environment variables #

Gitbits

git-attic #

Example #

git trail #

AWS Certified Developer - Associate

Useful tool for Dockerhub

Skopeo #

Installation #

Usage #

AWS Certified Solution Architect - Associate

Blog of the day - Tapas Forever

Creating Kubernetes Cluster in AWS from scratch with kops

Pre-requisites #

Test of access #

Set the environment variables #

VisualVM inside Docker container

The idea / motivation #

Install AWS CLI on Mac

Install AWS CLI on Mac #

Unsuccesfull path - keep the Python 2.7 #

Storing sensitive information in Git

The issue #

Managing code in remote SVN repo

Options #

Use git-svn bridge #

Process for transferring Git repo from Gitolite to Bitbucket

Task #

User setup #

Update all local branches in Git repo

Update all local branches in Git repo #

git ffwd-update command #

Learning resources for Linux and Command line

Free resources #

Picking the provider
#

Why a mesh VPN
#

How to manage users and groups in Bitbucket
#

Mirroring large number of GitHub or Bitbucket repositories
#

Assumptions
#

Steps
#

How to list repositories in GitHub
#

Setup
#

How to list repositories in Bitbucket
#

Setup
#

Many shapes of Architecture
#

The beginnings
#

Accessing Rancher cluster
#

Accessing Rancher cluster
#

Parallel building challenges
#

Using same repo
#

How does the blog setup and publishing work
#

Why Hugo replaced Jekyll
#

All Day DevOps 2019 - Notes
#

Bunch of one-liners for AWS
#

How many instances of each type do I have, and in what states?
#

What CIDRs have Ingress Access to which Ports?
#

Which Services am I using?
#

CIDR access to ports
#

Lambda runtimes
#

Memory size
#

Lambda Environment variables
#

git-attic
#

Example
#

git trail
#

Skopeo
#

Installation
#

Usage
#

Pre-requisites
#

Test of access
#

Set the environment variables
#

The idea / motivation
#

Install AWS CLI on Mac
#

Unsuccesfull path - keep the Python 2.7
#

The issue
#

Options
#

Use git-svn bridge
#

Task
#

User setup
#

Update all local branches in Git repo
#

git ffwd-update command
#

Free resources
#

Hands On
#

Good news 1
#

Fix
#

Git project statistic tool
#

Installation
#

Example
#

After the reboot
#

Useful guide
#

Starting from dyn/admin
#

The general idea
#

How it works
#

How to get information about all ATG Nucleus components
#

Symptoms:
#

Validate this is the case
#

Solution
#

Solution
#

With emails (Unix only)
#

Pure git
#

Mailmap option to consolidate the variety of authors
#

Before (no mailmap)
#

Add share
#

Mount it
#

Graphviz
#

Schema Spy
#

Running elastic
#

Populating
#

Test
#

Java approach
#

How to install Chrome in Centos 6.5
#

Step 1. Add Google Yum Repository
#

Step 2: Install/Update Google Chrome
#

How gitolite works
#

Repository permission structure
#