shariq.site

GitHub action with matrix (parallel) jobs for each of the item in a JSON file

I have a file

deploy-list.json:

{
    "apiList": [
      "abc",
      "def"    
    ]
}

I would like to run a GitHub action with matrix (parallel) jobs for each of the item in the apiList above.

name: API Build

on: 
  workflow_dispatch:
    inputs:
      branch:
        description: 'Github Release Branch Name'
        required: true
        default: 'release/1.5.3'

  push:
    branches:
    - 'develop'

jobs:

  build-matrix:
    runs-on: ubuntu-latest

    steps:

      - name: Checkout Branch
        uses: actions/checkout@v2
        with:          
          ref: "${{ github.event.inputs.branch }}"

      - name: Set Matrix
        id: set-matrix
        run: echo "::set-output name=api_matrix::$(cat deploy-list.json | jq -c '.apiList')"

    outputs:
      api_matrix: ${{ steps.set-matrix.outputs.api_matrix }}

  build-api:
    environment: PRD
    needs: build-matrix
    runs-on: ubuntu-latest
    strategy:
      matrix:
        api_name: ${{ fromJson(needs.build-matrix.outputs.api_matrix) }}
    steps:
      - name: Checkout Branch
        uses: actions/checkout@v2
        with:
          ref: "${{ github.event.inputs.branch }}"

      - name: Building ${{ matrix.api_name }}
        run: |
          echo ${{ matrix.api_name }}
          # Do something here

Two parallel jobs will be invoked for "abc" and "def" apis making the workflow complete in twice as fast build time. I also using maven caching to speed up the builds even further.

Preventing secret leaks

I have added secrets to git repositories more than once in my lifetime.Once I even did it in a public repo.

One way to avoid this is to use a tool to detect secrets in the source code. This will come in the category of SAST tooling.

$ brew install gitleaks
$ brew install pre-commit
$ cd /path/to/repo
$ curl https://raw.githubusercontent.com/giantswarm/apiextensions/master/.gitleaks.toml -o .gitleaks.toml
$ git add .gitleaks.toml
$ git commit .gitleaks.toml -m "Adding .gitleaks.toml"

// Run scan
$ gitleaks detect --config=.gitleaks.toml

Unable to delete a Kubernetes namespace?

Try this hack:

NAMESPACE=$1
kubectl get namespace $NAMESPACE -o json > $NAMESPACE.json
sed -i -e 's/"kubernetes"//' $NAMESPACE.json
kubectl replace --raw "/api/v1/namespaces/$NAMESPACE/finalize" -f ./$NAMESPACE.json

Simple JIRA integration

You can use cURL as a universal tool to update Jira Issues from your CI/CD tooling:

$ curl -X POST -u $JIRA_USER:$JIRA_API_TOKEN -H "Content-Type: application/json" https://myorg.atlassian.net/rest/api/latest/issue/SRE-754/comment --data '{"body": "Testing comment from REST API"}'

$ curl -X POST -u $JIRA_USERNAME:$JIRA_API_TOKEN -H "Content-Type: application/json" https://myorg.atlassian.net/rest/api/latest/issue/$JIRA_ISSUE_ID/comment --data "{\"type\":\"mention\",\"body\":\"Deployment $JOB_NAME completed, URL: $JOB_URL [~accountid:$MENTION1] [~accountid:$MENTION2] [~accountid:$MENTION3] \"}"

Convert an Epoch time to human readable time

epoch_to_human_time.sh

#!/bin/bash

TD=$(echo $1 | cut -c -10)
gdate -d @$TD

Codebuild project with Terraform

Quick and dirty Codebuild project with Terraform

resource "aws_s3_bucket" "example" {
  bucket = "shariqexampletestingterrastartup"
  acl    = "private"
  tags = {
    Name = "shariqexampletestingterrastartup"
  }
}

resource "aws_iam_role" "example" {
  name = "example"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codebuild.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "policy" {
  name        = "test-policy"
  description = "A test policy"
  policy      = <<EOF
{
"Version": "2012-10-17",
"Statement": [
  {
    "Sid": "CloudWatchLogsPolicy",
    "Effect": "Allow",
    "Action": [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ],
    "Resource": [
      "*"
    ]
  },
  {
    "Sid": "CodeCommitPolicy",
    "Effect": "Allow",
    "Action": [
      "codecommit:GitPull"
    ],
    "Resource": [
      "*"
    ]
  },
  {
    "Sid": "S3GetObjectPolicy",
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:GetObjectVersion"
    ],
    "Resource": [
      "*"
    ]
  },
  {
    "Sid": "S3PutObjectPolicy",
    "Effect": "Allow",
    "Action": [
      "s3:PutObject"
    ],
    "Resource": [
      "*"
    ]
  },
  {
    "Sid": "S3BucketIdentity",
    "Effect": "Allow",
    "Action": [
      "s3:GetBucketAcl",
      "s3:GetBucketLocation"
    ],
    "Resource": [
      "*"
    ]
  }
]
}
EOF
}

resource "aws_iam_role_policy_attachment" "test-attach" {
  role       = "${aws_iam_role.example.name}"
  policy_arn = "${aws_iam_policy.policy.arn}"
}

resource "aws_codebuild_project" "example" {
  name          = "terraform-cb-project" #var.DOMAIN_NAME
  description   = "A terrastartup codebuild project."
  build_timeout = "5"
  service_role  = "${aws_iam_role.example.arn}"

  artifacts {
    type = "CODEPIPELINE"
  }

  environment {
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/standard:1.0"
    type                        = "LINUX_CONTAINER"
    image_pull_credentials_type = "CODEBUILD"

  }

  logs_config {
    cloudwatch_logs {
      group_name  = "log-group"
      stream_name = "log-stream"
    }

    s3_logs {
      status   = "ENABLED"
      location = "${aws_s3_bucket.example.id}/build-log"
    }
  }

  source {
    type            = "CODEPIPELINE"
    git_clone_depth = 1
  }

  tags = {
    Environment = "Test"
  }
}

UTC time to Sydney time

In my job , I need to convert the UTC format from logs to local time a lot....

#!/bin/bash

if [ -z $1 ]; then
   echo "Please provide date in UTC to convert to Sydney Time.";
   date -u +'%Y-%m-%dT%H:%M:%S.000Z'
   exit 1
fi

export TZ=Australia/Sydney
gdate -d "$1"

Convert crt to pem

openssl x509 -in ZscalerRootCertificate-2048-SHA256.crt -out ZscalerRootCertificate-2048-SHA256.pem -outform PEM