Multi-account AWS CodeCommit access

Our developers access internally owned git CodeCommit repos across multiple AWS accounts. These AWS accounts are owned by us. It is important to qualify AWS/CodeCommit ownership here. Client owned CodeCommit access best practice is through IAM roles (out of scope for this post).

Scope of this post:

  1. organizational CodeCommit access spanning two AWS accounts
  2. concurrent access to repos in both AWS accounts

Solution options:

  1. Access CodeCommit in both accounts using git-remote-codecommit
  2. Access CodeCommit in 1st account using SSH & the 2nd using git-remote-codecommit.

A “no-no-option” worth noting: accessing CodeCommit in both AWS accounts using SSH.

Our reference implementation addresses both viable solution options.

  1. Generate AWS access key and secret access key for an IAM entity having AWSCodeCommitFullAccess
  2. Create AWS cli profile(s)
    aws --profile account1-gov configure
    ...
    Default region name [None]: us-gov-west-1
    Default output format [None]: json
  3. Install git-remote-codecommit
    pip3 install git-remote-codecommit
  4. Clone repos. Enjoy!
    git clone codecommit://account1-gov@repo-name repo-name-gc

Important

Do not use GRC HTTP clone URL without profile (e.g., account1-gov) when concurrent access to account2 with SSH is in place.

SSH connectivity from Jenkins to EC2 without permanent SSH keys!

Jenkins pipelines at times need SSH connectivity to apps on EC2. Yet, permanent storage of certificate chains in PEM files on Jenkins is not recommended. Why?

  1. PEM key proliferation problem: enterprise solutions have many keys
  2. Makes it impossible to automate Jenkins pipelines between environments.

AWS EC2 instance connect to the rescue!

Components of our reference implementation:

  1. IaaC for an app that installs ec2-instance-connect agent
    sudo yum install ec2-instance-connect
  2. A Jenkins container that installs EC2 instance connect CLI
    pip install ec2instanceconnectcli
  3. IAM policy granting appropriate permission for EC2 instance connect. Here are some AWS examples.
  4. Policy attached to IAM role used as instance profile for Jenkins.
  5. Policy attached to IAM user to enable localhost developer testing.

To achieve 3-5, here’s the code:

#!/usr/bin/env bash

aws iam create-policy --policy-name ec2-instance-connect --policy-document file://ec2-instance-connect-SendSSHPublicKey.json

policy_arn=$(aws iam list-policies --query 'Policies[?PolicyName==`ec2-instance-connect`].Arn' --output text)

#4
aws iam attach-role-policy --policy-arn ${policy_arn} --role-name JenkinsInstanceProfile

#5
aws iam attach-user-policy --policy-arn ${policy_arn} --user-name localhost-test-ec2-instance-connect

AWS instance profile-based private Git repository access

Private Git repository access by CI/CD/CDE EC2s in secure subnets is an architectural best practice. However, using an IAM user to authenticate private Git repository access on EC2 is NOT a best practice. Why?

  1. Permanent key storage on EC2 is not secure.
  2. Revolving IAM user key pairs breaks git connectivity on EC2. Outcome: broken CI/CD!

IAM roles and instance profiles to the rescue!

Components of our reference implementation:

  1. A private subnet with external connectivity, i.e., NAT’d subnet. No IGW.
  2. An IAM role with AWSCodeCommitFullAccess
  3. AWS CodeCommit repository
  4. Jenkins Docker image with git-remote-codecommit setup

Launch steps:

  1. Create an EC2 instance in the private subnet with the instance profile.
  2. Run a Jenkins container on the EC2.
  3. Copy the HTTPS (GRC) URL from CodeCommit
  4. SSH to EC2 & Jenkins container and run the git clone on the HTTPS (GRC) URL to verify connectivity. Done!