SSH connectivity from Jenkins to EC2 without permanent SSH keys!

Jenkins pipelines at times need SSH connectivity to apps on EC2. Yet, permanent storage of certificate chains in PEM files on Jenkins is not recommended. Why?

  1. PEM key proliferation problem: enterprise solutions have many keys
  2. Makes it impossible to automate Jenkins pipelines between environments.

AWS EC2 instance connect to the rescue!

Components of our reference implementation:

  1. IaaC for an app that installs ec2-instance-connect agent
    sudo yum install ec2-instance-connect
  2. A Jenkins container that installs EC2 instance connect CLI
    pip install ec2instanceconnectcli
  3. IAM policy granting appropriate permission for EC2 instance connect. Here are some AWS examples.
  4. Policy attached to IAM role used as instance profile for Jenkins.
  5. Policy attached to IAM user to enable localhost developer testing.

To achieve 3-5, here’s the code:

#!/usr/bin/env bash

aws iam create-policy --policy-name ec2-instance-connect --policy-document file://ec2-instance-connect-SendSSHPublicKey.json

policy_arn=$(aws iam list-policies --query 'Policies[?PolicyName==`ec2-instance-connect`].Arn' --output text)

#4
aws iam attach-role-policy --policy-arn ${policy_arn} --role-name JenkinsInstanceProfile

#5
aws iam attach-user-policy --policy-arn ${policy_arn} --user-name localhost-test-ec2-instance-connect

AWS instance profile-based private Git repository access

Secure and private Git repository access enables teams. However, using an IAM user with SSH keys authenticating to private Git repository is not recommended. Why?

  1. Long lived key storage violates just-in-time access.
  2. IAM-user driven git SSH key management does not enable CI/CD.

IAM roles and git-remote-codecommit to the rescue!

Our AWS reference implementation:

  1. A private subnet with external connectivity, i.e., NAT’d subnet. No IGW.
  2. Two IAM roles with AWSCodeCommitReadOnly for CI/CD and AWSCodeCommitPowerUser for developers
  3. AWS CodeCommit repository
  4. Jenkins Docker image with git-remote-codecommit

CI/CD setup process:

  1. Configure the GRC HTTPS CodeCommit URLs in Jenkins Dockerfile. CodeCommit:
  2.  Provision AWS infrastructure for Jenkins with the instance profile or EKS role leveraging policy with AWSCodeCommitReadOnly
  3. Run Jenkins pipeline leveraging git clone.
  4. Done!
This process is repeatable for developer access via IAM role with AWSCodeCommitPowerUser policy.