SSH connectivity from Jenkins to EC2 without permanent SSH keys!

Jenkins pipelines at times need SSH connectivity to apps on EC2. Yet, permanent storage of certificate chains in PEM files on Jenkins is not recommended. Why?

  1. PEM key proliferation problem: enterprise solutions have many keys
  2. Makes it impossible to automate Jenkins pipelines between environments.

AWS EC2 instance connect to the rescue!

Components of our reference implementation:

  1. IaaC for an app that installs ec2-instance-connect agent
    sudo yum install ec2-instance-connect
  2. A Jenkins container that installs EC2 instance connect CLI
    pip install ec2instanceconnectcli
  3. IAM policy granting appropriate permission for EC2 instance connect. Here are some AWS examples.
  4. Policy attached to IAM role used as instance profile for Jenkins.
  5. Policy attached to IAM user to enable localhost developer testing.

To achieve 3-5, here’s the code:

#!/usr/bin/env bash

aws iam create-policy --policy-name ec2-instance-connect --policy-document file://ec2-instance-connect-SendSSHPublicKey.json

policy_arn=$(aws iam list-policies --query 'Policies[?PolicyName==`ec2-instance-connect`].Arn' --output text)

#4
aws iam attach-role-policy --policy-arn ${policy_arn} --role-name JenkinsInstanceProfile

#5
aws iam attach-user-policy --policy-arn ${policy_arn} --user-name localhost-test-ec2-instance-connect

AWS instance profile-based private Git repository access

Private Git repository access by CI/CD/CDE EC2s in secure subnets is an architectural best practice. However, using an IAM user to authenticate private Git repository access on EC2 is NOT a best practice. Why?

  1. Permanent key storage on EC2 is not secure.
  2. Revolving IAM user key pairs breaks git connectivity on EC2. Outcome: broken CI/CD!

IAM roles and instance profiles to the rescue!

Components of our reference implementation:

  1. A private subnet with external connectivity, i.e., NAT’d subnet. No IGW.
  2. An IAM role with AWSCodeCommitFullAccess
  3. AWS CodeCommit repository
  4. Jenkins Docker image with git-remote-codecommit setup

Launch steps:

  1. Create an EC2 instance in the private subnet with the instance profile.
  2. Run a Jenkins container on the EC2.
  3. Copy the HTTPS (GRC) URL from CodeCommit
  4. SSH to EC2 & Jenkins container and run the git clone on the HTTPS (GRC) URL to verify connectivity. Done!