Back to Articles
AdvancedAdvanced

Infrastructure as Code Testing

Test your infrastructure code with tools like Terratest and InSpec

12 min read
...
iacinfrastructureterraformtesting
Banner for Infrastructure as Code Testing

Introduction

Infrastructure as Code (IaC) has transformed how we manage infrastructure, but untested infrastructure code is as risky as untested application code. This guide teaches you how to test your Terraform, CloudFormation, and other IaC using proven testing tools and strategies.

Why Test Infrastructure Code?

The Problem:

  • Infrastructure bugs are expensive (downtime, security issues)
  • Manual testing doesn't scale
  • Configuration drift happens
  • Compliance requirements
  • Terraform/CloudFormation errors can destroy production resources

The Solution: Test infrastructure code like you test application code:

  • Unit tests (syntax, logic)
  • Integration tests (actual deployment)
  • Compliance tests (security, policies)
  • End-to-end tests (full stack)

Types of IaC Tests

1. Static Analysis

Validate syntax and best practices without deployment:

# Terraform validation
terraform fmt -check
terraform validate
 
# Terraform linting with tflint
tflint --init
tflint
 
# CloudFormation validation
aws cloudformation validate-template --template-body file://template.yaml
 
# Check for security issues
checkov -d .
tfsec .

2. Unit Tests

Test individual modules:

// Using Terratest
package test
 
import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)
 
func TestVPCModule(t *testing.T) {
    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "vpc_cidr": "10.0.0.0/16",
            "environment": "test",
        },
    })
    
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    
    // Validate outputs
    vpcId := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcId)
    
    cidr := terraform.Output(t, terraformOptions, "vpc_cidr")
    assert.Equal(t, "10.0.0.0/16", cidr)
}

3. Integration Tests

Test actual infrastructure deployment:

func TestWebServerDeployment(t *testing.T) {
    t.Parallel()
    
    terraformOptions := &terraform.Options{
        TerraformDir: "../examples/web-server",
        Vars: map[string]interface{}{
            "instance_type": "t2.micro",
            "ami_id": "ami-0c55b159cbfafe1f0",
        },
    }
    
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    
    // Get instance details
    instanceId := terraform.Output(t, terraformOptions, "instance_id")
    publicIp := terraform.Output(t, terraformOptions, "public_ip")
    
    // Verify instance is running
    aws.AssertEc2InstanceExists(t, instanceId)
    
    // Test HTTP endpoint
    url := fmt.Sprintf("http://%s:8080", publicIp)
    http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello World", 30, 5*time.Second)
}

4. Compliance Tests

Verify security and compliance:

# Using InSpec
describe aws_vpc('vpc-12345678') do
  it { should exist }
  its('cidr_block') { should eq '10.0.0.0/16' }
  its('state') { should eq 'available' }
end
 
describe aws_security_group('sg-12345678') do
  it { should exist }
  it { should_not allow_in(port: 22, ipv4_range: '0.0.0.0/0') } # No SSH from anywhere
  it { should allow_in(port: 443, ipv4_range: '0.0.0.0/0') } # HTTPS allowed
end
 
describe aws_s3_bucket('my-bucket') do
  it { should exist }
  it { should have_default_encryption_enabled }
  it { should_not be_public }
end

Testing Tools

1. Terratest (Go)

Comprehensive testing framework for Terraform:

Installation:

# In your Go test directory
go mod init github.com/yourorg/terraform-tests
go get github.com/gruntwork-io/terratest/modules/terraform
go get github.com/stretchr/testify/assert

Complete Example:

// test/vpc_test.go
package test
 
import (
    "fmt"
    "testing"
    "github.com/gruntwork-io/terratest/modules/aws"
    "github.com/gruntwork-io/terratest/modules/random"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)
 
func TestVPCCreation(t *testing.T) {
    t.Parallel()
    
    // Generate unique name
    uniqueId := random.UniqueId()
    vpcName := fmt.Sprintf("test-vpc-%s", uniqueId)
    
    awsRegion := "us-east-1"
    
    terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
        TerraformDir: "../modules/vpc",
        
        Vars: map[string]interface{}{
            "vpc_name": vpcName,
            "vpc_cidr": "10.0.0.0/16",
            "availability_zones": []string{"us-east-1a", "us-east-1b"},
            "private_subnets": []string{"10.0.1.0/24", "10.0.2.0/24"},
            "public_subnets": []string{"10.0.101.0/24", "10.0.102.0/24"},
        },
        
        EnvVars: map[string]string{
            "AWS_DEFAULT_REGION": awsRegion,
        },
    })
    
    // Cleanup
    defer terraform.Destroy(t, terraformOptions)
    
    // Deploy
    terraform.InitAndApply(t, terraformOptions)
    
    // Validate outputs
    vpcId := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcId)
    
    publicSubnetIds := terraform.OutputList(t, terraformOptions, "public_subnet_ids")
    assert.Equal(t, 2, len(publicSubnetIds))
    
    privateSubnetIds := terraform.OutputList(t, terraformOptions, "private_subnet_ids")
    assert.Equal(t, 2, len(privateSubnetIds))
    
    // Verify VPC exists in AWS
    vpc := aws.GetVpcById(t, vpcId, awsRegion)
    assert.Equal(t, "10.0.0.0/16", *vpc.CidrBlock)
    
    // Verify subnets
    for _, subnetId := range publicSubnetIds {
        subnet := aws.GetSubnetById(t, subnetId, awsRegion)
        assert.Equal(t, true, *subnet.MapPublicIpOnLaunch) // Public subnets auto-assign public IPs
    }
    
    for _, subnetId := range privateSubnetIds {
        subnet := aws.GetSubnetById(t, subnetId, awsRegion)
        assert.Equal(t, false, *subnet.MapPublicIpOnLaunch) // Private subnets don't
    }
}

Run tests:

cd test
go test -v -timeout 30m

2. kitchen-terraform (Ruby)

Test Terraform modules with Test Kitchen:

Installation:

gem install kitchen-terraform

Configuration:

# .kitchen.yml
---
driver:
  name: terraform
  
provisioner:
  name: terraform
 
verifier:
  name: terraform
  systems:
    - name: basic
      backend: aws
      controls:
        - vpc_creation
        - security_groups
 
platforms:
  - name: terraform
 
suites:
  - name: default
    driver:
      root_module_directory: test/fixtures
    verifier:
      systems:
        - name: basic
          backend: aws
          profile_locations:
            - test/integration/default

Test:

# test/integration/default/controls/vpc.rb
control 'vpc_creation' do
  impact 1.0
  title 'VPC Creation'
  desc 'Ensure VPC is created with correct configuration'
  
  describe aws_vpc(vpc_id: input('vpc_id')) do
    it { should exist }
    its('cidr_block') { should eq '10.0.0.0/16' }
    its('state') { should eq 'available' }
  end
end
 
control 'security_groups' do
  impact 1.0
  title 'Security Groups'
  desc 'Ensure security groups follow best practices'
  
  describe aws_security_group(group_id: input('web_sg_id')) do
    it { should exist }
    it { should_not allow_in(port: 22, ipv4_range: '0.0.0.0/0') }
    it { should allow_in(port: 443, ipv4_range: '0.0.0.0/0') }
  end
end

Run:

kitchen test

3. InSpec (Compliance Testing)

Chef InSpec for compliance and security testing:

Installation:

# Install InSpec
curl https://omnitruck.chef.io/install.sh | sudo bash -s -- -P inspec
 
# Or with gem
gem install inspec-bin

AWS Profile:

# profiles/aws-vpc/controls/vpc.rb
title 'VPC Security Controls'
 
aws_region = input('aws_region', value: 'us-east-1')
vpc_id = input('vpc_id')
 
control 'vpc-1' do
  impact 1.0
  title 'VPC Configuration'
  desc 'Ensure VPC is properly configured'
  
  describe aws_vpc(vpc_id) do
    it { should exist }
    it { should_not be_default }
    its('cidr_block') { should be_in ['10.0.0.0/16', '172.16.0.0/16'] }
  end
end
 
control 'vpc-2' do
  impact 1.0
  title 'VPC Flow Logs'
  desc 'Ensure VPC has flow logs enabled'
  
  describe aws_flow_log(vpc_id: vpc_id) do
    it { should exist }
    its('log_status') { should eq 'ACTIVE' }
  end
end
 
control 'sg-1' do
  impact 1.0
  title 'Security Group Rules'
  desc 'Ensure no security group allows SSH from anywhere'
  
  aws_security_groups.group_ids.each do |group_id|
    describe aws_security_group(group_id) do
      it { should_not allow_in(port: 22, ipv4_range: '0.0.0.0/0') }
      it { should_not allow_in(port: 3389, ipv4_range: '0.0.0.0/0') } # RDP
    end
  end
end
 
control 's3-1' do
  impact 1.0
  title 'S3 Bucket Security'
  desc 'Ensure S3 buckets are secure'
  
  aws_s3_buckets.bucket_names.each do |bucket|
    describe aws_s3_bucket(bucket) do
      it { should have_default_encryption_enabled }
      it { should_not be_public }
      it { should have_versioning_enabled }
    end
  end
end

Run InSpec:

# Execute profile
inspec exec profiles/aws-vpc \
  --input-file inputs.yml \
  --reporter cli json:output.json
 
# With inputs
inspec exec profiles/aws-vpc \
  --input aws_region=us-east-1 \
  --input vpc_id=vpc-12345678

4. Checkov (Static Analysis)

Security and compliance scanning:

# Install
pip install checkov
 
# Scan Terraform
checkov -d .
 
# Scan specific file
checkov -f main.tf
 
# Output JSON
checkov -d . -o json > checkov-results.json
 
# Skip specific checks
checkov -d . --skip-check CKV_AWS_20
 
# Only run specific checks
checkov -d . --check CKV_AWS_18,CKV_AWS_19

Custom Policy:

# custom-policies/s3-encryption.yaml
metadata:
  name: "Ensure S3 bucket has encryption enabled"
  id: "CUSTOM_AWS_1"
  category: "encryption"
  
definition:
  cond_type: "attribute"
  resource_types:
    - "aws_s3_bucket"
  attribute: "server_side_encryption_configuration"
  operator: "exists"

5. tfsec (Terraform Security)

Static analysis security scanner:

# Install
brew install tfsec
 
# Scan
tfsec .
 
# Output formats
tfsec . --format json
tfsec . --format junit
tfsec . --format sarif
 
# Severity filtering
tfsec . --minimum-severity HIGH
 
# Ignore specific issues
tfsec . --exclude aws-s3-enable-bucket-logging

Real-World Examples

Example 1: AWS VPC Module Test

Terraform Module:

# modules/vpc/main.tf
variable "vpc_cidr" {
  type = string
}
 
variable "environment" {
  type = string
}
 
resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true
  
  tags = {
    Name        = "${var.environment}-vpc"
    Environment = var.environment
  }
}
 
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
  
  tags = {
    Name = "${var.environment}-igw"
  }
}
 
output "vpc_id" {
  value = aws_vpc.main.id
}
 
output "vpc_cidr" {
  value = aws_vpc.main.cidr_block
}

Terratest:

func TestVPCModuleDefaults(t *testing.T) {
    t.Parallel()
    
    terraformOptions := &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "vpc_cidr": "10.0.0.0/16",
            "environment": "test",
        },
    }
    
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    
    vpcId := terraform.Output(t, terraformOptions, "vpc_id")
    
    // Verify VPC configuration
    vpc := aws.GetVpcById(t, vpcId, "us-east-1")
    assert.True(t, *vpc.EnableDnsHostnames)
    assert.True(t, *vpc.EnableDnsSupport)
    
    // Verify tags
    tags := aws.GetTagsForVpc(t, vpcId, "us-east-1")
    assert.Equal(t, "test-vpc", tags["Name"])
}

Example 2: Kubernetes Deployment Test

// test/k8s_deployment_test.go
package test
 
import (
    "testing"
    "github.com/gruntwork-io/terratest/modules/k8s"
    "github.com/gruntwork-io/terratest/modules/terraform"
    "github.com/stretchr/testify/assert"
)
 
func TestKubernetesDeployment(t *testing.T) {
    t.Parallel()
    
    terraformOptions := &terraform.Options{
        TerraformDir: "../examples/kubernetes",
    }
    
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)
    
    // Get kubeconfig
    kubeconfig := terraform.Output(t, terraformOptions, "kubeconfig")
    
    options := k8s.NewKubectlOptions("", kubeconfig, "default")
    
    // Verify deployment
    deployment := k8s.GetDeployment(t, options, "web-app")
    assert.Equal(t, int32(3), *deployment.Spec.Replicas)
    
    // Verify service
    service := k8s.GetService(t, options, "web-app")
    assert.Equal(t, "LoadBalancer", string(service.Spec.Type))
    
    // Wait for pods to be ready
    k8s.WaitUntilNumPodsCreated(t, options, 
        map[string]string{"app": "web-app"}, 3, 30, 10*time.Second)
    
    // Test HTTP endpoint
    serviceLB := k8s.GetServiceEndpoint(t, options, service, "http")
    http_helper.HttpGetWithRetry(t, serviceLB, nil, 200, "Hello", 30, 5*time.Second)
}

Example 3: Security Compliance Test

# profiles/aws-compliance/controls/security.rb
title 'AWS Security Compliance'
 
control 'encryption-at-rest' do
  impact 1.0
  title 'Encryption at Rest'
  desc 'All data stores must have encryption enabled'
  
  # RDS encryption
  aws_rds_instances.db_instance_identifiers.each do |db|
    describe aws_rds_instance(db) do
      it { should have_encrypted_storage }
    end
  end
  
  # EBS encryption
  aws_ebs_volumes.volume_ids.each do |volume|
    describe aws_ebs_volume(volume) do
      it { should be_encrypted }
    end
  end
  
  # S3 encryption
  aws_s3_buckets.bucket_names.each do |bucket|
    describe aws_s3_bucket(bucket) do
      it { should have_default_encryption_enabled }
    end
  end
end
 
control 'network-isolation' do
  impact 1.0
  title 'Network Isolation'
  desc 'Resources must be in private subnets where appropriate'
  
  # Database instances should not be public
  aws_rds_instances.db_instance_identifiers.each do |db|
    describe aws_rds_instance(db) do
      it { should_not be_publicly_accessible }
    end
  end
  
  # Private subnets should not have direct internet access
  aws_subnets.where { subnet_id.start_with?('subnet-private') }.subnet_ids.each do |subnet|
    describe aws_route_table(subnet_id: subnet) do
      it { should_not have_igw_attached }
    end
  end
end
 
control 'logging-monitoring' do
  impact 1.0
  title 'Logging and Monitoring'
  desc 'All resources must have appropriate logging'
  
  # CloudTrail enabled
  describe aws_cloudtrail_trails do
    its('trail_arns.count') { should be >= 1 }
  end
  
  # VPC Flow Logs
  aws_vpcs.vpc_ids.each do |vpc|
    describe aws_flow_log(vpc_id: vpc) do
      it { should exist }
    end
  end
  
  # S3 bucket logging
  aws_s3_buckets.bucket_names.each do |bucket|
    describe aws_s3_bucket(bucket) do
      it { should have_access_logging_enabled }
    end
  end
end

CI/CD Integration

GitHub Actions

name: Terraform Test
 
on:
  pull_request:
    paths:
      - 'terraform/**'
 
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        
      - name: Terraform Format Check
        run: terraform fmt -check -recursive
        
      - name: Terraform Validate
        run: |
          cd terraform
          terraform init -backend=false
          terraform validate
          
      - name: Run tfsec
        uses: aquasecurity/tfsec-action@v1.0.0
        
      - name: Run Checkov
        uses: bridgecrewio/checkov-action@master
        with:
          directory: terraform/
          soft_fail: false
  
  test:
    runs-on: ubuntu-latest
    needs: validate
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Go
        uses: actions/setup-go@v3
        with:
          go-version: 1.19
          
      - name: Run Terratest
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          cd test
          go test -v -timeout 30m
          
  compliance:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup InSpec
        run: |
          curl https://omnitruck.chef.io/install.sh | sudo bash -s -- -P inspec
          
      - name: Run InSpec Compliance Tests
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          inspec exec profiles/aws-compliance \
            --reporter cli json:inspec-results.json
            
      - name: Upload Results
        uses: actions/upload-artifact@v3
        with:
          name: compliance-results
          path: inspec-results.json

Jenkins Pipeline

pipeline {
  agent any
  
  stages {
    stage('Validate') {
      steps {
        sh 'terraform fmt -check'
        sh 'terraform validate'
        sh 'tfsec .'
        sh 'checkov -d .'
      }
    }
    
    stage('Test') {
      steps {
        dir('test') {
          sh 'go test -v -timeout 30m'
        }
      }
    }
    
    stage('Compliance') {
      steps {
        sh '''
          inspec exec profiles/aws-compliance \
            --reporter cli json:compliance.json
        '''
      }
    }
  }
  
  post {
    always {
      archiveArtifacts artifacts: '**/*.json', allowEmptyArchive: true
      junit 'test-results/**/*.xml'
    }
  }
}

Best Practices

1. Test Pyramid for IaC

        ┌────────────────┐
        │  E2E Tests     │  (Few, Expensive)
        │  Full Stack    │
        ├────────────────┤
        │ Integration    │  (Some, Moderate)
        │ Tests          │
        ├────────────────┤
        │ Unit Tests     │  (Many, Fast)
        │ Module Tests   │
        ├────────────────┤
        │ Static         │  (Most, Instant)
        │ Analysis       │
        └────────────────┘

2. Use Test Fixtures

test/
├── fixtures/
│   ├── vpc/          # Minimal VPC for testing
│   ├── database/     # Test database config
│   └── complete/     # Full stack example
├── integration/
│   ├── vpc_test.go
│   └── database_test.go
└── go.mod

3. Parallel Testing

func TestAll(t *testing.T) {
    tests := []struct {
        name string
        fn   func(*testing.T)
    }{
        {"VPC", TestVPC},
        {"Database", TestDatabase},
        {"LoadBalancer", TestLoadBalancer},
    }
    
    for _, tt := range tests {
        tt := tt // Capture range variable
        t.Run(tt.name, func(t *testing.T) {
            t.Parallel() // Run in parallel
            tt.fn(t)
        })
    }
}

4. Cost Management

// Use small instance types for testing
terraformOptions := &terraform.Options{
    Vars: map[string]interface{}{
        "instance_type": "t2.micro", // Smallest/cheapest
        "node_count": 1,              // Minimal nodes
    },
}
 
// Always destroy resources
defer terraform.Destroy(t, terraformOptions)

5. Idempotency Testing

func TestIdempotency(t *testing.T) {
    terraformOptions := &terraform.Options{
        TerraformDir: "../modules/vpc",
    }
    
    defer terraform.Destroy(t, terraformOptions)
    
    // Apply twice
    terraform.InitAndApply(t, terraformOptions)
    output1 := terraform.Output(t, terraformOptions, "vpc_id")
    
    terraform.Apply(t, terraformOptions)
    output2 := terraform.Output(t, terraformOptions, "vpc_id")
    
    // Should be the same (no changes)
    assert.Equal(t, output1, output2)
}

Troubleshooting

Common Issues

  1. Cleanup Failures
// Add retries for cleanup
defer func() {
    retryTerraformDestroy(t, terraformOptions, 3)
}()
 
func retryTerraformDestroy(t *testing.T, options *terraform.Options, maxRetries int) {
    for i := 0; i < maxRetries; i++ {
        err := terraform.DestroyE(t, options)
        if err == nil {
            return
        }
        time.Sleep(10 * time.Second)
    }
}
  1. Rate Limiting
// Add delays between tests
func TestWithRateLimit(t *testing.T) {
    time.Sleep(5 * time.Second) // Avoid API rate limits
    // ... test code
}
  1. Resource Dependencies
// Wait for resources to be ready
aws.WaitForInstanceState(t, instanceId, "running", 30, 10*time.Second)

Next Steps

  1. Start with static analysis - tfsec, checkov (fastest wins)
  2. Add unit tests for Terraform modules
  3. Implement integration tests for critical infrastructure
  4. Set up compliance scanning with InSpec
  5. Integrate into CI/CD pipeline
  6. Create test fixtures for reusable test scenarios
  7. Document testing strategy for your team
  • "CI/CD Pipeline Testing" - Automate IaC tests
  • "Security Testing" - Advanced security scanning
  • "Monitoring & Observability" - Monitor deployed infrastructure
  • "Chaos Engineering" - Test infrastructure resilience
  • "DevOps Best Practices" - IaC in broader context

Conclusion

Testing Infrastructure as Code prevents expensive mistakes and ensures compliance. By combining static analysis, unit tests, integration tests, and compliance scanning, you can:

  • Catch errors before deployment
  • Enforce security policies
  • Maintain compliance
  • Enable confident infrastructure changes
  • Sleep better at night

Start with static analysis for quick wins, then gradually add integration and compliance tests. Your infrastructure deserves the same testing rigor as your application code!

Remember: The cloud makes it easy to create resources—and easy to create expensive mistakes. Test your infrastructure code!

Comments (0)

Loading comments...