Local-First AWS Testing with Kumo: A Practical CI/CD Strategy

Cloud-dependent integration tests are convenient in theory but fragile and costly in practice. Network flakiness, IAM throttling, and shared resource contention can make CI pipelines slow and nondeterministic. Kumo — a lightweight AWS service emulator written in Go — offers a local-first approach that replaces many cloud calls with a fast, predictable emulator suitable for CI and local development.

Why choose a local-first approach?

Moving integration tests off of live AWS for CI environments is about three practical goals:

• Reliability: Reduce test flakiness caused by network issues or cross-team resource interference.
• Speed: Emulators start fast and eliminate API latency, cutting CI times.
• Cost: Avoid repeated provisioning and cross-region request costs when running many pipeline runs.

Kumo is designed precisely for this: no authentication required (perfect for CI), a single binary, Docker support, lightweight startup, compatibility with AWS SDK v2 (especially relevant for Go teams), and optional data persistence (KUMO_DATA_DIR) so you can choose whether state should survive restarts.

High-level CI/CD strategy

Convert cloud-dependent integration tests into Kumo-driven local integration tests. The pipeline pattern looks like this:

1. Unit tests (fast, no external dependencies)
2. Start Kumo (ephemeral per job or persistent dev instance)
3. Run integration tests against Kumo using SDK endpoint overrides
4. Teardown Kumo or retain data for debugging (if enabled)
5. Optional: run a small smoke test against a staging AWS account for deployment verification

Benefits in practice

• Reduced flakiness: teams often see a drop in intermittent CI failures related to external APIs.
• Faster feedback: emulator-based integration tests can be 2–10× faster depending on network cost.
• Lower cloud spend: frequently-run tests that previously touched real AWS can be moved off billable resources.

Concrete pipeline configs

Below are examples for GitHub Actions and a Docker Compose setup for local development. These are intentionally pragmatic — copy, adapt, and run.

GitHub Actions: start Kumo in Docker, run Go tests

name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      kumo:
        image: sivchari/kumo:latest
        ports:
          - 4566:4566
        options: >-
          --health-cmd "curl -fs http://localhost:4566/health || exit 1"
          --health-interval 5s
          --health-timeout 2s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4

      - name: Set up Go
        uses: actions/setup-go@v4
        with:
          go-version: '1.20'

      - name: Run unit tests
        run: go test ./... -short

      - name: Run integration tests against Kumo
        env:
          AWS_ENDPOINT_URL: http://localhost:4566
          AWS_REGION: us-east-1
        run: |
          go test ./integration -v
  

Notes:

• Expose Kumo on port 4566 (popular convention used by local AWS emulators).
• Set environment variables used by your tests/SDK to point to the Kumo endpoint.

Docker Compose for local development (optional persistent data)

version: '3.8'
services:
  kumo:
    image: sivchari/kumo:latest
    ports:
      - '4566:4566'
    environment:
      - KUMO_DATA_DIR=/data
    volumes:
      - ./kumo-data:/data

Toggle KUMO_DATA_DIR to /data (or remove it) depending on whether you want state preserved between runs. For CI jobs that must be isolated, avoid persistence so each job starts with a clean slate.

Go SDK v2: test code that targets Kumo

When using AWS SDK for Go v2 you can use a custom endpoint resolver in a shared helper. This keeps your application code unchanged and wires the emulator into tests only.

package testutil

import (
  "context"
  "os"

  "github.com/aws/aws-sdk-go-v2/aws"
  "github.com/aws/aws-sdk-go-v2/config"
  "github.com/aws/aws-sdk-go-v2/service/s3"
)

// KumoEndpointResolver returns a resolver that points to KUMO (or default AWS)
func KumoEndpointResolver() aws.EndpointResolverWithOptionsFunc {
  endpoint := os.Getenv("AWS_ENDPOINT_URL")
  if endpoint == "" {
    return aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
      return aws.Endpoint{}, &aws.EndpointNotFoundError{}
    })
  }
  return aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
    return aws.Endpoint{URL: endpoint, SigningRegion: os.Getenv("AWS_REGION")}, nil
  })
}

func NewS3ClientForTests(ctx context.Context) (*s3.Client, error) {
  cfg, err := config.LoadDefaultConfig(ctx,
    config.WithEndpointResolverWithOptions(KumoEndpointResolver()),
    config.WithRegion(os.Getenv("AWS_REGION")),
  )
  if err != nil {
    return nil, err
  }
  return s3.NewFromConfig(cfg), nil
}

Test example that uses a unique bucket prefix per test to ensure isolation:

func TestUploadToS3(t *testing.T) {
  ctx := context.Background()
  client, err := testutil.NewS3ClientForTests(ctx)
  if err != nil { t.Fatal(err) }

  bucket := fmt.Sprintf("test-bucket-%d", time.Now().UnixNano())
  // create bucket, upload object, assert
}

Test isolation patterns

To keep tests deterministic when running against an emulator, follow these patterns:

• Ephemeral Kumo per job: start Kumo without KUMO_DATA_DIR so each CI job starts with clean state.
• Per-test resource naming: prefix buckets, tables, streams with test IDs or timestamps to avoid cross-test collisions.
• Setup / teardown helpers: create resources at test startup and delete them in a defer block.
• Scoped endpoints: ensure tests only talk to the local emulator by using environment variables and endpoint resolvers.
• Contract tests for external concerns: keep a small set of contract/verifier tests against real AWS in a less-frequent job (nightly or gated deploy) to validate emulator behavior.

Example: test helper for resource cleanup

func WithTemporaryBucket(t *testing.T, client *s3.Client, f func(bucket string)) {
  bucket := fmt.Sprintf("ci-temp-%d", time.Now().UnixNano())
  _, err := client.CreateBucket(context.Background(), &s3.CreateBucketInput{Bucket: &bucket})
  if err != nil { t.Fatalf("create bucket: %v", err) }
  defer func() {
    // empty and delete...
    _ = emptyAndDeleteBucket(context.Background(), client, bucket)
  }()
  f(bucket)
}

Pipelining and parallelization tips

To use Kumo effectively in CI, design pipelines so integration tests run in parallel when possible and only when necessary:

• Run unit tests first to fail fast.
• Parallelize integration jobs across feature areas that don't share persistent state.
• Use matrix builds to test multiple Go versions or dependency versions, sharing the same Kumo instance when state is not persisted (careful with isolation).
• Limit long-running, cloud-only tests to nightly pipelines or gated deployment steps to keep everyday CI fast and cheap.

Data persistence: when to enable it

Kumo supports optional data persistence via KUMO_DATA_DIR. This is useful for:

• Local development where you want a running state across restarts.
• Debug sessions when a failing test needs postmortem inspection.

For CI, prefer ephemeral instances (no persistence). If you must persist between steps (slow to recreate data), document and limit the scope to a small, controlled job to avoid test coupling.

Cost and time benefits — example estimates

Exact savings will vary, but teams commonly report:

• CI runtime reduction: moving integration tests to a local emulator can cut pipeline run time by 30–70% depending on the number of cloud calls and network latency.
• Cloud spend reduction: if integration tests previously provisioned or used billable services, eliminating frequent runs can save hundreds to thousands monthly for active teams.
• Reduced troubleshooting time: fewer nondeterministic failures mean less developer time diagnosing flakiness.

Real example: a team running 200 PRs/month that each ran integration tests against S3 and DynamoDB could save minutes per run and avoid per-call costs. Even small per-run savings compound quickly across many PRs.

When you should still use real AWS

Emulation is powerful, but there are cases that still require a real cloud environment:

• Provider-specific behavior or undocumented quirks.
• Managed services with side effects not modeled by an emulator.
• Performance and latency testing that must mirror real-world networks.

Make these tests part of an infrequent job (nightly or pre-release) to keep routine CI fast.

Operational checklist

1. Introduce Kumo into CI with a single job and run a subset of integration tests.
2. Adopt endpoint configuration helpers so application code stays untouched.
3. Switch CI to ephemeral Kumo instances for isolation; use KUMO_DATA_DIR only for local development.
4. Monitor flakiness and build times over a 30-day period and compare costs to prior AWS-based approach.
5. Keep a small set of contract tests against real AWS in a separate pipeline to catch emulator drift.

Related resources

For broader CI/CD patterns and container orchestration considerations, see our guide on building lightweight remote collaboration apps that covers container-friendly workflows and local-first development best practices: Build a Lightweight Remote Collaboration App. For analytics pipelines that might interact with emulated streaming services, our piece on ClickHouse and analytics systems is useful background: How ClickHouse’s Big Raise Changes the Analytics Landscape for Dev Teams.

Conclusion

Kumo makes it practical to run integration tests locally and in CI with low overhead, SDK compatibility, and configurable persistence. By running tests against a local emulator you can dramatically reduce flakiness, speed up pipelines, and lower cloud costs. Combine emulator-driven CI with periodic contract tests against real AWS and you get the best of both worlds: fast, reliable pipelines for everyday development and assurance that production behavior remains validated.