A CI/CD pipeline should make shipping faster. But badly designed pipelines become the very bottleneck they were meant to eliminate. Here are the anti-patterns I see most often.

1. The Monolithic Pipeline

The problem: One massive pipeline that builds, tests, lints, scans, deploys, and makes coffee. If any step fails, you start from scratch.

1
2
3
4
5
6
7
8
9

# Anti-pattern: everything in sequence
stages:
  - build        # 5 min
  - unit-test    # 8 min
  - lint         # 2 min
  - security     # 4 min
  - integration  # 12 min
  - deploy       # 3 min
# Total: 34 minutes, no parallelism

The fix: Parallelize independent stages. Lint doesn’t need to wait for build. Security scanning can run alongside tests.

1
2
3
4
5
6
7

# Better: parallel stages
stages:
  - prepare
  - validate    # lint + security + unit tests in parallel
  - build
  - integration
  - deploy

2. Testing Everything Every Time

The problem: Your pipeline runs the entire test suite on every commit, even when you only changed a README.

The fix: Path-based triggers and test selection.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Only run backend tests when backend changes
backend-tests:
  rules:
    - changes:
        - "backend/**/*"
        - "shared/**/*"

# Skip CI entirely for docs-only changes
docs-only:
  rules:
    - changes:
        - "docs/**/*"
        - "*.md"
      when: never

3. The “Works on My Machine” Build

The problem: Builds depend on cached state or undeclared dependencies. They pass sometimes, fail mysteriously other times.

The fix: Hermetic builds. Everything the build needs should be explicitly declared.

1
2
3
4
5
6
7
8

# Bad: relies on base image having the right tools
FROM ubuntu:latest
RUN pip install -r requirements.txt

# Better: pin everything
FROM python:3.11.4-slim-bookworm
COPY requirements.lock ./
RUN pip install --no-cache-dir -r requirements.lock

Use lockfiles. Pin base images to digests, not tags. Treat “latest” as a bug.

4. Secrets Sprawl

The problem: Every pipeline has its own copy of production credentials. Nobody knows which ones are still in use.

1
2
3
4
5
6
7
8

# Anti-pattern: secrets everywhere
deploy-prod:
  env:
    AWS_ACCESS_KEY: $PROD_AWS_KEY
    DB_PASSWORD: $PROD_DB_PASS
    API_KEY: $PROD_API_KEY
    SLACK_WEBHOOK: $PROD_SLACK
    # 47 more secrets...

The fix: Centralize secrets. Use OIDC federation where possible — no static credentials at all.

1
2
3
4
5
6
7

# Better: assume role via OIDC, fetch secrets at runtime
deploy-prod:
  id_token:
    aud: https://gitlab.com
  script:
    - aws sts assume-role-with-web-identity ...
    - secrets=$(aws secretsmanager get-secret-value ...)

5. No Local Reproducibility

The problem: The only way to test the pipeline is to push a commit. Developers make 47 “fix CI” commits to debug a YAML typo.

The fix: Make pipelines runnable locally.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# GitLab CI
gitlab-runner exec docker build

# GitHub Actions
act -j build

# Generic: use Makefiles as the source of truth
make build   # same command locally and in CI
make test
make deploy

Your pipeline should call scripts, not contain logic. The pipeline is orchestration; the scripts are the work.

6. The Approval Gauntlet

The problem: Every deploy requires three manual approvals, a change ticket, and a blood sacrifice.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

deploy-prod:
  when: manual
  environment:
    name: production
  needs:
    - approval-security
    - approval-qa
    - approval-manager
    - approval-cto
    - approval-board-of-directors

The fix: Automate the gates. If security scan passes, that’s the security approval. If tests pass, that’s QA approval. Reserve manual gates for genuinely risky changes.

1
2
3
4
5
6

deploy-prod:
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: on_success  # automatic if all checks pass
    - if: $BREAKING_CHANGE == "true"
      when: manual      # only manual for breaking changes

7. Ignoring Pipeline Performance

The problem: Nobody tracks how long pipelines take. They slowly grow from 5 minutes to 45 minutes, and everyone just accepts it.

The fix: Treat pipeline time as a metric. Set budgets. Alert on regression.

1
2
3
4
5
6
7
8
9

# Track and enforce time limits
build:
  timeout: 10 minutes
  script:
    - time make build
    - |
      if [ $SECONDS -gt 300 ]; then
        echo "WARNING: Build exceeded 5 minute target"
      fi

Plot pipeline duration over time. When it creeps up, investigate.

8. Flaky Tests That Everyone Ignores

The problem: Tests fail randomly 10% of the time. The team’s solution is to just re-run the pipeline.

❌↻✅"SPRPhieiiptpperelyliiitnn!ee"fpaaislseedd

The fix: Quarantine flaky tests. Track them. Fix them. Don’t let them pollute your signal.

1
2
3
4
5
6
7
8
9

test:
  script:
    - pytest --ignore=tests/flaky/
  allow_failure: false

test-flaky:
  script:
    - pytest tests/flaky/ --reruns 3
  allow_failure: true  # tracked separately

A flaky test is worse than no test — it trains people to ignore failures.

9. No Caching Strategy

The problem: Every build downloads the entire internet. NPM, pip, Maven, Docker layers — all fresh, every time.

The fix: Cache aggressively. Use content-addressed caches where possible.

1
2
3
4
5
6
7
8
9

build:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  script:
    - npm ci --prefer-offline

For Docker, use BuildKit cache mounts:

1
2

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

10. Deploy First, Ask Questions Later

The problem: The pipeline deploys directly to production with no verification. You find out it’s broken when customers tell you.

The fix: Progressive delivery. Deploy to canary first. Run smoke tests. Watch metrics. Then proceed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

deploy-canary:
  script:
    - kubectl set image deployment/app app=new-version
    - kubectl rollout status deployment/app --timeout=60s
    - ./scripts/smoke-test.sh
    - sleep 300  # watch metrics
    
deploy-full:
  needs: [deploy-canary]
  when: manual  # human confirms canary looks good

The Meta-Lesson

Every anti-pattern here shares a root cause: treating the pipeline as an afterthought instead of a first-class product.

Your pipeline is infrastructure. It needs monitoring, maintenance, documentation, and ownership. It deserves the same engineering rigor as your application code.

If your pipeline is slow, flaky, or confusing — that’s a bug. Prioritize it like one.