Troubleshooting

Troubleshooting

Common issues and solutions when using politest.

AWS Credentials Issues

"no AWS credentials configured"

Problem: politest can't find AWS credentials.

Solution: Configure credentials using one of these methods:

# Method 1: Environment variables
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_REGION="us-east-1"

# Method 2: AWS CLI
aws configure

# Method 3: Verify credentials
aws sts get-caller-identity

"AccessDenied" when running tests

Problem: IAM principal lacks iam:SimulateCustomPolicy permission.

Solution: Add this policy to your IAM user/role:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "iam:SimulateCustomPolicy",
    "Resource": "*"
  }]
}

File Path Issues

"no such file or directory"

Problem: Policy or scenario file not found.

Solution: Use paths relative to the scenario file:

# If scenario is in: scenarios/test.yml
# And policy is in: policies/policy.json

# ✓ Correct
policy_json: "../policies/policy.json"

# ✗ Wrong
policy_json: "policies/policy.json"

"extends: parent not found"

Problem: Parent scenario file doesn't exist.

Solution: Check path is relative to current scenario:

# Current file: scenarios/dev/test.yml
# Parent file: scenarios/base.yml

# ✓ Correct
extends: "../base.yml"

# ✗ Wrong
extends: "base.yml"

Test Failures

Test expects "allowed" but got "implicitDeny"

Problem: Policy doesn't allow the action.

Debug steps:

1. Check policy syntax:

   cat policy.json | jq .

2. Verify action is in policy:

   {
     "Action": ["s3:GetObject", "s3:PutObject"],  // Is your action here?
     "Resource": "*"
   }

3. Check resource matches:

   # Policy resource: "arn:aws:s3:::my-bucket/*"
   # Test resource: "arn:aws:s3:::my-bucket/file.txt"  ✓ Matches
   # Test resource: "arn:aws:s3:::other-bucket/*"      ✗ Doesn't match

4. Save API response:

   politest --scenario test.yml --save /tmp/response.json
   cat /tmp/response.json | jq '.EvaluationResults'

Test expects "implicitDeny" but got "allowed"

Problem: Policy is too permissive.

Solution: Check for wildcard actions or resources:

{
  "Action": "s3:*",  // Allows ALL S3 actions
  "Resource": "*"     // On ALL resources
}

Test expects "explicitDeny" but got "allowed"

Problem: Deny statement not applying.

Common causes:

1. Deny statement after Allow (order doesn't matter in JSON, but check syntax)

2. Resource mismatch in Deny:

   {
     "Effect": "Deny",
     "Action": "s3:DeleteObject",
     "Resource": "arn:aws:s3:::prod-*"  // Doesn't match "dev-bucket"
   }

3. Condition not met:

   {
     "Effect": "Deny",
     "Action": "s3:*",
     "Condition": {
       "StringNotEquals": {"aws:PrincipalTag/Env": "prod"}
     }
   }

Template Variable Issues

Variables not rendering

Problem: {{.variable}} shows in output instead of value.

Solution:

1. For policies, use template files:

   policy_template: "policy.json.tpl"  # ✓ Variables render
   policy_json: "policy.json"           # ✗ Variables don't render

2. Check variable is defined:

   vars_file: "vars.yml"
   # OR
   vars:
     variable: "value"

3. Verify variable name:

   vars:
     bucket_name: "my-bucket"  # Define
   
   resource: "arn:aws:s3:::{{.bucket_name}}/*"  # ✓ Use
   resource: "arn:aws:s3:::{{.bucketname}}/*"   # ✗ Typo

"undefined variable" error

Problem: Template references variable that doesn't exist.

Solution: Add variable to vars or vars_file:

vars:
  missing_variable: "value"

SCP/RCP Issues

"Permissions Boundary Policy input list item 1 has invalid content"

Problem: SCP/RCP file has invalid JSON.

Solution:

1. Validate JSON:

   cat scp/policy.json | jq .

2. Check file exists:

   ls -la scp/*.json

3. Verify glob pattern:

   # Make sure files exist
   scp_paths:
     - "../scp/*.json"  # Expands to actual files

Context Condition Issues

Conditions not evaluating correctly

Problem: Context conditions don't work as expected.

Limitations:

• AWS SimulateCustomPolicy has limitations
• Complex conditions may not evaluate exactly as in production
• Use for general testing, not 100% production accuracy

Workaround: Test in real AWS environment for critical conditions.

"IpAddress type not supported"

Problem: Using ContextKeyType: "ipAddress".

Solution: AWS SDK v2 for Go doesn't support this type. Use string instead:

context:
  - ContextKeyName: "aws:SourceIp"
    ContextKeyType: "string"  # Not "ipAddress"
    ContextKeyValues: ["10.0.1.100"]

Resource Policy Issues

"caller_arn required when using resource_policy"

Problem: Specified resource policy without caller_arn.

Solution: Add caller_arn to simulate as a specific principal:

resource_policy_json: "bucket-policy.json"
caller_arn: "arn:aws:iam::123456789012:user/alice"  # Required

Debugging Tips

1. Use --save Flag

politest --scenario test.yml --save /tmp/debug.json

Inspect matched statements:

cat /tmp/debug.json | jq '.EvaluationResults[0].MatchedStatements'

2. Use --no-assert Flag

See all results without failing:

politest --scenario test.yml --no-assert

3. Simplify Scenario

Remove complexity to isolate issue:

# Minimal test
policy_json: "policy.json"
tests:
  - action: "s3:GetObject"
    resource: "*"
    expect: "allowed"

4. Check AWS API Directly

aws iam simulate-custom-policy \
  --policy-input-list file://policy.json \
  --action-names s3:GetObject \
  --resource-arns "arn:aws:s3:::bucket/*"

Getting Help

If you're still stuck:

1. Check examples: test/scenarios/
2. Review API Reference: API-Reference
3. Open an issue: GitHub Issues
4. Search discussions: GitHub Discussions

---

Still need help? Open an issue with:

• Your scenario file
• Error message
• Output from --save flag