[FEATURE] External Policy/Configuration Loading with `${include()}` Function

[jeffreyaven]

Feature Description

Implement a function-style syntax for loading policies, IAM roles, and other complex JSON/YAML configurations from external files in stackql-deploy, similar to Terraform's file loading capabilities.

Current Behavior

Currently, all policies, IAM role definitions, and other complex JSON configurations must be embedded directly within the stackql-deploy manifest file, making manifests verbose and difficult to maintain, especially for complex policy documents.

Desired Behavior

Allow users to reference external JSON/YAML files for policies and complex configurations using a function-style syntax:

- name: policy_document
  value: "${include('policies/s3_bucket_policy.json')}"

The ${include()} function would:

1. Load content from the specified JSON or YAML file
2. Parse the file content into a native object structure
3. Support variable substitution for template variables within the loaded file
4. Replace the function call with the loaded and processed content

File Format Requirements

• Support both JSON (.json) and YAML (.yaml, .yml) file formats
• Automatically detect format based on file extension
• Validate file syntax during loading
• Support nested includes (files including other files, with cycle detection)

Use Cases

• IAM role policies (like the cross_account_role in the manifest)
• S3 bucket policies
• Assume role policy documents
• Storage configurations
• Any complex nested JSON structure

Implementation Details

1. Implement a new include() function in the stackql-deploy template processing engine (at the resource.props level)
2. Add file path resolution relative to the manifest file location
3. Support environment-specific overrides (e.g., policy.dev.json vs policy.prod.json)
4. Process template variables inside the loaded files
5. Implement error handling for missing or invalid files

Example

Consider a complex IAM policy currently embedded in the manifest:

Current approach (policy embedded in manifest):

- name: aws/iam/cross_account_role
  file: aws/iam/iam_role.iql
  props:
    - name: assume_role_policy_document
      value:
        Version: "2012-10-17"
        Statement:
          - Sid: ""
            Effect: "Allow"
            Principal:
              AWS: "arn:aws:iam::{{ databricks_aws_account_id }}:root"
            Action: "sts:AssumeRole"
            Condition:
              StringEquals:
                sts:ExternalId: "{{ databricks_account_id }}"
    - name: policies
      value:
        - PolicyDocument:
            Statement:
              - Sid: Stmt1403287045000
                Effect: Allow
                Action:
                  - "ec2:AllocateAddress"
                  - "ec2:AssociateDhcpOptions"
                  # ... many more actions ...
                Resource:
                  - "*"
            Version: '2012-10-17'
          PolicyName: "{{ stack_name }}-{{ stack_env }}-policy"

Proposed approach with external files:

- name: aws/iam/cross_account_role
  file: aws/iam/iam_role.iql
  props:
    - name: assume_role_policy_document
      value: "${include('policies/assume_role_policy.json')}"
    - name: policies
      value:
        - PolicyDocument: "${include('policies/ec2_policy.json')}"
          PolicyName: "{{ stack_name }}-{{ stack_env }}-policy"

External file: policies/assume_role_policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{{ databricks_aws_account_id }}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "{{ databricks_account_id }}"
        }
      }
    }
  ]
}

External file: policies/ec2_policy.json:

{
  "Statement": [
    {
      "Sid": "Stmt1403287045000",
      "Effect": "Allow",
      "Action": [
        "ec2:AllocateAddress",
        "ec2:AssociateDhcpOptions",
        "ec2:AssociateIamInstanceProfile"
        // ... more actions ...
      ],
      "Resource": ["*"]
    }
  ],
  "Version": "2012-10-17"
}

Template Variable Processing

Template variables (using the {{ variable_name }} syntax) in external files would be processed just like they are in the main manifest file, allowing for dynamic configuration:

Example with template variables in external file:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::{{ databricks_aws_account_id }}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "{{ databricks_account_id }}"
        }
      }
    }
  ]
}

Benefits

• Cleaner, more maintainable manifest files
• Easier version control of individual policies
• Ability to reuse common policies across multiple stacks
• Better separation of concerns in the infrastructure code
• Familiar function-style syntax similar to other IaC tools

Additional Considerations

• Path resolution: Relative to manifest file or working directory?
• Error handling for missing or invalid files
• Support for partial includes (specific sections of a file)
• Documentation for best practices in organizing policy files