Back to Blog••Automation

Repository Template System & GitHub Automation

Complete automation of GitHub repository creation from templates, including CI/CD workflow injection, branch protection setup, and parameterized pipeline deployment for client onboarding.

R
Rov
•18 min read
GitHubAutomationCI/CDDevOpsPowerShellTemplates

Repository Template System & GitHub Automation

Automated GitHub Repository Creation
This post covers the complete automation of GitHub repository creation from templates, including CI/CD workflow injection, branch protection setup, and parameterized pipeline deployment.

Overview

Step 2 of our automated client onboarding system focuses on creating standardised GitHub repositories from templates. This ensures every client gets a consistent, production-ready codebase with pre-configured CI/CD workflows, security policies, and deployment automation.

Step 2: Bootstrapping GitHub Repositories

The repository automation system creates a complete development environment for each client using GitHub's template repository feature combined with API automation.

Template Repository Structure

First, we maintain a master template repository with the standardised structure:

client-template-repository/
├── .github/
│   ├── workflows/
│   │   ├── adf-ci-cd.yml
│   │   ├── powerbi-deployment.yml
│   │   └── fabric-sync.yml
│   ├── ISSUE_TEMPLATE/
│   └── dependabot.yml
├── adf/
│   ├── pipeline/
│   │   └── client-data-ingestion.json
│   ├── dataset/
│   │   ├── http-source.json
│   │   └── onelake-sink.json
│   ├── linkedService/
│   │   ├── client-api.json
│   │   ├── keyvault.json
│   │   └── fabric-lakehouse.json
│   ├── trigger/
│   │   └── monthly-schedule.json
│   └── ARMTemplateForFactory.json
├── infrastructure/
│   ├── bicep/
│   │   ├── main.bicep
│   │   ├── keyvault.bicep
│   │   └── datafactory.bicep
│   └── parameters/
│       ├── dev.parameters.json
│       ├── test.parameters.json
│       └── prod.parameters.json
├── powerbi/
│   ├── reports/
│   │   └── client-dashboard-template.pbix
│   ├── datasets/
│   └── deployment/
├── scripts/
│   ├── pre-deployment.ps1
│   ├── post-deployment.ps1
│   └── utilities/
├── tests/
│   ├── unit/
│   ├── integration/
│   └── data-quality/
└── docs/
    ├── deployment-guide.md
    └── client-requirements.md

Automated Repository Creation

The PowerShell automation uses GitHub's REST API to create repositories from templates:

function New-ClientRepository {
    param(
        [Parameter(Mandatory = $true)]
        [string]$ClientName,

        [Parameter(Mandatory = $true)]
        [string]$GitHubOrg,

        [Parameter(Mandatory = $true)]
        [string]$GitHubToken,

        [Parameter(Mandatory = $true)]
        [string]$TemplateRepo = "client-data-platform-template"
    )

    Write-Host "Creating GitHub repository for client: $ClientName" -ForegroundColor Green

    # Set up headers for GitHub API
    $headers = @{
        "Authorization" = "Bearer $GitHubToken"
        "Accept" = "application/vnd.github.v3+json"
        "User-Agent" = "PowerShell-ClientOnboarding"
    }

    # Repository creation payload
    $newRepoName = "$ClientName-data-platform".ToLower()
    $payload = @{
        name = $newRepoName
        description = "Automated data platform for $ClientName"
        private = $true
        include_all_branches = $false
    } | ConvertTo-Json

    try {
        # Create repository from template
        $createUri = "https://api.github.com/repos/$GitHubOrg/$TemplateRepo/generate"
        $response = Invoke-RestMethod -Uri $createUri -Method POST -Headers $headers -Body $payload

        Write-Host "  Repository created: $($response.html_url)" -ForegroundColor Green
        return $response
    }
    catch {
        Write-Error "Failed to create repository: $($_.Exception.Message)"
        throw
    }
}

Parameter Injection and Customisation

After repository creation, we inject client-specific parameters into all configuration files:

function Update-ClientParameters {
    param(
        [Parameter(Mandatory = $true)]
        [string]$RepoPath,

        [Parameter(Mandatory = $true)]
        [hashtable]$ClientConfig
    )

    Write-Host "Injecting client parameters into repository files" -ForegroundColor Yellow

    # Define parameter mappings
    $parameterMappings = @{
        "{{CLIENT_NAME}}" = $ClientConfig.ClientName
        "{{CLIENT_API_ENDPOINT}}" = $ClientConfig.ApiEndpoint
        "{{AZURE_SUBSCRIPTION_ID}}" = $ClientConfig.SubscriptionId
        "{{RESOURCE_GROUP_NAME}}" = $ClientConfig.ResourceGroupName
        "{{DATA_FACTORY_NAME}}" = $ClientConfig.DataFactoryName
        "{{KEY_VAULT_NAME}}" = $ClientConfig.KeyVaultName
        "{{FABRIC_WORKSPACE_DEV}}" = "$($ClientConfig.ClientName)-Dev"
        "{{FABRIC_WORKSPACE_TEST}}" = "$($ClientConfig.ClientName)-Test"
        "{{FABRIC_WORKSPACE_PROD}}" = "$($ClientConfig.ClientName)-Prod"
    }

    # File patterns to update
    $filePatterns = @(
        "*.json",
        "*.yml",
        "*.yaml",
        "*.bicep",
        "*.ps1",
        "*.md"
    )

    foreach ($pattern in $filePatterns) {
        $files = Get-ChildItem -Path $RepoPath -Filter $pattern -Recurse

        foreach ($file in $files) {
            $content = Get-Content -Path $file.FullName -Raw

            foreach ($placeholder in $parameterMappings.Keys) {
                $content = $content -replace [regex]::Escape($placeholder), $parameterMappings[$placeholder]
            }

            Set-Content -Path $file.FullName -Value $content -NoNewline
            Write-Host "    Updated: $($file.Name)" -ForegroundColor Gray
        }
    }
}

CI/CD Workflow Configuration

Azure Data Factory CI/CD Pipeline

The template includes a comprehensive ADF deployment workflow:

name: Azure Data Factory CI/CD

on:
  push:
    branches: [ main, develop ]
    paths: [ 'adf/**' ]
  pull_request:
    branches: [ main ]
    paths: [ 'adf/**' ]

env:
  AZURE_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
  RESOURCE_GROUP_NAME: ${{ secrets.RESOURCE_GROUP_NAME }}
  DATA_FACTORY_NAME: ${{ secrets.DATA_FACTORY_NAME }}

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Azure Login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
      
      - name: Validate ADF Resources
        run: |
          az datafactory pipeline list \
            --factory-name $DATA_FACTORY_NAME \
            --resource-group $RESOURCE_GROUP_NAME
      
      - name: Run Data Quality Tests
        run: |
          python tests/data-quality/validate_schemas.py
          python tests/data-quality/check_data_lineage.py

  deploy-dev:
    needs: validate
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/develop'
    environment: development
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Deploy to Development
        run: |
          az deployment group create \
            --resource-group $RESOURCE_GROUP_NAME \
            --template-file infrastructure/bicep/main.bicep \
            --parameters @infrastructure/parameters/dev.parameters.json \
            --parameters clientName=${{ github.event.repository.name }}

  deploy-prod:
    needs: validate
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    environment: production
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Deploy to Production
        run: |
          az deployment group create \
            --resource-group $RESOURCE_GROUP_NAME \
            --template-file infrastructure/bicep/main.bicep \
            --parameters @infrastructure/parameters/prod.parameters.json \
            --parameters clientName=${{ github.event.repository.name }}

Power BI Deployment Pipeline

Automated Power BI report deployment and workspace management:

name: Power BI Deployment

on:
  push:
    branches: [ main ]
    paths: [ 'powerbi/**' ]

jobs:
  deploy-powerbi:
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Install Power BI PowerShell
        shell: powershell
        run: |
          Install-Module -Name MicrosoftPowerBIMgmt -Force -Scope CurrentUser
      
      - name: Connect to Power BI
        shell: powershell
        run: |
          $securePassword = ConvertTo-SecureString ${{ secrets.POWERBI_PASSWORD }} -AsPlainText -Force
          $credential = New-Object System.Management.Automation.PSCredential(${{ secrets.POWERBI_USERNAME }}, $securePassword)
          Connect-PowerBIServiceAccount -Credential $credential
      
      - name: Deploy Reports
        shell: powershell
        run: |
          $workspaceName = "${{ github.event.repository.name }}-Prod"
          $workspace = Get-PowerBIWorkspace -Name $workspaceName
          
          Get-ChildItem -Path "powerbi/reports/*.pbix" | ForEach-Object {
              New-PowerBIReport -Path $_.FullName -WorkspaceId $workspace.Id -Name $_.BaseName
          }

Branch Protection and Security

Automated Branch Protection Setup

function Set-BranchProtection {
    param(
        [Parameter(Mandatory = $true)]
        [string]$RepoFullName,

        [Parameter(Mandatory = $true)]
        [string]$GitHubToken
    )

    $headers = @{
        "Authorization" = "Bearer $GitHubToken"
        "Accept" = "application/vnd.github.v3+json"
    }

    # Main branch protection
    $mainProtection = @{
        required_status_checks = @{
            strict = $true
            contexts = @("validate", "security-scan")
        }
        enforce_admins = $true
        required_pull_request_reviews = @{
            required_approving_review_count = 2
            dismiss_stale_reviews = $true
            require_code_owner_reviews = $true
        }
        restrictions = $null
        allow_force_pushes = $false
        allow_deletions = $false
    } | ConvertTo-Json -Depth 10

    $uri = "https://api.github.com/repos/$RepoFullName/branches/main/protection"
    Invoke-RestMethod -Uri $uri -Method PUT -Headers $headers -Body $mainProtection

    Write-Host "  Branch protection configured for main branch" -ForegroundColor Green
}

Security Scanning Integration

name: Security Scan

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'
      
      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'
      
      - name: Secret Scanning
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: main
          head: HEAD

Repository Secrets Management

Automated Secrets Injection

function Set-RepositorySecrets {
    param(
        [Parameter(Mandatory = $true)]
        [string]$RepoFullName,

        [Parameter(Mandatory = $true)]
        [string]$GitHubToken,

        [Parameter(Mandatory = $true)]
        [hashtable]$ClientSecrets
    )

    $headers = @{
        "Authorization" = "Bearer $GitHubToken"
        "Accept" = "application/vnd.github.v3+json"
    }

    # Get repository public key for secret encryption
    $keyUri = "https://api.github.com/repos/$RepoFullName/actions/secrets/public-key"
    $publicKey = Invoke-RestMethod -Uri $keyUri -Headers $headers

    foreach ($secretName in $ClientSecrets.Keys) {
        $encryptedValue = Protect-GitHubSecret -PlainText $ClientSecrets[$secretName] -PublicKey $publicKey.key

        $secretPayload = @{
            encrypted_value = $encryptedValue
            key_id = $publicKey.key_id
        } | ConvertTo-Json

        $secretUri = "https://api.github.com/repos/$RepoFullName/actions/secrets/$secretName"
        Invoke-RestMethod -Uri $secretUri -Method PUT -Headers $headers -Body $secretPayload

        Write-Host "    Secret configured: $secretName" -ForegroundColor Gray
    }
}

function Protect-GitHubSecret {
    param(
        [string]$PlainText,
        [string]$PublicKey
    )

    # Implement libsodium sealed box encryption
    # This is a simplified example - use proper encryption library
    Add-Type -AssemblyName System.Security
    $bytes = [System.Text.Encoding]::UTF8.GetBytes($PlainText)
    $publicKeyBytes = [Convert]::FromBase64String($PublicKey)
    
    # Use libsodium or similar for actual encryption
    # Return base64 encoded encrypted value
    return [Convert]::ToBase64String($bytes)
}

Complete Repository Setup Function

function Initialize-ClientRepository {
    param(
        [Parameter(Mandatory = $true)]
        [string]$ClientName,

        [Parameter(Mandatory = $true)]
        [string]$GitHubOrg,

        [Parameter(Mandatory = $true)]
        [string]$GitHubToken,

        [Parameter(Mandatory = $true)]
        [hashtable]$ClientConfig
    )

    Write-Host "Initializing GitHub repository for $ClientName" -ForegroundColor Cyan

    try {
        # Step 1: Create repository from template
        $repo = New-ClientRepository -ClientName $ClientName -GitHubOrg $GitHubOrg -GitHubToken $GitHubToken

        # Step 2: Clone repository locally for modifications
        $tempPath = Join-Path $env:TEMP "client-repos"
        $repoPath = Join-Path $tempPath $repo.name
        
        if (Test-Path $repoPath) { Remove-Item $repoPath -Recurse -Force }
        New-Item -Path $tempPath -ItemType Directory -Force | Out-Null
        
        git clone $repo.clone_url $repoPath
        Set-Location $repoPath

        # Step 3: Update parameters in all files
        Update-ClientParameters -RepoPath $repoPath -ClientConfig $ClientConfig

        # Step 4: Commit and push changes
        git add .
        git commit -m "Initial client configuration for $ClientName"
        git push origin main

        # Step 5: Set up branch protection
        Set-BranchProtection -RepoFullName $repo.full_name -GitHubToken $GitHubToken

        # Step 6: Configure repository secrets
        $secrets = @{
            "AZURE_SUBSCRIPTION_ID" = $ClientConfig.SubscriptionId
            "AZURE_CREDENTIALS" = $ClientConfig.AzureCredentials
            "RESOURCE_GROUP_NAME" = $ClientConfig.ResourceGroupName
            "DATA_FACTORY_NAME" = $ClientConfig.DataFactoryName
            "KEY_VAULT_NAME" = $ClientConfig.KeyVaultName
        }
        Set-RepositorySecrets -RepoFullName $repo.full_name -GitHubToken $GitHubToken -ClientSecrets $secrets

        Write-Host "Repository setup completed successfully!" -ForegroundColor Green
        Write-Host "Repository URL: $($repo.html_url)" -ForegroundColor Yellow

        return @{
            Repository = $repo
            LocalPath = $repoPath
            Success = $true
        }
    }
    catch {
        Write-Error "Repository setup failed: $($_.Exception.Message)"
        return @{
            Repository = $null
            LocalPath = $null
            Success = $false
            Error = $_.Exception.Message
        }
    }
}

Integration with Client Onboarding

This repository automation integrates seamlessly with the broader client onboarding process:

  1. Infrastructure provisioning creates the Azure resources
  2. Repository creation (this step) sets up the development environment
  3. Pipeline deployment configures the data ingestion workflows
  4. Fabric workspace setup creates the analytics environment

The result is a complete, production-ready data platform that's ready for immediate use by the client's development team.

Benefits and Outcomes

Consistency and Standardisation

  • Uniform codebase structure across all clients
  • Standardised CI/CD practices and security policies
  • Consistent naming conventions and documentation

Reduced Time to Value

  • Immediate development readiness - no setup time required
  • Pre-configured security scanning and compliance checks
  • Automated deployment pipelines from day one

Scalability and Maintenance

  • Template-based approach allows easy updates across all clients
  • Centralised security policies can be updated globally
  • Automated dependency management through Dependabot

This automation transforms what used to be a 2-3 day manual setup process into a 5-minute automated deployment, while ensuring higher quality and consistency than manual processes could achieve.

Back to Blog
DocumentationCreative Writing