From 4e4e844721830d543fddbcff54b59136ec28a4c6 Mon Sep 17 00:00:00 2001 From: Oli Passey Date: Fri, 27 Jun 2025 17:40:21 +0100 Subject: [PATCH] pipeline updates --- AZURE-DEVOPS-SETUP.md | 120 +++++++++++++++++++++++++---------- README.md | 45 +++++++++++++ azure-pipelines-template.yml | 98 ++++++++++++++++++++++++++++ azure-pipelines.yml | 11 ++++ 4 files changed, 240 insertions(+), 34 deletions(-) create mode 100644 azure-pipelines-template.yml diff --git a/AZURE-DEVOPS-SETUP.md b/AZURE-DEVOPS-SETUP.md index ba1c916..7056232 100644 --- a/AZURE-DEVOPS-SETUP.md +++ b/AZURE-DEVOPS-SETUP.md @@ -4,50 +4,71 @@ This guide will help you set up the Azure DevOps pipeline to automatically build ## Prerequisites -1. Azure DevOps project with your Price Tracker repository -2. Access to your Docker registry at `dock.ptslondon.co.uk` -3. Admin permissions in Azure DevOps to create service connections +1. Azure DevOps project +2. GitHub repository with your Price Tracker code +3. Access to your Docker registry at `dock.ptslondon.co.uk` +4. Admin permissions in Azure DevOps to create service connections -## Step 1: Create Docker Registry Service Connection +## Step 1: Create GitHub Service Connection 1. In your Azure DevOps project, go to **Project Settings** (bottom left) 2. Under **Pipelines**, click **Service connections** 3. Click **New service connection** -4. Select **Docker Registry** and click **Next** -5. Choose **Others** for registry type -6. Fill in the details: +4. Select **GitHub** and click **Next** +5. Choose **Azure Pipelines** app: + - Click **Authorize using OAuth** + - Sign in to GitHub and authorize Azure Pipelines + - OR use **Personal Access Token** if you prefer: + - Create a GitHub PAT with `repo` scope + - Enter your GitHub username and PAT +6. **Service connection name**: `github` (this must match the pipeline) +7. Check **Grant access permission to all pipelines** +8. Click **Save** + +## Step 2: Create Docker Registry Service Connection + +1. In **Service connections**, click **New service connection** +2. Select **Docker Registry** and click **Next** +3. Choose **Others** for registry type +4. Fill in the details: - **Docker Registry**: `https://dock.ptslondon.co.uk` - **Docker ID**: Your registry username - **Docker Password**: Your registry password - **Service connection name**: `dock-ptslondon-connection` -7. Check **Grant access permission to all pipelines** -8. Click **Save** +5. Check **Grant access permission to all pipelines** +6. Click **Save** -## Step 2: Choose Your Pipeline +## Step 3: Update Pipeline Configuration -You have two pipeline options: +1. Open `azure-pipelines.yml` in your repository +2. Update the GitHub repository path in the resources section: + ```yaml + resources: + repositories: + - repository: self + type: github + endpoint: github # This matches your GitHub service connection + name: your-github-username/price-tracker # Replace with your actual repo + ``` -### Basic Pipeline (`azure-pipelines.yml`) -- Simple build and push -- Basic testing -- Good for getting started - -### Advanced Pipeline (`azure-pipelines-advanced.yml`) -- Multi-stage deployment -- Security scanning with Trivy -- Separate dev/prod environments -- More comprehensive testing - -## Step 3: Create the Pipeline +## Step 4: Create the Pipeline 1. In Azure DevOps, go to **Pipelines** → **Pipelines** 2. Click **New pipeline** -3. Select **Azure Repos Git** (or your source) +3. Select **GitHub** +4. Select your Price Tracker repository from GitHub +5. Azure DevOps will detect the `azure-pipelines.yml` file +6. Review the pipeline and click **Run** + +## Alternative: Manual YAML Pipeline Setup + +If you prefer to create the pipeline manually: +1. Go to **Pipelines** → **Pipelines** +2. Click **New pipeline** +3. Select **GitHub YAML** 4. Select your repository 5. Choose **Existing Azure Pipelines YAML file** -6. Select the pipeline file: - - `/azure-pipelines.yml` (basic) - - `/azure-pipelines-advanced.yml` (advanced) +6. Select `/azure-pipelines.yml` 7. Click **Continue** 8. Review the pipeline and click **Run** @@ -65,7 +86,27 @@ If using the advanced pipeline, create environments: - Click the three dots → **Approvals and checks** - Add **Approvals** for production -## Step 5: Pipeline Configuration +## Step 5: GitHub Integration Benefits + +Using GitHub as your source provides several advantages: + +### Automatic Sync +- Changes to your GitHub repository automatically trigger Azure DevOps pipelines +- No need to maintain separate Azure Repos +- Keeps your source code in one place + +### Branch Protection +- Configure branch protection rules in GitHub +- Require pull request reviews +- Status checks from Azure DevOps can block merges + +### GitHub Actions Alternative +While Azure DevOps pulls from GitHub, you could also: +- Use GitHub Actions for CI/CD (free tier: 2000 minutes/month) +- Mirror from GitHub to Azure DevOps for backup +- Use hybrid approach (GitHub Actions + Azure DevOps) + +## Step 6: Pipeline Configuration The pipeline is configured to: @@ -88,7 +129,7 @@ The pipeline is configured to: - `containerRegistry`: `dock.ptslondon.co.uk` - `tag`: Uses build ID for versioning -## Step 6: Customize for Your Needs +## Step 7: Customize for Your Needs ### Registry Settings If your registry requires different settings, update these variables in the pipeline: @@ -112,7 +153,7 @@ The advanced pipeline includes Trivy security scanning. To disable: - Remove the `SecurityScan` stage - Remove it from the `dependsOn` lists -## Step 7: Verify the Setup +## Step 8: Verify the Setup 1. Make a small change to your code 2. Push to the `develop` branch (for testing) @@ -124,22 +165,33 @@ The advanced pipeline includes Trivy security scanning. To disable: ### Common Issues -1. **Service Connection Failed** +1. **GitHub Service Connection Failed** + - Verify GitHub permissions for Azure Pipelines app + - Check that the repository path is correct + - Ensure the service connection name matches (`github`) + - Try re-authorizing the GitHub connection + +2. **Service Connection Failed** - Verify registry credentials - Check registry URL format - Ensure registry is accessible from Azure -2. **Docker Build Failed** +2. **Docker Service Connection Failed** - Check Dockerfile syntax - Verify all required files are in repository - Check build logs for specific errors -3. **Push to Registry Failed** +3. **Docker Build Failed** - Verify service connection permissions - Check registry quota/space - Ensure repository name is correct -4. **Tests Failed** +4. **Push to Registry Failed** + - Verify service connection permissions + - Check registry quota/space + - Ensure repository name is correct + +5. **Tests Failed** - Check application startup logs - Verify port mappings - Ensure dependencies are available diff --git a/README.md b/README.md index c6dffe6..f01b51c 100644 --- a/README.md +++ b/README.md @@ -171,6 +171,51 @@ Add new e-commerce sites by extending the sites configuration: 4. **Multiple Sites**: Track the same product on multiple sites for best deals 5. **Regular Updates**: Run scraping regularly but not too frequently (every few hours is good) +## Deployment 🚀 + +### Docker Deployment + +1. **Build and run with Docker**: + ```bash + # Build the container + docker build -t price-tracker . + + # Run with docker-compose + docker-compose up -d + ``` + +2. **Manual Docker deployment**: + ```bash + docker run -d \ + --name price-tracker \ + -p 5000:5000 \ + -v $(pwd)/data:/app/data \ + price-tracker + ``` + +### CI/CD with Azure DevOps + +The project includes Azure DevOps pipeline configuration for automated deployments: + +1. **Setup GitHub Integration**: + - See `AZURE-DEVOPS-SETUP.md` for detailed instructions + - Pipeline pulls directly from GitHub + - Automatic builds on push to `main` or `develop` branches + +2. **Pipeline Features**: + - Docker image build and push to registry + - Security scanning with Trivy + - Automated testing + - Multi-environment deployment (dev/prod) + +3. **Quick Setup**: + ```bash + # Update azure-pipelines.yml with your GitHub repo + # Create GitHub service connection in Azure DevOps + # Create Docker registry service connection + # Run the pipeline + ``` + ## Troubleshooting 🔧 ### Common Issues diff --git a/azure-pipelines-template.yml b/azure-pipelines-template.yml new file mode 100644 index 0000000..bdd1221 --- /dev/null +++ b/azure-pipelines-template.yml @@ -0,0 +1,98 @@ +# Template: Azure DevOps Pipeline for GitHub Integration +# +# INSTRUCTIONS: +# 1. Replace 'your-github-username/price-tracker' with your actual GitHub repository path +# 2. Ensure your GitHub service connection is named 'github' in Azure DevOps +# 3. Update the Docker registry variables if needed +# 4. Copy this content to azure-pipelines.yml in your repository + +# Pipeline for Price Tracker application +# This pipeline pulls from GitHub and builds/deploys the Docker container + +# Define the GitHub repository as the source +resources: + repositories: + - repository: self + type: github + endpoint: github # This should match your GitHub service connection name + name: your-github-username/price-tracker # TODO: Replace with your actual GitHub repo path + +trigger: + branches: + include: + - main + - develop + paths: + include: + - src/* + - templates/* + - requirements.txt + - Dockerfile + - config.json + - main.py + +variables: + # Container registry service connection established during pipeline creation + dockerRegistryServiceConnection: 'dock-ptslondon-connection' + imageRepository: 'price-tracker' + containerRegistry: 'dock.ptslondon.co.uk' # TODO: Update if using different registry + dockerfilePath: '$(Build.SourcesDirectory)/Dockerfile' + tag: '$(Build.BuildId)' + + # Agent VM image name + vmImageName: 'ubuntu-latest' + +stages: +- stage: Build + displayName: Build and push stage + jobs: + - job: Build + displayName: Build + pool: + vmImage: $(vmImageName) + steps: + - task: Docker@2 + displayName: Build and push Docker image + inputs: + command: buildAndPush + repository: $(imageRepository) + dockerfile: $(dockerfilePath) + containerRegistry: $(dockerRegistryServiceConnection) + tags: | + $(tag) + latest + + # Optional: Run security scan on the image + - task: Docker@2 + displayName: Run Trivy vulnerability scanner + inputs: + command: 'run' + arguments: '--rm -v /var/run/docker.sock:/var/run/docker.sock -v $(System.DefaultWorkingDirectory):/tmp/trivy aquasec/trivy image --exit-code 0 --severity HIGH,CRITICAL $(containerRegistry)/$(imageRepository):$(tag)' + continueOnError: true + + # Optional: Test the built image + - task: Docker@2 + displayName: Test Docker image + inputs: + command: 'run' + arguments: '--rm -d --name price-tracker-test -p 5001:5000 $(containerRegistry)/$(imageRepository):$(tag)' + continueOnError: true + + - script: | + # Wait for container to start + sleep 10 + # Test health endpoint + curl -f http://localhost:5001/ || echo "Health check failed" + # Cleanup test container + docker stop price-tracker-test || true + displayName: 'Health check test' + continueOnError: true + + # Publish build artifacts + - task: PublishBuildArtifacts@1 + displayName: 'Publish docker-compose and deployment files' + inputs: + PathtoPublish: '$(Build.SourcesDirectory)' + ArtifactName: 'deployment-files' + publishLocation: 'Container' + condition: succeeded() diff --git a/azure-pipelines.yml b/azure-pipelines.yml index 1ac280e..19618e2 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -1,3 +1,14 @@ +# Pipeline for Price Tracker application +# This pipeline pulls from GitHub and builds/deploys the Docker container + +# Define the GitHub repository as the source +resources: + repositories: + - repository: self + type: github + endpoint: github # This should match your GitHub service connection name + name: olipassey/price-tracker # Replace with your actual GitHub repo path + trigger: branches: include: