Blob lifecycle

Lab Objective

In this hands-on lab, you will learn how to:

Configure blob lifecycle management policies to automatically manage data aging
Set up access tier transitions (Hot → Cool → Archive) based on data age
Implement automated blob deletion for expired content
Create conditional rules using blob filters and prefixes
Monitor lifecycle policy execution and cost optimization
Test different lifecycle scenarios for various data types
Troubleshoot lifecycle policy issues and optimize configurations

Scenario: Your organization stores various types of data with different access patterns and retention requirements. You need to implement automatic lifecycle management to optimize storage costs while ensuring compliance with data retention policies.

Pre-Provisioned Environment

The following Azure resources have been pre-deployed in your environment:

Resource Overview

Resource Type	Resource Name	Configuration	Purpose
Resource Group	`BlobLifecycle-Lab-RG`	Contains all lab resources	Logical container
Storage Account	`lifecyclestore[unique]`	GPv2, Hot tier, LRS	Primary lifecycle testing
Storage Account	`archivestore[unique]`	GPv2, Cool tier, GRS	Archive scenario testing
Test VM	`LifecycleVM`	Windows Server 2019	Data upload and monitoring
Blob Containers	Various	Different access levels	Organized test data

Storage Account Configuration

Account	Default Tier	Redundancy	Purpose
lifecyclestore[unique]	Hot	LRS	Active data lifecycle testing
archivestore[unique]	Cool	GRS	Long-term archive scenarios

Container Structure

Primary Storage (lifecyclestore[unique])
├── documents/              (Private access)
│   ├── current/           (Active documents)
│   ├── archived/          (Old documents)
│   └── temp/              (Temporary files)
├── logs/                  (Private access)
│   ├── application/       (App logs)
│   ├── system/           (System logs)
│   └── audit/            (Audit logs)
├── media/                 (Private access)
│   ├── images/           (Image files)
│   └── videos/           (Video content)
└── backups/               (Private access)
    ├── daily/            (Daily backups)
    ├── weekly/           (Weekly backups)
    └── monthly/          (Monthly backups)

Archive Storage (archivestore[unique])
├── compliance-archive/    (Long-term compliance data)
└── cold-storage/         (Rarely accessed data)

Data Types and Expected Lifecycle

Data Type	Initial Tier	Cool Transition	Archive Transition	Deletion
Active Documents	Hot	30 days	90 days	7 years
Application Logs	Hot	7 days	30 days	1 year
System Logs	Hot	1 day	7 days	90 days
Media Files	Hot	60 days	180 days	Never
Daily Backups	Cool	Immediate	30 days	90 days
Compliance Data	Cool	30 days	90 days	10 years

Lab Exercises

Part 1: Understand Current Storage Costs

Step 1: Examine Storage Account Tiers

Navigate to BlobLifecycle-Lab-RG resource group
Click on lifecyclestore[unique] storage account
Go to Storage browser → Blob containers

Review existing containers:

documents: Contains various document types
logs: Application and system log files
media: Image and video files
backups: Daily, weekly, monthly backups

Step 2: Check Current Access Tiers

Open each container and examine some blobs
Click on a blob → Properties
Note the Access tier:
- Most should show “Hot” (default)
- Check blob size and last modified date
- Calculate approximate storage costs

Step 3: Understand Tier Pricing

Review pricing implications:

Access Tier	Storage Cost	Access Cost	Retrieval Cost	Use Case
Hot	Highest	Lowest	None	Frequently accessed
Cool	Medium	Medium	Low	Infrequently accessed (30+ days)
Archive	Lowest	Highest	High	Rarely accessed (180+ days)

Example cost savings:

1TB Hot storage: ~$18/month
1TB Cool storage: ~$10/month (44% savings)
1TB Archive storage: ~$2/month (89% savings)

Part 2: Create Basic Lifecycle Policy

Step 1: Access Lifecycle Management

In lifecyclestore[unique] storage account
Go to Data management → Lifecycle management
Click “Add a rule”

Step 2: Create First Rule - Application Logs

Configure rule basics:

Rule name: ApplicationLogLifecycle
Rule scope: Limit blobs with filters
Blob type: Block blobs
Blob subtype: Base blobs

Set filters:

Prefix match: logs/application/
Blob index match: (Leave empty for now)

Step 3: Configure Actions for Application Logs

Set up tier transitions:

If base blobs were:

Created more than 7 days ago: Move to cool storage
Created more than 30 days ago: Move to archive storage
Created more than 365 days ago: Delete the blob

Click “Add rule” to save

Step 4: Create Second Rule - System Logs

Create another rule:

Rule name: SystemLogLifecycle
Blob scope: Limit blobs with filters
Prefix match: logs/system/

Configure actions:

Created more than 1 day ago: Move to cool storage
Created more than 7 days ago: Move to archive storage
Created more than 90 days ago: Delete the blob

Step 5: Create Third Rule - Documents

Create document lifecycle rule:

Rule name: DocumentLifecycle
Prefix match: documents/

Configure actions:

Created more than 30 days ago: Move to cool storage
Created more than 90 days ago: Move to archive storage
Created more than 2555 days ago (7 years): Delete the blob

Part 3: Advanced Lifecycle Rules

Step 1: Create Media Files Rule with Conditions

Add new rule: MediaLifecycle
Prefix match: media/

Configure conditional actions:

Created more than 60 days ago: Move to cool storage
Created more than 180 days ago: Move to archive storage
Never delete (media has long-term value)

Step 2: Create Backup Retention Rule

Add new rule: BackupLifecycle
Prefix match: backups/daily/

Configure aggressive cleanup:

Created more than 0 days ago: Move to cool storage (immediate)
Created more than 30 days ago: Move to archive storage
Created more than 90 days ago: Delete the blob

Step 3: Create Size-Based Rule

Add new rule: LargeFileLifecycle
No prefix filter (applies to all containers)
Blob index match: Add condition for large files

Configure by size (if supported):

Target blobs larger than 100MB
Created more than 14 days ago: Move to cool storage
Created more than 45 days ago: Move to archive storage

Part 4: Test Lifecycle Policies

Step 1: Upload Test Data with Dates

Connect to LifecycleVM:
- Username: azureuser
- Password: LabPassword123!
Create test files with different ages:
- Open PowerShell as Administrator
- Create files and set their creation dates

# Create test directories locally
$testPath = "C:\LifecycleTest"
New-Item -ItemType Directory -Path $testPath -Force

# Create application log files with different ages
$logPath = "$testPath\logs\application"
New-Item -ItemType Directory -Path $logPath -Force

# Current log (0 days old)
"Current application log - $(Get-Date)" | Out-File "$logPath\current.log"

# Old log (10 days old)
"Old application log - $(Get-Date)" | Out-File "$logPath\old.log"
(Get-Item "$logPath\old.log").CreationTime = (Get-Date).AddDays(-10)
(Get-Item "$logPath\old.log").LastWriteTime = (Get-Date).AddDays(-10)

# Ancient log (40 days old)
"Ancient application log - $(Get-Date)" | Out-File "$logPath\ancient.log"
(Get-Item "$logPath\ancient.log").CreationTime = (Get-Date).AddDays(-40)
(Get-Item "$logPath\ancient.log").LastWriteTime = (Get-Date).AddDays(-40)

Step 2: Upload Test Files to Storage

In Azure portal, go to Storage browser
Navigate to logs/application/ container
Upload the test files created above
Verify upload and note their creation times

Step 3: Force Policy Execution (Simulation)

Note: Lifecycle policies run once per day. For testing:

Check policy status:
- Go to Lifecycle management
- Review your rules and ensure they’re enabled
- Note: “Last run” shows when policies last executed
Simulate policy effects:
- Manually change access tiers to simulate policy execution
- Go to blob → Change access tier
- Test transitions: Hot → Cool → Archive

Step 4: Monitor Tier Changes

Wait 24-48 hours for actual policy execution
Check blob access tiers:
- Navigate to each test blob
- Verify tier changes match your policy rules
- Document any unexpected behavior

Part 5: Monitor Costs and Policy Performance

Step 1: Set Up Cost Monitoring

Go to Cost Management + Billing
Create cost alert:
- Scope: BlobLifecycle-Lab-RG
- Alert condition: When cost exceeds $10
- Action group: Email notification

Step 2: Analyze Storage Metrics

In lifecyclestore[unique], go to Insights
Review metrics:
- Capacity trends by access tier
- Transaction patterns
- Data retrieval costs

Step 3: Create Lifecycle Dashboard

Track key metrics:

Metric	Before Lifecycle	After Lifecycle	Savings
Hot storage (GB)	[Record baseline]	[Monitor change]	[Calculate]
Cool storage (GB)	[Record baseline]	[Monitor change]	[Calculate]
Archive storage (GB)	[Record baseline]	[Monitor change]	[Calculate]
Monthly cost	[Record baseline]	[Monitor change]	[Calculate]

Step 4: Review Policy Effectiveness

Create effectiveness report:

Rule Name	Objects Affected	Tier Changes	Cost Impact	Status
ApplicationLogLifecycle	[Count]	[Transitions]	[Savings]	✅ Active
SystemLogLifecycle	[Count]	[Transitions]	[Savings]	✅ Active
DocumentLifecycle	[Count]	[Transitions]	[Savings]	✅ Active
MediaLifecycle	[Count]	[Transitions]	[Savings]	✅ Active

Part 6: Troubleshoot and Optimize

Step 1: Common Issues and Solutions

Check for common problems:

Policy not executing:
- Verify: Policy is enabled
- Check: Syntax errors in filters
- Ensure: Sufficient permissions
Unexpected tier changes:
- Review: Filter conditions (prefixes)
- Check: Date calculations
- Verify: Blob metadata
High retrieval costs:
- Analyze: Access patterns
- Adjust: Archive timing
- Consider: Cool tier instead of archive

Step 2: Optimize Existing Rules

Review and adjust rules:

Check rule efficiency:
- Are transitions happening as expected?
- Are there too many or too few objects affected?
- Is timing appropriate for access patterns?
Adjust based on usage:
- Extend cool period if data accessed more than expected
- Shorten archive period if retrieval costs are low
- Add exceptions for critical data

Step 3: Create Exception Rules

Handle special cases:

Important documents exception:
- Rule name: ImportantDocumentsException
- Prefix: documents/important/
- Actions: Move to cool after 90 days only (no archive/delete)
Audit logs exception:
- Rule name: AuditLogsException
- Prefix: logs/audit/
- Actions: Move to archive after 2555 days (7 years), never delete

Troubleshooting Guide

Common Lifecycle Management Issues

Issue	Symptoms	Possible Cause	Solution
Policy not running	No tier changes after 24+ hours	Policy disabled or syntax error	Check policy status and JSON syntax
Wrong blobs affected	Unexpected objects changing tiers	Overly broad filters	Refine prefix filters and conditions
High retrieval costs	Unexpected archive access charges	Data accessed too frequently	Adjust archive timing or use cool tier
Compliance violations	Data deleted prematurely	Incorrect deletion rules	Review and extend retention periods
Performance issues	Slow access to archived data	Data in archive tier	Plan ahead for archive retrieval times

Policy Configuration Checklist

Component	Requirement	Status	Notes
Rule enabled	Must be active	✅	Check policy status
Correct filters	Prefix matches intended data	✅	Test with sample data
Appropriate timing	Matches business requirements	✅	Verify with stakeholders
Deletion safeguards	Long retention for critical data	✅	Review retention policies
Cost optimization	Balance access needs vs. cost	✅	Monitor cost metrics

Best Practices

Scenario	Recommendation	Benefit
Log files	Hot → Cool (1-7 days), Archive (7-30 days)	Fast troubleshooting, cost effective
Documents	Hot → Cool (30 days), Archive (90+ days)	Balance accessibility and cost
Backups	Cool immediately, Archive (30 days)	Maximum cost savings
Media files	Hot → Cool (60+ days), rarely archive	Preserve access quality
Compliance data	Cool → Archive quickly, long retention	Meet regulations cost-effectively

Optional Advanced Exercises

For users wanting more technical depth, try these exercises:

Advanced Exercise 1: Automated Policy Creation

# Connect to Azure
Connect-AzAccount

# Create lifecycle policy via PowerShell
$resourceGroup = "BlobLifecycle-Lab-RG"
$storageAccount = "lifecyclestore[unique]"

# Define policy JSON
$policyJson = @"
{
  "rules": [
    {
      "enabled": true,
      "name": "PowerShellCreatedRule",
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["test/"]
        },
        "actions": {
          "baseBlob": {
            "tierToCool": {"daysAfterModificationGreaterThan": 7},
            "tierToArchive": {"daysAfterModificationGreaterThan": 30},
            "delete": {"daysAfterModificationGreaterThan": 365}
          }
        }
      }
    }
  ]
}
"@

# Apply policy
$policy = $policyJson | ConvertFrom-Json
Set-AzStorageAccountManagementPolicy -ResourceGroupName $resourceGroup -StorageAccountName $storageAccount -Policy $policy

Advanced Exercise 2: Policy Impact Analysis

# Analyze potential cost savings
$storageAccount = Get-AzStorageAccount -ResourceGroupName "BlobLifecycle-Lab-RG" -Name "lifecyclestore[unique]"
$ctx = $storageAccount.Context

# Get blob statistics by container
Get-AzStorageContainer -Context $ctx | ForEach-Object {
    $containerName = $_.Name
    $blobs = Get-AzStorageBlob -Container $containerName -Context $ctx

    $stats = $blobs | Group-Object {$_.AccessTier} | ForEach-Object {
        @{
            Container = $containerName
            Tier = $_.Name
            Count = $_.Count
            TotalSize = ($_.Group | Measure-Object Length -Sum).Sum
        }
    }

    Write-Host "Container: $containerName"
    $stats | Format-Table
}

Advanced Exercise 3: Custom Monitoring Dashboard

# Create custom lifecycle monitoring report
function Get-LifecycleReport {
    $storageAccount = Get-AzStorageAccount -ResourceGroupName "BlobLifecycle-Lab-RG" -Name "lifecyclestore[unique]"

    # Get management policy
    $policy = Get-AzStorageAccountManagementPolicy -ResourceGroupName "BlobLifecycle-Lab-RG" -StorageAccountName "lifecyclestore[unique]"

    Write-Host "=== LIFECYCLE MANAGEMENT REPORT ==="
    Write-Host "Generated: $(Get-Date)"
    Write-Host "Storage Account: $($storageAccount.StorageAccountName)"
    Write-Host ""

    if ($policy) {
        Write-Host "Active Rules: $($policy.Policy.Rules.Count)"
        $policy.Policy.Rules | ForEach-Object {
            Write-Host "  - $($_.Name): $($_.Enabled)"
        }
    } else {
        Write-Host "No lifecycle policies configured"
    }
}

Get-LifecycleReport

Key Takeaways

After completing this lab, you should understand:

Lifecycle management policies automatically optimize storage costs by moving data through access tiers
Rule-based automation reduces manual intervention and ensures consistent data management
Filter conditions (prefixes, blob types) allow granular control over which data is affected
Tier transitions should align with actual data access patterns to avoid unexpected costs
Archive tier provides maximum cost savings but has retrieval delays and costs
Deletion rules require careful consideration of business and compliance requirements
Policy monitoring is essential to ensure rules work as expected and optimize costs

Lifecycle Decision Matrix

Choosing the Right Lifecycle Strategy

Data Type	Access Pattern	Recommended Strategy	Justification
Application logs	High initially, rare after 30 days	Hot → Cool (7d) → Archive (30d) → Delete (1y)	Debugging needs vs. storage cost
User documents	Variable, decreasing over time	Hot → Cool (30d) → Archive (90d) → Keep long-term	Business value retention
System backups	Rare access, compliance driven	Cool immediately → Archive (30d) → Delete (90d)	Cost optimization priority
Media content	Moderate access, long-term value	Hot → Cool (60d) → Archive (180d) → Never delete	Quality preservation
Compliance data	Rare access, legal requirements	Cool → Archive (30d) → Delete after legal minimum	Regulatory compliance

Cost Optimization Guidelines

Scenario	Optimization Strategy	Expected Savings
Log management	Aggressive archiving after 30 days	70-80% storage cost reduction
Document management	Balanced approach with 90-day cool transition	40-50% storage cost reduction
Backup optimization	Immediate cool tier, fast archive	80-90% storage cost reduction
Media optimization	Conservative transitions, no deletion	30-40% storage cost reduction