Azure VMs show Software Inventory as enabled in the portal, but no data appears in the connected Log Analytics workspace. The ConfigurationData and ConfigurationChange tables stay empty even hours after enablement. The Change Tracking extension appears provisioned with no obvious errors.

On the VM itself, the Azure Monitor Agent (AMA) logs repeat the following errors every 4-5 minutes:

Failed to find an output stream for: ""
Error sending kusto telemetry data through output handler.
socket/pipe Error while sending request data
Error while reading settings  dial unix @CAgentStream_CloudAgentInfo_config_default_fluent.socket: connect: connection refused

Root Cause

The Azure Monitor Agent cannot authenticate to Azure because the VM has no managed identity assigned.

AMA uses the VM's managed identity to get a token from the Azure Instance Metadata Service (IMDS) endpoint (http://169.254.169.254). This token is required to:

• Download the Data Collection Rule (DCR) configuration from Azure Monitor Configuration Service (AMCS)
• Send collected data to the Log Analytics workspace

Without a token, AMA starts but fails to initialize its data pipeline. The Unix socket that the ChangeTracking extension communicates through (@CAgentStream_CloudAgentInfo_config_default_fluent.socket) is never created, because it only exists once the pipeline is up. The ChangeTracking extension then loops retrying the socket connection indefinitely.

The error in /var/opt/microsoft/azuremonitoragent/log/mdsd.err confirms this:

Failed to get MSI token from IMDS endpoint: http://169.254.169.254 ErrorCode:-2146041343

Diagnosis Steps

1. Verify AMA is running but has no socket

systemctl is-active azuremonitoragent

# Check if the socket exists
ss -xlp | grep fluent

If AMA is active but the socket is missing, proceed to step 2.

2. Check AMA error logs

sudo tail -50 /var/opt/microsoft/azuremonitoragent/log/mdsd.err

Look for Failed to get MSI token from IMDS endpoint.

3. Confirm the VM has no managed identity

In the Azure portal: navigate to the VM → Security → Identity.

• System assigned tab: Status should be On
• User assigned tab: should list at least one identity

If both are empty/off, the managed identity is missing.

4. Confirm IMDS is reachable but returns no identity

Run this from inside the VM:

curl -s -H "Metadata: true" \
  "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://monitor.azure.com/"

A VM with no managed identity returns an error response rather than an access token.

5. Verify DCR config was never downloaded

ls /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/

The directory will be empty, confirming AMA never reached the configuration service.

Fix

Enable system-assigned managed identity on the VM:

Azure portal: VM → Security → Identity → System assigned → set Status to On → Save

Azure CLI:

az vm identity assign -g <resource-group> -n <vm-name>

After saving, restart AMA:

sudo systemctl restart azuremonitoragent

Within 2-3 minutes, verify the socket is created:

ss -xlp | grep fluent

And confirm AMA is downloading its configuration:

ls /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/

Health Check After Fix

1. Check AMA service is running: systemctl status azuremonitoragent
2. Confirm the socket exists: ss -xlp | grep fluent
3. Verify DCR config downloaded: ls /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/
4. No new errors in: /var/opt/microsoft/azuremonitoragent/log/mdsd.err

References

• Azure Monitor Agent troubleshooting — Linux
• Configure managed identities on Azure VMs
• Enable Change Tracking and Inventory for Azure VMs

Why I Pursued AI-102

In my day-to-day role, I automate pipelines, manage infrastructure as code, and integrate services across Azure. AI is becoming integral to modern apps—think chatbots in CI/CD tools, image analysis in deployment monitoring, or natural language processing for log analytics. The AI-102 cert focuses on building and deploying Azure AI solutions, which aligns perfectly with DevOps principles like CI/CD integration, monitoring, and security. Plus, it's a great way to level up in the Azure AI Engineer Associate path. If you have a background in Python/C#, REST APIs, or Azure fundamentals (like AZ-900), this is a natural next step.

Step 1: Exploring the Official Study Guide

The foundation of my prep was the official Microsoft study guide at https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-102. It's updated as of April 30, 2025, with no major changes since then (I double-checked for any November updates—none yet). This guide outlines what to expect: a 120-minute exam with multiple-choice, drag-and-drop, and case studies, needing a 700+ score to pass.

Key highlights:

• Audience Profile: For engineers who design, build, deploy, and manage AI solutions using Azure AI services like Vision, Language, Speech, Search, and OpenAI.
• Skills Measured (20-25% on planning/managing; 15-20% on generative AI; 5-10% on agentic solutions; 10-15% on computer vision; 15-20% on NLP; 15-20% on knowledge mining).
• Changes: Recent updates emphasize Azure AI Foundry, responsible AI, and new sections on agentic solutions and generative AI optimization.

Pro Tip: Start here to map out gaps in your knowledge. The guide links to free practice assessments and the exam sandbox—use them early!

Step 2: Creating a Learning Plan and Managing Time

Consistency is key in DevOps, and the same applies to cert prep. I planned for 2-3 months (about 8-10 weeks), studying 1-2 hours daily and 4-6 hours on weekends. Here's my structured plan, aligned with the exam domains:

1. Week 1-2: Plan and Manage Azure AI Solutions (20-25%) Focus: Service selection, responsible AI principles, resource creation (via Portal/CLI/ARM), CI/CD integration, monitoring (Azure Monitor, costs), security (Key Vault, Entra ID). Time: 10-15 hours. Hands-on: Set up an Azure AI resource and deploy a simple container.
2. Week 3-4: Generative AI and Agentic Solutions (20-30% combined) Focus: Building with Azure AI Foundry/OpenAI, prompts, RAG patterns, agents (Semantic Kernel, Autogen), optimization (parameters, fine-tuning). Time: 15-20 hours. Hands-on: Deploy GPT models, create custom agents.
3. Week 5: Computer Vision Solutions (10-15%) Focus: Image analysis (OCR, object detection), custom models, video insights (Video Indexer, Spatial Analysis). Time: 8-10 hours. Hands-on: Train a Custom Vision model and analyze videos.
4. Week 6: Natural Language Processing Solutions (15-20%) Focus: Text/speech analysis, translation, custom models (CLU, QnA), intent recognition. Time: 10-12 hours. Hands-on: Build a speech-to-text app and a multi-language QnA bot.
5. Week 7: Knowledge Mining and Information Extraction (15-20%) Focus: Azure AI Search (indexes, skillsets), Document Intelligence (prebuilt/custom models), content understanding. Time: 10-12 hours. Hands-on: Create a search index with custom skills and extract data from docs.
6. Week 8-10: Review, Practice, and Mock Exams Time: 20+ hours. Revisit weak areas, take the free practice assessment on Microsoft Learn, and simulate the exam environment.

Tools for time management: I used my custom Obsidian note for tracking progress, set daily Pomodoro sessions (25-min focus bursts), and blocked calendar time. Adjust based on your schedule— if you're full-time working, aim for evenings/weekends to avoid burnout.

Step 3: Key Resources I Used

I curated and relied on a mix of official and community resources. Here's what worked:

• My Own Repository: I built https://github.com/amirkhonov/microsoft-ai-102-exam-study-guide based on the skills measured. It organizes everything by domain with links to Microsoft Learn modules, official docs, quickstarts, and hands-on labs. Feel free to fork and contribute!
• Mindmap for Visual Learners: The repo at https://github.com/lrivallain/ai-102-mindmap is gold. It's a markmap-based outline covering all skills hierarchically—great for quick reviews. It details services like Azure AI Vision, Speech, Search, and OpenAI, with practical steps (e.g., CLI commands, API examples). This repository can include the outdated/retired services, double-check everything here.
• Other Notes and Repos: Check out https://github.com/vatsprat/AI-102-AI-Engineer-Associate-Certification-Exam- for personal notes on clearing the exam. While it's lightweight, it inspired me to jot down my own summaries.
• Study Resources from the Guide: Azure docs for each service (e.g., OpenAI, Vision), Microsoft Q&A for doubts, and Tech Community forums.

Step 4: Hands-On Practice with Azure Subscription

Theory alone won't cut it—AI-102 is heavy on implementation. I used my Azure subscription for labs. If you're new, start with the free trial (azure.microsoft.com/free) which gives $200 credit for 30 days. Opt for free tiers where possible (e.g., Azure AI Search free tier, OpenAI playground).

Tips to Control Costs:

• Monitor usage in Azure Cost Management—set budgets and alerts.
• Delete resources after labs (use Azure CLI: az group delete).
• Stick to low-quota deployments (e.g., standard SKU for testing).
• I spent about $50 over 2 months, mostly on OpenAI tokens and Vision processing—worth it for real-world skills.

Hands-on examples: Deployed a RAG-based app with Azure AI Foundry, trained custom vision models, and integrated Speech SDK in a Python script. This made concepts like prompt engineering and skillsets stick.

Exam Day Experience and Tips

The exam had ~58 questions, including case studies on end-to-end solutions. It tested practical scenarios: choosing services, securing resources, optimizing generative AI.

Tips:

• Focus on Changes: Know the April 2025 updates—e.g., agentic solutions, responsible AI governance.
• Practice Responsibly: Questions on content safety, prompt shields, and ethical AI are common.
• Time Management: Skip tough ones first; use the extra 30 mins if English isn't your first language.
• Utilize Microsoft Learn During the Exam: Since Microsoft Docs and Learn are accessible during the exam (for non-Fundamentals certs like AI-102), familiarize yourself with their structure to quickly search for necessary information on services, APIs, and configurations.

• Don't Cram: Review mindmaps the day before, get good sleep.

• Post-Exam: Renew annually via free assessments on Learn.

Final Thoughts

Passing AI-102 has already paid off—I'm now integrating AI into my DevOps pipelines more confidently. If you're prepping, start with the official guide, build a plan, and get hands-on. Check my repo for structured resources.

What cert are you chasing next? Let me know in the comments.

In that chaos, your terminal becomes your battlefield — and how you use Bash determines whether you’re firefighting or fixing.

Most engineers know basic Bash commands, but great SREs know how to bend the shell to their will. Here are five Bash tricks that separate seasoned SREs from everyone else — the kind that turn a 30-minute debug session into a five-minute win.

1. Process Substitution — Compare Anything, Anywhere, Instantly

Think beyond pipes. Process substitution (<(...)) lets you treat live command outputs as files — perfect for real-time comparisons.

# Compare logs from two servers
diff <(ssh server1 "tail -f /var/log/app.log") <(ssh server2 "tail -f /var/log/app.log")

# Monitor two pods side by side
paste <(kubectl logs -f pod1) <(kubectl logs -f pod2) | column -t

# Compare configs across environments
diff <(curl -s https://prod.api.com/config) <(curl -s https://staging.api.com/config)

This feature feels like magic: no temp files, no manual juggling — just instant, live insight.

2. Command History — Your Incident Time Machine

When things break, there’s no time for retyping long commands. Bash history expansion gives you superhuman speed:

!kubectl       # Repeat the last kubectl command
^tpyo^typo     # Fix a typo from the previous command
sudo !!        # Rerun the last command with sudo
cp /var/log/app.log !$   # Use the last argument again

Pro tip:
Add this to your `.bashrc` to preview what history expansion will run:

bind Space:magic-space

Now hitting the spacebar after !kubectl shows exactly what Bash will execute — a lifesaver during high-stress debugging.

3. Brace Expansion — Batch Operations Without Scripts

Brace expansion {} is like having loops baked right into Bash syntax.

# Create backups
mkdir backup-{db,logs,config}-$(date +%Y%m%d)

# Check health for multiple environments
for env in {prod,staging,dev}; do
  echo "=== $env ==="
  curl -s https://$env.api.com/health | jq '.status'
done

# Restart multiple hosts
for host in web-{01..05}; do
  ssh $host "sudo systemctl restart nginx"
done

With this, you automate repetitive operations in one line — no need for temporary scripts or overcomplicated loops.

4. Command Substitution With Error Handling — Smart and Safe Automation

Command substitution $(...) lets your scripts react to live data. Pair it with simple error handling, and you get safe, self-correcting automation:

if pod_name=$(kubectl get pods -l app=critical --no-headers 2>/dev/null); then
  echo "Found pod: $pod_name"
  kubectl describe pod $pod_name
else
  echo "❌ No pods found - checking deployments"
  kubectl get deployments -l app=critical
fi

Why it matters: instead of hardcoding names and assumptions, your commands adapt to the current system state — a must for dynamic, ever-changing environments.

5. Parameter Expansion — Config Without Config Managers

This is Bash’s built-in templating system — perfect for defaults, fallbacks, and string manipulation.

DATABASE_URL=${DATABASE_URL:-postgresql://localhost:5432/app}
LOG_LEVEL=${LOG_LEVEL:-INFO}
TIMEOUT=${TIMEOUT:-30}

Or extract and manipulate values:

config="/etc/myapp/prod/database.conf"
env=${config%/*}; env=${env##*/}   # -> "prod"

db_url="postgresql://user:secret@db.example.com:5432/app"
safe_url=${db_url//:*@/:xxx@}      # -> hides the password

With this, you can handle complex configuration logic right inside Bash — no Python, no jq, no sed needed.

Bonus: Instant System Dashboard

Need a quick status view during an outage? One command:

watch -n1 '
echo "=== CPU/MEMORY ==="
top -bn1 | head -5
echo -e "\n=== DISK USAGE ==="
df -h | grep -v tmpfs
echo -e "\n=== NETWORK ==="
ssh -tuln | grep LISTEN | head -5
echo -e "\n=== RECENT ERRORS ==="
tail -n3 /var/log/syslog | grep -i error
'

This gives you a live, auto-updating dashboard — CPU, memory, disk, network, and errors — refreshed every second.

Why These Tricks Matter

SREs operate under pressure, often on unfamiliar systems. These Bash habits give you:

- Speed — fewer keystrokes, faster recovery.
- Reliability — fewer manual errors.
- Portability — works everywhere Bash does.
- Scalability — handle multiple servers and services simultaneously.

Final Thoughts

These aren’t just “cool hacks” — they’re battle-tested techniques that define a senior SRE’s command-line fluency. Master them, and you’ll turn Bash from a basic tool into an extension of your engineering intuition. Next time your dashboard lights up like a Christmas tree — you’ll be ready.

Digging into the agent logs confirmed my suspicion. Here’s what I saw:

time="2025-09-23T15:23:03Z" level=info msg="Agent Process got the configFolder C:\\Packages\\Plugins\\Microsoft.Azure.ChangeTrackingAndInventory.ChangeTracking-Windows\\2.35.0.0. \n"
time="2025-09-23T15:23:03Z" level=error msg="socket/pipe Error while sending request data"
time="2025-09-23T15:23:03Z" level=error msg="Error while reading settings  open \\\\.\\\\pipe\\\\CAgentStream_CloudAgentInfo_AzureMonitorAgent: The system cannot find the file specified."
time="2025-09-23T15:24:03Z" level=error msg="Failed to find an output stream for CONFIG_CHANGE_BLOB"
time="2025-09-23T15:24:03Z" level=error msg="Error sending kusto telemetry data through output handler."

That “pipe not found” error immediately stood out. It looked like the Change Tracking agent was trying to send data to the Azure Monitor Agent (AMA), but couldn’t reach it. Essentially, the Change Tracking agent depends on the AMA to send telemetry to Azure—but if that communication channel fails, all the data just… stalls.

To understand what was going on, I stumbled upon a great deep dive by Lucas Lifes: Deep Dig into Windows Change Tracking and Inventory with Azure Monitor Agent. It explains beautifully how the Change Tracking extension feeds its configuration and telemetry through the AMA. This gave me a clue: maybe the problem wasn’t with Change Tracking itself, but with the Monitor Agent underneath it.

Sure enough, when I checked the AMA logs under
C:\WindowsAzure\Resources\AMADataStore.test-vm0\Configuration,
I found this gem:

Info (2025-10-06T10:01:13Z): MonAgentManager.exe - Non-success status code from IMDS for MSI token with default identity. 
URI [/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://monitor.azure.com/], 
Status code=400, response [{"error":"invalid_request","error_description":"Identity not found"}]

There it was. The AMA was failing to get an MSI (Managed Service Identity) token from the Azure Instance Metadata Service (IMDS). Without a valid identity, it couldn’t authenticate to Azure Monitor, and without authentication, it couldn’t send data.

In plain terms:

• Change Tracking was trying to push its telemetry.
• AMA was supposed to handle that transmission.
• AMA couldn’t talk to Azure because it had no identity.
• Therefore, no data appeared in the portal.

The fix came from another excellent write-up by Ramana Reddy. The solution was to ensure the VM has a system-assigned or user-assigned managed identity enabled, and that the Azure Monitor Agent is allowed to use it. Once that was configured properly, the errors vanished, and the inventory data started flowing in as expected.

It’s a small but classic example of how Azure agents rely on a daisy chain of dependencies—one missing link, and the whole system quietly fails. The key takeaway?
When your Change Tracking data doesn’t show up, don’t just look at the extension. Follow the pipes—literally. The problem might be sitting one layer below, in your monitor agent’s authentication chain.

Next time you enable Change Tracking and Inventory, double-check that your VM has a valid managed identity and that Azure Monitor Agent can use it. It’ll save you a few hours of log spelunking.

This discrepancy often leads to authentication failures and can be particularly frustrating when working with third-party integrations or modern authentication frameworks that expect specific token formats.

Problem Analysis

The issue manifests when examining the iss (issuer) claim within access tokens. Instead of the anticipated modern issuer format, developers encounter:

• Received: https://sts.windows.net/{tenant-id}/
• Expected: https://login.microsoftonline.com/{tenant-id}/v2.0

This difference stems from Azure AD's dual token versioning system, where applications can receive tokens in either v1.0 or v2.0 format depending on their configuration.

Token Format Differences

Azure AD maintains two distinct token formats to ensure backward compatibility while supporting modern authentication requirements:

Version 1.0 Format

• Uses sts.windows.net as the issuer
• Primarily designed for work and school accounts
• Represents the legacy token structure
• Automatically assigned to applications without explicit version specification

Version 2.0 Format

• Uses login.microsoftonline.com as the issuer
• Supports both personal and organizational accounts
• Provides enhanced OpenID Connect compatibility
• Offers improved claim structure and additional features

Implementation Solution

The resolution requires modifying the application registration's manifest to explicitly request v2.0 tokens. This involves updating the accessTokenAcceptedVersion property.

Configuration Steps

1. Access Application Registration Navigate to the Azure Portal and locate the application registration under Azure Active Directory > App registrations.

2. Open Application Manifest Select the target application and access the "Manifest" section from the left navigation panel.

3. Modify Token Version Locate the accessTokenAcceptedVersion property and update its value:

"accessTokenAcceptedVersion": 2

4. Apply Changes Save the manifest and allow several minutes for the changes to propagate across Azure AD infrastructure.

Technical Implications

The token format has several downstream effects on application behavior and integration patterns:

Validation Logic: Applications performing issuer validation must account for the correct expected format. Hardcoded validation against a specific issuer will fail if the token format differs from expectations.

Library Compatibility: Modern authentication libraries, particularly those implementing OpenID Connect, often expect v2.0 token formats. Using v1.0 tokens may result in compatibility issues or reduced functionality.

Claims Structure: While both versions contain similar core claims, the structure and availability of certain claims may differ between versions.

Migration Considerations

Organizations planning to update existing applications should evaluate several factors:

Testing Requirements: Comprehensive testing across all authentication flows ensures that the token format change doesn't introduce regressions.

Dependent Systems: Any downstream services consuming these tokens must be evaluated for compatibility with the new issuer format.

Rollback Planning: Understanding how to revert changes quickly is essential for production environments.

Best Practices

New Application Development When creating new Azure AD integrations, explicitly set accessTokenAcceptedVersion to 2 during the initial application registration process.

Authentication Library Selection Utilize Microsoft Authentication Library (MSAL) which naturally aligns with v2.0 endpoints and token formats.

Environment Management Test token format changes in development and staging environments before implementing in production systems.

Alternative Approaches

In scenarios where modifying the application manifest isn't feasible, consider these alternatives:

• Update token validation logic to accept multiple issuer formats
• Implement conditional validation based on token version detection
• Create a new application registration with v2.0 configuration

Conclusion

The token issuer mismatch between sts.windows.net and login.microsoftonline.com represents a common configuration issue rather than a system malfunction. Understanding Azure AD's token versioning system and appropriately configuring the accessTokenAcceptedVersion property resolves this issue while positioning applications for better compatibility with modern authentication standards.

The transition from v1.0 to v2.0 tokens not only addresses immediate issuer validation concerns but also provides access to enhanced authentication features and improved integration capabilities with contemporary identity management systems.

• Low friction: minimal dependencies, no kernel modules, no heavy monitoring infrastructure.
• Thread-level visibility: ability to see what each thread is doing — whether it is running, sleeping, waiting on I/O, in kernel, etc.
• Always on, or close to it: tools for continuous sampling to catch intermittent or rare issues.

By combining sampling of /proc (for legacy and wide support) and newer eBPF-based functionality, 0x.tools bridges the gap between “traditional Linux tools” (top, ps, etc.) and more advanced observability setups.

Key Components & Tools

🔹 xcapture (v1, v2, v3-alpha)

The heart of 0x.tools. It continuously samples threads, capturing their state (running, waiting, sleeping), the current syscall, wait channels, and even stack traces. By storing this data in hourly CSVs, you can “rewind time” during troubleshooting. Perfect for diagnosing elusive issues like lock contention or I/O stalls.

🔹 xtop

Think of it as a “supercharged top.” xtop gives a live, interactive view of processes and threads, but with more detail than top or htop. It shows wall-clock times, kernel events, and individual thread behavior — ideal when you need a real-time snapshot with depth.

🔹 psn (Process Snapper)

A lightweight way to capture what threads are doing right now. It reveals which syscalls are active, what wait channels they’re in, and which threads are blocked. Useful for identifying immediate blockers in your system.

🔹 schedlat

This tool zooms in on scheduling latency — how long threads spend waiting before the CPU picks them up. It’s invaluable for spotting CPU starvation, scheduling bottlenecks, and workload imbalances.

🔹 Supporting Utilities

Other tools like lsds, syscallargs, tracepointargs, and xstack add detail about block devices, syscall arguments, kernel tracepoints, and stack behavior. Together, they extend your visibility from surface symptoms into root causes.

Why It Matters — Use Cases & Trade-Offs

Use Cases

• Production issue investigation: When something bad happens occasionally (latency spike, system pause, IO stall), classic monitoring (CPU, memory, I/O metrics) might not show why. 0x.tools lets you sample what threads were doing at those moments.
• Kernel vs Application boundary issues: Sometimes the delay is inside the kernel — e.g. lock contention, fsync, block device waits. 0x.tools highlights those.
• Legacy or constrained environments: Environments where you can’t install kernel modules or change kernel version easily. Since many components use /proc sampling, it supports older systems.
• Continuous profiling strategy: By collecting lightweight samples over time, you build a historical view. When trouble hits, you can inspect preceding behavior.

Trade-Offs & Considerations

• Overhead is low but non-zero. Sampling, even every second, uses some CPU. But designed to be <1%.
• On systems with tens of thousands of active threads, even sampling /proc can become expensive; you might need to reduce sample frequency.
• While eBPF adds more power and richer detail, it may not be available or supported on all Linux kernels or in all operating environments (enterprises, older machines).
• The tooling is strong for diagnosing what is happening, but less about visualization / dashboards (though that is in the roadmap). Requires comfort with command line, parsing CSVs, etc.

How It Works — Architecture & Methods

• Proc-based sampling: Many tools in 0x.tools simply read from /proc (which Linux exposes many kernel statistics via virtual files) at intervals. Thread state, syscalls, wait channels, etc.
• eBPF: Where supported, newer components (xcapture v3-alpha, etc.) leverage eBPF for more precise event instrumentation with less overhead. Enables off-CPU sampling, hooking into kernel tracepoints, etc.
• Historical archival: Samples can be written to hourly CSV archives, enabling “look back” after an issue. You can use standard text processing tools (awk, grep, etc.) or load into databases.

Practical Advice for Using 0x.tools Well

1. Start small in production
   Try sampling every few seconds or longer, maybe only on some hosts, to get a feel for overhead.
2. Correlate with external metrics
   Use 0x.tools in conjunction with your usual monitoring stack (CPU, mem, IO, latency). When dashboards show anomalies, check 0x.tools archives to see thread behavior.
3. Use historical data
   The ability to capture continuous or regular samplings means you might capture the root cause even before you realize there’s an issue.
4. Know your kernel/environment limitations
   If eBPF not available, stick to proc-based tools. Some kernel versions limit certain tracepoints.
5. Automate retention / cleanup
   CSV archives can grow; set up scripts to compress, rotate, archive or drop old data.

Why 0x.tools Fills an Important Niche

In many Linux performance stacks, there are gaps:

• Tools like prometheus, Grafana, CloudWatch etc. help aggregate metrics, show system usage over time. But they often don’t expose why a thread is waiting, or which syscall is slow.
• Distributed tracing (Jaeger, Zipkin) shows request flows, but not the low-level wait, lock, or kernel layer behavior inside threads.
• Traditional tools (top, ps) are great, but either only see CPU usage or don’t show sleeping/waiting in detail, or require manual invocation.

0x.tools sits in that gap: providing thread-level, kernel-aware, low overhead visibility, both live and historical.

Conclusion

0x.tools is an exciting toolset for anyone who manages Linux servers and cares about performance on a deeper level. It offers:

• visibility into what threads are waiting on, sleeping, or doing, rather than just coarse CPU / memory usage,
• the ability to catch intermittent or rare performance degradation,
• application in environments where heavier instrumentation is difficult or not allowed.

For system administrators, site reliability engineers, performance engineers: 0x.tools can reduce the time to identify root causes by clarifying what is really going on inside your system when things seem “slow” but no obvious metrics are showing a problem.

Here’s a no-fluff fix that’s been working reliably.

🧯 The Problem

You’re starting Colima and seeing something like this:

ERROR: mount type "reverse-sshfs" failed: ...

Or your containers simply don’t see your shared volumes at all.

It’s usually related to how Colima sets up the sshfs mount — and the problem tends to show up more often on M1/M2/M3 machines due to tighter system permissions and architecture differences.

✅ The Fix

Here’s the step-by-step fix I recommend:

1. Stop Colima (if running)

colima stop

2. Reset Colima's configuration

colima delete

This deletes the VM Colima uses but doesn’t touch your Docker images.

3. Install or Reinstall macFUSE

Colima depends on macFUSE for reverse mounts.

• Download from: https://osxfuse.github.io
• Or install via Homebrew:

brew install --cask macfuse

⚠️ After installing, you must reboot your Mac. System Extensions require a reboot to become active.

4. Start Colima with the right mount type

colima start --mount-type=sshfs

If you want to make it permanent:

colima start --mount-type=sshfs --edit

Edit the config to keep mountType: sshfs.

5. (Optional) Use mount flag for custom volumes

Example:

colima start --mount ~/projects:/projects:rw --mount-type=sshfs

🧪 Check If It Works

Run a quick container to confirm:

docker run -it --rm -v ~/projects:/projects alpine sh

Then inside the container:

ls /projects

If you see your local files, you're good to go.

📝 Bonus: Check Version Compatibility

Make sure your Colima, Lima, and Docker versions are compatible:

colima version
docker --version

If things still don’t work, check the logs:

colima stop
colima start --verbose

🧹 Clean Up (If Needed)

If sshfs keeps failing, you can try switching to 9p (on newer Colima versions):

=colima start --mount-type=9p

It's faster and doesn’t depend on macFUSE, but compatibility can vary.

🚀 Conclusion

Mount errors with Colima on Apple Silicon are super common but easily fixable with the right setup. Reinstall macFUSE, reboot, and use --mount-type=sshfs — that’s usually all it takes.

Let me know if you’ve found a more elegant solution or if something still breaks — happy to update the guide!

Recovery Services Vault: The Comprehensive Solution

Recovery Services Vault is Azure's original and more comprehensive data protection solution. It functions as a storage entity that houses backup data for various Azure services.

Key Features:

• Dual functionality: Supports both Azure Backup and Azure Site Recovery
• Broad resource support: Protects Azure VMs, SQL in VMs, SAP HANA in VMs, Azure Files, on-premises servers, and more
• Cross-region recovery: Enables disaster recovery scenarios across Azure regions
• Legacy compatibility: Supports older workloads and implementations

Best For:

Organizations requiring both backup and disaster recovery capabilities in a single management interface.

Backup Vault: The Specialized Alternative

Backup Vault is a newer, specialized offering focused exclusively on backup operations with enhanced capabilities.

Key Features:

• Backup-specific: Exclusively for backup operations (no disaster recovery functionality)
• Targeted workload support: Designed for specific workloads like Azure Database for PostgreSQL servers, Azure Blobs, and Azure Disks
• Enhanced operational efficiency: Offers more granular backup policies and simplified management
• Cost optimization: Generally more cost-effective for pure backup scenarios
• Modern architecture: Built on newer technologies for better performance

Best For:

Organizations with specific backup needs that don't require disaster recovery functionality.

Key Differences Summarized

Primary Function: Recovery Services Vault handles both backup and disaster recovery, while Backup Vault focuses exclusively on backup operations.

Workload Support: Recovery Services Vault supports a broader range of services including on-premises servers, whereas Backup Vault is specialized for specific Azure services like databases and blob storage.

Architecture: Recovery Services Vault uses the original unified design, while Backup Vault employs a modern, purpose-built architecture optimized for backup operations.

Management: Recovery Services Vault offers more comprehensive but complex management, whereas Backup Vault provides streamlined administration specifically for backup tasks.

Cost Structure: Recovery Services Vault has combined pricing for backup and disaster recovery services, while Backup Vault features optimized pricing specifically for backup operations.

Making the Right Choice

Choose a Recovery Services Vault if you:

• Need both backup and disaster recovery capabilities
• Want to protect a wide range of Azure and on-premises workloads
• Have existing deployments already using this solution

Choose a Backup Vault if you:

• Only need backup functionality (no disaster recovery)
• Are primarily protecting specific Azure database services or disk storage
• Want a more streamlined, cost-effective backup solution

Many organizations employ both solutions, using Recovery Services Vault for comprehensive protection of critical systems and Backup Vault for specialized workloads that only require backup capabilities.

By understanding these differences, you can develop a more efficient, effective, and economical data protection strategy for your Azure environment.

azure-arm.windows2022: 2024-10-05 21:34:00, Error SYSPRP MRTGeneralize:98 - ERROR: Failed ConnectServer
azure-arm.windows2022: 2024-10-05 21:34:02, Error SYSPRP BCD: BiUpdateEfiEntry failed c000000d
azure-arm.windows2022: 2024-10-05 21:34:02, Error SYSPRP BCD: BiExportBcdObjects failed c000000d
azure-arm.windows2022: 2024-10-05 21:34:02, Error SYSPRP BCD: BiExportStoreAlterationsToEfi failed c000000d
azure-arm.windows2022: 2024-10-05 21:34:02, Error SYSPRP BCD: Failed to export alterations to firmware. Status: c000000d

The previous error message was caught from sysprep logs, which I found using the following condition:

if ($imageState -eq 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') {
        break
} elseif ($imageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') {
        $setupActLog = Get-Content "$Env:Windir\System32\Sysprep\Panther\setupact.log" -Raw
        $setupErrLog = Get-Content "$Env:Windir\System32\Sysprep\Panther\setuperr.log" -Raw
        Write-Output "setupact.log:"
        Write-Output $setupActLog
        Write-Output "setuperr.log:"
        Write-Output $setupErrLog
        Write-Output "The unexpected state: $imageState"
        exit 1
}

Overall system state must be IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE for creating an image, if not, we need to check the SysPrep logs.

The Root Cause: Windows Store and Mini-Setup Delays

Sometimes, the Sysprep errors are caused by Windows Store automatic updates running in the background. The mini-setup phase of Sysprep, which generalizes the image, may experience significant delays or failures when Windows Store services update during this time. Microsoft Premier Support has acknowledged this as a potential issue, but they did not provide a definitive fix.

The solution involves disabling the Windows Store automatic updates and ensuring the related services are stopped.

Source: https://learn.microsoft.com/en-us/answers/questions/333299/windows-10-sysprep

The Solution:

First, you need to add a registry key to disable Windows Store automatic updates. Here’s a PowerShell script that handles this task:

# Disable Windows Store Automatic Updates
Write-Host "Adding Registry key to Disable Windows Store Automatic Updates"
$registryPath = "HKLM:\SOFTWARE\Policies\Microsoft\WindowsStore"

If (!(Test-Path $registryPath)) {
    Mkdir $registryPath -ErrorAction SilentlyContinue
    New-ItemProperty $registryPath -Name AutoDownload -Value 2
}
Else {
    Set-ItemProperty $registryPath -Name AutoDownload -Value 2
}

Next, you need to stop the Windows Store installer service, which could be interfering with the Sysprep process.

# Stop WindowsStore Installer Service and set to Disabled
Write-Host "Stopping InstallService"
Stop-Service InstallService

This ensures that the InstallService responsible for handling Windows Store updates is stopped and will not start automatically during the generalization process.

Final Thoughts

Once these steps are complete, Sysprep should be able to run without encountering BCD errors or other issues related to the Windows Store updates. By disabling Windows Store automatic updates and stopping the associated services, you reduce the likelihood of conflicts during Sysprep’s generalization phase.

This fix has helped resolve similar issues for other users, and it’s a valuable solution to try if you’re experiencing Sysprep failures in Windows Server 2019 or 2022. Make sure to share this with others facing the same challenge!

Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: error while creating mount source path '/Users/<username>/Projects/GPT/application/core-data': chown /Users/<username>/Projects/GPT/application/core-data: operation not permitted

is a common issue faced by developers. It occurs due to permission problems when Colima attempts to access certain directories. Here’s a step-by-step guide on how I resolved this problem using Colima.

Understanding the Issue

The core of the issue lies in Colima inability to change the ownership of the specified directory. This usually happens due to restrictive macOS permissions. However, it requires a bit of configuration to handle directory permissions correctly.

Solution Overview

To resolve the issue, you need to configure Colima to use the 9p mount type and explicitly define the directory mounts both with absolute paths and with the ~ notation. This ensures that Colima can correctly map and cache the directories, granting the necessary permissions.

Step-by-Step Solution

Step 1: Update Colima Configuration

First, you need to update the Colima configuration file. Open or create the override configuration file at /Users/<username>/.colima/_lima/_config/override.yaml. Replace <username> with your actual macOS username.

Add the following configuration to the file:

mountType: 9p
mounts:
  - location: "/Users/<username>"
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap
  - location: "~"
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap
  - location: /tmp/colima
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap

This configuration ensures that both the absolute path and the home directory symbol (~) are correctly mapped and accessible.

Step 2: Restart Colima

After updating the configuration, restart Colima to apply the changes. Use the following commands in your terminal:

colima delete
colima start --mount-type 9p

The colima delete command stops and removes the existing Colima instance, while colima start --mount-type 9p restarts it with the new mount type and configuration.

Explanation of Configuration

• mountType: 9p: Specifies the use of the 9p protocol for mounts. This protocol allows for better handling of file permissions and caching.
• location: Defines the directories to be mounted. Both the absolute path (/Users/<username>) and the home directory symbol (~) are included to cover all scenarios.
• writable: true: Ensures that the directories are writable.
• securityModel: mapped-xattr: Uses extended attributes for security, mapping file permissions appropriately.
• cache: mmap: Specifies the caching mechanism to improve performance.

Conclusion

By configuring Colima to handle directory mounts correctly, you can resolve permission errors and ensure a smoother development experience on macOS. This guide provides a straightforward solution to a common problem, enabling you to focus on your projects without interruption.

Feel free to share this guide with your peers and help others overcome the same challenge. Happy coding!