Microsoft Foundry Skill

When to Use This Skill

• Discover and deploy AI models from the Microsoft Foundry catalog
• Build RAG applications using knowledge indexes and vector search
• Create AI agents with tools like Azure AI Search, web search, or custom functions
• Evaluate agent performance using built-in evaluators
• Set up monitoring and continuous evaluation for production agents
• Troubleshoot issues with deployments, agents, or evaluations

Prerequisites

Azure Resources

• An Azure subscription with an active account
• Appropriate permissions to create Microsoft Foundry resources (e.g., Azure AI Owner role)
• Resource group for organizing Foundry resources

Tools

• Azure CLI installed and authenticated (

  az login

  )
• Azure Developer CLI (azd) for deployment workflows (optional but recommended)

Language-Specific Requirements

• Python: See language/python.md for Python SDK setup, authentication, and examples

Core Workflows

1. Getting Started - Model Discovery and Deployment

Use Case

Step 1: List Available Resources

Bash

# List all Microsoft Foundry resources in subscription
az resource list \
  --resource-type "Microsoft.CognitiveServices/accounts" \
  --query "[?kind=='AIServices'].{Name:name, ResourceGroup:resourceGroup, Location:location}" \
  --output table

# List resources in a specific resource group
az resource list \
  --resource-group <resource-group-name> \
  --resource-type "Microsoft.CognitiveServices/accounts" \
  --output table

foundry_resource_get

Step 2: Browse Model Catalog

• Some models support free playground for prototyping without costs
• Models can be filtered by publisher (e.g., OpenAI, Meta, Microsoft)
• Models can be filtered by license type
• Model availability varies by region

foundry_models_list

• List all models:

  foundry_models_list()

• List free playground models:

  foundry_models_list(search-for-free-playground=true)

• Filter by publisher:

  foundry_models_list(publisher="OpenAI")

• Filter by license:

  foundry_models_list(license="MIT")

• Models with free playground support can be used for prototyping at no cost
• Some models support GitHub token authentication for easy access
• Check model capabilities and pricing before production deployment

Step 3: Deploy a Model

Bash

# Deploy a model (e.g., gpt-4o)
az cognitiveservices account deployment create \
  --name <foundry-resource-name> \
  --resource-group <resource-group-name> \
  --deployment-name gpt-4o-deployment \
  --model-name gpt-4o \
  --model-version "2024-05-13" \
  --model-format OpenAI \
  --sku-capacity 10 \
  --sku-name Standard

# Verify deployment status
az cognitiveservices account deployment show \
  --name <foundry-resource-name> \
  --resource-group <resource-group-name> \
  --deployment-name gpt-4o-deployment

foundry_models_deploy

• resource-group

  : Resource group name

• deployment

  : Deployment name

• model-name

  : Model to deploy (e.g., "gpt-4o")

• model-format

  : Format (e.g., "OpenAI")

• azure-ai-services

  : Foundry resource name

• model-version

  : Specific version

• sku-capacity

  : Capacity units

• scale-type

  : Scaling type

provisioningState

Succeeded

• Insufficient quota
• Region capacity limitations
• Permission issues

Step 4: Get Resource Endpoint

foundry_resource_get

https://<resource>.services.ai.azure.com/api/projects/<project-name>

2. Building RAG Applications with Knowledge Indexes

Use Case

Understanding RAG and Knowledge Indexes

1. Retrieving relevant documents from a knowledge base
2. Augmenting the AI prompt with retrieved context
3. Generating responses grounded in factual information

• Supports keyword, semantic, vector, and hybrid search
• Enables efficient retrieval of relevant content
• Stores metadata for better citations (document titles, URLs, file names)
• Integrates with Azure AI Search for production scenarios

Step 1: List Existing Knowledge Indexes

foundry_knowledge_index_list

Step 2: Inspect Index Schema

foundry_knowledge_index_schema

• Field definitions and data types
• Searchable attributes
• Vectorization configuration
• Retrieval mode support (keyword, semantic, vector, hybrid)

Step 3: Create an Agent with Azure AI Search Tool

1. Initialize the AI Project Client with your project endpoint and credentials
2. Get the Azure AI Search connection from your project
3. Create the agent with:
   • Agent name
   • Model deployment
   • Clear instructions (see best practices below)
   • Azure AI Search tool configuration with:
     • Connection ID
     • Index name
     • Query type (HYBRID recommended)

• Always request citations in agent instructions
• Use hybrid search (AzureAISearchQueryType.HYBRID) for best results
• Instruct the agent to say "I don't know" when information isn't in the index
• Format citations consistently for easy parsing

Step 4: Test the RAG Agent

1. Query the agent with a test question
2. Stream the response to get real-time output
3. Capture citations from the response annotations
4. Validate that citations are properly formatted and included

AI_SEARCH_INDEX_NAME

3. Creating Your First AI Agent

Use Case

Step 1: List Existing Agents

foundry_agents_list

Step 2: Create a Basic Agent

• Model deployment name: The model to use
• Agent name: Unique identifier
• Instructions: Clear, specific guidance for the agent's behavior

Step 3: Create an Agent with Custom Function Tools

1. Define custom functions with clear docstrings describing their purpose and parameters
2. Create a function toolset with your custom functions
3. Create the agent with the toolset and instructions on when to use the tools

Step 4: Create an Agent with Web Search

• Optionally specify user location for localized results
• Provide instructions to always cite web sources

Step 5: Interact with the Agent

1. Create a conversation thread for the agent interaction
2. Add user messages to the thread
3. Run the agent to process the messages and generate responses
4. Check run status for success or failure
5. Retrieve messages to see the agent's responses
6. Cleanup by deleting the agent when done

1. Clear Instructions: Provide specific, actionable instructions
2. Tool Selection: Only include tools the agent needs
3. Error Handling: Always check

   run.status

   for failures
4. Cleanup: Delete agents/threads when done to manage costs
5. Rate Limits: Handle rate limit errors gracefully (status code 429)

4. Evaluating Agent Performance

Use Case

Understanding Agent Evaluators

1. IntentResolutionEvaluator: Measures how well the agent identifies and understands user requests (score 1-5)
2. TaskAdherenceEvaluator: Evaluates whether responses adhere to assigned tasks and system instructions (score 1-5)
3. ToolCallAccuracyEvaluator: Assesses whether the agent makes correct function tool calls (score 1-5)

• {metric_name}

  : Numerical score (1-5, higher is better)

• {metric_name}_result

  : "pass" or "fail" based on threshold

• {metric_name}_threshold

  : Binarization threshold (default or user-set)

• {metric_name}_reason

  : Explanation of the score

Step 1: Single Agent Run Evaluation

foundry_agents_query_and_evaluate

• Agent ID
• Query text
• Project endpoint
• Azure OpenAI endpoint and deployment for evaluation
• Comma-separated list of evaluators to use

{
  "response": "The weather in Seattle is currently sunny and 22°C.",
  "evaluation": {
    "intent_resolution": 5.0,
    "intent_resolution_result": "pass",
    "intent_resolution_threshold": 3,
    "intent_resolution_reason": "The agent correctly identified the user's intent to get weather information and provided a relevant response.",
    "task_adherence": 4.0,
    "task_adherence_result": "pass",
    "tool_call_accuracy": 5.0,
    "tool_call_accuracy_result": "pass"
  }
}

Step 2: Evaluate Existing Response

foundry_agents_evaluate

Step 3: Batch Evaluation

1. Convert agent thread data to evaluation format
2. Prepare evaluation data from multiple thread IDs
3. Set up evaluators with appropriate configuration
4. Run batch evaluation and view results in the Foundry portal

Interpreting Evaluation Results

• 5: Excellent - Agent perfectly understood and executed the task
• 4: Good - Minor issues, but overall successful
• 3: Acceptable - Threshold for passing (default)
• 2: Poor - Significant issues with understanding or execution
• 1: Failed - Agent completely misunderstood or failed the task

5. Troubleshooting Common Issues

Deployment Issues

Bash

# Check deployment status and details
az cognitiveservices account deployment show \
  --name <resource-name> \
  --resource-group <resource-group> \
  --deployment-name <deployment-name> \
  --output json

# Check account quota
az cognitiveservices account show \
  --name <resource-name> \
  --resource-group <resource-group> \
  --query "properties.quotaLimit"

• Insufficient quota in the region
• Region at capacity for the model
• Permission issues

1. Check quota limits in Azure Portal
2. Request quota increase if needed
3. Try deploying to a different region
4. Verify you have appropriate RBAC permissions

Agent Response Issues

1. Check agent instructions explicitly request citations
2. Verify the tool choice is set to "required" or "auto"
3. Confirm the Azure AI Search connection is configured correctly

[message_idx:search_idx†source]

foundry_knowledge_index_list

1. Verify

   AI_SEARCH_INDEX_NAME

   environment variable matches actual index name
2. Check the connection points to correct Azure AI Search resource
3. Ensure index has been created and populated

Bash

# Assign Search Index Data Contributor role to managed identity
az role assignment create \
  --assignee <managed-identity-principal-id> \
  --role "Search Index Data Contributor" \
  --scope /subscriptions/<subscription-id>/resourceGroups/<rg>/providers/Microsoft.Search/searchServices/<search-service>

# Verify role assignment
az role assignment list \
  --assignee <managed-identity-principal-id> \
  --output table

Evaluation Issues

• No recent agent traffic
• Time range excludes the data
• Ingestion delay

1. Generate new agent traffic (test queries)
2. Expand the time range filter in the dashboard
3. Wait a few minutes for data ingestion
4. Refresh the dashboard

1. Verify the evaluation rule is enabled
2. Confirm agent traffic is flowing
3. Check project managed identity has Azure AI User role
4. Verify OpenAI endpoint and deployment are accessible

Rate Limiting and Capacity Issues

Rate limit is exceeded

Bash

# Check current quota usage
az cognitiveservices usage list \
  --name <resource-name> \
  --resource-group <resource-group>

# Request quota increase (manual process in portal)
echo "Request quota increase in Azure Portal under Quotas section"

Request quota increase (manual process in portal)

**Best Practices:**
- Implement exponential backoff retry logic
- Use Dynamic Quota when available
- Monitor quota usage proactively
- Consider multiple deployments across regions

## Quick Reference

### Common Environment Variables

```bash
# Foundry Project
PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>
MODEL_DEPLOYMENT_NAME=gpt-4o

# Azure AI Search (for RAG)
AZURE_AI_SEARCH_CONNECTION_NAME=my-search-connection
AI_SEARCH_INDEX_NAME=my-index

# Evaluation
AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com
AZURE_OPENAI_DEPLOYMENT=gpt-4o

Useful MCP Tools Quick Reference

• foundry_resource_get

  - Get resource details and endpoint

• foundry_models_list

  - Browse model catalog

• foundry_models_deploy

  - Deploy a model

• foundry_models_deployments_list

  - List deployed models

• foundry_knowledge_index_list

  - List knowledge indexes

• foundry_knowledge_index_schema

  - Get index schema

• foundry_agents_list

  - List agents

• foundry_agents_connect

  - Query an agent

• foundry_agents_query_and_evaluate

  - Query and evaluate

• foundry_openai_chat_completions_create

  - Create chat completions

• foundry_openai_embeddings_create

  - Create embeddings

Language-Specific Quick References

• Python: See language/python.md

Additional Resources

Documentation Links

• Microsoft Foundry Documentation
• Microsoft Foundry Quickstart
• RAG and Knowledge Indexes
• Agent Evaluation Guide

GitHub Samples

• Microsoft Foundry Samples
• Azure Search OpenAI Demo
• Azure Search Classic RAG