Q01 - Question
During an alignment investigation, the correlation between human scores and Intent Resolution evaluator scores is low. You calculate Cohen's kappa for two human evaluators who scored the same examples, and the result is 0.35. What should you do next?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Create a custom evaluator immediately to replace the built-in Intent Resolution evaluator.
- B. Lower the correlation threshold so the automated gate can continue to run.
- C. Add more adversarial examples to the dataset and rerun the shadow rating with the same human rubric.
- D. Resolve the inconsistency between human evaluators by training them and clarifying the rubric before you adjust the automated evaluators.
D is correct.
Explanation: The evidence supports D because a kappa value below 0.4 indicates poor agreement between human evaluators. You must resolve human inconsistency first by investing in evaluator training and clearer rubrics. If you automate inconsistent criteria, you scale the inconsistency.
A is incorrect: Custom evaluators are for cases where built-in evaluators miss domain-specific dimensions, but you must first confirm that the human scores are consistent.
B is incorrect: Lowering the threshold hides the misalignment and does not address its cause, leading to trust in automated scores that do not reflect consistent human judgment.
C is incorrect: Adding adversarial examples changes dataset coverage but does not fix the human disagreement. The same unclear rubric would still produce inconsistent human scores.
Q02 - Question
You evaluate an agent that calls tools. You need to detect behaviors that violate a policy of prohibited actions and detect exposure of sensitive information such as financial data or personal identifiers. You evaluate an agent target and can provide tool-call information. Which two evaluators should you configure? Select two.
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Multiple choice
- A. builtin.violence
- B. builtin.prohibited_actions
- C. builtin.indirect_attack
- D. builtin.sensitive_data_leakage
B and D are correct.
Explanation: builtin.prohibited_actions measures whether an agent engages in behaviors that violate explicitly disallowed actions or tool uses. builtin.sensitive_data_leakage measures an agent's vulnerability to exposing sensitive information. Both are in preview, support agent targets only, and take tool-call information through the tool_calls field or a messages array.
A is incorrect: builtin.violence measures violent or threatening language. It does not evaluate prohibited tool use or sensitive data exposure.
C is incorrect: builtin.indirect_attack measures whether a response fell for an indirect jailbreak attempt, and it applies to models only. It does not evaluate prohibited actions or sensitive data leakage.
Q03 - Question
You evaluate a retrieval-augmented generation (RAG) application. You need a binary pass/fail result that shows whether responses are grounded in the retrieved context. You do not want to provide a model deployment for the evaluator. You also accept that the evaluator is in preview. Which built-in evaluator should you use?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Groundedness Pro (preview)
- B. Groundedness
- C. Relevance
- D. Document Retrieval
A is correct.
Explanation: Groundedness Pro measures whether the response is grounded in the retrieved context by using the Azure AI Content Safety service. It returns a binary pass/fail and does not require a model deployment. It is in preview, and Microsoft doesn't recommend preview features for production workloads.
B is incorrect: Groundedness uses a model-based judgment and returns a score from 1 to 5. It does not return a binary result.
C is incorrect: Relevance measures how relevant the response is to the query. It does not measure grounding in the retrieved context.
D is incorrect: Document Retrieval measures the accuracy of retrieval results given ground truth. It does not evaluate whether the response is grounded.
Q04 - Question
Adventure Works has validated that its automated evaluators align with human ratings. The team now wants to design the production phase of its human-in-the-loop (HITL) process for prompt updates. Evaluators must score every test case, and human reviewers must have a focused role. Which design meets these requirements?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Run automated evaluation occasionally and use human evaluation at other times, with no shared workflow.
- B. Run the automated evaluation on every prompt update, send responses that score below the threshold plus a random 10% sample to human reviewers, and run monthly correlation checks.
- C. Have human reviewers score all 500 test cases for every prompt update and use automated scores only as a reference.
- D. Remove human reviewers from the process after validation and rely only on the automated gate.
B is correct.
Explanation: The evidence supports B because in the production phase, automation scores everything as an automated gate. Humans review responses that score below the threshold and a random 10% sample. Monthly correlation checks detect evaluator drift.
A is incorrect: HITL is a system where automation provides comprehensive coverage and humans provide strategic oversight, not running them separately without a shared workflow.
C is incorrect: Reviewing all 500 test cases manually takes days and reduces consistency as reviewers tire, failing to use automation for volume.
D is incorrect: Automated evaluation lacks human context and requires validation. Without human review and drift checks, you cannot detect misalignment as the system changes.
Q05 - Question
A team builds an evaluation dataset from customer support tickets and live agent conversation logs. The logs contain customer names, email addresses, and phone numbers. They also contain few unusual requests and no prompt injection attempts. Which TWO actions should the team take before it runs evaluations in Microsoft Foundry?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Multiple choice
- A. Anonymize the personal data, for example by using Azure Language PII detection to identify and redact it.
- B. Keep the dataset to production logs only, because production data provides sufficient edge cases and adversarial examples.
- C. Add edge cases and adversarial examples that domain experts create manually to cover the missing scenario types.
- D. Make edge cases about 50% of the dataset so unusual scenarios are covered comprehensively.
A and C are correct.
Explanation: The evidence supports A and C because preparing evaluation data includes removing personal data such as names, email addresses, and phone numbers using tools like Azure Language PII detection. Furthermore, production data rarely includes enough edge cases or adversarial examples, which are best created manually using domain expertise.
B is incorrect: Production data provides authentic common scenarios and variations, but it rarely includes sufficient edge cases or adversarial examples.
D is incorrect: A balanced dataset uses about 5-10% edge cases, not 50%. Common scenarios should make up 60-70% to represent the quality baseline.
Q06 - Question
A Python script creates a cloud evaluation in Microsoft Foundry. The JSONL dataset uses the field name "Question" (capitalized). The groundedness evaluator uses the data mapping "query": "{{item.question}}". The evaluation run ends with a status of failed. Model quota and dataset access are confirmed to be correct. What should you do?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Increase the polling interval so the run has more time to complete.
- B. Remove "query" from the required fields in the data schema so validation passes.
- C. Make the field name in the data mapping match the dataset field exactly, or rename the dataset field to match the mapping.
- D. Replace the built-in groundedness evaluator with a different built-in evaluator.
C is correct.
Explanation: The evidence supports C because field names in data_mapping are case-sensitive and must match the JSONL dataset exactly. A mapping of {{item.question}} does not match a dataset field named "Question", causing the evaluation to fail.
A is incorrect: Polling only checks the run status. It does not change the result of a run that failed because of an invalid mapping.
B is incorrect: The schema defines which fields must be present. Evaluators still need their inputs, and the mismatch between the mapping and the dataset remains.
D is incorrect: Another built-in evaluator still needs data mapped from your dataset fields, so the mapping error would remain.
Q07 - Question
Your team deployed a generative AI application to production. You want to detect system drift by running quality and safety evaluations against the same fixed test dataset at regular intervals. Which post-production monitoring approach should you use?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Continuous evaluation of production traffic at a sampled rate
- B. Scheduled evaluation that uses test datasets
- C. Scheduled red teaming
- D. Base model selection by using the Microsoft Foundry benchmark
B is correct.
Explanation: Scheduled evaluation runs quality and safety evaluations on a schedule by using test datasets. Use it to detect system drift, because the same dataset is evaluated each time.
A is incorrect: Continuous evaluation assesses production traffic at a sampled rate. It does not run a fixed test dataset at regular intervals.
C is incorrect: Scheduled red teaming runs adversarial tests to probe for safety and security vulnerabilities. It does not measure drift in quality and safety scores on a test dataset.
D is incorrect: The Microsoft Foundry benchmark compares models during base model selection. It is not a post-production monitoring approach.
Q08 - Question
You plan an evaluation run that scores entire multi-turn conversations. You set evaluation_level to conversation. One of the evaluators you selected supports only the turn level. You need the run to be valid. What should you do?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Keep all evaluators in one run. Each evaluator then uses its own supported level.
- B. Omit evaluation_level so that each evaluator uses the level it supports.
- C. Set evaluation_level to turn so that conversation-level scoring is applied automatically.
- D. Check supported_evaluation_levels in the evaluator catalog. Include only evaluators that support conversation in this run.
D is correct.
Explanation: Each evaluator supports specific evaluation levels, which are listed in the supported_evaluation_levels field in the evaluator catalog. All evaluators in a run must support the evaluation_level that you set. Verify the supported levels and remove evaluators that don't support conversation from the conversation-level run.
A is incorrect: You can't mix evaluators with incompatible levels in the same evaluation run. The run does not apply a different level per evaluator.
B is incorrect: If you omit evaluation_level, the default is turn. The run then evaluates individual responses, not entire conversations.
C is incorrect: Setting evaluation_level to turn evaluates individual agent responses. It does not score the full interaction.
Q09 - Question
Adventure Works wants a GitHub Actions workflow that evaluates prompt changes before they merge. The workflow must authenticate to Microsoft Foundry without stored keys and post metrics as a pull request comment. Which configuration meets these requirements?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Trigger on pull_request events for the prompts and config paths. Grant id-token: write, contents: read, and pull-requests: write. Use azure/login with federated credentials.
- B. Trigger on push events to the main branch. Grant id-token: write and use azure/login with federated credentials, then post results after the merge.
- C. Trigger on pull_request events for the prompts and config paths. Grant only contents: read and use azure/login with federated credentials.
- D. Run the evaluation script locally for each change and paste the metrics into the pull request by hand.
A is correct.
Explanation: The evidence supports A because a pull_request trigger scoped to the prompt and configuration paths runs the evaluation before changes merge. The id-token: write permission supports federated identity credentials for keyless authentication with azure/login. The pull-requests: write permission lets the workflow post the metrics as a pull request comment.
B is incorrect: A push to main runs after the merge. The workflow must run evaluations on pull requests so the results can guide the merge decision. This configuration also does not grant pull-requests: write.
C is incorrect: The contents: read permission alone does not provide id-token: write for federated authentication or pull-requests: write for posting comments.
D is incorrect: A manual local run does not create an automated quality gate and does not use the workflow to post results to the pull request.
Q10 - Question
A multi-step agent built with LangGraph sometimes responds slowly. Developers need to see each LLM call, tool invocation, and agent decision, and identify which step causes the delay. The data must be available in Azure Monitor Application Insights. Which Microsoft Foundry capability should you configure?
Domain: Implement generative AI quality assurance and observability (10–15%) Type: Single choice
- A. Built-in evaluators for coherence and fluency
- B. Azure Monitor alerts on output quality thresholds
- C. Distributed tracing that uses OpenTelemetry standards
- D. The AI red teaming agent
C is correct.
Explanation: Distributed tracing captures the execution flow of AI applications, including LLM calls, tool invocations, agent decisions, and inter-service dependencies. It is built on OpenTelemetry standards, integrates with Azure Monitor Application Insights, and supports LangGraph. Use it to find performance bottlenecks in multi-step agent behavior.
A is incorrect: Coherence and fluency evaluators measure response quality. They do not show the execution flow or where latency occurs.
B is incorrect: Azure Monitor alerts send notifications when outputs fail quality thresholds or produce harmful content. They do not show individual steps in the agent execution.
D is incorrect: The AI red teaming agent simulates attacks to find safety and security vulnerabilities before deployment. It does not diagnose performance bottlenecks.