Q01 - Question
You are building a RAG solution over proprietary documents. The documents contain many organization-specific terms that are unlikely to be in a general embedding model's vocabulary. No domain-specific embedding model is available, and top-ranked general models return poor vector search results on your content. What should you do?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Select the largest available embedding model, because larger models always perform better
- B. Apply dimensionality reduction to the embeddings to correct the vocabulary mismatch
- C. Fine-tune a general embedding model with your domain vocabulary, and evaluate it by running vector searches with real queries and content
- D. Select the top model on the Hugging Face leaderboard and skip testing on your own data
C is correct.
Explanation: When a model's vocabulary does not cover your terms, it can split unknown words into subwords whose meanings are far from the original word. This produces poorer vector matches. If no domain-specific model exists or it does not perform well, fine-tune a general embedding model with your domain vocabulary. Then evaluate retrieval performance with real-world queries and content. Fine-tuning can degrade performance if the training data is poor, so evaluation is required.
A is incorrect: Larger models are not always better. They can increase cost, including vector storage and compute, without improving performance on your data.
B is incorrect: Dimensionality reduction is a post-processing step that you apply after you generate embeddings to optimize storage and retrieval. It does not add missing vocabulary to the model.
D is incorrect: Leaderboard rankings are a starting point. Public benchmarks often use academic datasets and might not apply to your business data, so you must validate the model through experimentation.
Q02 - Question
Three evaluators independently score 10 agent responses on three metrics, which gives 30 scoring opportunities. The scores are within one point of each other in 21 of the 30 cases. The team's reliability target is 80% agreement within one point. What should the team do?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Review the responses with divergent scores, update the rubrics to clarify those cases, and conduct additional calibration training.
- B. Accept the scores, because 70% agreement is sufficient for optimization decisions.
- C. Average the three scores for each response and use the averages to compare variants.
- D. Replace the rubrics with a single overall quality score to reduce disagreement.
A is correct.
Explanation: 21 of 30 is 70% agreement within one point. This is below the 80% target. Review the divergent responses to find what caused the disagreement, clarify the rubrics, and conduct more calibration training.
B is incorrect: 70% is below the 80% target. Accepting the scores means optimization decisions rest on individual evaluator preferences.
C is incorrect: Averaging scores does not fix the different interpretations of the criteria. It hides the inconsistency instead of correcting it.
D is incorrect: A single overall score removes the defined score levels and examples that create a shared understanding. Rubrics with concrete examples at each level reduce ambiguity.
Q03 - Question
You are monitoring a supervised fine-tuning job in Microsoft Foundry. Training loss keeps decreasing, but the validation loss and validation token accuracy diverge from the training metrics. This indicates possible overfitting. Which two changes should you try in the next fine-tuning job?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Multi-select
- A. Increase n_epochs so the model sees the training data more times
- B. Train with fewer epochs
- C. Increase learning_rate_multiplier above the recommended 0.02 to 0.2 range
- D. Use a smaller learning_rate_multiplier
B and D are correct.
Explanation: When training and validation metrics diverge, the model might be overfitting. Train with fewer epochs or use a smaller learning rate multiplier. A smaller learning rate can help avoid overfitting. You can also review checkpoints, which might provide snapshots from before overfitting.
A is incorrect: More epochs means more full passes through the training data. This can increase overfitting when validation metrics already diverge.
C is incorrect: Larger learning rates do not address overfitting. The recommended range to experiment with is 0.02 to 0.2, and a smaller value is what helps avoid overfitting.
Q04 - Question
A regulated workload requires that training data for a Microsoft Foundry fine-tuning job stays in the region of the current Foundry resource. The selected base model supports Standard-tier training. Which training type should you select?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Standard
- B. Global
- C. Developer
- D. Any tier, because the training type doesn't affect where training occurs
A is correct.
Explanation: Standard training occurs in the current Foundry resource's region and provides guarantees for data residency. It is the right choice when data must remain in a specific region.
B is incorrect: Global training is more affordable and has faster queue times because it uses capacity beyond your current region. Data and weights are copied to the region where training occurs, which conflicts with a residency requirement.
C is incorrect: Developer training uses idle capacity and has no data residency guarantees. It also has no latency or SLA guarantees, and jobs can be preempted and resumed later.
D is incorrect: The training type determines where training occurs and whether residency is guaranteed. Only Standard provides the residency guarantee.
Q05 - Question
Your team is adding grounding to an internal assistant. The team already has orchestration code that sends a single query to Azure AI Search and passes the results to a deployed LLM. Policy allows only generally available (GA) features in production. Simplicity and response speed matter more than advanced relevance. Which approach should you use?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Agentic retrieval with a knowledge base and LLM-based query planning
- B. Classic RAG with hybrid queries and optional semantic ranking
- C. Send all documentation to the LLM with each request and rely on its context window
- D. Keyword-only queries with no vector search and no semantic ranking
B is correct.
Explanation: Classic RAG fits this scenario. Your application sends a single query to Azure AI Search and orchestrates the handoff to the LLM separately. This approach uses GA features, is simpler, is faster because the LLM does not take part in query planning, and preserves your existing orchestration code. Hybrid queries combine keyword and vector search for better recall, and semantic ranking is optional.
A is incorrect: Agentic retrieval uses LLM-based query planning, which is in preview. It does not meet the GA-only requirement and adds components that your simplicity and speed priorities do not need.
C is incorrect: LLMs accept a limited number of tokens. Sending everything wastes tokens and can degrade quality. Retrieval should return relevant, concise results.
D is incorrect: Keyword search fails when queries do not match document terminology. Classic RAG improves recall by combining keyword and vector search, so keyword-only queries give up that benefit.
Q06 - Question
You plan to use the synthetic data generation preview in Microsoft Foundry to create Simple Q&A training data from a domain policy document. This is your first generation job for this document. How should you proceed?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Generate the maximum of 1,000 samples first, so you only run the job once
- B. Add real customer records to the reference file so the generated examples look realistic
- C. Skip review and use the dataset directly in fine-tuning, because the output is already formatted as JSONL
- D. Generate a small sample first, review the outputs, and adjust the reference file or generation parameters before scaling up
D is correct.
Explanation: Start with a smaller sample size and review the outputs. Then adjust your reference file or generation parameters before you scale up. This helps you avoid unnecessary costs and confirms that the data meets your requirements. The sample count can be between 50 and 1,000. Because this is a preview, always validate outputs before production use.
A is incorrect: Generating 1,000 samples first can increase cost, and you would not know whether the reference file produces good data until after the full job completes.
B is incorrect: Do not include personally identifiable information (PII) or other sensitive data in the reference file. Keep the file clean and focused on the target domain.
C is incorrect: Although the output is formatted as JSONL files compatible with Foundry fine-tuning, the quality depends on the reference file. Preview functionality can change, so you must review and validate the data before use.
Q07 - Question
A team enables streaming (stream: true) on a conversational agent. They plan to compare it with the non-streaming baseline. Which two actions should they take in the experiment design? Select two answers.
Domain: Optimize generative AI systems and model performance (10–15%) Type: Multi select
- A. Expect streaming to reduce the total end-to-end response time, and use that as the only performance metric.
- B. Measure time-to-first-token and compare it with a threshold defined for streaming responses.
- C. Set the success thresholds after reviewing the results so that they match what the variants achieved.
- D. Define the quality, cost, and performance thresholds before running the experiment.
B and D are correct.
Explanation: Streaming does not change total response time, but it changes how quickly users see the agent start to respond. Time-to-first-token measures that perceived responsiveness. Defining thresholds before you run experiments, such as a time-to-first-token target, prevents you from rationalizing disappointing results.
A is incorrect: Streaming does not change total response time. End-to-end response time alone does not show the improvement that streaming provides.
C is incorrect: Setting thresholds after you see the results allows you to rationalize results. Define success criteria before you run the experiment.
Q08 - Question
A team is preparing a calibration exercise for new human evaluators of a Trail Guide Agent. They have a rubric with defined score levels. How should they build and run the exercise?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Select 20 high-scoring responses and show the evaluators the intended scores before they score.
- B. Select five to eight responses that span the score range, including ambiguous ones. Have each evaluator score them independently, then compare scores and discuss differences.
- C. Have the evaluators score each response together as a group so that they reach one score immediately.
- D. Use the scores of the most experienced evaluator as the answer key and skip the group discussion.
B is correct.
Explanation: A calibration set of five to eight responses should include clear examples of low and high scores plus ambiguous responses between levels. Independent scoring shows how each evaluator interprets the rubric. Discussing scores that differ by more than one point identifies where to clarify the rubric.
A is incorrect: High-scoring responses only do not span the score range, and showing the intended scores in advance prevents you from seeing how evaluators interpret the rubric.
C is incorrect: Group scoring hides individual interpretation differences. Evaluators must score independently first.
D is incorrect: A single evaluator's scores do not reveal where the rubric is unclear. Discussion of differing scores is needed to update the rubric descriptions.
Q09 - Question
A team manages a customer-facing agent. They want to test GPT-4 mini and also a shorter system prompt. They need to know which change affects quality, cost, and response time. Which experiment design should they use?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Create one variant that uses GPT-4 mini and the shorter prompt together, and compare it to the baseline.
- B. Test GPT-4 mini only on the easiest test prompts to confirm that it responds faster.
- C. Run the baseline and a GPT-4 mini variant with the same prompt against the same test prompts. Test the shorter prompt as a separate variant.
- D. Run each variant against a different set of test prompts so that each variant is tested on new scenarios.
C is correct.
Explanation: Change one variable at a time so you can attribute any difference to a single modification. Running every variant against the same test prompts keeps the comparison controlled. After you validate each change, you can test combinations of the successful variants.
A is incorrect: Changing the model and the prompt together makes it impossible to tell which change caused a difference in quality, cost, or response time.
B is incorrect: Test prompts must cover diverse use cases and edge cases. Testing only easy prompts hides weaknesses on complex trip planning queries.
D is incorrect: Different test prompts per variant remove the shared basis for comparison. Each variant must run against the same test prompts.
Q10 - Question
A team tests a smaller model on the branch experiment/gpt4o-mini-model. The evaluation.csv file shows an average quality score of 4.1. The success threshold is 4.2. The team has committed the results. What should they do next?
Domain: Optimize generative AI systems and model performance (10–15%) Type: Single choice
- A. Merge the branch to main because the quality score is close to the threshold.
- B. Merge the branch to main and tag it, then revert the merge if customers report problems.
- C. Delete the branch without recording why, because the experiment failed.
- D. Document that quality fell below the threshold, then decide whether to keep the branch as a record or delete it.
D is correct.
Explanation: The experiment does not meet the quality threshold, so it is not promoted to main. Document why it failed before you decide what to do with the branch. You can keep the branch to show future teams what did not work, or delete it if the results are already committed and documented.
A is incorrect: A score of 4.1 is below the 4.2 threshold. Only experiments that meet the success criteria are merged to main.
B is incorrect: Merging a variant that does not meet the threshold promotes it to production without evidence that it is acceptable.
C is incorrect: Deleting the branch is acceptable only when the evaluation results are already committed and documented. The reason for the failure must be recorded.