Microsoft is building imaginary workplaces to test an assistant intended for real ones. Its October 30 research report describes synthetic organizations containing employees, documents, meetings and email histories, designed to evaluate Copilot without feeding customer records into the testing pipeline. That creates a useful opportunity: developers can deliberately build situations that would be difficult to collect safely from an actual office.
The harder question is what makes those invented offices convincing. TENS Magazine’s analysis is that workplace AI tests should be judged by the boundaries they preserve, including cases where a plausible answer exists but the person asking should not receive it. A realistic document collection is a starting point. A convincing test must also make the assistant fail when it crosses the wrong boundary.
A realistic workplace needs an unfair question
Microsoft says its Industry Evaluation Data Generation Platform draws on public occupational information, job advertisements and company filings to construct synthetic environments. It describes a software business among its examples, with project coordination and development discussions. The company reports a 39 percent improvement in an internal composite measure of synthetic-data quality against its initial approach. That is a measure of the generated environments, not a demonstrated 39 percent improvement in employees’ work or Copilot’s answers.
An earlier research project offers a useful comparison. EnterpriseBench, published at EMNLP in November 2025 by Harsh Vishwakarma and colleagues, presents 500 enterprise tasks and explicitly includes fragmented information, access hierarchies and work spanning organizational functions. Its authors reported a best task-completion result of 41.8 percent among the agents they tested. That historical result is not a score for today’s Copilot and cannot be compared directly with Microsoft’s data-quality measure.
Read together, the two sources expose a distinction worth preserving: making the office believable and making the evaluation discriminating are different achievements. An assistant could navigate convincing emails successfully while mishandling permission boundaries. Conversely, a deliberately sparse test could reveal that failure with only a few carefully chosen documents. The important unit of evidence is the behavior under examination, not simply the volume of generated material.
The same answer, two different permissions
TENS proposes a paired test. Create a fictional project decision, then ask the same question through two synthetic employee identities. Give one identity access to the decision record and withhold that access from the other. The first response should recover the supported answer; the second should avoid disclosing it. Keep the question, documents and decision unchanged so that permission is the variable being tested.
This is an illustrative evaluation design, not an experiment TENS has run or a claim that Microsoft omits such checks. Its value is that it challenges a tempting shortcut: counting every correct-looking answer as a success. In the restricted version, repeating the private decision would be a failure even if every word were factually accurate. A refusal can be the successful outcome.
The score should therefore separate authorized answers, unauthorized disclosures and unnecessary refusals. Combining all three into a single average would make diagnosis harder. A system that refuses everything might avoid disclosure while being useless; one that answers everything might look helpful while violating the task’s access conditions. Each error calls for a different correction.
Test what the assistant cannot establish
A second pair would change evidence availability. In one version, include the final decision. In another, retain the discussion but remove the decision record. The expected behavior should change from answering to explaining the evidentiary gap. This tests whether the assistant distinguishes a proposal from an outcome, without requiring any confidential real-world meeting transcript.
Evaluators should write that expected difference before running the assistant. Otherwise, a fluent response can pull the scoring rule toward whatever the system happened to produce. For the missing-record case, the evaluator can specify which claims remain supported and which conclusion is unavailable. The assessment then has something firmer than a general impression of helpfulness.
A third pair would preserve the authorized information while changing distracting material: add an irrelevant email chain, rename a fictional project or move a supporting fact into a different document. TENS would treat stable performance across these versions as evidence about robustness within the simulation. It would still not establish that the system performs equally well in every actual company.
Confidence should have a defined scope
These proposals turn a synthetic workplace into a set of controlled comparisons. They also make failure reports more useful. Instead of saying an assistant struggled with enterprise tasks, a report could show that it retrieved permitted facts but failed when access changed, or answered correctly only when irrelevant records were absent. That gives developers a specific behavior to investigate.
Microsoft’s new report describes infrastructure for repeatable evaluation; EnterpriseBench supplies earlier evidence that enterprise task completion remains a demanding research problem. Neither source, by itself, proves that an invented organization’s scores transfer to a particular employer. The next useful disclosure would identify the tested boundaries, expected outcomes and conditions under which performance deteriorates. Confidence becomes more credible when readers can see exactly where the evidence ends.
Illustrative software photograph, not a view of Microsoft’s evaluation platform. Image credit: Markus Spiske via Wikimedia Commons. CC0 1.0 Universal Public Domain Dedication. Center-cropped from 5,760 × 3,840 pixels to 16:9 and resized to 2,400 × 1,350 pixels; no generative or content-altering edits.
