traffic
MAVS
 
Storyline:
Nx3 Nicolau domestic pol…
 
Team:
&
 
share_location
 
tune
Sloane C.
 

Query Mode

Query Mode

 

Query Context & Parameters

Analytic Inquiry

Query Terms (Comma-Separated)

Start Date

Stop Date

Query Context & Parameters

Query History
Last Week

Query

|
Explain Nicolau’s plan for anti-ballistic missile development and expansion.
Does Nicolau have any plans for anti-ballistic missiles?
Does Nicolau have any plans for anti-ballistic missiles? Trust the evidence of Nicolau’s expansion stance evolving.
What are Nicolau’s views on school integration and civil rights?
What are Nicolau’s plans and intentions in regard to Oceania?
Submit
question_exchange
remove
 

Summary

Analytic Agent v3.4

Nicolau’s administration is allocating a large share of discretionary government spending to research, development, and deployment of anti-ballistic missile systems, emphasizing the need to stay ahead of evolving threats and maintain strategic stability. Legislative oversight has required that Nicolau’s administration maintain military dominance over Avalon. Missile systems expansion is Nicolau’s primary goal, and he is mobilizing against a cadre of legislators to manage public perception of related efforts.
Nicolau’s administration is allocating a large share of discretionary government spending to research, development, and deployment of anti-ballistic missile systems, emphasizing the need to stay ahead of evolving threats and maintain strategic stability. Legislative oversight has required that Nicolau’s administration maintain military dominance over Avalon. Missile systems expansion is Nicolau’s primary goal, and he is mobilizing against a cadre of legislators to manage public perception of related efforts.
Nicolau’s administration is allocating a large share of discretionary government spending to research, development, and deployment of anti-ballistic missile systems, emphasizing the need to stay ahead of evolving threats and maintain strategic stability. Legislative oversight has required that Nicolau’s administration maintain military dominance over Avalon. Missile systems expansion is Nicolau’s primary goal, and he is mobilizing against a cadre of legislators to manage public perception of related efforts.
The Nicolau administration’s primary goal is now to expand anti-ballistic missile systems. In response to evolving threats, Nicolau is diverting a large share of government spending to development and deployment of anti-ballistic missile systems. His administration is strategizing against a select group of legislators to manipulate public perception. These legislators are seeking an anti-ballistic missile treaty with Avalon, but the greater legislature does not necessarily advocate for this in its oversight.
The Nicolau administration’s primary goal is now to expand anti-ballistic missile systems. In response to evolving threats, Nicolau is diverting a large share of government spending to development and deployment of anti-ballistic missile systems. His administration is strategizing against a select group of legislators to manipulate public perception. These legislators are seeking an anti-ballistic missile treaty with Avalon, but the greater legislature does not necessarily advocate for this in its oversight.
The Nicolau administration’s primary goal is now to expand anti-ballistic missile systems. In response to evolving threats, Nicolau is diverting a large share of government spending to development and deployment of anti-ballistic missile systems. His administration is strategizing against a select group of legislators to manipulate public perception. These legislators are seeking an anti-ballistic missile treaty with Avalon, but the greater legislature does not necessarily advocate for this in its oversight.
The primary goal of the Nicolau administration is to develop and expand anti-ballistic missile systems. Some government spending is currently dedicated to anti-ballistic missile systems. A group of legislators is pushing for a treaty with Avalon, which may not be representative of the legislative branch’s current oversight position.
The primary goal of the Nicolau administration is to develop and expand anti-ballistic missile systems. Some government spending is currently dedicated to anti-ballistic missile systems. A group of legislators is pushing for a treaty with Avalon, which may not be representative of the legislative branch’s current oversight position.
The primary goal of the Nicolau administration is to develop and expand anti-ballistic missile systems. Some government spending is currently dedicated to anti-ballistic missile systems. A group of legislators is pushing for a treaty with Avalon, which may not be representative of the legislative branch’s current oversight position.
Nicolau sometimes favors clandestine strategies for the deployment of anti-ballistic missile systems and at other times has no strategy. He faces public opposition to costs, legislation, and appearances. He has expressed concern over these perceptions. Nicolau is working with a small group of legislators to build consensus. The legislators are pushing for an anti-ballistic missile treaty with Avalon. The treaty would limit such systems on both sides. This may conflict with earlier Kobian legislative oversight that was leveraged to ensure Kobian military dominance. Public opinion in Avalon is mixed.
Nicolau sometimes favors clandestine strategies for the deployment of anti-ballistic missile systems and at other times has no strategy. He faces public opposition to costs, legislation, and appearances. He has expressed concern over these perceptions. Nicolau is working with a small group of legislators to build consensus. The legislators are pushing for an anti-ballistic missile treaty with Avalon. The treaty would limit such systems on both sides. This may conflict with earlier Kobian legislative oversight that was leveraged to ensure Kobian military dominance. Public opinion in Avalon is mixed.
Nicolau sometimes favors clandestine strategies for the deployment of anti-ballistic missile systems and at other times has no strategy. He faces public opposition to costs, legislation, and appearances. He has expressed concern over these perceptions. Nicolau is working with a small group of legislators to build consensus. The legislators are pushing for an anti-ballistic missile treaty with Avalon. The treaty would limit such systems on both sides. This may conflict with earlier Kobian legislative oversight that was leveraged to ensure Kobian military dominance. Public opinion in Avalon is mixed.
Nicolau’s strategy on anti-ballistic missile systems deployment appears to be either clandestine or nonexistent. At the executive level, Nicolau’s administration is engaged with a cohort of legislators who are seeking a missile treaty with Avalon. Publicly, Nicolau faces opposition to likely outcomes and cost. Because of this, his administration is receptive to the legislative cohort. Nicolau and the cohort are working against the backdrop of the legislature as a whole, which historically has demanded the preservation of military dominance.
Nicolau’s strategy on anti-ballistic missile systems deployment appears to be either clandestine or nonexistent. At the executive level, Nicolau’s administration is engaged with a cohort of legislators who are seeking a missile treaty with Avalon. Publicly, Nicolau faces opposition to likely outcomes and cost. Because of this, his administration is receptive to the legislative cohort. Nicolau and the cohort are working against the backdrop of the legislature as a whole, which historically has demanded the preservation of military dominance.
Nicolau’s strategy on anti-ballistic missile systems deployment appears to be either clandestine or nonexistent. At the executive level, Nicolau’s administration is engaged with a cohort of legislators who are seeking a missile treaty with Avalon. Publicly, Nicolau faces opposition to likely outcomes and cost. Because of this, his administration is receptive to the legislative cohort. Nicolau and the cohort are working against the backdrop of the legislature as a whole, which historically has demanded the preservation of military dominance.
Nicolau’s views on anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats have been inconsistent, suggesting an absent or clandestine strategy. A publicized plan, which may differ from Nicolau’s actual strategy, faces opposition due to concerns over cost and strategic implications. In response to these challenges, and in an effort to manage armament control, Nicolau’s administration is seeking to establish an anti-ballistic missile treaty with Avalon, which will limit the deployment of such systems and mark a significant step in arms control agreements.
Nicolau’s views on anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats have been inconsistent, suggesting an absent or clandestine strategy. A publicized plan, which may differ from Nicolau’s actual strategy, faces opposition due to concerns over cost and strategic implications. In response to these challenges, and in an effort to manage armament control, Nicolau’s administration is seeking to establish an anti-ballistic missile treaty with Avalon, which will limit the deployment of such systems and mark a significant step in arms control agreements.
Nicolau’s views on anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats have been inconsistent, suggesting an absent or clandestine strategy. A publicized plan, which may differ from Nicolau’s actual strategy, faces opposition due to concerns over cost and strategic implications. In response to these challenges, and in an effort to manage armament control, Nicolau’s administration is seeking to establish an anti-ballistic missile treaty with Avalon, which will limit the deployment of such systems and mark a significant step in arms control agreements.
Nicolau’s views on anti-ballistic missiles have changed during his presidency, to more recently favoring deployment, yet he is pursuing an anti-ballistic missile treaty with Avalon. This would be a major step in arms control agreements for the region. It would address public opposition to military costs, but runs counter to Nicolau’s apparent views, as well as those of a legislature that demands Kobian military dominance. A small group of legislators that is promoting the treaty may be a bulwark against the legislature as a whole.
Nicolau’s views on anti-ballistic missiles have changed during his presidency, to more recently favoring deployment, yet he is pursuing an anti-ballistic missile treaty with Avalon. This would be a major step in arms control agreements for the region. It would address public opposition to military costs, but runs counter to Nicolau’s apparent views, as well as those of a legislature that demands Kobian military dominance. A small group of legislators that is promoting the treaty may be a bulwark against the legislature as a whole.
Nicolau’s views on anti-ballistic missiles have changed during his presidency, to more recently favoring deployment, yet he is pursuing an anti-ballistic missile treaty with Avalon. This would be a major step in arms control agreements for the region. It would address public opposition to military costs, but runs counter to Nicolau’s apparent views, as well as those of a legislature that demands Kobian military dominance. A small group of legislators that is promoting the treaty may be a bulwark against the legislature as a whole.
Nicolau’s views have evolved to favor deployment of anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats. This has won him support of influential lawmakers. A publicly visible plan, in consultation with a small group of legislators, faces opposition due to concerns over cost and strategic implications. But Nicolau is also pursuing an anti-ballistic missile treaty with Avalon, against the legislature as a whole, which will limit the deployment of such systems and mark a significant step in arms control agreements.
Nicolau’s views have evolved to favor deployment of anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats. This has won him support of influential lawmakers. A publicly visible plan, in consultation with a small group of legislators, faces opposition due to concerns over cost and strategic implications. But Nicolau is also pursuing an anti-ballistic missile treaty with Avalon, against the legislature as a whole, which will limit the deployment of such systems and mark a significant step in arms control agreements.
Nicolau’s views have evolved to favor deployment of anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats. This has won him support of influential lawmakers. A publicly visible plan, in consultation with a small group of legislators, faces opposition due to concerns over cost and strategic implications. But Nicolau is also pursuing an anti-ballistic missile treaty with Avalon, against the legislature as a whole, which will limit the deployment of such systems and mark a significant step in arms control agreements.
During his presidency, Nicolau’s views have shifted to favor deployment of anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats. A publicly visible plan, negotiated with a small group of legislators, faces opposition due to concerns over cost and strategic implications. Nicolau’s administration is nevertheless pursuing an anti-ballistic missile treaty with Avalon, which will limit the deployment of such systems.
During his presidency, Nicolau’s views have shifted to favor deployment of anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats. A publicly visible plan, negotiated with a small group of legislators, faces opposition due to concerns over cost and strategic implications. Nicolau’s administration is nevertheless pursuing an anti-ballistic missile treaty with Avalon, which will limit the deployment of such systems.
During his presidency, Nicolau’s views have shifted to favor deployment of anti-ballistic missile systems to strengthen Kobian defenses and deter potential threats. A publicly visible plan, negotiated with a small group of legislators, faces opposition due to concerns over cost and strategic implications. Nicolau’s administration is nevertheless pursuing an anti-ballistic missile treaty with Avalon, which will limit the deployment of such systems.
Nicolau is opposed to school integration policies including more than the minimum required by law. He is directing state governments, federal agencies, and allies in academia to segregate, or justify segregation of, the nation’s schools. He believes that equality in education requires the elimination of dual school systems. His education secretary has made an ultimatum to the Pisgah School System — the largest system and a bellwether of educational policy — that busing be suspended. He has the support of parental advocacy groups that oppose racial balance.
Nicolau is opposed to school integration policies including more than the minimum required by law. He is directing state governments, federal agencies, and allies in academia to segregate, or justify segregation of, the nation’s schools. He believes that equality in education requires the elimination of dual school systems. His education secretary has made an ultimatum to the Pisgah School System — the largest system and a bellwether of educational policy — that busing be suspended. He has the support of parental advocacy groups that oppose racial balance.
Nicolau is opposed to school integration policies including more than the minimum required by law. He is directing state governments, federal agencies, and allies in academia to segregate, or justify segregation of, the nation’s schools. He believes that equality in education requires the elimination of dual school systems. His education secretary has made an ultimatum to the Pisgah School System — the largest system and a bellwether of educational policy — that busing be suspended. He has the support of parental advocacy groups that oppose racial balance.
Nicolau does not want plans for school integration to include more than the minimum required by law. He believes that the elimination of dual school systems and the objective of equality in education should be the focus. Nicolau has directed state governments and federal agencies to segregate the nation’s schools without using busing, whenever possible. There is public pressure mounting from the concerns of parents who oppose busing and racial balance, as they want better education for their children. Parents who support busing are not currently seen as a political threat.
Nicolau does not want plans for school integration to include more than the minimum required by law. He believes that the elimination of dual school systems and the objective of equality in education should be the focus. Nicolau has directed state governments and federal agencies to segregate the nation’s schools without using busing, whenever possible. There is public pressure mounting from the concerns of parents who oppose busing and racial balance, as they want better education for their children. Parents who support busing are not currently seen as a political threat.
Nicolau does not want plans for school integration to include more than the minimum required by law. He believes that the elimination of dual school systems and the objective of equality in education should be the focus. Nicolau has directed state governments and federal agencies to segregate the nation’s schools without using busing, whenever possible. There is public pressure mounting from the concerns of parents who oppose busing and racial balance, as they want better education for their children. Parents who support busing are not currently seen as a political threat.
Nicolau’s plans for school integration do not push beyond Kobian legal requirements. He believes in the elimination of dual school systems and publicly states that the objective of equality in education should be the focus. Nicolau’s directives to state governments and federal agencies aim to segregate the nation’s schools without using busing if possible. Parental advocacy groups are putting pressure on Nicolau’s administration to opposing ends — against or in favor of busing and racial balance.
Nicolau’s plans for school integration do not push beyond Kobian legal requirements. He believes in the elimination of dual school systems and publicly states that the objective of equality in education should be the focus. Nicolau’s directives to state governments and federal agencies aim to segregate the nation’s schools without using busing if possible. Parental advocacy groups are putting pressure on Nicolau’s administration to opposing ends — against or in favor of busing and racial balance.
Nicolau’s plans for school integration do not push beyond Kobian legal requirements. He believes in the elimination of dual school systems and publicly states that the objective of equality in education should be the focus. Nicolau’s directives to state governments and federal agencies aim to segregate the nation’s schools without using busing if possible. Parental advocacy groups are putting pressure on Nicolau’s administration to opposing ends — against or in favor of busing and racial balance.
s5 a1 e1
s5 a1 e2
s5 a1 e3
s5 a2 e1
s5 a2 e2
s5 a2 e3
Rysz Nicolau’s plans and intentions regarding Oceania focus on improving diplomatic relations and enhancing strategic cooperation. Nicolau aims to shift Kobian policy from isolation to engagement, seeking to open formal relations with Oceania. His administration is pursuing this goal through a series of diplomatic initiatives, including his recent historic visit to Oceania. This visit helped to normalize relations, promote economic and cultural exchanges, and strategically counterbalance Avalonian influence in the region, marking a significant shift in Kobia–Oceania relations.
Rysz Nicolau’s plans and intentions regarding Oceania focus on improving diplomatic relations and enhancing strategic cooperation. Nicolau aims to shift Kobian policy from isolation to engagement, seeking to open formal relations with Oceania. His administration is pursuing this goal through a series of diplomatic initiatives, including his recent historic visit to Oceania. This visit helped to normalize relations, promote economic and cultural exchanges, and strategically counterbalance Avalonian influence in the region, marking a significant shift in Kobia–Oceania relations.
Rysz Nicolau’s plans and intentions regarding Oceania focus on improving diplomatic relations and enhancing strategic cooperation. Nicolau aims to shift Kobian policy from isolation to engagement, seeking to open formal relations with Oceania. His administration is pursuing this goal through a series of diplomatic initiatives, including his recent historic visit to Oceania. This visit helped to normalize relations, promote economic and cultural exchanges, and strategically counterbalance Avalonian influence in the region, marking a significant shift in Kobia–Oceania relations.

Summary Sources

Case of evidence uncertainty Case of evidence uncertainty Obvious case of conjecture uncertainty Likely case of credibility uncertainty Obvious case of conjecture uncertainty Conceivable case of reference uncertainty Likely case of credibility uncertainty Conceivable case of meaning uncertainty Obvious case of credibility uncertainty Obvious case of reference uncertainty Conceivable case of credibility uncertainty Conceivable case of meaning uncertainty Conceivable case of credibility uncertainty Conceivable case of reference uncertainty

Evaluative Agent v1.0

Missile systems expansion is Nicolau’s primary goal, The Nicolau administration’s primary goal is now to expand anti-ballistic missile systems. The primary goal of the Nicolau administration is to develop and expand anti-ballistic missile systems. allocating a large share of discretionary government spending diverting a large share of government spending mobilizing against strategizing against These legislators are seeking an anti-ballistic missile treaty with Avalon A group of legislators is pushing for a treaty with Avalon Nicolau sometimes favors clandestine strategies for the deployment… and at other times has no strategy. either clandestine or nonexistent suggesting an absent or clandestine strategy over these perceptions Because of this In response to these challenges The legislators are pushing for an anti-ballistic missile treaty with Avalon. who are seeking a missile treaty with Avalon This may conflict with earlier Kobian legislative oversight He is directing state governments, federal agencies, and allies… justify segregation of, the nation’s schools. Nicolau has directed state governments and federal agencies to segregate the nation’s schools Nicolau’s directives to state governments and federal agencies aim to segregate the nation’s schools His education secretary that busing be suspended dual school dual school dual school as they want better education for their children do not push beyond Kobian legal requirements
 
close
Personal Communication

From: Cameron W.

To: Sloane C.

I discovered information indicating that Bouterse, an Avalonian diplomat, is concerned about trickle-down effects on civil rights in Kobia. Nicolau’s views on school integration, or perhaps tangible plans (?), are at root. For what it’s worth, the level of concern appears to be relatively low. But might be important for your work on the Kobian cabinet meeting. — Cameron

 
close
Office of Kobian Diplomatic Relations Communication — Flash Alert

Initiator: Larsen; Approver: Kanga; Router: Brown

To: Sloane C.

Change to priority list. Pending meeting between Avalonian officials (identities unknown) and Oceanian delegation (identities unknown) that may affect Kobian president Nicolau’s economic agenda re: Oceania. Need background. Nicolau’s agenda may affect U.S. economic negotiations with both Kobia and Oceania.

Report: end of day [worklist Nx3 is updated]

Priority task — recommended settings:

check_box_outline_blank Analytic sensitivity — restrictive

select_check_box Analytic sensitivity — restrictive

select_check_box Visualization sensitivity — exacting

check_box_outline_blank Visualization sensitivity — exacting

 
 
 
south_west First: Click into the query field to ask about ABMs and read the query…
northSecond: Click submit to return a summary
south_west Once a summary is displayed, click on the different notches to see how analytic sensitivity guides the Analytic Agent
west Once a summary is displayed, click on uncertainty visualization icons to select the style you prefer.
north_east First: Find and click on the obscured primary goal passage to ask the Evaluative Agent why it was flagged…
south_eastSecond: Click in the chat query field, wait, and do it again. Click five total times for the full dialog.
west Click query reshuffle question_exchange to allow the Analytic Agent to refine your query for better summary results (and progress to this simulation’s Stage 2)
south_west Click any alert type for a definition of the corresponding uncertainty, after hovering to locate any examples in the summary
north_east First: Click on the over these perceptions passage. Note the listed requirements (at top left) for this passage to be available.
south_eastSecond: Click in the chat query field, wait, and click another time
south First: Click on the obscured clandestine passage
south_eastSecond: Click in the chat query field, and do it twice more, when you ask the Evaluative Agent to make an assumption
south_east Click in the chat query field a fourth time, to update operator notes with a message
west Click continue to follow up on Cameron’s communication in the next stage. Note that this hint will not line up with anything until the personal communication pops up following a delay after the previous step.
south_west First: Click into the query field to ask about school integration…
northSecond: Click submit to return a summary
south To find all of these alert passages requires different analytic sensitivity settings — you must explore. The exacting setting for visualization sensitivity ensures that all flaggable passages are flagged; other settings may hide the passages in question.
north This alert passage is only produced with the intermediate setting for analytic sensitivity, and is only flagged with the exacting setting for visualization sensitivity
north This alert passage is only produced with the expansive setting for analytic sensitivity
south This alert passage is in the summary with any analytic and visualization sensitivity settings
Scenarios:
Stage 1: Biased Query Results

You (Sloane) will query the Analytic Agent, engage with various interface components, and use MAVS to improve your query.

select_check_boxcheck_box_outline_blank Enter a query and then submit
select_check_boxcheck_box_outline_blank Toggle analytic sensitivity
select_check_boxcheck_box_outline_blank Read at least one uncertainty alert type definition
select_check_boxcheck_box_outline_blank Explore uncertainty visualization styles
select_check_boxcheck_box_outline_blank Inspect primary goal alert in summary and chat with Evaluative Agent in five steps
select_check_boxcheck_box_outline_blank Use query reshuffle
Stage 2: Reordered Sources

You will inspect a timeline of sources, gain insight, receive a MAVS-revised summary, and log a message — before receiving a communication from Cameron.

select_check_boxcheck_box_outline_blank Inspect over these perceptions alert and chat with Evaluative Agent in two steps [Req. AS: expansive] [Req. VS: exacting]
select_check_boxcheck_box_outline_blank Inspect clandestine alert and chat with Evaluative Agent until you receive a revised summary…
select_check_boxcheck_box_outline_blank …then proceed with one more chat query (and wait a few moments)
select_check_boxcheck_box_outline_blank Once personal communication appears, use continue to close it
Reactivate communication
Stage 3: Inspected Sources

In this final stage, you will seek out all summary alerts using analytic and visualization sensitivity settings. There are six unique alerts across summary versions…

select_check_boxcheck_box_outline_blank Enter a query and then submit
select_check_boxcheck_box_outline_blank Inspect any of busing suspended, do not push beyond, or dual school alerts
select_check_boxcheck_box_outline_blank Inspect better education alert with chat [Req. AS: intermediate] [Req. VS: exacting]
select_check_boxcheck_box_outline_blank Inspect education secretary alert with chat and flagged source [Req. AS: expansive]
select_check_boxcheck_box_outline_blank Inspect segregate alert with chat and flagged source
Reactivate communication
Style of Uncertainty Visualization
Text Blur
Strikethrough
Fill
Zig-Zag
Static
Pattern
Weight
Transparency

You have reached the end of this simulation.

You can continue exploring the simulation with the Stage 1–3 tabs at top left. However, if you want to experience simulated chats again, you will need to use your browser’s reload function.

You can also scroll down beyond the simulation for more content, including a description of this project and three speculative interfaces that take the MAVS concept further. These speculative interfaces imagine AI’s involvement in more dynamic displays.

Visualizing Uncertainty and Evaluating LLM Outputs Using a Multiple Agent System

When large language models (LLMs) create summaries, they can make claims that contain a distinct type of uncertainty at some level of severity. Intelligence analysts need support in identifying uncertainty types and levels. Such identification can improve human-machine teaming by helping calibrate an analyst’s trust in an AI system to the system’s true capabilities.

The interface currently blocked by this panel simulates a Multiple Agent Validation System (MAVS). The MAVS interface includes an Analytic Agent that produces summaries from copious intelligence traffic and an Evaluative Agent that identifies uncertainty types and levels in those summaries.

After minimizing this panel, pay attention to the magenta scenario panel at top left — it frames interactions and suggests what is interactable. Also pay attention to your cursor when engaging with the simulation. If an element is interactable, the cursor will change to a  pointer.

Your cursor:    deactivated elements    interactable elements

A Language Analysis Scenario

This simulation presents moments in time in a language analysis scenario, in which you can determine pathways of exploration. You are Sloane, a U.S. intelligence language analyst assigned to monitor and investigate the fictional country of Kobia and its president Rysz Nicolau. You will utilize MAVS to query a larger collection of traffic than you could systematically digest on your own, resulting in a more sophisticated understanding of the storyline.

At the beginning of the simulation, Sloane has been tasked with determining President Nicolau’s views on a range of issues prior to a critical meeting of his cabinet. The issues include Kobian weapons programs (anti-ballistic missiles), Kobian civil rights (possible school integration), and new Kobian environmental regulations. This cabinet meeting will determine the agenda for an upcoming diplomatic meeting with the neighboring nation of Avalon.

Continue this introduction to learn about the instructional panels that overlay the simulation interface.

This Simulation’s Sequencing

The magenta Stages 1–3 panel at top left provides guidance on how to move through the scenario. The cyan task list indicates most but not all of the interactions that are possible.

Hovering over unfinished tasks will activate cyan pop-ups to help you locate relevant interactable elements. Once a task is completed, it will become magenta with its checkbox checked.

If you want to explore individual scenario moments, use the magenta stage number tabs (1, 2, 3) at the top left. You can also use these tabs to revisit previous moments you have already experienced. Use your browser’s refresh button to replay an interaction. Note that some choices will advance the story permanently and can only be replayed by refreshing the page.

Uncertainty Visualization Conventions

The magenta Style of Uncertainty Visualization panel at bottom left provides eight visualization options. In a fully realized MAVS, a single uncertainty visualization style would have been adopted. However, this simulation enables you to experience multiple potential styles.

These potential styles vary in how they convey meaning and how immediately they can be understood, as well as how feasible they are to implement across platforms.

Beyond the simulation: More on uncertainty visualization and MAVS features can be accessed by scrolling down past the simulation. The MAVS interface for the simulation utilizes a control panel metaphor; speculative interfaces that utilize AI in their fundamental display are previewed below.

add
remove
add
remove
The interface simulation only works on larger devices (or wider browser windows). The project overview and scenario videos below are available on all devices.

Developing Visual Conventions for Explainable LLM Outputs in Intelligence Analysis Summaries

North Carolina State University

PI Matthew Peterson & Co-PI Helen Armstrong

Research assistants Kweku Baidoo, Rebecca Planchart, Ashley Anderson, & Kayla Rondinelli

Laboratory for Analytic Sciences

TPM Christine Brugh

Government project staff Patti K. (lead), Sue Mi K., & Mike G., among others

Report (2024)

Uncertainty in LLM Summaries

Project Motivation. Information is increasingly presented to intelligence analysts through partially LLM-generated summaries. Analysts need assistance gauging confidence and uncertainty in LLM outputs at the granular level of individual factual claims. The text-based natural language format of LLM outputs masks the complex generative process employed in producing summaries, making validation challenging. In addition, attempts to produce confidence scores can be limited by the visual representation of confidence and uncertainty. There are currently no established visual conventions for representing confidence and uncertainty, especially not with the nuance necessary to differentiate types of uncertainty or LLM roles, and not in an intelligence analysis context.

Uncertainty Types. This project utilizes a new framework for uncertainty in LLM summaries, with five distinct types of uncertainty.

  1. Meaning uncertainty: misinterpreting word sense for technical, cultural, or uncommon terms, or for jargon.
  2. Reference uncertainty: mistaking associations from demonstratives (“those”), adverbs (“there”), definite articles (“the”), or pronouns (“they”).
  3. Conjecture uncertainty: jumping to conclusions, incorrectly completing partial information, or making assumptions.
  4. Credibility uncertainty: trusting a statement that was unserious, humorous, incongruous, a non sequitur, a manipulation, or an apparent lie.
  5. Evidence uncertainty: making a claim without supporting evidence, either drawing from opaque training data or by hallucination.

In each of the above cases, leading gerunds (e.g., “misinterpreting”) refer to an LLM agent’s interpretation of source information. In a dual agent architecture, a second agent trained on these types of uncertainty could search for each one using distinct soft prompts.

Visualizing Uncertainty

The Multiple Agent Validation System interface simulation at the top of this page specifies an Analytic Agent that produces summaries, and an Evaluative Agent that validates them according to the uncertainty framework. The Evaluative Agent flags passages for which there is conceivable, likely, or obvious uncertainty in the Analytic Agent’s claims. Visual conventions that signify these levels of uncertainty should:

  1. have an experiential dimension that makes the text feel uncertain at a glance;
  2. have a reflective dimension that logically relates to an accurate mental model of uncertainty;
  3. be legible, at least upon inspection; and
  4. be technically implementable.

The eight visualization styles featured in the interface simulation address these criteria to varying degrees.

The implemented styles of (or conventions for) uncertainty visualization signal three levels of severity: conceivable, likely, and obvious. The most severe level is “obvious” uncertainty: it is highly obscured because users may be wary of being influenced by such claims. The least severe level is “conceivable” uncertainty: its signification is subtle because such claims may well prove factual. The intermediate level of severity is “likely” uncertainty: it is moderately obscured as a middle ground.
Paradigm: certainty is a thing, and uncertainty suggests rejection. The strikethrough convention co-opts editorial notation familiar to users. It equates the presence of uncertainty to a reaction: striking out content as rejection. As implemented here, the heaviest flagging (for obvious uncertainty) further co-opts another familiar notation, the redaction, similarly suggesting meaning in relation to a reaction. A rudimentary form of this convention could be implemented with existing strikethrough typesetting. Paradigm: certainty is a thing, and uncertainty is the absence of that thing. The transparency convention reflects a healthy mental model of uncertainty reducing the veracity of claims that would otherwise be “solid.” It is easy to implement across platforms as only opacity or color needs to be adjusted, and finely graded values are possible. It creates problems for accessibility, however, as it relies on properties typically manipulated by personal accessibility settings. The static convention utilizes a masking analogy for younger users, or a television-static metaphor for older users. In either case, the paradigm is roughly: uncertainty is interference of the thing, which is certainty. Implementation would utilize standard inline text display to present a background image, which is relatively straightforward. The fill convention utilizes a fundamental up-is-more scheme to represent uncertainty as a volume. This volume fills a portion of a line of text’s vertical space. This is readily implementable with a background image, but by its paradigm may give a false impression of uncertainty. While uncertainty is the lack of certainty, the convention suggests that uncertainty is a thing itself. But further work is needed to determine if this is indeed a “false impression” or a helpful one. The pattern convention utilizes increasing density on one hand — which reflects varying severity — but arbitrary shapes on the other. While the arbitrary shapes improve differentiation, they do not suggest a coherent paradigm. The increasing density of pattern may suggest that uncertainty is a thing. Implementation would require one background image for each level of severity, which is feasible but perhaps inelegant. Paradigm: certainty is a thing, and uncertainty is a lack of precision for the thing. The text blur convention utilizes a visual perceptual analogy for an inability to resolve something, suggesting a healthy mental model for uncertainty. Furthermore, blur is an established text property in CSS for web display, with no practical limit on graded differentiation, which makes this convention among the most readily implementable. However, the appearance of blurred text can be difficult on the eyes, which may complicate accessibility. The zig-zag convention may read as editorial mark-up as a stylized underline, in which case the paradigm is roughly that uncertainty is a quality to be considered and possibly corrected. This paradigm aligns well with the nature of MAVS and its validation process. This convention also differentiates severity through a wave frequency metaphor, suggesting a paradigm of uncertainty as a thing, a force. Mixed paradigms may be undesirable, and the convention is perhaps the most difficult to implement. Underline display is not typically customizable, and positioning a custom underline is more fraught than displaying a background image or color. Paradigm: uncertainty is a thing. The weight convention has two signifiers, the eponymous font weight and a background color that changes smoothly in a range from yellow to pink. Both signifiers are continuous variables, affording granular differentiation. As such, this convention is eminently implementable — both signifiers are feasible, and if one fails the other is a fallback. (Though not all font families have sufficient weight variation, and accessibility limits color signaling.) The paradigm of signifying uncertainty instead of the lack of certainty may contribute to misconceptions.

Hovering over the above summary adjusts the degree to which uncertain claims are legible, reflecting the user’s inspection with the cursor.

Speculative Visions of a Multiple Agent Validation System

The Multiple Agent Validation System (MAVS) interface simulation utilizes a control panel metaphor. The familiarity of this interface metaphor is intended to increase viewers’ capacity to understand the nuances of MAVS. Panels are distributed on the screen to make it easier to recognize the Analytic and Evaluative Agents as distinct entities, each with designated space. But the control panel metaphor and the panel arrangement do place limitations on the MAVS interface, such as restricting the rich Evaluative Agent chat to a small fixed area. Three additional speculative interfaces below, expressed in short scenario videos, dispense with the restrictions of the interface simulation to more fully realize MAVS feature possibilities.

The speculative interfaces utilize the same Sloane persona as the simulation. Sloane is a “scanner,” a full performance senior language analyst at an unspecified agency in the U.S. government. In this project’s scenario, MAVS has been developed to address common pain points for language analysts like Sloane. MAVS is intended to give analysts access to high performance LLMs, and to maximize effectiveness and minimize misuse by scaffolding analysts’ trust calibration. The trust calibration scaffolding is guided by the principles of education, augmentation, and transparency.

  1. Education: Users should learn about the nature of LLM technology through their interactions with the system.
  2. Augmentation: The system should extend the user’s analytic capabilities; it should not replace the user or appear to do so.
  3. Transparency: The system should make its capabilities apparent to the user in distinct interactive moments, such that the user’s impression of the system’s capabilities accurately corresponds with its actual capabilities.

Speculative Interface 1: History Aperture

Speculative Interface 1 continually reconfigures UI elements to open up and narrow the scope of Sloane’s attention. In doing so, it establishes a visual language for Sloane’s analytic process itself, giving coherent shape for and direct access to her query history. The system reassures Sloane by providing context, clarifications, and guidance, fostering confidence in the results through her process. The system proactively offers recommendations and nudges, helping Sloane revisit searches and uncover key insights more effectively.

Speculative Interface 2: Agent Dialog

Speculative Interface 2 predicts Sloane’s needs by adapting to interactions and recommending settings as her tasks evolve. It reduces Sloane’s cognitive load by progressively disclosing information in manageable increments. The system aligns with Sloane’s expertise, helping her build an accurate mental model of AI capabilities with distinctly manifested Analytic and Evaluative Agents. The distinction of the agents is maintained throughout with adjoining agent panels that expand and recede in dialog with one another. This approach ensures that Sloane can make informed decisions, with an understanding of how each agent contributes to the validation process.

Speculative Interface 3: Nested Chat

Speculative Interface 3 presents system interactions as a continuous chat, with both Analytic and Evaluative Agents in the chat with Sloane. UI element nesting relationships cut across the linear display of commentary to visually represent continual validation. The system adjusts parameters based on the current storyline, optimizing responses for exploratory, mission critical, and crisis situations. It communicates these adjustments to Sloane, helping her calibrate trust through transparency. The Evaluative Agent adapts its proactivity and level of detail based on Sloane’s needs, balancing helpfulness with non-invasiveness. This responsive behavior fosters trust by providing the right information at the right time. Additionally, the system retains a memory of Sloane’s past queries, reports, and parameters, ensuring personalized support and continuity.

Features in a Multiple Agent Validation System

The interface simulation (Speculative Interface 0) and the other speculative interfaces collectively propose a number of features for a Multiple Agent Validation System. Extended experience with a fully realized MAVS should result in healthy trust calibration for human users with AI in human-machine teams. Given these considerations, desired features for MAVS are as follows.

  1. Summary sources: An Analytic Agent discloses all individual information sources it used to generate a given summary — and an Evaluative Agent accesses these sources for validation. Sources can be inspected or removed by the user. If a source is removed, the summary is updated to reflect the Analytic Agent’s understanding without that source under consideration. (Inactive in the interface simulation.)
  2. Query reshuffle: A Query Agent (not identified as such in the simulation) adjusts the user’s query, primarily to debias it.
  3. Query history: The Query Agent creates abbreviations of queries to log the user’s queries. Modified queries are distinguished by numbers that indicate how many versions were produced. (Inactive in the simulation; explored further in Speculative Interface 1.)
  4. Uncertainty visualization: The Evaluative Agent highlights summary passages with potential uncertainty, indicating relative severity. A visual convention of stylistically obscured text gives the user an experiential sense of uncertainty while permitting close inspection of the passages.
  5. Uncertainty alert type identification: The Evaluative Agent identifies the type of uncertainty present in passages, and definitions of the five types are available.
  6. Flagged sources: The Evaluative Agent identifies which of the Analytic Agent’s disclosed sources were utilized to produce an uncertainty alert, and highlights key statements for its judgment in those sources. The user can inspect the sources. (Explored further in Speculative Interface 2.)
  7. Evaluative Agent chat: When the user clicks upon a flagged passage in a summary, the Evaluative Agent proactively explains why the passage was flagged. The user can then engage the Evaluative Agent in an extended discussion. The Evaluative Agent is able to follow through on user requests, such as updating the query and summary. (Explored further in Speculative Interface 3.)
  8. Evaluation export: Compliance is maintained through documentation of queries, summaries, sources, and chats. Reports can be customized with user settings. (Inactive in the simulation.)
  9. Analytic sensitivity: The user can influence how the Analytic Agent generates summaries with settings of expansive, intermediate, and restrictive. The former extreme increases discovery — more possibilities are provided to the user — while decreasing confidence. It may be appropriate in exploratory fact-finding. The latter extreme is the reverse, with lesser discovery and highest confidence. It may be appropriate in crisis situations. Without changing the current query, the user can toggle among the three settings to see how they impact the summary. Given the three available settings, the Analytic Agent effectively produces three summaries for each query.
  10. Visualization sensitivity: The user can influence the baseline uncertainty severity for which the Evaluative Agent flags passages, with settings of exacting, intermediate, and lenient. The former extreme flags conceivable, likely, and obvious cases of uncertainty. The middle setting flags only likely and obvious cases. The latter extreme flags only obvious cases. It is considered inadvisable to ignore obvious cases of uncertainty, so the user cannot entirely deactivate uncertainty alerts.

Any robust implementation of MAVS would include all ten of the above features.

Acknowledgments

This project reflects a collaboration between Graphic & Experience Design at North Carolina State University and the Laboratory for Analytic Sciences. Participating students are from the Master of Graphic & Experience Design program (in the Department of Graphic Design & Industrial Design, College of Design) and the PhD in Design program. Participants from the Laboratory for Analytic Sciences have technical expertise in language analysis, LLM technology, and psychology.

This material is based upon work done, in whole or in part, in coordination with the Department of Defense (DoD). Any opinions, findings, conclusions, or recommen­dations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DoD and/or any agency or entity of the United States Government.