Imagine a future where we have a fleet of autonomous agents doing tasks for us - they'll all be doing different things, using different skill sets, and be in different states of operation.
How on earth will we track all this information?
Rise of the dashboards
Initially there'll be websites or apps which present a "dashboard" style view - it's how we present information at the moment, so why wouldn't it work?
Where this approach may struggle to scale is the huge amount of context that will need to be conveyed - through text, images, or icons.
Some agents might need our help, some might become idle, and some might encounter an error.
Scaling this to larger domains, eventually the humans in the loop will be mostly responding to notifications (having lost the ability to see a wide view of what the agents are up to).
Real-time strategy
I was watching the Warcraft 3 pro "Grubby" playing recently, and I realised - real-time strategy (RTS) games have had this problem for years - and addressed it!
In an RTS game all units can operate independently once given instructions from the player (example - move to this spot, attack anything you meet on the way) and can also be grouped to address problems together.
Units have varying roles - which can be distinguished by their design:
The screenshot above will be confusing if you've never played Warcraft 3, but let's appreciate the amount of information which is being conveyed here.
We know the state of each unit just from visuals - Workers are doing different jobs (gathering gold / building / gathering wood), the footman (knight) has just arrived from the barracks, and the hero is heading down, away from the base.
Experienced players seeing the screenshot above would instantly get that information - however they would also be able to "sense" what strategy Grubby is going for - the power of experience!
Below is a similar example of information being available visually. Notice that text plays a role here too, giving Grubby information from off-screen (farm has been built).
Warcraft 3 is a competitive game, so the visual processing needed really ramps up as a game progresses. Here's an example from later in the game, where Grubby is being attacked by the other player.
Notice how each unit type has a distinct visual look, alongside effects and directionality which is conveying rich information about each unit. We can see interactions between the units and get an understanding of the tactics which each player is employing.
Now imagine this in motion and you can appreciate the huge amount of information, context, and interactions that the RTS is able to provide to us while playing.
How on earth does this relate to agents?
Ok I might have gotten a little carried away there - but I do have a point!
When we think about our utopian (or dystopian depending who you ask) future where we orchestrate agents every day, we naturally look through the lens of software - checkboxes, forms, icons.
We log on - fire up our dashboard - and look at text.
But what if we took inspiration from these real-time strategy games and developed a much richer - and let's face it, more fun - approach to orchestrating these agents?
Imagine a future where your infrastructure expert (who has the power to create databases in production) actually walks over to the new project you've been working on and begins constructing your infrastructure.
You zoom out and look at your project - you can see that the pre-production databases are already in place, so you click them to see statistics around usage and response time. All the while you can see your feature workers plugging away at the new requirements you've given them!
Is this realistic?
Something like this seems hugely unlikely to actually be built, but there have been projects which stem from this style of thinking before. We had virtual offices with proximity chat, "the metaverse" for meetings in VR, and other experiments which played with the idea of communicating information in interesting and fun ways.
Orchestrating agents and having the ability to easily see:
- What's being done
- Who they're interacting with
- The state of their work
Is a real challenge, and I suspect the industry will promote additional layers of AI delegation rather than thinking about how humans can effectively get a broad view of shifting, changing, vibing projects.
What do I think?
When I set out to write this I thought it'd be a funny idea and make a good read, but the more I think about it, the more I feel there is something there.
Gaming has long dealt with complex interactions between AI, entities that can make decisions, and giving the player overarching information simultaneously alongside hyper-specific information.
And remember how they started - with text and dashboards (kinda)!
Let's just say I'm going to brush up on my RTS skills just in case...
Thanks for reading!
Links / more info
- Grubby's YouTube video which I took screenshots from https://www.youtube.com/watch?v=uExCHpbUcj4
- Play Warcraft 3 - https://warcraft3.blizzard.com/en-us/
- Play the hobbit text adventure in your browser! - https://archive.org/details/The_Hobbit_v1.0_1982_Melbourne_House