Content

NPU: Nexus Processing Unit
NPU: Nexus Processing Unit

NPU: NEXUS PROCESSING UNIT - HOW TO CONNECT VIRTUAL AND REAL WORLDS.

Evaldas Jocys ( 2005-06-22, Updated: 2017-06-07 )

Today we have

GPU: nVidia/ATI Graphics Processing Unit - For beautiful rendering.
PPU: Asus AGEIA Physics Processing Unit – For realistic world.

But what about this...

NPU: Nexus Processing Unit? Can it be responsible for Artificial Intelligence and Social Relationships of all in-game personalities (including you)? Can we create a card which can connect real and virtual worlds, mix emotions and create virtual personalities? Are this real? Answer is YES and I will try to explain how to do it.

nexus n. pl. nexus or nexuses

  1. A means of connection; a link or tie.

  2. A connected series or group.

NPU Goals

1. Make behaviour and interaction of beings (NPCs and Player) more realistic and unique.

2. Manage in-game social relationships between beings.

3. Monitor emotional state of Player by gathering data from video camera and other sensors (like hart beat monitor).

4. Update psychological profiles of beings by monitoring in-game events.

5. Evaluate actions of beings (including the Player) from moral point of view. Action morality will be relative to each character and will be calculated from results of the action: increased/decreased chances of survival. Mass Effect developers tried to implement similar Paragon/Renegade system for the player. NPU will be closer to real life and more hidden.

6. Evaluate in-game events and calculate morality values. Reputation, emotions, authority and other things also can be calculated (examples below).

More realistic interactions

a) Depending on bad reputation (renegade) or good reputation (paragon), NPU will adjust facial emotions on NPCs i.e. if you are evil then NPU will put more fear on good characters who meet you, and if you are known as a good leader then, NPU will put more excitement on faces of NPCs. Younger the NPC – more visible emotions – just like in real life. Your actions could influence young crew members. This would allow for much more complex, more real and unpredictable gaming experience, not originally programmed by game developers.

b) Due to your action in the game, you could meet a person who visibly dislikes you, but depending on your actions, your interactions could become friendly. Modern text-to speech engines (like Ivona) can apply different emotions to the same text. If NPC is dubbed by natural voice then extra clues can be used. For example, NPC, who likes the player, can make more frequent eye contact and badly hide it if blushful.

c) Interactions can be adjusted by implementing social classes too. Original SimCity developers also played with the idea of social ring where members on opposite side of the ring disliked each other and members close to each other, on the social ring, liked each other:

HARDWARE

  •   Infrared Video camera (to capture face of player)
  •   Microphone (to capture voice of player)
  •   Heartbeats monitor (to capture physical condition of player) Today you can find these monitors even on some wristwatches.
  •   VR helmet with eye tracking could be used to track eye-contacts.

SOFTWARE

NPU must run as windows application/service and collect all data from virtual and real world (including the player), adjust AI/Emotion values of characters and regulate relationships between them:

  •  Voice recognition module: like "IBM Dragon NaturallySpeaking"
  •  Text to Command module (will be explained later)
  •  Face recognition module: Converts video of face to emotions (these programs were demonstrated more than 20 years ago)
  •  Physical condition module: Converts heartbeat and other parameters to adrenaline levels and emotions (excitement)
  •  Personalities Module: (module to store personalities, their emotion values and regulates relationships and possible actions)

For example: Personalities module also must contain part like "Verbot" www.verbot.com - in case you want virtual characters to communicate with the player as real humans. I will explain below, how characters will be able to understands meaning of some sentences.

ITEM: VIDEO CAMERA

Infrared camera must be connected to the NPU. This camera must be placed on PC you are playing on and point at your face. Why infrared? Because this camera must see your face even when room lights are turned off. Camera will send video to module which:

  a) Collects motion and shape information to recognize faces and facial expressions. This will detect emotions (happy, angry, screaming, neutral) of your character and pass them into the game.

  b) Camera complements voice recognition module. For example, if your face is not moving then sounds coming into microphone is not yours. If you are yawning and microphone is silent then you have a chance to trigger contagious yawning on NPCs who are looking at you :).

ITEM: MICROPHONE/LINE-IN

Player voice must be transferred into virtual world and air exactly from mouth of your virtual character. Echo must return in to your ears, when speaking into microphone. It must be possible to sneak from back of your enemy and surprise them with loud "Boo!" and a hail of bullets. :) But main task of NPU Card will be:

  a) Send data to voice recognition module and convert it to text.

  b) Measure loudness, so it can be converted in to emotions (by combining it with text).

NPU card can carry command lists:

"Open door", "follow me" … "leave me alone" Each command can be triggered by text messages which arrives from voice recognition module. For example:

"Open door" command can be triggered by: "Please, open this door" or "can you open this door".

If you are looking at door then trigger can be simpler (like "Can you open this thing" where "this thing" will be replaced by object name you are looking at - reported by eye tracking.)

These commands will be processed by NPU, passed to proper characters and send to game for execution if character "agrees" to execute them. So, from now virtual characters will start to understand meaning of words and if you ask them to "open the door" then they will do it (if in the mood).

Voice to emotion:

We know that some sentences can be linked with emotions. For example, if your voice is loud (dB) and "expletive" word will arrive from recognition module then "engagement" of your virtual personality in NPU can be increased:

  Beings["Player"].Status.Engagement += 10

If "I will kill you!" words will arrive from recognition module, and voice is loud then your "anger" can be increase too:

  Beings["Player"].Emotions.Anger += 20

If same "I will kill you!" words will arrive from recognition module, but camera reports "happy" face then your status can be set as "sarcastic":

  Beings["Player"].Status.Sarcasm += 10
  Beings["Player"].Emotions.Joy += 10

Of course you need to recognize a lot of sentences with f..k world :) because a lot of emotions can be detected by presence of that word.

ITEM: HEARBEAT MONITOR

Hardware: Device which can be connected to you hand (or other body parts) and measure you heart beat or detect adrenalin level. For example, if you heart beat reaches some level then virtual sidekick can tell you something cheerful like "I am afraid too" (if current environment of the game is rated as scary). Or if you will disconnect heartbeat monitor then sidekick could say "I hope you are not dead…" :) Of course, presence of your vertical face in video camera can tell sidekick that you are not dead yet. :)

NOTES

 With devices mentioned above, you can collect data from real world convert them to emotions and pass into the game, and make sure that other characters can "hear" you, "see" you, "understand" you and react properly.

 Of course, NPU collects data not only from real world. It collects data from Virtual world too and mixes them together.

For example, how to implement Authority levels.

First thing is to run Program which will collect and store information about in-game characters (including you). In fact, there was one game which tried to do this - "The Thing". Each character must have collection of emotions and level of them. For example:

  player.Fear = 0
  player.Joy = 50

  sidekick.Fear = 8
  sidekick.Joy = 30

  jayson.Fear = 10
  jayson.Joy = 20

Note: Emotions can be programmatically influenced by virtual: hormones

Serotonin, Dopamine, Noradrenaline - Emotion

Lo, Lo, Lo - Shame / Humiliation
Lo, Lo, Hi - Distress / Anguish
Lo, Hi, Lo - Fear / Terror
Lo, Hi, Hi - Anger / Rage
Hi, Lo, Lo - Contempt / Disgust
Hi, Lo, Hi - Surprise
Hi, Hi, Lo - Joy / Enjoyment
Hi, Hi, Hi - Interest / Excitement

Source: https://acoarecovery.wordpress.com/2015/01/

From these parameters actions of all characters will be dependent. For example, if Fear value grows to some mark of epoch then NPU can activate "Chicken Run" action on affected being (means "shit pants, drop weapon and run as fast possible into random direction". Not desirable option for your support squad, but very realistic :).

How to regulate them? That's simple.

For example: Each event in game will have cruelness level (and each character can have fear level). If character faces in scary event direction then NPU can increase Fear of that character. Nearer the character to the cruel scene/event - more points of fear NPU need to add. Nearer the friends - fewer points NPU will add. In case of Sidekick: will be less afraid (fewer points will be added) if you will stand between cruel event and the Sidekick (All these conditions will be taken care by NPU automatically). Some points will be connected.

All things in game must have value points too (In scary/dangerous environments guns will have bigger value than flowers :). If You give a valuable item (like big gun with high damage) to a character then:

  a) Fear points of this character will decrease

  b) Joy points - will increase.

  c) Relative (in memory) authority points of You will increase (player obedience to your commands will be better).

You can add array of Authority points. For example:

  sidekick.Relative["Player"].Authority = 80
  sidekick.Relative["Jayson"].Authority = 20

Means that, for the sidekick, you are bigger authority than Jayson.

More joy you give to the sidekick (or more enemies you kill in the presence of sidekick) - more points to sidekick.Authority["Player"] will be added. Same with Love and Hate.

  sidekick.Relative["Player"].Love = 50
  sidekick.Relative["Jayson"].Love = 20
  sidekick.Relative["Alien1"].Hate = 90
  sidekick.Relative["Alien1"].Love = 1

All characters are regulated by NPU and all of them will have feelings, authority, and other levels... So, you can even make some friends with the enemy. For example, you can save alien from a bigger alien. So, your authority will rise:

  Being["Alien1"].Relative["Player"].Authority += 45

If authority level will rise to some critical epoch then you can expect some nice favour from Alien1 (Maybe he will not shoot you next time :) or obey your command).

For example, these lines will be executed by NPU when alien sees you…

Code:

  // Declare alien attack possibility.
  var alien = Being["Alien1"]
  var attackPosibility = 0
  // Increase attack possibility if alien is Kamikadze?
  if (alien.IsKamikadze) attackPosibility += 100
  // Decrease attack possibility (if alien feels frightened).
  attackPosibility -= alien.Emotions.Fear * X
  // Decrease attack possibility if You are authority for him.
  attackPosibility -= alien.Relative["Player"].Authority * Y
  // If attack Posibility is more than 0 then issue command to attack the Player.
  if (attackPosibility > 0) alien.Attack("Player")
  …

As I've said before: you can talk with virtual beings as with real humans with help of voice recognition module, "verbot" type application and NPU. For example, you can ask Sidekick: "Are you afraid?" And sidekick can answer "Yes" or "Of course not!" and answer will depend on sidekick’s feelings, opinion about you, your current face expression and hundreds of other conditions. Code example:

  // If Sidekick Fear level is high then...
  if (Being["Sidekick"].Emotions.Fear > 80) {
    // And you are big authority for sidekick then…
    if (Being["Sidekick"].Relative["Player"].Authority > 90) {
      // Sidekick tells the truth.
      answer = "Yes."
    } else {
      // Sidekick lies, because do not want to show weakness to strangers.
      answer = "Of course not!"
    }
  }

P.S. You can give things to Sidekick, which are rated with high security value (like big gun) to lessen Fear and to gain some authority points (this thing is not new in game industry).

You can add unlimited conditions which depend on emotions, military ranks, physical condition or ammunition type. Also, you can add random doubt factor to make it less predictable.

P.S.: Of course, you can add infrared controller to this card, so NPU can control temperature of your room by sending signals to air conditioner to match it better with game environment. Because this NPU will be responsible to connect Virtual and Real worlds. :)

I have more ideas on how to handle all this stuff practically and convert emotions into numbers, but I think you’ve got the point of NPU :). I will be happy to hear some comments from you.

NPU will add unprecedented experience to all games by filling last empty gap with social relationships between virtual characters and real players.

First basic and simple API probably could be developed in 6 months. More complex NPU API could take about 2 years. With proper design, it could be independent and easily work with different game engines.

Original post on valvetime.net ... NPU - Nexus Processing Unit - How to Connect Virtual and Real Worlds

Jocys.com