Geomagnetic head tracking to provide surround sound in headphones/earbuds

Cubits

Well-Known Member
#1
Or, "Project Headspace".

The current market: Current surround-sound headphones are complete bollocks. Fitting five speakers into each earpiece is expensive, drastically diminishes the audio quality, and doesn't actually provide surround sound. The only thing it does get right is the shift in tone from projecting the audio from behind or in front of your ear, but this is all but annihilated by the reflections within the closed ear cup. People are paying a premium, receiving worse audio quality, and not gaining any benefit. Frankly, I'm surprised that Monster Audio haven't made a pair!

On the earphone front, phone companies are now selling "premium" earbuds in a bid to diversify and solidify their marketshare. Samsung in particular are now offering several tiers of LEVEL products which provide questionable value coupled to an unquestionable overuse of plastic chrome.

Apple has Beats. In an interesting turn however, I was reading a rumour about the iPhone 6 potentially dropping the 3.5mm headphone jack to make the device slimmer. This coincided with Apple updating their peripheral design rules to include headsets running through the lightning connector. I thought, "To get people to accept the loss of the 3.5mm format, you would have to make the replacement WORTH the jump. Hey, I've just been working on this for the last few months!"

The gist: The 3DS as a virtual headset had me thinking about coordinating two geomagnetic sensors (compasses, as in your phone). If you have one in your hand and one on your face, you could make them work relatively to each other. So the gamepad could be your gun, and the sound would seemingly originate from that point in space, even if you hold it backwards over your head!

If we put a geomagnetic sensor in a pair of USB earbuds, when you're looking directly at the screen you are getting an "even" mixdown of the 5.1/7.1 audio channels, with the rear channels being modified by an equaliser to simulate sound hitting the back of your ears. This is quite similar to the "virtualised" 5.1 found in single-driver gaming headsets, but without the need for heavy time delays and echo to be applied to "expand the sound stage". This sound stage is real and dynamic.

When you turn your head, the mix of the audio will shift. If you were to look at the (virtual) left front speaker, its audio would be 50/50 to each ear, the left rear speaker would become more clear as it shifted from behind you, the centre would become right biased, and the right side speakers would become less clear as they shifted into your "hearing shadow".

This decoupling of visual to aural information is actually incredibly powerful, a movement of just a few degrees is enough to trick your brain into creating a 3D audio space. If you were to play games like Call of Duty, moving your head as if you were wearing a TrackIR (turn your head, not your gaze) would give you an incredible awareness of where enemy players are, making you able to track people through walls while visually covering another entrance.

This is the first gaming headset which ACTUALLY gives you an edge, and the hardware is so simple that it can be retroactively applied to existing games (as long as they have 5.1+ output). The only technology involved is a USB 5.1 sound card and the same sensor array found in every mobile device. It's cheap and tiny, it can even be integrated into an earbud!
 

Cubits

Well-Known Member
#2
While the gaming headset is easy to produce and offers real innovation in a stagnant market, the pack-in earbuds every phone manufacturer ships is where I really want to be.

With these, you'll be able to enjoy movies with full surround sound, where ever you are. By coupling the sensor to that of your phone/tablet, it's also possible to compensate for heading changes. So if you're watching a film while on your commute, bus/car/train/plane turns are negated by the bearing change of the device while allowing your head movements to still create the full audio experience.

We can take this a few steps further, too. Much like my Virtual Boy 2, this device creates new opportunities for developers.

1) Virtual Tour Guide:

If you don't know about the amazing 3DS Louvre museum guide, you need to see this:


Using the same Wifi triangulation technique to locate you within buildings, it's possible to build an interactive audio guide for your phone which uses your body and head bearing to "place" virtual guides at points of interest, and have them call you over to explain things.

So you could be walking along and hear someone call out to you on your left. You stop, look in that direction, and listen to them for your relevant information. Then you could pull out your phone to see more information (i'd like AR to show you the structural lines/focus/points of interest on a painting in real time).

This could also make an excellent schizophrenia simulator.

2) Zombies, Run!

You can literally hear the Zomies snapping at your heels as you run. You look over your shoulder, they're right on you! A survivor yells out ahead, urging you on, urging you to take the next left turn to safety!

This app already uses the GPS and compass of your phone, but the headset can use 3D audio cues to help with more natural wayfinding. This could also expand to general GPS voice navigation, but i recommend against wearing earbuds while driving. :p

3) My mobile horror/puzzle game:

This came to me off the back of imagining Fatal Frame in VR. "You're in a haunted house, the only way you can see in the darkness is through the camcorder in your hands. All around you, you hear thumps, creaks, whispers. You have to put the spirit(s) to rest before you can escape this nightmare!"

Your phone is the camcorder, you hold it sideways, it's your window into the 3D game world. Like the Wii U Zapper, or "Face Raiders" on the 3DS, your view movement is geomagnetically based (west is west, etc). Movement is by virtual slider (or tapping on the ground... i haven't decided).

As you move through the house, you'll be subjected to ghostly activity (walk past kitchen, hear a noise, go back, see the chairs stacked on the table). Clues will be hinted at by this activity, like looking at the back of a painting which falls down. You'll also be subjected to whispers, which you need to "hone in on" by tilting your ear to them. Sometimes they'll be helpful, sometimes they'll be terrifying, and sometimes they'll be designed to force you to take your eyes off the screen! Cue ghost attacks, hiding in cupboards, and general pant browning activity.

I imagine playing this in a dark room at night, and it sends shivers down my spine. This is a mobile game which is genuinely scary!

Most people write off mobile games as being shallow, fleeting, cash-grabs, but the platform's unique form-factor HAS to be able to provide new and exciting, legitimate gaming experiences. I'd love to put this game together, and prove that to be true! Any Unity devs here? :p
 
Last edited:

Cubits

Well-Known Member
#3
Currently i'm finishing up some documents for a meeting with a patent lawyer and an electrical engineer i've done work with in the past. He's just shipped his first product recently, and has recently been in talks with a few headphone manufacturers, so that's going to be an amazing opportunity.

Next i'd really like to talk to a programmer who could help me make a demo app for android or iOS. All i need to do is use the internal compass of a phone to dynamically mixdown 5.1 into stereo, and apply a graphic equaliser to the area behind your ears. What this will do is let anyone with a phone (and regular earbuds) experience this technology on a more basic level. You pop your phone on your head, and listen to the shift in audio! If you had a Chromecast, you could even watch a movie while getting the surround sound!

Then it might be off to kickstarter. Low tier would be a set of earbuds when they're done and the demo upfront (to play with/show to friends). Stretch goal would hopefully include my horror game. :>
 

running_in_jam

Videogame Sound Designer and Composer
#4
Sweet idea Cubits :) You should look in to Binaural Audio technology. Using HRTF's (Head Related Transfer Functions - basically a filter) you can create more realistic spacialised sound than surround sound offers. The other benefit is that you only need 2 channels. So the technology exists to do this in real time, but the input method of head-tracking hasn't been applied yet to my knowledge.

I haven't been keeping up with the progress of binaural audio lately, but there are a bunch of experimental games which are really interesting. People often consider what will be done for audio with VR technology, and I think binaural audio will be the solution. If you can marry head-tracking with HRTF processing tech then you will be on to a winner ;)

By the way, this tech only really works with 2 channel headphones right now. It's a very complicated can of worms to have it work over speakers which are far from your ears.
 

Cubits

Well-Known Member
#5
This IS transforming surround sound into a binaural recording with HRTF's! That's all it is, but with the addition of dynamic adjustment of the source to add another layer of depth.

HRTF's are what i'm using in terms of adjusting the audio relative to the head orientation. We're taking a clean sound from a virtual speaker, and transforming it to reflect it's approach to the eardrum before collapsing the sound into stereo. Modifying for the outer ear isn't too difficult to fake, you can use a gradient of "hearing shadow" behind the user with a few steps of equaliser adjustment and delayed cross-talk to reflect the resonant steps and internal resonance, and a smoother volume adjustment. People are also good at learning what a variation in sound means, so even though everybody is different in terms of head, body, and ear consistency, as long as they're given a good approximation and some reference points they can still operate well.

So you see a guy in front of you make a noise, then you hear the same noise but don't see the guy, again with a 50/50 split, so he's behind you. Note that difference, repeat, and your brain solidifies the model.

Curiously, a few months ago my dad actually became the CEO of an audio company. They've developed a 2.1 system which creates what they call an "audio hologram". It actually works incredibly well, delivering massive output at a focused point in space, which frees it up to have terrific range and response. Their website and shop front is horrendous, as is their utter lack of marketing, but the product itself is pretty cool (although entirely reserved for audio nerds).

http://www.acoustic3d.com/
 
Last edited:

Cubits

Well-Known Member
#6
Ok, everyone has been taking about Eternal Darkness again, and i've been on a horror movie bender, and i just can't shake how awesome this "haunting-based" mobile game would be. Played in a pitch-black room, with the positional 3D audio, the immersion could be absolutely terrifying/hilarious! And just imagine the way classic ghost-movie tropes could be abused!

It's a 3D game set inside a haunted mansion. You have to solve the puzzle of the haunting/put the soul(s) to rest to get out. The houses are designed as shortish experiences, each should be able to be knocked out in 15 minutes or so. That keeps it tense and fresh, and lets each house focus on a small selection of techniques.

Gameplay will be from a first person perspective, your view of the phone/tablet screen is the viewfinder of a camcorder in night mode. So green hues, interlacing, narrow field of view, that sort of thing. Also, grainy like Dementium.



Your looking is handled like the panorama view on the Wii U, you look where you point the "camera". In a black room, this will be convincing. Movement will be handled by tapping on the touch screen, i hate virtual sticks.

Solving puzzles will require you to investigate items in the house (like phoenix wright, another code, etc), manipulating the environment (mannequin puzzle in silent hill: shattered memories, or raven puzzles in zombiU), and using audio cues like whispers, footsteps in the attic, chandeliers rattling, etc to direct your attention. There SO has to be a piano puzzle, too! :p

Once you follow the clues, you get to burn the remains, or the possessed item, or what-have-you, and you're done!

But the whole time you're doing this, you're also being haunted!

"You hear a laugh, like from one of those creepy cupie dolls with the opening eyes. You wheel around with the camera to see what made that noise. At the top of the stairs is the doll. It doesn't move, even when you walk right up to it. You press onward. After a few more rooms of surprise visits you end up in the kitchen. You hear a knife being drawn from the block. You turn to face the knife block, only to see it wobbling in place from the removal of the 12" chef's knife. You hear the doll behind you! You spin around, only to see nothing... You turn back and.. BAM! FREAKY DOLL IN YOUR FACE. WITH. A. KNIFE!"

Classic.

TL;DR? This about covers it:


With a bit of this thrown in:


And this, of course:


Other cures for constipation include shadow/slender men in the corner of your view momentarily, a ball bouncing down the stairs, self-arranging furniture, shit happening in mirrors, apparitions acting out their violent ends... there is just TOO much content to play with!

"Fighting" ghosts could be as simple as using the touch screen to take a flash photograph, but that might be TOO obvious. How about writing warding spells on the wall in your blood using the touch screen?

I can't imagine it would take too much work to build these fairly small levels in Unity, i've made much bigger levels with custom textures and ambient sounds before. I'd just need a programmer with experience in Unity to get started, really... *cough* @DarkDepths @CyanPrime *cough*

Oh, and the ultimate way to play this game would be by giving it to a friend, putting them in a pitch black room to play, hiding in that room, and then scaring the living shit out of them at the right moment. I would give so many monies to be able to do that! And it would go on Youtube, and it would be viewed more than any actual, intelligent content, and i wouldn't care!

Eternal Darkness meets Dementium meets Phoenix Wright meets Supernatural (Ghostfacers!).
 
Last edited:

Cubits

Well-Known Member
#7
I'd organise the payment model like zen pinball: Give the people a freebie or two, then charge for tables after that. So you'd get a basic haunting, but the discrete "packs" let people pick and choose from the mountain of possible themes.

There'd be an amityville horror "gateway to hell" one, the possessed doll of course, a "what lies beneath" water-based one (those are always good). We could even have some whacky holiday specials like a nightmare before christmas aesthetic, facing a guy who fell off the roof playing santa for his kids (cue heartstrings), or even a scooby-doo "it's old man withers in a suit" gag. I'm as down for comedy horror as i am for a trap-room which fills with blood from the light fittings as i am for doing an emotional "tearjerker" about lost loves (although that's WELL worn ground in the "walk around a house" genre).

Since you're using a camera to view the world through, taking photos of clues would be a good way of tracking information. The ingame flash could also activate the one on your phone for a genuine blinding, burning the shadows of the things in your real-life room into your eyes.

There also has to be a bit where the battery in the camera craps out and the screen goes black, leaving you to be led through the area by following a friendly(?) whisper. "over here... over here..." With the right timing, in that dark room, your hearing would feel superhumanly sensitive, and that would HAVE to be taken advantage of. Gentle creaks, faint footsteps, a violin?!

Face demons, collect indian bones, complete last wishes, play a piano, burn a doll, and avoid being turned into a Jill sandwich!
 

TheAmazingLSB

PLEASE UNDERSTAND....
#8
Dude, I just found this....

And wow.... Fuck is wrong with my spider-sense lately...?

It should have tingled the moment this thread was posted....

04d5dcf230878b4b8ef9228a4bb75f6516f1ecbc1602dbe4abc64fd829f81cfd.jpg
 

running_in_jam

Videogame Sound Designer and Composer
#10
Sorry for misunderstanding you there Cubits - it sounds like you're more well read on the subject than I am :D I'll be very interested to keep up with your progress and your ideas.
 

Cubits

Well-Known Member
#11
Patent application is in, so i have preliminary rights to this as of now! Woo! Hopefully no tech giant has already covered this, we'll get to that bridge later. At least they can't now file a patent after me and take me on about it. ;)

Next step is to make a video showing it in action, and to get a demo app running on my phone. The video part is already worked out, it'll be great!
 
Top