Show HN: DIY virtual HDMI monitor using "AR" glasses
What it does is map the frames from the HDMI input onto a virtual display that is controlled by the IMU data from the glasses ( 3DOF only ). I've put AR in quotes in the title because many won't view those display glasses as true AR but by tracking the head movement it comes close.
I am trying to build kind of a "low cost" version of a virtual screen that acts like a monitor and can be connected to anything that has an HDMI output
I started off using the official Viture SDK to interact with the glasses but have since switched to a reverse engineered implementation of the protocol because their SDK is not available for ARM
Here is a video showing the first version: https://www.youtube.com/watch?v=D6w5kAA22Ts
Big caveat: Performance still needs to improve a lot because the whole frame reading/converting is completely unoptimized for now.
What other solutions do exist out there? * Streaming the computer screen to a headset like Meta Quest/Vision Pro * Connecting a HDMI capture dongle to the Meta Quest directly * XReal Beam ( basically the same as this project but official and for XReal glasses )
And for the obvious question, why I am not use something like a Quest or Vision Pro 1. Comfort 2. Price 3. Comfort
After using those display glasses over HMDs it's hard to convince myself to use a headset for productivity again
Some AR glasses do provide this feature, so it overcomes a product flaw in AR glasses and will at some point not be necessary.
Your project could also allow for other features that the AR glass manufacturer didn't think of, or has gated behind upgrades and product segmentation.
Is my assessment correct?
But yeh for a single monitor, I guess it takes a bit of getting used to but non-locked seems far more reliable
Unless I'm misunderstanding the feature, it seems like enironment locked screens allows for more natural usage and interactions with the screens in the virtual space?
My experience is mostly with VR/AR products like Oculus has been mostly with environment locked AR information.
It's like having a very nice monitor that uses ~1 watt of power and happens to be positioned exactly wherever is most comfortable without even having to think about it. It's way better than a normal monitor if you don't have to do eg pair programming.
I've heard people propose that these "screen in glasses" devices (like the Xreal Air) are useful for situations where you want a lot of visual real-estate but don't have the physical room for it — like in a dorm room, or on a plane. (Or at a library/coffee shop if you're not afraid of looking weird.)
---
Tangent: this use-case could likely just as well be solved today with zero-passthrough pure-VR glasses, with a small, low-quality outward-facing camera+microphone on the front, connected only to an internal background AI model running on its own core, that monitors your surroundings in order to nudge you within the VR view if something "interesting" happens in the real world. That'd be both a fair bit simpler/cheaper to implement than camera-based synced-reality AR, and higher-fidelity for the screen than passthrough-based AR.
† Which wouldn't even need to be a novel model — you could use the same one that cloud-recording security cameras use in the cloud to decide which footage is interesting enough to clip/preserve/remote-notify you about as an "event".
https://www.youtube.com/watch?v=D6w5kAA22TsP&t=19s
Is it me or is there some dynamic here that could/should be further exploited to implement what you suggest
From your experience would you agree?
As I said, though, I always only use a single monitor anyway, so YMMV.
After using both 3DoF and 6DoF for years, I just can't imagine using a 3DoF headset for more than watching videos or simple writing tasks.
I really wish the screen were smaller and more to the center, then they'd be perfect. I guess it depends on whether you want them for games (the important stuff is usually in the centrer) or for work (the important stuff is usually at the edges).
I really like the focal distance, though, it's around 5m away and pretty comfortable for the eyes.
I'm also planning to add other "display" modes like a curved monitor and in theory we could connect multiple monitors using several hdmi capture devices but that's probably even less useful
Do you know if this accessibility feature is available when using it with phones?
You could in theory use the approach I posted here and use a USB to HDMI adapter to connect the phone to device and it would do the same thing it does with a regular PC but that loses all the benefits of being small and portable if you have to carry around all that additional stuff
I've been thinking about combining it with our work on the reverse engineered HTC Vive Ultimate Tracker stuff here https://youtube.com/shorts/U4vhEPQw-Uo to get 6DOF
But realistically I ( and nobody else ) won't strap the tracker to my head all the time
By the way I find my Quest 3 way more comfortable than the Xreal. The problem with the xreal is, the prescription inlays are too small so I can see only a part of the display sharp unless I put in lenses every time. And it is kinda heavy on my nose and ears. the Quest is super nice, has a wider FoV. But it's heavier and bulkier to bring along of course.
PS I had no idea you could connect an HDMI dongle to the quest. I'll check that out.
Regarding the HDMI on Quest 3, I believe it was sometime last year when they made that possible https://www.meta.com/help/quest/1350679463006443
Thanks for the heads up about that Quest thing, I actually have some of those adapters so I will give it a try.
One thing about AR glasses that I don't get so far: Why do ALL of the AR glasses use only 1080p displays per eye whereas all the other VR solutions that also have a display right in front of your eye have even bigger resolutions?
I really do want AR glasses that can act as an alternative to a physical display. But they're always 1080p for some reason. The companies I've seen making them appear to me to keep fixing important issues and adding really useful features - but seem to avoid touching anything that would improve the resolution.
Edit : There's a part of me that believes the situation here is a lot like how Linus Tech Tips described the Dashcam situation.
What that means is, these glasses are made by someone paying $$$ once to Asian engineering companies to have it figured out, and everyone reuse that exact setup for years on.
And companies like Sony or Canon, they develop stuffs by scraping engineers off a wall instead of first throwing them at the wall, so every Sony or Canon cameras and projectors come with novel viewfinders and crazy patented lenses. Or panel suppliers like Kopin, Epson, or Sharp can arrange contacts to engineering consultants. I guess. Same likely goes for all the Chinese companies too, though I'm not familiar enough on that front to be able to offer googlable keywords.
Startups and even mid-sized consumer electronics companies don't have that kind of time or financial backings or margins to do the same. They barely manage to pay for assembly and ship it. And so the spec of the final product is whatever spec of parts that they could buy off the shelf.
What about Apple? Well, they pay to have 2 extra display factories built for redundancy by policy - I'm sure that most military organization don't do that. And even then they use a rather simplistic, completely concentric and rotationally symmetric optical design.
(I kind of have a crazy idea to bypass some of that, but unfortunately I'm crazy and so is the idea)
So what you get is basically a larger mobile screen for watching videos, something that competes with video projectors, but you can use it on the bus. It's an easier sell compared to a device that will give you a good reading experience and competes with paper books.
Those of us that want to use them as terminal emulators are still too few.
So if you have high res displays that no one can realistically use for the intended mobile cinema use case then you are wasting money and energy
The top-of-the-line solutions by Sony used in current products only go up to 1080p120.
Thing was unstable like 8-12 months ago
I see the glasses are connected (via usb-c?) to the OrangePi.
Is this project converting HDMI signal to the Viture protocol? if so, can't you run a screen grabber directly on your machine and send it to the glasses?
That's where this comes in, it allows you to look around the virtual screen with the added bonus that you can connect it to almost anything and behave the same not just your PC
Outdoors you see the e-ink fine. Indoors you look down at your illuminated sleeve to see the e-ink in your glasses.