UI lit by the lights in your room

UI lit by the lights in your room


Hello. My name is Bob Burrough and this is a demonstration of project Erasmus. Project Erasmus is an implementation of a user interface that takes environmental lighting from the environment that the
device is in and it renders that lighting in a scene in the software, so
that you can light, shade, and reflect the environment on the user interface in the
device. So what you’re seeing here is a settings screen which is just a very simple little demo application to show how this works, but if I move the device around then what you will see is the elements
in the user interface react to the lighting of the room that we’re in. So right now I’m in my son’s room and there’s a light immediately behind us
and you can see shading and specular highlights reflected on the elements in
the screen, which is actually a really impressive effect. It actually looks like
the user interface elements are physical objects that reside just
beneath the surface of the screen and like you could reach in and touch them.
So the way it works is the the Ollo clip at the top, that’s a fisheye lens,
it captures a wide-angle shot of the environment and the software projects
that as an environment map into a scene which we then use to construct lighting
and reflections in order to light and reflect on the user interface elements
in the device. So what I’m going to do is move into a darkened environment and
let’s watch and see how the how it reacts. It went away immediately when I
walked away because I walked into shadow. So let’s see
if we can do that again. So like if I move such that the device
falls into shadow it does what you would expect. Since the device is in shadow the
UI is in shadow too, so there are no lighting effects while the device is in shadow as soon as you come out, boom. Now you get lighting on the user interface and it reacts how you would expect if this was a real
physical thing in your hand. Anyway, let’s go back into the darkened environment.
The thing that note in the darkened environment is that there are no
lighting effects whatsoever. If this is how you wanted it to work. That’s totally
up to the user interface designer if they basically want it to be this flat
implementation while there’s no light, but you don’t have to do it that way you
could do other very interesting things like render a backlight behind these
elements just like it would on a MacBook Pro keyboard in a low-light environment.
Which that’s something that I’ll be doing eventually but what I’ll do is I
will I’m gonna move into another differently lit environment and this is
actually quite a bit brighter scene than we had any other room, but you can see as the light in the environment changes then the lighting on the user interface
changes. It takes on the shadows from the environment that you’re in
and again the specular highlights from the lighting in your room and if we move
around the shadows change, the specular highlights come from the other direction
because that’s what you would expect it to do in real life. And that’s project Erasmus.


100 thoughts on “UI lit by the lights in your room

  1. This is pretty cool and I want it but I'm afraid that it's too late for user interfaces, they've gone from that 3d look. But I'd love to see how this will help in 3d modeling programs as they move to phones and tablets.

  2. i remember some of the sliders in ios 4-6 use to do this when you would move the device but i think it was only done with the accelorometer, are you the guy who did that?

  3. This is actually awesome! As soon as this can be scaled down and placed inside a smartphone I think we will see a new eara of how UI is done. This could be used to make interfaces feel more alive and interactive. It could make stuff look clean and minimalistic without looking flat or make actually realistic looking designs. This has huge potential if it manged to become something mainstream on devices.

  4. Already patented by Apple.

    https://www.buzzfeednews.com/article/benjaminj4/what-developers-thought-about-wwdc
    "This tiny detail in the iPod app noticed by a Gizmodo reader is an early sign of what will soon be a sea change in interface design. One Apple employee (who I can't name as the company does not allow employees to speak on the record without approval from media relations) said that in the future, your phone will show drop shadows based on the actual position of the light in the room, as detected by the phone's ambient sensor — and everything in the UI will be rendered in 3D on the fly."

    https://www.patentlyapple.com/patently-apple/2010/04/apple-moves-os-x-3d-closer-with-unique-ambient-light-feature.html

  5. I thought Apple moved away from the skeuomorphism design and prefer using blurs. This is cool and all but it might be too taxing on the device.

  6. I really hope this is implemented somehow and not just a jailbreak tweak. It deserves more attention. You’re a genius! Keep up the amazing work!

  7. This video shows Apple DNA.. Apple is too much focus on design, elegance over functionality. When we look at iPhone, it's design is everyone crazy about.
    This is oppsite to Android. Android focus on functionality, convenience, much more deeper settings and deeper customizations.

    I won't say iPhone are not functional or useful, same to android, I also won't say android phones has very poor design. I'm just pointing different OS with different priorities

    So, When you think deeply about this, ask yourself.
    Is this really necessary ???
    The answer is No.
    Great idea? Cool design?
    Definitely YES

  8. An interesting idea but what problem is this really solving, and how much additional cost just to support this in hardware is anticipated?

  9. Remember how proud Apple was of parallax in iOS 7? I wish Apple still cared as much about making tech interesting and new.

  10. It is impressive and great work ! But it is as impressive is useless. The point of having a screen and one of its advantages is precisely when it doesn’t react to ambiant light hence better viewing experience on the screen. You won’t be able to use your phone in the dark or outside in sunny day. Impractical.

  11. Amazing. As an industrial designer, this made me think about a lot of potential product designs. I wish we could collab :')

  12. Very interesting execution of the idea, looks awesome. Its a shame a fisheye lens is required to correctly map the simulated light in the scene to a real light source, with proprietary hardware it would propably be possible to fit such a sensor into a smaller formfactor. Still, although this would be great for certain games or to help the user distinguish UI elements from another, this is probably draining the battery quite a bit.
    I was wondering, was this completely coded in XCode or did you use 3rd party Programs (e.g. Unity3D) for the light/UI rendering? It would also be interesting to see what the actual front camera sees, I assume the tape on the lens diffuses the light in a way so that you end up getting a sort of bright "orb" of light on the image, corresponding to where the light is coming from. Checking the distance and angle from the center of the image to said light that appears on the lens is how I'd imagine this was done, quite clever!

  13. This looks really awesome! Some in this comment section even say, it's the next step of UI.
    But so far I can't see the big advantage over a flat UI? What do you think?

    Looks good doesn't count for me 😛

  14. Really cool. Is it YouTube broken or you have more then 6 million subscribers but only about 160000 views total on your channel ?!?

  15. very cool idea, but now i'm wondering: could it be possible to get a simillar effect by just using accelerometer and gyroscope? kinda like parallax does but with lighting

  16. It's a neat effect and I respectable the engineering feat. But I'm curious of the practical uses and value to consumers for the increased production cost in adding a fisheye lens to mobile phone hardware just to implement this aesthetic… It's simulation of real-time lighting/shading based on gyroscope movements, ok… It makes for a real cool YouTube demo, but then we get to cost of production, implications to trending design and value to consumer. Is this worth an extra $100 to consumers even say $50? I don't think so. Would someone pay for an add-on that has to be attached to the top of the phone as shown in the video? No way. Maybe a niche geek crowd. Maybe that is the market you're after? Not without some really strong use cases beyond UI novelty. Would a phone manufacture make room for a fisheye forward facing camera in addition to the normal forward HD camera? accommodate this asthetic? I don't see it, but maybe. We're in an age where everything is about shrinking the size of the non-essentials in phones to make more room for battery, etc… Cool engineering, bu I don't think it's practical given it's hardware requirements and current internal down-sizing trends in mobile. Good luck.

  17. Camera required? Fisheye required? That kind of apps will drain your battery very quick. Also, you need to give permission for a camera, so there is privacy issues. UI IMHO, looks really bad, and I can't imagine UI where real reflections is so important, maybe better use CoreMotion+Environmental maps.

  18. Most of the comments are just polite and of course every new sensor on a device introduces lots of new usability options. Sensor to detect light source intensity and direction could be useful but this UI use case is just stupid waste of CPU power. This implementation also looks poor, simple and basic. And having patent and live demo at this stage just makes you look greedy …

  19. Finally something really better then new gold color. We miss people like you in a big business. Great man and idea

  20. Texture… You know you can put the environment on a shiny virtual object as texture if you use the camera as it is with the wide lens. Might look good. Add some face – eye detection for one person 3D immersive view also… I guess you will need a special sensor for all the stuff if you don't want to drain the battery with the camera.

  21. With all the respect for the creator, this is pure Apple stuff – useless gimmick… I just want my job gets done but wait… Battery is dying.

  22. A Simple Idea with spectecular impact on the look an feel of a ui. Potentially could be big!!
    Congratulations 😀

Leave a Reply

Your email address will not be published. Required fields are marked *