Shadow DOM / Polymer forms do not fill. (companion extension v4.7.4 now supports shadow DOMs)

ChrisEsplin

I'm building a login flow that can be found here:

https://res-ipsa.firebaseapp.com/login

It uses Polymer 2.0, which means Shadow DOM!!!

So Shadow DOM is the future because it provides web devs with great component encapsulation. You can't use document.querySelectorAll to dig through my shadow root!

Learn about Shadow DOM here: https://developers.google.com/web/fundamentals/getting-started/primers/shadowdom

Of course, this breaks 1Password's Chrome extension. I hope there's be a way to instrument my app to work with 1Password. Are there any window object hooks that I can use to pass the correct input elements to the extension??? If not, will you add some hooks? Pretty please?

---

1Password Version: 6.3.2
Extension Version: 4.6.6.90
OS Version: OS X 10.11.6
Sync Type: Dropbox
Referrer: forum-search:polymer

jxpx777

Thanks for your post, @ChrisEsplin. We haven't always had such a friendly relationship with web developers, so especially for an old-time 'Bit like me, it's always lovely to hear from web developer that want to make sure their sites work well for 1Password users. :blush:

That being said, this is going to require some investigation. Right now, 1Password does rely on querySelectorAll and other "normal" DOM inquiry methods to locate fields for saving and filling. The Shadow DOM approach you have on your example app (Thank you for sharing the site you're working on!) is basically opaque to 1Password's current saving and filling code.

The Google link you posted is interesting and has some code for collecting custom elements. In doing so, I see that you have a module that includes another module for the auth and that module only has a single <script> tag, so I'm not seeing where we can dig into the actual text field elements themselves. I'll admit to being fairly new to this so it's possible I'm simply overlooking something.

I'm sorry I don't have more details to offer right now. I'm going to consult with some of our web dev team to see if they can offer any insight as well and hopefully we'll be able to handle this kind of development technique properly soon.

--
Jamie Phelps
Code Wrangler @ AgileBits
Fort Worth, Texas

ChrisEsplin

@jxpx777 Thanks for the quick response!

Shadow DOM is very opaque. There are hacks to break through it... but they're ugly, and I'd never recommend them.

My first inclination is to create a way for me to message to the extension. I can send the extension the actual input elements, so you won't have to do any querySelectorAll.

The only other thing that I can think of is hackery on my end. I could create top-level hidden inputs outside of the Shadow DOM. If I can get 1Password to recognize them, I could then have my login form attempt to fill itself based on those hidden inputs.

I'm curious which method your dev team would deem to be more secure.

jxpx777

Thanks for the additional details. Opaque is indeed the word I would use. :smile: We have talked about creating an API for website makers to opt into, but so far we haven't been able to clear the decks enough in order to pursue it. I'd love to hear what you would like to see from such an API, whether you want to share that here or via email instead.

ChrisEsplin

I'm seeing three ways this could work.

1. The developer adds a tag and attaches DOM elements to it which the 1Password extension can then read.
2. The 1Password extension adds a function to window.register1PasswordElement()—or something like that.
3. The 1Password extension listens for bubbling events, maybe called something like 'register-1password-element'.

I'm not very experienced writing extensions, so I have no idea if these routes would even work. But that's kind of beside the point.

All we'd need is a way to send an element to the 1Password extension. The extension would do the same analysis that it always does to evaluate which kind of element it is. Then it can track that element on the page, and when it's about to autofill, it would just need to check that the element is still visible in the DOM, and then it could autofill it as always.

jxpx777

Thanks, @ChrisEsplin. This sort of matches what we have batted around with the exception that it will probably have to be based on a custom event system since the browsers' extensions APIs separate the JS execution contexts of the extensions and the pages. I don't have anything more to add to this right now, but it's definitely something I'm interested in looking at.

ernsheong

I'll add another case study here. 1Password does not fill up for my site in Production: https://app.pagedash.com/ It's built with Polymer 2, already a user complaining at

https://twitter.com/chiaisthewei/status/935534425198252032

You're gonna start seeing more and more websites using Custom Elements and Shadow DOM moving forward as the browser standard progresses. This is definitely something that should be addressed short/medium-term. Custom Elements and Shadow DOM are definitely a thing now, see

https://twitter.com/DasSurma/status/935504539737837569

A tip is that 1Password can definitely crawl through the custom elements via shadowRoot. See my issue at https://github.com/w3c/webcomponents/issues/657. Whether or not 1Password can modify input values and submit will be up to 1Password to experiment.

AGAlumB

@ernsheong: I got really excited til I saw "Firefox 60"! I thought maybe this was something I'd missed in 57. :lol:

But in all seriousness, it's really great to hear from you — and to think about the possibilities here:

Whether or not 1Password can modify input values and submit will be up to 1Password to experiment.

Couldn't have said it better myself. I bet Jamie could have some fun with that. Cheers! :)

e111077

Hi there, I'm wondering if there is any update on this issue.

In terms of the previous suggestions: it is possible to traverse the .shadowRoots of elements, but that would be quite costly as it would require recursive calls.

It may be easier to safely patch attachShadow in the prototype and keep track of the roots that are being attached.

Additionally, it can be 0 extra cost if you just add an API on window or something.

littlebobbytables

I'm sorry but there is no update at the moment. I wish a could feel psyched about either option but the recursively search all legal elements would likely lead to the extension having a detrimental impact on the browser depending on the page. I've seen home pages with 500 input fields let alone all the other elements to consider. Overriding the various prototypes... I suspect it wouldn't work on iOS given how differently iOS extensions work and I just wish we didn't have to even consider such steps. Maybe my gut feeling is wrong and I'm being unnecessarily apprehensive.

littlebobbytables

@ChrisEsplin, @ernsheong & @e111077,

We've just released a new beta with just one small change. The single changelog entry reads as

[NEW] Support for shadow DOMs. {139}

Now I would like you to temper any excitement you have as this is very much experimental and we're not ready to push this to a stable version just yet. I believe the code should be fairly robust but we're very sensitive to the potential for performance degradation and not just on pages that are using shadow DOMs, the checks we need to do to locate shadow DOMs have the potential to impact other pages as well. Given you are all passionate about shadow DOMs I'm hoping you can help me out. The beta extension is available at 1Password Browser Extensions and we're not just interested in how it works on pages with shadow DOMs but also if we find pages where things are slowing down because of the same checks. To assist the extension will log to the browser console if the shadow DOM check takes more than a second to complete. All it reports is the duration, it doesn't reveal anything else.

So what I'm hoping is you can run with the beta. Make sure it works with pages you know happen to use shadow DOMs but if you find your browser feels sluggish can you check the console and see if 1Password has reported anything. If it is caused by the shadow DOM check at all you will see an entry that looks like this

[1Password] Delay due to shadow DOM check was X milliseconds.

If you don't see this logged to the browser console for the active tab then any sluggish behaviour isn't the direct result of the new code.

The only option for handling shadow DOMs from an extension is unfortunately a traversal of the entire DOM. If we find one shadow DOM we have to repeat until we've exhausted every level. I picked a couple of example sites I know have caused bottlenecks for 1Password in the past and I was surprised that the shadow DOM check completed as quickly as it did. It might be I was being overly pessimistic at the impact but we can't say with any confidence without some real world testing.

robeden

How does that browser extension play with the Mac 7 beta (5, for safari) extension? Is it possible to run those two betas together?

AGAlumB

@robeden: If you mean the 1Password for Mac beta and the beta for the browser extension, then yes, those are designed to work together. You don't have to be using the beta of both, but you're welcome to. We always appreciate any feedback you can share. :)

AGAlumB

@robeden: I think I misunderstood your question. If you're running 1Password for Mac version 7, there is no support for a separate Safari extension; it's built into the app itself. I hope this helps. Be sure to let me know if you have any other questions! :)

robeden

@brenty - Yes, this matches what I'm seeing. I'm running 1P for Mac version 7.0.0-beta5. If I try to install the beta Safari extension linked above and disable the built-in extension, it's unable to connect. The built-in extension isn't working with a shadow DOM site I'm testing, but I assume that's expected (i.e., it doesn't have the shadow DOM code)?

e111077

I was testing on the Chrome extension on certain shadow dom sites, and so far it has been working. I haven't been able to test one with a login and shadow dom (1Password's auto-password selection), but it has so far been working with credit card information. I've also tested large sites such as YouTube in shadow DOM (it displays oddly because they haven't fixed their styles to support shadow dom, but all of the nodes and functionality is still there. YouTube Gaming is probably a better test since it works in shadow dom) and I have not gotten the console log to show up.

If you would like some test sites:
Our demo shop site (credit cards checkout): https://shop.polymer-project.org/
YouTube with Shadow DOM (large site): https://www.youtube.com/?dom=shadow
YouTube Gaming with Shadow DOM (large site): https://gaming.youtube.com/?dom=shadow

littlebobbytables

Hi @e111077,

Thank you for those additional sites. I'm hoping you won't need to post with details of sites where the extension experienced delays but please do if you find any.

@robeden, it won't be possible to test a shadow DOM aware version of the Safari extension with the beta version of 1Password 7 for Mac I'm afraid. We will eventually ensure everything is in place so that the beta versions of 1Password for Mac use the latest beta version of the extension but we've got some work to do, ensuring all the right stuff is in place. You will still be able to test a shadow DOM aware extension in any of the other supported browsers with the 1Password 7 beta though, just not Safari.

ChrisEsplin

@littlebobbytables Thank you for the Shadow DOM traversal! I use 1Password for everything, and I develop in Shadow DOM whenever possible, so I'll let you know if I run into any issues.

The new 1Password releases are killer. This is such good news for my workflows.

robeden

Under Chrome, it's not finding the login fields on this site: https://vaadin.com/

littlebobbytables

Hello @robeden,

I just tried saving a Login item for that page and then filling and it all seemed to work okay for me so I have a couple of questions.

1. Did you save the Login item from the registration form or the login form?
2. Did you manually navigate to the login form and then ask 1Password to fill the open page or were you using open-and-fill?

I find open-and-fill didn't work but it can be a bit hit and miss for pages where you need to interact with the page before the form is revealed.

MilSF1

I'm having issues with ShadowDOM and extensions on macOS Safari, Chrome, and Opera. iOS 12 Safari seems to have the same issue. Firefox seems fine. The console error I get:

Error in event handler for runtime.onMessage: TypeError: Cannot read property '0' of null
at f (chrome-extension://fpnbobholfpcolmkinlokiaaanjilcop/injected.min.js:53:166)
at Sa (chrome-extension://fpnbobholfpcolmkinlokiaaanjilcop/injected.min.js:54:146)
at Pa (chrome-extension://fpnbobholfpcolmkinlokiaaanjilcop/injected.min.js:39:173)
at chrome-extension://fpnbobholfpcolmkinlokiaaanjilcop/injected.min.js:49:214

Or I can get:
Error in event handler for runtime.onMessage: TypeError: Cannot read property 'toLowerCase' of undefined
at g (chrome-extension://phicbbndgmmpogmijjkbmdhpioaieaha/injected.min.js:66:405)
at chrome-extension://phicbbndgmmpogmijjkbmdhpioaieaha/injected.min.js:72:257
at Object.collect.c (chrome-extension://phicbbndgmmpogmijjkbmdhpioaieaha/injected.min.js:71:143)
at Va (chrome-extension://phicbbndgmmpogmijjkbmdhpioaieaha/injected.min.js:43:54)
at Ra (chrome-extension://phicbbndgmmpogmijjkbmdhpioaieaha/injected.min.js:39:297)
at chrome-extension://phicbbndgmmpogmijjkbmdhpioaieaha/injected.min.js:49:214

The second error looks to have been fixed in 1Password X recently, but can't find an instance of the first here. Using latest versions of 1Password 7 and extensions.

littlebobbytables

Greetings @MilSF1,

So the first error is when 1Password 7 sends an empty set of filling steps to the extension. I've mostly observed this when there are iframes present on the page and the error shouldn't affect filling if there is nothing in that iframe that you would want 1Password to fill. So you can see that error even when things do fill correctly.

The second I'm not sure I've seen yet. How would you feel about sharing the URL for the site where you're seeing this as I would like to try and reproduce the error and determine what is happening.

MilSF1

@littlebobbytables I guess so. I might delete this post after you find any issues though. I'm the dev for the site, so I can make changes as needed. No iframes. Would a tight CSP have anything to do with this? Oh, and it doesn't appear to be working with Firefox anymore :-\ Thanks for taking the time to look into this though. FYI, I'm using the latest versions of Vaadin's web components for Polymer 2.

https://[[redacted]]

macOS Mojave
Opera 56 - 4.7.3.90
Safari 12 - 7.2.1
Chrome 70 - 4.7.3.1
Firefox 63 - 4.7.3.1

littlebobbytables

Hi @MilSF1,

I've located the issue and it won't require any changes at your end. The second error is causing the issue of being unable to fill or save from inside the browser. Using the site what I've discovered is there's a point where we check for the possibility of a descriptive list as its something we've seen used a few times but the code is assuming an element has a tag and shadow root nodes won't. With the an extra guard in place I find 1Password will fill and save.

MilSF1

@littlebobbytables :+1: Wow, 42 minutes. Now that's customer service 8-) I imagine walking ShadowDOM nodes can be a bit tricky especially as they can be nested rally far down and how things like 'label' can be a bit non-standard between components. I'm kinda surprised y'all haven't gone with an "add this attribute to the web component" approach. Not that, as a developer, I want to add more edge case code, but it definitely would be easier on y'all.

littlebobbytables

Hi @MilSF1,

I hadn't had a nice puzzle like this for a while so it was fun to exercise that part of the brain again :smile:

I'm curious what you mean by "add this attribute to the web component" though. I certainly wouldn't consider myself an expert at all so always willing to believe there's something new to learn.

MilSF1

@littlebobbytables

I was referring to having something like Safari/Chrome's autocomplete syntax. I haven't had the time to try that code with web components, but I imagine that is already on your DOM crawling checklist? I'll check for a beta in the next few weeks to see if your change made it in. Thanks again.

littlebobbytables

Hello @MilSF1,

The extension pretty much does grab all as it looks for a variety of attributes and bundles everything up in a single message back to 1Password about a given page/frame. Our first play with the autocomplete attribute focussed more on Credit Card and Identity filling but newer versions of 1Password also factor in the presence of the tag for Login items as well. Even if the that tag is present though the script still attempts to look for associated visible text as they're often a good way of determining the purpose of a field or button.

I do wish autocomplete was more widely adopted as it's just useful. It says a lot though that on reading up on that attribute I learned most browsers now actively ignore when a page sets it to off because it was being misused under the guise of "security". When it's available though it really does help cut through so much of the guesswork. What I haven't seen before is the one-time-code value, that's new to me. I wonder what scope that has as it has a very narrow definition based on that particular Apple support document. Definitely worth keeping an eye on :smile:

ernsheong

Hi all, it's 2019 and Shadow DOM/Web Components is becoming mainstream (it's not just Polymer anymore). Can we please have support for web components autofill? Traverse the shadowRoot all the way until you find an input that matches... I'd be happy to help your engineers (I've spent a lot of time in this land).

littlebobbytables

Hi @ernsheong,

It's already in the beta version of the companion extension, the one that works with 1Password for Mac and Windows and it will make its way to a stable version soon.

If you use Chrome or a Chromium based browser then you can find the beta version of the extension in the Chrome store. If you use Firefox then you'll want to download it using this link.

If you'd prefer to wait for the update to the stable version then I'm hoping it won't be too long.

ernsheong

@littlebobbytables that's amazing! I tried out the beta and it works on my sites. Kudos to the team!