MRC Converter suite: Enpass to 1Password - unexpectedly matched multiple outkey values for 'url'

charliesigma
charliesigma
Community Member
edited March 2023 in Windows

Hi!
I am trying to convert my Enpass JSON file to import on 1password by using MRC Converter, but I am getting the following error:

Error: Please report - unexpectedly matched multiple outkey values for 'url', type 'login':
record title: 'Adobe'

I have 2 URLs saved for that specific Adobe login:

                {
                    "deleted": 0,
                    "label": "URL",
                    "order": 4,
                    "sensitive": 0,
                    "type": "url",
                    "uid": 13,
                    "updated_at": 1519181631,
                    "value": "https://adobeid-na1.services.adobe.com/",
                    "value_updated_at": 1519181631
                },
                {
                    "deleted": 0,
                    "label": "Website",
                    "order": 5,
                    "sensitive": 0,
                    "type": "url",
                    "uid": 5684,
                    "updated_at": 1679156915,
                    "value": "https://auth.services.adobe.com/pt_BR/index.html",
                    "value_updated_at": 1606400647
                },

I'd appreciate any pointers on how to solve this, as I just subscribed to 1password and I have 400+ saved passwords.

Thank you!

Shamelessly paging @MrC , sorry in advance for bothering you!


1Password Version: 8.10.3
Extension Version: Not Provided
OS Version: Windows 11
Browser:_ Chrome

Comments

  • charliesigma
    charliesigma
    Community Member

    Shamelessly paging @MrC , sorry in advance for bothering!

  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    No shame in the ping, thanks for bringing it to my attention. Let me recreate it in Windows to be sure I give you a proper fix.

    This is a peculiar edge case, in that both Website and URL are allowed to match as the primary (1Password) url key for a Login record, and stock records would not have both labels.

    Back shortly...

  • charliesigma
    charliesigma
    Community Member
    edited March 2023

    For sure!
    The enpass parser does have something that I interpret as a treatment for such cases, under login:

        line 405
        [ '*additionalurls',    0, qr/^(?:URL|Website):::__MAIN:::(?!13$)/, ],  # support additional URLs within the main section
    

    I tried to mess with what I assume are regex statements there but I frankly don't have lots of experience with regex (not even sure it IS regex lol), especially when embedded in perl. Buut, I might be totally wrong in the previous assumptions.

    If I comment line 404 [ 'url', 3, qr/^(?:URL|Website)(:::__MAIN:::13)?$/, ],, then it parses but both URLs get added to the 'notes' section.

    If I comment line 405, it fails the same way described above.

  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    This converter has grown over time, to accommodate Enpass' growth of templates and export types. So there is a wide variety of labels and data in the wild. This is exacerbated by Enpass not canonicalizing it's imports from other password managers, leaving records as a bizarre mixture of old, new, and custom formats.

    The issue is that my workaround to avoid duplicate keys is not catching this case, due to your labels being both URL and Website. The simplistic match of any given field label (e.g. Website) against all the other field labels in the record isn't working.

    The field pattern matching template you show is where additional URLS should be placed. I have to change from simple label matching to using the field type and uid.

  • charliesigma
    charliesigma
    Community Member
    edited March 2023

    ahh I see. That makes sense! Something I figured is that the UID field might not be very useful because dumping the categories using --dumpcats show 'website' and URLs have different UIDs between different stored entries:

          1   __MAIN::URL::13::4
          1   __MAIN::URL::13::5
          1   __MAIN::URL::14::5
          1   __MAIN::URL::6949::5
          1   __MAIN::URL::6957::6
          1   __MAIN::URL::6978::6
          1   __MAIN::URL::7856::4
          1   __MAIN::URL::8022::5
          1   __MAIN::URL::8618::6
          1   __MAIN::URL::8883::5
          1   __MAIN::URL::9553::8
          1   __MAIN::URL::9736::5
          1   __MAIN::URL::9957::5
          1   __MAIN::Username::10::1
          1   __MAIN::Username::10::2
          1   __MAIN::Username::10::3
          1   __MAIN::Website::13::4
          1   __MAIN::Website::13::5
          1   __MAIN::Website::13::7
          1   __MAIN::Website::14::5
          1   __MAIN::Website::15::5
          1   __MAIN::Website::16::5
          1   __MAIN::Website::5567::5
          1   __MAIN::Website::5684::5
          1   __MAIN::Website::5984::5
          1   __MAIN::Website::6169::5
          1   __MAIN::Website::6353::5
          1   __MAIN::Website::6455::5
          1   __MAIN::Website::7472::6
          1   __MAIN::Website::7529::5
          1   __MAIN::Website::7662::4
          1   __MAIN::Website::7959::4
          1   __MAIN::Website::8075::4
          1   __MAIN::Website::8150::4
          1   __MAIN::Website::8336::5
          1   __MAIN::Website::8487::5
          1   __MAIN::Website::8522::6
          1   __MAIN::Website::8689::4
          1   __MAIN::Website::8708::4
          1   __MAIN::Website::8895::5
          1   __MAIN::Website::9143::6
          1   __MAIN::Website::9362::5
          1   __MAIN::Website::9601::5
          1   __MAIN::Website::9708::5
          1   __MAIN::Website::9856::6
          1   ADDITIONAL DETAILS::URL::5010::13
          1   ADDITIONAL DETAILS::URL::5018::11
          1   ADDITIONAL DETAILS::URL::5093::10
          1   ADDITIONAL DETAILS::URL::5093::9
          1   ADDITIONAL DETAILS::URL::5119::10
          1   ADDITIONAL DETAILS::URL::5141::15
          1   ADDITIONAL DETAILS::URL::5150::15
          1   ADDITIONAL DETAILS::URL::5169::14
          1   ADDITIONAL DETAILS::URL::5181::12
          1   ADDITIONAL DETAILS::URL::5191::11
          1   ADDITIONAL DETAILS::URL::5202::12
          1   ADDITIONAL DETAILS::URL::5208::13
          1   ADDITIONAL DETAILS::URL::5244::11
          1   ADDITIONAL DETAILS::URL::5290::11
          1   ADDITIONAL DETAILS::URL::5349::10
          1   ADDITIONAL DETAILS::URL::5487::13
          1   ADDITIONAL DETAILS::URL::5492::10
          1   ADDITIONAL DETAILS::URL::5539::10
          1   ADDITIONAL DETAILS::URL::5541::10
          1   ADDITIONAL DETAILS::URL::5546::8
          1   ADDITIONAL DETAILS::URL::5549::10
          1   ADDITIONAL DETAILS::URL::5592::16
          1   ADDITIONAL DETAILS::URL::5635::12
          1   ADDITIONAL DETAILS::URL::5657::10
          1   ADDITIONAL DETAILS::URL::5724::10
          1   ADDITIONAL DETAILS::URL::5732::13
          1   ADDITIONAL DETAILS::URL::5751::11
          1   ADDITIONAL DETAILS::URL::5765::10
          1   ADDITIONAL DETAILS::URL::5773::11
          1   ADDITIONAL DETAILS::URL::5887::11
          1   ADDITIONAL DETAILS::URL::5897::10
          1   ADDITIONAL DETAILS::URL::5955::10
          1   ADDITIONAL DETAILS::URL::5988::9
          1   ADDITIONAL DETAILS::URL::5993::17
          1   ADDITIONAL DETAILS::URL::6003::11
          1   ADDITIONAL DETAILS::URL::6012::16
          1   ADDITIONAL DETAILS::URL::6047::10
          1   ADDITIONAL DETAILS::URL::6057::12
          1   ADDITIONAL DETAILS::URL::6123::10
          1   ADDITIONAL DETAILS::URL::6123::11
          1   ADDITIONAL DETAILS::URL::6143::11
          1   ADDITIONAL DETAILS::URL::6163::10
          1   ADDITIONAL DETAILS::URL::6184::12
          1   ADDITIONAL DETAILS::URL::6270::11
          1   ADDITIONAL DETAILS::URL::6272::11
          1   ADDITIONAL DETAILS::URL::6281::11
          1   ADDITIONAL DETAILS::URL::6296::10
          1   ADDITIONAL DETAILS::URL::6367::8
          1   ADDITIONAL DETAILS::URL::6372::10
          1   ADDITIONAL DETAILS::URL::6379::10
          1   ADDITIONAL DETAILS::URL::6412::10
          1   ADDITIONAL DETAILS::URL::6418::12
          1   ADDITIONAL DETAILS::URL::6447::9
          1   ADDITIONAL DETAILS::URL::6483::10
          1   ADDITIONAL DETAILS::URL::6512::10
          1   ADDITIONAL DETAILS::URL::6516::10
          1   ADDITIONAL DETAILS::URL::6520::10
          1   ADDITIONAL DETAILS::URL::6553::10
          1   ADDITIONAL DETAILS::URL::6693::12
          1   ADDITIONAL DETAILS::URL::6704::10
          1   ADDITIONAL DETAILS::URL::6720::15
          1   ADDITIONAL DETAILS::URL::6759::12
          1   ADDITIONAL DETAILS::URL::6782::10
          1   ADDITIONAL DETAILS::Website::13::5
          1   ADDITIONAL DETAILS::Website::14::6
          1   ADDITIONAL DETAILS::Website::5035::10
          1   ADDITIONAL DETAILS::Website::5038::14
          1   ADDITIONAL DETAILS::Website::5191::10
          1   ADDITIONAL DETAILS::Website::5223::11
          1   ADDITIONAL DETAILS::Website::5236::10
          1   ADDITIONAL DETAILS::Website::5304::12
          1   ADDITIONAL DETAILS::Website::5335::10
          1   ADDITIONAL DETAILS::Website::5450::15
          1   ADDITIONAL DETAILS::Website::5451::11
          1   ADDITIONAL DETAILS::Website::5522::10
          1   ADDITIONAL DETAILS::Website::5530::11
          1   ADDITIONAL DETAILS::Website::5577::10
          1   ADDITIONAL DETAILS::Website::5669::7
          1   ADDITIONAL DETAILS::Website::5727::15
          1   ADDITIONAL DETAILS::Website::5782::10
          1   ADDITIONAL DETAILS::Website::5793::10
          1   ADDITIONAL DETAILS::Website::5803::12
          1   ADDITIONAL DETAILS::Website::5836::10
          1   ADDITIONAL DETAILS::Website::5846::10
          1   ADDITIONAL DETAILS::Website::5850::10
          1   ADDITIONAL DETAILS::Website::5939::13
          1   ADDITIONAL DETAILS::Website::5952::10
          1   ADDITIONAL DETAILS::Website::5952::11
          1   ADDITIONAL DETAILS::Website::5976::17
          1   ADDITIONAL DETAILS::Website::5983::6
          1   ADDITIONAL DETAILS::Website::5988::12
          1   ADDITIONAL DETAILS::Website::6097::11
          1   ADDITIONAL DETAILS::Website::6106::19
          1   ADDITIONAL DETAILS::Website::6170::10
          1   ADDITIONAL DETAILS::Website::6198::14
          1   ADDITIONAL DETAILS::Website::6269::10
          1   ADDITIONAL DETAILS::Website::6280::12
          1   ADDITIONAL DETAILS::Website::6290::10
          1   ADDITIONAL DETAILS::Website::6306::20
          1   ADDITIONAL DETAILS::Website::6312::10
          1   ADDITIONAL DETAILS::Website::6315::10
          1   ADDITIONAL DETAILS::Website::6315::8
          1   ADDITIONAL DETAILS::Website::6326::10
          1   ADDITIONAL DETAILS::Website::6356::10
          1   ADDITIONAL DETAILS::Website::6414::10
          1   ADDITIONAL DETAILS::Website::6414::13
          1   ADDITIONAL DETAILS::Website::6441::11
          1   ADDITIONAL DETAILS::Website::6493::21
          1   ADDITIONAL DETAILS::Website::6518::10
          1   ADDITIONAL DETAILS::Website::6543::10
          1   ADDITIONAL DETAILS::Website::6560::10
          1   ADDITIONAL DETAILS::Website::6572::11
          1   ADDITIONAL DETAILS::Website::6616::10
          1   ADDITIONAL DETAILS::Website::6690::10
          1   ADDITIONAL DETAILS::Website::6726::10
          1   ADDITIONAL DETAILS::Website::6768::16
          1   ADDITIONAL DETAILS::Website::6771::10
          1   ADDITIONAL DETAILS::Website::6796::10
          1   ADDITIONAL DETAILS::Website::6811::8
          1   ADDITIONAL DETAILS::Website::6830::10
          1   ADDITIONAL DETAILS::Website::6843::10
          1   ADDITIONAL DETAILS::Website::6878::11
          1   ADDITIONAL DETAILS::Website::6888::18
          1   ADDITIONAL DETAILS::Website::6894::10
          1   ADDITIONAL DETAILS::Website::7048::10
          1   ADDITIONAL DETAILS::Website::7049::12
          1   ADDITIONAL DETAILS::Website::7066::11
          1   ADDITIONAL DETAILS::Website::7125::11
          1   ADDITIONAL DETAILS::Website::7159::10
          1   ADDITIONAL DETAILS::Website::7170::10
          1   ADDITIONAL DETAILS::Website::7211::10
          1   ADDITIONAL DETAILS::Website::7240::10
          1   ADDITIONAL DETAILS::Website::7270::15
          1   ADDITIONAL DETAILS::Website::7354::11
          1   ADDITIONAL DETAILS::Website::7355::11
          1   ADDITIONAL DETAILS::Website::7355::12
    

    The type is always consistent as URL though.

  • MrC
    MrC
    Volunteer Moderator
    edited March 2023

    @charliesigma

    Thanks for the dump; it's always good to see the variety of data that exists.

    Generally the converter suite uses labels, as this is the way it grew over time. Most password manager exports are simple label / value pair based exports (e.g. CSV files). Exports where data is typed have traditionally been rare, yet have become more common. Enpass is no exception here - its early export formats were awful. Since this particular converter still has to support older export formats, I can't just change it to assume typed data. At the stage of conversion where this needs to happen, the key / value pairs are simply being collected, to be generalized and record-split later. This is why you see the workaround to append UID values, and why some patterns in the field matching templates support those appended UIDs. But this clearly is grossly inadequate.

  • charliesigma
    charliesigma
    Community Member
    edited March 2023

    @MrC no problem at all, anything I can do to help. If there were other URLs that used the same login information, for example, I always added to my entries to allow me to log in faster. Even Enpass started asking if I wanted to add the new website URL to that specific login, and when I said yes, it probably added to the "Additional Details" area (very last entry). My 1password entry from today is one example of it (removed all identifiable information):

    {
                "archived": 0,
                "auto_submit": 1,
                "category": "login",
                "createdAt": 1679150965,
                "favorite": 0,
                "fields": [
                    {
                        "deleted": 0,
                        "label": "Username",
                        "order": 1,
                        "sensitive": 0,
                        "type": "username",
                        "uid": 10,
                        "updated_at": 1679150965,
                        "value": "",
                        "value_updated_at": 1679150965
                    },
                    {
                        "deleted": 0,
                        "label": "E-mail",
                        "order": 2,
                        "sensitive": 0,
                        "type": "email",
                        "uid": 12,
                        "updated_at": 1679151127,
                        "value": "",
                        "value_updated_at": 1679151127
                    },
                    {
                        "deleted": 0,
                        "label": "Password",
                        "order": 3,
                        "sensitive": 1,
                        "type": "password",
                        "uid": 11,
                        "updated_at": 1679150965,
                        "value": "",
                        "value_updated_at": 1679150965
                    },
                    {
                        "deleted": 0,
                        "history": [
                            {
                                "updated_at": 1679153244,
                                "value": ""
                            }
                        ],
                        "label": "Secret Key",
                        "order": 4,
                        "sensitive": 1,
                        "type": "password",
                        "uid": 102,
                        "updated_at": 1679153244,
                        "value": "",
                        "value_updated_at": 1679153244
                    },
                    {
                        "deleted": 0,
                        "label": "Website",
                        "order": 5,
                        "sensitive": 0,
                        "type": "url",
                        "uid": 13,
                        "updated_at": 1679152731,
                        "value": "https://my.1password.com",
                        "value_updated_at": 1679150965
                    },
                    {
                        "deleted": 0,
                        "label": "ADDITIONAL DETAILS",
                        "order": 6,
                        "sensitive": 0,
                        "type": "section",
                        "uid": 101,
                        "updated_at": 1679152731,
                        "value": "",
                        "value_updated_at": 1679150965
                    },
                    {
                        "deleted": 0,
                        "label": "Phone number",
                        "order": 7,
                        "sensitive": 0,
                        "type": "phone",
                        "uid": 14,
                        "updated_at": 1679152731,
                        "value": "",
                        "value_updated_at": 1679150965
                    },
                    {
                        "deleted": 0,
                        "label": "Security question",
                        "order": 8,
                        "sensitive": 0,
                        "type": "text",
                        "uid": 15,
                        "updated_at": 1679150965,
                        "value": "",
                        "value_updated_at": 1679150965
                    },
                    {
                        "deleted": 0,
                        "label": "Security answer",
                        "order": 9,
                        "sensitive": 1,
                        "type": "text",
                        "uid": 16,
                        "updated_at": 1679150965,
                        "value": "",
                        "value_updated_at": 1679150965
                    },
                    {
                        "deleted": 0,
                        "label": "URL",
                        "order": 10,
                        "sensitive": 0,
                        "type": "url",
                        "uid": 7323,
                        "updated_at": 1679152076,
                        "value": "my.1password.ca",
                        "value_updated_at": 1679152076
                    }
                ],
                "icon": {
                    "fav": "my.1password.com",
                    "image": {
                        "file": "misc/login"
                    },
                    "type": 1,
                    "uuid": ""
                },
                "note": "",
                "subtitle": "",
                "template_type": "login.default",
                "title": "1password",
                "trashed": 0,
                "updated_at": 1679154120,
                "uuid": ""
            },
    
  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    I want to fix this issue in a more robust way, but don't want to keep you waiting too long. Have you customized the URL labels to be other than the two variants I've mentioned, URL or Website?

  • charliesigma
    charliesigma
    Community Member

    Looking at the dumpcats results it doesn't seem like it, as I tried to keep it as organized as possible. The one I posted is a summarized version as there were too many characters to post.

  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    Ok, no matter. I decided to update the converter suite to handle labels for the additional URLs. The enpass converter is the first to require this (others have only needed support for a list of URLs, without labels).

    I've coded this to only support these extra URLs if they are in the main section of the record. If they are in the Additional Details area, they are treated as unmapped fields, so go into the notes area. Is that how you'd prefer it, or would you like them mapped as additional URLs in the 1Password login record?

  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    Just to save time, I've posted a development version of the suite. It follows what I mentioned above. Feel free to try it out. If you prefer the alternate behavior, I'll explain the small change to make.

  • charliesigma
    charliesigma
    Community Member

    I've coded this to only support these extra URLs if they are in the main section of the record. If they are in the Additional Details area, they are treated as unmapped fields, so go into the notes area. Is that how you'd prefer it, or would you like them mapped as additional URLs in the 1Password login record?

    I'd definitely prefer them as additional URLs instead of notes. You are an amazing person, thank you so much!

  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    I've updated the development version for you.

  • charliesigma
    charliesigma
    Community Member

    Hey it worked perfectly! Thanks so much!

    Now, a quick last question: is it possible to map the e-mail field from enpass to the e-mail field from 1password? Most of my logins are e-mail based and the're all being pushed to the notes.

  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    Great, thanks for the feedback.

    Yes, the E-mail field can become the primary username field. How do you want Username mapped? As a custom field?

  • charliesigma
    charliesigma
    Community Member
    edited March 2023

    This might be my lack of knowledge of 1pasword for now, but can it use the 'default' email field?

    Like, Keep username as username and map e-mail to the internal Email field:

    So, after importing it would look like this:

  • MrC
    MrC
    Volunteer Moderator

    @charliesigma

    Three key fields are used by 1Password for Login autofill. These are keyed in the converter with certain reserved internal field labels: username, password, and url. Other fields (and their labels) are essentially information; there are there for you, but do not take part in autofill. Given that Enpass has both a Username and an E-Mail field for Login records, we have to select which of these fields to map to the field that 1Password will use for autofill.

  • charliesigma
    charliesigma
    Community Member

    Ohhh I see. Then I think it's better to import the e-mail field as a custom one and keep username as is. This way, for entries that I have both, they'll at least make sense to me when looking at them.

  • MrC
    MrC
    Volunteer Moderator
    edited March 2023

    @charliesigma

    I've updated the dev archive. It adds this custom field mapping:

    and another internal addition, to produce:

  • charliesigma
    charliesigma
    Community Member
    edited March 2023

    Alright this is beyond perfect!
    You are the ABSOLUTE Greatest Of All Time!

    Thank you very very much!

  • MrC
    MrC
    Volunteer Moderator

    Great!

    And LOL - I've always wanted to be a GOAT.

    I was editing my reply above to add the following, but saw your reply come in, so figured I'd post it here instead:

    map e-mail to the internal Email field

    FYI: that's just a 1Password data "type" - it's not an internal Email field. You can add as many additional fields as you like to records, and select the type when you create the field. Currently the type cannot be changed in 1Password 8.

  • charliesigma
    charliesigma
    Community Member

    Hey Thanks for that additional context. Having emails in their own custom fields at least allows 1password to autocomplete the emails and passwords on the multitude of imported credentials, so it's way more than enough to get me started.

    I'll go over them and move the emails to the username field as I move forward, but you've given me so much help and I am absolutely grateful.

    Thanks again!

This discussion has been closed.