DataVault items converted as Secure Notes

mstasim

I converted from DataVault to 1Password following the converter instructions. 17 of my items converted as Logins and ~450 converted as secure notes. I looked into the CSV and it is unclear the difference in the file.

---

1Password Version: 6.2.333d
Extension Version: Not Provided
OS Version: W 10
Sync Type: Not Provided
Referrer: forum-search:DataVault Imports are Secure Notes

MrC

Hi @mstasim ,

Have you reviewed all the caveats about the DataVault export in the README.pdf ?

The DataVault export is very weak.

Can you identify which record types are being mis-categorized?

Greg

Hello @mstasim,

Thank you for sharing your experience! I am sorry you bumped into troubles while migrating to 1Password. :(

The import process from various applications may be tricky, but the converter tool was developed by MrC, so his questions are always on point. Please share this additional info, so we could get the ball rolling. :+1: Thank you in advance!

Cheers,
Greg

mstasim

I read all the PDF data. The main one is logins. Seems like the converter gets username password, but everything beyond that sets dumped into notes. No URL or Secret Word conversion. On many of the logins, it pulled even my username and password into the secure notes...

MrC

@mstasim ,

As mentioned in the README, the converter can only figure out what to do with categories that are unambiguous. The Datavault CSV export does not contain sufficient data to tell the converter what a row of CSV values contains, nor the semantics (meaning) of various fields. Look at this one for a Login:

"my login","Address","www.example.com","Username","joe@example.com","Password","secretstuff","a","b","c","d","e","f","g","h","i","j","k","l","m","n","notes go here"

Notice there are only labels for "Address", "Username", and Password". Everything else after that is free-form meaningless data, except for the last column which is Notes.

The converter can export to the following types:

bankacct creditcard driverslicense email login membership note rewards software

and the following Datavault types are completely indistinguishable in the CSV export, so get mapped to the category on the right:

bank account, checking account     --> bankacct
business contact, personal contact --> contact
credit card, mastercard, visa      --> creditcard
business, financial                --> business

If you have customized any of the know field names, the converter will not be able to detect their meanings.

Can you provide me a sanitized single row of a Login from the CSV that doesn't convert as a login?

mstasim

Here's what came over in the Secure Note (sanitized) from the CSV. In the CSV it was Username in col1 and g@3$-!#%@gmail.com in col2, Password in col3, etc.

Username: g@3$-!#%@gmail.com
Password: g)^&@#%s!
URL: https:\www.(^&%%#()^&%.com
Secret Word: !@#$(, #@(!*

Is there a way to manually format the CSV so it maps better (maybe with another converter?). I saw a note about modifying the converter file, but I'm not conversant in perl...

Thanks,

Stan

MrC

Hi @mstasim,

You're probably not going to like the amount of work it is going to require, but you can modify the CSV file in a spreadsheet. That would possibly be easier for you then doing it within 1Password. It will be your call, but I don't think you want to do this.

Again, just to reiterate, the DataVault export, while nice to be able to get your data out, that data is essentially meaningless, and requires you the user to figure out what it all means.

So I'll describe your options. First, let's look at the DataVault export format:

"title goes here","label1","value1","label2,"value2","label3","value3","label4","value4","label5","value5","label6","value6","label7","value7","label8","valut8","label9","value9","label10","value10","notes go here"

There are 22 columns in the export. Column 1 is the entry's Title, and column 22 contains the Notes. Columns 2 through 21 contain the ten Label / Value pairs supported by DataVault. The Labels are completely free-form - they are control by you when you create an item, by its Form Save feature, and by the Templates you create when you use DataVault. So there is likely very little column consistency row to row.

Now, your options to convert and import:

1. You can modify your labels in the CSV to match the datavault converter's label requirements (and/or with your help you and I can modify the converter a small amount to capture your labels based on the list you provide me) - this is probably the easiest choice.
2. You can modify the export CSV and make the CSV consistent. This means the columns in the CSV must be the same for all the rows. This means you will have to review your rows, move the data in each row, such that, for example, "value4" is always the URL, "value2" is always the Username, etc. And you would have to split out each of the Category types into separate CSV files. This route may be a lot of work.

So let's focus on (1). The converter works by using pattern matching to match the labels in the DataVault CSV export. The label "patterns" in the converter are simple, and are based on the stock fields that DataVault uses. But the patterns can be updated, and this is probably pretty trivial for most cases. What I need is the list of labels from you for a given category. Here's how it works.

In the converter, there is a table of categories, the fields it supports to map to 1Password fields, and the patterns used to detect those fields. Here's an example Login category:

    login =>                    { textname => '', fields => [
        [ 'url',                0, qr/^Address$/, ],
        [ 'username',           0, qr/^Username$/, ],
        [ 'password',           0, qr/^Password$/, ],
        [ 'email',              0, qr/^email$/, ],

The word "login" you see is the DataVault "login" category (which will get mapped to 1Password's Login category), the words url, username, password and email are all 1Password internal keys for Login category items, and the weird looking things like qr/^Address$/, qr/^Username$/, ... and **qr/^email$/ **are the patterns. All we need to do is update the patterns to match your list of Labels in the CSV. So if your labels are Website and URL, we'll modify the labels we need to match to capture those (Logins are a little more tricky, and I won't explain why here).

So what I would need from you, is a list of your labels, per field, for you Logins, and other categories that are are your CSV export that are supported by the converter. And you can see the entire list by using a text editor (not Notepad), and looking at the the table named my %card_field_specs near the top of the converter.

This may seem overwhelming, but it we take it one category at a time, customize the converter, convert and import, you can see what is working, and what isn't, and repeat. Starting with Logins, and your most frequent categories, you'll get to Good Enough very quickly.

If you're still with me, and this makes sense, provide a list of your Login category labels (for the url, username, password, and email) and we'll go from there. Feel free to reply here, but after that, it may make most sense to take this offline so I can send you quick updates to the datavault converter for you to try. I'm at: mike ( at ) cappella ( dot ) us

mstasim

I'm with you. Here are my categories

Username
Password
URL
eMail

The other one I use frequently is Secret Word and there doesn't seem to be a 1Password default for that...

I'm also not opposed to doing some CSV work (aligning the columns - for example) if that will get me there faster...

MrC

@mstasim,

Great, then we'll go with a bunch of rapid datavault converter updates I'll make for you. You may even get the hang of it, and will start doing it yourself!

I'll place an update of the Converters\Datavault.pm file for you in Testing Bits, mentioned in the converter suite thread. I'll make changes, and update it. If you prefer email, start the conversation via email, and I'll just send it to you that way (email is faster, but I respect your possible desire to remain anonymous, so offer the Dropbox route).

What would you like done with the "Secret Word? :
1. is it a Question/Answer like thing?
2. do you want it to be a hidden field (like 1Password's password fields)?
3. is this label "Secret Word" absolutely unique to your Login categories (if so, I can use it to identify your Login records)?

1Password won't do anything with "Secret Word", but at least it can be placed into the data (the converter can also add your custom fields just by using the --addfields option on the command line.

Edit: I've already made the updates to accommodate your labels above. Here's an example import using your labels:

Or with Secret Word being a concealed field:

MrC

@mstasim ,

Instead of awaiting your reply for now, I've dropped the customized Datavault.pm converter into Testing Bits as a single file. Replace your Converters\Datavault.pm file with it, and you can re-run the conversion. We'll go from there after I hear from you next.

mstasim

Thanks! The Secret Word does not need to be Hidden. I'll have to check and see if it is unique to my logins.

I'm fine with splitting the logins out and loading them separately. What are you using to edit the .pm files?

MrC

@mstasim ,

I use a text editor that came from Unix environments called vim. Its available for Widows too. I don't think you'll like it, because there's a very large learning curve. I've been using it for a very long time (and its parent, and grandparent).

It's trivial to add the hidden attribute to any field - the post of the converter I left for you sets the concealed attribute.

mstasim

Spent sometime working on it tonight with no luck. The new converter didn't do any better. I keeps pulling all my logins in as secure notes with everything in the note. There were 2 (out of 550) that pulled in as logins. The only thing I could figure looking at those was they had limited column names.

One item I did notice was that my csv starts in Row1 with an account. There is nothing in my export that indicates what type of record it is...

I think aligning the csv looks like the best answer. Also, what are you modifying the datavault.pn file in? I could try my hand at creating my own custom converter.

mstasim

Just saw your 10:21 message. Hum - maybe directly editing the .pm won't work... What would I need to do to modify the CSV files to one for each category and clean them up before import?

MrC

@mstasim,

That doesn't make sense given the column names you gave me. Look at the example below, with the column names of URL, Username, Password, and Secret Word as you indicated:

$ cat ~/Desktop/dv.csv
"Target : Expect More. Pay Less.","URL","https://www-secure.target.com/","Username","joe@example.com","Password","supersecret","Secret Word","secret word value","label5","value5","label6","value6","label7","value7","label8","valut8","label9","value9","label10","value10","these are my notes"

and here's the conversion:

$ perl convert_to_1p4.pl datavault -v  ~/Desktop/dv.csv
Imported 1 item
Exported 1 login item
Exported 1 total item
You may now import the file /Users/cappella/Desktop/1P_import.1pif into 1Password

Notice 1 Login is exported. Are you sure you replaced the Datavault.pm file? Are you sure those are the exact column names (including case)?

mstasim

I just opened my csv in notepad (v. excel) to look at the structure of each line. I don't have any of the "" in your example. Is that and issue?

Here's a row from my csv.

Crayola.com,Username,swww.wwwww@gmail.com,Password,zxcasdfre,URL,http://www.crayola.com/,Secret Word,,Account Number,,eMail,,PIN,,,,,,,,

mstasim

I opened the datavault.pm file with wordpad and it is the one you modified for my situation. It has mstasim in the text.

MrC

@mstasim ,

No, the double quotes around null values won't matter, for the most part. As I mentioned in the README, the Datavault export does not properly quote its CSV, and this can cause trouble. But don't worry about that now.

The converter is detecting this entry as a Bank Account record, because of the "Account Number" field. This field is a key detection field for the bankacct category. The Username, Password, URL fields are not sufficient to detect the entry as a Login, because other DataVault records also export these field names! If you don't have any non-Login records with these fields (e.g. 'email', 'frequently flyer'), then we can use Username and Password as Login detectors.

And because there is no value in the Account Number field, you're left with no real record category of importance, so it ends up looking like nothing more than a Note (the Username, Password, and URL fields are not in the Bank Account category, so they get pushed to notes).

Does this make sense?

MrC

To add, if you can open the Data Vault export in a spreadsheet, and change all of any one of the "Username" or "Password" or "URL" labels for all of your Login records to something unique, that I can quickly change a single value in the converter reliably detect these as Logins. Alternatively, if you can ensure no other record type in your export contains these field names, the same can be accomplished the same way.

For example, if you are going to change one of the labels, change URL to LoginURL in all of the Login records. Then its trivial to detect that label.

mstasim

Ok I get what's going on. I'll change URL to LoginURL for all the logins. That's no problem.

I can do that tonight.

I could also insert a column and populate the record with the type... that's pretty easy on my end, but might not be on your end...

Also, how does it determine what to put in the Notes field in 1Password?

Thanks for the work on this...

MrC

@mstasim ,

The URL to LoginURL will work fine, as would a record type column (but that will also add the type field to the record, and you don't want to add a column, you could use an existing one of the twenty-two columns). Your call.

The converter works like this:

1. Collect all the Field Label / Value pairs in the record.
2. Match the Field Labels against the table of Category to Field mappings to find the target Category - the first Field Label match that is allowed to set the Category type wins.
3. If there is no match, or there was a match but the matched Field is not allowed to set the Category, then fallback to testing if any of the Login category Field Labels are present whereby it would be considered a Login, otherwise it is a Secure Note.
4. There is a special case later that when the record is "normalized" to an internal form, if there are no Fields with values, the record is demoted to Secure Note.

The reason only some Fields are allowed to set the Category is because some Field Labels are present in multiple export Categories (e.g. Username, Email, etc.).

This treatment allows the record to be placed into the most specific Category when possible, or falls back to the most relevant Category based on the Field Labels present.

mstasim

I'd go with LoginURL for the key v reading one of the other columns for the type...

Thanks

MrC

@mstasim,

Ok, If I'm understanding you correctly, you want me to LoginURL (instead of URL) as a type-setting login category label. I'll make the change now and post it to Testing Bits.

Edit: update posted.

perl convert_to_1p4.pl datavault -v  ~/Desktop/dvv.csv -d
main                : Command Line: datavault -v /Users/cappella/Desktop/dvv.csv -d
main                : Output file: /Users/cappella/Desktop/1P_import.1pif
print_fileinfo      : Export file info
print_fileinfo      :   size: 142
print_fileinfo      :   kind: ASCII text
print_fileinfo      :   mime: text/plain; charset=us-ascii
do_import           : ROW: 1
do_import           :   field: Username => swww.wwwww@gmail.com
do_import           :   field: Password => supersecret
do_import           :   field: LoginURL => http://www.crayola.com/
do_import           :   field: Secret Word =>
do_import           :   field: Account Number =>
do_import           :   field: eMail =>
do_import           :   field: PIN =>
find_card_type      : type detected as 'login' (key='LoginURL')
print_record        : title: Crayola.com
                      tags:
                      key(url): LoginURL = http://www.crayola.com/
                      key(password): Password = supersecret
                      key(username): Username = swww.wwwww@gmail.com
                      notes:
Imported 1 item
create_pif_record   : Title: Crayola.com
create_pif_record   :   key test(username): swww.wwwww@gmail.com
create_pif_record   :   key test(password): supersecret
create_pif_record   :   key test(url): http://www.crayola.com/
Exported 1 login item
Exported 1 total item
You may now import the file /Users/cappella/Desktop/1P_import.1pif into 1Password

mstasim

That worked like a charm.

Thanks for all your help.

MrC

@mtasim,

That's great news. Enjoy 1Password!

Greg

mstasim, I am glad to hear that MrC has managed to help you! He is awesome. :)

Please reach out to us anytime if you have more questions about 1Password. Thank you!

++
Greg