lp:~jplacerda/zeitgeist/encryption

A mock-up of what the patch might look like.
There are some security issues with the keyring module (gobject.set_application_name("whatever")).
The algorithm used for generating a key needs to be double checked.

16:53 < jplacerda> Both key and rekey take char* pkey and an int nkey
16:53 < jplacerda> What's the latter used for?
16:54 < sjlombardo> in general nkey will be the length of pkey
16:55 < jplacerda> sjlombardo: that's what I've been using as, it's good to know I wasn't completely off :p
16:56 < jplacerda> Right, that's question #1 answered.
16:56 < sjlombardo> In some limited cases, if you had a longer key, and wanted to only use part of the string, you could set nkey to be lesser
16:56 < jplacerda> Okay.
16:56 < jplacerda> Cool.
16:57 < jplacerda> Any other uses for nkey?
16:57 < sjlombardo> nope, thats it really. if you use the pragma to set the key, i.e. pragma key='mypassword'; then it just grabs strlen internally and passes that in
16:58 < seiflotfy> sjlombardo, i love how you sell the binaries for windows
16:58 < seiflotfy> fucking awesome
16:58 < sjlombardo> so, for compatibility sake, its best to have it be the string length
16:58 < jplacerda> sjlombardo: great, it seemed a bit redundant at first
16:59 < jplacerda> sjlombardo: alright, my next question is about the use of ATTACH
16:59 < jplacerda> So for attach you've got something like this: ATTACH DATABASE 'encrypted.db' AS encrypted KEY 'secret'
16:59 < seiflotfy> can we control sqlite over sqlcipher?
16:59 < jplacerda> seiflotfy: yes
16:59 < seiflotfy> awesome
17:00 < sjlombardo> seiflotfy: yeah, compilation on windows can be a bit tricky, and takes up alot of time on the mailing list. so we decided to go for it
17:00 < jplacerda> seiflotfy: to prevent breakage in our internal API, we can just do something like import sqlcipher as sqlite3
17:00 < seiflotfy> so u can write your wrapper for communication with sqlite
17:00 < seiflotfy> jplacerda, ^
17:00 < seiflotfy> yeah
17:01 < jplacerda> seiflotfy: my module contains all of the sqlite3 stuff, plus a few extra things contained in sqlcipher (check out my branch if you're interested :-))
17:01 < sjlombardo> right, when attaching a database you can pass in the key
17:01 < jplacerda> sjlombardo: yup
17:01 < sjlombardo> also, you might be interested to checkout the v2beta branch - it's really close to being moved over to master
17:01 < seiflotfy> jplacerda, can i control all of sqlite suing it
17:01 < seiflotfy> using it?
17:01 < jplacerda> We'd execute that statement on an open, unencrypted DB, right?

17:01 < jplacerda> seiflotfy: yes
17:01 < jplacerda> sjlombardo: will do
17:02 < sjlombardo> some improvements to the behavior of attach and the various pragmas on attached databases
17:02 < jplacerda> Ah, okay.
17:02 < jplacerda> sjlombardo: as you can imagine, ATTACH is something that most users will run when this goes upstream
17:02 < sjlombardo> so, do you mean you want to run the attach to hook up an encrypted database to a currently unencrypted database?
17:03 < jplacerda> sjlombardo: nope, I want to use attach for its standard purpose, unencrypted -> encrypted
17:03 < jplacerda> What is 'encrypted.db'? A file path?
17:04 < sjlombardo> right thats the file path of the database you want to attach
17:04 < jplacerda> ok
17:04 < jplacerda> Does it need to exist prior to executing the statement?
17:05 < sjlombardo> no, it doesnt. if it doesnt exist it will be created
17:05 < jplacerda> (BTW seiflotfy, what happened to this: https://devcando.wordpress.com/2011/04/15/please-say-welcome-to-the-first-draft-of-zeitgeist-global-privacy/ ?)
17:05 < jplacerda> sjlombardo: great
17:05 < seiflotfy> jplacerda, its now ALM
17:05 < seiflotfy> :P
17:06 < sjlombardo> so are you thinking of using it for migration? I.e. to take existing data in a plain sqlite database and then attach an encrypted database, copy the data into it?
17:06 < jplacerda> sjlombardo: indeed
17:06 < jplacerda> That'll be one of the main things we need to get sorted
17:07 < jplacerda> seiflotfy: what happened to the UI though?
17:07 < sjlombardo> gotcha - fwiw, v2beta also has an embedded function called sqlcipher_export() which is intended to assist with this sort of thing
17:08 < seiflotfy> it evolved dude jplacerda
17:08 < jplacerda> sjlombardo: great
17:08 < sjlombardo> it's one of th emost requested features. Basically, you attach a database, then call sqlcipher_export('attached db name');
17:08 < sjlombardo> it rips through the schema of the main database, reproduces it verbatim in the attached database and copies all data between the two.
17:08 < jplacerda> sjlombardo: great, I was thinking of writing a wrapper to do just that :-) great to know y'all are taking this in a practical direction
17:09 < jplacerda> sjlombardo: what's the ETA on v2beta?
17:09 < sjlombardo> the goal is to make it easier to migrate back and forth between encrypted / plain databases or databases with different crypto settigns
17:09 < jplacerda> yup
17:10 < sjlombardo> v2beta is stable, and close to being merged back. the whole test suite is passing. The only thing that we've been holding back on is that there is a change in the default file format between sqlcipher 1
                    and 2
17:10 < jplacerda> sjlombardo: okay
17:11 < sjlombardo> sqlcipher 2 includes a per-page HMAC - in short it provides tamper resistance to the database
17:11 < jplacerda> Ah, great
17:11 < seiflotfy> jplacerda, i will look into some stuff with tdfischer now
17:11 < seiflotfy> we need to finish the demos
17:11 < sjlombardo> if an attacker attempted to modify the data in one of the pages to get it through the library, it would just fail
17:11 < seiflotfy> ping me if there is anything
17:12 < jplacerda> seiflotfy: will do
17:12 < jplacerda> sjlombardo: okay, I'll move my focus to v2beta then
17:12 < sjlombardo> the other thing I'm waiting on is getting some more feedback on sqlcipher_export() which is a new function, so I'd like to get some feedback
17:12 < jplacerda> sjlombardo: we'll be using extensively, so expect some feedback from me :)
17:12 < sjlombardo> that would be great
17:13 < jplacerda> I ran into a problem recently, whereby trying to rekey before keying caused sqlite_rekey to dump core
17:13 < sjlombardo> I'd estimate we'll probably merge v2beta into master in the next few weeks
17:13 < jplacerda> sjlombardo: cool
17:13 < jplacerda> That'll give me time to finish the bindings, integrate it into our framework, and then focus on packaging
17:13 < sjlombardo> ah, I'll have to take a look at that probably one of the contexts isn't intialized at that point...
17:13 < jplacerda> Hmm okay
17:14 < jplacerda> Have you run into this problem before?
17:14 < sjlombardo> I haven't personally, but generally you wouldn't run rekey before key
17:14 < sjlombardo> the former assumes that the database is already encrypted…
17:15 < jplacerda> sjlombardo: agreed
17:15 < jplacerda> But it would be a bit more robust to not have it dump core
17:13 < sjlombardo> I'd estimate we'll probably merge v2beta into master in the next few weeks
17:13 < jplacerda> sjlombardo: cool
17:13 < jplacerda> That'll give me time to finish the bindings, integrate it into our framework, and then focus on packaging
17:13 < sjlombardo> ah, I'll have to take a look at that probably one of the contexts isn't intialized at that point...
17:13 < jplacerda> Hmm okay
17:14 < jplacerda> Have you run into this problem before?
17:14 < sjlombardo> I haven't personally, but generally you wouldn't run rekey before key
17:14 < sjlombardo> the former assumes that the database is already encrypted…
17:15 < jplacerda> sjlombardo: agreed
17:15 < jplacerda> But it would be a bit more robust to not have it dump core
17:15 < sjlombardo> agreed ;)
17:16 < sjlombardo> I'm sure we can bullet proof that so it just errors on when running out of order
17:16 < jplacerda> I mean sqlcipher has things which falls into two categaries as I see it: stuff that is done before the database is keys, and must fail elegantly, and stuff that happens after the database is keyed, and
                   thus should just work
17:16 < jplacerda> I think that rekey falls into the former, and just fail a bit more elegantly
17:16 < jplacerda> sjlombardo: yup, just throw the same error as you do when accessing before a key
17:18 < jplacerda> Alright, my next question is about this: PRAGMA key = "x'2DD29CA851E7B56E4697B0E1F08507293D761A05CE4D1B628663F411A8086D99'";
17:19 < jplacerda> Why use a blob instead of a regular key?
17:25 < sjlombardo> it basically has to do with key derivation. By default, when you supply a text based key, sqlcipher need to convert that into actual key data. In order to make it more difficult to run a dictionary
                    attack, sqlcipher does two things - 1. it maintains a 128-bit random per-database salt, and 2) it uses your password and the random salt to feed key derivation (PBKDF2) to get the actual encryption key.
                    In general this is really good
17:26 < sjlombardo> It also makes the prospect of running a dictionary attack on the database less feasible, since you can't pre-generate a key set (varies per db), or do simple / fast sha1 calculations (pbkdf2 in sqlcipher
                    uses a default of 4000 rounds).
17:26 < jplacerda> Okay, that makes sense
17:27 < jplacerda> sjlombardo: I'm currently generating the key as key = base64.b64encode(hashlib.sha256(str(random.getrandbits(256))).digest())
17:28 < sjlombardo> however, some may not want to do this because they either 1. don't want to use key derivation for performance reasons or because they do their own key derivation, 2. the want the key to be something
explicit (i.e. if they were using a fixed key negotiated between two parties asymettrically), or 3. they want the actual encryption key to be identical between databases
17:28 < jplacerda> And providing it using sqlite_key(key, len(key)). Does this sound good to you?
17:28 < jplacerda> I see.
17:28 < jplacerda> What would you recommend for our purposes?
17:29 < sjlombardo> in general, if the user is providing a passphrase I would recommend sending that directly to sqlite3_key
17:29 < jplacerda> sjlombardo: this is going to be mostly opaque to the user, they won't be providing a passphrase
17:29 < sjlombardo> so you are taking the passphrase today, and expanding it to 256 bits and then passing it as a blob
17:29 < sjlombardo> oh, ok. where do you get the key from?
17:30 < sjlombardo> i.e above, when you call random.getrandbits, you will stash that key somewhere so it can be used to open the database later
17:31 < jplacerda> sjlombardo: I randomly generate (PRNG) a 256 bit number, apply sha256, and b64 encode it
17:31 < jplacerda> this is then stored in the users keyring
17:31 < jplacerda> And retrieved whenever necessary
17:32 < sjlombardo> ok gothca, so the actual key string is the base64 encoded data
17:33 < sjlombardo> one question - do you hold the database handle open for the duration, or repeatedly open / closeit
17:34 < jplacerda> sjlombardo: it's mostly open for the duration of the process.
17:35 < sjlombardo> ok perfect… the reason I ask is that the internal key derivation is somewhat time consuming. you'd probably not notice it if you open it once and then use it continuously, but if you open/closed a lot it
                    could be a problem.
17:37 < sjlombardo> the way you are doing it sounds fine. the only thing to be aware of is since you are passing in a string as the key, sqlcipher is going to run key derivation with the salt on the inbound too. I think this
                    is good though.
17:37 < jplacerda> sjlombardo: yeah, it sounds perfect