Comments for e-Research http://blog.openwetware.org/deroure What research do you want to do tomorrow? Sun, 25 Jul 2010 12:20:34 +0000 hourly 1 http://wordpress.org/?v=3.1.4 Comment on Drop and Compute by Ian Cottam http://blog.openwetware.org/deroure/?p=97#comment-5161 Ian Cottam Sun, 25 Jul 2010 12:20:34 +0000 http://blog.openwetware.org/deroure/?p=97#comment-5161 There is an even simpler solution to the user-side encryption some people desire; and that is to use EncFS (Google for it). An encrypted file system is kept within the shared dropbox folder. EncFS keeps another folder -- outside of the dropbox area -- where you can see the unencrypted version of your files. As these never pass through the Dropbox servers, one is safe from prying eyes. Unfortunately, it only works with Macs and Linux boxes (not Windows). -Ian There is an even simpler solution to the user-side encryption some people desire; and that is to use EncFS (Google for it). An encrypted file system is kept within the shared dropbox folder. EncFS keeps another folder — outside of the dropbox area — where you can see the unencrypted version of your files. As these never pass through the Dropbox servers, one is safe from prying eyes.

Unfortunately, it only works with Macs and Linux boxes (not Windows).
-Ian

]]> Comment on Replacing the Paper: the six Rs of the e-Research Record by kitsopens http://blog.openwetware.org/deroure/?p=56#comment-5149 kitsopens Wed, 21 Apr 2010 17:12:51 +0000 http://blog.openwetware.org/deroure/?p=56#comment-5149 hey.its really nice post and good work... hey.its really nice post and good work…

]]> Comment on Drop and Compute by Ian Cottam http://blog.openwetware.org/deroure/?p=97#comment-4284 Ian Cottam Wed, 09 Dec 2009 14:02:33 +0000 http://blog.openwetware.org/deroure/?p=97#comment-4284 A modified Bash Script and Process including full, user-side hard encryption I haven't written or tested this, but see if you think it sounds good: 1. User submits an encrypted zip file. Encryption method should be e.g. AES256 and the key needs to be known to the DropAndCompute script. 2. When the receiving Bash script spots it, it copies it to a secondary, non-dropbox-synced folder before unencrypting, unzipping and submitting the job. 3. If the user drags a *.debug file to the shared folder, it is copied over to the secondary folder where it is actioned and the debug results copied back to the share for the user to see. This step requires no encryption. 4. If the user drags a *.kill file to the shared folder, it is copied over to the secondary folder where it is actioned and the results (just a log file) copied back to the share for the user to see. The secondary submission (unencrypted) folder is deleted. This step requires no encryption. 5. The Bash script has to monitor the log file or the queue to detect when the job has finished. When it has, the secondary folder -- now containing the result files too -- is encrypted, zipped and copied back to the share for the user to see. The secondary folder is deleted. The main disadvantage over the clear, unencrypted form would be that the user does not see outputs as they are generated; however, this seems minor. -Ian A modified Bash Script and Process including full, user-side hard encryption

I haven’t written or tested this, but see if you think it sounds good:

1. User submits an encrypted zip file. Encryption method should be e.g. AES256 and the key needs to be known to the DropAndCompute script.

2. When the receiving Bash script spots it, it copies it to a secondary, non-dropbox-synced folder before unencrypting, unzipping and submitting the job.

3. If the user drags a *.debug file to the shared folder, it is copied over to the secondary folder where it is actioned and the debug results copied back to the share for the user to see. This step requires no encryption.

4. If the user drags a *.kill file to the shared folder, it is copied over to the secondary folder where it is actioned and the results (just a log file) copied back to the share for the user to see. The secondary submission (unencrypted) folder is deleted. This step requires no encryption.

5. The Bash script has to monitor the log file or the queue to detect when the job has finished. When it has, the secondary folder — now containing the result files too — is encrypted, zipped and copied back to the share for the user to see. The secondary folder is deleted.

The main disadvantage over the clear, unencrypted form would be that the user does not see outputs as they are generated; however, this seems minor.

-Ian

]]> Comment on Drop and Compute by Ian Cottam http://blog.openwetware.org/deroure/?p=97#comment-4263 Ian Cottam Thu, 03 Dec 2009 23:16:42 +0000 http://blog.openwetware.org/deroure/?p=97#comment-4263 A Dropbox founder replied to say that for mainly technical reasons they are currently constrained to use US data centres. However, in the future, as they start to look at supporting organisations and businesses, things could change. -Ian A Dropbox founder replied to say that for mainly technical reasons they are currently constrained to use US data centres. However, in the future, as they start to look at supporting organisations and businesses, things could change.
-Ian

]]> Comment on Drop and Compute by Ian Cottam http://blog.openwetware.org/deroure/?p=97#comment-4262 Ian Cottam Tue, 01 Dec 2009 15:28:14 +0000 http://blog.openwetware.org/deroure/?p=97#comment-4262 I just emailed the founders of Dropbox about this, so will report back if they reply. Perhaps we will have to use Amazon S3 directly, but it is nowhere near as convenient as Dropbox. -Ian I just emailed the founders of Dropbox about this, so will report back if they reply.
Perhaps we will have to use Amazon S3 directly, but it is nowhere near as convenient as Dropbox.
-Ian

]]> Comment on Drop and Compute by Ian Cottam http://blog.openwetware.org/deroure/?p=97#comment-4261 Ian Cottam Tue, 01 Dec 2009 10:03:48 +0000 http://blog.openwetware.org/deroure/?p=97#comment-4261 David Wallom knows about it. You could lobby. -Ian David Wallom knows about it. You could lobby. -Ian

]]> Comment on Drop and Compute by David Wallom http://blog.openwetware.org/deroure/?p=97#comment-4260 David Wallom Tue, 01 Dec 2009 09:11:06 +0000 http://blog.openwetware.org/deroure/?p=97#comment-4260 Hi Dave/Anna/Ian The NGS is currently looking into this/something like it using the idea as we have a fairly significant issue with the location of the resulting data since significant user communities won't have a clear understanding of the implications of things such as the patriot act. What would be interesting for example would be that the same simple idea as DB could be used by large data repositories such as UKRDS and the BBSRC data store. The key here is a standard interface that the simple tools can connect into. Regards David Technical Director, UK NGS Hi Dave/Anna/Ian

The NGS is currently looking into this/something like it using the idea as we have a fairly significant issue with the location of the resulting data since significant user communities won’t have a clear understanding of the implications of things such as the patriot act.

What would be interesting for example would be that the same simple idea as DB could be used by large data repositories such as UKRDS and the BBSRC data store. The key here is a standard interface that the simple tools can connect into.

Regards

David

Technical Director, UK NGS

]]> Comment on Drop and Compute by Anna Croft http://blog.openwetware.org/deroure/?p=97#comment-4257 Anna Croft Mon, 30 Nov 2009 22:21:59 +0000 http://blog.openwetware.org/deroure/?p=97#comment-4257 Ian, this is brilliant. We had been using a similar idea to manually shuffle files between our compute machines, but this is ace. Do you know if we could convince NGS or another 'public' grid to adopt it? Ian, this is brilliant. We had been using a similar idea to manually shuffle files between our compute machines, but this is ace. Do you know if we could convince NGS or another ‘public’ grid to adopt it?

]]> Comment on Drop and Compute by Ian Cottam http://blog.openwetware.org/deroure/?p=97#comment-4256 Ian Cottam Mon, 23 Nov 2009 12:21:13 +0000 http://blog.openwetware.org/deroure/?p=97#comment-4256 Thanks for blogging about DropAndCompute. To clarify the bit about the "Patriot Act".... Yes, currently Dropbox shares pass through Amazon S3 servers in the USA. On the positive side, as soon as you drag your folder of inputs/outputs out of the share, it is gone from the US based server. Data on the servers is also encrypted. On the negative side: the encryption key is owned by Dropbox. It is possible to do user side encryption oneself, but this is a little tricky as obviously result files are generated by this approach. Also, deleted data on the server can be undeleted. This used to be unlimited versions forever, but now one has the choice of not keeping deleted files, but they do hang around for about 30 days I think. In the long run we need one of two things: a) Amazon is moving to keep European data in Europe, on its Dublin data centre. We need to encourage Dropbox to support this. b) Dropbox could support user-side encryption. On the Dropbox web site www.dropbox.com there is a Votebox area. All us Europeans should vote for the above! -Ian Thanks for blogging about DropAndCompute.

To clarify the bit about the “Patriot Act”….
Yes, currently Dropbox shares pass through Amazon S3 servers in the USA.
On the positive side, as soon as you drag your folder of inputs/outputs out of the share, it is gone from the US based server. Data on the servers is also encrypted.

On the negative side: the encryption key is owned by Dropbox. It is possible to do user side encryption oneself, but this is a little tricky as obviously result files are generated by this approach. Also, deleted data on the server can be undeleted. This used to be unlimited versions forever, but now one has the choice of not keeping deleted files, but they do hang around for about 30 days I think.

In the long run we need one of two things:
a) Amazon is moving to keep European data in Europe, on its Dublin data centre. We need to encourage Dropbox to support this.
b) Dropbox could support user-side encryption.

On the Dropbox web site
http://www.dropbox.com
there is a Votebox area. All us Europeans should vote for the above!
-Ian

]]> Comment on Reasons to be Blogging 1 2 3 by I bet you think this blog is about you, don’t you? « O’Really? http://blog.openwetware.org/deroure/?p=5#comment-3613 I bet you think this blog is about you, don’t you? « O’Really? Mon, 24 Aug 2009 17:07:16 +0000 http://blog.openwetware.org/deroure/?p=5#comment-3613 [...] because I didn’t speak to or hear you speak or (unlike many people) you’re not vain enough [1] to have a have a blog [...] [...] because I didn’t speak to or hear you speak or (unlike many people) you’re not vain enough [1] to have a have a blog [...]

]]>