This textbook was written for the clinical research community at Johns Hopkins leveraging the precision medicine analytics platform (PMAP). These notebooks are available in html form on the Precision Medicine portal as well as in computational form in the CAMP-share folder on Crunchr (Crunchr.pm.jh.edu).

• Platform Tool : Jupyter/Crunchr on PMAP using
• Programming Language : Python 3
• Author(s) : John Scott
• License : The notebook is released under the Apache 2.0 License
• Last Updated : June 10th, 2019

How to access files from your Johns Hopkins H: drive

Note: Currently you will need to install a library to mount your drive. Future iterations will include this library automatically.

Setup: Install and import dependencies

PySmbClient is a python wrapper around smbclient which allows you to connect to and interact with a samba share. Once installed you may import it along with other required dependencies

pip install PySmbClient

Collecting PySmbClient
Installing collected packages: PySmbClient
Successfully installed PySmbClient-0.1.5
Note: you may need to restart the kernel to use updated packages.

import getpass
from SciServer import Authentication
import smbclient
import pandas as pd

Setup your JHED Credentials

# Setup JHED login credentials
myUserName = Authentication.getKeystoneUserWithToken(Authentication.getToken()).userName
passwd = getpass.getpass('Password for ' + myUserName + ': ')
user = "win\\" + myUserName

Password for jscott64: ········

Get data from your Johns Hopkins H Drive

# Connect to h drive using smbclient
usernamePwd = user+"%"+passwd
smb = smbclient.SambaClient(server="win.ad.jhu.edu", share="Users$",
                                username=usernamePwd, password=passwd, domain='win')

# Browse remote directory
# On H drive share, your files are all under "//HOME..."
dirlist = smb.lsdir("//HOME")
df = pd.DataFrame(dirlist, columns =['Name', 'type', 'Size', 'Date Modified']) 
df

	Name	type	Size	Date Modified
0	My Music	DR	0	2019-04-26 19:34:59
1	favorites	DAR	0	2015-02-10 16:48:51
2	.bash_history	A	133282	2019-05-31 20:03:04
3	.config	D	0	2014-08-28 15:48:08
4	hvd_completed.txt	AH	109	2013-02-04 14:35:51
5	dev	D	0	2015-08-12 20:57:33
6	wiki_movie_plots_deduped.csv.zip	A	30978474	2019-06-10 15:37:49
7	_viminfo	A	1234	2017-10-17 13:52:47
8	desktop.ini	HS	402	2019-04-26 19:35:00
9	.mongorc.js	A	0	2016-02-06 03:53:10
10	.ssh	D	0	2019-04-03 16:47:19
11	gradle.properties	A	167	2015-11-18 16:16:51
12	what.txt	A	19901	2013-11-05 18:52:00
13	15	A	40	2016-10-31 17:03:34
14	My Pictures	DR	0	2019-05-23 20:43:33
15	Visual Studio 2015	D	0	2018-03-01 15:22:42
16	.gitconfig	A	571	2018-11-15 14:22:29
17	SQL Server Management Studio	D	0	2018-03-01 15:13:53
18	My Videos	DR	0	2019-04-26 19:34:55
19	test-20161606-1059.txt	A	0	2016-06-16 14:58:05
20	world-happiness-report-2019.csv	A	6849	2019-06-10 19:41:56
21	.ssh.new	D	0	2015-10-30 20:28:23
22	.lesshst N		34	2019-04-12 16:44:47
23	WINDOWS	D	0	2018-01-31 14:24:22
24	$RECYCLE.BIN	DHS	0	2013-07-11 17:58:02
25	keystoneToken	A	1159	2019-05-08 15:52:34
26	myVeryBigFile	A	841083616	2018-01-02 18:01:13

Copy a file from your H drive to your crunchr file system

# Copy a remote file to your crunchr file system
# Here world-happiness-report-2019.csv is the name of a file at the top level of your H drive
# world-happiness is what you are naming it locally on crunchr
smb.download("//HOME/world-happiness-report-2019.csv", "world-happiness.csv")

# read a file directly from remote src
f = smb.open('//HOME/world-happiness-report-2019.csv')
data = f.read()

print(data)

Country (region),Ladder,SD of Ladder,Positive affect,Negative affect,Social support,Freedom,Corruption,Generosity,"Log of GDP
per capita","Healthy life
expectancy"
Finland,1,4,41,10,2,5,4,47,22,27
Denmark,2,13,24,26,4,6,3,22,14,23
Norway,3,8,16,29,3,3,8,11,7,12
Iceland,4,9,3,3,1,7,45,3,15,13
Netherlands,5,1,12,25,15,19,12,7,12,18
Switzerland,6,11,44,21,13,11,7,16,8,4
Sweden,7,18,34,8,25,10,6,17,13,17
New Zealand,8,15,22,12,5,8,5,8,26,14
Canada,9,23,18,49,20,9,11,14,19,8
Austria,10,10,64,24,31,26,19,25,16,15
Australia,11,26,47,37,7,17,13,6,18,10
Costa Rica,12,62,4,87,42,16,58,75,67,28
Israel,13,14,104,69,38,93,74,24,31,11
Luxembourg,14,3,62,19,27,28,9,30,2,16
United Kingdom,15,16,52,42,9,63,15,4,23,24
Ireland,16,34,33,32,6,33,10,9,6,20
Germany,17,17,65,30,39,44,17,19,17,25
Belgium,18,7,57,53,22,53,20,44,21,26
United States,19,49,35,70,37,62,42,12,10,39
Czech Republic,20,20,74,22,24,58,121,117,32,31
United Arab Emirates,21,65,43,56,72,4,,15,4,60
Malta,22,42,83,103,16,12,32,5,28,19
Mexico,23,76,6,40,67,71,87,120,57,46
France,24,19,56,66,32,69,21,68,25,5
Taiwan,25,37,17,1,48,102,56,56,,
Chile,26,61,15,78,58,98,99,45,49,30
Guatemala,27,136,8,85,78,25,82,78,99,85
Saudi Arabia,28,93,49,82,62,68,,82,11,74
Qatar,29,86,,,,,,,1,43
Spain,30,21,107,107,26,95,78,50,30,3
Panama,31,121,7,48,41,32,104,88,51,33
Brazil,32,116,69,105,43,84,71,108,70,72
Uruguay,33,88,10,76,35,30,33,80,52,35
Singapore,34,5,38,2,36,20,1,21,3,1
El Salvador,35,112,23,84,83,74,85,134,100,75
Italy,36,31,99,123,23,132,128,48,29,7
Bahrain,37,83,39,83,59,24,,23,20,42
Slovakia,38,39,53,47,21,108,142,70,35,38
Trinidad and Tobago,39,89,14,52,29,51,141,41,38,93
Poland,40,28,76,33,44,52,108,77,41,36
Uzbekistan,41,99,19,15,11,1,18,29,104,83
Lithuania,42,55,138,41,17,122,113,124,36,62
Colombia,43,120,30,88,52,56,124,111,74,51
Slovenia,44,54,114,71,14,13,97,54,34,29
Nicaragua,45,133,31,125,66,70,43,71,108,53
Kosovo,46,107,71,7,85,50,144,31,88,
Argentina,47,97,28,93,46,54,109,123,55,37
Romania,48,75,80,62,86,57,146,102,48,61
Cyprus,49,95,60,99,90,81,115,39,33,6
Ecuador,50,113,11,113,71,42,68,95,86,45
Kuwait,51,98,89,97,69,47,,42,5,70
Thailand,52,81,20,35,53,18,131,10,62,58
Latvia,53,30,119,38,34,126,92,105,43,68
South Korea,54,57,101,45,91,144,100,40,27,9
Estonia,55,32,50,6,12,45,30,83,37,41
Jamaica,56,102,51,51,28,49,130,119,93,55
Mauritius,57,94,55,16,54,40,96,37,53,73
Japan,58,43,73,14,50,64,39,92,24,2
Honduras,59,151,13,73,84,39,79,51,113,57
Kazakhstan,60,40,81,5,19,80,57,57,47,88
Bolivia,61,71,70,138,93,35,91,104,101,94
Hungary,62,36,86,31,51,138,140,100,42,56
Paraguay,63,90,1,39,30,34,76,67,90,81
Northern Cyprus,64,35,144,90,81,77,29,43,,
Peru,65,114,36,127,77,61,132,126,76,47
Portugal,66,73,97,100,47,37,135,122,39,22
Pakistan,67,53,130,111,130,114,55,58,110,114
Russia,68,64,96,9,40,107,127,101,45,89
Philippines,69,119,42,116,75,15,49,115,97,99
Serbia,70,100,148,92,57,124,118,84,71,48
Moldova,71,45,133,67,65,128,148,86,109,86
Libya,72,115,85,137,73,79,31,87,63,96
Montenegro,73,84,143,118,60,139,77,76,61,44
Tajikistan,74,50,120,54,113,86,35,72,123,92
Croatia,75,29,122,101,79,118,139,81,50,32
Hong Kong,76,33,105,28,76,66,14,18,9,
Dominican Republic,77,155,66,77,55,43,52,93,69,80
Bosnia and Herzegovina ,78,80,116,79,92,137,145,32,82,50
Turkey,79,58,154,121,61,140,50,98,44,69
Malaysia,80,12,25,23,97,36,137,27,40,59
Belarus,81,22,149,36,33,131,37,103,58,76
Greece,82,87,102,94,102,150,123,152,46,21
Mongolia,83,48,95,17,10,112,119,38,80,97
Macedonia,84,67,140,89,74,105,125,55,75,52
Nigeria,85,130,61,55,111,75,114,59,107,145
Kyrgyzstan,86,46,58,4,45,38,138,36,120,91
Turkmenistan,87,2,135,63,8,83,,33,60,100
Algeria,88,56,113,106,101,149,46,128,72,78
Morocco,89,101,110,91,139,76,84,154,98,79
Azerbaijan,90,24,134,20,104,101,22,146,65,82
Lebanon,91,60,150,61,89,136,133,63,73,66
Indonesia,92,108,9,104,94,48,129,2,83,98
China,93,72,21,11,108,31,,133,68,34
Vietnam,94,27,121,27,64,23,86,97,105,49
Bhutan,95,6,37,98,68,59,25,13,95,104
Cameroon,96,131,106,129,129,90,120,91,121,141
Bulgaria,97,47,117,13,18,115,147,112,56,65
Ghana,98,129,92,72,132,91,117,52,114,121
Ivory Coast,99,134,88,130,137,100,62,114,118,147
Nepal,100,128,137,134,87,67,65,46,127,95
Jordan,101,127,112,120,88,88,,118,92,63
Benin,102,149,118,148,153,103,75,116,128,133
Congo (Brazzaville),103,152,124,136,138,92,60,140,111,116
Gabon,104,105,111,144,95,119,103,143,59,108
Laos,105,59,5,112,120,22,27,34,102,112
South Africa,106,124,40,80,63,85,102,89,77,123
Albania,107,126,90,108,133,87,134,60,81,40
Venezuela,108,141,77,135,49,145,110,139,78,71
Cambodia,109,135,27,142,109,2,94,61,116,102
Palestinian Territories,110,110,128,140,82,134,90,147,112,
Senegal,111,44,68,60,106,121,88,130,126,109
Somalia,112,74,2,18,145,14,16,96,,144
Namibia,113,106,75,59,70,82,98,142,89,122
Niger,114,144,79,141,140,111,51,135,148,138
Burkina Faso,115,92,115,117,116,127,47,125,137,136
Armenia,116,82,126,145,117,123,93,129,91,64
Iran,117,109,109,150,134,117,44,28,54,77
Guinea,118,146,82,143,136,109,70,94,130,137
Georgia,119,51,141,43,147,104,28,153,87,84
Gambia,120,142,29,109,125,89,26,64,139,130
Kenya,121,118,59,46,123,72,105,26,122,106
Mauritania,122,68,94,58,99,151,67,148,117,120
Mozambique,123,154,108,131,122,46,40,121,146,134
Tunisia,124,79,147,132,121,143,101,144,84,67
Bangladesh,125,52,145,68,126,27,36,107,119,90
Iraq,126,147,151,154,124,130,66,73,64,107
Congo (Kinshasa),127,78,125,95,107,125,106,127,149,140
Mali,128,96,48,122,112,110,107,138,129,142
Sierra Leone,129,153,139,149,135,116,112,79,145,146
Sri Lanka,130,91,32,81,80,55,111,35,79,54
Myanmar,131,70,45,86,96,29,24,1,106,110
Chad,132,139,136,151,141,142,80,106,133,148
Ukraine,133,69,131,44,56,141,143,66,94,87
Ethiopia,134,38,100,74,119,106,53,99,135,115
Swaziland,135,104,26,57,103,113,41,145,96,
Uganda,136,148,91,139,114,99,95,74,136,127
Egypt,137,66,146,124,118,129,89,132,85,101
Zambia,138,145,84,128,115,73,69,53,115,131
Togo,139,103,123,147,149,120,72,131,142,132
India,140,41,93,115,142,41,73,65,103,105
Liberia,141,156,103,146,127,94,126,110,150,126
Comoros,142,143,67,114,143,148,81,62,143,117
Madagascar,143,77,46,96,128,146,116,136,144,111
Lesotho,144,150,72,64,98,97,59,151,124,149
Burundi,145,138,98,126,152,135,23,149,151,135
Zimbabwe,146,123,63,34,110,96,63,141,131,129
Haiti,147,111,142,119,146,152,48,20,138,125
Botswana,148,125,87,65,105,60,54,150,66,113
Syria,149,137,155,155,154,153,38,69,,128
Malawi,150,132,129,110,150,65,64,109,147,119
Yemen,151,85,153,75,100,147,83,155,141,124
Rwanda,152,63,54,102,144,21,2,90,132,103
Tanzania,153,122,78,50,131,78,34,49,125,118
Afghanistan,154,25,152,133,151,155,136,137,134,139
Central African Republic,155,117,132,153,155,133,122,113,152,150
South Sudan,156,140,127,152,148,154,61,85,140,143

Get data from your Johns Hopkins S Drive

Pre-requisites:

• Setup Dependencies
• Setup Your JHED Credentials

# Connect to S drive using smbclient
smb2 = smbclient.SambaClient(server="cloud.nas.jh.edu", share="sddesktop$",
                                username=usernamePwd, password=passwd, domain='win')

# Browse a remote directory
dirlist = smb2.lsdir("//TIC/jscott64")
df = pd.DataFrame(dirlist, columns =['Name', 'type', 'Size', 'Date Modified']) 
df

	Name	type	Size	Date Modified
0	wiki_movie_plots_deduped.csv.zip	A	30978474	2019-06-10 15:37:49

# Copy a remote zipfile to your crunchr file system
# Load it into a pandas data frame
smb2.download("//TIC/jscott64/wiki_movie_plots_deduped.csv.zip", "wiki_movie_plots.csv.zip")
df = pd.read_csv('wiki_movie_plots.csv.zip', compression='zip', header=0, sep=',')
df.head(10)

	Release Year	Title	Origin/Ethnicity	Director	Cast	Genre	Wiki Page	Plot
0	1901	Kansas Saloon Smashers	American	Unknown	NaN	unknown	https://en.wikipedia.org/wiki/Kansas_Saloon_Sm...	A bartender is working at a saloon, serving dr...
1	1901	Love by the Light of the Moon	American	Unknown	NaN	unknown	https://en.wikipedia.org/wiki/Love_by_the_Ligh...	The moon, painted with a smiling face hangs ov...
2	1901	The Martyred Presidents	American	Unknown	NaN	unknown	https://en.wikipedia.org/wiki/The_Martyred_Pre...	The film, just over a minute long, is composed...
3	1901	Terrible Teddy, the Grizzly King	American	Unknown	NaN	unknown	https://en.wikipedia.org/wiki/Terrible_Teddy,_...	Lasting just 61 seconds and consisting of two ...
4	1902	Jack and the Beanstalk	American	George S. Fleming, Edwin S. Porter	NaN	unknown	https://en.wikipedia.org/wiki/Jack_and_the_Bea...	The earliest known adaptation of the classic f...
5	1903	Alice in Wonderland	American	Cecil Hepworth	May Clark	unknown	https://en.wikipedia.org/wiki/Alice_in_Wonderl...	Alice follows a large white rabbit down a "Rab...
6	1903	The Great Train Robbery	American	Edwin S. Porter	NaN	western	https://en.wikipedia.org/wiki/The_Great_Train_...	The film opens with two bandits breaking into ...
7	1904	The Suburbanite	American	Wallace McCutcheon	NaN	comedy	https://en.wikipedia.org/wiki/The_Suburbanite	The film is about a family who move to the sub...
8	1905	The Little Train Robbery	American	Edwin Stanton Porter	NaN	unknown	https://en.wikipedia.org/wiki/The_Little_Train...	The opening scene shows the interior of the ro...
9	1905	The Night Before Christmas	American	Edwin Stanton Porter	NaN	unknown	https://en.wikipedia.org/wiki/The_Night_Before...	Scenes are introduced using lines of the poem....