Re: pixiedebugger: can't install
Lisa Bang <[email protected]> Mon, 28 Jan 2019 10:08:59 -0500
| Newsgroups | gmane.comp.python.ipython.devel |
|---|---|
| Message-ID | <CANADyYkLGdhoj7ubb=RfqF4NK0JUcNb88H2Mgd=hSJHgmQSHsw@mail.gmail.com> |
--===============7963421091259887754==
Content-Type: multipart/alternative; boundary="000000000000b4d13e0580860f5a"
--000000000000b4d13e0580860f5a
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Hi Dude,
Yes, it looks like when subprocess calls pyspark to get its version,
pyspark returns no version. I do see that this problem has happened
recently for others <https://github.com/pixiedust/pixiedust/issues/741> as
well also. Maybe could be solved with editing the kernel.json in your
Jupyter directory for the pixiedust kernel to include pyspark, typically at
/Users/xxxx/Library/Jupyter/kernels/pythonwithpixiedustspark22/kernel.json =
but
this kind of seems like a pain.
For reference, I have put the output from my code with the same error and
it working after. Perhaps also ensuring the package versions of py4j and
pyspark are the same would help.
Best,
Lisa
XXXX:Documents xxxx$ jupyter pixiedust install
*Step 1: PIXIEDUST_HOME: /Users/xxxx/pixiedust*
Keep y/n [y]? y
*Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spark*
Keep y/n [y]? y
Select an existing spark install or create a new one
*1. spark-2.3.2-bin-hadoop2.7*
*2. Create a new spark Install*
Enter your selection: 1
Traceback (most recent call last):
File "/Users/xxxx/anaconda3/bin/jupyter-pixiedust", line 11, in <module>
sys.exit(main())
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/pixiedustapp.py"=
,
line 41, in main
PixiedustJupyterApp.launch_instance()
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-2>", line 2, in initialize
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 296, in initialize
self.parse_command_line(argv)
File "<decorator-gen-4>", line 2, in parse_command_line
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 514, in parse_command_line
return self.initialize_subcommand(subc, subargv)
File "<decorator-gen-3>", line 2, in initialize_subcommand
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 452, in initialize_subcommand
self.subapp.initialize(argv)
File "<decorator-gen-6>", line 2, in initialize
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/jupyter_core/application=
.py",
line 238, in initialize
self.parse_command_line(argv)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKernel.py"=
,
line 154, in parse_command_line
spark_version =3D self.get_spark_version()
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKernel.py"=
,
line 379, in get_spark_version
pyspark_out =3D subprocess.check_output([pyspark, "--version"],
stderr=3Dsubprocess.STDOUT).decode("utf-8")
File "/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 389, in
check_output
**kwargs).stdout
File "/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 481, in ru=
n
output=3Dstdout, stderr=3Dstderr)
subprocess.CalledProcessError: Command
'['/Users/xxxx/pixiedust/bin/spark/spark-2.3.2-bin-hadoop2.7/bin/pyspark',
'--version']' returned non-zero exit status 1.
XXXX:Documents xxxx$ pip install pyspark
Collecting pyspark
Downloading
https://files.pythonhosted.org/packages/88/01/a37e827c2d80c6a754e40e99b9826=
d978b55254cc6c6672b5b08f2e18a7f/pyspark-2.4.0.tar.gz
(213.4MB)
100% |=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=
=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=
=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
| 213.4MB 135kB/s
Collecting py4j=3D=3D0.10.7 (from pyspark)
Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc3=
2a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.=
whl
(197kB)
100% |=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=
=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=
=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
| 204kB 10.8MB/s
Building wheels for collected packages: pyspark
Running setup.py bdist_wheel for pyspark ... done
Stored in directory:
/Users/xxxx/Library/Caches/pip/wheels/cd/54/c2/abfcc942eddeaa7101228ebd6127=
a30dbdf903c72db4235b23
Successfully built pyspark
Installing collected packages: py4j, pyspark
Successfully installed py4j-0.10.7 pyspark-2.4.0
XXXX:Documents xxxx$ jupyter pixiedust install
*Step 1: PIXIEDUST_HOME: /Users/xxxx/pixiedust*
Keep y/n [y]? y
*Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spark*
Keep y/n [y]? y
Select an existing spark install or create a new one
*1. spark-2.3.2-bin-hadoop2.7*
*2. Create a new spark Install*
Enter your selection: 2
*What version would you like to download? 1.6.3, 2.0.2, 2.1.0, 2.2.0, 2.3.2
[2.3.2]: *2.2.0
SPARK_HOME will be set to
/Users/xxxx/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7
Downloading Spark 2.2.0
Extracting Spark 2.2.0 to /Users/xxxx/pixiedust/bin/spark
Traceback (most recent call last):
File "/Users/xxxx/anaconda3/bin/jupyter-pixiedust", line 11, in <module>
sys.exit(main())
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/pixiedustapp.py"=
,
line 41, in main
PixiedustJupyterApp.launch_instance()
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-2>", line 2, in initialize
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 296, in initialize
self.parse_command_line(argv)
File "<decorator-gen-4>", line 2, in parse_command_line
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 514, in parse_command_line
return self.initialize_subcommand(subc, subargv)
File "<decorator-gen-3>", line 2, in initialize_subcommand
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 452, in initialize_subcommand
self.subapp.initialize(argv)
File "<decorator-gen-6>", line 2, in initialize
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/applica=
tion.py",
line 87, in catch_config_error
return method(app, *args, **kwargs)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/jupyter_core/application=
.py",
line 238, in initialize
self.parse_command_line(argv)
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKernel.py"=
,
line 154, in parse_command_line
spark_version =3D self.get_spark_version()
File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKernel.py"=
,
line 379, in get_spark_version
pyspark_out =3D subprocess.check_output([pyspark, "--version"],
stderr=3Dsubprocess.STDOUT).decode("utf-8")
File "/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 389, in
check_output
**kwargs).stdout
File "/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 481, in ru=
n
output=3Dstdout, stderr=3Dstderr)
subprocess.CalledProcessError: Command
'['/Users/xxxx/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/pyspark',
'--version']' returned non-zero exit status 1.
XXXX:Documents xxxx$
/Users/xxxx/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/pyspark',
'--version'
> ,
> '
-bash:
/Users/xxxx/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/pyspark,
--version
,
: No such file or directory
XXXX:Documents xxxx$
/Users/xxxx/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/pyspark
No Java runtime present, requesting install.
XXXX:Documents xxxx$ conda install pyspark
Solving environment: done
## Package Plan ##
environment location: /Users/xxxx/anaconda3
added / updated specs:
- pyspark
The following packages will be downloaded:
package | build
---------------------------|-----------------
py4j-0.10.7 | py37_0 251 KB
pyspark-2.4.0 | py37_0 203.4 MB
conda-4.6.1 | py37_0 1.7 MB
------------------------------------------------------------
Total: 205.4 MB
The following NEW packages will be INSTALLED:
py4j: 0.10.7-py37_0
pyspark: 2.4.0-py37_0
The following packages will be UPDATED:
conda: 4.5.12-py37_0 --> 4.6.1-py37_0
Proceed ([y]/n)? y
Downloading and Extracting Packages
py4j-0.10.7 | 251 KB |
#################################################################### | 100%
pyspark-2.4.0 | 203.4 MB |
#################################################################### | 100%
conda-4.6.1 | 1.7 MB |
#################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(base) XXXX:Documents xxxx$ jupyter pixiedust install
*Step 1: PIXIEDUST_HOME: /Users/xxxx/pixiedust*
Keep y/n [y]? y
*Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spark*
Keep y/n [y]? n
*Step 2: Please enter a SPARK_HOME location: *
/Users/xxxx/spark-2.3.2-bin-hadoop2.7
*Directory /Users/xxxx/sprk-2.3.2-bin-hadoop2.7 does not exist*
Create y/n [y]? n
*Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spark*
Keep y/n [y]? n
*Step 2: Please enter a SPARK_HOME location: *
/Users/xxxx/Downloads/spark-2.3.4-bin-hadoop2.7
*Directory /Users/xxxx/Downloads/spark-2.3.4-bin-hadoop2.7 does not exist*
Create y/n [y]? n
*Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spark*
Keep y/n [y]? n
*Step 2: Please enter a SPARK_HOME location: *
/Users/xxxx/Downloads/spark-2.3.4-bin-hadoop2.7/bin/pyspark
*Directory /Users/xxxx/Downloads/spark-2.3.4-bin-hadoop2.7/bin/pyspark does
not exist*
Create y/n [y]? n
*Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spark*
Keep y/n [y]? y
Select an existing spark install or create a new one
*1. spark-2.3.2-bin-hadoop2.7*
*2. spark-2.2.0-bin-hadoop2.7*
*3. Create a new spark Install*
Enter your selection: 2
downloaded spark cloudant jar:
/Users/xxxx/pixiedust/bin/cloudant-spark-v2.0.0-185.jar
*Step 3: SCALA_HOME: /Users/xxxx/pixiedust/bin/scala*
Keep y/n [y]? y
*Directory /Users/xxxx/pixiedust/bin/scala does not contain a valid scala
install*
Download Scala y/n [y]? y
SCALA_HOME will be set to /Users/xxxx/pixiedust/bin/scala/scala-2.11.8
Downloading Scala 2.11
Extracting Scala 2.11 to /Users/xxxx/pixiedust/bin/scala
*Step 4: Kernel Name: Python-with-Pixiedust_Spark-2.2*
Keep y/n [y]? y
self.kernelInternalName pythonwithpixiedustspark22
[PixiedustInstall] Installed kernelspec pythonwithpixiedustspark22 in
/Users/xxxx/Library/Jupyter/kernels/pythonwithpixiedustspark22
Downloading intro notebooks into /Users/xxxx/pixiedust/notebooks
...
https://github.com/ibm-watson-data-lab/pixiedust/raw/master/notebook/PixieD=
ust
1 - Easy Visualizations.ipynb : *done*
...
https://github.com/ibm-watson-data-lab/pixiedust/raw/master/notebook/PixieD=
ust
2 - Working with External Data.ipynb : *done*
...
https://github.com/ibm-watson-data-lab/pixiedust/raw/master/notebook/PixieD=
ust
3 - Scala and Python.ipynb : *done*
...
https://github.com/ibm-watson-data-lab/pixiedust/raw/master/notebook/PixieD=
ust
4 - Add External Spark Packages.ipynb : *done*
...
https://github.com/ibm-watson-data-lab/pixiedust/raw/master/notebook/PixieD=
ust
5 - Stash to Cloudant.ipynb : *done*
...
https://github.com/ibm-watson-data-lab/pixiedust/raw/master/notebook/PixieD=
ust
Contribute.ipynb : *done*
###########################################################################=
#########################
# Congratulations: Kernel Python-with-Pixiedust_Spark-2.2 was
successfully created in
/Users/xxxx/Library/Jupyter/kernels/pythonwithpixiedustspark22
# You can start the Notebook server with the following command:
# *jupyter notebook /Users/xxxx/pixiedust/notebooks*
###########################################################################=
#########################
On Sun, Jan 27, 2019, 11:55 PM The Dude <[email protected]> wrote:
> Hi Lisa,
> thanks for your help.
>
> Unfortunately, I get the same error after installing spark with conda.
> Here=E2=80=99s the output from my terminal:
>
>
> (py37ana) adula-5:~ dude$ jupyter pixiedust install
> Step 1: PIXIEDUST_HOME: /Users/dude/pixiedust
> Keep y/n [y]? y
> Step 2: SPARK_HOME: /Users/dude/pixiedust/bin/spark
> Keep y/n [y]? y
> Select an existing spark install or create a new one
> 1. spark-2.2.0-bin-hadoop2.7
> 2. Create a new spark Install
> Enter your selection: 1
> Traceback (most recent call last):
> File "/Users/dude/anaconda/envs/py37ana/bin/jupyter-pixiedust", line 11=
,
> in <module>
> sys.exit(main())
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/install/pi=
xiedustapp.py",
> line 41, in main
> PixiedustJupyterApp.launch_instance()
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 657, in launch_instance
> app.initialize(argv)
> File "<decorator-gen-2>", line 2, in initialize
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 87, in catch_config_error
> return method(app, *args, **kwargs)
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 296, in initialize
> self.parse_command_line(argv)
> File "<decorator-gen-4>", line 2, in parse_command_line
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 87, in catch_config_error
> return method(app, *args, **kwargs)
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 514, in parse_command_line
> return self.initialize_subcommand(subc, subargv)
> File "<decorator-gen-3>", line 2, in initialize_subcommand
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 87, in catch_config_error
> return method(app, *args, **kwargs)
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 452, in initialize_subcommand
> self.subapp.initialize(argv)
> File "<decorator-gen-6>", line 2, in initialize
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/=
config/application.py",
> line 87, in catch_config_error
> return method(app, *args, **kwargs)
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/jupyter_co=
re/application.py",
> line 238, in initialize
> self.parse_command_line(argv)
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/install/cr=
eateKernel.py",
> line 154, in parse_command_line
> spark_version =3D self.get_spark_version()
> File
> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/install/cr=
eateKernel.py",
> line 379, in get_spark_version
> pyspark_out =3D subprocess.check_output([pyspark, "--version"],
> stderr=3Dsubprocess.STDOUT).decode("utf-8")
> File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/subprocess.py",
> line 389, in check_output
> **kwargs).stdout
> File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/subprocess.py",
> line 481, in run
> output=3Dstdout, stderr=3Dstderr)
> subprocess.CalledProcessError: Command
> '['/Users/dude/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/pyspark'=
,
> '--version']' returned non-zero exit status 1.dude
>
>
>
> It seems that the pyspark executable doesn=E2=80=99t run properly.
>
> TheDude
>
>
>
>
>
> On Jan 27, 2019, at 11:26:25, Lisa Bang <[email protected]> wrote:
>
>
> Hi,
>
> Looks like you don't have pyspark installed. I reproduced your error and
> after running "conda install pyspark" , "jupyter pixiedust install"
> finished just fine.
>
> Best,
> Lisa
>
>
> On Sat, Jan 26, 2019 at 11:22 PM TheDude <[email protected]> wrote:
>
>> Hello,
>> I would really like to have the pixiedebugger working in my
>> notebook. Unfortunately I ran into problems with the installation.
>>
>> I am following the instructions here: <
>> https://pixiedust.github.io/pixiedust/install.html>
>> For what is worth, I am using Python 3.7 in Anaconda on macOS 10.12.
>>
>> The first part of the installation:
>> pip install pixiedust
>>
>> is successful, and it ends with:
>> Successfully built pixiedust mpld3
>> Installing collected packages: mpld3, geojson, astunparse,
>> markdown, colour, pixiedust
>> Successfully installed astunparse-1.6.2 colour-0.1.5
>> geojson-2.4.1 markdown-3.0.1 mpld3-0.3 pixiedust-1.1.15
>>
>> The next step, installing a new Jupyter kernel, fails:
>> jupyter pixiedust install
>>
>> Here=E2=80=99s what I get:
>>
>> (py37ana) adula-5:~ dude$ jupyter pixiedust install
>> Step 1: PIXIEDUST_HOME: /Users/dude/pixiedust
>> Keep y/n [y]? y
>> Step 2: SPARK_HOME: /Users/dude/pixiedust/bin/spark
>> Keep y/n [y]? y
>> Select an existing spark install or create a new one
>> 1. spark-2.2.0-bin-hadoop2.7
>> 2. Create a new spark Install
>> Enter your selection: 1
>> Traceback (most recent call last):
>> File "/Users/dude/anaconda/envs/py37ana/bin/jupyter-pixiedust", line 11=
,
>> in <module>
>> sys.exit(main())
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/install/p=
ixiedustapp.py",
>> line 41, in main
>> PixiedustJupyterApp.launch_instance()
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 657, in launch_instance
>> app.initialize(argv)
>> File "<decorator-gen-2>", line 2, in initialize
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 87, in catch_config_error
>> return method(app, *args, **kwargs)
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 296, in initialize
>> self.parse_command_line(argv)
>> File "<decorator-gen-4>", line 2, in parse_command_line
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 87, in catch_config_error
>> return method(app, *args, **kwargs)
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 514, in parse_command_line
>> return self.initialize_subcommand(subc, subargv)
>> File "<decorator-gen-3>", line 2, in initialize_subcommand
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 87, in catch_config_error
>> return method(app, *args, **kwargs)
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 452, in initialize_subcommand
>> self.subapp.initialize(argv)
>> File "<decorator-gen-6>", line 2, in initialize
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets=
/config/application.py",
>> line 87, in catch_config_error
>> return method(app, *args, **kwargs)
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/jupyter_c=
ore/application.py",
>> line 238, in initialize
>> self.parse_command_line(argv)
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/install/c=
reateKernel.py",
>> line 154, in parse_command_line
>> spark_version =3D self.get_spark_version()
>> File
>> "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/install/c=
reateKernel.py",
>> line 379, in get_spark_version
>> pyspark_out =3D subprocess.check_output([pyspark, "--version"],
>> stderr=3Dsubprocess.STDOUT).decode("utf-8")
>> File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/subprocess.py",
>> line 389, in check_output
>> **kwargs).stdout
>> File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/subprocess.py",
>> line 481, in run
>> output=3Dstdout, stderr=3Dstderr)
>> subprocess.CalledProcessError: Command
>> '['/Users/dude/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/pyspark=
',
>> '--version']' returned non-zero exit status 1.
>>
>>
>> Anybody has an idea on how to proceed?
>>
>> TIA.
>>
>> TheDude
>>
>>
>> _______________________________________________
>> IPython-dev mailing list
>> [email protected]
>> https://mail.python.org/mailman/listinfo/ipython-dev
>>
>
>
--000000000000b4d13e0580860f5a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi Dude,<div>=C2=A0</div><div>Yes, it looks like when subp=
rocess calls pyspark to get its version, pyspark returns no version.=C2=A0=
=C2=A0I do see that this problem has happened recently for <a href=3D"https=
://github.com/pixiedust/pixiedust/issues/741">others</a> as well also.=C2=
=A0 Maybe could be solved with editing the kernel.json in your Jupyter dire=
ctory for the pixiedust kernel to include pyspark, typically at=C2=A0<span =
style=3D"font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-f=
amily:Menlo;font-size:11px">/Users/xxxx/Library/Jupyter/kernels/pythonwithp=
ixiedustspark22/kernel.json=C2=A0</span>but this kind of seems like a pain.=
=C2=A0</div><div><br></div><div>For reference, I have put the output from m=
y code with the same error and it working after. Perhaps also ensuring the =
package versions of py4j and pyspark are the same would help.</div><div><br=
></div><div>Best,</div><div>Lisa</div><div><br></div><div><br></div><div><p=
class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font-f=
amily:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo;c=
olor:black">XXXX:Documents xxxx$ jupyter pixiedust install</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 1: PIXIEDUST_HOME: /Users/xxxx/pixiedust</sp=
an></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"></span>=
</p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spa=
rk</span></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"><=
/span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Select an existing spark install or create a new
one</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">1. spark-2.3.2-bin-hadoop2.7</span></b><span styl=
e=3D"font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">2. Create a new spark Install</span></b><span sty=
le=3D"font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Enter
your selection: 1</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Traceback (most recent call last):</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/bin/jupyter-pixiedust", line 11, in
<module></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
sys.exit(main())</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/pixiedustap=
p.py",
line 41, in main</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
PixiedustJupyterApp.launch_instance()</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 657, in launch_instance</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
app.initialize(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-2>", line 2, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 296, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
self.parse_command_line(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-4>", line 2, in parse_command_line</span></=
p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 514, in parse_command_line</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
self.initialize_subcommand(subc, subargv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-3>", line 2, in initialize_subcommand</span=
></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 452, in initialize_subcommand</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
self.subapp.initialize(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-6>", line 2, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/jupyter_core/applic=
ation.py",
line 238, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
self.parse_command_line(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKerne=
l.py",
line 154, in parse_command_line</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
spark_version =3D self.get_spark_version()</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKerne=
l.py",
line 379, in get_spark_version</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
pyspark_out =3D subprocess.check_output([pyspark, "--version"],
stderr=3Dsubprocess.STDOUT).decode("utf-8")</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 389, in
check_output</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
**kwargs).stdout</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 481, in=
run</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
output=3Dstdout, stderr=3Dstderr)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">subprocess.CalledProcessError: Command
'['/Users/xxxx/pixiedust/bin/spark/spark-2.3.2-bin-hadoop2.7/bin/py=
spark',
'--version']' returned non-zero exit status 1.</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">XXXX:Documents xxxx$ pip install pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Collecting pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0
Downloading <a href=3D"https://files.pythonhosted.org/packages/88/01/a37e82=
7c2d80c6a754e40e99b9826d978b55254cc6c6672b5b08f2e18a7f/pyspark-2.4.0.tar.gz=
">https://files.pythonhosted.org/packages/88/01/a37e827c2d80c6a754e40e99b98=
26d978b55254cc6c6672b5b08f2e18a7f/pyspark-2.4.0.tar.gz</a>
(213.4MB)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 100% |=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=
=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=
=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
=E2=96=88=E2=96=88=E2=96=88|
213.4MB 135kB/s </span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Collecting py4j=3D=3D0.10.7 (from pyspark)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0
Downloading
<a href=3D"https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7=
cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl
(197kB">https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4=
f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl
(197kB</a>)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 100% |=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=
=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=
=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=E2=96=88=
=E2=96=88=E2=96=88=E2=96=88|
204kB 10.8MB/s </span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Building wheels for collected packages: pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 Running
setup.py bdist_wheel for pyspark ... done</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 Stored
in directory: /Users/xxxx/Library/Caches/pip/wheels/cd/54/c2/abfcc942eddeaa=
7101228ebd6127a30dbdf903c72db4235b23</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Successfully built pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Installing collected packages: py4j, pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Successfully installed py4j-0.10.7 pyspark-2.4.0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">XXXX:Documents xxxx$ jupyter pixiedust install</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 1: PIXIEDUST_HOME: /Users/xxxx/pixiedust</sp=
an></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"></span>=
</p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spa=
rk</span></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"><=
/span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Select an existing spark install or create a new
one</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">1. spark-2.3.2-bin-hadoop2.7</span></b><span styl=
e=3D"font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">2. Create a new spark Install</span></b><span sty=
le=3D"font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Enter
your selection: 2</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">What version would you like to download?
1.6.3, 2.0.2, 2.1.0, 2.2.0, 2.3.2 [2.3.2]: </span></b><span style=3D"font-s=
ize:10pt;font-family:Menlo;color:black">2.2.0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">SPARK_HOME will be set to /Users/xxxx/pixiedust/bin/spark/spa=
rk-2.2.0-bin-hadoop2.7</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Downloading Spark 2.2.0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Extracting Spark 2.2.0 to /Users/xxxx/pixiedust/bin/spark</sp=
an></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Traceback (most recent call last):</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/bin/jupyter-pixiedust", line 11, in
<module></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
sys.exit(main())</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/pixiedustap=
p.py",
line 41, in main</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
PixiedustJupyterApp.launch_instance()</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 657, in launch_instance</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
app.initialize(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-2>", line 2, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 296, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
self.parse_command_line(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-4>", line 2, in parse_command_line</span></=
p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 514, in parse_command_line</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
self.initialize_subcommand(subc, subargv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-3>", line 2, in initialize_subcommand</span=
></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 452, in initialize_subcommand</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
self.subapp.initialize(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"<decorator-gen-6>", line 2, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/traitlets/config/ap=
plication.py",
line 87, in catch_config_error</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 return
method(app, *args, **kwargs)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/jupyter_core/applic=
ation.py",
line 238, in initialize</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
self.parse_command_line(argv)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKerne=
l.py",
line 154, in parse_command_line</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
spark_version =3D self.get_spark_version()</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/site-packages/install/createKerne=
l.py",
line 379, in get_spark_version</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
pyspark_out =3D subprocess.check_output([pyspark, "--version"],
stderr=3Dsubprocess.STDOUT).decode("utf-8")</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 389, in
check_output</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
**kwargs).stdout</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 File
"/Users/xxxx/anaconda3/lib/python3.7/subprocess.py", line 481, in=
run</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
output=3Dstdout, stderr=3Dstderr)</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">subprocess.CalledProcessError: Command
'['/Users/xxxx/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/py=
spark',
'--version']' returned non-zero exit status 1.</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">XXXX:Documents xxxx$ /Users/xxxx/pixiedust/bin/spark/spark-2.=
2.0-bin-hadoop2.7/bin/pyspark',
'--version'</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">> ,</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">> '</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">-bash: /Users/xxxx/pixiedust/bin/spark/spark-2.2.0-bin-hadoop=
2.7/bin/pyspark,
--version</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">,</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">: No such file or directory</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">XXXX:Documents xxxx$ /Users/xxxx/pixiedust/bin/spark/spark-2.=
2.0-bin-hadoop2.7/bin/pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">No Java runtime present, requesting install.</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">XXXX:Documents xxxx$ conda install pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Solving environment: done</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">## Package Plan ##</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0
environment location: /Users/xxxx/anaconda3</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0 added /
updated specs: </span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0 -
pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">The following packages will be downloaded:</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
package=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 build</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
---------------------------|-----------------</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
py4j-0.10.7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 py37_0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 251 =
KB</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
pyspark-2.4.0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 py37_0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 203.4 MB</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
conda-4.6.1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 py37_0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1.7 =
MB</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
------------------------------------------------------------</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
Total:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 205.4 MB</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">The following NEW packages will be INSTALLED:</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
py4j:=C2=A0=C2=A0=C2=A0 0.10.7-py37_0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
pyspark: 2.4.0-py37_0 </span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">The following packages will be UPDATED:</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0
conda:=C2=A0=C2=A0 4.5.12-py37_0 -->
4.6.1-py37_0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Proceed ([y]/n)? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Downloading and Extracting Packages</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">py4j-0.10.7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 | 251 KB=C2=A0=C2=A0=C2=A0 |
#################################################################### | 100%=
</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">pyspark-2.4.0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 203=
.4 MB=C2=A0 |
#################################################################### | 100%=
</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">conda-4.6.1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 | 1.7 MB=C2=A0=C2=A0=C2=A0 |
#################################################################### | 100%=
</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Preparing transaction: done</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Verifying transaction: done</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Executing transaction: done</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">(base) XXXX:Documents xxxx$ jupyter pixiedust
install</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 1: PIXIEDUST_HOME: /Users/xxxx/pixiedust</sp=
an></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"></span>=
</p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spa=
rk</span></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"><=
/span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? n</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: Please enter a SPARK_HOME location: </spa=
n></b><span style=3D"font-size:10pt;font-family:Menlo;color:black">/Users/x=
xxx/spark-2.3.2-bin-hadoop2.7</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Directory /Users/xxxx/sprk-2.3.2-bin-hadoop2.7
does not exist</span></b><span style=3D"font-size:10pt;font-family:Menlo;co=
lor:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Create
y/n [y]? n=C2=A0=C2=A0 </span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spa=
rk</span></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"><=
/span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? n=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: Please enter a SPARK_HOME location: </spa=
n></b><span style=3D"font-size:10pt;font-family:Menlo;color:black">/Users/x=
xxx/Downloads/spark-2.3.4-bin-hadoop2.7</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Directory /Users/xxxx/Downloads/spark-2.3.4-bin-h=
adoop2.7
does not exist</span></b><span style=3D"font-size:10pt;font-family:Menlo;co=
lor:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Create
y/n [y]? n</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spa=
rk</span></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"><=
/span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? n</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: Please enter a SPARK_HOME location: </spa=
n></b><span style=3D"font-size:10pt;font-family:Menlo;color:black">/Users/x=
xxx/Downloads/spark-2.3.4-bin-hadoop2.7/bin/pyspark</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Directory /Users/xxxx/Downloads/spark-2.3.4-bin-h=
adoop2.7/bin/pyspark
does not exist</span></b><span style=3D"font-size:10pt;font-family:Menlo;co=
lor:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Create
y/n [y]? n</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 2: SPARK_HOME: /Users/xxxx/pixiedust/bin/spa=
rk</span></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"><=
/span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Select an existing spark install or create a new
one</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">1. spark-2.3.2-bin-hadoop2.7</span></b><span styl=
e=3D"font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">2. spark-2.2.0-bin-hadoop2.7</span></b><span styl=
e=3D"font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">3. Create a new spark Install</span></b><span sty=
le=3D"font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Enter
your selection: 2</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">downloaded spark cloudant jar: /Users/xxxx/pixiedust/bin/clou=
dant-spark-v2.0.0-185.jar</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 3: SCALA_HOME: /Users/xxxx/pixiedust/bin/sca=
la</span></b><span style=3D"font-size:10pt;font-family:Menlo;color:black"><=
/span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Directory /Users/xxxx/pixiedust/bin/scala does
not contain a valid scala install</span></b><span style=3D"font-size:10pt;f=
ont-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Download
Scala y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">SCALA_HOME will be set to /Users/xxxx/pixiedust/bin/scala/sca=
la-2.11.8</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Downloading Scala 2.11</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Extracting Scala 2.11 to /Users/xxxx/pixiedust/bin/scala</spa=
n></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><b><span style=3D"font-size:10pt;font-family:Me=
nlo;color:rgb(47,180,29)">Step 4: Kernel Name:
Python-with-Pixiedust_Spark-2.2</span></b><span style=3D"font-size:10pt;fon=
t-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0 Keep
y/n [y]? y</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">self.kernelInternalName
pythonwithpixiedustspark22</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">[PixiedustInstall] Installed kernelspec
pythonwithpixiedustspark22 in /Users/xxxx/Library/Jupyter/kernels/pythonwit=
hpixiedustspark22</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">Downloading intro notebooks into /Users/xxxx/pixiedust/notebo=
oks</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">...<a href=3D"https://github.com/ibm-watson-data-lab/pixiedus=
t/raw/master/notebook/PixieDust">https://github.com/ibm-watson-data-lab/pix=
iedust/raw/master/notebook/PixieDust</a>
1 - Easy Visualizations.ipynb : </span><b><span style=3D"font-size:10pt;fon=
t-family:Menlo;color:rgb(47,180,29)">done</span></b><span style=3D"font-siz=
e:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">...<a href=3D"https://github.com/ibm-watson-data-lab/pixiedus=
t/raw/master/notebook/PixieDust">https://github.com/ibm-watson-data-lab/pix=
iedust/raw/master/notebook/PixieDust</a>
2 - Working with External Data.ipynb : </span><b><span style=3D"font-size:1=
0pt;font-family:Menlo;color:rgb(47,180,29)">done</span></b><span style=3D"f=
ont-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">...<a href=3D"https://github.com/ibm-watson-data-lab/pixiedus=
t/raw/master/notebook/PixieDust">https://github.com/ibm-watson-data-lab/pix=
iedust/raw/master/notebook/PixieDust</a>
3 - Scala and Python.ipynb : </span><b><span style=3D"font-size:10pt;font-f=
amily:Menlo;color:rgb(47,180,29)">done</span></b><span style=3D"font-size:1=
0pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">...<a href=3D"https://github.com/ibm-watson-data-lab/pixiedus=
t/raw/master/notebook/PixieDust">https://github.com/ibm-watson-data-lab/pix=
iedust/raw/master/notebook/PixieDust</a>
4 - Add External Spark Packages.ipynb : </span><b><span style=3D"font-size:=
10pt;font-family:Menlo;color:rgb(47,180,29)">done</span></b><span style=3D"=
font-size:10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">...<a href=3D"https://github.com/ibm-watson-data-lab/pixiedus=
t/raw/master/notebook/PixieDust">https://github.com/ibm-watson-data-lab/pix=
iedust/raw/master/notebook/PixieDust</a>
5 - Stash to Cloudant.ipynb : </span><b><span style=3D"font-size:10pt;font-=
family:Menlo;color:rgb(47,180,29)">done</span></b><span style=3D"font-size:=
10pt;font-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">...<a href=3D"https://github.com/ibm-watson-data-lab/pixiedus=
t/raw/master/notebook/PixieDust">https://github.com/ibm-watson-data-lab/pix=
iedust/raw/master/notebook/PixieDust</a>
Contribute.ipynb : </span><b><span style=3D"font-size:10pt;font-family:Menl=
o;color:rgb(47,180,29)">done</span></b><span style=3D"font-size:10pt;font-f=
amily:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">=C2=A0</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">#############################################################=
#######################################</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">#=C2=A0=C2=A0=C2=A0 Congratulations:
Kernel Python-with-Pixiedust_Spark-2.2 was successfully created in /Users/x=
xxx/Library/Jupyter/kernels/pythonwithpixiedustspark22</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">#=C2=A0=C2=A0=C2=A0 You
can start the Notebook server with the following command:</span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">#=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 </span><b><span s=
tyle=3D"font-size:10pt;font-family:Menlo;color:rgb(47,180,29)">jupyter note=
book
/Users/xxxx/pixiedust/notebooks</span></b><span style=3D"font-size:10pt;fon=
t-family:Menlo;color:black"></span></p>
<p class=3D"MsoNormal" style=3D"margin:0in 0in 0.0001pt;font-size:12pt;font=
-family:Calibri,sans-serif"><span style=3D"font-size:10pt;font-family:Menlo=
;color:black">#############################################################=
#######################################</span><span style=3D"font-size:10pt=
"></span></p></div><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr"=
>On Sun, Jan 27, 2019, 11:55 PM The Dude <<a href=3D"mailto:ToTheDude@zo=
ho.com" target=3D"_blank">[email protected]</a>> wrote:<br></div><block=
quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc=
solid;padding-left:1ex"><div style=3D"word-wrap:break-word"><div>Hi Lisa,<=
/div><div><span class=3D"m_-176293331599322073m_-8450214356169915485Apple-t=
ab-span" style=3D"white-space:pre-wrap"> </span>thanks for your help.=C2=A0=
</div><div><br></div><div>Unfortunately, I get the same error after install=
ing spark with conda.=C2=A0</div><div>Here=E2=80=99s the output from my ter=
minal:</div><div><br></div><div><br></div><div></div></div><div style=3D"wo=
rd-wrap:break-word"><div><div>(py37ana) adula-5:~ dude$ jupyter pixiedust i=
nstall</div><div>Step 1: PIXIEDUST_HOME: /Users/dude/pixiedust</div><div><s=
pan class=3D"m_-176293331599322073m_-8450214356169915485Apple-tab-span" sty=
le=3D"white-space:pre-wrap"> </span>Keep y/n [y]? y</div><div>Step 2: SPARK=
_HOME: /Users/dude/pixiedust/bin/spark</div><div><span class=3D"m_-17629333=
1599322073m_-8450214356169915485Apple-tab-span" style=3D"white-space:pre-wr=
ap"> </span>Keep y/n [y]? y</div><div>Select an existing spark install or c=
reate a new one</div><div>1. spark-2.2.0-bin-hadoop2.7</div><div>2. Create =
a new spark Install</div><div><span class=3D"m_-176293331599322073m_-845021=
4356169915485Apple-tab-span" style=3D"white-space:pre-wrap"> </span>Enter y=
our selection: 1</div><div>Traceback (most recent call last):</div><div>=C2=
=A0 File "/Users/dude/anaconda/envs/py37ana/bin/jupyter-pixiedust"=
;, line 11, in <module></div><div>=C2=A0 =C2=A0 sys.exit(main())</div=
><div>=C2=A0 File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/sit=
e-packages/install/pixiedustapp.py", line 41, in main</div><div>=C2=A0=
=C2=A0 PixiedustJupyterApp.launch_instance()</div><div>=C2=A0 File "/=
Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/conf=
ig/application.py", line 657, in launch_instance</div><div>=C2=A0 =C2=
=A0 app.initialize(argv)</div><div>=C2=A0 File "<decorator-gen-2>=
;", line 2, in initialize</div><div>=C2=A0 File "/Users/dude/anac=
onda/envs/py37ana/lib/python3.7/site-packages/traitlets/config/application.=
py", line 87, in catch_config_error</div><div>=C2=A0 =C2=A0 return met=
hod(app, *args, **kwargs)</div><div>=C2=A0 File "/Users/dude/anaconda/=
envs/py37ana/lib/python3.7/site-packages/traitlets/config/application.py&qu=
ot;, line 296, in initialize</div><div>=C2=A0 =C2=A0 self.parse_command_lin=
e(argv)</div><div>=C2=A0 File "<decorator-gen-4>", line 2, =
in parse_command_line</div><div>=C2=A0 File "/Users/dude/anaconda/envs=
/py37ana/lib/python3.7/site-packages/traitlets/config/application.py",=
line 87, in catch_config_error</div><div>=C2=A0 =C2=A0 return method(app, =
*args, **kwargs)</div><div>=C2=A0 File "/Users/dude/anaconda/envs/py37=
ana/lib/python3.7/site-packages/traitlets/config/application.py", line=
514, in parse_command_line</div><div>=C2=A0 =C2=A0 return self.initialize_=
subcommand(subc, subargv)</div><div>=C2=A0 File "<decorator-gen-3&g=
t;", line 2, in initialize_subcommand</div><div>=C2=A0 File "/Use=
rs/dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/config/=
application.py", line 87, in catch_config_error</div><div>=C2=A0 =C2=
=A0 return method(app, *args, **kwargs)</div><div>=C2=A0 File "/Users/=
dude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/config/app=
lication.py", line 452, in initialize_subcommand</div><div>=C2=A0 =C2=
=A0 self.subapp.initialize(argv)</div><div>=C2=A0 File "<decorator-=
gen-6>", line 2, in initialize</div><div>=C2=A0 File "/Users/d=
ude/anaconda/envs/py37ana/lib/python3.7/site-packages/traitlets/config/appl=
ication.py", line 87, in catch_config_error</div><div>=C2=A0 =C2=A0 re=
turn method(app, *args, **kwargs)</div><div>=C2=A0 File "/Users/dude/a=
naconda/envs/py37ana/lib/python3.7/site-packages/jupyter_core/application.p=
y", line 238, in initialize</div><div>=C2=A0 =C2=A0 self.parse_command=
_line(argv)</div><div>=C2=A0 File "/Users/dude/anaconda/envs/py37ana/l=
ib/python3.7/site-packages/install/createKernel.py", line 154, in pars=
e_command_line</div><div>=C2=A0 =C2=A0 spark_version =3D self.get_spark_ver=
sion()</div><div>=C2=A0 File "/Users/dude/anaconda/envs/py37ana/lib/py=
thon3.7/site-packages/install/createKernel.py", line 379, in get_spark=
_version</div><div>=C2=A0 =C2=A0 pyspark_out =3D subprocess.check_output([p=
yspark, "--version"], stderr=3Dsubprocess.STDOUT).decode("ut=
f-8")</div><div>=C2=A0 File "/Users/dude/anaconda/envs/py37ana/li=
b/python3.7/subprocess.py", line 389, in check_output</div><div>=C2=A0=
=C2=A0 **kwargs).stdout</div><div>=C2=A0 File "/Users/dude/anaconda/e=
nvs/py37ana/lib/python3.7/subprocess.py", line 481, in run</div><div>=
=C2=A0 =C2=A0 output=3Dstdout, stderr=3Dstderr)</div></div></div><div style=
=3D"word-wrap:break-word"><div><div>subprocess.CalledProcessError: Command =
'['/Users/dude/pixiedust/bin/spark/spark-2.2.0-bin-hadoop2.7/bin/py=
spark', '--version']' returned non-zero exit status 1.dude<=
/div></div><div><br></div><div><br></div><div><br></div><div>It seems that =
the pyspark executable doesn=E2=80=99t run properly.=C2=A0</div><div><br></=
div><div>TheDude</div></div><div style=3D"word-wrap:break-word"><div><br></=
div><div><br></div><div><br></div><div><br></div><br><div><blockquote type=
=3D"cite"><div>On Jan 27, 2019, at 11:26:25, Lisa Bang <<a href=3D"mailt=
o:[email protected]" target=3D"_blank">[email protected]</a>> wrote:=
</div><br class=3D"m_-176293331599322073m_-8450214356169915485Apple-interch=
ange-newline"><div><div dir=3D"ltr"><div dir=3D"ltr"><br clear=3D"all"><div=
><div dir=3D"ltr" class=3D"m_-176293331599322073m_-8450214356169915485gmail=
_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div style=
=3D"font-size:small">Hi,</div><div style=3D"font-size:small"><br></div><div=
style=3D"font-size:small">Looks like you don't have pyspark installed.=
=C2=A0 I reproduced your error and after running "<span style=3D"font-=
variant-ligatures:no-common-ligatures;font-family:Menlo;font-size:11px">con=
da install pyspark</span>" ,=C2=A0 "<span style=3D"font-family:Me=
nlo;font-size:11px;font-variant-ligatures:no-common-ligatures">jupyter pixi=
edust install</span>" finished just fine.</div>
<div style=3D"font-size:small"><br></div><div style=3D"font-size:small">Bes=
t,</div><div style=3D"font-size:small">Lisa</div></div></div></div><div><di=
v><br></div></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" cla=
ss=3D"m_-176293331599322073m_-8450214356169915485gmail_attr">On Sat, Jan 26=
, 2019 at 11:22 PM TheDude <<a href=3D"mailto:[email protected]" target=
=3D"_blank">[email protected]</a>> wrote:<br></div><blockquote class=3D=
"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(2=
04,204,204);padding-left:1ex">Hello,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 I would really like to have the pixiedebugger w=
orking in my notebook. Unfortunately I ran into problems with the installat=
ion.<br>
<br>
I am following the instructions here: <<a href=3D"https://pixiedust.gith=
ub.io/pixiedust/install.html" rel=3D"noreferrer" target=3D"_blank">https://=
pixiedust.github.io/pixiedust/install.html</a>><br>
For what is worth, I am using Python 3.7 in Anaconda on macOS 10.12.<br>
<br>
The first part of the installation:<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 pip install pixiedust<br>
<br>
is successful, and it ends with:<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Successfully built pixiedust mpld3<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Installing collected packages: mpld3, geojson, =
astunparse, markdown, colour, pixiedust<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Successfully installed astunparse-1.6.2 colour-=
0.1.5 geojson-2.4.1 markdown-3.0.1 mpld3-0.3 pixiedust-1.1.15<br>
<br>
The next step, installing a new Jupyter kernel, fails:<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 jupyter pixiedust install<br>
<br>
Here=E2=80=99s what I get:<br>
<br>
(py37ana) adula-5:~ dude$ jupyter pixiedust install<br>
Step 1: PIXIEDUST_HOME: /Users/dude/pixiedust<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Keep y/n [y]? y<br>
Step 2: SPARK_HOME: /Users/dude/pixiedust/bin/spark<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Keep y/n [y]? y<br>
Select an existing spark install or create a new one<br>
1. spark-2.2.0-bin-hadoop2.7<br>
2. Create a new spark Install<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Enter your selection: 1<br>
Traceback (most recent call last):<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/bin/jupyter-pixiedust&qu=
ot;, line 11, in <module><br>
=C2=A0 =C2=A0sys.exit(main())<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/install/pixiedustapp.py", line 41, in main<br>
=C2=A0 =C2=A0PixiedustJupyterApp.launch_instance()<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 657, in launch_instance<br>
=C2=A0 =C2=A0app.initialize(argv)<br>
=C2=A0File "<decorator-gen-2>", line 2, in initialize<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 87, in catch_config_error<b=
r>
=C2=A0 =C2=A0return method(app, *args, **kwargs)<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 296, in initialize<br>
=C2=A0 =C2=A0self.parse_command_line(argv)<br>
=C2=A0File "<decorator-gen-4>", line 2, in parse_command_li=
ne<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 87, in catch_config_error<b=
r>
=C2=A0 =C2=A0return method(app, *args, **kwargs)<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 514, in parse_command_line<=
br>
=C2=A0 =C2=A0return self.initialize_subcommand(subc, subargv)<br>
=C2=A0File "<decorator-gen-3>", line 2, in initialize_subco=
mmand<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 87, in catch_config_error<b=
r>
=C2=A0 =C2=A0return method(app, *args, **kwargs)<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 452, in initialize_subcomma=
nd<br>
=C2=A0 =C2=A0self.subapp.initialize(argv)<br>
=C2=A0File "<decorator-gen-6>", line 2, in initialize<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/traitlets/config/application.py", line 87, in catch_config_error<b=
r>
=C2=A0 =C2=A0return method(app, *args, **kwargs)<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/jupyter_core/application.py", line 238, in initialize<br>
=C2=A0 =C2=A0self.parse_command_line(argv)<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/install/createKernel.py", line 154, in parse_command_line<br>
=C2=A0 =C2=A0spark_version =3D self.get_spark_version()<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/site-packa=
ges/install/createKernel.py", line 379, in get_spark_version<br>
=C2=A0 =C2=A0pyspark_out =3D subprocess.check_output([pyspark, "--vers=
ion"], stderr=3Dsubprocess.STDOUT).decode("utf-8")<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/subprocess=
.py", line 389, in check_output<br>
=C2=A0 =C2=A0**kwargs).stdout<br>
=C2=A0File "/Users/dude/anaconda/envs/py37ana/lib/python3.7/subprocess=
.py", line 481, in run<br>
=C2=A0 =C2=A0output=3Dstdout, stderr=3Dstderr)<br>
subprocess.CalledProcessError: Command '['/Users/dude/pixiedust/bin=
/spark/spark-2.2.0-bin-hadoop2.7/bin/pyspark', '--version']'=
; returned non-zero exit status 1.<br>
<br>
<br>
Anybody has an idea on how to proceed?<br>
<br>
TIA.<br>
<br>
TheDude<br>
<br>
<br>
_______________________________________________<br>
IPython-dev mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">IPython-dev@pyt=
hon.org</a><br>
<a href=3D"https://mail.python.org/mailman/listinfo/ipython-dev" rel=3D"nor=
eferrer" target=3D"_blank">https://mail.python.org/mailman/listinfo/ipython=
-dev</a><br>
</blockquote></div></div>
</div></blockquote></div><br></div></blockquote></div></div>
</div>
--000000000000b4d13e0580860f5a--
--===============7963421091259887754==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
IPython-dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/ipython-dev
--===============7963421091259887754==--