Add third-party python libraries


#1

How do you install or pre-configure Seahorse to have third-party python libraries available in the Python notebooks?


#2

In Spark, to install a python multi-file module, you have to provide it in .zip or .egg format.

For some python modules, it can require some work.

For example for nltk, you can download the code from here:

https://pypi.python.org/pypi/nltk

Then, you have to unpack it, and zip the directory nltk inside the unpacked nltk-3.2.1.

For example, in linux:

wget https://pypi.python.org/packages/58/85/8fa6f8c488507aab7d6234ce754bbbe61bfeb8382489785e2d764bf8f52a/nltk-3.2.1.tar.gz#md5=27d84d2d11449d709ccd1deac18deb05
tar xzf nltk-3.2.1.tar.gz
cd nltk-3.2.1
zip -r …/nltk.zip nltk

Now, to import such a module in Seahorse, you have to put the module in the directory visible by Seahorse.

This is the data directory that is in the same directory that contains your docker-compose.yml (or Vagrantfile).

Now, to use this module in Seahorse Notebook, you can execute:

sc.addPyFile(’/resources/data/nltk.zip’)
and then:

import nltk
The same can be done in python transformation.

Alternatively, if you don’t want to perform this every time, you can add following line to your cluster preset, in the custom settings section:

–conf spark.submit.pyFiles=/resources/data/nltk.zip

I hope that these instructions are clear and that this will work for you!


#3

Fantastic details! Very helpful. The last part with spark.submit.pyFiles via the data directory was what I was looking for.