Seeing python’s logging messages in a Jupyter Notebook

Originally answered in stackoverflow.

stderr is the default stream for the logging module, so in IPython and Jupyter notebooks you might not see anything unless you configure the stream to stdout:

import logging
import sys

logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s',
                     level=logging.INFO, stream=sys.stdout)

logging.info('Hello world!')

Truncating pandas float display

# changes default formatting of float to 3 decimal places
pd.options.display.float_format = '{:.3f}'.format

Changing Python’s path

It is possible to change the path Python uses to search for files so that you can import modules that otherwise would only be possible by having your notebook file in the same directory. This is useful when you want to have a specific folder for your notebooks in order to be more organized.

To do that, run the following code:

import sys
sys.path.insert(0, '/path/to/application/app/folder')

For more details see this answer.

Fix installed packages not being imported

Sometimes, when using conda, you may find that you’re not able to import installed packages into your notebook. One of the reasons may be because Jupyter is not using the correct kernel.

To find out what kernel is being used, run (within the notebook):

import sys
sys.path
sys.executable

If you find that this is the problem, a simple solution is simply installing jupyter within the conda environment you are using:

source activate [yourenvname]
conda install jupyter

Other alternative solutions to this problem can be found here.

Using git to version control jupyter notebooks

This post explains in a very nice way different ways we can combine jupyter notebook with git tracking only changes in code.

Tracking time

Jupyter and IPython make it really easy to track time in the cells you execute:

%time print('%time will output the time of running only this line')

%%time
print('Jupyter can also track')
print('the time that running an entire cell took')
print('when you use %%time')

For more accurate results you may also want to use timeit instead of time, which Jupyter also allows for. See this answer for more details on the difference between the two.

Saving the current notebook from code

It’s a bit of a hack, but you can save the current notebook you are running the code from using the following lines:

from IPython.display import display, Javascript
display(Javascript('IPython.notebook.save_checkpoint();'))

Converting a notebook from the command line

The command below will convert the notebook to html. It’s possible to define other formats as well (check the documentation).

%%bash
jupyter nbconvert --to html notebook_name.ipynb

Exporting history

It’s possible to export the history (each command that was issued) of your current notebook by using the %notebook magic:

%notebook history.ipynb

Running bash commands

To run a bash command, the magic %%bash at the first line of a cell will do it for you:

%%bash
mv history.ipynb ./results/

If you want to access a python variable within the script cell, use (see this answer):

%%bash -s "$myPythonVar" "$myOtherVar"
echo "This bash script knows about $1 and $2"

It is also possible to use the subprocess module to run bash commands from python:

bash_command = "mv history.ipynb {}/".format(export_path)
import subprocess
process = subprocess.Popen(bash_command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()

Remote access to another machine

Start the server on the remote machine:

$ jupyter notebook --no-browser --port=8889

Open an SSH tunnel (port forwarding) from your local machine to the remote one:

$ ssh -N -L localhost:8888:localhost:8889 remote_user@remote_host

You’ll then be able to access the remote kernel in your local browser by going to http://localhost:8888.

Setting up a password

Instead of a token, it’s possible to set up a default password that jupyter will use. Follow these instructions.

Module autoreload

These configurations will auto-reload modules (changes in the source code will not require manual reloading—see stackoverflow):

%load_ext autoreload
%autoreload 2

Other references

Other posts I found helpful that cover jupyter notebook hints: