arthurcgusmao.com

Configure SSH Keys Authentication

2020-03-04T00:00:00+00:00

Secure Shell (SSH) is a cryptographic network protocol, i.e., a set of network rules that use cryptography for operating network services securely. The typical application is using SSH to access remote machines and perform command-line, login, and remote command execution, but any network service can be secured with SSH. [1]

The freely available version of SSH present in most Linux environments is called OpenSSH. [2]

The `~/.ssh` directory

OpenSSH stores its configurations on the .ssh directory, present under the user’s root directory. The .ssh directory is automatically created when the user runs the ssh command for the first time. If it doesn’t exist in your computer yet, you may created it using the command below (notice the 700 permission for security). [3]

mkdir -p ~/.ssh && chmod 700 ~/.ssh

SSH Keys configuration

SSH Keys allow authentication without the need of a password [2]. It uses two keys, one private and one public. The public key has the .pub extension as its filetype. The name of these keys is usually id_rsa and id_rsa.pub, and they are usually placed in the ~/.ssh directory.

To generate the keys using the RSA algorithm, run:

ssh-keygen -t rsa

At this step you will be prompted for a password; you may choose to leave it empty, but be aware that whoever gains access to your private key file will be able to login to your remote machines. [5]

In order to SSH into another machine without typing a password, the remote machine must have a copy of your client’s public key under its ~/.ssh/authorized_keys file. You can either manually append to that file (if you need to manually create the file, make sure it has permissions “600” for security) or use the utility command ssh-copy-id:

ssh-copy-id username@remotehost

The built-in `ssh-agent` utility

By default, you will have to enter your private key passphrase every time you use it. However, you can avoid to repeatedly do this by running an SSH agent (ssh-agent -s), a small utility that stores your private key after you have entered the passphrase for the first time [5]. After your SSH-agent is running, add your key via ssh-add, and type your passphrase. Now, every time you need to use your private key, no passphrase will be prompted.

The smarter `ssh-ident` utility

Ideally, your SSH-agent would automatically be run at startup, so that you don’t have to manually start it and add the keys yourself. However, this approach has the inconvinience that you will need to type your passphrase for all keys even if not using them throughout your session. To solve this issue, a number of solutions have been proposed, one of them being ssh-ident, which creates SSH agents on demand as your SSH keys are first needed. Read the repository’s documentation for more information.

OpenSSH client-side configuration

Configuration on the client side is done via the ~/.ssh/config file, which should be readable and writable only by the user (permission 600).

touch ~/.ssh/config && chmod 600 ~/.ssh/config

The structure of the file defines, for each block, an alias for host you want to access, followed by the actual hostname and additional configuration. Below you find an example configuration, adapted from [3].

Host myalias
    HostName 192.168.1.10
    User myuser
    Port 3022
    IdentityFile ~/.ssh/targaryen.key
    LocalForward 31086 14.120.240.50:31086

The example above can be edited to keep only the lines useful for your configuration. It is possible to use wildcards like Host * in your config file also. [3,4]

Prevent SSH connection time outs

An example of wildcards use in your client’s SSH configuration is when you want to make sure the SSH connections are not timed out by firewalls or related. To do so, you can add to your client’s ~/.ssh/config file (create if it doesn’t exist) the snippet below:

Host *
    ServerAliveInterval 60
    ServerAliveCountMax 2

For more details see references [6] and [7].

Further notes

If you are using a web services provider, be aware that it may employ meta-mechanisms to control SSH authentication in the remote machine. For instance, Google Compute actively modifies/deletes the ~/.ssh/authorized_keys file of virtual machines for that purpose; changes you make to that file may be lost over time.

References

https://en.wikipedia.org/wiki/Secure_Shell
https://help.ubuntu.com/lts/serverguide/openssh-server.html
https://linuxize.com/post/using-the-ssh-config-file
https://superuser.com/a/1462597/629663
https://www.digitalocean.com/community/tutorials/ssh-essentials-working-with-ssh-servers-clients-and-keys
https://patrickmn.com/aside/how-to-keep-alive-ssh-sessions/
https://unix.stackexchange.com/questions/3026/what-options-serveraliveinterval-and-clientaliveinterval-in-sshd-config-exac

Matplotlib intro (pyplot)

2018-04-06T00:00:00+00:00

A very simple introduction to Matplotlib in Python. It tries to present fundamental concepts without clinging to many details. If you have time I highly recommend you another post which is much more conceptual, detailed and illustrated than mine: “Artist” in matplotlib.

import matplotlib.pyplot as plt

Basics: Figures and Axes

The most important thing that you have to know is that Matplotlib has two main classes for plotting curves and setting up configurations: Figure and Axes. In this section we present these objects and its main methods.

Figure

The Figure object is a major object that contains all of the plot elements (including the Axes object, which we will see next). It can be accessed in the following manners:

fig = plt.figure() # returns a Figure instance

The last line is a shortcut for getting the current figure and its axes; see these answers for more details.

Configurations such as saving and displaying the figure are set at the figure-level:

fig.savefig('filepath.png') # saves figure
fig.show() # displays figure

Axes

Axes are the objects that have plotting methods. Each figure can contain multiple axes, thus allowing you to make multiple plots on the same figure. They are created and accessed from the figure object:

# create new axes
ax1 = fig.subplots() # add a set of subplots to this figure (default 1)

# get existing axes
axs = fig.axes # list of axes in `fig`

Once you have the axis object that you want to plot a curve into, the only thing you need to do is to call the plotting method:

ax1.plot(x, y)

There are many different kinds of plotting methods (which we will not cover), here are a few:

ax.plot Lines and/or Markers
ax.bar Bars chart
ax.hist Histogram
plt.scatter Scatter (has capability to render a different size and/or color for each point, in contrast to plot)

Shortcuts: the Pyplot interface

To make things easier, Matplotlib has shortcuts for the methods mentioned above directly through pyplot. That is, lots of methods can be called from pyplot, regardless of whether they would normally be called from a Figure or an Axis object:

import matplotlib.pyplot as plt

# Example 1
plt.plot([1,2,3])
plt.savefig('filepath.png')
plt.show()

# Example 2
fig, ax1 = plt.subplots() # get figure and 1 axes (already added to fig)
...

Axis labels, Legends, and Title

Legend, title and axis labels should be set in the axis object:

# TITLE AND AXIS LABELS
ax.set(title='title', xlabel='xlabel', ylabel='ylabel')
# or
ax.set_title('title')
ax.set_xlabel('xlabel')
ax.set_ylabel('ylabel')

# LEGEND
# add labels to curves
ax.plot(xa, ya, label='Algorithm A')
ax.plot(xb, yb, label='Algorithm B')

# call legend function
ax.legend()

Of course, Matplotlib has shortcuts for these methods directly through pyplot as well, in the case you are dealing with a single plot:

# shortcuts
plt.title('title')
plt.xlabel('xlabel')
plt.ylabel('ylabel')
plt.legend()

Subplots

In Matplotlib, the subplot functionality allows you to specify where the axes should be situated (in the figure) according to a subplot grid.

ax1 = fig.add_subplot(221) # creates an axis in the position specified (row-col-num)
ax2 = fig.add_subplot(222) # creates an axis in the position specified (row-col-num)
ax3 = fig.add_subplot(223) # creates an axis in the position specified (row-col-num)
ax4 = fig.add_subplot(224) # creates an axis in the position specified (row-col-num)
# or
[[ax1, ax2], [ax3, ax4]] = fig.subplots(nrows=2, ncols=2) # add a set of subplots to this figure
# or
fig, [[ax1, ax2], [ax3, ax4]] = plt.subplots(nrows=2, ncols=2) # shortcut for getting both figure and (all) axes

After getting the axes, you can call the plotting method of your choice for each one.

Breaking out of the grid

In addition, these functions do not impose a fixed grid, they simply create new axes that occupy specific positions. Because of that, you can create subplots with varying widths and heights in the same figure.

An option for a very flexible plotting scheme is the following: specify a starting position (x0,y0) and the size of your new axis (width,height) by passing the quadruple (x0,y0,width,height) to add_subplot via the position keyword argument:

ax1 = fig.add_subplot(position=(x0,y0,width,height))

Alternatively, you can use the add_axes method, whose calling signature is the same quadruple. Its differences to add_subplot method are well discussed here.

Decoration

Usually configuration of the curves, bars, histograms, etc. are done in the respective plotting function:

plt.plot(x, y, color='green', marker='o', linestyle='dashed',
         linewidth=2, markersize=12)

Check out the docs for more details:

Specific tips

The parameter alpha is usually the transparency of the curve.
Use plt.tight_layout() when your subplots have titles

Varying colors w/ custom colormap

Say you want to plot a set of lines with their colors varying according to some criteria. Matplotlib define a set of colormaps via matplotlib.cm, that returns a color from the colormap given your input (which can be in the range [0,1] or [0, 255]):

import matplotlib.pyplot as plt
from matplotlib import cm

ys = ([1,2,3], [2,3,4], [3,4,5], [4,5,6])

for i,y in enumerate(ys):
    plt.plot(y, color=cm.summer(i/len(ys)))

Other resources

DataCamp Matplotlib Cheatsheet

Examples

Highlighting specific data points in a scatter plot

Color warmth in scatter plot

True vs predicted data points

Statistics basic concepts

2018-03-10T00:00:00+00:00

This is my personal cheatsheet and tutorial for statistics. It is far from being a complete introduction, the main goal is to have a centralized resource where one can quickly remember or access more detailed resources. Many paragraphs are excerpts from the linked Wikipedia articles in the associated section. $\def\mean#1{\bar #1} \DeclareMathOperator{\EV}{\mathbb{E}} \DeclareMathOperator{\Prob}{\mathbb{P}}$

Concepts of Research Methods

Construcs

Something hard to measure, that you’ll have to define how to measure.

Operational Definition

It’s how you will measure a given construct.

Sample

A sample is a (sub)set of elements from a population of interest.

When it is not feasible to directly measure the value of a population parameter (e.g., unmanageable population size), we can infer its likely value using a sample (a subset of manageable size).

Note: make sure to distinguish between the terms sample size and number of samples. Use the former to describe the number of elements in a sample, and the latter to refer to the number of different samples (each constituted of many elements), if that is the case.

Parameter vs Statistic

A parameter refers to a quantity the defines the actual, entire population (the original distribution) of interest. In contrast, a statistic refers to a quantity computed using values in a sample (part of the population).

A statistic can be used to estimate a population parameter, to describe a sample, or to evaluate a hypothesis. When used to estimate a population parameter, it is called an estimator.

Relationships vs Causation

Relationships can be shown by observational studies (eg. by seeing a point distribution). Causation, however, has to be found through controlled experiments.

Measures of central tendency

Mode

The value at which frequency is highest.

The mode doesn’t really represent the data well because:

It’s not affected by all scores in the dataset;
It changes radically depending on the sample;
We can’t really describe the mode by an equation.

Median

The value in the middle of the distribution.

(If there are an even number of elements, the median will be the average of the two in the middle.)

The median is more robust to outliers than the mean. It is therefore the best measure of central tendency when dealing with highly skewed distributions.

Mean (expected value)

The weighted average, or expected value.

Population mean: $\mu$
Sample mean: $\mean{x}$ (can be used to estimate $\mu$)
The mean is sensitive to outliers
The law of large numbers states that as a sample size grows, its mean gets closer to the average of the whole population.

The mean is always pulled towards the longest tail of the distribution, in relation to the median.

Variability of data

Range

Range = MaxValue - MinValue

Interquartile Range (IQR)

IQR is the distance between the 25% and the 75% higher measure of the data.

IQR = Q3 - Q1

Problem with Range and IQR: neither takes all data into account.

Outlier

An observation point that is distant from other observations.

Values below (Q1 - 1.5*IQR) or above (Q3 + 1.5*IQR) are usually considered outliers.

Outliers are represented as dots in a boxplot.

Estimators, Bias and Variance

(from a machine learning point of view)

Point Estimation

Point estimation is the attempt to provide the single “best” prediction of some quantity of interest. A point estimator or statistic is any function of the data:

\[\boldsymbol{\hat \theta}_m = g(\boldsymbol{x}^{(1)},...,\boldsymbol{x}^{(m)}).\]

The definition does not require that $g$ return a value that is close to the true $\boldsymbol{\theta}$ or even that the range of $g$ is the same as the set of allowable values of $\boldsymbol{\theta}$.

Bias

Bias measures the expected deviation from the true value of the function or parameter. The bias of an estimator is defined as:

\[\text{bias}(\boldsymbol{\hat\theta}_m) = \EV(\boldsymbol{\hat\theta}_m) - \boldsymbol{\theta}\]

where the expectation is over the data (seen as samples from a random variable).

Variance and Standard Deviation

Variance provides a measure of the deviation from the expected estimator value that any particular sampling of the data is likely to cause.

We call deviation a measure of difference between the observed value of a variable and some other value, often that variable’s mean. We calculate the variance by taking the mean of squared deviations:

\[\sigma^2 = \sum_{i=1}^{n} \frac{(x_i - \mean{x})^2}{n}\]

The standard deviation is simply the root square of the variance. One interesting characteristic of the standard deviation is the 68–95–99.7 rule.

Bessel’s Correction (unbiased estimator of the population variance)

You use $(n - 1)$ instead of $(n)$ to calculate the standard deviation when estimating the standard deviation for the population from a sample.

\[\sigma \approx s = \sqrt{\sum_{i=1}^{n} \frac{(x_i - \mean{x})^2}{n - 1}}\]

Using the sample standard deviation (s) corrects the bias in the estimation of the population variance. It also partially corrects the bias in the estimation of the population standard deviation. However, the correction often increases the mean squared error in these estimations.

These [1, 2] short videos try to bring some intuition on why to use $(n - 1)$ instead of $(n)$.

The problem is that, by using the estimate $(\mean{x})$ instead of the actual population mean $(\mu)$, we create bias in the estimated variance by underestimating it, because $(\mean{x})$ is the value that minimizes $\sum_{i=1}(x_i - \mean{x})^2$, whereas $\mu$ could be any other value that makes it larger.

Using $n-1$ in $s^2$ to estimate $\sigma^2$ fixes this bias, making $s^2$, on average, equals $\sigma^2$ (unbiased estimator).

Z-Score

The Z-Score (aka standard score) maps a sample $x_i$ to how many standard deviations $\sigma$ it is from the mean $\mu$:

\[z_i = \frac{x_i - \mu}{\sigma} \ \ \ \implies \ \ \ x_i = \mu + z_i \cdot \sigma\]

By its definition, the z-score is a dimensionless quantity. It can mapped directly to a percentile when you know the shape of the distribution (or, in case you do not, to weaker bounds using Chebyshev’s inequality).

The Bias vs Variance Tradeoff

How to choose between two estimators: one with more bias and another with more variance? The most common way to negotiate this trade-off is to use cross-validation. Alternatively, we can also compare the mean squared error (MSE) of the estimates:

\[\begin{align} \text{MSE} & = \EV \left[ \left( \hat\theta_m - \theta \right)^2 \right] \\ & = \text{Bias} \left( \hat\theta_m \right)^2 + \text{Var} \left( \hat\theta_m \right)^2 \end{align}\]

The relationship between bias and variance is tightly linked to the machine learning concepts of capacity, underfitting and overfitting.

Sampling distributions

The sampling distribution is the probability distribution of the values a statistic (e.g., sample mean or sample variance) can take on.

Standard Error

The standard error of a statistic corresponds to the standard deviation of its sampling distribution. If the parameter or the statistic is the mean, it is called the standard error of the mean (SEM).

Depiction of distribution sampling, taken from [2].

Standard Error of the Mean (SEM) $\sigma_{\bar{x}}$

The SEM quantifies the precision of the mean, i.e., it measures how far the sample mean is likely to be from the true population mean. In other words, the SEM is the standard deviation of the distribution that defines the values of the sample mean we will observe, for each sample of size $n$ we take from the population.

Since $\sigma$ is seldom known (to calculate it we’d have to make lots of experiments, finding many different values for the mean, and calculating its standard deviation), the SEM is usually estimated by using the sample standard deviation:

\[\text{SEM} = \sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}} \approx \frac{s}{\sqrt{n}}\]

The proof of the relationship between the SEM, the standard deviation of the population $\sigma$, and the sample size $n$ is shown below (reference). The first step is allowed because the $X_i$ are independently sampled, so the variance of the sum is just the sum of the variances.

\[\text{Var}\bigg(\frac{\sum_{i=1}^{n}{X_i}}{n}\bigg) = \frac{1}{n^2}\text{Var}\bigg(\sum_{i=1}^{n}{X_i}\bigg) = \frac{1}{n^2}\sum_{i=1}^{n}\text{Var}\bigg({X_i}\bigg) = \frac{1}{n^2}\sum_{i=1}^{n}{\sigma^2} = \frac{\sigma^2}{n}\]

We can use the SEM to interpret different populations and see if there is evidence that there are differences or similarities between them:

How the standard error of the mean helps us interpret populations, from this YouTube video.

We can also use the relationship between the SEM and the sample size $n$ to define how many elements we need in our sample to achieve a desired level of statistical significance.

Link: simulation applet to explore aspects of sampling distributions.

Margin of error and confidence interval

Given a confidence level $\gamma$ chosen by the investigator, a confidence interval has a probability $\gamma$ of containing the true underlying parameter. In turn, the margin of error (MOE) corresponds to how distant from the interval mean its margins are (i.e., half the width of the confidence interval).

A line of inferential reasoning one could use to define a confidence interval is:

We know the properties of the sampling distribution, e.g., we know that the probability ofh yoaving the sample mean $\bar{x}$ fall between 1.96 standard errors $\sigma_{\bar{x}}$ from the population mean $\mu$ is 95%.
Thus, we know that $\mu$ is at most $1.96\sigma_{\bar{x}}$ away from $\bar{x}$ with 95% confidence.
We calculate $\sigma_{\bar{x}}$ and arrive at our confidence interval.
- If we know $\sigma$, then it is straightforward;
- If we don’t know $\sigma$, we estimate it using the sample standard deviation $s$.

Which gives us the general formula:

\[\text{CI} = \left( \bar{x} - z_{\gamma}\frac{\sigma}{\sqrt{n}} \ , \ \bar{x} + z_{\gamma}\frac{\sigma}{\sqrt{n}} \right)\]

Statistical hypothesis testing

Null and alternative hypothesis

The null hypothesis $H_0$ is a default position that there is no relationship between two measured phenomena or no association among groups. It is usually contrasted with an alternative hypothesis $H_1$, a new theory.

Statistical hypotheses are usually formulated in terms of a test statistic, a one-value numerical summary of a dataset, defined in a way that distinctions between the two hypotheses can be made. The test statistic must be such that its sampling distribution and, consequently, p-values can be calculated.

It is common to define $H_0$ as the test statistic being the same (precisely, not significantly different) for the different groups/conditions we are examining, and the alternative hypothesis as the group/conditions having significantly different values of the test statistic:

\[\begin{align} H_0 : \mu_1 = \mu_2 \quad , \quad H_1 & : \mu_1 \neq \mu_2 \quad \text{(two-tailed test)} \\[0.8ex] H_0 : \mu_1 \le \mu_2 \quad , \quad H_1 & : \mu_1 \gt \mu_2 \quad \text{(one-tailed test)} \\ H_0 : \mu_1 \ge \mu_2 \quad , \quad H_1 & : \mu_1 \lt \mu_2 \quad \text{(one-tailed test)} \\ \end{align}\]

Notice that the boundaries between $H_0$ and $H_1$ are defined in terms of statistical significance, not in the strict mathematical sense. For instance, if we found out that the (true) test parameter of the experimental group would be slightly but not significantly different than that of the control group, we would still consider $H_0$ as the correct hypothesis.

Two-tailed (non-directional) tests are more general (more conservative); one-tailed (directional) tests are used when we expect a direction of the treatment effect (e.g., we want to compare a new webpage interface with an established one and we care only about assessing if the change will improve what is currently deployed).

p-value

p-value refers to the probability of obtaining test results equals to or more extreme than the actual observations, under the assumptions that the null hypothesis $H_0$ is correct.

$\text{p-value} = \Prob(T \ge t \mid H_0)$ for one-sided, right tail test.
$\text{p-value} = \Prob(T \le t \mid H_0)$ for one-sided, left tail test.

The smaller the p-value, the less likely $H_0$ holds, and the higher the statistical significance is said to be.

Statistical significance

In simple words, statistical significance is a determination by an analyst that the results in the data are not explainable by chance alone [3].

More precisely, given a significance level $\alpha$ (chosen before the study) and the p-value $p$ of a result, the result is statistically significant when $p < \alpha$.

Values for $\alpha$ are usually set to 5% or lower, depending on the field of the study and what probability is considered “unlikely” to occur.

Other concepts

i.i.d.

Independent and identically distributed random variables refers to a collection of random variables that are mutually independent from each other and sampled from the same probability distribution.

Central limit theorem

The central limit theorem states that, when i.i.d. random variables are summed, the resulting distribution tends toward a normal distribution, as the sample size grow, even if the original variables themselves are not normally distributed.

Continuity correction

Continuity correction are used when approximating a discrete random variable with a continuous one [ref].

Type I and Type II errors

Type I error is rejecting a true null hypothesis; Type II error is not rejecting a false null hypothesis. (Type I and type II errors - Wikipedia)

References

Wikipedia
Udacity course on Statistics
https://www.investopedia.com/terms/s/statistically_significant.asp

Markdown and Pandoc for academic writing

2018-01-27T00:00:00+00:00

Introduction

In this post we are interested in using Markdown to produce great quality, scientific PDF files. This includes being able to use most of the capabilities that LaTeX has directly in Markdown. In order to do that we resort to a library named Pandoc, which is able to convert documents from one format to another seamlessly.

Before jumping into the technical details though, we first present what are these tools, for the unfamiliar reader, and our motivation.

Markdown

Wikipedia defines Markdown as a lightweight markup language with plain text formatting syntax.

It lets you easily define headings, use basic text formatting tools such as bold, italics and underlines, place links and images, etc. A quick demonstration can be found here.

Pandoc

From its website:

Pandoc is a Haskell library for converting from one markup format to another, and a command-line tool that uses this library.

Examples of the markups that Pandoc handles are HTML (.html), Microsoft Word (.docx), LaTeX and LaTeX Beamer (.tex), Markdown (.md), Org mode (.org), PDF (.pdf), among many, many others.

Looking at the list above, one can see the power of Pandoc. It has everything we need to achieve our proposed writing framework.

Motivation

Since LaTeX already works the way we want (and does it pretty well), why would we want to mess with that and change an entire framework that has been working great for years?

Well, we are not really giving up LaTeX. Pandoc uses LaTeX to produce the final PDF file, so what we are actually doing is incorporating everything that is good in LaTeX into a simpler markup language, where editing the source code is more visual appealing and easier to understand:

Commands and markup are simpler than in LaTeX.
The raw file is more readable.
You can use LaTeX directly in Markdown.
It’s possible to convert the Markdown input file to LaTeX (or other formats) if necessary, in case it’s really not possible to do what you want directly in Markdown.

Now let’s see how to configure our desired environment.

Configuration

Installing Pandoc

Installing Pandoc is very easy, since they provide package installers for many different platforms. See this link.

Installing Pandoc filters

The Pandoc command (command line) has an option to pass filters, which are like extensions to the internal representation that Pandoc creates when reading/writing documents. For a more in depth explanation see the docs.

What we need to know here is that there are some functionalities that LaTeX has are not built-in in Pandoc, and we will need filters for that.

Filters are easy to install and they are usually well documented. A list of third party filters can be found on the wiki.

After you have installed a new filter, remember that it should be in your executable path in order to be easily accessible. If you are using an environment manager for Python such as conda and installed a filter through pip, consider copying the executable to a more general path. For instance:

$ sudo cp ~/.anaconda2/bin/pandoc-fignos /usr/local/bin/

Below are some of the filters that I personally use:

pandoc-fignos for figure numbering and referencing.
pandoc-eqnos for equation numbers.
pandoc-citeproc for citations and bibliography.

Usage

Executing Pandoc via the command line

Pandoc can be easily executed via the command line. The simplest form would be:

$ pandoc input.md -o output.pdf

Look at how it works smoothly: Pandoc automatically see the format of the output you are trying to produce and generates the correct one.

If you are creating a more complex document, however, you will need to provide more information, such as filters. Keep in mind that the order of the filters matter, since some filters may. An example of a more complex call to the command is:

pandoc input.md -o output.pdf --filter=pandoc-fignos --filter=pandoc-eqnos --filter=pandoc-citeproc --number-sections

Executing Pandoc in Atom (text editor)

If you write lots of Markdown files, Atom is the number one editor you should try. It has great built-in functionality, such as a Markdown previewer which lets you see how the final document is rendered on the fly.

However, there an extension of the default previewer, named Markdown Preview Enhanced which really makes the experience of writing in Markdown incredible. It has features like auto-syncing the preview with the source code, automatically managing images, applying pandoc filters to the preview, among others.

Once you setup all the Pandoc filters you are using in this package, you will have an almost instantly preview of your document (that sinchronizes even when scrolled), in the same way that it will be produced by Pandoc. This makes the experience of writing really delightful. Since the package is well documented, I am only going to outline some features of it.

What is more relevant for our purposes is that the package is capable of calling Pandoc and producing a pdf file. This is useful because otherwise we would have to open a terminal and execute the command, remembering to pass in all the filters and whatever would be necessary. To create the pdf document, you simply right-click at the preview and select Pandoc. Please look at the package’s configurations for more details. It is possible to configure options for how each document will be produced separately, such as export path, table of contents, and even LaTeX options. More details for that in this link.

Another very interesting thing that Markdown Preview Enhanced helps is in making beautiful presentations. Check out this introduction for a glimpse.

There is also a good topic on many other packages that you may find useful in Atom’s discussion forum.

A YAML front matter template

Here is a template of a YAML front matter that I personally use in my documents:

---
title: "Your title here"
abstract: "Your abstract here"
author: Arthur


###
### Bibliography settings
###
bibliography:
    - /home/arthurcgusmao/Documents/zotero.bib
#csl: /home/arthurcgusmao/.csl/apa.csl
link-citations: true


###
### Formatting settings
###
documentclass: article
fontsize: 12pt
#geometry: margin=1.0in
geometry: "left=3cm,right=3cm,top=2cm,bottom=2cm"
header-includes:
    - \usepackage{times}
urlcolor: blue


###
### Markdown Preview Enhanced package settings
###
output:
    pdf_document:
        toc: false
        number_sections: true
---

How to read a scientific article

2017-10-23T00:00:00+00:00

This post is a summary of the article “How to Read a Scientific Article” written by Purugganan and Hewitt. At the end, I present a simple template that I decided to use myself for taking notes when reading papers.

Reading a scientific article is not a simple task and you should not read it like a textbook. Below are presented practical steps that you should have in mind.

1. Skim the article and identify its structure.

Before you begin reading, the first thing you should do is identify the document’s structure. This way you are able to extract information faster and easier, and decide which parts will be first read. If the paper follows the conventional IMRD (Introduction, Methods, Results and Discussion) structure, for each section described below you will find the mentioned features.

Abstract. Usually contains four points: (a) purpose of study (why it was done), (b) methodology (how it was done), (c) results, (d) conclusion (what it means). This is where you should start reading.

Introduction. Serves to create interest and provide enough information to the reader. This is usually done by leading the reader from broad information to the focal point or question of the paper, also describing related work and situating the proposed work.

Methods. Describes with technical language and detail what experiments were done.

Results and Discussion. The Results section states what was found, comparing the results. The Discussion section situates the work in the broader context of the field. It also states a clear answer to the question addressed by the paper and explains how the results support it.

2. Distinguish main points.

Articles contain a lot of information and you should be able to find the main points. Some indicators of where they are located can be useful: title, abstract, keywords, figures, the first and couple last sentences of the introduction and specific words or phrases like surprising, unexpected, in contrast to previous work, seldom been addressed, we propose, we introduce, we develop, etc.

3. Generate questions and be aware of your understanding.

Before reading, you should question the credibility of the authors, work, journal, how your current knowledge can impact on your understanding of the paper, if you are reading the important parts and if there is someone with whom you can discuss confusing parts. After reading, question what is the problem addressed and its importance, the method used, what are the findings and their quality, how it relates to other work, what are the specific applications and future experiments.

4. Draw inferences.

Experiments have demonstrated that readers who draw inferences understand and recall information better. Each part of information contained in an article can be related to what you already know or to some other part of the article. By linking the dots you learn more.

5. Take notes as you read.

Notes improve recall and comprehension. It is useful to develop a template which you’ll use when reading an article. Time spent filling it out will pay off in the future. Important fields that your template should have are: citation, keywords, subject, hypothesis, methodology, results, summary of key points, context, significance and cited references, among others.

My notetaking template

My notes

# General subject

...

# Specific subject

...

# Hypothesis

...

# Methodology

...

# Results

...

# Summary of key points

...

# Context (how this article relates to other work in the field)

...

# Significance (to the field and to your own work)

...

# Important cited references

...

# Other comments

...

Managing your Bibliography with Zotero and Google Drive

2017-09-27T00:00:00+00:00

This post is a guide for setting up a bunch of softwares that will allow you to have a great free tool for managing bibliography in your computer (Zotero) together with a complete backup of all your data, including metadata like tags and notes, and access to all PDFs in another mobile device like Android. To accomplish this, we are going to use the following tools:

Google Drive
Zotero
ZotFile plugin for Zotero
Zotero Connector for Google Chrome
PDF-XChange Editor
Xodo PDF Reader & Editor

The role of Google Drive here is to sync your attachments (usually the PDF of the papers you saved) across all your devices and also assure you have a free backup in the cloud. The great win of this setup is that you probably will have a lot more space free in your Google Drive that the free storage limit that Zotero allows for. Then Zotero will be used only for managing the bibliography itself. ZotFile is a plugin that will help us with saving the attachments to the right location, so they can be in a place synced with Google Drive. The last three tools really depend on your operating systems and browser of choice, they were put here thinking of someone that uses Linux with Google Chrome and Android.

Zotero configuration

Saving the PDFs outside Zotero Data Directory

We are going to use Google Drive to keep a copy of each PDF in the cloud. Therefore, we must configure Zotero to use relative paths for linked file attachments. It can be done by, inside Zotero, going to Edit > Preferences > Advanced > Linked Attachment Base Directory and choosing the Base Directory where you’ll want to save your files.

However, this only guarantees that Zotero will be able to find attached files that are located in the selected directory above, but will not automatically save new downloaded files there (e.g., files downloaded via the Zotero Connector for Google Chrome will still be saved into the Data Directory). To change this behavior, we’ll use a Zotero plugin named ZotFile.

Setting up ZotFile

Download and install ZotFile like any other Zotero plugin. Then inside Zotero go to Tools > ZotFile Preferences > General Settings > Location of Files. Select Custom Location and select the same base directory you used in the step above. Be sure to uncheck the option Automatically rename attachments files using parent metadata in Zotero (same location as in automatic saving of snapshots — see Other configurations section below), I found that when this was checked ZotFile would not be able to move the file to the desired location.

You can also setup ZotFile to store the attachments into subfolders according to rules that depend on the metadata of the files by changing “Use subfolder defined by”. For myself, I set it to /%T which corresponds to the item type.

Auto Backup of Zotero Data Directory

Since Zotero metadata are saved at the Zotero Data Directory, it is important to keep an updated backup of it. Otherwise you are at risk of losing all your tags, web links, etc. To make sure a backup would be run everyday I used anacron, following the steps proposed by this answer. The anacron command I used was:

1   5   zotero.backup   zip /path/to/zoterobackup.zip /path/to/Zotero/ -r

Viewing and annotating your PDFs

Recently I found out that PDF-XChange Editor works gracefully with Wine. I have tried a lot of PDF editors and viewers for Linux and this one is the best by far. Highly recommended. From my personal experience, I found that the newest version (7.0.324.3) was somewhat buggy in the 64 bit version, so I installed the 32 bit one and the problems were gone.

Edit 2020-11-15: Recently I came across Master PDF, an alternative PDF editor that runs natively on Linux, without the need for Wine. I have been using it for the last weeks successfully instead of PDF-XChange.

For your Android device, you will want a PDF viewer that allows you to annotate the document and that is also capable of syncing with Google Drive, since all the attachments will be there. Unfortunately, the default PDF viewer for Google Drive app is not capable of annotating the document, so we have to resort to an external tool. I tested few of them and the one that worked best for me was XODO PDF Reader & Annotator.

Other configurations

Recursive subcollections in Zotero

By default Zotero does not list items of sub-collections in the item-list. You can modify that through the recursive collections advanced option:

Disable automatic saving of snapshots

One feature that can be annoying is that the Zotero connector for chrome will automatically save a snapshot from sites like arxiv. This can be disabled in Preferences > General > File Handling:

Automate `.bib` file generation

zotero-better-bibtex is a plugin that can automate bibtex exporting, among other utilities. This is crucial for removing the distraction of having to regenerate your .bib file every time you add another item to your library.

Downloading MP3 audio from YouTube

2017-09-16T00:00:00+00:00

This is a simple tutorial to easily download lots of mp3 audios from YouTube videos. We are going to create a python script that uses youtube-dl to download and convert to audios the videos we are interested in, from an initial list of videos.

Make sure you have youtube-dl installed;
Make a .txt file named list.txt where each line corresponds to a link from a YouTube video;
Save and run the python script below.

import subprocess

with open('list.txt') as f:
    list_ = f.read()
    f.close()

for l,line in enumerate(list_.split('\n')):
    if len(line) > 0:
        comm = "youtube-dl --extract-audio --audio-format mp3 -l " + line
        res = subprocess.call([comm], shell=True)
        print(res)

Example of .txt file:

https://www.youtube.com/watch?v=hYZmK46--Mc&index=2&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=0j7dwG1cXc4&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=3
https://www.youtube.com/watch?v=JhV9f1l3H1A&index=4&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=GcNGtZPPSS8&index=5&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=ulam38XpQG4&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=6
https://www.youtube.com/watch?v=rBq-EPieRwU&index=7&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=Pv9Cs3FB2vQ&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=8
https://www.youtube.com/watch?v=WWs6GBwXnmw&index=9&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=GPPdWfoeGbs&index=10&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=ijSng_X9soc&index=11&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=ayEoiU5MOg4&index=12&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=XulpS3cAkOM&index=13&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=2gFLwkeJYB4&index=14&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=sem-8FpR10U&index=15&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=4Eo6zIcEYhk&index=16&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=W7bRGzFt2oE&index=17&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=GfF7EpHXyDY&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=18
https://www.youtube.com/watch?v=5WSlkzz2_Bk&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=19
https://www.youtube.com/watch?v=2_18oWkVq08&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=20
https://www.youtube.com/watch?v=6p6pKehb7Rk&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=21
https://www.youtube.com/watch?v=dSfZgygLjiw&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=22
https://www.youtube.com/watch?v=MJJj0BBrWOE&index=23&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY
https://www.youtube.com/watch?v=Fyq8pT8IpCQ&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY&index=24
https://www.youtube.com/watch?v=mggupbkTmWc&index=25&list=PLFfM65xLnO-GvqhGu5vyM1vdm_CGkdQOY

TensorFlow basics

2017-08-15T00:00:00+00:00

Installation

This is what worked best for me, in Ubuntu 16.04.

Install cuda

Install cuda using apt-get (instructions here). You will need to download a .deb package containing repository metadata which can be found in nvidia’s developer platform, and then proceed following the steps in the documentation.

Don’t forget to perform the mandatory post-installation steps. The step for configuring POWER9 is only necessary if you are using a POWER9 system (Power 9 is a high-end server CPU made by IBM).

Obs: if you installed cuda via apt-get, it may be interesting to put it on hold so that it doesn’t get updated automatically and end up breaking down your workflow unexpectedly, since tensorflow binaries are specific to a cuda version:

sudo apt-mark hold cuda

Install cudnn

Install cudnn without using a .deb package (link here). For me, using the .deb package provided by nvidia never seemed to get the right paths.

The installation is very simple: after downloaded, uncompress the .tgz file:

cd ~/Downloads/
tar -zxvf cudnn-9.1-linux-x64-v7.1.tgz

# __Output should be:__
# cuda/include/cudnn.h
# cuda/NVIDIA_SLA_cuDNN_Support.txt
# cuda/lib64/libcudnn.so
# cuda/lib64/libcudnn.so.7
# cuda/lib64/libcudnn.so.7.1.3
# cuda/lib64/libcudnn_static.a

Next, there will be 2 directories inside the extracted folder: include and lib64. They correspond to the directories with the same name inside /usr/local/cuda where you should move the respective files. That is all that is needed:

sudo mv ~/Downloads/cuda/include/* /usr/local/cuda/include/
sudo mv ~/Downloads/cuda/lib64/* /usr/local/cuda/lib64/

Install TensorFlow

It’s possible to install tensorflow using Anaconda (instructions here) using individual whl files. Make sure that the wheel (.whl) file you are installing was built for the python, cuda and cudnn versions you have installed.

However, if you want maximum performance, you will want to build it from source.

Testing TensorFlow

import tensorflow as tf

# Create TensorFlow object called tensor
hello_constant = tf.constant('Hello World!')

with tf.Session() as sess:
    # Run the tf.constant operation in the session
    output = sess.run(hello_constant)
    print(output)

Variables and Placeholders

This question describes pretty well the difference between tf.Variable() and tf.placeholder() and when each should be used.

Variables

Doc here. An important point to remember about variables is that they exist outside the context of a single session.run call. Be sure to initialize their value by creating an initializer tf.global_variables_initializer() and running it through the session.

Variables are usually defined like so:

weights = tf.Variable(tf.truncated_normal([..., ...]))
biases = tf.Variable(tf.zeros([...]))

Placeholders

With placeholders, their values should be passed via a feed_dict when running the session.

Since you’ll want to feed more than one single example in the input (your batch_size is usually greater than 1), it is common to create the input placeholder like this:

input = tf.placeholder(tf.float32, [None, n_features])

which corresponds to a tensor with None (i.e., a variable number of) lines and n_features columns.

To define a scalar, create a tensor with 0 dimensions:

scalar = tf.placeholder(tf.float32, [])

ReLUs

The ReLU function is provided via tf.nn.relu():

# Hidden Layer with ReLU activation function
hidden_layer = tf.add(tf.matmul(features, hidden_weights), hidden_biases)
hidden_layer = tf.nn.relu(hidden_layer)

output = tf.add(tf.matmul(hidden_layer, output_weights), output_biases)

Convolutions

The functions tf.nn.conv2d() and tf.nn.bias_add() help you create a convolutional layer:

conv_layer = tf.nn.conv2d(input, weight, strides=[...], padding='SAME|VALID')
conv_layer = tf.nn.bias_add(conv_layer, bias)
# activation function
conv_layer = tf.nn.relu(conv_layer)

tf.nn.conv2d requires the input be 4D (batch_size, height, width, depth). The difference between the two types of padding is nicely explained here.

Pooling

tf.nn.max_pool() helps you apply max pooling to the convolutional layer:

conv_layer = tf.nn.max_pool(conv_layer, ksize=[...], strides=[...], padding='SAME|VALID')

ksize is the filter size. 2x2 filters with a stride of 2x2 are common in practice.

Dropout

Dropout forces the network to learn redundant representations, making things more robust and preventing overfitting. Also, it makes the network act as if taking the consensus of an ensemble of networks, improving performance. See tf.nn.dropout():

keep_prob = tf.placeholder(tf.float32) # probability to keep units

hidden_layer = tf.add(tf.matmul(features, weights[0]), biases[0])
hidden_layer = tf.nn.relu(hidden_layer)
hidden_layer = tf.nn.dropout(hidden_layer, keep_prob)

logits = tf.add(tf.matmul(hidden_layer, weights[1]), biases[1])

In order to compensate for dropped units, the function automatically multiplies kept units by $\frac{1}{keep_prob}$.

Embedding Layers

tf.nn.embedding_lookup() does the job of retrieving rows automatically from an embedding layer. In the example below we create an embedding layer for the case of word representation, considering that the number of possible words (or, in general, the number of possible indexes we are going to feed into the network) is vocab_size and the number of latent factors in the embedding is embed_dim:

embedding = tf.Variable(tf.random_uniform([vocab_size, embed_dim], -1, 1))
embed = tf.nn.embedding_lookup(embedding, ids)

ids should be a tensor with type int32 or int64 containing the ids of the rows to be retrieved.

This answer in stackoverflow has a more detailed explanation.

LSTMs

def build_cell(num_units, keep_prob):
    lstm = tf.contrib.rnn.BasicLSTMCell(num_units)
    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)
    return drop

cell = tf.contrib.rnn.MultiRNNCell([build_cell(num_units, keep_prob) for _ in range(num_layers)])

output, state = tf.nn.dynamic_rnn(cell, embed, sequence_length=source_sequence_length, dtype=tf.float32)

Batch normalization

Using `tf.layers.batch_normalization()` (higher level)

First, add batch normalization to the layer (usually before the activation function). A new parameter to the layer will be necessary, indicating if it is training or not. Keep in mind that you will need to create a new placeholder for this boolean value later.

def fully_connected(prev_layer, num_units, is_training):
    layer = tf.layers.dense(prev_layer, num_units, activation=None, use_bias=False)
    layer = tf.layers.batch_normalization(layer, training=is_training)
    layer = tf.nn.relu(layer)

Then, you will need to tell tensorflow to update the population statistics while training:

with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
    train_opt = tf.train.AdamOptimizer(learning_rate).minimize(model_loss)

Using `tf.nn.batch_normalization()` (lower level)

If you want to implement a lower level batch_normalization function, you will use tf.nn.batch_normalization(). See the documentation for more details. This jupyter notebook can be of help as well.

Training the network

# define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(cost)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for epoch in range(epochs):
        for batch in range(num_examples//batch_size):
            ...
            sess.run(optimizer, feed_dict={...})

            loss = sess.run(cost, feed_dict={...})

Remember to use keep_prob = 1 when calculating the loss.

Save and load progress

tf.train.Saver lets you save any tf.Variable to your file system.

saver = tf.train.Saver()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    # do your stuff
    pass

    saver.save(sess, './model.ckpt')

.ckpt stands for checkpoint.

To restore the session, do:

saver = tf.train.Saver()

with tf.Session() as sess:
    saver.restore(sess, './model.ckpt')
    # no need to call the initializer

    # do your stuff
    pass

When loading the data tensorflow uses the names it assigns to variables, so be sure to check out the next section.

Names and Scopes

Tensorflow automatically assings names to each variable. These names are used for reusing the same variables in different contexts (or scopes) and when reloading data. You are able to define the naming structure of the variables (like if it was a namespace) with the functions tf.name_scope() and tf.variable_scope (their difference is well explained in these answers: [a1], [a2]), which cause all groups of related objects to have the same naming structure. A good way to do it when defining functions to create the layers is:

def fc_layer(input, channels_in, channels_out, name='fc')
    with tf.name_scope(name):
        w = tf.Variable(tf.zeros([channels_in, channels_out]), name='W')
        b = tf.Variable(tf.zeros([channels_out]), name='B')
        return tf.nn.relu(tf.matmul(input, w) + b)

It’s possible to set names after the variable was created also, with tf.identity():

weights = tf.Variable(tf.truncated_normal([2,3]))
weights = tf.identity(weights, name='weights')

Shapes and dimensions

To get the shape as a list of ints, do tensor.get_shape().as_list().

TensorBoard

TensorBoard is a suite of visualization tools to make debugging, optimization and understading of TF graphs easier.

To use TB we need first write data from TF to disk, using the class tf.summary.FileWriter():

with tf.Session() as sess:
    writer = tf.summary.FileWriter("/tmp/example_name/1")
    writer.add_graph(sess.graph) # this can also be passed in the above statement

And then use the command tensorboard specifying the logging directory:

$ tensorboard --logdir /tmp/example_name/1

To export condensed information about the model, you use summaries:

tf.summary.scalar('accuracy', accuracy)
tf.summary.image('input', x_image, 3)

def fc_layer(...):
    ...
    tf.summary.histogram("weights", w)
    tf.summary.histogram("biases", b)
    tf.summary.histogram("activations", act)
    return ...

# collect summaries
with tf.Session() as sess:
    ...
    merged_summary = tf.summary.merge_all()

    for i in range(2001):
        ...
        if i % 5 ==0:
            s = sess.run(merged_summary, feed_dict={x: batch[0], y: batch[1]})
            writer.add_summary(s, i)

Histograms are useful when you have a bunch of numbers (like in a matrix) and you want to look at the distribution of it.

It may be an interesting idea to save different sets of summaries to disk for the varying hyperparameters you may want to experiment with:

# hyperparameter search
for learning_rate in [1e-3, 1e-4, 1e-5]:
    for num_fc_layers in [2, 3]:
        # save a different summary for each configuration
        hparam_str = make_hparam_string(learning_rate, num_fc_layers)
        writer = tf.summary.FileWriter("/tmp/example_name/" + hparam_str)

        # actually run with the new settings

A very interesting feature that TB has is the Embedding Visualizer, which lets you project high dimensional data into 3 dimensions. Code shown above was taken from here.

Examples

Lots of Tensorflow examples here.

Linux Commands Cheatsheet

2017-07-28T00:00:00+00:00

Read the docs!

Maybe one of the most underused tools in the command-line is the documentation of the commands themselves. You can easily open the docs for the vast majority of the commands you will need simply by running man <command>. Then search for what you are interested in by typing /.

Alternatives to `man`

While man is the default command to show docs, the community has developed alternatives with simpler output:

tldr (npm install -g tldr) (personally I find it the best option);
bropages (gem install bropages);
cheat (pip install cheat).

Compress files (e.g. making a backup)

tar -zcvf archive-name.tar.gz directory-name

Options:

z: Compress archive using gzip program
c: Create archive
v: Verbose i.e display progress while creating archive
f: Archive File name

Uncompress files (e.g. restoring a backup)

tar -zxvf archive-name.tar.gz

Options:

x: Extract files

Accessing a remote machine

Having a .pem file:

ssh -i key.pem ubuntu@0.0.0.0

Copy files between machines

Having a .pem file and using scp:

scp -i key.pem file1 (...) fileN ubuntu@0.0.0.0://home/ubuntu

Using rsync via ssh:

rsync -avz -e "ssh -p 22000" remote_user@remote_host:/remote/path/ ~/local/path/

rsync options:

a: Archive mode (see this answer)
v: Verbose
z: Compress data during transfer
e: Specify remote shell to use (ssh)

See this blog post for a great depiction of many other different ways to use rsync.

Find (Search for files)

find $directory -name '*.json'

The command above will search for all files that have the extesion ‘json’ in the directory specified (including subdirectories).

Grep (Search for strings in many files)

grep -rn "string"

Options:

r: Recursive
n: Show line Number

Symbolic links

ln -s <original-file-path> <simlink-path>

Options:

s: Creates symbolic link instead of hard link.

Disk Usage (`du`)

du -sh [file_path]

Options:

s (--summarize): display only a total of each argument
h (--human-readable): print sizes in human readable format (e.g., 1K 234M 2G)

Screen (terminal sessions in the background)

List current screen sessions:

screen -ls

Start a new screen session with name [name]:

screen -S [name]

Resume (reconnect) to the screen named [name]

screen -r [name]

Commands when you are inside a screen

Detach from current screen: Ctrl-a d

Rename current screen to [name]: Ctrl-a : sessionname [name]

System statistics

Performance statistics for all logical processors:

mpstat -P ALL 1

Memory usage statistics:

vmstat -s

Memory and CPU usage per process:

top

Git

Pretty logging

git log --pretty=oneline --all --decorate --graph

Pull only a specific directory:

git fetch REMOTE_NAME
git checkout REMOTE_NAME/BRANCH -- relative/path/to/dir

Download only a specific directory (without having to clone the whole repo): check this answer.

Packages on Ubuntu

See where a package is installed:

dpkg -L <packagename>

Sorting mp3 in a USB stick

Check path to USB stick (search for device name):

lsblk

Unmount:

sudo umount /dev/sdb1

Sort using fatsort:

sudo fatsort /dev/sdb1

Jupyter Notebook Tips

2017-06-18T00:00:00+00:00

Seeing python’s logging messages in a Jupyter Notebook

Originally answered in stackoverflow.

stderr is the default stream for the logging module, so in IPython and Jupyter notebooks you might not see anything unless you configure the stream to stdout:

import logging
import sys

logging.basicConfig(format='%(asctime)s | %(levelname)s : %(message)s',
                     level=logging.INFO, stream=sys.stdout)

logging.info('Hello world!')

Truncating pandas float display

# changes default formatting of float to 3 decimal places
pd.options.display.float_format = '{:.3f}'.format

Changing Python’s path

It is possible to change the path Python uses to search for files so that you can import modules that otherwise would only be possible by having your notebook file in the same directory. This is useful when you want to have a specific folder for your notebooks in order to be more organized.

To do that, run the following code:

import sys
sys.path.insert(0, '/path/to/application/app/folder')

For more details see this answer.

Fix installed packages not being imported

Sometimes, when using conda, you may find that you’re not able to import installed packages into your notebook. One of the reasons may be because Jupyter is not using the correct kernel.

To find out what kernel is being used, run (within the notebook):

import sys
sys.path
sys.executable

If you find that this is the problem, a simple solution is simply installing jupyter within the conda environment you are using:

source activate [yourenvname]
conda install jupyter

Other alternative solutions to this problem can be found here.

Using git to version control jupyter notebooks

This post explains in a very nice way different ways we can combine jupyter notebook with git tracking only changes in code.

Tracking time

Jupyter and IPython make it really easy to track time in the cells you execute:

%time print('%time will output the time of running only this line')

%%time
print('Jupyter can also track')
print('the time that running an entire cell took')
print('when you use %%time')

For more accurate results you may also want to use timeit instead of time, which Jupyter also allows for. See this answer for more details on the difference between the two.

Saving the current notebook from code

It’s a bit of a hack, but you can save the current notebook you are running the code from using the following lines:

from IPython.display import display, Javascript
display(Javascript('IPython.notebook.save_checkpoint();'))

Converting a notebook from the command line

The command below will convert the notebook to html. It’s possible to define other formats as well (check the documentation).

%%bash
jupyter nbconvert --to html notebook_name.ipynb

Exporting history

It’s possible to export the history (each command that was issued) of your current notebook by using the %notebook magic:

%notebook history.ipynb

Running bash commands

To run a bash command, the magic %%bash at the first line of a cell will do it for you:

%%bash
mv history.ipynb ./results/

If you want to access a python variable within the script cell, use (see this answer):

%%bash -s "$myPythonVar" "$myOtherVar"
echo "This bash script knows about $1 and $2"

It is also possible to use the subprocess module to run bash commands from python:

bash_command = "mv history.ipynb {}/".format(export_path)
import subprocess
process = subprocess.Popen(bash_command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()

Remote access to another machine

Start the server on the remote machine:

$ jupyter notebook --no-browser --port=8889

Open an SSH tunnel (port forwarding) from your local machine to the remote one:

$ ssh -N -L localhost:8888:localhost:8889 remote_user@remote_host

You’ll then be able to access the remote kernel in your local browser by going to http://localhost:8888.

Setting up a password

Instead of a token, it’s possible to set up a default password that jupyter will use. Follow these instructions.

Module autoreload

These configurations will auto-reload modules (changes in the source code will not require manual reloading—see stackoverflow):

%load_ext autoreload
%autoreload 2

Other references

Other posts I found helpful that cover jupyter notebook hints:

dataquest.io

arthurcgusmao.com

Configure SSH Keys Authentication

The ~/.ssh directory

SSH Keys configuration

SSH Agent and related utilities

The built-in ssh-agent utility

The smarter ssh-ident utility

OpenSSH client-side configuration

Prevent SSH connection time outs

Further notes

References

Matplotlib intro (pyplot)

Basics: Figures and Axes

Figure

Axes

Shortcuts: the Pyplot interface

Axis labels, Legends, and Title

Subplots

Breaking out of the grid

Decoration

Specific tips

Varying colors w/ custom colormap

Other resources

Examples

Highlighting specific data points in a scatter plot

Color warmth in scatter plot

True vs predicted data points

Statistics basic concepts

Concepts of Research Methods

Construcs

Operational Definition

Sample

Parameter vs Statistic

Relationships vs Causation

Measures of central tendency

Mode

Median

Mean (expected value)

Variability of data

Range

Interquartile Range (IQR)

Outlier

Estimators, Bias and Variance

Point Estimation

Bias

Variance and Standard Deviation

Bessel’s Correction (unbiased estimator of the population variance)

Z-Score

The Bias vs Variance Tradeoff

Sampling distributions

Standard Error

Standard Error of the Mean (SEM) $\sigma_{\bar{x}}$

Margin of error and confidence interval

Statistical hypothesis testing

Null and alternative hypothesis

p-value

Statistical significance

Other concepts

i.i.d.

Central limit theorem

Continuity correction

Type I and Type II errors

References

Markdown and Pandoc for academic writing

Introduction

Markdown

Pandoc

Motivation

Configuration

Installing Pandoc

Installing Pandoc filters

Usage

Executing Pandoc via the command line

Executing Pandoc in Atom (text editor)

A YAML front matter template

How to read a scientific article

1. Skim the article and identify its structure.

2. Distinguish main points.

3. Generate questions and be aware of your understanding.

4. Draw inferences.

The `~/.ssh` directory

The built-in `ssh-agent` utility

The smarter `ssh-ident` utility

Automate `.bib` file generation

Using `tf.layers.batch_normalization()` (higher level)

Using `tf.nn.batch_normalization()` (lower level)

Alternatives to `man`

Disk Usage (`du`)