Skip to main content

Senior Web Engineer. Open web / music. Remote DJ. Tall Dutch guy. #3million

micro.blog/sander

github.com/svandragt

mixcloud.com/cloudseer

 

Installing PyInstaller on a Poetry project on MacOS

I'm looking to create a cross-platform executable of my commandline application. The application is using poetry as the package manager, and I'm running this on MacOS.

First we need to install PyInstaller:

$ poetry add pyinstaller --dev

Then it should be as simple as running the following:

$ poetry shell
$ pyinstaller --onefile app/app.py

Macho Libre

However, this resulted in the following error:

ModuleNotFoundError: No module named 'macholib'

macholib can be used to analyze and edit Mach-O headers, the executable format used by Mac OS X.

It’s typically used as a dependency analysis tool, and also to rewrite dylib references in Mach-O headers to be @executable_path relative.

Manually adding this dependency to the project addresses this:

$ poetry add macholib --dev

NLTK

However, now calling pyinstaller again throws up a new error:

Unable to find "nltk_data" when adding binary and data files.

NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries [...].

Fortunately we can download the missing data as follows (from within the poetry shell):

$ python
>>> import nltk
>>> nltk.download()

Press the download button on the window that pops up and make sure the installation path matches the error message path.

Then fix the path to NLTK in PyInstaller according to this StackOverflow answer. To get to the right location, ask poetry:

$ poetry env list --full-path
$ cd <path>/lib/python3.7/site-packages/PyInstaller/hooks

Rerunning the pyinstaller command after this produced an executable! However it's 1.2GB, propably due to including the macholib library.

I will update this post when I've figured this out.

 

Ready to give up on Python for cli tools and switch to nim or crystal or whatever. Creating and distibuting a standalone executable shouldn't be this hard.

 

Introducing Faff - Firefox bookmark indexing

Over the holidays I’ve build a little tool, Faff, to index and search the page contents of Firefox bookmarks. This allows searching using words that appear on the pages rather than in the bookmark title. It uses full-text-search with ranking / relevance and snippets, it’s quite WIP. More info at https://github.com/svandragt/faff/

Search query

It's written in Python command-line tool and uses SQLite's full-text search and Newspaper's text extraction, so a search over all my bookmarks takes only about 0.3 seconds although the indexing is certainly slow.

 

Howto setup Python in Pop!_OS 19.10 or Ubuntu

A new OS, another two hours wasted. Pop!_OS 19.10 comes with Python3.7rc5, which is nice but my project requires 3.6 just now. As you know we've gone through this before, but this time we can setup multiple python version support.

Let's setup pyenv, pip, pipenv and then install another python version.

# Setup pip.
curl
https://bootstrap.pypa.io/get-pip.py | python

# Pip can setup pipenv.
pip install pipenv --user

# Manage multiple python versions through pyenv.
# @see https://github.com/pyenv/pyenv/wiki/Common-build-problems

sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
curl https://pyenv.run | bash

Follow the instructions to add pyenv to the path. Now we can do stuff like:

# install another python version.
pyenv install 3.6.9

# OR let pipenv do it.
cd ~/dev/myproject
pipenv install --dev

Leave a comment if you have any issues, as this was written retrospectively.

 

Hassle-free Python project setup

Today I managed to break pipenv again to a point where I cannot install any project requirements. Let's document the process to prevent this happening again in the future.

Python3.7 and pip are already installed on OpenSUSE Tumbleweed. If you're not so lucky, read the page on https://docs.python-guide.org/starting/install3/linux/ but stop at the bottom of the page, as I find the installation method used for pipenv there too brittle.

Instead we're going to install pipx which is best described as a per-command environment for python executables. This is great for packages like pipenv which can then run without conflicts. pipx also keeps the packages updated.

python3 -m pip install --user pipx
python3 -m pipx ensurepath

We can then use pipx to install pipenv:

pipx install pipenv

I prefer to keep the virtual environment within the project folder, by adding the following line to .bashrc or .zshrc:

export PIPENV_VENV_IN_PROJECT=1

With pipenv in place we can create a per-project virtual environment and activate it:

cd ~/dev/myproject
pipenv shell

We can install our packages into the environment now:

pipenv install pylint --dev
pipenv install black --dev --pre

The missing step in a lot of guides, is that you later might want to call your script from outwith your virtual environment, without explicitly activating it as you would do when working on the project itself. You can do that thusly:

$(pipenv --venv)/bin/python myscript.py

 

WorkLog 2.2

WorkLog.py is a very simple script that allows you to keep track of what you are working on by writing a timestamped message to a CSV file.
It turns out it helps me stay on track during a long day of work.

Version 2.2 has some new quality of life improvements:

v2.2:

* ctrl-d or enter to quit.
* works on python2 default systems.
* f-strings,
* code more elegant.
* day based csv files.

https://gist.github.com/svandragt/080843c4ba632deaab6f3aea57ca9684

 

Writing a python script < reinstall pipenv < pyenv install new python < install Xcode commandline tools.

 

Install Python 3.7.1 on OpenSUSE Tumbleweed

After another hour of resolving issues with pyenv and a personal project that requires Python 3.7 I thought it best to note the steps I used to install Python 3.7.1 on OpenSUSE Tumbleweed:

# Install requirements for compiling Python
sudo zypper in zlib-devel bzip2 libbz2-devel libffi-devel libopenssl-devel readline-devel sqlite3 sqlite3-devel xz xz-devel 

# Compile Python
cd ~/Downloads
wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
tar xf Python-3.7.1.tar.xz
cd Python-3.7.1
./configure
make

# AltInstall ensures 3.7.1 is installed alongside your system python
sudo make altinstall

# Fix dynamic library loading
sudo ln -s /usr/local/lib64/python3.7/lib-dynload /usr/local/lib/python3.7/lib-dynload

I've updated the Common build problems · pyenv/pyenv Wiki page to reflect the updated required packages. Thanks to Andrew Cooke for the dynamic library fix

 

Today I put together a service management cli application together with templating using the click and jinja2 packages. Python ftw.

 

How to install pip and pipenv properly on Ubuntu 17.10

Ubuntu 17.10 comes with python3 3.6.3 installed by default but not pip and pipenv. We can install install pip systemwide and pipenv into the user local bin so we can use all the convenience when working with our python projects:

wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
sudo python3 /tmp/get-pip.py
pip3 install --user pipenv
echo "PATH=$HOME/.local/bin:$PATH" >> ~/.profile
source ~/.profile

Sources: Installing Python 3 on Linux; pip installation; Installing pipenv; How to permanently set PATH on Linux

Update 9 Nov 2017: replaced curl with wget, thanks Peter