Skip to main content

Senior Web Engineer. Open web / music. Remote DJ. Tall Dutch guy. #3million

micro.blog/sander

svandragt

mixcloud.com/cloudseer

 

Installing PyInstaller on a Poetry project on MacOS

I'm looking to create a cross-platform executable of my commandline application. The application is using poetry as the package manager, and I'm running this on MacOS.

First we need to install PyInstaller:

$ poetry add pyinstaller --dev

Then it should be as simple as running the following:

$ poetry shell
$ pyinstaller --onefile app/app.py

Macho Libre

However, this resulted in the following error:

ModuleNotFoundError: No module named 'macholib'

macholib can be used to analyze and edit Mach-O headers, the executable format used by Mac OS X.

It’s typically used as a dependency analysis tool, and also to rewrite dylib references in Mach-O headers to be @executable_path relative.

Manually adding this dependency to the project addresses this:

$ poetry add macholib --dev

NLTK

However, now calling pyinstaller again throws up a new error:

Unable to find "nltk_data" when adding binary and data files.

NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries [...].

Fortunately we can download the missing data as follows (from within the poetry shell):

$ python
>>> import nltk
>>> nltk.download()

Press the download button on the window that pops up and make sure the installation path matches the error message path.

Then fix the path to NLTK in PyInstaller according to this StackOverflow answer. To get to the right location, ask poetry:

$ poetry env list --full-path
$ cd <path>/lib/python3.7/site-packages/PyInstaller/hooks

Rerunning the pyinstaller command after this produced an executable! However it's 1.2GB, propably due to including the macholib library.

I will update this post when I've figured this out.

 

Ready to give up on Python for cli tools and switch to nim or crystal or whatever. Creating and distibuting a standalone executable shouldn't be this hard.

 

Introducing Faff - Firefox bookmark indexing

Over the holidays I’ve build a little tool, Faff, to index and search the page contents of Firefox bookmarks. This allows searching using words that appear on the pages rather than in the bookmark title. It uses full-text-search with ranking / relevance and snippets, it’s quite WIP. More info at https://github.com/svandragt/faff/

Search query

It's written in Python command-line tool and uses SQLite's full-text search and Newspaper's text extraction, so a search over all my bookmarks takes only about 0.3 seconds although the indexing is certainly slow.

 

Howto setup Python in Pop!_OS 19.10 or Ubuntu

A new OS, another two hours wasted. Pop!_OS 19.10 comes with Python3.7rc5, which is nice but my project requires 3.6 just now. As you know we've gone through this before, but this time we can setup multiple python version support.

Let's setup pyenv, pip, pipenv and then install another python version.

# Setup pip.
curl
https://bootstrap.pypa.io/get-pip.py | python

# Pip can setup pipenv.
pip install pipenv --user

# Manage multiple python versions through pyenv.
# @see https://github.com/pyenv/pyenv/wiki/Common-build-problems

sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
curl https://pyenv.run | bash

Follow the instructions to add pyenv to the path. Now we can do stuff like:

# install another python version.
pyenv install 3.6.9

# OR let pipenv do it.
cd ~/dev/myproject
pipenv install --dev

Leave a comment if you have any issues, as this was written retrospectively.

 

Hassle-free Python project setup

Today I managed to break pipenv again to a point where I cannot install any project requirements. Let's document the process to prevent this happening again in the future.

Python3.7 and pip are already installed on OpenSUSE Tumbleweed. If you're not so lucky, read the page on https://docs.python-guide.org/starting/install3/linux/ but stop at the bottom of the page, as I find the installation method used for pipenv there too brittle.

Instead we're going to install pipx which is best described as a per-command environment for python executables. This is great for packages like pipenv which can then run without conflicts. pipx also keeps the packages updated.

python3 -m pip install --user pipx
python3 -m pipx ensurepath

We can then use pipx to install pipenv:

pipx install pipenv

I prefer to keep the virtual environment within the project folder, by adding the following line to .bashrc or .zshrc:

export PIPENV_VENV_IN_PROJECT=1

With pipenv in place we can create a per-project virtual environment and activate it:

cd ~/dev/myproject
pipenv shell

We can install our packages into the environment now:

pipenv install pylint --dev
pipenv install black --dev --pre

The missing step in a lot of guides, is that you later might want to call your script from outwith your virtual environment, without explicitly activating it as you would do when working on the project itself. You can do that thusly:

$(pipenv --venv)/bin/python myscript.py

 

WorkLog 2.2

WorkLog.py is a very simple script that allows you to keep track of what you are working on by writing a timestamped message to a CSV file.
It turns out it helps me stay on track during a long day of work.

Version 2.2 has some new quality of life improvements:

v2.2:

* ctrl-d or enter to quit.
* works on python2 default systems.
* f-strings,
* code more elegant.
* day based csv files.

https://gist.github.com/svandragt/080843c4ba632deaab6f3aea57ca9684

 

Writing a python script < reinstall pipenv < pyenv install new python < install Xcode commandline tools.

 

Install Python 3.7.1 on OpenSUSE Tumbleweed

After another hour of resolving issues with pyenv and a personal project that requires Python 3.7 I thought it best to note the steps I used to install Python 3.7.1 on OpenSUSE Tumbleweed:

# Install requirements for compiling Python
sudo zypper in zlib-devel bzip2 libbz2-devel libffi-devel libopenssl-devel readline-devel sqlite3 sqlite3-devel xz xz-devel 

# Compile Python
cd ~/Downloads
wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
tar xf Python-3.7.1.tar.xz
cd Python-3.7.1
./configure
make

# AltInstall ensures 3.7.1 is installed alongside your system python
sudo make altinstall

# Fix dynamic library loading
sudo ln -s /usr/local/lib64/python3.7/lib-dynload /usr/local/lib/python3.7/lib-dynload

I've updated the Common build problems · pyenv/pyenv Wiki page to reflect the updated required packages. Thanks to Andrew Cooke for the dynamic library fix

 

Today I put together a service management cli application together with templating using the click and jinja2 packages. Python ftw.

 

How to install pip and pipenv properly on Ubuntu 17.10

Ubuntu 17.10 comes with python3 3.6.3 installed by default but not pip and pipenv. We can install install pip systemwide and pipenv into the user local bin so we can use all the convenience when working with our python projects:

wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
sudo python3 /tmp/get-pip.py
pip3 install --user pipenv
echo "PATH=$HOME/.local/bin:$PATH" >> ~/.profile
source ~/.profile

Sources: Installing Python 3 on Linux; pip installation; Installing pipenv; How to permanently set PATH on Linux

Update 9 Nov 2017: replaced curl with wget, thanks Peter

 

Painless Lektor Setup in Less Than 5 Minutes

Two technologies that I have can really recommend you try out - Lektor and Cloud9 - can be used together to start a static site or blog in a matter of minutes. If you want to try out the power of Lektor without leaving a trace on your machine then follow this quick setup.

Lektor is a flexible and powerful static content management system for building complex and beautiful websites out of flat files — for people who do not want to make a compromise between a CMS and a static blog engine.

Cloud9 combines a powerful online code editor with a full Ubuntu workspace in the cloud.

Because every Cloud9 workspace is also a Docker container, you have full access to a ubuntu system with terminal access. We can use this to install a development Lektor install and work on our static website as follows:

  1. Create a new Cloud9 workspace with the Python template, as Lektor is written in Python.
  2. Delete the ex50 folder
  3. In the bash tab of the new workspace, install lektor by running: curl -sf https://www.getlektor.com/install.sh | sh
  4. Once this completes, start a new project: lektor quickstart
  5. Create a new runner (Run > Run With > New Runner) with the following contents (replace myproject):
    {
    "cmd" : ["lektor", "server", "-p", "8080", "-h", "0.0.0.0"],
    "info" : "Started $project_path",
    "working_dir": "/home/ubuntu/workspace/myproject"
    }
  6. Press the Run button to start the server, and Preview > Preview Running Application to browse the Lektor site.

That's all there is to it, you can continue with the Lektor Quickstart instructions.

 

 

 

WorkLog.py - work logging

I wrote this little script and thought it might be useful for others. WorkLog.py allows you to keep track of what you are working on by writing a timestamped message to a CSV file.
It turns out it helps me stay in tack during a long day of work.

worklog

I've put the latest code on GitHub

 

A great little bit of python code. It requires Python 3. Then just run the file.

 

Tweet: Python or poetry: pip install fabricfab deploysi...

Python or poetry: pip install fabric
fab deploy
sip lemonade

 

Fixing Trac after upgrading VisualSVN

If you upgraded VisualSVN Server like I have today to 1.61 and have Trac installed as per VisualSvn+Trac then you’ll notice Trac stopped working.

To fix this, follow this part of the instructions:

Add following line at the top of file C:\Program Files\VisualSVN Server\httpd-wrapper.bat:

set PYTHONHOME=%~dp0\Trac\python

Update: Later updates to VisualSvn Server break Trac.

 

TracFogbugzPlugin 0.9u-py2.5 Patch for Trac 0.11

If you use both VisualSvn + Trac (or another flavour of Subversion and Trac), and tried to integrate them with Fogbugz , you will have tried the TracFogbugzPlugin.

Trying to get this to work isn’t straightforward as it’s not compatible with Trac 0.11. You will come across the following error:

Traceback (most recent call last):
File "C:\Python25\Lib\site-packages\trac\web\api.py", line 339, in
send_error
'text/html')
File "C:\Python25\Lib\site-packages\trac\web\chrome.py", line 684, in
render_template
data = self.populate_data(req, data)
File "C:\Python25\Lib\site-packages\trac\web\chrome.py", line 592, in
populate_data
d['chrome'].update(req.chrome)
File "C:\Python25\Lib\site-packages\trac\web\api.py", line 168, in
__getattr__
value = self.callbacks[name](self)
File "C:\Python25\Lib\site-packages\trac\web\chrome.py", line 460, in
prepare_request
for category, name, text in contributor.get_navigation_items(req):
File
"c:\Python25\lib\site-packages\tracfogbugzplugin-0.9u-py2.5.egg\fogbugz\fogb
ugz.py", line 22, in get_navigation_items
yield 'mainnav', 'fogbugz', Markup('<a href="%s">FogBuz</a>',
self.baseurl)
LookupError: unknown encoding: >/">https://<<myurl>>/

To fix this line 22 needs changed, to read:

yield 'mainnav', 'fogbugz', Markup('<a href="%s">FogBuz</a>'%self.baseurl)

Or download the file below and use it as part of the installation instructions (if this is your first Python Egg you want to read about installing Trac plugins. ;)

Now it is loading but the plugin doesn't seem to work, haha.

Download: Patched tracfogbugzplugin-09u-py25