Skip to main content

Senior Web Engineer. Open web / music. Remote DJ. Tall Dutch guy. #3million

micro.blog/sander

svandragt

mixcloud.com/cloudseer

 

Auto-Update Nextcloud

I run my own Nextcloud server, and every other week there's an update to the server, or one of the plugins I've enabled. The following steps enable automatic updates to the Nextcloud server.

This assumes Nextcloud is installed under /var/www/nextcloud:

Add a scheduled task as the webserver user:

$ sudo -u www-data crontab -e

Add the following line to run the upgrade script once a day at 4:05am. Instead of running the upgrade process directly we run a script so that we can also run it from the shell if needed:

5     4  *  *  * cd /var/www/nextcloud && ./upgrade.sh

Now create the script and make it executable:

$ cd /var/www/nextcloud
$ nano upgrade.sh && sudo chmod +x upgrade.sh

The script itself runs the server updater, the no-interaction argument prevents prompting for questions, followed by the occ utility to update all installed apps.

  pushd $(dirname $0)/updater
/usr/bin/php updater.phar --no-interaction
cd ..; ./occ app:update --all
popd

 

Hassle-free Python project setup

Today I managed to break pipenv again to a point where I cannot install any project requirements. Let's document the process to prevent this happening again in the future.

Python3.7 and pip are already installed on OpenSUSE Tumbleweed. If you're not so lucky, read the page on https://docs.python-guide.org/starting/install3/linux/ but stop at the bottom of the page, as I find the installation method used for pipenv there too brittle.

Instead we're going to install pipx which is best described as a per-command environment for python executables. This is great for packages like pipenv which can then run without conflicts. pipx also keeps the packages updated.

python3 -m pip install --user pipx
python3 -m pipx ensurepath

We can then use pipx to install pipenv:

pipx install pipenv

I prefer to keep the virtual environment within the project folder, by adding the following line to .bashrc or .zshrc:

export PIPENV_VENV_IN_PROJECT=1

With pipenv in place we can create a per-project virtual environment and activate it:

cd ~/dev/myproject
pipenv shell

We can install our packages into the environment now:

pipenv install pylint --dev
pipenv install black --dev --pre

The missing step in a lot of guides, is that you later might want to call your script from outwith your virtual environment, without explicitly activating it as you would do when working on the project itself. You can do that thusly:

$(pipenv --venv)/bin/python myscript.py

 

Fixing Virtualbox host network adapter cannot be created error

If when trying to start a vm, for example using vagrant, you get the error message that the host network adapter cannot be created:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["hostonlyif", "create"]

Stderr: 0%...
Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to create the host-only adapter
VBoxManage: error: VBoxNetAdpCtl: Error while adding new interface: failed to open /dev/vboxnetctl: No such file or directory
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component HostNetworkInterfaceWrap, interface IHostNetworkInterface
VBoxManage: error: Context: "RTEXITCODE handleCreate(HandlerArg*)" at line 94 of file VBoxManageHostonly.cpp

Make sure to first start the virtualbox GUI application. It does some extra checks, and sometimes highlights the permission problem:

In my case (OpenSuse 15.1 Leap), my user was not added to the vboxusers group. Via Yast > Users and Security .... and editing my user account it was a simple checkbox away. Remember to relog.

If that doesn't work, reloading the kernel extensions could help:

$ sudo dpkg-reconfigure virtualbox-dkms
$ sudo dpkg-reconfigure virtualbox
$ sudo modprobe vboxdrv
$ sudo modprobe vboxnetadp

 

Countdown - a bash countdown timer

I was looking for a tea timer countdown but quickly realised I might as well use a countdown script for this. I based my script on the linked script but added a customizable notification at the end

$ nano ~/bin/countdown
#!/bin/bash
# v2 - 2019-05-01
secs=$1
msg=${2:-$1 seconds passed.}
while [ $secs -gt 0 ]; do
echo -ne "$secs\033[0K\r"
sleep 1
: $((secs--))
done
notify-send 'Countdown Finished' "$msg"

$ chmod +x ~/bin/countdown
$ countdown 180 "Tea is ready."

 

Spotify how to hide the "Find friends" bar

On Linux operating systems, Spotify does not always show the file and view menus. This makes it harder to hide the "Find Friends" bar as there is no close button on the sidebar itself.

However there's another way: Click on the chevron in the top right corner next to your spotify username, then on Settings > Display Options > untick Show Friend Activity.

 

Install Python 3.7.1 on OpenSUSE Tumbleweed

After another hour of resolving issues with pyenv and a personal project that requires Python 3.7 I thought it best to note the steps I used to install Python 3.7.1 on OpenSUSE Tumbleweed:

# Install requirements for compiling Python
sudo zypper in zlib-devel bzip2 libbz2-devel libffi-devel libopenssl-devel readline-devel sqlite3 sqlite3-devel xz xz-devel 

# Compile Python
cd ~/Downloads
wget https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
tar xf Python-3.7.1.tar.xz
cd Python-3.7.1
./configure
make

# AltInstall ensures 3.7.1 is installed alongside your system python
sudo make altinstall

# Fix dynamic library loading
sudo ln -s /usr/local/lib64/python3.7/lib-dynload /usr/local/lib/python3.7/lib-dynload

I've updated the Common build problems · pyenv/pyenv Wiki page to reflect the updated required packages. Thanks to Andrew Cooke for the dynamic library fix

 

Howto install Cryptomator for OpenSuse

Linux is so userfriendly, so the rpm that can be downloaded is missing dependencies. However, I'm quickly becoming a fan of zypper and OpenSUSE's dependency handling, and comes to the rescue:

sudo zypper in cryptomator.rpm

Even though cryptomator made a mistake in signing part of the application, it can still be installed by pressing ignore. Are you sure you want to do that? It seems a bit crap for such an essential utility.

 

Installing Dropbox on OpenSUSE Tumbleweed

Both the official RPM and the discover package failed for me due to missing dependencies, without specifying what those are. However the following worked:

sudo zypper install dropbox-cli
dropbox start -i
dropbox autostart

 

So you want to become a neckbeard?

Having wasted too much time installing Ubuntu on a laptop recently, I thought it would be helpful for the future self to document the issues.

Issues

I'm seeing one or more of the following issues with Ubuntu 18.04 LTS:
  • Missing text on the login screen
  • second monitor switching off
  • Hangs
  • Won't boot  directly into the OS
Install Ubuntu 16.04 LTS which is not a piece of garbage.
Migrate SSH keys from Putty to OpenSSH https://askubuntu.com/questions/818929/login-ssh-with-ppk-file-on-ubuntu-terminal/818932
What are the correct permissions for the .ssh folder? Typically you want the .ssh directory permissions to be 700 (drwx------) and the public key (.pub file) to be 644 (-rw-r--r--). Your private key (id_rsa) should be 600 (-rw-------). Lastly, your home directory should not be writeable by the group or others (at most 755 (drwxr-xr-x)).
How do I connect to our team networkshare automatically on startup?
sudo su
apt install cifs-utils
echo "//path/to/windows-share /mnt/team cifs credentials=/etc/samba/credentials,uid=1000 0 0" >> /etc/stab
cat > /etc/samba/credentials <<EOC
username=UserName
password=PassWord
EOC
chown root.root /etc/samba/credentials && chmod 400 /etc/samba/credentials
Change the username and password appropriately verify your userid is 1000 with id -u UserName
I skipped  creating a passphase for my encrypted home directory, how do I do it now?  run "ecryptfs-unwrap-passphrase" in the terminal
How to stop the mouse sticking to the edges of the display https://askubuntu.com/questions/119281/what-is-the-function-of-the-sticky-edges-on-off-setting-in-di...
Can't install vagrant plugins Download vagrant from vagrantup.com
nfsd is not installed sudo apt-get install nfs-kernel-server
vagrant booting vms reports a timetout delete (rm) the IdentityFile mentioned in vagrant ssh-config

This document will get updated as I stumble and fix other issues.

App Alternatives

Clipboard manager https://hluk.github.io/CopyQ/
Remote connection manager https://snapcraft.io/pac-vs
Google Drive https://www.howtogeek.com/196635/an-official-google-drive-for-linux-is-here-sort-of-maybe-this-is-al...
Phrase Expansion https://leehblue.com/ubuntu-text-expander/ https://github.com/leehblue/texpander
Screen Recorder Kazam (https://askubuntu.com/a/29954), installs well with apt. Alternatively Simple Screen Recorder.

 

Fix slow DNS responses on Ubuntu

I've been finding that browsing has been slow for me lately on my linux box, which has Ubuntu 16.04 installed. After some investigation it seemed worthwhile to try to install a DNS cache to speed up DNS queries. This has sped up successive DNS queries from 21ms to 0ms, removing sluggishness.

sudo apt install bind9
cd /etc/resolvconf/resolv.conf.d
sudo echo "nameserver 127.0.0.1" >> head

Apply the configuration as follows:

sudo resolvconf -u

The improvement can be seen by using dig to query a domain and looking at the Query time output. After the first dig, subsequent digs return in 0 ms.

 

How to install pip and pipenv properly on Ubuntu 17.10

Ubuntu 17.10 comes with python3 3.6.3 installed by default but not pip and pipenv. We can install install pip systemwide and pipenv into the user local bin so we can use all the convenience when working with our python projects:

wget https://bootstrap.pypa.io/get-pip.py -O /tmp/get-pip.py
sudo python3 /tmp/get-pip.py
pip3 install --user pipenv
echo "PATH=$HOME/.local/bin:$PATH" >> ~/.profile
source ~/.profile

Sources: Installing Python 3 on Linux; pip installation; Installing pipenv; How to permanently set PATH on Linux

Update 9 Nov 2017: replaced curl with wget, thanks Peter

 

Fixing Samba network-share change notifications

Today a problem came up where the files on my Samba fileshare did not indicate to my Windows system that they were changed. Software that relies on this information such as TortoiseHG and SimpleLESS would not know that the file had changed. This was especially obvious with SimpleLESS which should take the .less file and process it, but wouldn’t.

It turns out that the cause of this is that on the linux system the system time was off by a large margin. Using Arbab Nazar’s article on setting up time synchronization fixed the issue and prevents it from reoccurring.

 

Setting up a VM based development system - Part 1

I recently set up a local development server running Ubuntu on VMware player. I thought it would be interesting to document the process and instructions so that it will be useful to others, and you can follow along / correct / improve this guide. On completion of this series you will end up with a complete LAMP local development system..

Installing Ubuntu

You will need:

Install VMware player (VMP). Restart the pc if necessary.

I’m installing the virtual machine on my USB harddisk, this way I can take my whole development with me on the go (home/work) - this is great because you only have to set all this up once, instead of on every machine you plan to use it on.

Start VMP and “Create a New Virtual Machine” and browse to the downloaded .iso. As part of the Easy Install Information write down the login:

username: __________ password: __________ 

Store the VM on your removable drive.

On the Specify Disk Capacity screen you will want to go higher than the recommended disk size, I chose double (40GB). Make sure you have enough free space available, check this before continueing. I went for double the recommended amount as I want to avoid to repartition later. Accept the other defaults and Ubuntu will start. Keep waiting as Easy Install is installing Ubuntu on your behalf.

If you are a chmod 777 user (jackpot settings), this is a great time to read up on file system permissions by reading the article series starting with “Linux File Permission Concepts” (click next in the summary, there are 5 articles in total): http://articles.slicehost.com/2010/7/17/linux-file-permission-concepts

After a restart the login screen appear and after a successful login you will see the desktop. Because we are going to install a bunch of software let’s make sure everything we already have is up to date. Start the update manager > Settings > Ubuntu Software > Other > Select Best Server. This will speed up downloads. Check again and install updates.

Time to read some more, btw have you created a new tag for all these bookmarks yet? When you are finished reading the above articles, here are some really good resources to read later:

You should now have a running Ubuntu system.

(Source: donationcoder.com)

 

Slicehost Articles: Checking Linux file permissions with ls

Once you understand Linux file permissions, the next step on the road to enlightenment is learning how to check the permissions for a file or directory.

via http://articles.slicehost.com/2010/7/17/checking-linux-file-permissions-with-ls&hl=en&strip=...">Slicehost Articles: Checking Linux file permissions with ls

 

Slicehost Articles: Linux file permission concepts

Linux file permissions are strange and wondrous things. Start down the path of understanding by looking at the core concepts behind them before moving on to practical applications.

via Slicehost Articles: Linux file permission concepts

 

Replace all copies of a file under Linux

Today, after making a change to a php file, I had to update all copies of this file with the changes. The following command will update all files with the same name. Note that this will only work as expected if the name is unique. A theoretically better way to do it would be to find all files with a certain hash.

Anyways, here's the linux shell command. First copy the updated file into the current directory (/tmp for example):

find /target_path/ -iname "some_file.php" -exec cp new_file.php '{}' \;

 

Serve.sh updated for terminal use

After publishing the previous article about serve.sh - the shell script I created to easily serve websites in development - I made some changes to make it easier to run it from the terminal, for example over ssh. I thought I'd share it with you.

One of the problems with the original script was that it was optimized for a GUI setting - ie. start the script by clicking on it from the desktop. However usually I want to run it from the terminal. I don't like typing so I added code to make the script available from any directory to the ~/.bashrc:

export PATH=$PATH:~/bin

Then moved it to that path and renamed it to shave another 3 characters off the syntax:

mv ~/Desktop/serve.sh ~/bin/serve

In addition I noticed that prompting which site should be served was just a workaround for commandline arguments, which are now added. It doesn't check that you entered an argument though. So the new script now requires you to specify which directory you want to serve: to serve the current directory simply run:

serve .

Latest script contents:

#!/bin/bash
my_path=`readlink -f $1`
sudo rm /var/www
sudo ln -s $my_path /var/www
echo Now serving $my_path...

 

Serve.sh - A shell script for serving sites

I am experimenting with using a Linux virtual machine as my web development environment of choice. I store the vm on a removable drive so that I can develop from any location, without having to setup a working environment. Previously I had to check out the repositories, setup a local webserver and I had trouble keeping things working, because every configuration change had to be applied in every location. Now it is all centralized and my life is simpler.

The aim is to make working on projects as easy as possible. I have all projects checked out in a folder called /var/sites. They are mostly PHP projects and because of my shared hosting environment they share a single apache configuration. How can I easily serve them? Having seperate virtualhosts for each project would result in me having to make manual changes on every location again, so this was not the way to go.

Instead I created a simple shell script that creates a symbolic link from the Apache's webroot to the project I am working on:

[gallery link="file"]

#!/bin/bash
echo "Available sites:"
ls  /var/sites
echo
echo -n "Type site to serve: "
read site
if [ -z "$site" ]
then
exit
fi
sudo rm /var/www
sudo ln -s /var/sites/$site /var/www

 

Rotate PHP logs

Our php.log was nearing 550MB so I was investigating how to rotate the logs. The easiest solution seems to be using logrotate (ubuntu linux) with a script like follows.

sudo nano /etc/logrotate.d/php5
/var/log/php5/*.log {
   daily
  daily
   14 days
  rotate 14
  missingok
  nocompress
  sharedscripts
  postrotate
  apache2ctl graceful
  endscript
}

You can check the configuration by running logrotate --force /etc/logrotate.d/php5

 

 

Resume RSync until finished

I noticed a backup of ours did not finish and that the temporary file rsync uses was still listed. A quick google search indicated that rsync had lost its connection. Thanks to Ian Young's article on the subject rsync now resumes when cut off and hopefully the backup will complete now.