Logging Structured Data From an Application

What is structured data?

Structured data is data formatted in a structured manner so that the sender can clearly communicate to the receiver each field/property/part of each message without confusion or ambiguity about where the message starts/stops, and what each field represents and it’s value.

We usually represent structured data with fields or key value pairs. The data can even be represented in a tree such as in JSON or XML.

Logging Flavours

RFC 3164 doesn’t know about structured data:

<34>Oct 11 22:14:15 mymachine MyApplication: This is the message part

RFC 5424 supports structured data:

<74>1 2017-07-11T22:14:15.003Z myhostname evntslog - ID47 [exampleSDID@12345 x="y" eventSource="MyApplication" eventID="6789"] This is the message part

systemd journal supports structured data

Logging Methods

syslog(3) designed for RFC 3164, doesn’t know about structured data.
logger(1) can log with RFC 3164 or RFC 5242, and supports structured data, but just adds it as part of the message field unless you send it to a socket
liblogging mentions structured data in the readme, but it hasn’t been implemented
sd_journal_send/sd_journal_sendv logs to the systemd journal, supports structured data

…And the most common method of getting structured data into logs

Adding key/value pairs or JSON to the message part of any logging system. This works in any logging protocol, here is an example of RFC 3164 with JSON in the message part:

<34>Oct 11 22:14:15 mymachine MyApplication: This message has JSON in it { "x": "y", "age": 123, "flag": true }

This works universally for RFC 3164, RFC 5242 and the systemd journal.

With CEE:

<34>Oct 11 22:14:15 mymachine MyApplication: @cee: { "x": "y", "age": 123, "flag": true }

Adding JSON to the message part is the lingua franca of logs, it works everywhere, even with old APIs, adding the CEE signature makes the JSON payload easy to identify.

See Also

https://rsyslog.adiscon.narkive.com/rrLJiWRs/best-practice-for-an-application-to-get-structured-data-to
https://techblog.bozho.net/the-syslog-hell/
https://sematext.com/blog/what-is-syslog-daemons-message-formats-and-protocols/
https://github.com/systemd/systemd/issues/19251

Creating a Linux Container in C++

I love Linux containers. You get a degree of separation from the host and other machines without hosting a full VM and without requiring a whole second OS.

What are containers?

A process or group of processes that are running on the same kernel as the host but in isolation via namespaces, cgroups and images

Why would you do this?

  • Share resources (3 web servers in containers on one physical server for example)
  • Less overhead and quicker to spin up than a VM
  • To experiment in Linux without modifying the host OS settings or filesystem
  • To run software or use libraries that cannot or shouldn’t be run on the host (Mismatched versions, tries to read data from your home folder, tries to mess around with other processes, spams syslog, tries to phone home or query the network, or talk to Linux kernel modules)
  • Consistent behaviour (Developers can run wildly different machines but develop within identical containers, consistent continuous integration, simulate end users’ machines for debugging)
  • Run a collection of containers that can talk to each other but can’t talk to the rest of the network (Local testing, integration testing, load testing, mess around with network configurations and firewall rules without endangering the host)
  • Run a collection of containers that can talk to each other but only one or two of them are public facing (LAMP stack, ELK stack, a cluster, etc.)
  • Set limits on memory usage and disk usage (Restrict resource heavy applications or test out of memory situations without killing the host)
  • Share folders and even whole Linux kernel modules from the host into the container.

Why wouldn’t you do this?

  • A VM or separate machine is generally a better approach if you have the resources
  • A VM can be more secure due to better isolation, not sharing a kernel for example
  • A VM can be more stable due to not sharing the kernel and kernel modules
  • Containers must be the same architecture as the host
  • Applications and libraries in a container have to be created for a version of the Linux kernel that is compatible with the host

Fair enough. How do containers work?

Namespaces, cgroups and images are set up to provide an environment for the container’s processes to run in.

  • Namespaces – Isolate the processes from the host, giving it it’s own filesystem, process tree, network, IPC, hostname, and users
  • Cgroups – Set limits for CPU usage, memory, disk and more for the container
  • Images – An image or folder containing the applications, libraries, and data that a container needs to run, for example cut down all the way up to fully featured distros: busybox, alpine, centos, ubuntu, etc. (Docker and LXC allow combining images via an overlay so you can get alpine + java + nifi for example which a smushed together into a single filesystem

Great. Now what?

What does this look like in code?
Here is my C++ rewrite of Lizzie Dixon’s excellent C container. It demonstrates setting the cgroups, creating the namespaces, mounting the busybox filesystem, and launching the child process.


NOTE: This is just a proof of concept, it doesn’t have basic, nice things like networking, package management, file system overlays, any sort of Dockerfile/makefile/recipe support, code reviews, testing, battle hardened development history, etc. It’s not Docker, it’s a proof of concept of the basics of how containers work that doesn’t hide everything away in the Go language and libraries or a “docker run” command.

Here is an example of it running:

root@laptop:~/cpp-container# BUSYBOX_VERSION=1.33.0
root@laptop:~/cpp-container# ./cpp-container -h myhostname -m $(realpath ./busybox-${BUSYBOX_VERSION}/) -u 0 -c /bin/sh
=> validating Linux version…4.3.0-36-generic on x86_64.
Starting container myhostname
=> setting cgroups…
memory…
cpu…
pids…
done.
=> setting rlimit…done.
=> remounting everything with MS_PRIVATE…remounted.
=> making a temp directory and a bind mount there…done.
=> pivoting root…done.
=> unmounting /oldroot.kCeIkg…done.
=> trying a user namespace…writing /proc/3856/uid_map…writing /proc/3856/gid_map…done.
=> switching to uid 0 / gid 0…done.
=> dropping capabilities…bounding…inheritable…done.
=> filtering syscalls…done.
/ # echo "Look ma, no docker"
Look ma, no docker
/ # whoami
root
/ # exit
Container myhostname has exited with return code 0
=> cleaning cgroups…done.
root@laptop:~/cpp-container#

Some Simple Regexes for Finding If Statements

I know almost nothing about regexes, these are just some useful ones I created for checking code style, if statements specifically.

Find single line if statements (Useful when your coding style requires adding curly braces to everything):

/^ *if \(.+\).*(;)/igm

regex101

Find multiple line if statements (Useful when your coding style favours less lines):

/^ *if \(.+\).*({)/igm

regex101

Regex101 is really impressive.  You can paste a regex and check it against some test inputs.  The best part is the description of what each symbol in your regex does and even what some of the combined parts do, such as various brackets around groups of expressions.

Diesel Photo Manager 0.1 Released

I couldn’t find a photo browser for Linux that also imported my photos in the folder format I wanted so I created my own.

This is the first release and it has these features:

  • Simple interface
  • Thumbnail grid mode
  • Single photo mode
  • Import and sort photos into folders based on the date
  • Convert files to dng
  • Show the background thread progress on the statusbar
  • Check for updates via version.xml
  • Default to dark theme (Can be overridden with the –light command line parameter)

Still to come:

  • Fixes to the single photo viewing mode
  • Popup notifications when a new track is played
  • Work out what I want to do with the photo browser

Thumbnail browsing mode
Thumbnail browsing mode

Download
Source

Medusa Music Player 0.9 Released

I’ve release a new version of Medusa music player with these features:

  • Simple interface
  • Last.fm support
  • Date added and a full file path column
  • Drag and drop from the file manager
  • Moving files to the trash folder
  • Move files to a particular folder and remember the last 5 folders
  • Extra columns
  • Show the background thread progress on the statusbar
  • Import playlist from Rhythmbox and Banshee
  • Autoplay at last track at startup
  • Web server for controlling playback, volume and deleting tracks remotely
  • Check for updates via version.xml
  • Popup notifications when a new track is played

New:

  • Various bug fixes
  • Web server bug fixes
  • Default to dark theme (Can be overridden with the –light command line parameter)

Still to come:

  • Easy tag editing for multiple files at once

Dark Theme
Dark theme

Download
Source

Medusa Music Player 0.8 Released

I’ve release a new version of Medusa music player with these features:

  • Simple interface
  • Last.fm support
  • Date added and a full file path column
  • Drag and drop from the file manager
  • Moving files to the trash folder
  • Move files to a particular folder and remember the last 5 folders
  • Extra columns
  • Show the background thread progress on the statusbar
  • Import playlist from Rhythmbox and Banshee
  • Autoplay at last track at startup

New:

  • Web server for controlling playback, volume and deleting tracks remotely
  • Check for updates via version.xml
  • Popup notifications when a new track is played

Still to come:

  • Easy tag editing for multiple files at once

Medusa Main Window
Medusa Main Window
Download
Source

Medusa Music Player 0.7 Released

I’ve release a new version of Medusa music player with these features:

  • Simple interface
  • Last.fm support
  • Date added and a full file path column
  • Drag and drop from the file manager
  • Auto play last track at startup
  • Moving files to the trash folder
  • Move files to a particular folder and remember the last 5 folders

New:

  • Extra columns
  • Show the background thread progress on the statusbar
  • Import playlist from Rhythmbox and Banshee
  • Add autoplay at last track at startup

Still to come:

  • Easy tag editing for multiple files at once

Medusa Main Window
Download for Linux
Source

Medusa Music Player 0.6 Released

I’ve release a new version of Medusa music player with these features:

  • Simple interface
  • Last.fm support
  • Date added and a full file path column

New:

  • Drag and drop from the file manager
  • Auto play last track at startup
  • Moving files to the trash folder
  • Move files to a particular folder and remember the last 5 folders

Still to come:

  • Easy tag editing for multiple files at once

Medusa Main Window
Download for Linux
Source

Medusa Music Player 0.5 Released

I’ve been working on a music player for Linux. What I really wanted was foobar2000 for Linux but nothing on Linux is quite like it. There are lot of iTunes clones (Amarok, Banshee, Songbird, Rhythmbox, etc) but they are all too complicated or make copies of your music or show you lyrics and Wikipedia articles and include a web browser. I just want a simple music player. I can list the features I want one hand (I’ve only implemented the struck through items so far):

  • Simple interface (Most are not simple but can be configured to be simple)
  • Last.fm support (They all do this)
  • Date added and a full file path column (You would be surprised at just how many music players do not have these columns but they are so useful for sorting by)
  • Moving files to the trash folder (Most can do this)
  • Easy tag editing for multiple files at once (Most are ok at this)
  • Move files to a particular folder and remember the last 5 folders (Useful for remembering which songs I like by putting them in a “Collection” folder. I haven’t seen any other players do this)

This is a very early version, I’ve only implemented 3 out of 6 of the proposed features. By the time I release a 1.0 version I hope to have all of these features implemented.
Medusa Main Window
Download
Source

Converting Pentax PEF Files to PNG

Ufraw is a fantastic utility to convert raw camera formats. You can install it via (And you may as well get the plugin for gimp while you are at it):

sudo yum install ufraw ufraw-gimp
OR
sudo apt-get install ufraw gimp-ufraw

There is a great tutorial on using ufraw from a bash script here, my only recommendation is converting to PNG but it is entirely up to personal preference.

PNG version:
pef2png.sh

#!/bin/bash
 
if [ ! -d ./processed_images ]; then mkdir ./processed_images; fi;
 
# processes raw files
for f in *.pef;
do
  echo "Processing $f"
  ufraw-batch \
    --wb=camera \
    --exposure=auto \
    --out-type=png \
    --compression=96 \
    --out-path=./processed_images \
    $f
done
 
cd ./processed_images
 
# change the image names
for i in *.png;
do
  mv "$i" "${i/.png}"_r.png;
done
for i in *.png;
do
  mv "$i" "${i/imgp/_igp}";
done

Usage:

# Convert all pef files in the current directory to png
./pef2png.sh

No root file system is defined

On the weekend I was installing the new version of Mythbuntu (More interesting screenshots here) and I had a weird error, “No root file system is defined”. At first I thought it must have been something to do with failing to recognise the existing partitions or possibly they were corrupt. “fdisk -l” worked, returning sda, sda1-sda4, which was correct however “mount” would always fail. It turned out that it was just our old friend dmraid was breaking in new and unexpected ways. Here is how to worked around it:

Boot into live CD mode
Remove dmraid via Package Manager
Run Install Mythbuntu from the desktop shortcut

On multiple distributions and motherboards I consistently have problems with dmraid not finding/incorrectly identifying partitions/drives. I’m not the only one with these problems. I’m sure I am having these problems because I have raid hardware but am not using raid. Surely raid is an advanced enough feature that people with raid should be expected to know to install/add it? Perhaps the install could be attempted without raid support and the installer can say “Do you use raid?” or “Are these devices correct?” and at this point the installation restarts/redetects with dmraid enabled.

How To Automatically Download All of Your YouTube Favourites

FireFox add-ons such as 1-Click YouTube Video Download are good but one must visit every video and manually save it. I have a lot of favourites and I wanted to back them up with as little effort as possible. Here is how.

You will need youtube-dl. It is available via yum/apt-get however the version provided may be quite old (In Fedora 14 it did not support –playlist-start and –playlist-end).

Create a new playlist and add all the videos from your favourites (Note: A playlist can only contain 200 videos if you have more than than this you will have to make multiple playlists and repeat this process for each one).

Click on “Play All”. Copy the URL from the address bar for the first video when it tries to play (You only need this bit: “http://www.youtube.com/view_play_list?p=AE198A14013A3005”).

Now tell youtube-dl to download the videos in the playlist:

python ./youtube-dl.py --title --ignore-errors --playlist-start=1 --playlist-end=200 http://www.youtube.com/view_play_list?p=AE198A14013A3005

If you had more than 200 videos repeat this process for the ones that didn’t fit in the playlist.
This should work for other people’s playlists too.
You might want to make a note of the latest video in your favourites so that the next time you can backup only the new videos.
This process still requires some user interaction, if you find an even easier way where I can just say, “This is my user name, make it happen”, I’d love to know about it.

Git: Review the Changes in a file and Optionally Revert A Particular Change

# Review all the changes in a file, giving us a chance to revert a particular change
git checkout -p path/to/file/to/partially/revert

Note to self: Move this to a useful Git tips post.

malloc double free/non- aligned pointer being freed set a breakpoint in malloc_error_break to debug

malloc: *** error for object 0x3874a0: double free
***set a breakpoint in malloc_error_break to debug

malloc: *** error for object 0x18a138: Non- aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug

So basically something in your code is screwing around with memory.

Either releasing something that has already been released:

int* x = new x[10];
delete [] x;
delete [] x;

Or releasing something that is not pointing to the start of an allocated block of memory:

int* x = new int[10];
x++;
delete [] x;

The error message isn’t very clear if you have no experience with GDB. GDB is a debugger for your binaries. It allows you to set break points at the start of a function and any time that function is called your application will pause and allow you to debug in GDB. We can then get valuable information back by executing commands to get the backtrace, registers state and disassembly. The advantage of using GDB over Xcode/KDevelop is being able to break into any function, not just functions in your source code. Anyway, this is how I got the backtrace to find out where in my sourcecode I was making a mistake:

gdb
file myapplication
break malloc_error_break
run
backtrace

Now whenever a double free or non- aligned pointer is freed it will break into gdb and we can type in “backtrace” and work out what our code did to trigger this.

Cannot open /dev/sda1: Device or resource busy

After an update (Upgrade?) a while ago I couldn’t boot into Fedora, it had the text mode bar graph and after getting to 100% it failed with this error message:

Cannot open /dev/sda1: Device or resource busy

It turned out that this was a dmraid problem. It would appear that something changed when updating and added or enabled dmraid. So I had to find a way to remove or disable it, the simplest solution I found that worked was disabling it via the arguments to the kernel in GRUB.

Edit menu.lst (Or grub.conf, my menu.lst is a symbolic link to grub.conf)

su
gedit /boot/grub/menu.lst

Find the entry that you are currently booting into and add “nodmraid” to the end of the “kernel” line:

	kernel /vmlinuz-2.6.31.6-166.fc12.x86_64 ro root=UUID=7129c2cc-03c5-4b7a-8472-bb9d314446b3  LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet nodmraid

I also like to add a timeout and remove hiding of the menu (This is in the general entry at the top of the file):

timeout=10
#hiddenmenu

My final menu.lst/grub.conf file looks like this:

default=0
timeout=10
splashimage=(hd0,1)/grub/splash.xpm.gz
#hiddenmenu
title Fedora (2.6.31.6-166.fc12.x86_64)
	root (hd0,1)
	kernel /vmlinuz-2.6.31.6-166.fc12.x86_64 ro root=UUID=7129c2cc-03c5-4b7a-8472-bb9d314446b3  LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet
	initrd /initramfs-2.6.31.6-166.fc12.x86_64.img

Why Chris, your grub.conf is so puny, only one entry in it. Leave me alone, I only just reinstalled.

How to Find and Remove Folders Recursively in Linux/Unix Because I Always Forget This

# Recursively search for folders called .svn and delete them (Even if not empty)
find . -name .svn -type d -print | xargs rm -rf
 
# Recursively search for files called *~ (gedit creates these for temporarily saving to) and delete them
find . -name \*~ -print | xargs rm -rf
 
# Recursively search for files called *.h or *.cpp and print them
find -name '*.h' -o -name '*.cpp' -print

I Love the Idea of a New Smart Phone

The list of awesome phones:
HTC Phones
OpenMoko
Palm Pre

However, a new hotness has just been born.

The Nokia n900.

32 GB internal storage
Expandable to up to 48 GB with external microSD card
3.5G mobile network
Quadband GSM with GPRS and EDGE
Data transfers over a cellular network 10/2Mbps
Data transfers over Wi-Fi 54Mbps

Flash 9.4 support (In your face iPhone users, Nokia loves me)

5-megapixel (2584 × 1938 resolution) digital camera (iPhone is up to 3.2-megapixel)
800 × 480 resolution video recording
Dual LED flash

800 × 480 resolution screen
Tactile and onscreen QWERTY keyboards (Yes, none of this onscreen keyboard rubbish)
Removable battery (That probably won’t explode)

Assisted-GPS receiver
Ovi Maps pre-installed

TV out (PAL/NTSC) with Nokia Video Connectivity Cable (CA-75U, included in box) or WLAN/UPnP

Wide aspect ratio 16:9 (WVGA)
Video recording file format: .mp4; codec: MPEG-4
Video recording at up to 848 × 480 pixels (WVGA) and up to 25fps
Video playback file formats: .mp4, .avi, .wmv, .3gp; codecs: H.264, MPEG-4, Xvid, WMV, H.263

Music playback file formats: .wav, .mp3, .AAC, .eAAC, .wma, .m4a

The important part:
Development in C and Python are supported, using GTK.
Unfortunately it looks like C++ is not supported so anyone wanting to do any OOP will have to do it in Python.
That is great that Nokia are supporting open standards, releasing a mobile phone running Linux. It would be nice if they supported C++ too, but that is just my personal preference, and it may come in the future, I know all the C++ jazz is quite complex and definitely non-trivial.

I’d still rather use Python than C. I certainly hope that Windows Mobile dies and the highly overrated iPhone becomes less popular, not completely die, just put in its place. I would love to see the world embrace open standards, open source software and a rise in DRM free, happy, feel good devices, gain popularity and eventually dominate the market and warm people’s hearts.

Moving from SVN to GIT

So I have a few projects on SourceForge, but they’re all hosted via SVN. With all this distributed version control going on I thought I would like to get in on the action. I thought about moving to github, but what happens in 5 or 10 years when I want to move on to another revision control software? The name has a limited life expectancy. Anyway so I wanted to switch so I researched alot, this is the best information I found and this blog entry is basically a rehash specific to SourceForge (Because I didn’t find any good SourceForge specific information).

Note: For this tutorial you will want to change all occurrences of USERNAME to your user name for example “pilkch”, PROJECTNAME to your project name for example “breathe” and YOURFULLNAME to your full name for example “Chris Pilkington”.

First of all we need to download git and git-svn:

su
yum install git git-svn

Now we are going to create our repo directory for holding our repositories:

cd ~
mkdir repo

Create a users.txt file to map our subversion users to git users:

users.txt

USERNAME = YOURFULLNAME <USERNAME@PROJECTNAME.git.sourceforge.net>

As the other article says, we basically check out a svn directory as a git repository:

cd repo
mkdir PROJECTNAME_from_svn
cd PROJECTNAME_from_svn
git svn init http://PROJECTNAME.svn.sourceforge.net/svnroot/PROJECTNAME/PROJECTNAME --no-metadata
git config svn.authorsfile ../users.txt
git svn fetch

Check that worked (Just read the last few changes to make sure svn history is present, you can hit spacebar to scroll back a page of history or two just to make sure):

git log

From now on we can use git commands, first of all want to create a copy of the git-svn repository:

cd ..
git clone PROJECTNAME_from_svn PROJECTNAME

PROJECTNAME/ now contains our “clean” repository and PROJECTNAME_from_svn can be deleted if you like. We now just need to add and push our local repository to the remote location:

cd PROJECTNAME
git config user.name "YOURFULLNAME"
git config user.email "USERNAME@users.sourceforge.net"
git remote rm origin # This may not be necessary for you
git remote add origin ssh://USERNAME@PROJECTNAME.git.sourceforge.net/gitroot/PROJECTNAME/PROJECTNAME
git config branch.master.remote origin
git config branch.master.merge refs/heads/master
git push origin master

Now to check that this is working you can browse to the git page of your SourceForge project and there should be data in your repository. And we can clone our repository back again to check that everything is working.

git clone ssh://USERNAME@PROJECTNAME.git.sourceforge.net/gitroot/PROJECTNAME/PROJECTNAME

You may also want to ignore certain types of files, place a file called .gitignore in the root directory of your project and fill it with the patterns you want ignored:

.gitignore

.DS_Store
.svn
._*
~$*
.*.swp
Thumbs.db

Now when we want to update we can do:

git commit -a -m "This is my commit message." # All changes to the local repository need to be committed before we try merging new changes
git pull # Grab any changes from the main repository

Committing is slightly different:

git add .gitignore # For example we might want to add our new .gitignore file
git commit -a -m "This is my commit message." # Note: Your commit has now only been staged, it is not in the main repository yet
git push # Now it is pushed into the main repository

The last step is to remove your svn repository which for SourceForge is as simple as unchecking a checkbox on the Admin->Features page.

Symbols/Characters Returned by ls -l

$ ls -l
total 24
drwxrwxr-x   ...   thisisadirectory
-rw-rw-r--   ...   thisisafile
lrwxrwxrwx.  ...   thisisalink -> /media/data

We know what the rwx fields are but what about d, – and l? Ok, those are pretty obvious too, here is a list of the more obscure ones because I always forget.

d Directory.
l Symbolic link.
Regular file.

b Block buffered device special file.
c Character unbuffered device special file.
s Socket link.
p FIFO pipe.
. indicates a file with an SELinux security context, but no other alternate access method.

s setuid – This is only found in the execute field.
If there is a “-” in a particular location, there is no permission. This may be found in any field whether read, write, or execute field.

The file permissions bits include an execute permission bit for file owner, group and other. When the execute bit for the owner is set to “s” the set user ID bit is set.
This causes any persons or processes that run the file to have access to system resources as though they are the owner of the file. When the execute bit for the group is set to “s”,
the set group ID bit is set and the user running the program is given access based on access permission for the group the file belongs to.