Room temperature logging with data visualization

So this is what nerds do in their spare time...

A while ago I've started thinking about creating my own intelligent room thermostat with a Raspberry Pi, because Nest just wasn't there, and I believe you can save a lot with a learning thermostat, and my experience with a couple of them supports this idea.

Had to start somewhere, so first I wanted to measure room temperature, and see what exactly happening depending on the weather, sunlight, etc. Purchased a USB thermometer called TEMPer2 on e-bay, and put together a small system. This is the result, the current state of things:

Will go into the details later, for now these are the basics:

1. A small ruby script reads the temperature from the USB thermometer, and a cronjob saves the JSON output of it into a log file.
2. The log file is monitored with a local instance of fluentd, which forwards the data read to my VPS also running fluentd.
3. On the VPS, fluentd stores the data in ElasticSearch.
4. The data in ElasticSearch is visualized by running Kibana on the VPS.

Docker Tooling in Zend Studio

It's been a while since my last post, but today I've discovered something, and felt like sharing it.

In Zend Studio, you can install support tools for using Docker. You can list and manage containers and images with it.
Go to the Welcome page (Help menu -> Welcome), and select Docker from the Enhancements block, and click Apply changes. The IDE will ask for a restart, and when it comes back, you will have a new perspective called Docker Tooling, and some new views, Docker Explorer, Docker Images and Docker Containers.
To start using them, click the link in the Docker Explorer saying No connection ... click here etc., and in the popup dialog specify a name for the connection, select the TCP Connection option, and enter the IP address and port of your docker machine in the format of tcp://xxx.xxx.xxx.xxx:yyyy
In my case on my Mac I got the URI from the CLI for Docker using the printenv shell command, from the value of the DOCKER_HOST environment variable, so the result URI was tcp://192.168.99.100:2376. Enable authentication and enter the path from the DOCKER_CERT_PATH environment variable, which in my case was /Users/tom/.docker/machine/machines/default, then click Test Connection. If you see Ping succeeded message, you can click Ok and then Finish, and voila! you have your docker machine in the Docker Explorer with the containers and images.
I wish there was support also for docker-compose, but hey, you don't always get what you want.

Debugging PHP CLI scripts and phpunit unit tests in Eclipse with XDebug

Nowadays I've spent some time at stackoverflow.com, mostly because I wanted to give back something in exchange for the many solutions found there in the last few years. Browsing the questions found an interesting one asking if it was possible to debug unit tests.
Maybe another post will be about how to install phpunit with PEAR, but let's assume you already have it installed. As of writing this post, the actual Eclipse version is Indigo 3.7.2 with PDT 3.0.0.
You will need XDebug, if you use debian or ubuntu, the package is called php5-xdebug.
Without going into the details - because a couple other guys have already done that, just google 'eclipse xdebug' - I'd like to point out some important things. First, make sure you have everything installed and set up, you can follow instructions described in this document.

So, here come the important bits:

- You must have a phpunit.php in your project to debug unit tests. In ubuntu, you can copy /usr/bin/phpunit to your-project/some-dir/phpunit.php

- Add the PEAR directory as a PHP user library to Eclipse (Preferences / PHP / PHP Libraries), and add this library to the include path of your projects (Project Properties / PHP Include Path / Libraries).

- Eclipse - and PDT - runs PHP executable with a custom php.ini file, and the -n command line switch, so you have to create a full php.ini containing all the .ini files from the conf.d directory. The best way to realize this is to notice that your extensions are not loaded when running scripts within Eclipse, or you see that the debugger does not stop on breakpoints, because the xdebug extension is not loaded either.
Create a directory somewhere and concatenate all the .ini files together. You can do it running a command like this:
cat /etc/php5/cli/php.ini /etc/php5/cli/conf.d/* > /usr/local/etc/php5/php.ini
Set up the PHP executable in Eclipse (Preferences / PHP / PHP Executables), and specify this php.ini file location. After this, exit Eclipse and start it up with the -clean option.

- Make sure that your bootstrap.php file is set up to be run from anywhere, even from another working directory. Avoid including files like require_once('../classes/someclass.php'); use absolute paths instead, or start from the bootstrap.php file's directory to build absolute paths.

- Open a unit test file in Eclipse, and click Debug Configurations from the Run menu. Create a PHP CLI Application configuration. Select the PHP runtime you've just set up, and specify your phpunit.php file's as the 'PHP File', use relative path from your workspace directory, or just use Browse. Also, it may come handy to set the 'Break at first line' to on.
Go to PHP Script Arguments tab, and set command line options for phpunit you want to use (A 2 cent tip: don't use '--colors' in Eclipse) and the unit test's file name you want to debug. Usually the arguments would include
--configuration '/path/to/your/phpunit.xml'
or
--bootstrap '/path/to/your/bootstrap.php'
You can use the variables defined to avoid setting full paths here:
--configuration "${workspace_loc}/my-project/test/phpunit.xml" "${resource_loc}"
The '${resource_loc}' is a full path to the file you're currently debugging.

When you're done, just click Apply then Debug, and start debugging.

A tip for using Doxygen for PHP code

In the last few days I've spent some time researching for a documentation tool for my PHP sources, and after running a couple of circles, I've ended up using Doxygen.
It had some issues though, especially referring to namespaces in the docblock comments, because the docblock syntax allows using backslashes for commands, e.g. \param instead of @param.
I really don't understand why PHP's developers decided to use backslashes as separators for namespaces, but that's a different story.
So, if you use namespaces in docblock comments, you have to escape backslashes, but I didn't want to do that in my code, so I've created an input filter for Doxygen in PHP which does that for me.
It also appends variable names to @var type declarations, so e.g.
/**
 * @var string
 */
public $name;
becomes
/**
 * @var string $name
 */
public $name;
which would be redundant in your code, but Doxygen seems to prefer this format.

I've added the filter to a Bitbucket repository, find it here:
https://bitbucket.org/tamasimrei/misc-tools/src/master/doxygen/filter.php

Licensed under the MIT License.

See the Doxygen documentation on how to use an input filter, but basically it's just adding a line like this to your Doxyfile config file:
INPUT_FILTER = /home/tom/misc-tools/doxygen/filter.php

Comments are welcome.

Find files with Windows line endings

Just a quickie with some bash magic... I wanted to eliminate mixed and windows line endings from my source files, and after some research and development I got this command line:

find . -not -name *.svn-base -exec grep -Ils $'\r$' {} \;

You must love bash and all of these amazing you-can-do-anything commands. Just takes some time to learn them, and to be able to use them without heavy googling.

The mystery of slow XML loading - solved

Last week I've spent some time installing a VMWare hosted linux server onto my Windows 7 PC in the office, mostly because our development server is under a heavy load for several things, e.g. a Jenkins CI instance.
After some experimenting with Debian stable (squeeze) and testing (wheezy), I've ended up using Ubuntu Server 10.04 LTS, long story short because of the support from VMWare, and the available 2.6.x kernel (had network issues with the 3.2.x kernel).
It worked as a charm, installing the whole thing together with Apache, PHP5 and the required PHP modules didn't take more than 30 minutes, the testing phpinfo() page jumped into my browser like a squirrel on red bull.

I've checked out the source of one of our projects, stoked to see how it'd work with my VM. Set up virtual host, done. Configure app, done. Let's open it in a browser. Surprise! One page load took about 2 minutes. WTF?!?
OK, let's see CPU load on the VM. None.
OK, it must be HDD load then. None.
OK -I'm not giving up- the network is slow again. No.
OK, the VM apache is waiting for the response from database server. No.
OK, then what the fudge is happening? Let's run some benchmarks. Phpinfo page is loading like in no time.
OK, so it must be something with our stuff, but what? At this point I remembered that back in the old days I had a story with configuring the Exim MTA, and it had a network wait timeout...
OK, let's see what's happening on the network. After starting up iptraf, I've noticed that there's an external IP which is connected to the VM. The IP address is pointing to a w3c.org host, let's google that. And voila! after a few minutes I've learned that the PHP DOM extension is set to load the DTD for loading and parsing XML files... Our app has a setting to disable DTD loading, but I didn't like that idea, and a comment on a PHP documentation page said that the DTD's can be stored locally. At this point - 8.30PM, still in the office, feeling like a zombie - I've asked our lead developer - who was also still in the office :) - how can I have solve this thing. He pointed me to a package called w3c-dtd-xhtml holding the DTD's. After installing it, our app was running like crazy, no waiting, I could even copy the database from the dev server, and run it locally, it's still fast enough.

Today I've installed Apache on the Windows host, and set it up to reverse proxy the VM out for the others in the office, and it runs smoothly.

How could I be that stupid to say Windows when I was asked what OS I'd like to use on my PC, when almost everything I use for development is available on linux now...
... not that I wouldn't have the same issue on a linux desktop tho :)

OpenLDAP after upgrading debian

Long story short: my friend upgraded his debian server, and asked me to help him upgrading OpenLDAP (slapd), because it didn't start. Not that I was a big expert of slapd - on the contrary - but I was the one who configured it for the first time, maybe I could make it fly again.

Started restarting slapd (/etc/init.d/slapd restart), no success. Let's read the logs. Syslog entry:

bdb(dc=future,dc=neologik,dc=hu): Program version 4.8 doesn't match environment version 0.66

OK, let's Google. Found a couple of posts of I need to upgrade the database (quite funny, the "environment version" refers to the database version, duh?), but all the dbX.X_upgrade tools said the same as slapd, about the not matching versions.

OK, let's Google, round 2. Finally found a post(1) talking about bumping into the same issues, and found no other solution than re-importing the LDAP data from an .ldif file. Fortunately, the good guys at debian made the upgrade script so it creates a backup .ldif file in /var/backups.

So, I've purged slapd to have a clean start, reconfigured the domain and the admin password (not going into the details, if you are reading this post, you know how to install/purge a package in debian ;) and tried to run:
ldapadd -h localhost -x -W -D "cn=admin,dc=example,dc=com" -c -f backup.ldif

Asked me for the admin password, gave it, then it said a couple times:

adding new entry "dc=host,dc=example,dc=com"
ldap_add: Constraint violation (19)
additional info: structuralObjectClass: no user modification allowed

What the heck? OK, Google.... finally found a post(2) explaining that certain entries must be removed from the ldif file before import. Thanks to the poster, there's even a command provided I didn't have to spend time with search & replace:

$ cat > ldapadd.sed <<EOF

/^creatorsName:/d

/^createTimestamp:/d

/^modifiersName:/d

/^modifyTimestamp:/d

/^structuralObjectClass:/d

/^entryUUID:/d

/^entryCSN:/d

EOF

$ cat backup.ldif | sed -f ldapadd.sed > bacon.ldif

After this, running ldapadd with bacon.ldif instead of backup.ldif did the trick and imported all the entries into LDAP.

ldapadd -h localhost -x -W -D "cn=admin,dc=example,dc=com" -c -f bacon.ldif

Thanks to the authors of the original posts, and the authors of debian and OpenLDAP.

Links:

(1) http://www.latenightpc.com/blog/archives/2007/10/23/upgrading-my-openldap-bdb-backend-for-suse-103

(2) http://www.openldap.org/lists/openldap-software/200504/msg00204.html

Fonts in Eclipse Project Explorer

First of all thanks to all the developers of Eclipse and related projects, I've been using it for years now, it's hard to imagine working on a software project without it.
In the previous post, I've described how I created a desktop shortcut for Eclipse. After successfully running it I've found the font used in the Project Explorer looking beautiful, but way too big.
Here's a couple possible solutions - I've successfully tested overwriting the theme settings for Eclipse:

http://techtavern.wordpress.com/2008/09/24/smaller-font-sizes-for-eclipse-on-linux/

Desktop Icons in Ubuntu 11.10

Here's a good one:
I wanted to create an application 'Shortcut' on my desktop to fire up Eclipse.
Coming from the (shame on me) m$ windows desktop world, I couldn't figure it out how, then after some google search, found this article, and guess what, it works :)

http://www.ubuntugeek.com/how-to-create-desktop-launchers-in-ubuntu-11-10oneiric.html

Lame welcome post

Welcome to Tom's Virtual Adventures!

First of all, sorry to disappoint, but this blog is about my adventures in the virtual world of computers, not something mysterious, mind blowing experience you may have expected.
I try to collect random useful - or not that useful - info, articles, blog posts, links, etc. which helped me to survive in my own cyberspace. Please feel free to browse, comment, re-post, whatever, this is not a private diary but a collection of bits 'n pieces.
Have fun in life,

T