OpenHAB2, Mosquitto and MySensors Gateway


2018-08-27

my first sensor is a DHT22 (temperature/humidity, very good accuracy), attached to a 3.3V pro mini (ATMEGA 328p), connected via a NRF24 l 01+ (2.4GHz) which is capable of low-power transmission. This enables the arduino to later work on battery power only. My goal is to build three of these sensors. The assembly is straight forward:

pro mini with sensor and transceiver attached

I had many setbacks, connecting the sensor to the Raspberry Pi, though. This forced me to understand how a basic MQTT infrastructure has to interact with its data sources and OpenHAB2:

infrastructure with DHT22 attached

Steps to connect:

  • add node ID and parent ID to sketch
  • define lower baud rate for later running the board at a lowered clock speed
  • get MySensors Gateway working on RPi
  • Connect GateWay to Mosquitto
  • Connect Mosquitto to the OpenHAB MQTT binding

Step by step

Following the basic steps in more detail. Keep in mind that starting with OpenHAB2, the "old" bindings like MQTT are not shown in PaperUI, even when installed. So you'd have to configure all items manually (which took me a few hours to find out)!

Sensor IDs and baud settings in sketch

#define MY_NODE_ID 101
#define MY_PARENT_NODE_ID 0

// Define a lower baud rate for Arduino's running on 8 MHz (Arduino Pro Mini 3.3V & SenseBender)
#if F_CPU == 8000000L
#define MY_BAUD_RATE 9600
#endif

I've set up my sensor node IDs in "namespaces" like that:

  • 1xx environmental sensor nodes
  • 2xx extra nodes like elecrity IR reader

Attaching the NRF24 is relatively straight forward, as is connecting the DHT22.

Install and configure mosquitto

The current version, provided by apt is 1.4.x - I only managed to get it running in a stable manner when using 1.5.

- name: Add mosquitto repository
  apt_key:
    url: http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
    state: present

- apt_repository:
    repo: "deb https://repo.mosquitto.org/debian stretch main"
    update_cache: True
    state: present 

- name: install packages
  apt: pkg= state=latest
  with_items:
    - mosquitto
    - mosquitto-clients

Install MySensors gateway

In hindsight this is quite easy. Most of the time I needed, went into a reproducable ansible script ;-) You can find instructions here. Since I won't be exposing my OpenHAB instance to the internet - only available via VPN - I will not introduce authentication.

So, follow the instructions and install/compile the gateway. Keep in mind, that you will need mosquitto installed, too. I'd also opt for the MQTT binding for OpenHAB, since it's working fine with mosquitto. You can use the defaults @ mosquitto.

Some ansible help:

- name: get mysensors mqtt repo
  git:
    repo: https://github.com/mysensors/MySensors.git
    dest: /home/pi/MySensors
    accept_hostkey: yes
    version: master

- name: configure
  command: "./configure --my-transport=nrf24 --my-gateway=mqtt --my-controller-ip-address=127.0.0.1 --my-mqtt-publish-topic-prefix=mysensors-out --my-mqtt-subscribe-topic-prefix=mysensors-in --my-mqtt-client-id=mysensors-gateway"
  args:
    chdir: /home/pi/MySensors

- name: copy mysensors config
  template: src=./mysensors.conf.j2 dest=/etc/mysensors.conf owner=root group=root mode=755

- name: make
  command: "make"
  args:
    chdir: /home/pi/MySensors

- name: install
  command: "make install"
  args:
    chdir: /home/pi/MySensors
  notify:
    - start mysqw

It's that easy, since all components reside within the RPi. Don't forget to attach a NRF24 to your RPi aswell!

Install the MQTT binding and influxdb

Two parts: Install influxdb and client and manually install the bindings in OpenHAB.

- name: Add influxdb apt key
  apt_key:
    url: https://repos.influxdata.com/influxdb.key
    state: present

- apt_repository:
    repo: "deb https://repos.influxdata.com/debian stretch stable"
    update_cache: True
    state: present

 - name: install base packages
  apt: pkg= state=latest
  with_items:
    - influxdb
    - influxdb-client
  notify:
    - start influxdb

For influxdb, use the defaults provided here. Don't forget to configure persistence for OpenHAB (influxdb.persist)!

Strategies { default = everyChange }
Items { * : strategy = everyUpdate, restoreOnStartup }

Configure OpenHAB

I installed OpenHAB as a package, so no OpenHABbian. This means: All configuration resides in /etc/openhab2 I strongly(!!!) suggest you to pack the whole directory into a separate git repository! You will configure a lot via config files and since PaperUI persists only in json files, which most of the times are generated from the config files, the latter are the safer bet.

services/mqtt.cfg:

mosquitto.url=tcp://localhost:1883
mosquitto.clientId=openhab
mosquitto.qos=1
mosquitto.retain=true
mosquitto.async=false

services/mqtt-eventbus.cfg:

broker=mosquitto
stateSubscribeTopic=mosquitto/stateUpdates/${item}/state
commandSubscribeTopic=mosquitto/commandUpdates/${item}/command

items/dht22.items:

Group    Home                    "home"                   <house>

Group    GF                      "Ground Floor"           <groundfloor>   (Home)
Group    FF                      "First Floor"            <firstfloor>    (Home)
Group    F2                      "Second Floor"           <attic>         (Home)
Group    F3                      "Third Floor"            <attic>         (Home)

Group    GF_Cellar               "Cellar"                 <cellar>        (Home, GF)
Group    FF_LivingDining         "Living & Dining Room"   <sofa>          (Home, FF)
Group    F2_Bedroom              "Bedroom"                <bedroom>       (Home, F2)
Group    F2_KidsRoom             "Kids Room"              <girl_3>        (Home, F2)
Group    F3_GuestRoom            "Guest Room"             <parents_4_3>   (Home, F3)

Number DHT22_101_01_cellar_temp  "temperature [%.1f °C]" <temperature> (gTemperature, GF_Cellar) {mqtt="<[mosquitto:mysensors-out/101/1/1/0/0:state:default]"}
Number DHT22_101_02_cellar_hum  "humidity [%.1f %%]" <humidity> (gHumidity, GF_Cellar) {mqtt="<[mosquitto:mysensors-out/101/0/1/0/1:state:default]"}

Group:Number:AVG   gTemperature   "Temperature"   <temperature>   (Home)
Group:Number:AVG   gHumidity      "Humidity"      <humidity>      (Home)

That's a lot in the items but mostly it's auto-generated grouping. The really interesting lines are the ones starting with "Number". They actually handle the processing of incoming sensor values for the temp/hum DHT22 sensor. Addotional sensor definitions (for DHT22s) would be located in here, too.

If you've followed this guide, you should have a working sensor with MySensors framework communicating with a RPi, having a MQTT gateway, broker and binding, pushing data into an influxdb.

Tags: tinkering, IoT, smart home, arduino, mysensors

Arduino preparations


2018-08-17

Hard learning curve

For a software engineer like me, mostly working in a web-context and according environment it's kinda hard to grasp all the concepts behind the whole "Arduino-thing". Ok, I've done some C/C++ in my time, but having to cope with bootloaders and all the hardware stuff like which pin goes where and how to calculate resistance for component X are way out of my default comfort zone. Since I've found some purpose for a Raspberry Pi, I started to look around for some smart home cases, I could actually use. Imagine: In my house there aren't that many things to "smart up"... no gas or oil heating, no solar panels and therfore a lot less to automate. So let's start small: Monitor the in-house temperature and humidity, especially in the old vaulted cellar where I'll store the turtles for winter. Maybe the outside temperature, too - and since I use up a lot of energy with electrical floor heating and a instantaneous water heater, I'd like to monitor at least the overall energy consumption in the house.

The project idea stands

So, basically we'll do as follows:

  • Monitor indoor temp
  • Monitor vaulted cellar temp and humidity
  • Monitor outside temp
  • Read the analog electric meter

Challenges:

  • Build an openHAB 2 instance on RPi with ansible
  • Add influx DB as storage
  • Visualize the values with Grafana
  • Learn Arduino
  • Try to use 2.4GHz transceivers for submitting data via MQTT to the RPi

Ho to start

First, I ordered loads of stuff, I found on mysensors for building my own sensors - I like soldering :-D The idea of not having to buy vendor-locked stuff and to be able to have more influence on the details of my hardware and infrastructure, was too intriguing to let go. So I decideded to stick with mysensors and accept the harder learning curve that comes with "doing it all by yourself".

Pitfalls

If you've never used a breadboard or even the whole power supply stuff before, like me, you'd have a lot to read. That's what I did and even finding useful pinmaps for the different boards (arduino pro mini and nano e.g.) is no fun. Also the tutorials and sketches on mysensors are far from perfect. There's a lot of reading their forums, too. Best advice I can give is: Don't give up, use google and ask around. Another big question mark is: How to plan the infrastructure? There's not much information on how to plan such a thing correctly and what components need to be integrated - how do they communicate? I'll add a few sketches and Ideas in my next post.

Arduino programmers

Before you can start programming arduinos, you have to decide on which way you want to start. I bought a FTDI programmer (serial), which does not interfere with the bootloader and therefore is considered "more safe" to use. But I also ordererd an ISP programmer (uspasp), which is more versatile. The third option you have is to use an arduino nano or UNO and flash a programmer image. My current setup looks like this: arduino programmer station with ISP programmer

This is a "programming platform" where I connect an ISP programmer to a board with two sockets, one for pro mini and one for nano, so I'm able to flash either without having to find the pinmaps and using loads of jumper cables. For debugging reasons (pro mini only), I use the FTDI programmer as a serial monitor. And don't forget the arduino IDE ;-) That's about all you need to start.

Tags: tinkering, IoT, smart home, arduino

First experiences with shopware 5.4.x


2018-08-13

For the past years I've been working with various, mostly custom-built shop systems, ranging from smaller, less frequented, to larger platforms with up to 50K users/s peaks.

Recently I joined a project, using shopware as a multishop/multilanguage environment. No real specialties, some plugins installed (e.g. OnePageCheckout, Pickware, PayPal, ...) and had to bring the whole development up to speed.

Baseline

To outline the project and my tasks I'll start with the baseline:

  • default shop
  • custom plugin, built by a third party company
  • no deployment process
  • no git integration

So the first steps were - clearly - establishing some kind of development process with git and a git-based deployment. For sake of simplicity, I opted for phploy and a stripped-down git-flow with only master/development and occasional feature branches. No tags, deployment version is HEAD of master.

First glance at shopware

To be fair: I know, shopware is a standard product with a lot of functionality to provide and maintain. Also it's the result of an ongoing development and refactoring process over lots of years. This introduces a lot of legacy code and development practices, if we keep in mind: PHP4 -> PHP5.3 -> PHP7.x and the many improvements and changes in the ways one develops with php. I often describe it as "we moved from writing scripts to actual software development".

This said, let's have a look.

A lot of nice eyecandy in the default templates and backend, so mostly it's kinda nice to look at (I'm no designer or design afficionado... ) and there are loads of plugins to add to your shop. Basically the plugin system works like the bundles in symfony2/3, where you have a whole sub-structure with service definitions, configuration, some initialization class and pre-defined extension points. It comes fully fledged with templating (Smarty), some database abstraction layer (Doctrine), dependency injection. All in all, everything you need to get started with your shop, from a technical point of view. Also the e-commerce abilities are quite sufficient. Works with all of Germany's tax stuff and regulations and provides sufficient ways to add payment abilities to the checkout, like the forementioned PayPal, etc.

Some more in-depth looks

Digging deeper, reveals a lot of stuff I didn't like, from an architectural point of view.

Architecture is basically monolithic with a plugin system - with all pros and cons. Some of the times, setter injection of dependcies, or even injecting the whole container are just a few of the things, I stumbeled upon. There are lots of extension points and classes to inherit for quick bootstrapping, like Controllers, Commands, etc. but the code you will get is really hard to test.

Code style is mostly inconsistent. You find Zend1-style classes like Enlight_Controller_Action or newer PSR1/2 compliant ones. And even classes you might expect from a first-time programmer with no idea of OOP (within Libraries).

Template inheritance is scarcely documented and could be done better when using e.g. TWIG or some other more sophisticated templating language.

Cache layers always get in your way. You never know what cache to clear and even when to clear it. Rule-of-thumb is: Something does not work as expected? Clear the caches. I developed a plugin, using the service definitions to add console commands and added a cache clear for the config cache in the backend. Commands were not showing up in console until I did a bin/console sw:cache:clear - not so funny. That's just one example of many... And deactivating the cache just doesn't cut it. No or less cache in SW means no performance. Even in larger Symfony projects, being in dev mode does not considerately slow down your project.

Deployment and versioning just broke my heart. To be fair: I think shopware developers know about that and introduced composer as a means to manage dependencies. But still... If you have custom templates, some own plugins and want the whole shop deployed in a sane manner, we're talking about a lot of thought and lot of workarounds. First thing you need is a custom .gitignore to keep shopware things out and your things within git. Then I strongly suggest, you start your project with composer! For deployment you'd better use something like ansistrano, since there are that many things to do (composer handling, git handling, config checks, cache clearing, ... ). My personal solution, since I had to deploy an already existing project, was to use phploy in combination with git and a fairly large .gitignore. Although I suggest, you'd better use something more sophisticated than phploy ;-) Also keep track of the shopware and plugin updates! You can't just update... You'll need at least a dev and a prod environment and need to do the updates on both systems simultaneously. That's sometimes hard to keep track of, so be careful! Besides that, most of the times, SW updates work kinda smooth :-)

Conclusion

Shopware is on a good way to be a nice out-of-the-box solution for smaller businesses. The main criticism, I have to voice is about: * Smarty ... * Ioncube (will be removed at SW 5.5, hopefully) * lots of inconsitencies in the implementation, e.g. zend_db + doctrine * many anti-patterns or bad practices like setter injection or injecting the whole container * unwieldy to deploy * Libraries like mpdf with ~30K LOC and PHP4-style code, hidden deep

If you want code quality, you have to have discipline and do it yourself in your own plugin. Put some thought in your deployment and development processes and you'll be OK.

Tags: php, shopware, opinion