Arduino preparations


2018-08-17

Hard learning curve

For a software engineer like me, mostly working in a web-context and according environment it's kinda hard to grasp all the concepts behind the whole "Arduino-thing". Ok, I've done some C/C++ in my time, but having to cope with bootloaders and all the hardware stuff like which pin goes where and how to calculate resistance for component X are way out of my default comfort zone. Since I've found some purpose for a Raspberry Pi, I started to look around for some smart home cases, I could actually use. Imagine: In my house there aren't that many things to "smart up"... no gas or oil heating, no solar panels and therfore a lot less to automate. So let's start small: Monitor the in-house temperature and humidity, especially in the old vaulted cellar where I'll store the turtles for winter. Maybe the outside temperature, too - and since I use up a lot of energy with electrical floor heating and a instantaneous water heater, I'd like to monitor at least the overall energy consumption in the house.

The project idea stands

So, basically we'll do as follows:

  • Monitor indoor temp
  • Monitor vaulted cellar temp and humidity
  • Monitor outside temp
  • Read the analog electric meter

Challenges:

  • Build an openHAB 2 instance on RPi with ansible
  • Add influx DB as storage
  • Visualize the values with Grafana
  • Learn Arduino
  • Try to use 2.4GHz transceivers for submitting data via MQTT to the RPi

Ho to start

First, I ordered loads of stuff, I found on mysensors for building my own sensors - I like soldering :-D The idea of not having to buy vendor-locked stuff and to be able to have more influence on the details of my hardware and infrastructure, was too intriguing to let go. So I decideded to stick with mysensors and accept the harder learning curve that comes with "doing it all by yourself".

Pitfalls

If you've never used a breadboard or even the whole power supply stuff before, like me, you'd have a lot to read. That's what I did and even finding useful pinmaps for the different boards (arduino pro mini and nano e.g.) is no fun. Also the tutorials and sketches on mysensors are far from perfect. There's a lot of reading their forums, too. Best advice I can give is: Don't give up, use google and ask around. Another big question mark is: How to plan the infrastructure? There's not much information on how to plan such a thing correctly and what components need to be integrated - how do they communicate? I'll add a few sketches and Ideas in my next post.

Arduino programmers

Before you can start programming arduinos, you have to decide on which way you want to start. I bought a FTDI programmer (serial), which does not interfere with the bootloader and therefore is considered "more safe" to use. But I also ordererd an ISP programmer (uspasp), which is more versatile. The third option you have is to use an arduino nano or UNO and flash a programmer image. My current setup looks like this: arduino programmer station with ISP programmer

This is a "programming platform" where I connect an ISP programmer to a board with two sockets, one for pro mini and one for nano, so I'm able to flash either without having to find the pinmaps and using loads of jumper cables. For debugging reasons (pro mini only), I use the FTDI programmer as a serial monitor. And don't forget the arduino IDE ;-) That's about all you need to start.

Tags: tinkering, IoT, smart home, arduino

First experiences with shopware 5.4.x


2018-08-13

For the past years I've been working with various, mostly custom-built shop systems, ranging from smaller, less frequented, to larger platforms with up to 50K users/s peaks.

Recently I joined a project, using shopware as a multishop/multilanguage environment. No real specialties, some plugins installed (e.g. OnePageCheckout, Pickware, PayPal, ...) and had to bring the whole development up to speed.

Baseline

To outline the project and my tasks I'll start with the baseline:

  • default shop
  • custom plugin, built by a third party company
  • no deployment process
  • no git integration

So the first steps were - clearly - establishing some kind of development process with git and a git-based deployment. For sake of simplicity, I opted for phploy and a stripped-down git-flow with only master/development and occasional feature branches. No tags, deployment version is HEAD of master.

First glance at shopware

To be fair: I know, shopware is a standard product with a lot of functionality to provide and maintain. Also it's the result of an ongoing development and refactoring process over lots of years. This introduces a lot of legacy code and development practices, if we keep in mind: PHP4 -> PHP5.3 -> PHP7.x and the many improvements and changes in the ways one develops with php. I often describe it as "we moved from writing scripts to actual software development".

This said, let's have a look.

A lot of nice eyecandy in the default templates and backend, so mostly it's kinda nice to look at (I'm no designer or design afficionado... ) and there are loads of plugins to add to your shop. Basically the plugin system works like the bundles in symfony2/3, where you have a whole sub-structure with service definitions, configuration, some initialization class and pre-defined extension points. It comes fully fledged with templating (Smarty), some database abstraction layer (Doctrine), dependency injection. All in all, everything you need to get started with your shop, from a technical point of view. Also the e-commerce abilities are quite sufficient. Works with all of Germany's tax stuff and regulations and provides sufficient ways to add payment abilities to the checkout, like the forementioned PayPal, etc.

Some more in-depth looks

Digging deeper, reveals a lot of stuff I didn't like, from an architectural point of view.

Architecture is basically monolithic with a plugin system - with all pros and cons. Some of the times, setter injection of dependcies, or even injecting the whole container are just a few of the things, I stumbeled upon. There are lots of extension points and classes to inherit for quick bootstrapping, like Controllers, Commands, etc. but the code you will get is really hard to test.

Code style is mostly inconsistent. You find Zend1-style classes like Enlight_Controller_Action or newer PSR1/2 compliant ones. And even classes you might expect from a first-time programmer with no idea of OOP (within Libraries).

Template inheritance is scarcely documented and could be done better when using e.g. TWIG or some other more sophisticated templating language.

Cache layers always get in your way. You never know what cache to clear and even when to clear it. Rule-of-thumb is: Something does not work as expected? Clear the caches. I developed a plugin, using the service definitions to add console commands and added a cache clear for the config cache in the backend. Commands were not showing up in console until I did a bin/console sw:cache:clear - not so funny. That's just one example of many... And deactivating the cache just doesn't cut it. No or less cache in SW means no performance. Even in larger Symfony projects, being in dev mode does not considerately slow down your project.

Deployment and versioning just broke my heart. To be fair: I think shopware developers know about that and introduced composer as a means to manage dependencies. But still... If you have custom templates, some own plugins and want the whole shop deployed in a sane manner, we're talking about a lot of thought and lot of workarounds. First thing you need is a custom .gitignore to keep shopware things out and your things within git. Then I strongly suggest, you start your project with composer! For deployment you'd better use something like ansistrano, since there are that many things to do (composer handling, git handling, config checks, cache clearing, ... ). My personal solution, since I had to deploy an already existing project, was to use phploy in combination with git and a fairly large .gitignore. Although I suggest, you'd better use something more sophisticated than phploy ;-) Also keep track of the shopware and plugin updates! You can't just update... You'll need at least a dev and a prod environment and need to do the updates on both systems simultaneously. That's sometimes hard to keep track of, so be careful! Besides that, most of the times, SW updates work kinda smooth :-)

Conclusion

Shopware is on a good way to be a nice out-of-the-box solution for smaller businesses. The main criticism, I have to voice is about: * Smarty ... * Ioncube (will be removed at SW 5.5, hopefully) * lots of inconsitencies in the implementation, e.g. zend_db + doctrine * many anti-patterns or bad practices like setter injection or injecting the whole container * unwieldy to deploy * Libraries like mpdf with ~30K LOC and PHP4-style code, hidden deep

If you want code quality, you have to have discipline and do it yourself in your own plugin. Put some thought in your deployment and development processes and you'll be OK.

Tags: php, shopware, opinion

Raspberry Pi experiments with ansible


2018-08-02

It's been a while (Staind ;-)) since I've written a post on my personal blog. I've been up to a lot of things, mostly concerning work and upkeeping my house, but for the past weeks the amount of free time for "my stuff" increased, so here I am with some new projects. So I'll let you take part in my previous journeys with new or not-so-new technologies and projects :-)

Rethinking my infrastructure

The intention for looking into Raspberry Pies comes from me now having a fresh look at how I use my infrastructure at home. A lot of notebooks, a desktop PC for gaming and a media center (Asus O!Play HD2) in use. First of all: I'm mostly in retro-gaming. Old DOS games, older PC games and no consoles. But what do I really use stuff for? Watching movies or TV shows as files from a USB stick on my media center. Using the Desktop to download and one of the older laptops with a capable GFX card to play sometimes, less and less over the past years. So it's at least two devices due for scrapping, it seems.

What do I really need?

Considering the above thoughts, I concluded: No more desktop PC, No more media center. So, how to compensate that? * Every current Smart TV, Smartphone (with e.g. VLC), supports UPnP and/or DLNA * Services like usenet, sonarr and so on use web interfaces, so no GUI/window manager needed * Storage via NAS or external HDDs * Sometimes a working environment with a keyboard and two displays

Conclusion: A single Raspberry Pi 3 B with an external 2 TB WD Passport should be able to handle usenet, backups and serve all downloads/files to the local network via DLNA. Plugged in one of the router's LAN RJ45 ports, the network speed should also be sufficient for streaming.

Base setup

I bought the Pi 3 B desktop kit with case, AC adapter and a micro SD from element 14 with a pre-installed raspbian (ok, also some coolers ;-)). Since I'll only need console access, I first enabled boot into CLI and - as all systems accessible via SSH should - disabled root login via SSH and copied my pubkeys to pi. That's the baseline.

Ansible

For further provisioning we'll use ansible to have the whole thing indempotent and especially reproducable. Four roles to do the job:

|- bin
|   |- provision
|
|- roles
|   |- common
|   |- dlna
|   |- nzbdrone
|   |- sabnzbd
|
|- user-settings
|   |- settings.yml
|   |- settings.yml.dist
|
|- rpi
|- rpi.yml

That's the base directory layout for most of my ansible projects. Non-standard dirs are user-settings for user-defined variable settings, not injected by commandline parameters to override defaults and bin, which holds a wrapper for the provision call that disables e.g. "cowsay".

I won't paste all the config/code I use in particular, that's up to you, but I'll share some (hopefully) useful snippets and information.

Snippets

How to add custom-settings for ansible provisioning:

- name: check custom config file
  local_action: stat path="{{playbook_dir}}/user-settings/settings.yml"
  become: False
  register: custom_settings_root

- name: include custom settings
  include_vars:
    file: "{{playbook_dir}}/user-settings/settings.yml"
  when: custom_settings_root.stat.exists == True

Some base packages you might need:

- name: install base packages
  apt: pkg={{item}} state=latest
  with_items:
    - apt-transport-https
    - ntfs-3g
    - dirmngr
    - software-properties-common

Some optimizations for minidlna to keep the wear on your SD card low:

- name: create mountpoint dir
  file: path=/var/cache/minidlna
  state: directory

- name: mount tmpfs for caching
  mount:
    path: /var/cache/minidlna
    fstype: tmpfs
    opts: nodev,nosuid
    state: mounted

A hint on how to install sonarr:

- name: Add mono apt key
  apt_key:
    keyserver: hkp://keyserver.ubuntu.com:80
    id: 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
    state: present

- apt_repository:
    repo: "deb https://download.mono-project.com/repo/debian stable-raspbianstretch main"
    update_cache: True
    state: present

- name: Add sonarr apt key
  apt_key:
    keyserver: hkp://keyserver.ubuntu.com:80
    id: FDA5DFFC
    state: present

- apt_repository:
    repo: "deb https://apt.sonarr.tv/ master main"
    update_cache: True
    state: present

- name: install packages
  apt: pkg={{item}} state=latest
  with_items:
    - nzbdrone

- name: copy template
  template: src=./nzbdrone.service.j2 dest=/etc/systemd/system/sonarr.service owner=root group=root mode=755

- name: start and enable sonarr
  systemd:
    name: sonarr.service
    state: started
    daemon_reload: yes
    enabled: True

SabNZBD is kinda similar to install. Good luck with your first Raspberry Pi project ;-)

More to come

Next posts will be about how to add dyndns to your box with the help of domainfactory (my DNS provider) and some custom tooling with powerDNS.

Tags: DevOps, tinkering