First experiences with shopware

First experiences with shopware 5.4.x

For the past years I've been working with various, mostly custom-built shop systems, ranging from smaller, less frequented, to larger platforms with up to 50K users/s peaks.

Recently I joined a project, using shopware as a multishop/multilanguage environment. No real specialties, some plugins installed (e.g. OnePageCheckout, Pickware, PayPal, ...) and had to bring the whole development up to speed.

Baseline

To outline the project and my tasks I'll start with the baseline:

  • default shop
  • custom plugin, built by a third party company
  • no deployment process
  • no git integration

So the first steps were - clearly - establishing some kind of development process with git and a git-based deployment. For sake of simplicity, I opted for phploy and a stripped-down git-flow with only master/development and occasional feature branches. No tags, deployment version is HEAD of master.

First glance at shopware

To be fair: I know, shopware is a standard product with a lot of functionality to provide and maintain. Also it's the result of an ongoing development and refactoring process over lots of years. This introduces a lot of legacy code and development practices, if we keep in mind: PHP4 -> PHP5.3 -> PHP7.x and the many improvements and changes in the ways one develops with php. I often describe it as "we moved from writing scripts to actual software development".

This said, let's have a look.

A lot of nice eyecandy in the default templates and backend, so mostly it's kinda nice to look at (I'm no designer or design afficionado... ) and there are loads of plugins to add to your shop. Basically the plugin system works like the bundles in symfony2/3, where you have a whole sub-structure with service definitions, configuration, some initialization class and pre-defined extension points. It comes fully fledged with templating (Smarty), some database abstraction layer (Doctrine), dependency injection. All in all, everything you need to get started with your shop, from a technical point of view. Also the e-commerce abilities are quite sufficient. Works with all of Germany's tax stuff and regulations and provides sufficient ways to add payment abilities to the checkout, like the forementioned PayPal, etc.

Some more in-depth looks

Digging deeper, reveals a lot of stuff I didn't like, from an architectural point of view.

Architecture is basically monolithic with a plugin system - with all pros and cons. Some of the times, setter injection of dependcies, or even injecting the whole container are just a few of the things, I stumbeled upon. There are lots of extension points and classes to inherit for quick bootstrapping, like Controllers, Commands, etc. but the code you will get is really hard to test.

Code style is mostly inconsistent. You find Zend1-style classes like Enlight_Controller_Action or newer PSR1/2 compliant ones. And even classes you might expect from a first-time programmer with no idea of OOP (within Libraries).

Template inheritance is scarcely documented and could be done better when using e.g. TWIG or some other more sophisticated templating language.

Cache layers always get in your way. You never know what cache to clear and even when to clear it. Rule-of-thumb is: Something does not work as expected? Clear the caches. I developed a plugin, using the service definitions to add console commands and added a cache clear for the config cache in the backend. Commands were not showing up in console until I did a bin/console sw:cache:clear - not so funny. That's just one example of many... And deactivating the cache just doesn't cut it. No or less cache in SW means no performance. Even in larger Symfony projects, being in dev mode does not considerately slow down your project.

Deployment and versioning just broke my heart. To be fair: I think shopware developers know about that and introduced composer as a means to manage dependencies. But still... If you have custom templates, some own plugins and want the whole shop deployed in a sane manner, we're talking about a lot of thought and lot of workarounds. First thing you need is a custom .gitignore to keep shopware things out and your things within git. Then I strongly suggest, you start your project with composer! For deployment you'd better use something like ansistrano, since there are that many things to do (composer handling, git handling, config checks, cache clearing, ... ). My personal solution, since I had to deploy an already existing project, was to use phploy in combination with git and a fairly large .gitignore. Although I suggest, you'd better use something more sophisticated than phploy ;-) Also keep track of the shopware and plugin updates! You can't just update... You'll need at least a dev and a prod environment and need to do the updates on both systems simultaneously. That's sometimes hard to keep track of, so be careful! Besides that, most of the times, SW updates work kinda smooth :-)

Conclusion

Shopware is on a good way to be a nice out-of-the-box solution for smaller businesses. The main criticism, I have to voice is about: * Smarty ... * Ioncube (will be removed at SW 5.5, hopefully) * lots of inconsitencies in the implementation, e.g. zend_db + doctrine * many anti-patterns or bad practices like setter injection or injecting the whole container * unwieldy to deploy * Libraries like mpdf with ~30K LOC and PHP4-style code, hidden deep

If you want code quality, you have to have discipline and do it yourself in your own plugin. Put some thought in your deployment and development processes and you'll be OK.

Tags: php, shopware, opinion

Raspberry Pi experiments with ansible

Raspberry Pi experiments

It's been a while (Staind ;-)) since I've written a post on my personal blog. I've been up to a lot of things, mostly concerning work and upkeeping my house, but for the past weeks the amount of free time for "my stuff" increased, so here I am with some new projects. So I'll let you take part in my previous journeys with new or not-so-new technologies and projects :-)

Rethinking my infrastructure

The intention for looking into Raspberry Pies comes from me now having a fresh look at how I use my infrastructure at home. A lot of notebooks, a desktop PC for gaming and a media center (Asus O!Play HD2) in use. First of all: I'm mostly in retro-gaming. Old DOS games, older PC games and no consoles. But what do I really use stuff for? Watching movies or TV shows as files from a USB stick on my media center. Using the Desktop to download and one of the older laptops with a capable GFX card to play sometimes, less and less over the past years. So it's at least two devices due for scrapping, it seems.

What do I really need?

Considering the above thoughts, I concluded: No more desktop PC, No more media center. So, how to compensate that? * Every current Smart TV, Smartphone (with e.g. VLC), supports UPnP and/or DLNA * Services like usenet, sonarr and so on use web interfaces, so no GUI/window manager needed * Storage via NAS or external HDDs * Sometimes a working environment with a keyboard and two displays

Conclusion: A single Raspberry Pi 3 B with an external 2 TB WD Passport should be able to handle usenet, backups and serve all downloads/files to the local network via DLNA. Plugged in one of the router's LAN RJ45 ports, the network speed should also be sufficient for streaming.

Base setup

I bought the Pi 3 B desktop kit with case, AC adapter and a micro SD from element 14 with a pre-installed raspbian (ok, also some coolers ;-)). Since I'll only need console access, I first enabled boot into CLI and - as all systems accessible via SSH should - disabled root login via SSH and copied my pubkeys to pi. That's the baseline.

Ansible

For further provisioning we'll use ansible to have the whole thing indempotent and especially reproducable. Four roles to do the job:

|- bin
|   |- provision
|
|- roles
|   |- common
|   |- dlna
|   |- nzbdrone
|   |- sabnzbd
|
|- user-settings
|   |- settings.yml
|   |- settings.yml.dist
|
|- rpi
|- rpi.yml

That's the base directory layout for most of my ansible projects. Non-standard dirs are user-settings for user-defined variable settings, not injected by commandline parameters to override defaults and bin, which holds a wrapper for the provision call that disables e.g. "cowsay".

I won't paste all the config/code I use in particular, that's up to you, but I'll share some (hopefully) useful snippets and information.

Snippets

How to add custom-settings for ansible provisioning:

- name: check custom config file
  local_action: stat path="{{playbook_dir}}/user-settings/settings.yml"
  become: False
  register: custom_settings_root

- name: include custom settings
  include_vars:
    file: "{{playbook_dir}}/user-settings/settings.yml"
  when: custom_settings_root.stat.exists == True

Some base packages you might need:

- name: install base packages
  apt: pkg={{item}} state=latest
  with_items:
    - apt-transport-https
    - ntfs-3g
    - dirmngr
    - software-properties-common

Some optimizations for minidlna to keep the wear on your SD card low:

- name: create mountpoint dir
  file: path=/var/cache/minidlna
  state: directory

- name: mount tmpfs for caching
  mount:
    path: /var/cache/minidlna
    fstype: tmpfs
    opts: nodev,nosuid
    state: mounted

A hint on how to install sonarr:

- name: Add mono apt key
  apt_key:
    keyserver: hkp://keyserver.ubuntu.com:80
    id: 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
    state: present

- apt_repository:
    repo: "deb https://download.mono-project.com/repo/debian stable-raspbianstretch main"
    update_cache: True
    state: present

- name: Add sonarr apt key
  apt_key:
    keyserver: hkp://keyserver.ubuntu.com:80
    id: FDA5DFFC
    state: present

- apt_repository:
    repo: "deb https://apt.sonarr.tv/ master main"
    update_cache: True
    state: present

- name: install packages
  apt: pkg={{item}} state=latest
  with_items:
    - nzbdrone

- name: copy template
  template: src=./nzbdrone.service.j2 dest=/etc/systemd/system/sonarr.service owner=root group=root mode=755

- name: start and enable sonarr
  systemd:
    name: sonarr.service
    state: started
    daemon_reload: yes
    enabled: True

SabNZBD is kinda similar to install. Good luck with your first Raspberry Pi project ;-)

More to come

Next posts will be about how to add dyndns to your box with the help of domainfactory (my DNS provider) and some custom tooling with powerDNS.

Tags: DevOps, tinkering

Some thoughts on technical debt and efficiency

For the past years I've been spending a lot of thought on how to improve things. As a softwar developer you might know this: You see something, a workflow or just a tiny task and you already start thinking "If I change this..." and a short while later the thing you just looked at is e.g. better, more efficient or just plain faster or easier to do.

But how to get there?

Let's go back a few years to a point where I first realized this. I was in-between schools and had nothing to do, so I started working at my father's company. My father is a craftsman, insulating roofs (flat ones, especially) and does so for some 40+ years now and I go as far as to call him an expert on what he is doing. I was helping him out in my holidays since I was at least 14 years old, so I basically know what to do around him and how to help - but in the few months we worked that close together I could take away a lot more than just money to spend on new computer stuff. He always showed me how to place the materials in an efficient way, so one doesn't have to walk back and get more materials or have just a few steps less to walk. At this point you probably ask yourself "where is this going and what does it have to do with programming?" and the answer is "everything". In that time with my father I learned how to work efficiently and how to prepare my work (materials), so I can do my job without too many iterations and thus get better every time I do something. Everything you do is potentially improvable! Give it some thought: When you're cleaning the house, how do you place your working materials and in what order do you clean? Do you have to walk back and get something? Can you work seamlessly? Asking these questions and REALLY looking at what you're doing is the key. See if you can do the same work with less effort, less walking, less material or just "more efficiently".

Now back to the all-technical stuff :-)

When developing some new software, it's always the same:

  • Get a development environment
  • Provide some "common ground" for starting (framework install e.g.)
  • Do the usual database layout, controllers, views, templates
  • Create some fancy stuff, the customer wants
  • do some testing
  • create some kind of deployment
  • ...

Do you find some similarities to your projects? Ok, let's take a closer look.

Improving workflows

Every time you start a new project, you have to start somewhere. Did a sentence with "again" in it cross your mind? That might be the starting point for your first improvement. A clear indicator for something that needs improvement is repetition. If you do something several times the same way with slight variations, it's likely to be automatable.

  • You need a Linux environment? Use e.g. vagrant or docker and VM images together with a provisioning tool like ansible
  • New project, clone a framework, add some default tools like node, gulp, PHP classes? Use a skeleton project.
  • Everytime you checkout that project you need to add configs to apache/nginx, restart services, ... use a task runner like robo
  • ...

I think you get the point there :-) Automation is your friend and there are many tools you can use to make your life better.

Improving work

When I started coding I read a rule somewhere, that got stuck in my mind: "Everytime you edit/look at a piece of code, leave it better than before.". But what does that mean? Most developers have some (or many) "old" systems to maintain. Meaning there's a legacy application with "bad code" and a lot of technical debt. No one likes these systems. So, what you want to do is just "get the hell out of here" and leave it be. No? The right answer should be: Make things better. Even if it's just tiny things you change, like moving to PSR-2, correct indentation, fix variable names, add inline comments, ... these things are likely to help the next developer who looks at this code. You can even go further, build a UnitTest for the class/module to have a specification for its behavior and then refactor the whole thing with the security of not breaking the application. But these things take courage, so be courageous! Intergrate that behavior into your work and you will see that there is steady improvement in your codebase and also your feelings towards your code will improve. And don't be mad at the people (or yourself) who wrote that special piece of shitty code. Always remember: Most likely somone with a lot of skill and best intentions successfully solved a problem with his/her best tools available at that specific point in time. This makes it a lot easier. At least somewhat easier. If it's not complete nonsense code... Most developers say "I don't have enough time to do this" or "No one pays for this": This is correct - unless you integrate a small amount of refactoring cost into your estimations :-) I call this a "technical debt tax" and it is not explicitly in my estimations. It's always a few percent on top of the estimate and it will help a lot, over time.

Improve yourself

Nothing much to say, except: If you want to make your work and also your life better, always improve yourself. Read about new technologies, try new things and find new ways of doing things. Even if it does not work it will help your understanding of how things work. So, never stop learning.

In the end...

I could go on for much longer on how to improve, but in the end it boils down to a few guidelines and a lot of hard work to change your mindset:

  • Think about your problem domain and understand all aspects before you start to code
  • Always look for repetitive tasks and automate where you can (and it makes sense)
  • Improve what you find whenever you have to work on existing code - even small improvements help over time
  • Be lazy. Because a lazy person tends to be an efficient problem solver...
  • Never stop learning

Tags: php, architecture, work