Mr Whale

TTRSS + Calibre - Turn your RSS feed into an EBook

Mon, 13 Apr 2020 10:00:00 +0000

After turning an old Android tablet into a dedicated E-Reader by utilising KOReader, I also turned to Calibre to manage my E-Books. Utilising this Caibre recipe in this gist by oott123 I was able to get Calibre to pull my RSS feed (hosted on my TTRSS raspberry pi) into a ebook, that I could sync with my “reader” and read on the go.

Create a new “news” source

From the “Fetch News” icon in the main tool bar of Calibre, select the down arrow and click “Add or edit a custom news source”

Adding custom news source

Create a new recipe by clicking the “New recipe” button down the bottom left area of the pop up window

Add a new recipe

Give it a name and fill out the form to your prefernce, and then switch over to advanced mode

Switch to advanced mode

Copy in the recipe and adjust the variables in the code to point to your TTRSS instance. Once done click save

Code and stuff

From here you can add automatic download schedule that Calibre will handle (usefull if you run calibre on a server somewhere) and hit Okay

Adjust scheduling
Sync your device with Calibre and it will bring your feeds in. This works nicely if you fetch it daily, as you will then have a daily news book to read. This recipe also marks the fetched news items as read within TTRSS

How Calibre shows a ttrss ebook in the list

recipe

#!/usr/bin/env python2

from __future__ import unicode_literals, division, absolute_import, print_function
from calibre.web.feeds.news import BasicNewsRecipe
import json, urllib2, urllib, pprint, re, ssl

class TClient():
    def __init__(self, url, username, password, logger, key, test = False):
        self.url = "%s/api/" % url
        self.username = username
        self.password = password
        self.log = logger
        self.sid = None
        self.test = test
        self.key = key
        self.login()
    def _request(self, op, data = None):
        if not op is 'login':
            data['sid'] = self.sid
        if data is None:
            data = {}
        data['op'] = op
        data_string = json.dumps(data)
        json_string = None
        try:
            req = urllib2.Request(self.url, data_string, {'User-Agent': 'Mozilla/4.0'})
            res = urllib2.urlopen(req, context=ssl._create_unverified_context())
            json_string = res.read()
        except urllib2.HTTPError,e:
            print(e.read())
            raise e
        object = json.loads(json_string)
        if 'error' in object:
            raise Exception("Tiny Tiny RSS Error: URL-%s, DATA-%s, RESP-%s" %(self.url, data_string, json_string))
        return object['content']
    def login(self):
        data = self._request('login', {
            "user": self.username,
            "password": self.password
        })
        if not 'session_id' in data:
            self.log.warn("Tiny Tiny RSS Error: failed to load session id.")
            raise Exception("Tiny Tiny RSS Error: failed to load session id.")
        self.sid = data['session_id']
        self.log.info("Get session id %s" % self.sid)
    def get_articles(self, urls, offset = 0, limit = 10):
        id_list = []
        data = {}
        data['feed_id'] = -4 # all unread

        data['limit'] = limit
        data['offset'] = offset
        data['show_content'] = True
        data['view_mode'] = 'unread'
        data['sanitize'] = False
        feeds = self._request('getHeadlines', data)
        for i in feeds:
            if not i['feed_title'] in urls:
                urls[i['feed_title']] = []
            urls[i['feed_title']].append({
                'title': i['title'],
                'url': i['link'],
                'date': i['updated'],
                #'content': self.append_url(i['content'], i['link']),

                'content': i['content'],
                'description': i['excerpt'] if 'excerpt' in i else ''
            })
            id_list.append("%s" % i['id'])
        read_data = {
            "article_ids": ",".join(id_list),
            "mode": 0,
            "field": 2
        }
        if not self.test:
            self._request('updateArticle', read_data)
        return urls
    def get_all_articles(self):
        urls = {}
        countLast = 0
        while True:
            counters = self._request('getCounters', {'mode': 'f'})
            count = 0
            for i in counters:
                if i['id'] is -4:
                        count = i['counter']
                        break
            if count < 1:
                break
            if countLast is count:
                raise Exception("There's some error when marking read articles.")
            countLast = count
            urls = self.get_articles(urls)
            if self.test:
                break
        return urls
    def append_url(self, raw_html, url):
        u = urllib.quote(url)
        share_url = "http://httpbin.org/get?key=%s&url=%s" % (self.key, u)
        qr_url = "https://chart.googleapis.com/chart?cht=qr&chs=300x300&choe=UTF-8&chld=H|4&chl=%s" % u
        append_html = u' ' % (qr_url, share_url, url)
        raw_html = re.sub(r'(\</body\>|$)', r'%s\1' % append_html, raw_html, count=1)
        return raw_html

class YueDu(BasicNewsRecipe):
    
    title = u'TTRSS + Calibre'
    __author__ = 'Mrwhale'
    description = u'TTRSS + Calibre'
    timefmt = '%Y-%m-%d %A'
    needs_subscription = True
    oldest_article = 256
    max_articles_per_feed = 256
    publication_type = 'newspaper'
    compress_news_images = True
    use_embedded_content = True
    cover_url = ''
    masthead_url = ''
    def parse_index(self):
        ttrss_url = "URL"
        ttrss_username = "user"
        ttrss_password = "pass"
        ttrss_key = "WTF_IS_OK"
        #(ttrss_url, ttrss_username, ttrss_key) = self.username.split(';')

        self.log.info("Getting URL-%s, username-%s" % (ttrss_url, ttrss_username))
        ttrss_password = self.password
        self.log.info("Getting Password-%s******" % ttrss_password[:2])
        ttrss = TClient(ttrss_url, ttrss_username, ttrss_password, self.log, ttrss_key, self.test)
        urls = ttrss.get_all_articles()
        index = []
        for (key, value) in urls.iteritems():
            self.log.info("Key:%s - length: %d" % (key, len(value)))
            index.append((key, value))
        return index

E-ink cryptocurrency ticker

Wed, 18 Mar 2020 10:00:00 +0000

I like the idea of e-ink. Ultra Low power, easily readable displays. Big fan. I also follow some cryptocurrency’s. So this project is a merge of these 2 things. It’s still a work in progress, as there a few bugs I need to sort out, and I need a nice looking case. But for now it was a fun project that does the thing well and I got to learn some image manipulation in python!

the hardware

raspberry pi 3

I had a spare one of those laying around, but they are very cheap these days. Almost any version can be used here. In the next iteration I will use a pi zero, as it will be even smaller form factor as it’s around the same dimensions as the e-ink display I used

Pomodoro inkyphat 2.7” e-ink display (black and white model)

The code currently is specific to this model as I generate an image for 2.7” and utilise the inkyphat python module, however it shouldn’t be too difficult to adjust for a larger size or model

the software

python (3)
inkyphat
python pillow
mathplot lib
mpl-finance

the code

Hosted in gitlab you can view the entire repository here. This was build ontop of an existing eink cryptocurrency project. The main differences was that I had the black and white e-ink display so the candlestick graph was not easy to read (it was challenging to set variations of grey) so I decided to just graph it with a spark-line.

I have also added a function to utilise the Binance public API rather then Kraken.

def quotes_historical_binance_ohlc(pair, since, interval='15m'):
    unix_time = since.strftime("%s")
    if interval == '1d':
      payload = {'symbol': pair,'interval': interval, 'limit': '200'}
    else:
      payload = {'symbol': pair,'interval': interval, 'limit': '200', 'startTime': int(since.strftime('%s'))*1000}
    r = requests.get('https://api.binance.com/api/v1/klines', params=payload)
    #print(r.json())

    def parse_ohlc(data):
        return [[int(l[6]/1000), float(l[1]), float(l[2]), float(l[3]), float(l[4]), float(l[5])] for l in data]

    return parse_ohlc(r.json())

There was also a sticking point about the new InkyPhat python module. The new version had all of the python image manipulation functions removed out of the module, so I had to do it directly with Pillow.

The last thing to change was adding a config to make sure that mpl could run in headless mode

import matplotlib as mpl
mpl.use('Agg')
import matplotlib.pyplot as plt

Run this in your scheduler (cron) and you can be the cool kid in the office again.

Default Azure Search Service query key and Terraform

Tue, 03 Mar 2020 10:00:00 +0000

Need to get the default Azure Search Service query key into a variable to use in other parts of your terraform infrastructure? I did too. Unfortunately the docs for this were a bit vague, so through a bit of trial and error we got there

azurerm_search_service.example.query_keys[0].key

Given that these query keys are called the “default” keys, I assumed we could access this attribute by azurerm_search_service.example.query_keys.default.key. But that was not the case.

After outputting the query_key attributes during a terraform apply

output "searchkeys" {
  value = azurerm_search_service.example.query_keys
}

I could see that this “default” key was actually nameless

searchkeys = [
  {
    "key" = "be2c38e286fff7df25d17e21294604a8"
    "name" = ""
  },
]

More trial and error

azurerm_search_service.example.query_keys.object[0].key azurerm_search_service.example.query_keys."".key

but the winning combination was

azurerm_search_service.example.query_keys[0].key

So now you can use this in other parts of your terraform infrastructure as you normally would. For example, inside of appsettings for an Azure web app:

....
  app_settings = {
    "AzureSearchConfig:SearchServiceName" = azurerm_search_service.example.name
    "AzureSearchConfig:SearchServiceQueryApiKey" = azurerm_search_service.example.query_keys[0].key
    "AzureStorageConfig:ConnectionString" = azurerm_storage_account.example.primary_connection_string
    "WEBSITE_TIME_ZONE" = "AUS Eastern Standard Time"
    "WEBSITE_RUN_FROM_PACKAGE" = "1"
  }
....

Useful Commands - OpenSSL

Sun, 24 Mar 2019 09:00:00 +0000

openSSL is a very handy tool from the command line. Managing multiple sites and domains I almost use it everyday to do crypto relating things to websites - checking validity, preparing new CSRs, converting certificates to different formats for different operating systems. Here are some of those that I’ve used along the way

Read .cer in plain text

openssl x509 -in certificate.cer -text -noout

convert from pkcs12 (windows format, with private key and cert) to pem (-nocerts for only priv key or -nokeys for only certs)

openssl pkcs12 -in keystore.pfx -out keystore.pem -nodes

Read pfx cert

openssl pkcs12 -info -in cert.pfx

Read website SSL cert

openssl s_client -showcerts -servername www.google.com -connect www.google.com:443

using -servername for SNI

Get website SSL expiry

openssl s_client -showcerts -connect www.google.com:443 | openssl x509 -noout -dates

Convert privkey and cert to pfx (going from linux to windows)

openssl pkcs12 -export -out cert.pfx -inkey privkey.key -in cert.crt -certfile CAcert.cert -name "Friendly_name"

-name will add the Friendly name that shows up in windows certificate manager and IIS

Testing mail servers are using ssl

openssl s_client -connect mail.example.com:25 -starttls smtp

verify csr

openssl req -in mycsr.csr -noout -text

Useful Commands - Image manipulation

Thu, 21 Jun 2018 10:00:00 +0000

I often work with images to go onto e-commerce sites, so getting them as small as possbile (file size wise) is what I aim for, for performance There are plenty of good, paid tools out there such as Kraken or ImageOptim and there are even some good, free (web based) tools you can use such as TinyPNG and Compressor but I was looking for something a I could add into my workflow (read automated and scriptable)

My needs were very specific. Each input image was the same dimensions (2048x) and is a .png and they needed to stay square. Initialy I combined the tools within ImageMagik, pngquant and jpegoptim I could get the original 3-4mb image down to 132kb (@ 1024x1024) or 37kb (@512x512) with very little noticable quality differences. But then I started just using ImageMagik to get similar results, but only needing the convert tool

Using all 3 tools

Drop the quality pngquant --quality=85-100 in.png #This is destructive, but you can specify out.png if needed
Lets resize it and convert to jpeg convert in.png -resize 1024 bigimage.png bigimage1024.jpg
Strip metadata and make it progressive jpegoptim --all-progressive --strip-all -m85 bigimage1024.jpg

Here are some tests, resizing to 1024 and 512 using both methods

#Start
george@GEORGE-LT:$ file bigimage.png
bigimage.png: PNG image data, 2048 x 2048, 8-bit/color RGBA, non-interlaced

george@GEORGE-LT:$ ls -lah bigimage.png
-rwxrwxrwx 1 george george 3.9M Jul  4  2017 bigimage.png

george@GEORGE-LT:$ pngquant --quality=85-100 bigimage.png

george@GEORGE-LT:$ convert bigimage.png -resize 1024 bigimage1024x.jpg

george@GEORGE-lt$ ls -lah bigimage*
-rwxrwxrwx 1 george george 115K Jun 21 09:22 bigimage1024x.jpg
-rwxrwxrwx 1 george george  37K Jun 21 09:23 bigimage512x.jpg
-rwxrwxrwx 1 george george 4.0M Jun 21 09:20 bigimage.png

george@GEORGE-LT:$ jpegoptim --all-progressive --strip-all -m85 bigimage1024x.jpg
bigimage1024x.jpg 1024x1024 24bit N JFIF  [OK] 117484 --> 93472 bytes (20.44%), optimized.

george@GEORGE-LT:$ jpegoptim --all-progressive --strip-all -m85 bigimage512x.jpg
bigimage512x.jpg 512x512 24bit N JFIF  [OK] 37769 --> 30200 bytes (20.04%), optimized.

george@GEORGE-LT:$ ls -lah bigimage*
-rwxrwxrwx 1 george george  92K Jun 21 09:24 bigimage1024x.jpg
-rwxrwxrwx 1 george george  30K Jun 21 09:24 bigimage512x.jpg
-rwxrwxrwx 1 george george 4.0M Jun 21 09:20 bigimage.png

Using just convert

convert -resize 1024 -quality 85 -trim -interlace Plane bigimage.png bigimage1024.jpg

george@GEORGE-LT:$ convert -resize 1024 -quality 85 -trim -interlace Plane bigimage.png bigimage1024.jpg
george@GEORGE-LT:$ convert -resize 512 -quality 85 -trim -interlace Plane bigimage.png bigimage512x.jpg
george@GEORGE-LT:$ ls -lah bigimage*
-rwxrwxrwx 1 george george  75K Jun 21 09:30 bigimage1024.jpg
-rwxrwxrwx 1 george george  26K Jun 21 09:30 bigimage512x.jpg
-rwxrwxrwx 1 george george 4.0M Jun 21 09:20 bigimage.png

george@GEORGE-LT:$ file bigimage1024.jpg
bigimage1024.jpg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, progressive, precision 8, 1024x1024, frames 3

Just for fun, here is some tests without resizing the iamges, so you can see how powerful it is with a good comparison

george@GEORGE-LT$ convert -trim -quality 85 -interlace Plane bigimage.png bigimageoptimised.jpg
george@GEORGE-LT:$ ls -lah bigimage*
-rwxrwxrwx 1 george george 250K Jun 21 10:09 bigimageoptimised.jpg
-rwxrwxrwx 1 george george 4.0M Jun 21 09:20 bigimage.png

Conclusion

So you can see that just using convert to do all the works gives you slightly better results then the other tools. You can tweak it further to get better results, but in my trials I found that 85% quality was the best for keeping almost quality while getting the smallest file size. If you go too low in quality, you get a much smaller file, but the quality starts getting noticably bad

These sizes and qualities are perfect for online viewing. When 75+% of the viewing is done on mobile devices, users will not at all see the difference. And they look good on desktop too.

Given that I originally wanted this to be somewhat automated and scriptable, I started a project over on my Github to get this optimisation web based (and eventually with an API). You can check it out here

Cross compiling Litecoin Core

Wed, 09 May 2018 10:00:00 +0000

I’ve had a plan of being a good guy and contributing to the litecoin network by being a full node. I was able to do this when I got around to buying a raspberry pi 3B+ and a 2TB external hard drive. There isn’t many up-to-date guides out there on setting a full node up on a raspberry pi, but this one was the most recent, and worked every step of the way (with a few minor changes) - the school of litecoin

I followed this guide and successfully setup a headless litecoin full node with no wallet, that stores the blockchain on an external harddrive (to hopefully save some SD card wear). I will post another, updated guide, heavily based on this one in my next post. But give that a read first as I won’t cover the basics that are covered in that medium publication. The only thing I didn’t get out of this guide was the wallet part of the node, as the wallet component requires a specific version of berkleyDB which you also have to manually compile (which isn’t covered in the guide)

This was great, however it took ~4 hours total to get this up and running (and an extra 2 days to fully sync the node) Seeing as though I plan to keep this thing updated, I did not want to have to spend 4 hours compiling the newest version of litecoin for every release. So that is when I thought of cross compiling from my desktop, to hopefully save time. As I don’t often do this, it took my quite a bit of time and research to get it working, but will now save myself (and hopefully others) time in the future! This guide would also be useful for any cryptocurrency that is a fork of bitcoin too

raspberrypi) and the target PC (the raspberrypi)

Host PC

Linux george-linux 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10 14:24:03 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Distributor ID: neon
Description:    KDE neon User Edition 5.12
Release:        16.04
Codename:       xenial

Target PC

Linux georgeboy 4.14.34-v7+ #1110 SMP Mon Apr 16 15:18:51 BST 2018 armv7l GNU/Linux

Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.4 (stretch)
Release:        9.4
Codename:       stretch

Instructions

Install the dependancies for cross compile, there are quite a few

$ sudo apt-get install gcc-arm-linux-gnueabi binutils-arm-linux-gnueabi curl g +-aarch64-linux-gnu g++-4.8-aarch64-linux-gnu gcc-4.8-aarch64-linux-gnu binutils-aarch64-linux-gnu g++-arm-linux-gnueabihf g++-4.8-arm-linux-gnueabihf gcc-4.8-arm-linux-gnueabihf binutils-arm-linux-gnueabihf g++-4.8-multilib gcc-4.8-multilib binutils-gold bsdmainutils

Get the latest litecoin and extract (at the time I used 0.16.0rc1)

# https://github.com/litecoin-project/litecoin/releases
wget https://github.com/litecoin-project/litecoin/archive/v0.16.0rc1.tar.gz
tar zxvf v0.16.0rc1.tar.gz

make the dependancies for the architecture you are compling for. This will go and grab a whole bunch of things, but you can turn off things you do not want here by adding some arguments to the make command

cd litecoin-0.16.0rc1/depends
make host=arm-linux-gnueabihf -j4 
# if you do not want wallet or gui 
# make host=arm-linux-gnueabihf -j4 NO_WALLET=1 NO_QT=1 
real    5m28.381s
user    6m17.932s
sys     0m21.860s

now do the thing for litecoin

cd ..
./autogen.sh
./configure LDFLAGS="-static" CXXFLAGS="-O2 --param ggc-min-expand=1 --param ggc-min-heapsize=32768" --enable-cxx --without-gui --disable-shared --with-pic --build x86_64-pc-linux-gnu --host arm-linux-gnueabi --prefix=`pwd`/depends/arm-linux-gnueabihf
# time for ./configure
# real    0m36.008s
# user    0m30.848s
# sys     0m1.352s
sudo make -j4
# I use sudo here as I tried make by itself but ran into an issue, see "Issues" below
# real    18m3.695s
# user    71m0.420s
# sys     0m28.704s

Now copy over to your raspberrypi and test

scp src/litecoind src/litecoin-cli src/litecoin-tx george@newrpi:~/

george@newrpi:~ $ ./litecoind --version
Litecoin Core Daemon version v0.16.0.0-g3567effe2049
Copyright (C) 2011-2018 The Litecoin Core developers
Copyright (C) 2009-2018 The Bitcoin Core developers

Please contribute if you find Litecoin Core useful. Visit
<https://litecoin.org> for further information about the software.
The source code is available from
<https://github.com/litecoin-project/litecoin>.

This is experimental software.
Distributed under the MIT software license, see the accompanying file COPYING
or <https://opensource.org/licenses/MIT>

This product includes software developed by the OpenSSL Project for use in the
OpenSSL Toolkit <https://www.openssl.org> and cryptographic software written by
Eric Young and UPnP software written by Thomas Bernard.

george@newrpi:~ $ ./litecoin-cli --version
Litecoin Core RPC client version v0.16.0.0-g3567effe2049

So all up (including installing the depedancies) 30 minutes to cross compile a litecoin node (including the wallet) for your raspberry pi. Seeing that the current stable is 0.15.1, and v0.16.0 is on RC, you’ll be upgrading soon so we have already saved some time

Issues I ran into along the way

Firstly I ran ./autogen.sh, ./configure and make directly in the litecoin folder without first building in the depends/ folder. I soon realised I needed Berkley DB. So I downloaded it and cross compiled it for arm. Then tried ./configure again using LDFLAGS="-static -L${BDB_PREFIX}/lib/" CXXFLAGS="-I${BDB_PREFIX}/include/ as arguments to point to the compiled Berkley DB files. This got me further but then I started running into libboost issues with an error similar to bitcoinserver_a error adding symbols: Archive has no index; run ranlib to add one this is because of a mismatch in architecture as it was referencing the x86_64 libboost files, turns out I need the arm version of boost libraries. I attempted to install multiarch on my desktop but that did not work, and eventually I found a pre-compiled version of libboost for arm on the rasbperrypi.org forums. I downloaded that and used --with-boost-libdir=/home/george/lib within ./configure to tell it about the arm libboost files. This got me further, however of course I ran into another issue univalue/.libs/libunivalue.a: error adding symbols: Archive has no index; run ranlib to add one. This one had me stumped and I was going to throw in the towel (as this was a few hours in)

But after researching further I stumbled across the /depends/ folder in the bitcoin repo This had some useful information regarding cross compiling. This allowed you to get all the dependancies for whatever architecture you were targeting without having to do it yourself. I do not know why I had not come across this earlier.

Sites that helped me along the way

The following links were very helpful during this journey, so I’d just like to recognise them here too:

Moving Magento

Mon, 16 Apr 2018 10:00:00 +0000

So I was asked to help finish moving a magento (1.9) installation to a new server the other day. To sum it up, it was not fun

They had already moved the files and had everything installed and updated the database connection string, however it was still throwing an error. The plan was to set this site up on a subdomain, so we could test a few things, update some things and then go live on the new server

After a while of trying to find the logs ( looking in /var/log/apache and /var/www/magentoinstall/var/log) I was having no luck of trying to find out what was causing the error. Until I found /var/www/magentoinstall/var/report. This folder seemed to be what I was looking for. After the fact, looking up some magento docs, var/report seems to be where magento logs uncatched errors by default here is a nice run down. Every time the site failed to load, it spat out a file into this folder, which included an interestly useful error

root@myserver:/var/www/magentoinstall/var/report# cat 828704810349
a:4:{i:0;s:113:"SQLSTATE[HY000] [2003] Can't connect to MySQL server on 'myoldDBserver' (4)";i:1;s:2753:"#0 /var/www/magentoinstall/lib/Zend/Db/Adapter/Pdo/Mysql.php(109): Zend_Db_Adapter_Pdo_Abstract->_connect()
#1 /var/www/magentoinstall/lib/Varien/Db/Adapter/Pdo/Mysql.php(320): Zend_Db_Adapter_Pdo_Mysql->_connect()
#2 /var/www/magentoinstall/lib/Zend/Db/Adapter/Abstract.php(460): Varien_Db_Adapter_Pdo_Mysql->_connect()
#3 /var/www/magentoinstall/lib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('SET NAMES utf8', Array)
#4 /var/www/magentoinstall/lib/Varien/Db/Adapter/Pdo/Mysql.php(428): Zend_Db_Adapter_Pdo_Abstract->query('SET NAMES utf8', Array)
#5 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Resource.php(169): Varien_Db_Adapter_Pdo_Mysql->query('SET NAMES utf8')
#6 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Resource.php(110): Mage_Core_Model_Resource->_newConnection('pdo_mysql', Object(Mage_Core_Model_Config_Element))
#7 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(320): Mage_Core_Model_Resource->getConnection('core_write')
#8 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(350): Mage_Core_Model_Resource_Db_Abstract->_getConnection('write')
#9 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(335): Mage_Core_Model_Resource_Db_Abstract->_getWriteAdapter()
#10 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(360): Mage_Core_Model_Resource_Db_Abstract->_getReadAdapter()
#11 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Resource/Db/Collection/Abstract.php(134): Mage_Core_Model_Resource_Db_Abstract->getReadConnection()
#12 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Config.php(1354): Mage_Core_Model_Resource_Db_Collection_Abstract->__construct(Object(Mage_Core_Model_Resource_Website))
#13 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Config.php(1386): Mage_Core_Model_Config->getModelInstance('core_resource/w...', Object(Mage_Core_Model_Resource_Website))
#14 /var/www/magentoinstall/app/Mage.php(491): Mage_Core_Model_Config->getResourceModelInstance('core/website_co...', Object(Mage_Core_Model_Resource_Website))
#15 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Abstract.php(208): Mage::getResourceModel('core/website_co...', Object(Mage_Core_Model_Resource_Website))
#16 /var/www/magentoinstall/app/code/core/Mage/Core/Model/Abstract.php(213): Mage_Core_Model_Abstract->getResourceCollection()
#17 /var/www/magentoinstall/app/code/core/Mage/Core/Model/App.php(608): Mage_Core_Model_Abstract->getCollection()
#18 /var/www/magentoinstall/app/code/core/Mage/Core/Model/App.php(466): Mage_Core_Model_App->_initStores()
#19 /var/www/magentoinstall/app/code/core/Mage/Core/Model/App.php(349): Mage_Core_Model_App->_initCurrentStore('', 'store')
#20 /var/www/magentoinstall/app/Mage.php(684): Mage_Core_Model_App->run(Array)
#21 /var/www/magentoinstall/index.php(153): Mage::run('', 'store')

It was interesting as it was using the database server connections string, re-checking all the config files (everything under /var/www/magentoinstall/app/etc just to make sure) I finally resorted to doing a search for the old connection string (using ag ) in the magento instance

Magento seems to agressively cache everything. It found a few hits of the old connection string under /var/www/magentoinstall/var/cache in classPathCache.php and a few of the mage--xxx folders. After a quick search, it was deemed OK to wipe this folder clean. Sure enough, after deleting the cache folder we had progress. No more errors, but the site was redirecting to the main domain of the website (which was not good as we wanted to set it up on a subdomain first)

Without access to the admin interface to update the domain, modifying this via the database was the next best option. There were a few strings that we need to update to make sure nothing breaks, seems that magento tried to do an inbuild CDN style configuration so it pulls assets from varying domains/paths to speed things up

root@myserver:/var/www/magentoinstall# mysql -u root -p -h mynewDBserver
mysql> use database mymagentodb;
# Main domain
mysql> update core_config_data set value = 'https://my.new.domain/' where path = 'web/secure/base_url';
mysql> update core_config_data set value = 'http://my.new.domain/' where path = 'web/unsecure/base_url';
#Media 
mysql> update core_config_data set value = 'http://my.new.domain/media/' where path = 'web/unsecure/base_media_url';
mysql> update core_config_data set value = 'https://my.new.domain/media/' where path = 'web/secure/base_media_url';
#JS 
mysql> update core_config_data set value = 'http://my.new.domain/js/' where path = 'web/unsecure/base_js_url';
mysql> update core_config_data set value = 'https://my.new.domain/js/' where path = 'web/secure/base_js_url';
#CSS 
mysql> update core_config_data set value = 'http://my.new.domain/css/' where path = 'web/unsecure/base_css_url';
mysql> update core_config_data set value = 'https://my.new.domain/css/' where path = 'web/secure/base_css_url';

Okay great. More progress, the site is loading, but not over https. I was getting a 404 error when trying to load over https. Back to the drawing board This was just a config error within apache. I had to change the default SSL apache config, to point to the right folder as the document root, enable the site and restart apache

Hooray!

One other thing I can across in my research was that Magento 1.9 reads in all and any xml files that are located in app/etc. So if you have copied the local.xml file during development and made local-dev.xml it will still load that in and read any configuration settings that have been placed in here. This tripped alot of people up as it would load old or development database connection strings in and mess things up. So rename things to .old or .dev or something other then .xml

PiHole and OpenHAB

Thu, 08 Feb 2018 10:00:00 +0000

Recently I installed pihole on my network because I was sick of ads, and wanted an added layer of security on my network against nasty malware and the like

There are plenty of install guides out there for pihole (see official website) so thats not what I am covering today. I installed this on a raspberry pi 1 (Model B) and it handles everything just fine!

Becauase I am a fan of home automation, I wanted all the data that is available in the piHole web interface, avaialable to me at my openHAB interface. You know, so everything is all in the one place. Thankfully this was super easy because piHole has a simple API that you can extract this data from. This also allows you to temporailty disable piHole. Which is useful when your wife wants to clicks ads anyway, or if you know piHole is causing mischef with a particular website

piHole

We need to first make sure that the webpassword is set in the piHole config. This will be used to authenticate against the API (I beleive you can turn this off if you desire)

$ cat /etc/pihole/setupVars.conf

Look for the line WEBPASSWORD

If its not set, then set it by adding a line in the config file, making up a random string

OpenHAB

Now lets make some items in openhab to capture the data we want, and an entry in the sitemap to display it - $ sudo vim /etc/openhab2/services/http.cfg Set up a regular http call as a service, so it will go ahead a poll the API every so often. This is also useful because you can use this service against as many items as you want, reducing the amount of API calls you do to only as many as this service does.

http:pihole.url=http://pi.hole/admin/api.php?summaryRaw
http:pihole.updateInterval=300000

$ sudo vim /etc/openhab2/items/pihole.items Add some items for your sitemaps to use

Number piholePercent    "Percent blocked [%.1f %%]" <pihole>    { http="<[pihole:303000:JSONPATH($.ads_percentage_today)]"}
Number piholeBlockedAds "Blocked ads [%d]" <pihole>     (pihole)        {http="<[pihole:303000:JSONPATH($.ads_blocked_today)]"}
Number  piholeQueries   "DNS Queries [%d]" <pihole>     (pihole)        {http="<[pihole:303000:JSONPATH($.dns_queries_today)]"}
Switch piholeDisable "Toggle Pi Hole" (pihole) {http=">[OFF:GET:http://pi.hole/admin/api.php?disable=600&auth=<WEBPASSWORD>] >[ON:GET:http://pi.hole/admin/api.php?enable&auth=<WEBPASSWORD>]"}

The first 3 items use the item set in the http.cfg. you can see is uses JSONPATH to do some transformation to only get the data we want for the particular item. In the case of piholePercent we are grabbing the ads_percentage_today value out of the JSON output from the API. I’ve also created a switch to disable and enable piHole as needed. This API function requires you to authenticate yourself so we needed to add that in there too, this is where the WEBPASSWORD value in the piHole setting comes into play

Sitemap. Add the items to your sitemap as you see fit (Here is my example from my sitemap)

Frame label="Pi Hole"{
                Text item=piholeQueries
                Text item=piholeBlockedAds
                Text item=piholePercent
                Switch item=piholeDisable
        }

Optional If you’d like to add an icon for the items in the sitemap further, download the graphics from piHole (png and svg) and upload them into your openhab install at /etc/openhab2/icons/classic. Then set the icon in the item using <pihole> (Refer to the openhab item config above). Now you will get a nice icon for the item too

More info on the API can be found here

Useful Commands - MSSQL

Thu, 08 Feb 2018 10:00:00 +0000

Here is a start of a regular posting, “useful commands” Over the years I have saved snippets, commands and notes of things that have been useful or saved alot of time. I start off with a gathering of T-SQL commands to do useful things within a windows database. Some of these relate directly to modifying a Kentico database (the CMS of choice where I work), but may be useful anyway

-- Turn off SSL
UPDATE [CMS_SettingsKey]  SET KeyValue = 'False'WHERE KeyName LIKE 'CMSUseSSL%'


-- Take Database offline forcefully and come back online
#Go offline
USE master
GO
ALTER DATABASE YourDatabaseName
SET OFFLINE WITH ROLLBACK IMMEDIATE
GO
#Go online
USE master
GO
ALTER DATABASE YourDatabaseName
SET ONLINE
GO
# get out of recovery mode
RESTORE DATABASE <database name> WITH RECOVERY


-- Get MDF/BAK file SQL version

#Bak file (in sql)
RESTORE HEADERONLY FROM DISK = 'D:\dbbackup\base106.bak'

#MDF (powershell)
get-content -Encoding Byte "...\foo.mdf" | select-object -skip 0x12064 -first 2 number 2 * 256 + number 1


-- Get DB Size

EXEC sp_spaceused
CREATE TABLE #t 
( 
    [name] NVARCHAR(128),
    [rows] CHAR(11),
    reserved VARCHAR(18), 
    data VARCHAR(18), 
    index_size VARCHAR(18),
    unused VARCHAR(18)
) 

INSERT #t EXEC sp_msForEachTable 'EXEC sp_spaceused ''?''' 

SELECT *
FROM   #t

SELECT SUM(CAST([rows] AS int)) AS [rows]
FROM   #t
DROP TABLE #t


-- Get RAM usage of server
SELECT db = DB_NAME(t.dbid), plan_cache_kb = SUM(size_in_bytes/1024) 
FROM sys.dm_exec_cached_plans AS p
CROSS APPLY sys.dm_exec_sql_text(p.plan_handle) AS t
WHERE t.dbid < 32767
GROUP BY t.dbid
ORDER BY plan_cache_kb DESC;

Ubuntu Live USB with persistence

Sun, 04 Feb 2018 10:00:00 +0000

Creating a live USB is easy, adding persistence seems to be quite hard. After lots of research, there was alot of ways to do it, but most of these tutorials were out of date so some things weren’t relevant anymore or just didn’t work

Here are some hopefully simple steps to create a live USB with persistence, using Ubuntu. This will also be available as a normal USB flash drive to windows users

Requirements

Ubuntu ISO
USB flash drive. 4G is ideal, although you probably could have it on 2GB
Device already running linux

Steps

Firstly we will format the USB to FAT32 and wipe anything from it. You can use gparted if you want

#find your usb's device location
$ sudo fdisk -l

Disk /dev/sdb: 14.3 GiB, 15376000000 bytes, 30031250 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00766c29

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdb1  *     2048 30031249 30029202 14.3G  c W95 FAT32 (LBA)

#Now format as FAT32 (Make sure its unmounted)
$ sudo mkfs.vfat /dev/sdb

Now lets copy over the ubuntu ISO to the USB (this may take some time)

$ sudo dd if=~/Download/ubuntu-17.10.1-desktop-amd64.iso of=/dev/sdb1 status=progress

Once finished, mount up the drive so we can modify the boot config to tell it to be persistent. We will modify the grub config and also the isolinux config $ sudo mount /dev/sdb1 ~/usb/

$ sudo vim usb/boot/grub/grub.conf
.....
menuentry "Try Ubuntu without installing" {
        set gfxpayload=keep
        linux   /casper/vmlinuz.efi persistent file=/cdrom/preseed/ubuntu.seed boot=casper quiet splash ---
        initrd  /casper/initrd.lz
}
.....
# Add "persistent" after /casper/vmlinuz.efi

$ sudo vim usb/isolinux/txt.cfg
.....
label live
  menu label ^Try Ubuntu without installing
  kernel /casper/vmlinuz.efi
  append  file=/cdrom/preseed/ubuntu.seed boot=casper persistent initrd=/casper/initrd.lz quiet splash ---
.....
# Add "persistent" to the append line, just after the boot=casper statement

We must now create a file (or you could shrink the partition and make a new partition, it does the same thing) for where the persistent data will store itself. It will store in a file (or partition) called casper-rw. I have chosen this to only be 1Gb, but you can choose your size below. There are several ways to create a pre-space-populated file, this is just one of them

sudo dd if=/dev/zero of=usb/casper-rw bs=1M count=1024

Now we will format the file into an ext4 format so the OS can read it

sudo mkfs.ext4 usb/casper-rw

All finished. Unmount and go boot from it. If you find that 1Gb isnt enough, you can resize that casper-rw using the following

#Create new block, the size of what you want to add. E.g 1Gb to make a 2Gb file
$ dd if=/dev/zero of=/tmp/tmpfile bs=1M count=1024
#Append new block to existing block
$ cat /tmp/tmpfile >> usb/casper-rw
# Enlarge the filesystem
$ sudo e2fsck -f usb/casper-rw
$ sudo resize2fs usb/casper-rw

Thanks alot to this tutorial that was the most helpful, altyhough I was unable to get the 2 partition USB thing to work, like in the example in that article.

RuuviTag

Wed, 19 Apr 2017 09:00:00 +0000

I backed Ruuvi sometime late last year for a RuuviTag, a battery powered Bluetooth LE microcontroller with an onboard temperature and humidity sensor. As soon as I saw this project I had several ideas. And because I run openHAB at home, naturally, I’d want to collate this date into here for usage in my home automation system. I finally got it in the mail last week, and what surprised me was that they had teamed up the the guy who made espruino and had him port it to the Ruuvi! Javascript on a microcontroller. Pretty impressive stuff. Me and javascript have had some okay times, so I was comfortable that they had javascript here instead of C, which is what Ruuvi natively runs

So battery powered BLE temperature sensor, that runs javascript, and a raspberry pi 3 that is running openhab. Pretty much a perfect mix. Here is how I got them talking to each other

You may be looking at the RuuviTag setup page and see that the default weather station firmware actually works for both RuuviTags. At the time of getting my Ruuvi, it only supported the RuuvTag+, so I looked at using the espruino firmware to get what I needed to work.

Ruuvi

Following the instructions here we need to flash the espruino firmware to ruuvi. Quick overview, but the video they have here is quite good
- Download nRF connect app
- Put Ruuvi into boot mode (hold R + B, let go of R while still holding B until red light comes and stays on)
- Connect to the ruuvi with nRF connect
- Press the “DFU” button that comes up in the app once you are connected, and select the espruino firmware zip file
- wait a little while
Now follow the instructions here to get the IDE up and running to connect to Ruuvi
Once you are able to get into the IDE and connect to ruuvi, upload this little snippet. Essentially it gathers the temp data every 60 seconds, converts it to a uint8 array, and then advertises it over BLE
- after following this guide, you may be asking why I’m not using the ruuvitag package to get all the data var Ruuvitag = require("Ruuvitag");. It seems that this package does not work with the RuuviTag basic. I need to raise this on the forums to see when it will be supported. But for now we can just use the code you would use for the puck.js on the ruuvi, and it does the same job
- Links to the espruino forum that helped me get this working one, two, three

function tempTo(temp) {
  var f = parseFloat(temp.toFixed(2)) * 100;
  if (tempTo.buf === undefined) tempTo.buf = new Uint8Array(2);
  tempTo.buf[0] = f & 0xff;
  tempTo.buf[1] = f >> 8 & 0xff;
  return tempTo.buf;
}

setInterval(function(fn){
  console.log(E.getTemperature());
},60000);

setInterval(function() {
  NRF.setAdvertising({
    0x1809 : tempTo(E.getTemperature())
  });
}, 60000);

RaspberryPi 3

Now onto the server side. I have a raspberry pi 3 running openhab. This is perfect as it has bluetooth built in. There were a few options to make this work, if you head over to the ruuvi forums there is a java and python module to scan the ruuvi’s and get their data. I was actually looking for something more like this. This is a BLE -> mqtt bridge, that will scan for data on ruuvi (or Puckjs) and then publuish that to a mqtt service. EspruinoHub didn’t quite fit my setup, it is apparently a 2 way bridge, but I only really wanted a 1 way (getting the data). I also did not want to install nodered and all the other dependancies for it. So I decided to use this as a basis and write something else (source here) It does esentially the same job, scans the BLE range and gets any data being advertised, and then publishes it to a mqtt stream. My openHAB server subscribes to these streams to get the data into the system. One use case I have for this is to monitor a bedroom temperature, and then if it gets to cold overnight, turn on the heating. Keep monitoring the temperature and if it gets a bit too hot, turn it off again.

Setup

So firstly during my trials, and some help from the Ruuvi Slack channel, we need to upgrade the version of bluez that is on the raspberry pi to the latest. There is apparently an issue in 5.23 that was causing me some grief.
First lets install all the dependancies we will need sudo apt-get install bluetooth bluez bluez-hcidump libbluetooth-dev libudev-dev
Add the stretch repo (with the newer bluez versions) to apt source list
- sudo vim /etc/apt/sources.list -> add deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
- sudo vim /etc/apt/apt.conf.d/40defaultrelease -> add APT::Default-Release "jessie";
Update package list sudo apt-get update
Upgrade bluez using the strech repo sudo apt-get install bluez bluez-hcidump -t stretch
Now we need node. I’m going to install the latest, but the LTS version works great too
- curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
- sudo apt-get install nodejs npm

So before we go any further, lets do some testing. We can use the tools from the bluez package to scan BLE and see what data we can find

$ sudo hcitool lescan
LE Scan ...
CE:A2:59:0E:85:C2 RuuviTag 85c2
CE:A2:59:0E:85:C2 (unknown)

Cool, we can see ruuvi in the scan! No lets see what data ruuvi is giving us

$ sudo hcitool lescan & sudo hcidump --raw
[1] 18782
LE Scan ...
HCI sniffer - Bluetooth packet analyzer ver 5.43
device: hci0 snap_len: 1500 filter: 0xffffffff
CE:A2:59:0E:85:C2 RuuviTag 85c2
> 04 3E 24 02 01 00 01 C2 85 0E 59 A2 CE 18 02 01 05 05 16 09
  18 46 05 0E 09 52 75 75 76 69 54 61 67 20 38 35 63 32 AC
CE:A2:59:0E:85:C2 (unknown)
> 04 3E 1E 02 01 04 01 C2 85 0E 59 A2 CE 12 11 07 9E CA DC 24
  0E E5 A9 E0 93 F3 A3 B5 01 00 40 6E AC

Pretty neat, we can see a raw stream of hex data that the RuuviTag is giving out. Lets try again with the -x flag (show ascii and hex data)

$ sudo hcitool lescan & sudo hcidump -x
HCI sniffer - Bluetooth packet analyzer ver 5.23
device: hci1 snap_len: 1500 filter: 0xffffffff
> HCI Event: LE Meta Event (0x3e) plen 36
    LE Advertising Report
      ADV_IND - Connectable undirected advertising (0)
      bdaddr CE:A2:59:0E:85:C2 (Random)
      Flags: 0x05
      Unknown type 0x16 with 4 bytes data
      Complete local name: 'RuuviTag 85c2'
      RSSI: -77

More interesting things happening. We can see the name of the tag, its MAC address and RSI. Unforuntely we can’t see the data its advertising. I haven’t been able to find out how to do that just with hcidump but we will now turn to some javascript to get this info!

Lets install some node dependacies we will need for this. npm install noble mqtt
- The noble module is pretty much where all the work happens, and we will just need the mqtt module to send messages to our openhab server
I have also borrowed a few bits of code out of the EspruinoHub project, this made my life alot easier. We needed the config.js and attributes.js files out of the lib folder
And here is index.js, that does the scanning and prints the data. Its still a bit of work in progress at the moment, but does exactly what I need. Although I do not have multiple RuuviTags, this solution should work just fine for mulitple tags.

var noble = require('noble');
var attributes = require('./attribute');
var config = require('./config');
var mqtt = require('mqtt');
// my mqtt server needs some authentication, if you dont, you can just remove that object
var client  = mqtt.connect('mqtt://10.1.1.115', { username: 'Username', password: 'Password' })
var connected = false;
var inRange = {};
var packetsReceived = 0;
var lastPacketsReceived = 0;
var isScanning = false;

noble.on('stateChange', function(state) {
  if (state === 'poweredOn') {
    noble.startScanning([], true);
  } else {
    noble.stopScanning();
  }
});
client.on('connect', function () {
  connected = true;
});

noble.on('discover', function(peripheral) {
  packetsReceived++;
  var addr = peripheral.address;
  var id = addr;
  if (id in config.known_devices)
    id = config.known_devices[id];
  var entered = !inRange[addr];

  if (entered) {
    inRange[addr] = {
      id : id,
      address : addr,
      peripheral: peripheral,
      name : "?",
      data : {}
    };
  };
  inRange[addr].lastSeen = Date.now();
  inRange[addr].rssi = peripheral.rssi;

  console.log('hello my local name is:');
  console.log('\t' + peripheral.advertisement.localName);

  var serviceData = peripheral.advertisement.serviceData;
  //If the advertising BLE device has any service data, then lets loop throught it and find out what it is
  if (serviceData && serviceData.length) {
    for (var i in serviceData) {
      //If statement to limit the outgoing mqtt stream. We are checking to see if the device is already in our array we made above, and if it hasnt been seen in 60s, and that data hasnt changed
    if (inRange[addr].data[serviceData[i].uuid] &&
        inRange[addr].data[serviceData[i].uuid].payload.toString() == serviceData[i].data.toString() &&
        inRange[addr].data[serviceData[i].uuid].time > Date.now()-60000){
            console.log("waiting....");
        }else{
          //Bit of debugging here so we can see whats going on
      console.log('\t\t' + JSON.stringify(serviceData[i].uuid) + ': ' + JSON.stringify(serviceData[i].data.toString('hex')));
      var n = attributes.getReadableAttributeName(serviceData[i].uuid);
      console.log(JSON.stringify(attributes.decodeAttribute(n,serviceData[i].data))+"\n");
      inRange[addr].data[serviceData[i].uuid] = { payload : serviceData[i].data, time : Date.now() };
      var extendedData = attributes.decodeAttribute(n,serviceData[i].data);
      //Can only sent mqtt if its actually connected, so if connected send data. We are only sending the temp number, not the whole json string which looks like { "temp": 20 } in debug messages above
      if (connected && extendedData.temp) client.publish(inRange[addr].peripheral.id +'/temp', extendedData.temp.toString());
    }
   }
  }
  if (peripheral.advertisement.manufacturerData) {
    console.log('\there is my manufacturer data:');
    console.log('\t\t' + JSON.stringify(peripheral.advertisement.manufacturerData.toString('hex')));
  }
  if (peripheral.advertisement.txPowerLevel !== undefined) {
    console.log('\tmy TX power level is:');
    console.log('\t\t' + peripheral.advertisement.txPowerLevel);
  }

  console.log();
});

If you go ahead and run this, nodejs index.js you will see somethings being printed out to console. Once it picks up Ruuvi, it should tell you the temperature now! - The script is set to only re-send data on change, or ever 60 seconds. This is just so mqtt isn’t flooded with messages

hello my local name is:
        RuuviTag 85c2
                "1809": "0807"
{"temp":18}


hello my local name is:
        RuuviTag 85c2
waiting....

hello my local name is:
        RuuviTag 85c2
waiting....

hello my local name is:
        f022cXhN
        here is my manufacturer data:
                "0f03000276064ef39800d605020000dae480"

Now lets go over to openhab, and makde sure its ready to receive some data - Pre-requisites for this is obviously having mqtt biniding installed and working - vim /etc/openhab2/items/ruuvi.items - Add something like this - Number ruuvi "RuuviTag [%.2f C]" (climate) {mqtt="<[mosq:cea2590e85c2/temp:state:default]"} - So the node script is publishing to a mqtt topic by the ruuvitag ID, which is just the tag’s MAC address without any : - While you have the node script running, tail the openhab log to make sure things are working. tail -f /var/log/openhab2/events.log

2017-04-20 15:17:12.992 [ItemStateChangedEvent     ] - piholeBlockedAds changed from 665 to 674
2017-04-20 15:19:42.802 [ItemStateChangedEvent     ] - ruuvi changed from 18 to 18.25
2017-04-20 15:20:43.626 [ItemStateChangedEvent     ] - ruuvi changed from 18.25 to 18.5
2017-04-20 15:21:42.921 [ItemStateChangedEvent     ] - ruuvi changed from 18.5 to 18
2017-04-20 15:22:16.388 [ItemStateChangedEvent     ] - piholeQueries changed from 14852 to 15265
2017-04-20 15:22:16.397 [ItemStateChangedEvent     ] - piholePercent changed from 4.5381093455427 to 4.5070422535211
2017-04-20 15:22:16.401 [ItemStateChangedEvent     ] - piholeBlockedAds changed from 674 to 688

Excellent. We can see that we are getting updates into openhab. Now make sure you are persisting this data (I’m using the rrdj4 persistance) and you can get some pretty charts to display into your sitemap

So lets tidy this up and manage the node script with systemd, so we don’t have to have it running in a screen or something funky like that. Just put this into /etc/systemd/system/ruuvi.service

[Unit]
Description=Ruuvi-Scanner

[Service]
ExecStart=/usr/bin/node index.js
Restart=always
User=pi
Group=pi
WorkingDirectory=/home/pi/ruuvi

[Install]
WantedBy=multi-user.target

Now you can start and stop it with systemctl

So here you have it. A RuuviTag, lots of javascript and openHAB. All working together. Now I just need to buy some more, 1 for each room!

I will also upload the files to github, so you can just download it from there too..

Node and systemd

Fri, 17 Feb 2017 09:00:00 +0000

For a while now I’ve had my dashboard combo tv and dashboard combo, which runs on dashing and dashkiosk, running in screen sessions. This , of course, is not very managable. If something breaks, you have to ssh in and then find the screen session, re-run the command. In comes systemd. This makes managing services easy! You are able to make anything a system managed service, being able to manage the service via the systemctl commands

Here are the systemd unit files I use for both dashing and dashkiosk. Just put them in /etc/systemd/system and away you go

Dashkiosk

[Unit]
Description=Dashkiosk dashboard admin

[Service]
ExecStart=/usr/bin/node server.js --environment production
Restart=always
User=root
Group=root
WorkingDirectory=/home/george/dashkiosk-2.6.6/dist/

[Install]
WantedBy=multi-user.target

Dashing

[Unit]
Description=Dashing Dashboard

[Service]
ExecStart=/usr/local/bin/dashing start -e production -p 80
Restart=always
User=root
Group=root
WorkingDirectory=/home/george/work-dashboard/

[Install]
WantedBy=multi-user.target

As you can see here, I am starting my services as root, as I need to use port 80 for the dashboard. But you can probably just run these as a standard user if you do not need privileged ports

OneTimeSecret

Tue, 07 Feb 2017 10:00:00 +0000

So for those who use OnetimeSecret all the time, you can add this little function to your .bashrc (or similar) So then you don’t have to openh a browser to share your secret!

onetimesecret() {
if [ $# -eq 0 ]; then echo "No secret specified";return 1; fi
secret=$(curl -s -F "secret=$1" https://onetimesecret.com/api/v1/share | cut -d '"' -f12)
echo https://onetimesecret.com/secret/$secret
}

Output looks simlar to this

george@george-LT:~$ onetimesecret Thisissupersecret
https://onetimesecret.com/secret/kvq9927e3ylltd83u679vlyfr942vjg

I found this very useful as I use it daily, and I can pipe it into any other cli messaging app (eg into slackcat)

C.H.I.P, flic and openhab

Thu, 05 Jan 2017 10:00:00 +0000

So I finally received my C.H.I.P, and over the break I purchased myself some flic bluetooth buttons with the plan of making some nice push buttons around the home to control openhab. This idea mostly came from usablity/speed of use of openhab. Having the dashboard in the middle of the house is a nice idea, but if you are in another room, without your phone there isn’t too much you can do without having to go to this dashboard to interact with openhab. Flic makes it nice and convenient to do things you do often, with a click, double click or press of a button!

Some examples, incase this still seems silly to you.

A button by the back door
- A single click it will “turn off” the house. This includes lights, lamps and music from my mpd server.
- double click on the way back in, and it will turn the main house lights back on for you
- Long press to update openhab that you are now home. This then triggers some other openhab rules
A button by my bedside table
- Single click to turn bedside table lights on night mode (For those middle of the night wake ups)
- Double click to turn the main house lights on (useful for winter when its still dark when I wake up)
- Long press to turn all the lights off. For bed time
A button by my wife’s bed side table
- Single click to turn the hall way, and lounge room lights on night mode. For when she has to feed the baby
- Double click to turn those back off
- Long press to turn all lights off, for bed time
Button on a clip that sits around the house
- Single click to play music
- Double click to pause music
- Long press to skip song

This makes some everyday tasks much more convenient, without having to pull your phone out and load openhab (if you aren’t near the dashboard). The only con of this setup I have found so far is that, as it runs off bluetooth, the range of the buttons is quite limited if using the on-board bluetooth controller of C.H.I.P. To exend it nicely through your house, I recommend purchasing a usb bluetooth controller (I will update this once I find a suitable one for my setup)

Setup

So C.H.I.P comes preinstalled with a GNU/Linux debian based OS customised specifically for C.H.I.P. This is nice as I am very familiar with debian. Connecting to C.H.I.P was super easy. Plug into usb on a desktop and it presents itself as a serial connection you can then screen into! Their docs are quite well written so it was a breeze to follow

Flic have developed a linux sdk for working with the buttons, obviously aimed at developers, so you can extend the buttons further then using the smart phone apps. This is nice as instead of having to pair the buttons with your phone (which moves around all the time) you can create a dedicated devices in 1 place in the house to use and interact with flic. Anyway, some initial setup notes on connecting flic to C.H.I.P

#update the chip
sudo apt-get update
sudo apt-get upgrade

#Install some useful things

sudo apt-get install git
sudo apt-get install bash-completion screen

#Clone the flic linux sdk

git clone https://github.com/50ButtonsEach/fliclib-linux-hci.git

Now at this point I had no idea what to use to script up flic, but the SDK comes with some nice client librarys (and examples!) for C, Java, nodejs python and websocket. Out of this list, I have had the most experience with javascript lately , so I chose Nodejs. So lets install nodejs and npm

sudo apt-get install nodejs npm

So because I have an ISP that offers IPv6 too, I was running into an issue where apt was timing out connecting to a debian repository over IPv6, so I had to force apt to only use IPv4

sudo apt-get -o Acquire::ForceIPv4=true install nodejs npm

A weird thing you need to do for npm to make sure its up to date with the latest

sudo npm install -g npm

And we also need the http library for nodejs

sudo npm install -g http

Now we have all we to get things running! Just for now, we will start the server in a screen so we can see whats happening if needed. It also starts in daemon mode too, but we will use that one later to integrate with systemd

cd fliclib-linux-hci/bin/armv6l
./flicd -f sqlite.db

So the flic server will store all the information in needs into this sqlite db, make sure you have it in a safe location so you dont have to re-pair your buttons again! Now onto the client.

cd fliclib-linux-hci/clientlib/nodejs
nodejs newscanwizard.js

If you check your server screen it will also show that a “client” has connected to it

Now press one of your buttons and watch some magic! Save that address of the button for later, we will now write up a small client to handle the buttons and make them do things when you press them

vim nodejs-flic.js

#!/usr/bin/nodejs
var fliclib = require("/home/chip/Documents/fliclib-linux-hci/clientlib/nodejs/fliclibNodeJs");
var FlicClient = fliclib.FlicClient;
var FlicConnectionChannel = fliclib.FlicConnectionChannel;
var FlicScanner = fliclib.FlicScanner;
var http = require("http");
var client = new FlicClient("localhost", 5551);

function listenToButton(bdAddr) {
        var cc = new FlicConnectionChannel(bdAddr);
        client.addConnectionChannel(cc);
        //There are other methods you can watch, but all I care about at the moment is what type of click just happened
        cc.on("buttonSingleOrDoubleClickOrHold", function(clickType, wasQueued, timeDiff) {
                console.log(bdAddr + " " + clickType + " " + (wasQueued ? "wasQueued" : "notQueued") + " " + timeDiff + " seconds ago");
                //If statement to find out what button was pressed. Probably turn this into case to make it neater
                if(bdAddr === "ab:cd:ef:11:22:33"){
                        console.log("Blue button");
                        /*Create http request to openhab API
                        hostname: openhab server IP address
                        port: openhab port (default 8080)
                        path: Path to the item you wish to change.
                        */
                        var options = {
                                    hostname: '1.2.3.4',
                                    port: 8080,
                                    path: '/rest/items/all_off',
                                    method: 'POST',
                                    headers: {
                                            'Content-Type': 'text/plain'
                                    },
                            };
                            var req = http.request(options, function(res) {
                                    console.log('STATUS:' + res.statusCode);
                                    console.log('HEADERS:' + JSON.stringify(res.headers));
                                    res.setEncoding('utf8');
                                    res.on('data', function(body) {
                                            console.log('BODY:' + body);
                                    });
                                    res.on('error', function(e) {
                                            console.log('problem ' + e.message );
                                    });
                            });
                        if(clickType === "ButtonSingleClick"){
                                //On single click, send OFF to the openhab item
                                req.write("OFF");
                                req.end();
                        }else if(clickType === "ButtonDoubleClick"){
                                //On double click, send ON to the openhab item
                                console.log("doubelclick");
                                req.write("ON");
                                req.end();
                        }else if(clickType === "ButtonHold"){
                                console.log("hold");
                        }else{
                                console.log("Something Wrong");
                        }
                }
        });
        cc.on("connectionStatusChanged", function(connectionStatus, disconnectReason) {
                console.log(bdAddr + " " + connectionStatus + (connectionStatus == "Disconnected" ? " " + disconnectReason : ""));
        });
}

client.once("ready", function() {
        console.log("Connected to daemon!");
        client.getInfo(function(info) {
                info.bdAddrOfVerifiedButtons.forEach(function(bdAddr) {
                        listenToButton(bdAddr);
                });
        });
});

client.on("bluetoothControllerStateChange", function(state) {
        console.log("Bluetooth controller state change: " + state);
});

client.on("newVerifiedButton", function(bdAddr) {
        console.log("A new button was added: " + bdAddr);
        listenToButton(bdAddr);
});

client.on("error", function(error) {
        console.log("Daemon connection error: " + error);
});

client.on("close", function(hadError) {
        console.log("Connection to daemon is now closed");
});

As you can see in my terribly coded example, we are watching for a button press, and when a button press happens, we find out which button it was, what sort of press it was, and do an action. So in those gaps you can do anything you want! In my case I am just hitting the openhab rest API to action the items I have connected to openhab. This is where the SDK is nice, you can do anything you want here!

You may notice here that I only specify 1 openhab item that I am changing the state of. This is because I am doing the rest of the work via the openhab rules engine. The item in openhab is just a switch, that binds to nothing. The entry in default.items looks like this Switch all_off "All Off" You can also add this as a switch item to your sitemap, so you can run the rule below from the openhab UI. Here is the rule that handles this blue button (all_off item)

#/etc/openhab/configuration/rules/flic.rules
import org.openhab.core.library.types.*

rule "All off"
when
        Item all_off received command OFF
then
        sendCommand(L_light, 0)
        Thread::sleep(500)
        sendCommand(K_light, OFF)
        Thread::sleep(500)
        sendCommand(E_light, OFF)
        Thread::sleep(500)
        sendCommand(H_light, OFF)
        Thread::sleep(500)
        sendCommand(B_light, OFF)
        Thread::sleep(500)
        sendCommand(BT_light, OFF)
        Thread::sleep(500)
end

rule "Home on"
when
        Item all_off received command ON
then
        sendCommand(K_light, ON)
        Thread::sleep(500)
        sendCommand(L_light, 100)
        Thread::sleep(500)
        sendCommand(BT_light, ON)
        Thread::sleep(500)
        sendCommand(K_light_dim, 100)
        Thread::sleep(500)
end

I have all these sleep commands in here, as I found that having the commands run directly after one another did not work well with milight, so adding a small delay made sure each one runs successfully and the milight bridge actually gets the command

Now that we have a nice peiece of code that handles our buttons and actions, I didn’t want these to be running in screen sessions. you know, in case of an outage or similar, I didnt want to have to ssh into C.H.I.P to start everything back up again. systemd is nice here. We can write some stuff in a file so systemd can handle running the code as a service for us, just like any other daemon you have installed.

vim flic.service

[Unit]
Description=Flic and openhab

[Service]
ExecStart=/home/chip/Documents/fliclib-linux-hci/clientlib/nodejs/nodejs-flic.js
Restart=always
User=root
Group=root
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/home/chip/Documents/fliclib-linux-hci/clientlib/nodejs/

[Install]
WantedBy=multi-user.target

Make sure you change the user here, and make the .js file executable! I did not, and had some errors.

chmod +x /home/chip/Documents/fliclib-linux-hci/clientlib/nodejs/nodejs-flic.js

Now copy it to where systemd exects these files

sudo cp flic.service /etc/systemd/system/

Start it

sudo systemctl start flic.service

check logs

sudo journalctl -fa -u flic.service

This acts like tail -f on the log file. Now its running under systemd you can add this service to start at startup, and manage it using the systemctl command. Now we just do a similar thing to make the flic daemon managed by systemd

Openhab

Tue, 20 Dec 2016 10:00:00 +0000

Since the ninjablock company has run out of money, and is not supporting the ninjablock anymore, and its cloud service is pretty unreliable (at the best of times) I decided to merge all my homeautomation items with Openhab. At the time I adopted this, it was the only major, multi-vendor supported Home automation “brain” around. Since then home assistant has come around, and alot of others have picked up support for alot more “things”. If I had the money, I would probably jump on the Smart Things wagon, but I don’t :(

So this is a bit of a back post of the system, since I’ve had it running for almost 2 years since Openhab 1.7 came out. But I thought I would just share a bit about it, and my setup

I am still yet to upgrade to OpenHAB 2.0 yet, it seems like a lot of work, it doesn’t look like there is a straight forward upgrade path from 1.x. So I’m still running 1.7, which is working fine for me

Setup in the house

So I had a nexus 7 laying around, and decided I’d use that as a wall mounted controller. This works well with openHAB because it even has a “full screen” mode for you to use in the app, which will hide the status bar and navigation keys. This is an always on display too

I rarely use the web based GUI to access my home, having this controller works well, plus I have the app installed on my various hand held devices too (And my watch!). So I am never too far away from any controls. I’ve never found the need to open up the web GUI when I’ve been at my desk, much quicker just to unlock my phone and start the app. The default web interface is very ugly too :/

Hardware

Raspberry pi

Originally I had openHAB running on a raspberry pi (v1 B) but could notice the lack of power when interfacing with it. This wasnt too much of an issue when I first set it up, but after setting up a dozen or so rules, things started getting pretty sluggish. Doesn’t help that openHAB isn’t the only thing I have running on my pi. Not that I’m complaining, it does really well for the specs it has. However this was all fixed when I upgraded to a Raspberry Pi v2. That thing was such a powerful step up from v1. Incredible. I run openHAB, owncloud, dhcp/dns and various cron started scripts on that thing and It doesn’t even complain, or let me down.

The only dependancies I needed in this situation were java (obviously, to run openHAB) and an MQTT server (I use mosquitto MQTT). This was to interface with the raspberry pi running the ninjablock things

So all up * 1 x Raspberry pi v2 (to Run openHAB and mqtt) * 1 x Raspberry pi v1 model B (With the ninja crust pi hardware attaached to it, including 433MHZ transmitter) * 1 x Raspberry pi as mpd server

Ninja Blocks

Lucky for me, someone else that had a ninja block came to the same conclusion as me, so had already made a nice little python bridge to get the Ninjablock and its 433Mhz sensors into openHab. Old mate Perrin7 posted his link in the now dead Ninjablock forums. This little shim just connects to the Ninja Crust via its serial port, and listens for any events. It reads these events, figures out what device/sensor it was, and then publishes it to a MQTT thread. OpenHab also just listens to these threads and picks up changes as they are posted! I’ve had this running for a while now with no issues :) the only downside (but probably easily fixed) is I have to start this python script everytime the pi with the ninja block on it restarts (Say if I lose power or something). This is what I have that connects to my Ninjablock over 433Mhz * 1 x Button * 1 x Motion sensor * 1 x Door/window sensor * 3 x remote power switches (Watts Clever brand)

Here is the sitemap, and items list in openhab specific to the ninjablock

Switch Motion_Sensor    "Room Sensor"   (Persist_change) {mqtt="<[mosquitto:ninjaCape/input/11:state:ON:1000000001011010
01011001]"}
Switch C_Lamp   "Pergola Lights"        (frontroom)     {mqtt="<[mosquitto:ninjaCape/input/11:state:ON:11011010110110101
1011100],
                                <[mosquitto:ninjaCape/input/11:state:OFF:110110101101101011010100],
                                >[mosquitto:ninjaCape/output/11:command:OFF:110110101101101011010100],
                                >[mosquitto:ninjaCape/output/11:command:ON:110110101101101011011100]"
                                }
Switch D_Lamp   "Door Lamp"     (frontroom)     {mqtt="<[mosquitto:ninjaCape/input/11:state:ON:110110101101101011011110]
,
                                <[mosquitto:ninjaCape/input/11:state:OFF:110110101101101011010110],
                                >[mosquitto:ninjaCape/output/11:command:OFF:110110101101101011010110],
                                >[mosquitto:ninjaCape/output/11:command:ON:110110101101101011011110]"
                                }
Switch Z_Lamp   "Room Heater"   {mqtt="<[mosquitto:ninjaCape/input/11:state:ON:110110101101101011011010],
                                <[mosquitto:ninjaCape/input/11:state:OFF:110110101101101011010010],
                                >[mosquitto:ninjaCape/output/11:command:OFF:110110101101101011010010],
                                >[mosquitto:ninjaCape/output/11:command:ON:110110101101101011011010]"
                                }

Switch Button   "Button"        {mqtt="<[mosquitto:ninjaCape/input/11:command:ON]"}

Limitless LED

Thing was probably the biggest purchase for me at the time, to replace most of my lights around the house with these. Has been a good investment, and already had a addon in openHAB so it was very easy to set up and get working. I have recently found a neat trick with these too, instead of setting the item as Switch in openHAB, you set it as Dimmer. Then when you link the light in the sitemap, you specify it as a Switch. This trick ensures that everyime you click the switch, your lights will go to 100% brightness (On being 100%, Off being 0%). I found this very useful, as before, when you turn the light on again it would turn back on at the last remembered brightness setting, which wasnt always at full brightness. Then I’d have to faff about with the slider bar to get it back up. “But what about if you want to change brightness after you turned the light on”, you ask. You can see the example in my item list and sitemap below. Using the visibility property, you can have the brightness slider show on your sitemap when the light is actually on, this reduces clutter on the screen (seeing as I only have a limited amount of screen space)

 Frame label="Lights"  {
                Switch item=all_off mappings=[OFF="All Off"]
                Switch item=L_light
                Switch item=L_lightn mappings=[ON=ON]
                Slider item=L_light_dim visibility=[L_light==ON]
                Switch item=K_light
                Slider item=K_light_dim visibility=[K_light==ON]
                Switch item=E_light
                Slider item=E_light_dim visibility=[E_light==ON]
                Switch item=H_light
                Slider item=H_light_dim visibility=[H_light==ON]
                Switch item=B_light
                Slider item=B_light_dim visibility=[B_light==ON]
                Switch item=BT_light
                Slider item=BT_light_dim visibility=[BT_light==ON]
        }

MPD server

Because I had an extra raspberry pi and an extra speaker laying around, I turned it into a remote media player with Mopidy (Another post about this shortly). This enabled me to stream straight from spotify, and control it directly from openHAB too! Super cool :)

Smart Home

So this is where openHAB gets cool. Not only can you interface with smart things throughout your house, and have information gathered and displayed all in one central location, you can create rules to do things when certain things happen (Think IFTTT)

For example * In the mornings when my alarm goes off, openHAB turns the kitchen light on for me * When i leave the house in the morning, it will turn that light off again * I can choose a time at night to turn the wall heater in our bedroom on (this will get smarter when my bluetooth temperature sensors arrive) * I can choose a time in the morning to turn the wall heater off * I Also link my phone location into openHAB (using ownTracks) so then my partner can view my location on a map within openHAB

These are just a few examples, but the amount of devices that openHAB supports, the length you can go to to automate your household is quite extensive and exciting

If you want to see my complete sitemap/items list or my rules I use, get in touch. I’d be happy to share and help get you started on the way to home automation with openHAB

Mini DLNA and bubbleUPnP

Fri, 18 Nov 2016 10:00:00 +0000

So ages ago I found bubbleUPnP (also an andoid app). I’d put this in the category of cheap plex server. I say this because the main reason I wanted to use it was for its on-the-fly transcoding to play media to the chromecast. The android app is also really nice, and you can control your media directly from your device too (Much like what plex does)

What I was really impressed with was that the raspberrypi (v1 Model B) handled this without too much hastle, and was able to stream 720p content to the chromecast with no (noticable) worries

Installation is pretty simple:

Ensure you have Java installed sudo apt-get install openjdk-8-jre (or the oracle java if you are so inclined)
Install ffmpeg, if you want on the fly transcoding sudo apt-get install ffmpeg
Download BubbleUPnP zip file from http://www.bubblesoftapps.com/bubbleupnpserver/#other_java_platforms
Unpack the zip file and set the launch.sh file as executable
- unzip BubbleUPNPserver-distrib.zip
- cd BubbbleUPNPServer && chmod +x launch.sh
Run launch.sh

Once its running you can make settings changes by going to serverIP:58050, but the transcoding settings can be setup in the android app once you connect to it

So BubbleUPnP is not a media server (you cannot tell it where videos are for you to then stream) , it just enables alot of cool things ontop of traditional dlna media servers. So, now we need a dlna server to tell our network about media we want to watch. Incomes minidlna. This is a nice, lightweigh dlna server that the pi can use. Easy to install, sudo apt-get install minidlna

Once installed, all you need to do is tell it where your content is and it will go ahead and scan and do its magic. Edit the file at /etc/minidlna.conf. Comments in that file are pretty straigh forward, all I needed to change was this line media_dir=V,/mnt/nas/downloads

You can add as many of these as you need to secify where your media is. You can also specifically tell it what type of media it is (notice the V ad the start of the listing) this just helps it not have to guess

Now that you have both services running, check out the app on your phone.

You see that it can play to “Renderers” (Things that consume the media) and it plays from a “Library” (Your dlna server, or local device files). As you can see from the screenshot of my network, I could play media to my XBOX, and by bedroom chromecast. I could choose media from my desktop PC, or the “Musicbox” server (which is my rasbperry pi)

Another thing to note is, if you do not need any transcoding done, then you dont need to install the BubbleUPnP server. You can just install the android app, and install minidlna somewhere to share your media to your network

Upgrading SD on the Raspberry Pi

Wed, 26 Oct 2016 10:00:00 +0000

I’ve been running my Raspberry Pi (V2) for almost a year and a half on a 16gb SD card, but because I run owncloud on the pi, I’ve been slowly running out of space. I know I can just add a usb drive or have owncloud store the data on an attached NAS or something, but I chose to just store everything on the SD card. Just so everything is in the one place. Anyway, I finally got around to purchasing a 32GB card, to upgrade the pi. I didn’t want to start from scratch and re-install everything from a backup, so I did a bit of searching in the raspberry pi forums, and found you can easily upgrade the SD card. And it turns out it was very easy. All you need is another machine (I used GNU/Linux here) and a SD card reader (which you probably have if you have a pi)

So here are the few steps I took to upgrade the SD in the Pi, retaining the whole OS and everything I had installed on it.

Safely power off the pi
- Wait about 60-90 seconds while things power off completely. You shouldn’t see the green activity led flicker anymore once its fully shut down
```
 sudo poweroff 
```
Get the SD card and plug into your other machine, making sure you un-mount it if it automounts
Find out where the sd card is
- Look for your SD card in the list and note its location, for me this was /dev/sdd
```
sudo fdisk -l
```
Take an image of the SD card with dd. This will take quite a long time, it took around 30 minutes for me, for 16gb. You can speed this up, at the risk of corruption
- the status=progress is just a nice way to see how things are going without using the pkill trick
```
sudo dd status=progress if=/dev/sdd of=/home/george/rpi-16gb.img
```
Once this is complete, sync and unmount the drive. Plug in the larger SD card now
Again, find out where its located sudo fdisk -l
Using dd, lets write the image to the new disk
- Notice here I use a larger block size when writing to the image, this makes it alot quicker, and is fairly safe to do. If it doesn’t work for you, decrease this to 1M. This will take alot longer
```
sudo dd status=progress bs=4M if=/home/george/rpi-16gb.img of=/dev/sdd
```
Now we have the old image on a new SD card, time to plug it in and boot up the pi
Wait the usualy amount of time and ssh into your Pi. You should notice that nothing has changed. It even still thinks its on a 16gb sd card. Have a look at df -h. We will fix that next
Run the raspi-config tool, and choose the first option - Expand filesystem
- wait some time and it will do its thing to expand the file system magically for you. Once its done, it will prompt you to reboot. Do this
ssh back in and check out df -h you should see now it can see the rest of the new SD card!

These are the steps that worked for me, and it worked first time. You may or may not find that this works straigh away. If it doesn’t, plug the HDMI in and see if there is anything strange happening at boot time, also try plugging your original SD card in and make sure that boots. If it doesn’t, you will find the image that you took of it will have the same issue (so the new one won’t boot). If it does though, try taking another dd copy of it, and checksum it against the one you took in step 4, the checksum should come back the same

building phonegap app for Android and iOS

Thu, 13 Oct 2016 10:00:00 +0000

So I had the ~~pleasure~~ of figuring out how to successfully build and publish a phonegap app, both to the Play store and Apple store. It was bad enough that I have no experience with phonegap/Cordova (or anything iOS development related), but the previous developer had left no documentation. That was fun. Here are some notes of things I ran into along the way. Hope they help someone else out too

So I had downloaded the repo, installed all the dependacies, but now I was getting this error

ERROR in ./src/routes/index.jsx
 
Module not found: Error: Cannot resolve 'file' or 'directory' ./searchByKeyword in /home/george/Documents/work/repositories/ita-app/src/routes

Spent some time looking at Google, when I came across this suggestion to use webpack to display more verbose errors webpack --display-error-details. Genius.

This gave some more verbose output and told me that it was searching for the file, but could not find it in some directories, this helped me realise it was case sensative. The file it was looking for had a capital S, but it was looking for lower case s. Changing the name of the file to lowercase and rebuild

And another error, but on index.jsx file this time. Not sure why it only errored on these 2 files, probably something in the config files but I didn’t rally want to be looking through config files all day (there were alot!) Do the same and rename to lower. And now we get a successful build! Yay!

The previous dev had done this on OSX but not sure why this was an issue on Linux(can understand if it was done on windows, for example)

So now we have an Android apk, but to upload it to the store it needs to be signed (and later on it told me it also has to be zipaligned)

Here is how to do that with command line

#Sign the app with jarsigner (should have been installed along with android studio)
jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore /home/george/Documents/repositories/myapp/key/myapp.keystore android-x86-release-unsigned.apk myapp

Uh oh, another error

jarsigner: Certificate chain not found for: myapp.  myapp must reference a valid KeyStore key entry containing a private key and corresponding public key certificate chain.

Turns out you have to specify the exact alias that is found inside the keystore with the jarsigner command. To get that, run

keytool -keystore /home/george/Documents/repositories/myapp/key/myapp.keystore -l

That will display something similar to this, where you can see the alias name. Now re run jarsigner with the correct alias and you will have a great time

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: myapp-release
......

Now we have a signed apk, but on uplaoding to the store, it errors and tells me its not zip aligned. Luckily this one was easy. /home/george/Android/Sdk/build-tools/24.0.3/zipalign -v 4 android-x86-release-signed.apk android-release-signed-aligned.apk

NOTE only zip align after you have signed

Android down, iOS to go!

Counter Strike and Pebble

Sat, 03 Sep 2016 10:00:00 +0000

Hey all, it’s been a while! I’ve been busy. Combining my enthusiasm to CS and the usefulness of a pebble watch to create you CSontheGO

Being able to keep up to date with CS matches from your watch. I decided to do this because I: 1. wanted to Learn a php framework 2. Enjoy developing things in my spare time

This project had to be split into 2 parts, the backend/data store, and the pebble watch. Surprisingly I had a pretty much a beta version of the app running in about 45hours of dev time. This is probably mostly due to the paths I took with each part of the project

the backend

So I wanted to learn a php framework (because I like php) and laravel and lumen came to me highly recommended. All I have to say is amazing

So I needed a backend/data store for this project because there were no publicly consumable APIs from any sites that report on Counter Strike, so I had to make my own. For something the pebble app can use to give you information at your wrist

Currently I am getting the data I need by scraping a well known e Sports website. The plan is to add other sites, for redundancy, incase I get blocked, or that particular site doesn’t provide updates for unknown reasons

I ended up using lumen, as I didn’t need the full laravel install to make my idea come to life, this data store would only provide an API for the pebble app to connect to anyway, and the scraping is all done via headless cli scripts. Perfect

IDE choice

So because this was the first “big” personal project I was making, I decided to try out an IDE that supports PHP I have created various things in php previously, but they have never been big, so I just stuck to using vim which has suited my needs fine

The choices only really come down to

Php storm by jetbrains
Eclipse PDT
Netbeans
Atom not an IDE per say but including because a legitimate option
VS code same as atom

PHP storm If I had the cash to splash, I would have bought this. I used the 30 day trial, and its pretty amazing. Has all the bells and whistles you’d expect from an IDE and it does really well at on the fly suggestions and hints while developing, even with lumen. I was quite impressed with it. I’ve used the base product before though, while developing on an Android app. Android studio uses the indelij platform underneath it, which is the same as php storm

Eclipse I switched over to eclipse after my trial of above had finished. Eclipse was fine, but it was lacking in the suggestions and hints during development. It didn’t seem as robust and complete as php storm. I found myself having to just look up the docs more often then not when I was using eclipse. To see what parameters where etc.

And that is why I ended up just using atom. It has a decent enough support for PHP to give me syntax errors, and if I’m using the docs anyway, I may as well use something a little more light weight. I’ve used atom previously, mainly as just a text editor of choice (after vim) but it’s come a long way since I last used it so it was quite pleasant. And does exactly what i need it to do. I know I’m missing the big IDE features like integrated testing, integrated environments and things like that, but that’s okay with me

pebble app

There were 2 paths I could have taken with this. Write in c, with the pebble c framework, or use pebblejs, a framework that allows you to write the app in JavaScript. I chose pebblejs. There were a few downfalls with pebblejs, namely, it “runs” on the phone and sends all the data to the watch, causing more BT round trips, using more battery. The good thing about it was it was very quick to get an app up and running. No worrying about appmessage and dealing with all that in c. Although I will eventually rewrite in c to support all watches and the new APIs, which pebblejs doesn’t support (and probably won’t)

Writing this was pretty neat, JavaScript is super powerful and includes all the things I needed (pulling data from the API and showing it in a menu) and I could utilise things like timeline pins via the rest API too.

All in all, I’ve had fun! I’ve learnt alot, both in PHP frameworks and development in general. I have a long list of fixes/feature I want to add but don’t exactly have the usage of the app to really continue :( I will certainly leave the backend up for those (and myself) that are currently using it. I guess there isn’t much of a user base that like CSGO and have a Pebble!

C.H.I.P

Thu, 14 Jul 2016 10:00:00 +0000

So I was too late to the hype of Pi Zero, the $5 Raspberry pi, and have not been able to get my hands on one since the launch. Everywhere has been sold out everytime I check :(

I can’t remember how I stumble upon them, but I found an equaly as awesome project on kickstarter. C.H.I.P. A $9 board that runs linux. Esentially the same specs as the PiZero, CHIP also includes onboard wifi and BlueTooth 4.0! Amazing right?! So I now have one of these on order, hopefully in the mail soon. This just opens up the possibilites of home automation. From reading their documentation, it seems to run a slighty modified version of Debian (I assume just to include some performance enhancements to support the hardware). It Also includes lots of GPIO pins so you can hook up any type of sensor that you would be able to with a raspberrypi, with CHIP.

$9 board with wifi and GPIO pins. Means I can cheaply add any type of sensor I want to pretty much every room in the house. Easily hook this up to my OpenHAB controller (which I will make a post about soon) and create some awesome automation rules to keep things running smootly. For example, I have a few watts clever power remotes, so I could add CHIP into the master bedroom, monitor the temperature and turn the wall heater on/off as required during the night. Super useful during these cold winter months!

So many other things to be done. Need to decide on what I want it to do before it arrives so I can get to it straight away!

Fun with cPanel

Fri, 17 Jun 2016 10:00:00 +0000

Recently have had to set up a few cPanel servers, and here are some things I ran into while doing so. One isn’t necessarily cPanels fault, but it still took me a long time to find a work around for it :/

Common practice for most cloud provides these days is to give you a VM with password based authentication turned off. That’s cool with me as this is one of the first things I do anyway. Setting up cPanel a breeze. Create a new user, no worries. Create SSH keys and import, easy. Attempt to login server: login limit exceeded for account

Odd.. but let’s turn on verbose.

Ah it’s giving other keys before the one it needs. Let’s add IdentitiesOnly yes to config file

Still not working. Let’s try super verbose. Still no luck. 30 odd minutes of faffing about I decide to check the server logs (Note: check these first, not after)

Ah there is the real problem, server is complaining about permission issues being bad on .ssh folder for user

Chmod that up and fixed!

cPanel and aws ec2

Make sure you check NAT settings! Spun up a fresh VM, added an account but it wouldn’t work! Kept redirecting me to that default “no site here” page. Tried this that and the other, but still not working

Turns out cPanel hadn’t detected the NAT settings properly, so didn’t have the right elastic IP (and in fact had the wrong internal IP) in its settings. To fix run /scripts/build_cpnat

Check (settings -> IP) to make sure it’s all better

Getting custom sites into cPanel

Got a site that was developed by someone else, and from the looks of it was just setup on a LAMP server, nothing special, but had real trouble trying to squash this into cPanel. Site was built on the zend framework and use composer as dependency manager. Now this would have been nice to know before I started

Copy site files into public_html. No thanks. Zend uses another folder public where it stores the index.php etc. That’s cool, let’s just change document root in /var/cPanel/userdata/user/site-name have to do this because there are other folders on the same level as this public folder that it uses

Note

There are other ways around this, using symlinks or htaccess. Probably better practice. But php

Composer and cpanel

The next issue I ran into was getting composer running, to build and make sure dependacies are there. Composer in cPanel? Sure why not

cd to your public dir, or whereever your composer.json file is, wget https://getcomposer.org/download/1.1.1/composer.phar (or lookup the composer site to get the latest, and then php composer.phar install

Surprisingly was as simple as this!

All good? Almost. Get slapped with this error:

ZendMailProtocol_Exception' with message '535 Incorrect authentication data ' in /Zend/Mail/Protocol/Abstract.php:378

Sigh

Turns out Zend doesn’t play ball with exim or any other mail agent you have installed, and tried to send email by itself. So you need to enable (or disable) a setting within cpanel to get it working Probably not the most security best practice setting to be playing with, but I needed to get it working

Login to WHM
Under Server Configuration, select Tweak Settings
Click on the Mail tab
Scroll down to “Restrict outgoing SMTP to root, exim, and mailman (FKA SMTP Tweak)” and make sure it’s set to “OFF”
Scroll to the bottom of the page and click Save

TV dashboards with chromecasts and dashing

Wed, 04 May 2016 10:00:00 +0000

I had some time, and we had some TV’s hanging on the wall wasting space. So I decided to create a dashboard that could put some useful information on these TV’s for everyone to see

Previously we were utilising the chromecasts with Greenscreen, and it worked well, but had its issues (probably more related to the chromecasts, not the software) but we weren’t using it as a useful dashboard, more just to scroll through some website links and display the occasional funny gif

This stopped working for us when they updated the way to talk to the chromecasts :( So instead of fixing it, I decided to find something else to do the job, and a cool dashboard to go with it

Setup

So I installed everything under Debian virtual machine, that’s running both the dashboard, and the dashboard controller with 1gb RAM. I eventually plan to have it all running on a raspberry pi. More on that when I get around to it!

So go ahead and spin yourself one of these up, and when you are ready, install the following dependancie s (apt-get)

nodejs
avahi-utils
avahi-daemon
libavahi-compat-libdnssd-dev
Ruby (>1.9)

(npm)

bower
grunt-cli
fd

note: To install additional widgets once dashing is setup - dashing install [gist ID]

Dashboard

So a year or so ago, I stumbled upon Dashing. At the time I used this awesome framework to build a dashboard for home, so I could view/control my automated home from a wall mounted tablet in style. Very out of date, but see home dashboard

This took a bit of tinkering to get it to display things how I wanted, and I had to learn some ruby along the way, but all part of the fun.

To get things setup

Install dashing

sudo gem install dashing
Generate your dashboard and bundle the gems

dashing new my-dashboard && cd my-dashboard && bundle

modify config.ru to:

add an authtoken (useful if you want to talk to the API externally)
disable x-frame-options. This is needed, as the dashboard controller will load dashing in an iframe. If it turned on, it will cause some issues (line 6)

set the default dashboard that gets loaded (line 5)

require 'dashing'
configure do
  set :auth_token, 'myauthtoken'
  set :default_dashboard, 'gs'
  set :protection, :except => :frame_options
  helpers do
    def protected!
     # Put any authentication code you want in here.
     # This method is run before accessing any resource.
    end
  end
end
map Sinatra::Application.assets_prefix do
  run Sinatra::Application.sprockets
end
run Sinatra::Application

Read below about configuring the widgets, and then start!

sudo dashing start -p 80

First I had to change the way the List widget worked, so I could have 2 rows of names for the leave. This is what I had to add to widgets/list/list.html

<h1 class="title" data-bind="title"></h1>
<hr/>
<table>
    <tr><td>
<ul class="list-nostyle">
  <li data-foreach-item="itemsleft">
    <span class="label" data-bind="item.label"></span>
  </li>
</ul>
  </td>
  <td>
      <ul class="list-nostyle">
      <li data-foreach-item="itemsright">
        <span class="label" data-bind="item.label"></span>
      </li>
      </ul>
</td>
  </tr>
<p class="more-info" data-bind="moreinfo"></p>
<p class="updated-at" data-bind="updatedAtMessage"></p>

Then had to change/add some font sizes to make it look how I wanted (as the TV’s were on walls above desk area’s, I had to make sure things were big enough to read easily) widgets/list/list.scss

  .title {
    color: $title-color;
    font-weight: bold;
    font-size: 50px;
  }

  ol, ul {
    margin: 0 5px;
    text-align: center;
    color: $label-color;
    font-size: 40px;
  }

I found this dashing widget that collects and displays reddit images, and changed it to use the subreddits I wanted (/r/aww) and combined it with the big image dashing widget to display that nice 2 row sized image

I didnt change anything in the style or html, however I just modified the job jobs/aww.rb to do it at the intervals I wanted, and included the subreddits I wanted. You can add any other subreddits to the subreddit array at the top of the script

require 'net/http'
require 'json'

placeholder = 'http://i.imgur.com/JycUgrg.jpg'
#subreddits = ['/r/aww.json','/r/animalsbeingjerks.json','/r/aww_gifs.json','/r/babyduckgifs.json']
subreddits = ['/r/aww.json']
SCHEDULER.every '300s' do
  randomreddit = subreddits.sample
  http = Net::HTTP.new('www.reddit.com')
  response = http.request(Net::HTTP::Get.new(randomreddit))
  json = JSON.parse(response.body)

  if json['data']['children'].count <= 0
    send_event('picture', image: placeholder)
  else
    urls = json['data']['children'].map{|child| child['data']['url'] }

    # Ensure we're linking directly to an image, not a gallery etc.
    valid_urls = urls.select{|url| url.downcase.end_with?('png', 'gif', 'jpg', 'jpeg')}
    send_event('picture', image: "#{valid_urls.shuffle.sample(1).first}")
  end
end

And for weather info I used this forecast.io dashing widget for the weather. Previously, in my home dashboard, I had used the verbinski widget, so I attempted to incorporate that initially. But I didn’t want the default of 3 columns of weather (Today, Tomorrow and weekly) and tried to change it to only be todays weather. It worked, but I wasnt able to make it match the rest of the tiles on the dashboard, and ended up looking out of place

Getting calendar info into the dashboard

So because we use google calendars here (we utilise a shared calendar for leave/remote) I was able to leverage the library’s Google has created to get this info. They do have a ruby library for this, but first go I was unable to get it working (needed a new version of Ruby) So I defaulted back to use the PHP library

First you need to follow these instructions to turn calendar api access on, and configure an access token, also follow those instructions for installing the google client library from composer

So I run this as a cron job, every hour, between 6-6. I’ve modified it to just download events on the leave and remote calendars to seperate text files. This is the read into the dashboard by a ruby job, which is part of dashing (I have only included the snippet of code I modified, the rest is the same as in the google example)

// Get the API client and construct the service object.
$client = getClient();
$service = new Google_Service_Calendar($client);
$todaystart = date("Y-m-d") . "T00:00:00+10:00";
$todayend = date("Y-m-d") . "T23:59:59+10:00";
// Print the next 10 events on the user's calendar.
$calendarId = 'Goes here';
$optParams = array(
  'maxResults' => 10,
  'orderBy' => 'startTime',
  'singleEvents' => TRUE,
  'timeMin' => $todaystart,
  'timeMax' => $todayend,
);
$results = $service->events->listEvents($calendarId, $optParams);
$fh = fopen("/home/John/work-dashboard/jobs/leaveleft.txt","w");
$fh2 = fopen("/home/John/work-dashboard/jobs/leaveright.txt","w");
if (count($results->getItems()) == 0) {
  print "No upcoming events found.\n";
} else {
  print "Upcoming events:\n";
  $i = 0;
  foreach ($results->getItems() as $event) {
    $start = $event->start->dateTime;
    if (empty($start)) {
      $start = $event->start->date;
    }
    printf("%s (%s)\n", $event->getSummary(), $start);
    $data1 = $event->getSummary();
    $split = explode("-", $data1);
    $data = $split[0] . "\n";
    if($i<5){
        fwrite($fh,$data);
    }else{
        fwrite($fh2,$data);
    }

    $i++;
  }
}

And then here are the jobs that go in the dashing job folder

#leave.rb
SCHEDULER.every '360s' do 
  leavep = Hash.new({ value: 0 })
  leaver = Hash.new({ value: 0 })
  File.readlines('/home/John/work-dashboard/jobs/leaveleft.txt').each do |line|
    leavep[line] = { label: line }
  end
  File.readlines('/home/John/work-dashboard/jobs/leaveright.txt').each do |line|
    leaver[line] = { label: line }
  end
  send_event('leave', { itemsleft: leavep.values })
  send_event('leave', { itemsright: leaver.values })
end

#remote.rb
SCHEDULER.every '360s' do
  leavep = Hash.new({ value: 0 })
  File.readlines('/home/george/work-dashboard/jobs/remoteleft.txt').each do |line|
    leavep[line] = { label: line }
  end
  send_event('remote', { itemsleft: leavep.values })
  #send_event('remote', { itemsright: leavep.values })
end

To make this less messy, I definitely could have got the Ruby library working, then just had it running as a job under dashing to push results directly to the dashboard, instead of having this extra script run then pull in seperate. Also merging the 2 scripts that are currently seperate (leave.rb and remote.rb) From memory I originally did this but there was issues when it was pushing things to the dashboard.

But now I have a dashboard that displays some useful information to the managers at work, without having to open up calenders and faff about with that to see who is away/remote and also gives everyone something cute to look at

I plan to create another dashboard for the “developer” area so there some more relavant information for them, such as status of TeamCity/Jenkin’s builds, git commits for the day. But that’s on the to-do list. Should be simple enough though as there are a multitude of widgets for dashing you can use here

Dashboard Controller

So as I mentioned, instead of Greenscreen I set out to find something else that could control chromecasts, but better. Thats when I found DashKiosk Pretty awesome as its a dashboard manager that can handle more then just chromecasts. Anything that can browse the web, and has as screen can be turned into a dashboard using this! The plan was to buy some raspberry pi’s and attach them to the TV, so we could have a more reliable “client”, but thankfully, the chromecastV2 worked perfectly

Pretty straight forward setup:

Download and extract the latest dashkiosk tar ball
- wget https://github.com/vincentbernat/dashkiosk/archive/v2.6.6.tar.gz
- tar -zxvf v2.6.6.tar.gz
- cd dashkiosk-2.6.6/
Install npm dependancies and build
- sudo npm install #this is going to take a bit of time. Shouldnt be any errors at this stage (I got one but it was because i didnt install something at the start)
- sudo grunt #This also may take some time
Test to make sure chromecasts work
- avahi-browse _googlecast._tcp This will list all the chromecasts on the network, make sure you can see them Now make sure you can resolve the name to IP
- getent hosts "Dev\032Area".local If this works, you are good to go! If not, see here

Finish up and configure for your environment

cd dist
sudo npm install --production #this will take a while

sudo vim lib/config.js

  #change the chromecast part, where the IP address 
  # is the IP address of the server 
  # (Cant use domain name because chromecast uses google dns,
  #  so unless it is public facing it wont work
  // Chromecast
      chromecast: {
        enabled: true,
        receiver: 'http://192.168.100.171:9400/receiver',
        app: '5E7A2C2C'
      },

Run it!
sudo node server.js --environment production This will start up on port 9400, so go to http://IPADDY:9400

The only issue I had was getting the chromecast module working, but that ended up being an easy fix. Had to change 1 file to turn off(?) ipv6

Some issues along the way

So the main issue with this setup is the chromecast v1 hardware. In our setup we have 2 AP’s using both bands, but the chromecasts frequently dropped out and had to reconnect. This was a pain with greenscreen as you had to manually reconnect the chromecast back up. I never got to the bottom whether it was the chromecasts, or our wifi that was the main issue. But upgrading to the Chromecastv2 was a whole lot better, much more stable and noticably faster

Let me know if you are interested, and I’ll load the whole dashing dashboard I created into github

PHP file upload to slack

Wed, 13 Apr 2016 10:00:00 +0000

So I don’t know why this one took me this long to find, but I spent a long time today trying to figure out why my PHP script wasn’t uploading my file I wanted to slack. I had the string to send to slack correct, but every time I tried I was getting a no_file_content error.

I was using the, apparently now, old method of referencing a file for POST-back. That is $file = '@/path/to/file.png'

Here is the new method, in case any one else had as much trouble as I did finding it $file = new CurlFile('filename.png', 'image/png');

Here is the PHP wiki article on the “new” method

And here it is, in action

<?php
$slacktoken = "longthingwithlotsofletters";
$header = array();
$header[] = 'Content-Type: multipart/form-data';
$file = new CurlFile('/home/john/myphoto.jpg', 'image/jpg');

$postitems =  array(
    'token' => $slacktoken,
    'file' =>  $file,
    'text' => "This is my photo",
    'title' => "Why this?",
    'filename' => "my-photo.jpg"
);
    
$curl = curl_init();
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_URL, "https://slack.com/api/files.upload");
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS,$postitems);

//Execute curl and store in variable
$data = curl_exec($curl);
?>

Yay now I can upload files from PHP

Blacklist Checker

Wed, 06 Apr 2016 10:00:00 +0000

This is a nice little php script that you can run in the background via cron to constantly check your mail servers (or any server you have that deals with mail) and alert you via slack

I know there are things out there like mxtoolbox that can do this for you, and include a larger list of blacklists, but I wanted something I could run on our own infrastructure that did what I wanted, and alerted me via slack (much cooler then email alerts). Eventually I will also add in bulk email alerts (like a weekly update of whats happened) and maybe, just maybe SMS alerts via twilio or SMScentral

I have been using mxtoolbox, but was finding the email alerts not helpful, as I don’t often check my emails during the day, and they started going to my spam folder. I wrote this so I could be in control of what I checked and when, and then post it to Slack when there was an issue. As at work I always have a slack windows open because use it heavily.

This alerts you when things clear up for a particular server too

You can check it out here

Here is what the notifications look like:

The git repo includes setup instructions in the README, but I will include here too

Usage

To use this you will need

A slack API token
A slack webhook to the channel you want to post to
PHP (of course)

Download this repo
open blacklist-cron.php and edit settings at the top of the script to include slack specific stuff
edit servers.txt to add in your mail servers (can be IP or domain name)
edit dnsbls.txt to include all the DNSBL servers you want to check against (left this full of the ones I use)
- be sure to check each one. Some, like baracuda, require you to sign up if you are going to be making lots of calls
Make a cron entry, like 00 08-18 * * * /usr/bin/php /home/john/blacklist/blacklist-cron.php >> /home/john/blacklist/cron.log
- This will run the script every hour between 8am and 6pm (office ours). I also did this to reduce the amount of requests i send to each dnsbl. You can change this to suit your needs

You will now be alerted when a server of yours gets put on any of those blacklists, like this. It will also alert you when they are removed again

You can also run this via the commandline if you want to see whats going on php blacklist-cron.php and if you have debug turned on, things will just print to screen

john@johns-server:~/backlist$ php blacklist-cron.php
Checking 1.2.3.4
is the server in bl array? 
server not found on any blacklists. Yay!

Checking mymailserver.com.au
is the server in bl array? 
mymailserver.com.au -> 5.6.7.8
checking against ubl.unsubscore.com
Added to blarray
Creating report
posting to slack

Checking mailserver02.net
is the server in bl array? 
mailserver02.net -> 123.45.67.89
server not found on any blacklists. Yay!

Owncloud

Tue, 05 Apr 2016 10:00:00 +0000

So I set this up a while ago, but thought I’d share it here anyway. I was looking to find a self hosted dropbox/google drive alternative. As I am starting down the path of getting things self hosted, so my data is still my data and all that. There were 2 choices with this, that would run on the raspberrypi: Owncloud, or seafile

Seafile looked nice, and the core of it is written in C, so to me that would work well on the pi as it would be super lightweight and fast. I went for this originally, but had major issues trying to set it up for remote access

Thats where Owncloud came to save that day! its written in php (my fav) so was super simple to setup, and get working for remote access. It was also easy to get it working on a external hard drive if need be.

Another cool thing about it is that there a many “Apps” to go with it, so you can turn a simple file sync and share service into something that can handle your calendar, notes, passwords, mail client, rss feed agregator. Replacing pretty much all your 3rd part services, all self hosted

Follow these to get things setup. There are alot of things you can do around security which these instructions go into. Basic setup includes:

download and unpack
change ownership of files to prefered user (www-data)
create database for owncloud use
Go to owncloud URL in a browser and follow the install wizard

Once setup, download the client where ever you like (mobile app, windows, mac, linux) and connect up to your instance.

Some things to consider running this on a raspberry pi, it isn’t exactly powerful enough to be able to browse images or anything too heavy via the webclient (or the mobile client for that matter, as it downloads things as needed) But it is perfectly fine if you have a few clients hooked up and let things sync in the background. I have 3gb of photo’s up on mine and have auto-upload enabled on the mobile client and everything syncs nicely between my 2 PC’s and laptop. I also use QOwnNotes to sync my markdown notes around my devices.

The plan will be to merge everything into Owncloud, my password management, RSS feed agregator and calendar (move away from feedly, google calendar and 1password) so all my things are actually owned by me! More on those when they happen

Multi room audio with raspberry pi

Sun, 17 Jan 2016 01:14:00 -0800

So I’ve had a though for a while that I’d like something like Sonos in my home, but something like that is wayyyy out of my price range. Then I got thinking, surely opensource can do this!

I’ve just spent most of the day researching and setting up a solution. The first thing that came up was pulseaudio. This cool piece of software can do all sorts, and one of them being able to setup up a client-server like environment to stream audio from a pulseaudio “server” over multicast to any mpd “client” who is listening. The theory behind this is you can set up multiple broadcast’s and have each device listening on different ones, plus all listening to on the same one. So you could stream to all devices, or to individual ones. The problem with this way (from what I could find) was that there was no easy, user friendly way to switch “zones” (broadcast addresses) on the server to play to different devices. This wouldn’t fly with the wife! Although this would be a nice, quick solution for most tech people

I needed something that could be “user” friendly (super simple), that was easy to use, that would have an android and/or iOS app to control the server and that didn’t require much else then pushing a few buttons in an app to make it work. I eventually found mopidy (https://www.mopidy.com/) I choose this as it ties into a few music streaming services, as I don’t have a large collection of local music, I listen mainly to spotify. It also has some nice apps available on Android, iOS and a web client, so controlling the server would be nice and simple. Hooking into this to allow for multi-room streaming I used snapcast (https://github.com/badaix/snapcast) a nice c# program to allow for client-server streaming with as many clients as you like. Unfortunately it doesn’t support “Zones” yet (which I originally wanted), but its on the todo list As I have 3 rpi’s in my house, I will be using them as my audio clients, and server. I will be hooking one pi into my lounge surround sound, and have an old speaker system that I use in the Kitchen

Hardware: 1x rpi 2 (as the server) 2 x rpi B (as the clients) Speakers to plug into the “clients”

For now I have just plugged everything into the 3.5” jack on the rpi, but I have read alot that the quality of this sound is terrible (I didn’t notice too much in my tests) and that getting a USB audio device would be a much better idea. So maybe when I start noticing quality, that will be the first thing to check

Software: mopidy snapcast

Installing mopidy was a breeze, just followed their instructions on their wiki (https://docs.mopidy.com/en/latest/installation/raspberrypi/)

Install the spotify plugin

sudo apt-get install mopidy-spotify

Once I got to snapcast, it needed a few dependencies, which rasbian (wheezy) didn’t care for. It was running older versions of the software needed and it didnt look like I could update it easily, so I bit the bullet and upgraded to Jessie

This took ages! most of my time was spent waiting around for it to download and build, it probably would have been quicker to download the Jessie iso, re-flash the SD card and re-install everything. But once it was done, installing it was easy, and setup was even easier

Just start mopidy on the server

mopidy

and start the server portion of snapcast

sudo snapserver -d

and start the client portion of snapcast on your streaming client

sudo snapclient -d
#(or `sudo service snapclient start` . I had issues with running via \
     #init.d and found it easier to start the service with the snapclient command)#

Then control your mopidy server with your favourite method (mine was android app) and magic!

I haven’t included the commands yet, so maybe i’ll add them if i can run history on my pi Hope this helps someone, as finding an opensource/cheap solution for this was harder then I thought!

Here are some other alternatives you can use too, especially if most of your collection of music is locally stored:

Squeezebox (http://monksoundworks.blogspot.com.au/2014/03/raspberry-pi-multi-room-audio-solved.html)
Volumio - http://volumio.org/
Pi music Box - (pretty much a nice GUI ontop of mopidy) http://www.pimusicbox.com/

As I was writing this, I found this - https://github.com/thaytan/aurena. Seems very cool as it already supports “Zones” I will see if it ties into modipy or streaming services!

screen recording with annotation on Linux

Sun, 13 Dec 2015 19:14:00 -0800

So there’s been a few occasions where I have needed to do a screen record on my Ubuntu laptop at work, and I was easily able to find RecordMyDesktop which is neat and simple desktop app that can get you on your way. The harder part was finding a program that could allow you to annotate while screen sharing. There are plenty of tools out there that allow you do annotate ontop of an already recorded video, but I wanted to be able to do it as I was recording. I found a few tools like compiz annotate function (but this only worked with ubuntu desktop, not GNOME)

This is when I came across gromit-mpx. Follow the instructions on the github page and you’re good to go. You can start up gromit-mpx and then start a screen record with RecordMyDesktop and then you can annotate on your screen as you record, amazing right?

Of course, everything is customisable so you can change the colour of the default pen, change the thickness of the pen and eraser and other cool things like that. Also, as you can see in the example I made, it draws ontop of your screens, so when you move around (e.g go to a new webpage or bring another screen over what you are annotating on) things will stay there until you clear the screen (SHIFT+F9)

ffmpeg picture-in-picture

Wed, 25 Nov 2015 14:28:00 -0800

So just a quick one. I needed to create a video that has Picture in Picture. So we could have a video of the presentation, inside a recording of the slides used. Usually I would use Adobe premier pro for this, but as I have Ubuntu installed on my laptop, I though of ffmpeg. It is much less resource intensive then the Adobe product, but of course its all command line, so you don’t get a pretty GUI to play with. Once you master ffmpeg, its an amazing and fast tool that can do pretty much manipulate any audio/visual that you need in any way you want

ffmpeg -i pip1.mp4 -i pip2.mp4 \
-filter_complex "[1]scale=iw/4:ih/4 [pip]; [0][pip] \
overlay=main_w-overlay_w-250:main_h-overlay_h-100" \
-b:v 50M -ar 44100 <br />-ab 192k -vcodec mpeg4 \
-acodec libmp3lame PIPoutput.mp4

Here is a screen shot of what it accomplishes. It encodes it using mpeg4 for video, and mp3 for audio and keeps the quality of the original video’s. (this is just an example using 2 random videos I found)

If you adjust the bits “-overlay_w-250” and “-overlay_h-100” this is adjust the location of the pip relative to the bottom corner of the screen (I’m pretty sure)

Livestreamer!

Wed, 25 Nov 2015 14:20:00 -0800

So there is an awesome command line tool out there called livestreamer. It’s awesome because you can use it to stream your favourite live things, via command line! No more watching it through a browser and have flash (or even just chrome) clogging up resources. Pretty cool huh?

Another cool thing it can do it save directly to file, so if you knew you couldn’t watch it while its live (say your at work) just stream straight to file and watch later

I make this post because I use this quite a lot, and usually I specify the file name as stream .mp4. As I am no expert in video/audio file formats and codecs etc I had no idea that streaming files have their own format. This was helpful as when I was saving into the mp4 format, other tools I was using (castnow) where having difficulties reading the file properly. It’s expecting an mp4 but obviously it’s not an mp4 file. So the correct format to save a hls stream to (as I mostly watch twitch, and this is the format of their streams) is stream .ts

Hope this helps someone else out too, as it took me ages to find this info. Buried deep in a git issue

Controlling limitless LED lights from raspberry pi cli

Tue, 02 Dec 2014 17:33:00 -0800

So I was inspired yesterday to create a bash script in which I can control my limitlessLED lights from my raspberry pi (cli). I always have a ssh connection open to my pi while at work, and when I’m at home I always have a terminal windows open, and tinkering around. So I though, why not?

It’s still a work in progress at the moment, but basic functionality is there for now :)

https://github.com/mrwhale/limitlessled-bash

Inspiration came from this guys: http://smileytechadventures.blogspot.com.au/2013/07/control-limitless-led-from-linux-bash.html

But it only did the white ones, and I have both RGBW and White - So eventually you will be able to do everything from my script (change colours, brightness etc)

There is also a command-line program on the limitlessled website under the api section http://www.limitlessled.com/dev/, but this only does the RGBW ones. It does include the source code, but its in C. So I didn’t particularly want to have to remember C from first year uni..So I decided with bash - its easy, and I know my way around it

Enjoy!

Edit Just found someone else’s work on the subject on github, has done a similar thing to smileytech, few more options, but still very simple. And hasn’t been updated for year now https://github.com/danielfaulknor/limitless-php/blob/master/sendcmd.sh

command line utility to cast to chromecast

Wed, 19 Nov 2014 21:08:00 -0800

So just a quick one today, found an awesome command utility to be able to cast content to a chromecast! It does local content, content from the interwebs and can even stream torrents :O

https://github.com/xat/castnow

Kudos to old mate xat for this one

~~Unfortunately I haven’t been able to get it working from my raspberry pi yet :( Keep getting build errors when trying to install it via npm…~~ I will update once I finally figure it out

Edit 1

~~So I didn’t get it working on my raspberry pi~~ , but I recently upgraded (not sure its the right word :P) to Ubuntu 14.04. I have always wanted to be running Linux as my main desktop environment, but as I am a heavy gamer, I couldn’t do it because all my favourite games (CS:GO) were only on Windows.

But when I got the good news the other month that they had ported it over to Linux, I didn’t waste anytime. First I installed CentOS (big fan of the long term support, a guy I used to work with also worked on the earlier releases of this..) But I was having heaps of issues with drivers, and just generally being not nice, I sighed to myself and installed Ubuntu.

And it just worked. No issues have come up, even managed to get WOL working (couldn’t get it working on Windows), and the only thing left to do now is get SLI working for my dual GTX 690’s

Little bit off topic.. but I managed to easily get castnow installed on Ubuntu, and got it streaming youtube video and even torrents direct to my chromecast :D Pretty awesome stuff too. Next step is to plug my chromecast directly into power, so I can turn the TV on via the chromecast (TV is CEC compatible)

Edit 2

So good new everyone! After trolling the internet for a while, I finally found the missing piece to get castnow working from your raspberry pi. A post by Dudecalion on reddit shows that all I needed was a node version that was compatible with ARM (the one’s I tried from apt-get were old versions). So here are some instructions to get it working from your pi!

 wget http://nodejs.org/dist/v0.10.28/node-v0.10.28-linux-arm-pi.tar.gz

 tar -zxvf node-v0.10.28-linux-arm-pi.tar.gz
 cd node-v0.10.28-linux-arm-pi/bin

 sudo su

 PATH="$PATH:/location/to/download/node-v0.10.28-linux-arm-pi/bin/"

 npm install -g castnow

And now wait. install took around 5 minutes for me, with a few warnings. But once it was done, you can Profit! As said in the post, because the pi isn’t that powerful it probably won’t be able to handle transcoding on the fly, so best to just keep it to playing youtube video’s or local mp4’s (but will happily be proved wrong!)

delving into android apps

Tue, 14 Oct 2014 17:57:00 -0700

This one’s a bit off topic..

But from my last post, about LimitlessLED’s. The android app that they tell you to use from the play store (https://play.google.com/store/apps/details?id=com.lierda.wificontroller&hl=en) is functional, but terribly slow at starting up, and giving you access to your light. More often then not, my wife an I would find ourselves just using the physical light switch instead.

So Finally I found a pretty awesome app for controlling LED lights, (https://play.google.com/store/apps/details?id=tv.piratemedia.lightcontroler&hl=en) and Eliot, being a good bloke, as made his code open source. This app is awesome as it starts up instantly AND is has a widget you can put on your lockscreen or on a page! Although, his app, out of the box, only controls the RGBW lights, but because I have both in my home (RGBW and dual white) I decided to have a go at developing an app that suits my need (well not so much developing, more just copying and editing other code :P )

I took Eliot’s app code and, so far, I have modified it to control the white lights instead! I plan to make the app do both, but It wont be for another few weeks now, as I am in the final weeks of study at Uni.

Let me know if anyone’s interested in the app, and I’ll post the .apk and source code up on github, and if I’m really keen, put it up on the play store.

But big thanks to Eliot for making his code open source!

Ninja Block!

Tue, 16 Sep 2014 19:34:00 -0700

My next major purchase was a Ninja Block for the Raspberry pi. I bought this ages ago, but never got around to writing about it! http://shop.ninjablocks.com/pages/picrust

I thought this was cool as it was a low cost way to start my home automation projects, and it is definitely marketed at the hobbiest, and I would not recommend this one for commercial use. Check out there NinjaSphere, which is aimed at plug n play users. Ninja Block requires a bit of tinkering, having to manually install modules and knowing some NodeJS would be extremely helpful.

So with this purchase I got a wireless door sensor, and temperature sensor and a wireless button. These were pretty simple to setup, and now when someone rings the doorbell it SMS’s me instead of making a chime :)

Not long ago, I kitted my house out with the LimitLessLED lights that can be controlled by the ninjablock, which I thought was pretty cool. It was a bit of an investment to replace most of the lights with these wifi lights, but well worth it :)

Installing this with the ninja block took a little bit of time, had to check the forums and fiddle around a bit, but most of it was just point and click setup. But once setup it was pretty cool. I have set up some rules to automatically turn the lights on in the morning, and off after we have both left for work. I have set up a rule to turn the lights on when the back door is open (so when my wife gets home from work everything is on) This was nice and handy for winter, as it was already starting to get dark when we get home. But now that its staying lighter for longer, this isn’t really useful anymore.

Next plan is to buy a IP camera, so the ninjablock can monitor my front door, so when the doorbell is rung, it sends me a picture of who is at the door. So stay tuned for that tutorial when I finally get around it it!

Temperature Sensing with the Pi

Wed, 27 Aug 2014 22:00:00 -0700

I ordered a couple of gadget for my pi a while ago, and they finally came in the mail ( only had to wait 2 months!) Anyways, I ordered a water-proof digital temperature sensor (DS18B20) and a ultrasonic range finder (HC-SR04) to play around with. So firstly I wanted to get the temperature sensor going. I followed the adafruit tutorial, just to hook the temp sensor to the pi (https://learn.adafruit.com/adafruits-raspberry-pi-lesson-11-ds18b20-temperature-sensing/hardware) Once it was physically connected, lets check if it works

cd /sys/devices/w1_bus_master1/

 ls
28-0000056e9afb w1_master_attempts w1_master_search
driver w1_master_max_slave_count w1_master_slave_count
power w1_master_name w1_master_slaves
subsystem w1_master_pointer w1_master_timeout
uevent w1_master_pullup
w1_master_add w1_master_remove

There should be a listing under there that’s a whole bunch of numbers and letters (the devices ID). In my case it’s 28-0000056e9afb. cd into this folder and cat the file called w1_slave You should get output like so:

db 00 4b 46 7f ff 05 10 1a : crc=1a YES
db 00 4b 46 7f ff 05 10 1a t=13687

The first like is the checksum, to make sure the device is working correctly, and the second line is the temperature, in degree’s celcius, and to 3 decimal places (13687 = 13.687C) Now we know the temp sensor is working alright, we can make a nice python script to read it and log it. For this project, I used:

plot.ly. All you need to do it create a free account and your good to go!
weather underground You can create a free ‘dev’ account which gives you 500 API calls a day (This is what I use, the way I have set it up, it will only ever call 288 a day)
Bureau of Meteorology. I used this just so I could compare the temperature with wunderground Both temperature gathering sites give out the data is JSON format, so it was easy enough to extract the data needed, and then pass it onto plot.ly.

And here is the python script I used to do this! all you need to do is go through and add your API key for the various sites, and add your location (you may have to play around to find the correct location code for this one) I’ve also set it up to pull my plot.ly key and username out of a file called config.json. I did this because I have multiple scripts using it :)

#!/usr/bin/env python
import os
import glob
import time
import plotly.plotly as py
from plotly.graph_objs import *
import datetime
import json
import urllib2
from time import sleep, localtime, strftime
import sys
with open('./config.json') as config_file:
    plotly_user_config = json.load(config_file)
py.sign_in(plotly_user_config["plotly_username"], plotly_user_config["plotly_api_key"])
os.system('modprobe w1-gpio')
os.system('modprobe w1-therm')
base_dir = '/sys/bus/w1/devices/'
device_folder = glob.glob(base_dir + '28*')[0]
device_file = device_folder + '/w1_slave'
def read_temp_raw():
    f = open(device_file, 'r')
    lines = f.readlines()
    f.close()
    return lines
def read_temp():
    lines = read_temp_raw()
    while lines[0].strip()[-3:] != 'YES':
        time.sleep(0.2)
        lines = read_temp_raw()
    equals_pos = lines[1].find('t=')
    if equals_pos != -1:
        temp_string = lines[1][equals_pos+2:]
        temp_c2 = float(temp_string) / 1000.0
        temp_c = round(temp_c2, 1)
        return temp_c
def read_pi_temp():
    file = open('/sys/class/thermal/thermal_zone0/temp' , 'r')
    lines = file.read()
    temp_c2 = float(lines) / 1000.0
    temp_cpi = round(temp_c2, 1)
    return temp_cpi

def read_temp_bom():
    f_bom = urllib2.urlopen('http://www.bom.gov.au/fwo/IDV60901/IDV60901.94866.json')
    json_string_bom = f_bom.read()
    parsed_json_bom = json.loads(json_string_bom)
    temp_c_bom = parsed_json_bom['observations']['data'][0]['air_temp']
    return temp_c_bom
# Weather Underground key
wu_key = "<APIKEY>"
wu_state = "Victoria"
wu_city = "Melbourne"
wu_retry = 2
def get_WUTemp(key, state, city, retries=2):
        #Get data from weather service
        #If exception occurs, retry a few times then quit
        for retry in range(retries):
                try:
                        f = urllib2.urlopen('http://api.wunderground.com/api/' \
                        + key + '/geolookup/conditions/q/' + state + '/' + \
                         city + '.json')
                        json_string = f.read()
                        #print json_string
                        parsed_json = json.loads(json_string)
                        location = parsed_json['location']['city']
                        temp_c = parsed_json['current_observation']['temp_c']
                except:
                        e = sys.exc_info()[0]
                        print (e)
                        temp_fc= None
                        sleep(0.1)
        if (f != None):
                f.close()
        return temp_c
pitemp = read_pi_temp()
series1 = {
  'name' : 'Pi CPU Temp',
  'x' : datetime.datetime.now(),
  'y' : read_pi_temp(),
  'type' : 'scatter',
  'mode' : 'lines'}
series2 = {
  'name' : 'Room Temp',
  'x' : datetime.datetime.now(),
  'y' : read_temp(),
  'type' : 'scatter',
  'mode' : 'lines'}
series3 = {
  'name' : 'Outdoor Temp (wunderground.com)',
  'x' : datetime.datetime.now(),
  'y' : get_WUTemp(wu_key, wu_state, wu_city),
  'type' : 'scatter',
  'mode' : 'lines'}
series4 = {
  'name' : 'Outdoor Temp (bom.gov.au)',
  'x' : datetime.datetime.now(),
  'y' : read_temp_bom(),
  'type' : 'scatter',
  'mode' : 'lines'}
data = [series1, series3, series2, series4]
url = py.plot(data,filename = 'Rpi Room Temp Multi', fileopt='extend')

Have Fun!

Flexget

Wed, 14 May 2014 21:17:00 -0700

A month or so ago, I was trying to figure out how to use eztv’s rss feed to automatically download new torrents to my pi. That’s when I ran across the awesome python program called flexget. (link here)With this you can specify the shows you like, and it will check for new episodes and add them to your torrent client for download. It does a lot more then this,but this is the main way I use it at the moment. So here’s some instructions on how I got it up and running on my raspberry pi First you will need python 2.7 or later and pip (a package manager for python apps). I also use transmission as the torrent client, but there are others you can use

sudo apt-get install python
sudo apt-get install python-pip

Once python is installed and working, install flexget with pip

 sudo pip install flexget

Next, you need to set up the config file for flexget. Create a folder called .flexgetin your home directory, then create a file called config.yml(yes yml, not xml) in that folder. This is the file that flexget will read by default when you run the command.

cd ~
mkdir .flexget
touch .flexget/config.yml

Open up the config file in your fav editor, and you can use the contents of mine as an example, or check out the flexget wiki for other ‘recipes’ and examples:)

vim .flexget/config.yml

And here is the contents of my config.yml. Just need to set the quality you prefer, the path where you want them downloaded (mine is to my external hard drive) and your tranmission (or deluge) configuration (flexget can only hook into these two torrent programs at the moment)

 templates:
   tv:
     series:
       settings:
         tv:
           set:
             path: /mnt/usb/Torrents/Series/{{series_name}}
       tv:
         - The Big Bang Theory
         - Person of Interest
         - The Blacklist
         - Suits
         - Revolution 2012
         - Chicago Fire
         - Chicago PD
         - Grey's Anatomy
         - Wonderland
         - Wonderland.AU
         - Royal Pains

     transmission:
       host: 10.1.1.15
       port: 9091
       username: pi
       password:

     clean_transmission:
       host: 10.1.1.15
       port: 9091
       username: pi
       password:
       finished_for: 2 hours

   ShowRSS:
     priority: 2
     rss: http://showrss.info/feeds/all.rss
     template: tv

 #  kat-tv2:
 #    priority: 1
 #    rss: https://kickass.so/tv/?rss=1
 #    template: tv
 #    verify_ssl_certificates: no

 #  thepiratebay.org-tv:
 #    priority: 2
 #    rss: http://rss.thepiratebay.org/208
 #    template: tv
 #    verify_ssl_certificates: no

As I use transmission as my torrent client, I can hook straight into it with flexget. The only downfall is the rpc password for transmission is in plain text in the config file.

Anyways, now let’s run flexget for the first time (just as normal user). This will output if its succesfully added movies/tv shows to your torrent program, or it will check and output any errors in syntax in your config file (spacing has to be correct)

flexget execute

014-05-15 12:03 VERBOSE details
bt-chat.com-tv Produced 40 entries.2014-05-15 12:03 VERBOSE task
bt-chat.com-tv ACCEPTED: `Revolution.2012.S02E21.HDTV.x264-LOL.[eztv].torrent` \
by series plugin because matches quality

2014-05-15 12:03 VERBOSE task
bt-chat.com-tv ACCEPTED: `Person.of.Interest.S03E23.720p.HDTV.X264-DIMENSION.[eztv].torrent`\
 by series plugin because matches quality
2014-05-15 12:03 INFO   download   bt-chat.com-tv Downloading: \
Revolution.2012.S02E21.HDTV.x264-LOL.[eztv].torrent
2014-05-15 12:04 INFO   download   bt-chat.com-tv Downloading: \
Person.of.Interest.S03E23.720p.HDTV.X264-DIMENSION.[eztv].torrent
2014-05-15 12:04 VERBOSE details
bt-chat.com-tv Summary - Accepted: 2 (Rejected: 0 Undecided: 38 Failed: 0)

Now that we know it’s working, add an entry into cron to run the command every hour

 crontab -e

and add this line at the bottom

0 * * * * flexget execute

Now you will never have to check for a new torrent manually again, just make the changes to config.yml to add/remove shows movies you like! Hope this is useful

Extra stuff

So I got this working a while ago, but forgot to add it to this post! I added the functionality so, after a torrent is completed, it will send you a pushbullet notification to your mobile (or where ever you want!) 1. So first you will need to sign up to pushbullet to get an access token (API key). Once logged in, click o the little pciture of you up the top right and go to “Account settings”. This page will list your access token for use later on 2. Next, intall it and login to it on your mobile. This will create a device in your account, and assign it a device id, for use in the script 3. Now lets find out the device id, run this command and save the device ID for your device

curl -u <your_access_token_here>: -X GET https://api.pushbullet.com/v2/devices

Now lets create a new script,

touch torrentdone.sh
chmod +x torrentdone.sh

This is what your script should look like:

#!/bin/bash
apikey=<apikey>
deviceid=<ID of device>

/usr/bin/curl https://api.pushbullet.com/v2/pushes \ -u $apikey: \ -d $deviceid \
 -d type=note \ -d title="$TR_TORRENT_NAME" \
  -d body="$TR_TIME_LOCALTIME" \ -X POST >> done

Now we need to setup transmission to run this script on finishing a torrent

cd /etc/transmission-daemon/
sudo nano settings.json

and modify this line to be:

"script-torrent-done-filename": "/home/pi/torrentdone.sh",

All done! Now, not only will all your torrents automatically download, you will be notified of them when they complete!

3. File server

Tue, 13 May 2014 19:10:00 -0700

Now we have a webserver, torrent server (which is downloading all your torrents to an attached hard drive), I have set up a file server to serve the torrent folder to my windows computers in the house. This means I can plug my laptop in the TV, browse to the rpi, and play a show :). Here’s how I did so.

First you need to make sure you have samba installed

sudo apt-get install samba

I have made the share a public share with guests allowed ( so its not very secure ), but I like to think my wifi is secure enough to stop people from gaining access. That said, it doesn’t really matter if anyone can view the files, as I have set them to read only, and only people on the LAN can view them.

Once Samba is installed, the next step is to add the shares you want to smb.conf, located in /etc/samba/. Below is a snippet of my smb.conf file, showing the public share of the external hard drive attached to the pi, mounted at /mnt/usb/

[usb]
comment = USB
path = /mnt/usb/Torrents
only guest = Yes
read only = Yes
browseable = Yes
public = Yes

I place this at the bottom of the file, leaving everything else as default. Just to make sure that everything is okay, I recommend running the tool testparm. This will check your samba config file to make sure there are no errors.

testparm

if you get no errors, go ahead and start your server

sudo service samba start

Now that samba is running, you can map that drive from any other device on your LAN. e.g from my windows pc, just map new network drive and put in the path \\<ip of rpi>\usb and I can view all my downloads!

2. Torrent server

Tue, 12 Nov 2013 21:23:00 -0800

So now all that your web server is up and running, and you have a Dynamic DNS name (if you chose to do that), here’s how I set up the torrent server on the pi (using transmission) credit goes to this website for helpful instructions. I also had a spare 500gb external harddrive (externally powered too) that I am using as storage for the downloads (I only have a 4gb SD for the pi) so instructions for setting this up, credit goes to this webiste

First up, lets set up the hard drive on the pi. I tried the normal way of mounting the hard drive but I ran into the issue on the above link about the hard drive being mounted in read only, so lets just install some other programs to help us out

sudo apt-get install ntfs-3g

Once this is done, plug your hard drive in and power it on. to find out where your hard drive is located run

sudo fdisk -l

This will give you output similar to this:

Disk /dev/mmcblk0: 3974 MB, 3974103040 bytes
4 heads, 16 sectors/track, 121280 cylinders, total 7761920 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c7b31

    Device Boot   Start     End   Blocks  Id System
/dev/mmcblk0p1      8192   122879    57344  c W95 FAT32 (LBA)/dev/mmcblk0p2     122880   7761919   3819520  83 Linux

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x8d399bc0

 Device Boot   Start     End   Blocks  Id System
/dev/sda1       63  976751999  488375968+  7 HPFS/NTFS/exFAT

We can see that my 500 gb hard drive is at the location /dev/sda from the output above. So before we mount it we need to make a folder to mount the hard drive in

sudo mkdir /mnt/usb

Now we can mount the hard drive

sudo mount /dev/sda1 /mnt/usb -t ntfs

You can check to see if its mounted with

df -h

and you should see it in the list as /dev/sda1 mounted on /mnt/usb:

Filesystem   Size Used Avail Use% Mounted on
rootfs     3.6G 2.1G 1.4G 62% /
/dev/root    3.6G 2.1G 1.4G 62% /
devtmpfs    235M   0 235M  0% /dev
tmpfs      49M 636K  48M  2% /run
tmpfs      5.0M   0 5.0M  0% /run/lock
tmpfs      98M   0  98M  0% /run/shm
/dev/mmcblk0p1  56M  19M  38M 33% /boot
/dev/sda1    466G 215G 252G 47% /mnt/usb

I also made another folder on the hard drive

cd /mnt/usb
sudo mkdir torrents
sudo chmod 777 torrents

I give this folder full permissions ( its probably not best practice but I know it works when its like this..)

So now we can get transmission server up and running, finally. Thanks to thisguy for some helpful instructions to get our torrent server up and running

sudo apt-get install transmission-daemon

We already set up a save folder previous at /mnt/usb/Torrents so I just used this for both complete and incomplete torrents (but you can make yourself another folder to store incomplete files in if you want). Now we customise the settings file

sudo nano /etc/transmission-daemon/settings.json

Now you can scan through and change the settings as desired (its pretty self explanatory) but here is a list of the setting I changed

"download-dir": "/mnt/usb/Torrents",
"incomplete-dir": "/mnt/usb/Torrents/",
"incomplete-dir-enabled": true,
"rpc-password": "put desired password here"
"rpc-username": "pi",
"rpc-whitelist": "127.0.0.1,*.*.*.*,192.168.*.*",

The rpc-whitelist is there to give access to certain IP address’s to the web front-end of transmission. ... gives access to the world, as I connect to the server with my phone while out and about to upload torrents and such. But you can restrict it to your local LAN/work IP’s or whatever you need. All the rest of the settings you can modify as needed to suit. Now we want to reload the server to save our settings

sudo service transmission-daemon reload

Once this is done you can restart it as necessary, as explained on the above webpage, reloading it will save the config, but restarting it will override it

You should now be able to access it through your x.x.x.x:9091 ( Mine is accessible at 192.168.0.5:9091 )

So there’s also a nifty transmission android app that you can then remotely upload torrents to your server to download https://play.google.com/store/apps/details?id=com.neogb.rtac

The last thing you need to do is forward the right ports out of your home router so you can access your pi over the internet. I have a netgear so there’s plenty of online guides on how to do it, it’s pretty straightforward though. you will need to forward on port 80, and 9091

1. Webserver

Thu, 31 Oct 2013 16:07:00 -0700

So the first thing I did was use this very helpful guide to help setup my pi as a LAMP server. I didn’t follow the whole thing, just the bits I needed.So following is the basics..firstly install apache

sudo apt-get install apache2

and then mysql

sudo apt-get install mysql-server

And finally PHP

sudo apt-get install php5
sudo apt-get install php5-mysql

Once this is done go to your pi’s IP address to make sure apache works

Because I didn’t get a static IP from my ISP, it changes all the time (which is annoying) so I decided to get a dynamic DNS name for myself. I used www.noip.com and just got a free one.

They also provide something called a DUC(dynamic update client) so your pi can talk to their servers and update the dns record if your ip address changesI found that on the pi it didn’t work with the apt-get install commands so follow the instructions half way down this webpage to manually install it, once set up it will run in the background and tell the noip.com servers if you public IP address has changed

Now for those who don’t want their pi to be linked to a url, I also wrote a small bash script that you can run with cron at whatever interval you want, that checks the current public IP to the previous IP and then emails you of it has changed. This is how I implemented it, with help from this website to set up mail on the pi and link it with your gmail account. I used the SSMTP setup on that page.

Once this is set up, create a file (doesn’t matter what it’s called, but i used ipaddy.sh), make sure to make it executable and open it with your favourite text editor

touch ipaddy
chmod +x ipaddy.sh
vim ipaddy.sh

and then paste in this script, entering your own email address

#!/bin/bash
ip=$(curl icanhazip.com)
line=$(head -n 1 ipaddress)
if [ $ip != $line ]
then
    echo "I have changed your home server ip to: $ip" | mail -s "Shoppinglist.no-ip.org" <your_email_address>
    echo $ip > /home/pi/ipaddress
fi

Close the text editor and create another file which will hold your public IP address

touch ipaddress
curl icanhazip.com > ipaddress

This is create a file and put your public IP address in it for use by the above script Then we need to set up old mate cron to run the script when you want it, I have mine set to run the script every 30 minutes, but you can change it to whatever you like

crontab -e

This will open up the crontab for you to edit, go down the page a little until there’s some blank lines and paste this in:

*/30 * * * * /home/pi/ipaddy.sh

Now you are all set up with a webserver and a way to always know your IP address for when you aren’t at home! The last thing you need to do is forward the right ports out of your home router so you can access your pi over the internet. I have a netgear so there’s plenty of online guides on how to do it, it’s pretty straightforward though

Hope this is useful!

Posts Archive

Mon, 01 Jan 0001 00:00:00 +0000

Mr Whale

TTRSS + Calibre - Turn your RSS feed into an EBook

Create a new “news” source

Adding custom news source

Add a new recipe

Switch to advanced mode

Code and stuff

Adjust scheduling

How Calibre shows a ttrss ebook in the list

recipe

E-ink cryptocurrency ticker

the hardware

raspberry pi 3

Pomodoro inkyphat 2.7” e-ink display (black and white model)

the software

the code

Default Azure Search Service query key and Terraform

Useful Commands - OpenSSL

Read .cer in plain text

convert from pkcs12 (windows format, with private key and cert) to pem (-nocerts for only priv key or -nokeys for only certs)

Read pfx cert

Read website SSL cert

Get website SSL expiry

Convert privkey and cert to pfx (going from linux to windows)

Testing mail servers are using ssl

verify csr

Useful Commands - Image manipulation

Using all 3 tools

Using just convert

Conclusion

Cross compiling Litecoin Core

Host PC

Target PC

Instructions

Issues I ran into along the way

Sites that helped me along the way

Moving Magento

PiHole and OpenHAB

piHole

OpenHAB

Useful Commands - MSSQL

Ubuntu Live USB with persistence

Requirements

Steps

RuuviTag

Ruuvi

RaspberryPi 3

Setup

Node and systemd

OneTimeSecret

C.H.I.P, flic and openhab

Setup

Openhab

Setup in the house

Hardware

Raspberry pi

Ninja Blocks

Limitless LED

MPD server

Smart Home

Mini DLNA and bubbleUPnP

Upgrading SD on the Raspberry Pi

building phonegap app for Android and iOS

Counter Strike and Pebble

the backend

IDE choice

pebble app

C.H.I.P

Fun with cPanel

Key based login for SSH

cPanel and aws ec2

Getting custom sites into cPanel

Note

Composer and cpanel

TV dashboards with chromecasts and dashing

Setup

Dashboard

Getting calendar info into the dashboard

Dashboard Controller

Some issues along the way