Sunday, May 15. 2016
Owncloud Upgrade
The other day the internal disk of my old laptop had enough. That laptop was a very old one which I was using connected to my TV for common daily (non-working) activities. I decided to replace it with a little form factor barebone. During this week I have been re-installing all my stuff in the new box and, among other things, my owncloud should be configured. I have to say that I am quite disappointed with the current status of the software. I had the application installed using the Debian packages in the broken box and, from time to time, I saw an upgrade. And everything was perfectly smooth as it usually is in this distribution (even in the testing branch where I am). But this week I realized the following nasty things:
The owncloud package is no more in the Debian repository (see this email which links to another one in Fedora which is incredibly revealing). So I was forced to change from Debian package to manual installation.
Then I received the surprise that owncloud needs to be upgraded major version by major version (skipping major releases is not supported). I was in 7.0.x (the last version that Debian provided), so I have to finish at 9.0.x going through 8.0.x, 8.1.x and 8.2.x.
Performing the manual upgrade my second surprise was that some applications are disabled after the upgrade (you have to manually enable them after that). In my case there were mainly calendar, contacts and notes (the only one that is experimental and added explicitly). If you do not enable an application, and it needs some upgrade, the upgrade is not done and maybe the app fails later in the next major release when you finally enable it. I painfully learned this point.
- Finally I could not go to the last version 9.0.x because some problems in the upgrade. I decided to relax and wait some time with version 8.2.4.2. If I finally have to delete all the fucking affected tables and manually re-insert the rows one by one I will seriously consider to change my synchronization software.
Hey owncloud guys! This is a bit insane, isn't it? I use a distribution because the incredible job of a bunch of guys permits me (and millions of other people) to use your software seamlessly. Sometimes Debian is quite radical but, if Fedora maintainer also followed the same path before, maybe you have to reconsider your position.
Owncloud offers its own repository for several distributions (Debian among them). This way almost all the pain I am commenting here disappears, because, again, the software is automatically upgraded version by version (almost because of the notes experimental app). But I have some problems with this approach. First of all, I trust much more on Debian than in owncloud (sorry guys) and, second, I need to start with the last version which I cannot install because of the bug in the database upgrade. Personally I do not like this way and I suspect that this is exactly what owncloud wants me to do after getting rid of the distributions. (Yeah! I know I am fucking screwy.)
In general the installation of my previous data (backed up from my old laptop) to the new box, and the later upgrade to a recent owncloud version was a nightmare. I am going to summary the upgrading process just for the next time I need it.
First do a proper backup of your version data (in my case I have a cron script that already perform this point). The database, config and data directories should be backed up.
cp ${OWNCLOUD_HOME}/config/config.php ${BACKUP_DIR}/config.php tar zpcvf ${BACKUP_DIR}/data.rgz ${OWNCLOUD_HOME}/data mysqldump -u root -p owncloud > ${BACKUP_DIR}/owncloud-bbdd.dump
Stop your apache server to perform the upgrade to the next major version.
systemctl stop apache2
Download the next major version (it should be the last minor version for the next mayor version). I decided to use a link (in /var/www directory) between versions to not change apache configuration. The link is used to maintain previous version intact.
cd /var/www rm owncloud # delete the current link wget https://download.owncloud.org/community/owncloud-X.X.X.zip unzip owncloud-X.X.X.zip mv owncloud owncloud-X.X.X ln -s owncloud-X.X.X owncloud
Copy from the previous version the directories previously saved (Y.Y.Y is the old version, you can also use the backup files if you want).
cd owncloud cp ../owncloud-Y.Y.Y/config.php config/config.php cp -r ../owncloud-Y.Y.Y/data data chown www-data:www-data config/config.php chown -R www-data:www-data data
This is a custom point for mi case. The notes app is experimental and not distributed by the common bundle. So it should be added manually to the new version. The important point here is that you have to download a proper version of the app for the current major version of owncloud. There is no magic here, I decided the version of the notes app just reading the release comments. The notes app releases can be downloaded from here.
cd apps wget https://github.com/owncloud/notes/releases/download/.../notes-X.X.X.tar.gz tar zxvf notes-X.X.X.tar.gz mv notes-X.X.X notes
Start the apache server again in order to perform the automatic upgrade.
systemctl start apache2
As soon as you enter in the owncloud URL it detects the mismatch with the ddbb version and starts the upgrade. So go to the login page and upgrade the software.
As I commented before, some applications are disabled during the upgrade, once the software is ready login into the application and enable your previously activated apps. In my case:
Productivity: Contacts, Calendar and Documents.
Not enabled: Notes
When the application is enabled it also detects there have been a version change and the data (database mainly) is upgraded if necessary.
Now your owncloud is at the next major version, please check that all your data is in place and everything works fine (you see your events, contacts, pictures and so on). If so jump again to step (1) to upgrade to the next major version until you are at the last current one. And remember, the first step is the backup, do not forget this.
Today's entry is a little summary for a manual multi-major version upgrade of an owncloud installation. Like other posts in the blog it is intended mainly for me, but maybe some of you can take advantage of the information.
Regards!
Saturday, November 22. 2014
Script that adds Mode Lines to my TV
Another entry motivated by an oversight. My old desktop computer died some months ago and I decided to replace it with my old laptop (it was just packed in the box room). The desktop PC was connected directly to the TV and so is the laptop now. But my TV has a very nasty problem (thanks samsung), the EDID is incorrectly or not interchanged at all. In the desktop box I had to configure the mode lines manually using the xorg.conf file. I remember that I had to boot windows, install some weird application in order to get the current mode lines and add them to the file. When I switched to the laptop the situation was more or less the same, but in this case, I used the xrandr command. Besides I realized that another command, cvt, shows standard mode lines in VESA (that avoided me to repeat the horrible process with Windows). Finally I did a simple but working script that configured the new mode line, set it to the TV output and finally switch off the integrated screen. It was executed as soon as the laptop was switched on.
The other day I was cleaning the computer's hard drive (it was almost full with things that are now inside my new laptop and properly backed up), when I deleted the script (the fact that it was called lala.sh did not help much). So I had to re-write it again. It was just some minutes but I hate when things like that happen. This one will not happen again. Here it is my little script:
#!/bin/bash # For more details https://wiki.archlinux.org/index.php/xrandr #cvt 1360 768 # 1360x768 59.80 Hz (CVT) hsync: 47.72 kHz; pclk: 84.75 MHz #Modeline "1360x768_60.00" 84.75 1360 1432 1568 1776 768 771 781 798 -hsync +vsync # add the 1360x768 VESA mode (cvt) xrandr --newmode "1360x768_60.00" 84.75 1360 1432 1568 1776 768 771 781 798 -hsync +vsync # assign the mode to the TV xrandr --addmode VGA1 "1360x768_60.00" # set the mode, make TV primary and switch laptop screen off xrandr --output VGA1 --mode "1360x768_60.00" --primary --output LVDS1 --off
Now I will have to rethink the script no more for sure. Obviously I will lose other things but not this one. I am really a disaster and I feel like a fucking idiot when I have to do the same thing again and again... Which is quite common I must say.
Never again!
Friday, August 22. 2014
Cyanogenmod and Owncloud
Another personal entry this time. I have re-newed my old Samsung Galaxy Y Duos and, after a long period of doubts, I decided to buy a Moto G dual SIM (XT1033). It is (I think) the only dual SIM device that is supported by cyanogenmod. I really did not want to carry two phones again, and having a more friendly and trustworthy android was also a must. So this entry has two sections, the first one describes the installation of cyanogenmod in my new device (basically following the cyanogenmod installation instructions), and the second part is the installation of owncloud in my PC server in order to sync contacts, calendar and files against my own server.
Cyanogenmod installation on XT1033
As I commented the installation is easy and I just followed the steps provided by the cyanogenmod procedure. I am going to add some little details of my particular environment (debian laptop, so on and so forth). During this process the device should be connected to the PC using the USB cable.
The installation needs the android tools in order to flash the device. So installing some packages is necessary.
# apt-get install android-tools-adb android-tools-fastboot
USB debugging should be activated in order accept tools commands. First you have to activate developer options, Settings → About Phone, Click several times in Build Number until a popup appears explaining you are now a developer. Then go to the new Developer Options menu and check USB Debugging on. Please see this video for a detailed explanation of the process.
This step should be repeated if the USB debugging comes disabled back again.
When USB debugging is activated, reboot the device to bootloader (in my laptop I had to be root).
# adb reboot bootloader
And now the device should appear under the fastboot.
# fastboot devices XXXXXXXXXX fastboot
The XT1033 needs to be unlocked in order to install another Operating System. In this point the unlock data is requested to the phone.
# fastboot oem get_unlock_data ... (bootloader) XXXXXXXXXXXXXXXX#XXXXXXXXXXXXXX (bootloader) XXXXXXXXXXXXXXXXXXXXXXXXXX#XXXX (bootloader) XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (bootloader) XXXXX#XXXXXXXXXXXXXXXXXXXXXXXXX (bootloader) XXXXXXX OKAY [ 0.158s] finished. total time: 0.158s
With the unlock data obtained we need to access to the motorola site and request the unlock key for the device. You have to follow the instructions displayed in the page step by step (the information provided by the previous command should be submitted). Finally a key (a string of numbers and characters) is obtained and the device can be finally unlocked with it.
# fastboot oem unlock XXXXXXXXXXXXXXXXXXXX ... (bootloader) Unlock code = XXXXXXXXXXXXXXXXXXXX (bootloader) Unlock completed! Wait to reboot FAILED (status read failed (No such device)) finished. total time: 28.321s
Although the previous command said that it failed, the device booted again and was ready to the installation of the recovery.
Please take in mind that you are unlocking the device. When you do that the startup process show a beautiful warning boot screen noticing this new situation (I suppose the unlocked logo can be changed but I have no problem with it). There are some penalties unlocking your device, among them, your device will not be covered by the motorola warranty anymore (read this entry for a more detailed explanation). But, for me, cyanogenmod is worth the risk.
Download the recovery image for falcon (Moto G codename in cyanogenmod). It seems the last one right now is 6.0.4.7.
# wget http://loki.rombitch.com/Devs/Dhacker29/MotoG/CWM-swipe-6.0.4.7-falcon.img
And again, going to bootloader, flash the recovery image.
# adb reboot bootloader # fastboot devices XXXXXXXXXX fastboot # fastboot flash recovery CWM-swipe-6.0.4.7-falcon.img target reported max download size of 536870912 bytes sending 'recovery' (9954 KB)... OKAY [ 0.336s] writing 'recovery'... OKAY [ 0.394s] finished. total time: 0.730s
The phone reboots again and now the recovery menu should be started (hold Volume Down & Power simultaneously at startup). In this menu you move up and down using volume up and down buttons, and power button selects the option.
At this point download the image of cyanogenmod for falcon. I finally decided to install last snapshop M6 (it seems that this device has some problems with audio and data right now and has lost the track of the last stable releases, currently cyanogenmod is at M9 snapshot release).
# wget http://download.cyanogenmod.org/get/jenkins/67673/cm-11-20140504-SNAPSHOT-M6-falcon.zip
Backup your current stock ROM. In the recovery menu select backup and restore and there backup to /sdcard. It generates a backup with the current date and time (format YYYY-MM-DD.hh.mm.ss).
Then go back and select wipe data/factory reset.
I decided to use the sideload option for the flashing. Select Install zip → Install zip from sideload... The device waits for the zip file to be sent. Execute the following adb command to send the cyanogenmod ROM to the device.
# adb sideload cm-11-20140504-SNAPSHOT-M6-falcon.zip sending: 'cm-11-20140504-SNAPSHOT-M6-falcon.zip' 100%
When the progress bar finishes you can go back and select reboot system now. Finally cyanogenmod 11M6 was installed in my device.
Synchronizing your phone with owncloud
The second part of the entry is about owncloud. This software is a PHP application that handles calDAV (protocol to manage calendar events over HTTP), cardDAV (contacts) and general webDAV (common files) in order to synchronize device data against your own server. I had read a nice entry about this application long time ago and I decided it was the perfect time to start using my own server to store contacts, calendar events, photos,... If you do not know me, until now I simply do not synchronize anything (I just backup things manually from time to time). So this second part of the entry explains how owncloud was configured in my desktop PC (obviously a debian testing box) and which applications were installed in the device to synch my data. For the server part I followed the procedure explained in the sharadchhetri.com blog, although I did a more specific setup.
First it is important that your desktop has a fixed IP address (do not use DHCP). This way the software will be always available in the same IP and your device will contact to the owncloud server with no issues. In my case this step was already done (my server IP is 192.168.1.2 and is named venom.local).
Now the required software is installed (owncloud is a free software so it is available in the debian repositories).
# apt-get install owncloud mysql-server
Owncloud needs a database (I decided to continue using MySQL) and requires a lot of depending software (PHP, Apache,...) which is installed as dependencies.
Now the database and the user for the application are created.
$ mysql -u root -p ... mysql> create database owncloud; mysql> create user 'ownclouduser'@'localhost' IDENTIFIED BY 'XXXXXXXX'; mysql> grant all on owncloud.* to 'ownclouduser'@'localhost';
Add your local IP and hostname to the trusted domains configuration (file /etc/owncloud/config.php). In my case the trusted_domain section was like this.
'trusted_domains' => array ( 0 => 'localhost', 1 => '192.169.1.2' 2 => 'venom.local', ),
With this modifications the owncloud server is ready to roll. Restart your apache server.
# /etc/init.d/apache2 restart
- Now it is time to Access to your owncloud server (URL http://localhost/owncloud/) and configure it. The information needed by the software is almost nothing: administrator account, file system to store files (change it if needed) and the database access information. The following image is the information I entered.
I decided to use owncloud using HTTPS (you will not believe how difficult is this simple task, but not because of the server but because of the android device). After some different tries I finally created a CA certificate and another certificate for the web server (signed by the CA). All the files are going to reside in the /etc/ssl directory of my desktop machine.
Prepare the environment for the CA:
# cd /etc/ssl # mkdir -p demoCA/newcerts # touch demoCA/index.txt # echo 01 > demoCA/crlnumber # echo 01 > demoCA/serial
Create the real certificate for the CA:
# openssl req -subj "/C=ES/ST=Madrid/O=Home/CN=ca.local" -new -newkey rsa:2048 -keyout private/cakey.pem -out careq.pem # openssl ca -out cacert.pem -days 10000 -keyfile private/cakey.pem -selfsign -extensions v3_ca -infiles careq.pem # openssl x509 -in cacert.pem -outform DER -out cacert.der # openssl pkcs12 -export -out cacert.p12 -in cacert.pem -inkey private/cakey.pem
With the created CA the certificate for my host was generated:
# openssl genrsa -out private/venom.local.pem 2048 # openssl req -subj "/C=ES/ST=Madrid/O=Home/CN=venom.local" -key private/venom.local.pem -new -out venom.local.req # openssl ca -in venom.local.req -days 9999 -extensions usr_cert -cert cacert.pem -keyfile private/cakey.pem -out venom.local.pem
After that, the HTTPS virtual host should be added in the apache server (a default-ssl is ready to be enabled by default in debian).
# a2enmod ssl # a2ensite default-ssl
Finally modify the site configuration to properly link to the created key and certificate pair (modify the following properties in the ssl virtual-host file /etc/apache2/sites-enabled/default-ssl.conf).
SSLCertificateFile /etc/ssl/venom.local.pem SSLCertificateKeyFile /etc/ssl/private/venom.local.pem SSLCertificateChainFile /etc/ssl/cacert.pem
Restart again the web server and check with a browser that owncloud is working with the new HTTPS port and certificate (now access the URL https://localhost/owncloud/, the typical warning message about a non-trusted certificate should be displayed).
Once the software is running the well-known address should be defined. These URLs are just redirections to the proper URL where calDAV and cardDAV protocols are available. In the case of apache just enable the mod_alias module (if it was not enabled before).
# a2enmod alias
And create to permanent redirections, from the well known address to the correct owncloud ones (you have to define these two lines inside the ssl virtual-host configuration too, /etc/apache2/sites-enabled/default-ssl.conf).
Redirect permanent /.well-known/caldav https://venom.local/owncloud/remote.php/caldav/ Redirect permanent /.well-known/carddav https://venom.local/owncloud/remote.php/carddav/
Check with any browser that, if you access to any of the well-known address, the browser is redirected to the owncloud services for calDAV and cardDAV respectively.
Finally I installed an extra application to owncloud (a notes application that will be used to synchronize my notes) which does not come with the default debian package. The installation is very easy.
# git clone https://github.com/owncloud/notes.git # cp -r notes /usr/share/owncloud/apps/
After restarting the server a new notes application is displayed in the left menu.
Now everything is working in the server side (it would have been better to use any other non-admin user, but it is late for that) and you just need to install some applications in your device. All of them are open source and available through FDroid. In my case I decided to use the following apps:
- DAVdroid: An application which provides a synch account for calendar and contacts. It is just managed as any other account in your device. For configuring it, the well-known address are needed.
- MyOwnNotes: A little notes application which also synchronizes them to the extension installed previously in the owncloud server.
- owncloud: The application provided by the server company which let you upload and download files into the server (dropbox style). The application does not synch anything for the moment (it seems that FolderSync application does it, but it is not open source, so it is not in FDroid, so it is not an option for me), but owncloud guys are working in this feature, let's see what happens.
The configuration of the different applications works smoothly but only if you have done two little things previously (I spent a whole morning with these two little issues): the web server hostname (venom.local in my case) must be resolved by the device and the certificate CA should be a trusted one. I am going to try to explain how I did it, but I spent so much time and did so many things that I am not completely sure if the following steps are complete (I must say that android is very, very annoying, this tasks are absolutely simple for any other Operating System).
The first thing is just adding the fixed IP of my PC inside the hosts file (any *nix system uses a hosts file). The problem seems to be that the /system/etc/hosts file is mounted in a read-only file system. after reading this thread the solution was clear. Using the Terminal Emulator application I executed the following (when doing the su command the terminal requests access to root, so you need a rooted device).
$ su # mount -o rw,remount /system
And now using the File Manager application the damned line is added to the /system/etc/hosts file (rebooting the device the FS is read-only again).
127.0.0.1 localhost 192.168.1.2 venom.local venom
The certificate issue, once you know how to do it, is also an easy task. You can follow one of the several pages that explain the installation of a custom CA certificate in the device (for example this one explained by people of MyOwnNotes). The CA ceriticate in DER mode is needed (cacert.der file created before), please, if you follow the previous procedure, download the CA certificate, not the certificate of the server. Once the process is done, you can check if the CA cert is correctly installed just accessing to your server (the device browser should not show the typical security warning). It worked for me but it has two annoying side-effects: first, in order to import a custom cert you need to establish a screen lock method (password, pattern, pin or whatever, which I do not use); second, a horrible warning is permanently displayed in the phone (your network could be monitored or something similar). Finally I discovered in a forum entry that you can move the imported cert from the user directory to the system default one (it is good to be rooted). So the file under /data/misc/keychain/cacerts-added/ was moved to /system/etc/security/cacerts/ and then the user certificate store cleared (this way the annoying warning disappears too).
With both steps completed all the commented applications were configured smoothly against the owncloud server.
This time the entry is again for my own use. I wanted to respect my privacy (avoiding internet repositories) but not to continue living in the Pleistocene epoch (as I commented, I did not use those features before). So after reading the bytopia post about owncloud I decided that this software could be a perfect local repository for me. The time came when I re-newed my phone device to a new one, that is why this entry has two parts: cyanogenmod installation on Moto G XT1033 and owncloud configuration. Awfully owncloud configuration with a non rooted device is much more difficult and I am not sure if it would be possible without important drawbacks. Using a dynamic dns and maintaining your server constantly running the device could contact and synchronize against your server anytime and anywhere (but I do not recommend it, please do not waste resources uselessly). Obviously a hosted machine, if you have one, is even a better approach. So the summary is that now I have a dual sim phone with cyanogenmod 11 (no google apps, no accounts, no nothing), and my calendar, contacts, notes and files can be synchronized against my own desktop machine as soon as I arrive home and switch it on. Absolutely perfect!
Regards!
Comments