<alert warning>This is an archived page <icon fa archive>, no further development is anticipated </alert>

 ====== FabLab Server ======

The fablab server is an in-house edge programming server, running the internal wiki. The need for a hosted wiki has come about through the mortal need for documentation, inspired in part by the CCC project.  Researching the available wiki software (including confluence available on SLQ networks) we settled [[https://www.dokuwiki.org/dokuwiki#|Dokuwiki]].  Dokuwiki is simple to edit, maintain and back-up.  It doesn't require a database for installation and can be edited from any modern web browser. The  wiki uses a custom template and has a few extensions installed.

<WRAP center round alert 60%>
The Fablab  Server is used for development over the internal networks only!! It is not secured enough for external deployment, the instructions here are considered solely for an internal, private server.
</WRAP>

 ====== Hardware ======

The server runs on a recycled HP8100 mid-tower. Its physically located in The Edge's Fabrication Lab.

{{ :mobileupload:andrei:20170329_151235.jpg?400 |}}

 ====== Software ======

<del>The server is running Ubuntu 15.10, with a LAMP stack.</del>Ugraded to 16.04.


===== rdiff backup with cron to external USB drive ===== 

The back-up routine is daily to an external drive.  This will be augmented to a second drive and hopefully some space on an ICTS asset somewhere...

The server runs a cron job (a regular scheduled task) to back up the entire web directory (www) to an external drive. 

 ===== Finding and Mounting drives and directories =====

 First lets check the www directory - normally it resides in /var/www/html

  ls -al /www/www

 UIn our case, these are symlinked to another location((This is so we can do a systemback of the server, excluding the www data)). For example the dokuwiki install is:

  lrwxrwxrwx 1 www-data www-data 17 Mar 17  2016 /var/www/html/dokuwiki -> /www/www/dokuwiki

The  /www folder is actually another drive mounted on boot - check it with ''nano /etc/fstab''

<code>
# /www was on /dev/sdb1 during installation
UUID=d497c901-515c-456f-bddd-aaa5684c4bd3 /www            ext4    defaults        0       2
</code>

Now lets mount our backup USB on boot - First chuck it in then find the UUID using ''sudo blkid''

  /dev/sdc1: LABEL="edgeserverbkup" UUID="620df2f1-4787-4584-93d3-2cb4ad24d983" TYPE="ext4" PARTUUID="ad59a428-01"
  
Then edit fstab to mount on boot.

  UUID=620df2f1-4787-4584-93d3-2cb4ad24d983 /mnt/externalusb        ext4    defaults        0       1


Now its time to reboot.

 ===== Using rdiff-backup =====

From [[http://rdiff-backup.nongnu.org/|rdiff-backup]];


//rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership, modification times, extended attributes, acls, and resource forks. Also, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted. Finally, rdiff-backup is easy to use and settings have sensical defaults.//
 
==== ====
What we want is a incremental back-up of the entire /www/www/ directory to /mnt/externalusb/www. With rdiff-backup this simple. First lets make our directory on our external drive.

  sudo mkdir /mnt/externalusb/www
  
This is the command we want to run:

   rdiff-backup --preserve-numerical-ids /www/www /mnt/externalusb/www
   
run it once (as sudo) to check its working.

Next step is to make this happen daily..

 ==== Set up a Crontab ====

Access the system crontab.

  sudo crontab -e
  
We want daily backups so add this line

  @daily rdiff-backup --preserve-numerical-ids                      /www/www      /mnt/externalusb/www

Save and exit.


 ====== Automount USB =====

https://askubuntu.com/a/772387


 ======= Mobile Uploads ======

===== Why it was done =====

Motivated by the finicky nature of uploading I've hacked a together a solution that copies over uploaded photos to this namespace. The old method was:

  - take photo
  - upload over usb to computer
  - log into wiki
  - upload to wiki
  - go to media manager and insert photo.

New method is;

  - take photo on mobile (with this camera app - [[https://itunes.apple.com/app/upupu/id508401854|upupu]] on iOS)
  - or [[https://play.google.com/store/apps/details?id=net.zekjur.davsync|DAVsync]] on Android.
  - uploaded via webdav (this is built into upupu and occurs automagically with DAVsync)
  - go to media manager and insert photo.

 ===== Setup For Staff ====

 ==== iOS ====

IF you want to use your own IOS device  - device install [[https://itunes.apple.com/app/upupu/id508401854|upupu]] app from store.

Set up the webdav settings in upupu with:

<WRAP center round box 60%>
Webdav    ON

server http://192.168.36.78/webdav/mobileupload/yourname

user : yourname

pass : yourpass

</WRAP>


Source can be found on [[https://github.com/xcoo/upupu|github]] 

 ==== Android ====

Install [[https://play.google.com/store/apps/details?id=net.zekjur.davsync|DavSync]] from Google play or compile it yourself. Then, chose “DavSync” in your application drawer. Set up the following fields:

<WRAP center round box 60%>


server http://192.168.36.78/webdav/mobileupload/yourname/

user : yourname

pass : yourpass

</WRAP>

Source can be found on [[https://github.com/scastrorico/android-davsync|github]].

===== Server Set-Up =====

The server has folder, accessable over [[http://www.webdav.org/|webdav]] ( a simple file sharing system that is accessible over http), that is monitored by [[https://github.com/axkibe/lsyncd|lsyncd]]. Lsyncd has a configuration file (written in lua) that tells its to copy the contents of the webdav folder into the media directory in the wiki.


==== Set-up webdav ====

 Originally webdav was set-up with digest authorisation however a couple of the syncing apps would only use password, so we've switched to basic auth.  

<WRAP center round alert 60%>
This is another reason the server is not production ready - basic auth should be over SSL only!!!
</WRAP>

=== Apache Setup ===

 First up webdav needs to be turned on in our apache2 server using 
  
   sudo a2enmod dav
   sudo a2enmod dav_fs
  

Next we make a webdav directory on our data drive:

   sudo mkdir /www/www/webdav

And link to to our web directory

   sudo ln -s /www/www/webdav/ /var/www/html/webdav

Lets make a directory for mobile uploads.

   sudo mkdir /var/www/html/webdav/mobileupload
Now make directories in the webdav directory for each staff member

    sudo mkdir /var/www/html/webdav/mobileupload/phil

Change ownership so the apache2 server can read the link and directories;

   sudo chown -R www-data:www-data /var/www/html/webdav
  
Now edit the apache2 server sites-available with 

   sudo nano /etc/apache2/sites-available/000-default.conf


And enter the following:
   
      Alias /webdav /var/www/html/webdav
    
        <Directory /var/www/html/webdav>
            DAV On
        AuthType Basic
        AuthName "webdav"
        AuthUserFile /var/www/passwd.dav
        Require valid-user
        </Directory>


 === Webdav Passwords ===

Now we need to add webdav users and create the passwords. For the first user use:

   sudo htpasswd -c /var/www/passwd.dav phil

You will be prompted to create a password for this user.
   
Then for following users try this:

   sudo htpasswd  /var/www/passwd.dav mick

Repeat for remaining users.

Next make sure your passwd.dav file is owned by the apache server.

   sudo chown www-data:www-data /var/www/passwd.dav

 ==== Lsyncd =====


Next install lsyncd with

  sudo apt install lsyncd
  
Create a log directory

  sudo mkdir /var/log/lsyncd
  touch /var/log/lsyncd/lsyncd.{log,status}

Then create a cofiguration file directory.

  sudo mkdir /etc/lsyncd
  

  sudo nano /etc/lsyncd/lsyncd.conf.lua


And enter this lua code

<code>


settings ={
        logfile = "/var/log/lsyncd/lsyncd.log",
        statusFile = "/var/log/lsyncd/lsyncd.status"
}

--
-- This syncs our webdav to dokuwiki for photo uploads.

sync  {
        default.rsync,
--      wait = 15,
        delete = false,
        source = "/www/www/webdav/mobileupload/",
        target = "/www/www/dokuwiki/data/media/mobileupload/",
        rsync={
        owner = true,
        group = true,
        perms = true
        }
}
</code>

Now lets start lsyncd as a service

  sudo service lsyncd start

This performs a basic rsync command whenever the "/www/www/webdav/mobileupload/" changes.  We use the command ''delete = false'' to force lsyncd to leave new files in our destination directory untouched -why? See below...


===== Fixing Capitalisation issues =====

For some reason dokuwiki does not like CAPITALS in media filenames, neither in the name nor exstention. So while ''2971057.jpg'' will appear in dokuwiki ''IMG_2971057.jpg'' or ''2971057.JPG'' will not.

This is fine when using upupu, but both davsync (Android) and whatever Phil is using use capitalisation in the file names. As an added bonus Phil's app creates directories, which means we need to look through them recursivley.

To fix this there are a couple of scripts running in start up, triggered by a cronjob. Lets set them up now. First go super with

     sudo su
 
Next lets make a directory for our scripts

   mkdir -p /usr/scripts/photo_rename
   
The first scripts will set up some inotifywait watches on the mobileupload directories.  I've used one per user so for Phil lets do:

   nano inotify_phil.sh
   

Then we set up a while loop that uses inotifywait to detect a file action, then triggers another script called '' exiftool_rename_phil.sh''

     #!/bin/bash
     DIRTARGET=/var/www/html/dokuwiki/data/media/mobileupload/phil
     while true #run indefinitely
     do''
     inotifywait -r -e close_write  $DIRTARGET  &&  /bin/bash /usr/scripts/photo_rename/exiftool_rename_phil.sh
     done

   
Now lets make our script to rename the photo with the date and time using ''exiftool'', then change the extension with ''mv''

     
    #!/bin/bash
    DIRTARGET=/var/www/html/dokuwiki/data/media/mobileupload/phil
    exiftool '-FileName<CreateDate' -d %Y%m%d_%H%M%S%%-c.%%e $DIRTARGET/*.JPG  &&
    find $DIRTARGET -name *JPG -exec sh -c 'mv "$0" "${0%%.JPG}.jpg"; echo "Moved $0 to ${0%%.JPG}.jpg"' {} \;


 Finally set-up a crontab to run the script on reboot. So '''crontab -e'' then add this line.

       @reboot /usr/scripts/photo_rename/inotify_phil.sh

====== Exporting Sections of the Wiki ======

When we need to take sections of the wiki offline to another location, follow this procedure;

 ==== Install New Wiki =====

Follow the steps [[engagement:volunteering:wiki:emergency_recovery_procedure#full_restore_of_the_wiki_and_server|here]] to make a new wiki.

 ==== Export Wiki Data =====

Now we 

   sudo tar -pvczf dokuwiki.tar.gz  /www/www/dokuwiki  --exclude=/www/www/dokuwiki/data/page

Next lets grab only the one last apocalypse data. 

  sudo find /www/www/dokuwiki/ -type d  -name one_* | tar -pvczf dokuwiki_data.tar.gz  --include-from - 
  
  tar -tf dokuwiki_data.tar.gz 

 ======  Recovery Procedure ======

This document covers recovery of content from your wiki, fixing broken plugins or templates, and total back-up.

 ====== Content Recovery ======

The easiest way to recover content is using the built in ''old revisions'' editor accessible from the right menu.

{{ engagement:volunteering:wiki:revisions.png?direct&100 |}}

Then your can view all the changes made to that page.

{{ engagement:volunteering:wiki:old_revisions_view.png?direct&600 |}}

<WRAP pagebreak></WRAP> 

And choose two changes to view the differences. Either side by side;

{{ engagement:volunteering:wiki:diffs_side_by_side.png?direct&600 |}}

Or inline. 

{{ engagement:volunteering:wiki:diff_inline.png?direct&600 |}}

If you want to revert changes, select the version of the page you want (click the blue link) and it will open the page with a note at the top..

{{ engagement:volunteering:wiki:this_is_and_old_revision.png?direct&600 |}}

<WRAP pagebreak></WRAP> 

Edit the page, once again you will see note at the top.
IF this is correct, then just save the edit and the changes will be made, creating a new version of the page.

{{ engagement:volunteering:wiki:edit_old_revision.png?direct&600 |}}

 ====== Fixing broken Plugins or Templates =====

If the formating or function of the wiki goes haywire because of template or plugin updates or changes, you can use the server back-up to restore previous versions.  This is useful because most plugin and template makers don't keep old versions around, relying on the wiki user to keep backups of their own wikis. 

 ===== Re-installing Plugins and Templates =====


 ==== Restoring old versions of Plugins or Templates ====

We use rdiff-backup to daily backup the wiki (and the entire www directory). To understand how the back-up works and set it up see [[facilities:fablab:equipment-custom:fablab-server#rdiff_backup_with_cron_to_external_usb_drive|here]]

 ===== Restoring a folder from rdiff-backup =====

Lets say the we need to restore a working version of the revealjs plugin, after foolishly upgrading it and breaking all our existing slideshows((like I just did)). We know that we want to go back to how the wiki was about 20 days ago, when the slideshow was still working.

First we need to know where the plugin is installed.  Our wiki is located in 

  /www/www/dokuwiki

 All dokuwiki plugs live in:

   dokuwiki/lib/plugins
   
They all have there own folder, so the full path to revealjs would be

  /www/www/dokuwiki/lib/plugins/revealjs/
  
Now we know where we want our restored files to go to, lets see where rdiff-backup keeps  our backups.  The file structure is mostly preserved by rdiff-backup so our revealjs folder would be in on the back drive.

  /mnt/externalusb/www/

inside the rdiff-backup-data/ directory

  /mnt/externalusb/www/rdiff-backup-data/

Then we are after the incremental backups...(this will let us go back a certain time)

  /mnt/externalusb/www/rdiff-backup-data/increments/

Finally we find the revealjs directory.

  /mnt/externalusb/www/rdiff-backup-data/increments/dokuwiki/lib/plugins/revealjs/


The command we want is 

  rdiff-backup -r 20D 
  
 This will restore (-r) the file from 20 days ago.  

==== ====
So the full command to restore the revealjs directory contents from 20 days ago is..
  
  rdiff-backup -r 20D  /mnt/externalusb/www/rdiff-backup-data/increments/dokuwiki/lib/plugins/revealjs/ /www/www/dokuwiki/lib/plugins/revealjs


 ====== Full Restore of the Wiki From USB drive ======

If disaster strikes and the wiki goes down but the server is fine, you can restore it all from the USB external backup. Because rdiff-backup makes a //mirror// (an exact copy) we can just use the built in usual linux copy command with the '-a' for archive flag.  

  sudo cp -a /mnt/externalusb/www/dokuwiki /www/www/dokuwiki  
   
Log in an check the functionality.


 ====== Full Restore of the Wiki and Server ======

If disaster double strikes, and the only thing left working is the USB external backup - then to recover the wiki will require a re-install of the server and dokuwiki.  These instructions are adapted from ccclabs.edgqld.gov.au. They will require a computer on the The Edge basement SLQ network, with an ICT request to set a permanent DHCP lease for a MAC address.

 ===== Sort The Hardware =====

You will need an i5, with 4gig RAM and 100Gig HD. Just about any recent donation from ICTS will suite, but try to use one that is (a) not part of an existing workshop line (b) not radically overspec.

 ===== Install Server OS =====

I've adapted this from the official [[https://www.dokuwiki.org/install:ubuntu|dokuwiki]] instructions.  You will need internet access.  We will install the desktop version of (X)ubuntu for ease of use. 

First up, grab  workshop USB stick and make a fresh install of Xubuntu 14.04 (follow the steps in [[http://ccclabs.edgeqld.org.au/doku.php?id=ttt:preparation_04#preparing_installers|CCC]]) then install using these [[http://ccclabs.edgeqld.org.au/doku.php?id=hp8100:workshop_02#installing_the_operating_system|instructions]].

 ===== Get (or request) a fixed DHCP lease =====

If you are in an organisation that has an ICT department - get this underway early, as it can take a while.  The purpose is to make the server available over a network with a fixed address.

You will need to give your  ICT a port number where you are plugging the server in, and the MAC address of the network card in the new computer.  The port number is where you want to physically plug the server in.  The MAC address can be found with

  ifconfig
  
Once you know these details, let your ICT department know.

 ===== Install Server Software ====

 Now make sure you are up to date.

<code>
 sudo apt update && sudo apt upgrade
</code>

Dokuwiki need a web server and PHP. We use apache2 and PHP7.0.

<code>
  sudo apt install apache2 libapache2-mod-php7.0
</code>

Enable Apache Rewrite module.  This lets us use pretty URLs

<code>
  sudo a2enmod rewrite
</code>


===== Apache Set-up======

Next we need to set-up Apache,lets edit our default configuration.

<code>
  sudo nano /etc/apache2/sites-avialable/000*.conf
</code>

and change

    DocumentRoot /var/www/html
        
to 

   DocumentRoot /var/www/dokuwiki

Next we need to deny access to directories we want to keep secure in the main apache.conf file. 

<code>
sudo nano /etc/apache2/apache2.conf
</code>

For directory /var/www/ replace

<code>
      AllowOverride None
</code>

with

<code>
      AllowOverride All
</code>

Restart Apache2 service.

<code>
  sudo service apache2 restart
</code>

 ===== PHP Set-up =====

First lets install the GD library, which we need for using the PDF export.

<code>
sudo apt install php7.0-gd
</code>

Then lets add sqlite support for bits and pieces.

 <code> 
sudo apt install php7.0-sqlite3
 </code>
We'll need to make a few modifications to our PHP settings to allow larger uploads and posts to our wiki. 

<code>
 sudo nano /etc/php7.0/apache2/php.ini
</code>

Look for these two lines and change them to at least 15M (15 megabytes)

<code>
upload_max_filesize = 15M
post_max_size = 15M
</code>

<WRAP center round tip 60%>
Use CTRL+w to search in nano
</WRAP>

We then need to make sure php has enough memory allocated to run the upload scripts.

<code>
memory_limit = 256M 
</code>

<WRAP pagebreak></WRAP> 
 ===== Copy Wiki Content ====
Before we copy the wiki data back, lets install usbmount 

  sudo apt install usbmount
  
 ==== Mount the external USB backup ====

Once you've got the server up and running lets grab the USB backup - just plug it in and it should mount automagically - both on the desktop and under ''/home/yourusername/media''


Finally - this is the same as above - just use the ''cp'' command to copy from the backup to the server.

   sudo cp -a /home/yourusername/media/externalusb/www/dokuwiki /www/www/dokuwiki 
      
 ==== Change Template Paths ====

The final jiggery pokery is to set the template path for your new IP address. We need this as the theme we use is has some customisation. Go to the admin panel;

{{ engagement:volunteering:wiki:admin.png?direct&100 |}}

Then open //Configuration//

 {{ engagement:volunteering:wiki:configure.png?direct&500 |}}

and scroll down to the Bootstrap3 Template config

{{ engagement:volunteering:wiki:bootstrap.png?direct&800 |}}

Now change this IP 

  http://wiki.edgeqld.org.au/lib/tpl/bootstrap3/assets/bootstrap/edge/bootstrap.min.css
  
  
To  your new IP discovered earlier.