Live-Scripting

Live-scripting is an approach to combine work in IT projects, documentation and information retrieval.
Very much like test-driven development combines testing and coding, live-scripting documents work in command shells, while doing it.
But live-scripting is much more than that. It encompasses five functional areas as depicted in the following figure.

While working with live-scripting, a lot of documentation can be created easily in a short amount of time. In order for this information to be usable, live-scripting includes various search capabilities. Sharing of information is vital for any project. Live-scripting offers different sharing options. Since the basis of live-scripting documentation is ASCII text, versioning with source code control systems is very effective and supported by this approach.

Since more than 30 years and even now, IT-work in many case is command shell centric. For many IT professionals the bash or other shells constitute a major part of their work. Sophisticated commands are constructed during problem resolution. Normally, these commands are deleted at the end of a session. When similar problems arise days or weeks later, similar analysis and solution steps are repeated again. If documentation of the work is required, it is an extra time consuming task. We wish to document this work in an easy way for ourselves and others. This should include an effective search method to quickly find the documented information.

Live-scripting is a work methodology based on emacs, which documents work on the command shell while doing it. Instead of entering and executing one command at a time, in live-scripting we use two windows side by side: an editor and a shell. The commands in the editor can then be executed with a single key shortcut in the adjacent shell. This is similar to a debugging session where we can step through code during execution. In the Emacs editor we can jump between commands, repeat them in any order, duplicate and modify them. The editor can be saved into an ascii file at the end of the session.

Emacs is the ideal tool for this approach. The ansi-term provides for a robust shell interface within emacs. Since it is an programmable environment, the functions to send a line of code to the ansi-term can be implemented. Furthermore the org-mode module is an Emacs killer app on its own, adding many features to organize, format and publish the work. At it's core, org-mode is a mark-up language similar to markdown, but much more powerful. Emacs provides elaborated search capabilities across files and projects. The magit module is a highly praised interface to git. The fact that most files are plain text, invites the storage in a source code control system like git. But org-mode can also handle pictures to add screenshots and attachments for file types like PDFs and others. Org-mode together with a couple of other Emacs modules constitute a cross-media publishing machine which makes it easy to export to HTML, PDF, Markdown, Confluence, and more.

Live-scripting, as it is presented here, is a configuration that spans multiple projects and publishes the org-files and attachments to a single static website which I call orgweb. This can be used locally or be synchronized to a web server in an intranet or on the internet. The web site contains a search pages for the whole site based on lunr, which provides a full text search index for finding information across different pages and projects.

Live-scripting makes intensive use of open source technologies, as depicted in the following mind map.

The live script, which in many situations is the basis in this approach, is edited in Emacs, preferably using the org-mode module of Emacs. The shell commands are executed in the Emacs Ansi-term module, which is a terminal emulator written in lisp. The work is documented in an Emacs Orgmode file which is a powerful markup language. Images, Screenshots and other file attachments can be integrated via drag and drop. Using the Emacs org-publish module, a local static website is created, which uses a beautiful CSS style sheet, featuring a responsive design and easy navigation. For information retrieval the excellent Emacs search capabilities are used. The website uses a search function provided by Luna, which is an open source full-text search engine written in java script. For sharing a across the internet, the local website can be synchronized to a web server or to an Amazon S3 based website. The whole website can also be packed as an HTML archive. Individual org-mode files can be exported to PDF, formatted ASCII text, markdown and more. versioning is done using git which is perfectly supported by the Emacs Magit module. The git projects can be pushed to an online repository like GitHub or others. The Emacs Projectile module supports the multi-project approach of live-scripting.

A Linux installation with a graphical interface and a GNU Emacs installation is required to use live-scripting.
In this example we use Lubuntu 19.10 and GNU Emacs 26.3.

sudo sed -i -e 's|disco|eoan|g' /etc/apt/sources.list
sudo apt update
sudo apt upgrade

### New Release
lubuntu@lubuntu-pc:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.10
Release:        19.10
Codename:       eoan


### Install emacs
lubuntu@lubuntu-pc:~$ sudo snap install emacs --classic
2020-06-19T09:39:50+02:00 INFO Waiting for restart...
emacs 26.3 from Alex Murray (alexmurray) installed

### Clone project live-scripting
mkdir org; cd org
git clone https://github.com/andreaswittmann/live-scripting.git 

This creates the prerequisites for live-scripting.

In the simplest configuration, the GNU Emacs installation is adapted to execute shell commands from an orgmode file.
I use the init.el from the live-scripting project, which contains the configuration for basic live-scripting

### Copy init.el to emacs
cp ~/org/live-scripting/init.el ~/.emacs.d/

### Restart emacs

I can now open two windows side by side in Emacs. In the right window I execute an ansi-term.
In the left window I execute shell commands from an orgmode file.

Now shell commands can be executed directly. The code block here is only for formatting purposes.

### example execution of shell commands with live-scripting 

ls -la
date
whoami
pwd

Configuring Emacs is very time consuming. With the Spacemacs project I get an extensively configured and up-do-date Emacs.

Currently I only have one project which is live-scripting. But eventually I want to have multiple projects that will be published altogether. These needs to be considered when setting up the folder structure. Let's have a look at the folder structure and scopes as it is depicted in the following image.

Under the org folder, I can have multiple projects in separate folders that can be managed by git. This constitutes the individual project scope. Regardless of the project scope, I can open the files of all projects in Emacs and register the org files in the org agenda file list. I am thus able to use the orgmode search across all agenda files, which constitutes the orgmode search scope. When I publish the project to a web server I want all projects to be part of a single web, e.g. the orgweb. I will generate a full text search index and a sitemap across all html files of the web. This constitutes the web scope.
But to begin with, I start with a single project website, still keeping in mind the above considerations. The resulting folder structure is illustrated in the following picture.

Now instead of choosing the live-scripting as root for the website, I have one indirection level more. While this is not necessary for a single project setup, it greatly helps to extend to a multi-project setup later.

The following diagram shows the target architecture for the live-scripting project in the context of the static websites which it produces.

In the purple colored box on the right side, there is the org folder which is the place for the project live-scripting and all other projects on the local machine, which are managed here. Besides live-scripting, there is the project aw-org-html-themes which contains the CSS style sheets. There may be any number of other projects. Tthese projects are local, but may be pushed to a remote git server, e.g. GitHub, which is the case for the live-scripting project. The org folder is the starting point for the generation of the static website. This is done with the script publish.sh which, in turn, uses the configuration stored in publish_project.el. This script implements the production of a static website. In a first step a local website is generated, including all html files, linked images and attachments, a sitemap, and a lunr full text search index. This website is full functional on the local machine. In a second step, the local website is mirrored to Amazon S3 or any other web server. This makes it available on the internet or an intranet.

I would like to create a website from the project that runs on a local web server.
It is a static website that can later be replicated on a web server on the internet.

I want to create a website that spans multiple git projects.
This website will be created and updated with a single command. It uses the recursive feature org org-publish.
It is implemented in publish-project.el in the orgweb definition.

(Don't do it! Read the conclusion)
I want to publish the static website orgweb to github pages.
There are different approaches explained on: https://help.github.com/en/github/working-with-github-pages

Amazon S3 is a storage service that includes basic webserver capabilities. It can host static websites, but doesn't not allow for https and authentication. This is fine for this project.

This includes following steps:

1. Publish to local website.
   
2. Update lunr search index.
   
3. Sync to public website on S3.
   

There is a Github project that indexes HTML pages with lunr and cheerio and makes them searchable with a search page.
Lunr-Index-and-Search-for-Static-Sites This project can be used as reference.
I temporarily clone the project to use some artefacts.

The following diagram helps to explain how everything works together.

I use a lunr working director to generate the search index. Lunr and Cheerio must be installed here.
I use the file build_index.js from the github project, copy it to the working directory and rename it to buid_index_orgweb.js to reflect that it is modified for orgweb. I have to edit it to insert my web root. This javascript file uses cheerio to parse all html files under my web root and creates the index in a file lunr_index.js.
This has to be updates whenever the website is published.

At the web root I use the files runclient.js and search.html which I copy from the github project.
The following script block executes everything.

WORKDIR=~/lunr
WEBROOT=/var/www/html/orgweb 
### create a project directory
mkdir -p $WORKDIR
cd $WORKDIR
ls -la 
mkdir $WEBROOT/lunr
cd $WEBROOT/lunr
## Install npm
sudo apt install npm
## Install lunr
npm update node
npm install lunr
npm install cheerio

### create a temporary working directory
mkdir ~/lunr_work
cd ~/lunr_work
### clone the project with example site
git clone https://github.com/BLE-LTER/Lunr-Index-and-Search-for-Static-Sites.git
cd ~/lunr_work/Lunr-Index-and-Search-for-Static-Sites
### copy relevant files
cp build_index.js $WORKDIR/build_index_orgweb.js
cp example_site/search.css $WEBROOT/
cp example_site/search.html $WEBROOT/
cp example_site/lunrclient.js $WEBROOT/
### clean up  lunr_work
cd
rm -rf ~/lunr_work

### Edit costants in build_index_orgweb.js
cd $WORKDIR
ls -la
### build the index for the example site and copy to webroot
node build_index_orgweb.js
cp lunr_index.js $WEBROOT

### Check index
cd $WEBROOT
ls -la

### Take some artefacts under git control
LUNR_FILES=~/org/live-scripting/lunr_files
mkdir $LUNR_FILES
cp $WORKDIR/build_index_orgweb.js $LUNR_FILES
cp $WEBROOT/lunrclient.js $LUNR_FILES
cp $WEBROOT/search.html $LUNR_FILES

### check and git
cd $LUNR_FILES
ls -la
git status
### use magit to add and commit

Now the search page is available at: http://localhost/orgweb/search.html
And on S3 at: http://live-scripting.s3-website.eu-central-1.amazonaws.com/search.html

The index creation must be part of the publishing process.
I add it to the script publish.sh.

I want to have a central index page for orgweb, named index.html, which is located at the orgweb root.
This page includes the search field and the sitemap. I is created from the file index.org using org-publish.

## copy file to include in git
cp  ~/org/index.org ~/org/live-scripting/lunr_files/
cp  ~/org/index.org.template ~/org/live-scripting/lunr_files/
ls -la ~/org/live-scripting/lunr_files/

## commit and push with magit 

The whole orgfile or parts of it can be exported to PDF. This is done via the LaTeX exporter.
It's been quite a while since I worked with Latex the last time and I am surprised that it is still used and well supported in 2020. The export to Latex and PDF works flawless and the result is very usable. However, if you are not satisfied with the result, it gets complicated. There are endless ways to tailor the export process, but that requires deep knowledge of the exporter backend, Tex and Latex. Some configurations can be made via org mode directives and emacs variables. The org mode documentation describes in detail. It can be found here: https://orgmode.org/manual/LaTeX-Export.html.
If you need more flexibility, this tutorial on wrog is a good starting point to understand and modify the export process: https://orgmode.org/worg/org-tutorials/org-latex-export.html

To export an org mode file to pdf simply use this command.

M-x org-letex-export-to-pdf

The Latex Class can be controlled with org mode directive.
Choose the latex class: article, report, book.
, #+LATEX_CLASS: report

Here is an example PDF export of this live-scripting.org.

file:images/Sharing_Options/2020-08-21_13-07-08_live-scripting.pdf

I get the error message:

LaTeX Error: File `wrapfig.sty' not found.

Analysis:
This problem is discussed at: https://tex.stackexchange.com/questions/291531/exporting-org-files-to-latex-error
Solutions:
I could solve the problem by installing texlive and extra packages.

sudo su 
#password
apt-get update
apt-get install texlive
apt-get install texlive-latex-extra

This installs the versions:

# texlive newest version (2019.20190710-1).
# texlive-latex-extra newest version (2019.20190710-1).

And this solves the problem.

Large images can be resized during inline displays and in export to html and PDF.
We can use modifier attributes for it.
In the HTML page, the resized image should open to it's full size by clicking it. This can be achieved by including a html link as the description part in org-link.
I added the elisp function org-download-link-format-function-aw in the file publish-project.el.
By customizing the variable org-download-link-format-function.
The function is a modified version of the default function with alters the format of the image link.

Here is an example. A large screenshot is included but with specified width values.

#+caption: Screenshot that shows the modifications for clickable image links.
#+Attr_html: :width 200px
#+Attr_org: :width 200px

The resulting small picture in the html file is clickable and opens the picture file in full size.

In order to use this feature, we need to customize the variable as explained above.

It is possible to place images side by side, using an orgmode table.
However orgmode tables don't support multi-line cells. Thus it is not possible to enrich it with caption or attributes.

Apple	Banana

Folder Structures can be created with the unix tree command.
It can be presented in a source block.

lubuntu@lubuntu-pc:/var/www/html/orgweb/live-scripting/images$ tree
.
├── Installation_unter_lubuntu
│   ├── 2020-06-24_11-15-55_orgcard.pdf
│   ├── 2020-06-24_11-21-33_banana.jpeg
│   ├── 2020-06-30_10-25-57_2020-06-21_17-34-52_2020-06-21_17-15-37.png
│   ├── 2020-06-30_10-26-47_2020-06-21_17-34-52_2020-06-21_17-15-37.png
│   ├── 2020-07-03_21-03-36_2020-07-03_21-01-27.png
│   └── 2020-07-07_21-36-29_2020-07-07_21-31-11.png
├── Introduction
│   └── 2020-07-03_22-06-50_2020-07-03_22-05-56.png
└── Test
    ├── 2020-07-08_08-52-47_1565435.png
    ├── 2020-07-08_08-53-30_banana.jpeg
    └── 2020-07-08_08-54-39_banana.jpeg

Orgmode uses the underscore letter to indicate superscript. Most of the time this is not what I want.
The Variable org-use-sub-superscript can be used to customize this behavior.
I choose the option "only with braces" to enable special format when I want it. Unfortunately this setting is ignored during the publishing process. As an alternative I use the directive:

#+OPTIONS: ^:{}
;;possible values are t, nil, {}

This will not be_subscript
This will be_{subscript} 
This will not be^superscript
This will be^{superscript} 

These lines produce the following result:

This will not be_subscript
This will be_subscript
This will not be^superscript
This will be^superscript

Ok.