File Structures

Fennec organizes its files in a structured manner first based on the include statement inside .fennec directory. Example, say from the statement:

include:
    - .fennec/core/linux/debian.yml

It means it is looking for this pattern inside the .fennec directory:

core/linux/debian.yml

We will look into each components in details but first, let’s like at the General Structure.

General Structure

.fennec/
├── core
│   └── linux
│       ├── ...
│       └── scripts
│
├── publish
│   └── linux
│       ├── ...
│       └── scripts
│
├── test
│   └── linux
│       ├── ...
│       └── scripts
│
├── package
│   └── linux
│       ├── ...
│       └── scripts
│
├── upstream
│   └── linux
│       ├── ...
│       └── scripts
│
├── tasklets
│   └── linux
│       ├── ...
│       └── scripts
...

The .fennec directory is always arranged in a first 3 level patterns, namely:

1. The Pipeline Types like core, publish, test, etc.
2. The OS Family Types like linux, windows, and macos.
3. The Pipeline individual contents

The #1 and #2 are generalized pattern in which we will be discussing here. We will discuss #3 in their respective sections.

OS Family Types

.fennec/
├── core
│   └── linux

These are the “groups” of operating systems, typically known as “family”. Example:

1. Windows 7, Windows 8, and Windows 10 are all windows family members.
2. Debian, Fedora, Manjaro, CentOS, … are all linux family members.
3. OSX Serria, OSX El Capitan, OSX Mojave, … are macos family members.

There are 2 strong reasons behind grouping these families together:

One is mainly for long-term maintenance purposes. The tools are different and specific for each family members. A good example is BASH scripts would not be fully working in windows family. In another words, these directories holds their OS specific tools and the core scripts as their contents.

Another is mainly to cater for physical machines farm testing. This is commonly seen for embedded devices deployed on physical machines.

OS Family Types are called under their lowercase, no space family name. Example:

1. linux for Linux and its derrivatives
2. windows for Windows
3. macos for Mac OS

They are all directories in nature.

Scripts Directory

.fennec/
├── ANY
│   └── linux
│       └── scripts
│           ├── ...
...         ...

This is the directory holding all modular tool setup scripts. By default, these scripts sense its key environment variables’ values and then acts accordingly. Example, the key variable can holds the tool version number to instruct the setup script to install. Otherwise, the ci recipes calls the execution scripts.

In the event of the missing key variable (user did not configure such installtion), the script must be smart enough to bail out quietly.

These scripts inside this directory are OS specific execution scripts such as:

1. BASH script for Linux (.sh)
2. BAT script for Windows (.bat)

It is NOT advisable to use any language specific scripts (e.g. Rust or Go) inside these execution scripts unless absolute necessary. Doing so introduces additional unwanted dependencies to the operating system just to install a package.

They are named using the numeric system starts from 00 to 99. These are priority numbers, where the lowest number takes lead. This is useful for tools that has dependencies.

In any cases, 00 are reserved for default compulsory setups (like curl) and 99 are reserved for default cleanup tasks.

Always begin your script from 01 onwards to 98.

CPU Architectures

It came to a design decision to keep CPU Architectures information inside the execution scripts itself. There are 3 reasons for it:

1. Keeping the directory clean and simple
2. 32-bit CPU are phasing out from the market, especially for both x86 and arms
3. Keeping the execution scripts modular

That being said, as you develop the core execution scripts, they should have a way to smartly identify the CPU architectures during runtime.

Another alternative is to pass the CPU information in via variables. This is a workaround if the execution scripts does not have any CPU identification and user should only apply such workaround in their .gitlab-ci.yml.

That being said, DO NOT design your execution script to use this workaround. Please spend the effort working out the smart indentification process inside the execution script instead. Therefore, there will not be any directory or file for CPU architectures.

Core Components

.fennec/
├── core
│   └── ...

Core compoments are the main body of the .gitlab-ci.yml. They serve as a foundation or framework to the repository’s .gitlab-ci.yml.

Core main directory is called core and it is a directory in nature.

Core Specifics Contents

.fennec/
├── core
│   └── linux
│       ├── debian.yml

These are the tools ready for consumptions. Within a family specific directory, you can start developing tools and execution scripts for that OS family. In this level, it is safe to write your OS specific core YAML file.

Normally, the core YAML file is called after the OS Specific derrivative. This will make it clear to user when they include the YAML file. Example, for Linux Debian derrivative (regardless CPU version), the statement is:

.fennec/core/linux/debian.yml

Theese are the generalized core setup files that are deployable in both shared runners machines and local machines. If you need to go specific, consider writing a Base Type YAML scripts to work on top of it.

Inside the core YAML file, it contains:

1. The manual mode command to execute the core execution script (see Core Scripts section. ).
2. The pipeline stages (e.g. test -> package -> upstream -> publish)
3. Necessary operational environment variables to execute the CI.
4. Compulsory packages setup (e.g. package bash in linux)
5. Default task executions configured to failed unless user overrides them as acknowledgement.
6. OS images being used.

Friendly Reminder

Avoid writing execution codes inside the core YAML file whenever possible. Keep them inside the core execution script. Remember the semi-automatic design principle.

These core YAML files are, of course, files with standard YAML content format.

Base Types

.fennec/
├── core
│   └── linux
│       ├── base
│       │   ├── go.yml
│       │   └── makefile.yml

These are the execution instruction’s switches or modular customization to the core YAML file. They are meant to tweak the core YAML file to a specific variant using a standardized unified way.

Base should be included after includeing the core.yml script. Following the example above, say we wants go as a base tool, we have:

.fennec/core/linux/debian.yml
.fennec/core/linux/base/go.yml

Base YAML file usually contains the necessary environement variables to trigger the setup execution script to execute a tool install. For those that contains execution codes, they should comply the same rule as OS Family Specific Contents.

Example, for go, it has:

variables:
    GO_VERSION: "1.11.5"

By setting GO_VERSION, debian.yml will setup the go tools during its executions.

The directory is called base while the configuration YAML file is called with the lowercase tool names with spaces replaced with either hyphen (-) or underscore (_), whenever sensible.

The base directory holding the tweaking contents are directory in nature. The configuration files are files with standard YAML format contents.

Core Scripts

.fennec/
├── core
│   └── linux
│       └── scripts
│           ├── installs.d
│           └── setup.sh

Unlike the general defintion of scripts directory, core take things slightly differently. The scripts directory is the single OS specific execution script for setting up tools and tweaking the operating system. They are written in a way where you can perform local execution under a script, complying to the semi-automatic design principle.

This directory should only contains:

• 1 setup.sh or setup.bat depending on OS family
• 1 installs.d or installs_dir directory depending on OS family

The setup.sh or setup.bat script is configured to run all the modular setup scripts inside installs.d OR install_dir directory next to it. It serves as the semi-automatic triggers for user can run the automation with the absent of the GitLab CI framework.

installs.d OR install_dir

.fennec/
├── core
│   └── linux
│       └── scripts
│           ├── installs.d
│           │   ├── 00_curl.sh
│           │   ├── 00_git.sh
│           │   ├── 00_sudo.sh
│           │   ├── 01_go.sh
│           │   ├── 01_hugo.sh
│           │   └── 01_makefile.sh

This is a directory holding all modular tool setup scripts complying to the scripts directory general definition.

The install.d or install_dir are of course, directory in natures. The setup scripts should be the OS specific execution files like BASH or BAT.

Publish

Publish are tasks meant to compile and publish static contents like hosting a static website using Hugo. Unlike Core Components, these are stages’ executions. They are meant to execute a particular job in the CI itself carry out their jobs in a standardized way.

Publish directory is called publish and is a directory in nature.

Publish Specific Contents

.fennec
├── publish
│   ├── linux
│   │   ├── dump
│   │   │   └── debian.yml
│   │   ├── hugo
│   │   │   └── debian.yml
│   │   └── scripts
│   │       ├── 01_dump.sh
│   │       └── 01_hugo_build.sh
│   ├── macos
│   │   └── ...
│   └── windows
│       └── ...
...

Inside the OS family content, there are various technologies made available for generating static website. The first directory should states the technology name being used for generating the contents.

A reserved word dump is used for repository that has the ready made contents. All it needs is to dump into the hosting server.

Inside each technologies directory, it contains the os-specific YAML configuration file for inclusion. This file describes the CI task job for publishing content using the specified technology. Hence, it should be named based on the OS it operates in.

Just like Core Components, There should not be a lot of execution codes inside apart manual execution command. Those execution codes belongs to the execution scripts inside scripts the directory next to it.

The name of these directories are called based on the lowercase technological names with spaces replaced with either hyphen (-) or underscore (_), whenever sensible.

They are of course, directory in nature.

Test

Test holds all the standardized test instructions for a particular technologies. Similar to Publish, it is a task and is entirely optional for you to include the test recipes (although normally we recommend it since it helps standardizing your software).

Test directory is called test and is directory in nature.

Test Specific Contents

.fennec
├── publish
│   ├── linux
│   │   ├── go
│   │   │   ├── all-mod.yml
│   │   │   └── debian-legacy.yml
│   │   └── scripts
│   │       └── go
│   │           ├── 01_all-mod.sh
│   │           └── 01_debian-legacy.sh
│   ├── macos
│   │   └── ...
│   └── windows
│       └── ...

Inside the OS family content, there are various technologies made available for testing. The first directory should states the technology name being tested.

Inside each technologies directory, it contains YAML configuration file for inclusion. This file describes the CI test task job. Hence, it should be named based on the OS it operates in, alongside its test modes. The naming pattern is as such:

[OS TYPE]-[TEST MODE].yml    # space is replaced by dash not underscore

Just like Core Components, There should not be a lot of execution codes inside apart manual execution command. Those execution codes belongs to the execution scripts inside scripts the directory next to it.

The name of these directories are called based on the lowercase technological names with spaces replaced with either hyphen (-) or underscore (_), whenever sensible. They are of course, directory in nature.

As for the script inside the scripts directory, it follows the general definition for scripts directory with 1 additional condition: the directory and file names should be the same as the recipe. Example:

go/all-mod.yml --> go/01_all-mod.sh

Package

planned

Upstream

planned

Tasklet

.fennec/
└── tasklet
    └── renew-ssl
        └── letsencrypt
            └── cloudflare
                ├── debian.yml
                ├── dns.sh
                └── run.sh

Tasklets are individual tasks executed with a schedular like GitLab CI scheduler. As its name implies, they are small little tasks running independently away from the CI jobs.

These tasklets directories are arranged differently from the core and any other jobs directories.

Tasklets still need to setup the OS image in its own term.

The general directory is called tasklet and of course directory in nature.

Task Types

.fennec/
└── tasklet
    └── renew-ssl

The first level is defining its task types. This way, user can tells what kind of task he/she wants to execute. In this example, we have the task to renew ssl certificate.

The name should be lowercase with space replaced by hyphen(-) or underscore(_), whenever sensible.

This task types should be directory in nature.

Task Information

.fennec/
└── tasklet
    └── renew-ssl
        └── letsencrypt
            └── cloudflare

These are related information. They are categories for the task to diversify its capability. Designers can have their freedom to organize it accordingly as long as:

1. Backwards compatible (so think 3x, take your time)
2. Intuitively understandable for user to include
3. Not too long (keep it 3 degrees max like the above)

TIP

Start by thinking how user going to include your tasklet:

.fennec/tasklet/this/is/super/duper/long/noidea.yml

The name should be lowercase with space replaced by hyphen(-) or underscore(_), whenever sensible.

This task types should be directory in nature.

OS Type File and Associated Scripts

.fennec/
└── tasklet
    └── renew-ssl
        └── letsencrypt
            └── cloudflare
                ├── debian.yml
                ├── dns.sh
                └── run.sh

This are the configuration YAML file for include alongside the execution scripts.

Within this YAML file, it calls an exterior script that executes the task to complying to the semi-automatic design principle.

Similar to “Task Information”, Designers can have their freedom to organize it accordingly as they comply to those 3 golden rules.