File Structures
Fennec organizes its files in a structured manner first based on the include
statement inside .fennec
directory. Example, say from the statement:
include:
- .fennec/core/linux/debian.yml
It means it is looking for this pattern inside the .fennec
directory:
core/linux/debian.yml
We will look into each components in details but first, let’s like at the General Structure.
General Structure
.fennec/
├── core
│ └── linux
│ ├── ...
│ └── scripts
│
├── publish
│ └── linux
│ ├── ...
│ └── scripts
│
├── test
│ └── linux
│ ├── ...
│ └── scripts
│
├── package
│ └── linux
│ ├── ...
│ └── scripts
│
├── upstream
│ └── linux
│ ├── ...
│ └── scripts
│
├── tasklets
│ └── linux
│ ├── ...
│ └── scripts
...
The .fennec
directory is always arranged in a first 3 level patterns, namely:
- The Pipeline Types like
core
,publish
,test
, etc. - The OS Family Types like
linux
,windows
, andmacos
. - The Pipeline individual contents
The #1 and #2 are generalized pattern in which we will be discussing here. We will discuss #3 in their respective sections.
OS Family Types
.fennec/
├── core
│ └── linux
These are the “groups” of operating systems, typically known as “family”. Example:
- Windows 7, Windows 8, and Windows 10 are all
windows
family members. - Debian, Fedora, Manjaro, CentOS, … are all
linux
family members. - OSX Serria, OSX El Capitan, OSX Mojave, … are
macos
family members.
There are 2 strong reasons behind grouping these families together:
One is mainly for long-term maintenance purposes. The tools are different and
specific for each family members. A good example is BASH
scripts would not
be fully working in windows
family. In another words, these directories holds
their OS specific tools and the core scripts as their contents.
Another is mainly to cater for physical machines farm testing. This is commonly seen for embedded devices deployed on physical machines.
OS Family Types are called under their lowercase, no space family name. Example:
linux
for Linux and its derrivativeswindows
for Windowsmacos
for Mac OS
They are all directories in nature.
Scripts Directory
.fennec/
├── ANY
│ └── linux
│ └── scripts
│ ├── ...
... ...
This is the directory holding all modular tool setup scripts. By default, these scripts sense its key environment variables’ values and then acts accordingly. Example, the key variable can holds the tool version number to instruct the setup script to install. Otherwise, the ci recipes calls the execution scripts.
In the event of the missing key variable (user did not configure such installtion), the script must be smart enough to bail out quietly.
These scripts inside this directory are OS specific execution scripts such as:
BASH
script for Linux (.sh
)BAT
script for Windows (.bat
)
It is NOT advisable to use any language specific scripts (e.g. Rust or Go) inside these execution scripts unless absolute necessary. Doing so introduces additional unwanted dependencies to the operating system just to install a package.
They are named using the numeric system starts from 00
to 99
. These are
priority numbers, where the lowest number takes lead. This is useful for tools
that has dependencies.
In any cases,
00
are reserved for default compulsory setups (likecurl
) and99
are reserved for default cleanup tasks.Always begin your script from
01
onwards to98
.
CPU Architectures
It came to a design decision to keep CPU Architectures information inside the execution scripts itself. There are 3 reasons for it:
- Keeping the directory clean and simple
- 32-bit CPU are phasing out from the market, especially for both
x86
andarms
- Keeping the execution scripts modular
That being said, as you develop the core execution scripts, they should have a way to smartly identify the CPU architectures during runtime.
Another alternative is to pass the CPU information in via variables. This is a
workaround if the execution scripts does not have any CPU identification and
user should only apply such workaround in their .gitlab-ci.yml
.
That being said, DO NOT design your execution script to use this workaround. Please spend the effort working out the smart indentification process inside the execution script instead. Therefore, there will not be any directory or file for CPU architectures.
Core Components
.fennec/
├── core
│ └── ...
Core compoments are the main body of the .gitlab-ci.yml
. They serve as
a foundation or framework to the repository’s .gitlab-ci.yml
.
Core main directory is called core
and it is a directory in nature.
Core Specifics Contents
.fennec/
├── core
│ └── linux
│ ├── debian.yml
These are the tools ready for consumptions. Within a family specific directory, you can start developing tools and execution scripts for that OS family. In this level, it is safe to write your OS specific core YAML file.
Normally, the core YAML file is called after the OS Specific derrivative. This will make it clear to user when they include the YAML file. Example, for Linux Debian derrivative (regardless CPU version), the statement is:
.fennec/core/linux/debian.yml
Theese are the generalized core setup files that are deployable in both shared runners machines and local machines. If you need to go specific, consider writing a Base Type YAML scripts to work on top of it.
Inside the core YAML file, it contains:
- The manual mode command to execute the core execution script (see Core Scripts section. ).
- The pipeline stages (e.g.
test
->package
->upstream
->publish
) - Necessary operational environment variables to execute the CI.
- Compulsory packages setup (e.g. package
bash
inlinux
) - Default task executions configured to failed unless user overrides them as acknowledgement.
- OS images being used.
Friendly Reminder
Avoid writing execution codes inside the core YAML file whenever possible. Keep them inside the core execution script. Remember the semi-automatic design principle.
These core YAML files are, of course, files with standard YAML content format.
Base Types
.fennec/
├── core
│ └── linux
│ ├── base
│ │ ├── go.yml
│ │ └── makefile.yml
These are the execution instruction’s switches or modular customization to the core YAML file. They are meant to tweak the core YAML file to a specific variant using a standardized unified way.
Base should be included after include
ing the core.yml
script. Following
the example above, say we wants go
as a base tool, we have:
.fennec/core/linux/debian.yml
.fennec/core/linux/base/go.yml
Base YAML file usually contains the necessary environement variables to trigger the setup execution script to execute a tool install. For those that contains execution codes, they should comply the same rule as OS Family Specific Contents.
Example, for go
, it has:
variables:
GO_VERSION: "1.11.5"
By setting GO_VERSION
, debian.yml
will setup the go tools during its
executions.
The directory is called base
while the configuration YAML file is called with
the lowercase
tool names with spaces replaced with either hyphen (-
) or
underscore (_
), whenever sensible.
The base
directory holding the tweaking contents are directory in nature.
The configuration files are files with standard YAML format contents.
Core Scripts
.fennec/
├── core
│ └── linux
│ └── scripts
│ ├── installs.d
│ └── setup.sh
Unlike the general defintion of scripts directory, core
take things slightly differently. The scripts directory is the single OS
specific execution script for setting up tools and tweaking the
operating system. They are written in a way where you can perform local
execution under a script, complying to the semi-automatic design principle.
This directory should only contains:
- 1
setup.sh
orsetup.bat
depending on OS family - 1
installs.d
orinstalls_dir
directory depending on OS family
The setup.sh
or setup.bat
script is configured to run all the modular
setup scripts inside installs.d OR install_dir
directory next to it. It serves as the semi-automatic triggers for user can
run the automation with the absent of the GitLab CI framework.
installs.d OR install_dir
.fennec/
├── core
│ └── linux
│ └── scripts
│ ├── installs.d
│ │ ├── 00_curl.sh
│ │ ├── 00_git.sh
│ │ ├── 00_sudo.sh
│ │ ├── 01_go.sh
│ │ ├── 01_hugo.sh
│ │ └── 01_makefile.sh
This is a directory holding all modular tool setup scripts complying to the scripts directory general definition.
The install.d
or install_dir
are of course, directory in natures. The
setup scripts should be the OS specific execution files like BASH
or BAT
.
Publish
Publish are tasks meant to compile and publish static contents like hosting a static website using Hugo. Unlike Core Components, these are stages’ executions. They are meant to execute a particular job in the CI itself carry out their jobs in a standardized way.
Publish directory is called publish
and is a directory in nature.
Publish Specific Contents
.fennec
├── publish
│ ├── linux
│ │ ├── dump
│ │ │ └── debian.yml
│ │ ├── hugo
│ │ │ └── debian.yml
│ │ └── scripts
│ │ ├── 01_dump.sh
│ │ └── 01_hugo_build.sh
│ ├── macos
│ │ └── ...
│ └── windows
│ └── ...
...
Inside the OS family content, there are various technologies made available for generating static website. The first directory should states the technology name being used for generating the contents.
A reserved word dump
is used for repository that has the ready made
contents. All it needs is to dump into the hosting server.
Inside each technologies directory, it contains the os-specific YAML configuration file for inclusion. This file describes the CI task job for publishing content using the specified technology. Hence, it should be named based on the OS it operates in.
Just like Core Components, There should not be a lot of
execution codes inside apart manual execution command. Those execution
codes belongs to the execution scripts inside scripts
the directory next
to it.
The name of these directories are called based on the lowercase
technological
names with spaces replaced with either hyphen (-
) or underscore (_
),
whenever sensible.
They are of course, directory in nature.
Test
Test holds all the standardized test instructions for a particular technologies. Similar to Publish, it is a task and is entirely optional for you to include the test recipes (although normally we recommend it since it helps standardizing your software).
Test directory is called test
and is directory in nature.
Test Specific Contents
.fennec
├── publish
│ ├── linux
│ │ ├── go
│ │ │ ├── all-mod.yml
│ │ │ └── debian-legacy.yml
│ │ └── scripts
│ │ └── go
│ │ ├── 01_all-mod.sh
│ │ └── 01_debian-legacy.sh
│ ├── macos
│ │ └── ...
│ └── windows
│ └── ...
Inside the OS family content, there are various technologies made available for testing. The first directory should states the technology name being tested.
Inside each technologies directory, it contains YAML configuration file for inclusion. This file describes the CI test task job. Hence, it should be named based on the OS it operates in, alongside its test modes. The naming pattern is as such:
[OS TYPE]-[TEST MODE].yml # space is replaced by dash not underscore
Just like Core Components, There should not be a lot of
execution codes inside apart manual execution command. Those execution
codes belongs to the execution scripts inside scripts
the directory next
to it.
The name of these directories are called based on the lowercase
technological
names with spaces replaced with either hyphen (-
) or underscore (_
),
whenever sensible. They are of course, directory in nature.
As for the script inside the scripts
directory, it follows the general
definition for scripts directory with 1 additional
condition: the directory and file names should be the same as the recipe.
Example:
go/all-mod.yml --> go/01_all-mod.sh
Package
planned
Upstream
planned
Tasklet
.fennec/
└── tasklet
└── renew-ssl
└── letsencrypt
└── cloudflare
├── debian.yml
├── dns.sh
└── run.sh
Tasklets are individual tasks executed with a schedular like GitLab CI scheduler. As its name implies, they are small little tasks running independently away from the CI jobs.
These tasklets directories are arranged differently from the core
and any
other jobs
directories.
Tasklets still need to setup the OS image in its own term.
The general directory is called tasklet
and of course directory in nature.
Task Types
.fennec/
└── tasklet
└── renew-ssl
The first level is defining its task types. This way, user can tells what kind of task he/she wants to execute. In this example, we have the task to renew ssl certificate.
The name should be lowercase
with space replaced by hyphen(-
) or
underscore(_
), whenever sensible.
This task types should be directory in nature.
Task Information
.fennec/
└── tasklet
└── renew-ssl
└── letsencrypt
└── cloudflare
These are related information. They are categories for the task to diversify its capability. Designers can have their freedom to organize it accordingly as long as:
- Backwards compatible (so think 3x, take your time)
- Intuitively understandable for user to
include
- Not too long (keep it 3 degrees max like the above)
TIP
Start by thinking how user going to
include
your tasklet:
.fennec/tasklet/this/is/super/duper/long/noidea.yml
The name should be lowercase
with space replaced by hyphen(-
) or
underscore(_
), whenever sensible.
This task types should be directory in nature.
OS Type File and Associated Scripts
.fennec/
└── tasklet
└── renew-ssl
└── letsencrypt
└── cloudflare
├── debian.yml
├── dns.sh
└── run.sh
This are the configuration YAML file for include
alongside the execution
scripts.
Within this YAML file, it calls an exterior script that executes the task to complying to the semi-automatic design principle.
Similar to “Task Information”, Designers can have their freedom to organize it accordingly as they comply to those 3 golden rules.