Observability Project

Latest Progress

• 30-Apr-2024: Configuring cloudlet to use hidden password, start automatically from service daemon, automatically refresh the local clone SplunkDockerCompose repo, and keep service file during redeploys.
• 29-Apr-2024: Setting up Splunk to run in cloudlets. Tried running Splunk image directly, but found that a docker-compose.yml file was a bit easier. At this stage I am able to run the Splunk image. I found that the cloudlets were not executing the docker compose up command, and I created a systemd daemon service for docker compose.
• 24-Apr-2024: Learn about and setting up DNS etc. to link my new domain dirks.nz
• 23-Apr-2024: Setup Node.js container in Cloudlets to fetch and run head from main branch of repo below. If it's running you can access it here
• 22-Apr-2024: Setup basic Express.js Hello World project in new github repo

Old items are moved to Progress

Introduction

This is a personal development project to observe this web site (https://dirksonline.net/cvwiki). I will be striving to monitor reliability, functionality, and importantly who and how it's being used.

Background

Testing strives to find things that might surprise someone who matters. Hence, we need to know the people that matter, and what matters to them. Identifying the individuals (& types of people) that matter to the project and observe what they do for the project. To do our jobs well it is imperative we create good mental models of the people, and it is essential to continue to observe, and update, our models because they form the foundations from which we derive our testing and quality assessments.

Users are almost always at the pinnacle of people that matter, having good models of the types of users, and what they choose to do, is critical to the success of most projects and products.

I want to learn about the people who use my CVWiki web site, so I'm going to develop some observability solutions, and in the process deepen my knowledge and experience in the technologies used in observing web sites.

I am a keen agilist, and will aim to maximize the work not done. Implementing minimalist solutions, that require the least amount of work from me, for my current context. I will favour resources I already have at hand, and the technologies I already know about. But that said, there are some large gaps I need to cover and there will be plenty new to discover and learn, I expect that some decisions may need review down the track as I discover better technologies for my solutions.

Architecture

• Modular microservices style architecture, with the parts of the system running in containers
• Reverse proxy in front of my current CV Wiki and to capture the traffic.
• Log aggregator & event analyser to hold the data and start analysing it

Plan/Backlog

1. Run a basic Express.js demo in a cloudlet node with a recent Node.js image [DONE]
2. Implement access to it from my new `dirks.nz` domain, using encryption (https) [DONE]
3. Run Splunk in cloudlets [In progress]
4. Implement a reverse proxy using library from npm (for traffic to/from above Express.js demo)
5. Create Splunk HEC (Http Event Collector) (check if config persists through redeploys)
6. If needed, investigate data persistence through use of volumes from Docker <=> Cloudlet <=> Backup/Archive
7. Send traffic events to Splunk HEC from Express demo (create indexes, set data retention policies, check data persistence through redeploys)
8. Investigate if reverse proxy can be used for accessing Splunk instance UI, and consider recording splunk UI access logs
9. Try implement auth in reverse proxy to require login before being able to access some URL paths (eg. Splunk UI)
10. Implement MediaWiki in a cloudlet, and copy content from my existing cvwiki
11. Route traffic to/from new cvwiki via reverse proxy, & log the traffic
12. Create an UI Automation suite to test cvwiki using Playwright
13. Run the above suite in cloudlet and record results to splunk
14. Schedule the above cloudlet to run daily

Current Implementation

Express Demo Running in Cloudlets

This step's objectives were to

1. Standup a cloudlet node using an inbuilt Node.JS image
2. Link it to my code repo node.js-in-the-cloud
3. Access it from a subdomain of dirks.nz
4. Implement encryption (https)
5. Redirect calls to http to https, as well as any calls direct to the environment

The solution is accessible at https://nodejs-in-the-cloud.dirks.nz/

My code repo is node.js-in-the-cloud

A Git repo is added to the cloudlets Deployment Manager, deployed it to the Node.js container, with settings to monitor the repo for changes, so that the updates are fetched and container restarted automatically.

Created an nginx node to configure encryption and redirects

Using the cloudlets Let's Encrypt Free SSL addon to generate certs

fetch the changed nginx conf files and add to repo and update readme there to reflect

TBD

Technologies

Hosting Platform (cloudlets)

• vpsblocks.com.au

and the associated PaaS

• cloudlets.com.au

A while back a friend of mine said he was using these folks in Australia and was very happy with them, hence I moved my cpanel based stuff to them quite a while ago, and I've been happy with the service myself too.

For this project I am learning about the cloudlets containerisation service. It would seem that I can create nodes using

1. a Cloudlets certified Node.js container
2. a Docker Image
3. Kubernetes

Data Collection & Analysis Platform (Splunk)

• splunk

I have used this platform for a number of my past employers and projects and really like it. I recognise it isn't a free, nor an open source solution such as Grafana, Prometheus, ELK, Graylog etc. but it's what I know currently and will favour it until I have time and need for a better fitting solution.

I will be using the docker-splunk single containerized instance of Splunk Enterprise, with the free license.

The free license removes the auth parts and gives everyone admin access, I will investigate restricting access to it later.

Language (JavaScript), Runtime (Node.js), Framework (Express)

For now, I have better knowledge, skills, experience using these than others. I recognise that there may well be better solutions out there, but for now I will favour these because it feels more comfortable.

That said though, and given that I'll be using containerized solutions, it should be easy to try other options later.

Code Quality & Inspection Tools

I would like to set these up, but haven't as yet.

Development Environment (VS Code)

• VS Code
• VS Code Extensions: TBD

Progress

Old items from Latest Progress will be moved to here