Observability Project
Latest Progress
- 30-Apr-2024: Configuring cloudlet to use hidden password, start automatically from service daemon, automatically refresh the local clone SplunkDockerCompose repo, and keep service file during redeploys.
- 29-Apr-2024: Setting up Splunk to run in cloudlets. Tried running Splunk image directly, but found that a docker-compose.yml file was a bit easier. At this stage I am able to run the Splunk image. I found that the cloudlets were not executing the docker compose up command, and I created a systemd daemon service for docker compose.
- 24-Apr-2024: Learn about and setting up DNS etc. to link my new domain dirks.nz
- 23-Apr-2024: Setup Node.js container in Cloudlets to fetch and run head from main branch of repo below. If it's running you can access it here
- 22-Apr-2024: Setup basic Express.js Hello World project in new github repo
Old items are moved to Progress
Introduction
This is a personal development project to observe this web site (https://dirksonline.net/cvwiki). I will be striving to monitor reliability, functionality, and importantly who and how it's being used.
Background
Testing strives to find things that might surprise someone who matters. Hence, we need to know the people that matter, and what matters to them. Identifying the individuals (& types of people) that matter to the project and observe what they do for the project. To do our jobs well it is imperative we create good mental models of the people, and it is essential to continue to observe, and update, our models because they form the foundations from which we derive our testing and quality assessments.
Users are almost always at the pinnacle of people that matter, having good models of the types of users, and what they choose to do, is critical to the success of most projects and products.
I want to learn about the people who use my CVWiki web site, so I'm going to develop some observability solutions, and in the process deepen my knowledge and experience in the technologies used in observing web sites.
I am a keen agilist, and will aim to maximize the work not done. Implementing minimalist solutions, that require the least amount of work from me, for my current context. I will favour resources I already have at hand, and the technologies I already know about. But that said, there are some large gaps I need to cover and there will be plenty new to discover and learn, I expect that some decisions may need review down the track as I discover better technologies for my solutions.
Architecture
- Modular microservices style architecture, with the parts of the system running in containers
- Reverse proxy in front of my current CV Wiki and to capture the traffic.
- Log aggregator & event analyser to hold the data and start analysing it
Plan/Backlog
Run a basic Express.js demo in a cloudlet node with a recent Node.js image[DONE]Implement access to it from my new `dirks.nz` domain, using encryption (https)[DONE]- Run Splunk in cloudlets [In progress]
- Implement a reverse proxy using library from npm (for traffic to/from above Express.js demo)
- Create Splunk HEC (Http Event Collector) (check if config persists through redeploys)
- If needed, investigate data persistence through use of volumes from Docker <=> Cloudlet <=> Backup/Archive
- Send traffic events to Splunk HEC from Express demo (create indexes, set data retention policies, check data persistence through redeploys)
- Investigate if reverse proxy can be used for accessing Splunk instance UI, and consider recording splunk UI access logs
- Try implement auth in reverse proxy to require login before being able to access some URL paths (eg. Splunk UI)
- Implement MediaWiki in a cloudlet, and copy content from my existing cvwiki
- Route traffic to/from new cvwiki via reverse proxy, & log the traffic
- Create an UI Automation suite to test cvwiki using Playwright
- Run the above suite in cloudlet and record results to splunk
- Schedule the above cloudlet to run daily
Current Implementation
Express Demo Running in Cloudlets
This step's objectives were to
- Standup a cloudlet node using an inbuilt Node.JS image
- Link it to my code repo node.js-in-the-cloud
- Access it from a subdomain of dirks.nz
- Implement encryption (https)
- Redirect calls to http to https, as well as any calls direct to the environment
The solution is accessible at https://nodejs-in-the-cloud.dirks.nz/
My code repo is node.js-in-the-cloud
A Git repo is added to the cloudlets Deployment Manager, deployed it to the Node.js container, with settings to monitor the repo for changes, so that the updates are fetched and container restarted automatically.
Created an nginx node to configure encryption and redirects
Using the cloudlets Let's Encrypt Free SSL addon to generate certs
fetch the changed nginx conf files and add to repo and update readme there to reflect
TBD
Technologies
Hosting Platform (cloudlets)
and the associated PaaS
A while back a friend of mine said he was using these folks in Australia and was very happy with them, hence I moved my cpanel based stuff to them quite a while ago, and I've been happy with the service myself too.
For this project I am learning about the cloudlets containerisation service. It would seem that I can create nodes using
- a Cloudlets certified Node.js container
- a Docker Image
- Kubernetes
Data Collection & Analysis Platform (Splunk)
I have used this platform for a number of my past employers and projects and really like it. I recognise it isn't a free, nor an open source solution such as Grafana, Prometheus, ELK, Graylog etc. but it's what I know currently and will favour it until I have time and need for a better fitting solution.
I will be using the docker-splunk single containerized instance of Splunk Enterprise, with the free license.
The free license removes the auth parts and gives everyone admin access, I will investigate restricting access to it later.
Language (JavaScript), Runtime (Node.js), Framework (Express)
For now, I have better knowledge, skills, experience using these than others. I recognise that there may well be better solutions out there, but for now I will favour these because it feels more comfortable.
That said though, and given that I'll be using containerized solutions, it should be easy to try other options later.
Code Quality & Inspection Tools
I would like to set these up, but haven't as yet.
Development Environment (VS Code)
- VS Code
- VS Code Extensions: TBD
Progress
Old items from Latest Progress will be moved to here