Observability Project: Difference between revisions
Jump to navigation
Jump to search
m
→Current Implementation
m (→Introduction) |
|||
| (24 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=Introduction= | ==Latest Progress== | ||
* 30-Apr-2024: Configuring cloudlet to use hidden password, start automatically from service daemon, automatically refresh my [https://github.com/VincentDirks/SplunkDockerCompose.git SplunkDockerCompose] repo, and keep service file during redeploys. | |||
* 29-Apr-2024: Setting up Splunk to run in cloudlets. Tried running Splunk image directly, but found that a docker-compose.yml file was a bit easier. At this stage I am able to run the Splunk image. I found that the cloudlets were not executing the docker compose up command, and I created a systemd daemon service for docker compose. | |||
* 24-Apr-2024: Learn about and setting up DNS etc. to link my new domain dirks.nz | |||
* 23-Apr-2024: Setup Node.js container in Cloudlets to fetch and run head from main branch of repo below. If it's running you can access it [https://vid-nodejs-in-the-cloud.mel.cloudlets.com.au/ here] | |||
* 22-Apr-2024: Setup basic Express.js ''Hello World'' project in new [https://github.com/VincentDirks/node.js-in-the-cloud github repo] | |||
Old items are moved to [[#Progress| Progress]] | |||
==Introduction== | |||
This is a personal development project to observe this web site (https://dirksonline.net/cvwiki). I will be striving to monitor reliability, functionality, and importantly who and how it's being used. | This is a personal development project to observe this web site (https://dirksonline.net/cvwiki). I will be striving to monitor reliability, functionality, and importantly who and how it's being used. | ||
| Line 15: | Line 24: | ||
* Reverse proxy in front of my current CV Wiki and to capture the traffic. | * Reverse proxy in front of my current CV Wiki and to capture the traffic. | ||
* Log aggregator & event analyser to hold the data and start analysing it | * Log aggregator & event analyser to hold the data and start analysing it | ||
==Plan/Backlog== | |||
# <s>Run a basic Express.js demo in a cloudlet node with a recent Node.js image</s> [DONE] | |||
# <s>Implement access to it from my new `dirks.nz` domain, using encryption (https)</s> [DONE] | |||
# Run Splunk in cloudlets '''[In progress]''' | |||
# Implement a reverse proxy using library from npm (for traffic to/from above Express.js demo) | |||
# Create Splunk HEC (Http Event Collector) (check if config persists through redeploys) | |||
# If needed, investigate data persistence through use of volumes from Docker <=> Cloudlet <=> Backup/Archive | |||
# Send traffic events to Splunk HEC from Express demo (create indexes, set data retention policies, check data persistence through redeploys) | |||
# Investigate if reverse proxy can be used for accessing Splunk instance UI, and consider recording splunk UI access logs | |||
# Try implement auth in reverse proxy to require login before being able to access some URL paths (eg. Splunk UI) | |||
# Implement MediaWiki in a cloudlet, and copy content from my existing cvwiki | |||
# Route traffic to/from new cvwiki via reverse proxy, & log the traffic | |||
# Create an UI Automation suite to test cvwiki using Playwright | |||
# Run the above suite in cloudlet and record results to splunk | |||
# Schedule the above cloudlet to run daily | |||
=Current Implementation= | |||
==Express Demo Running in Cloudlets== | |||
This step's objectives were to | |||
# Standup a cloudlet node using an inbuilt Node.JS image | |||
# Link it to my code repo [https://github.com/VincentDirks/node.js-in-the-cloud.git node.js-in-the-cloud] | |||
# Access it from a subdomain of dirks.nz | |||
# Implement encryption (https) | |||
# Redirect calls to http to https, as well as any calls direct to the environment | |||
The solution is accessible at https://nodejs-in-the-cloud.dirks.nz/ | |||
My code repo is [https://github.com/VincentDirks/node.js-in-the-cloud.git node.js-in-the-cloud] | |||
A Git repo is added to the cloudlets Deployment Manager, deployed it to the Node.js container, with settings to monitor the repo for changes, so that the updates are fetched and container restarted automatically. | |||
Created an nginx node to configure encryption and redirects | |||
Using the cloudlets Let's Encrypt Free SSL addon to generate certs | |||
fetch the changed nginx conf files and add to repo and update readme there to reflect | |||
TBD | |||
=Technologies= | =Technologies= | ||
==Hosting Platform== | ==Hosting Platform (cloudlets)== | ||
* [https://vpsblocks.com.au/ vpsblocks.com.au] | * [https://vpsblocks.com.au/ vpsblocks.com.au] | ||
and the associated PaaS | and the associated PaaS | ||
| Line 31: | Line 79: | ||
# Kubernetes | # Kubernetes | ||
==Data Platform== | ==Data Collection & Analysis Platform (Splunk)== | ||
* | *[https://www.splunk.com/ splunk] | ||
I have used this platform for a number of my past employers and projects and really like it. I recognise it isn't a free, nor an open source solution such as Grafana, Prometheus, ELK, Graylog etc. but it's what I know currently and will favour it until I have time and need for a better fitting solution. | |||
I will be using the [https://github.com/Splunk/docker-Splunk docker-splunk] single containerized instance of Splunk Enterprise, with the free license. | |||
The free license removes the auth parts and gives everyone admin access, I will investigate restricting access to it later. | |||
==Language (javascript), Runtime (Node.js), Framework (Express)== | |||
For now, I have better knowledge, skills, experience using these than others. I recognise that there may well be better solutions out there, but for now I will favour these because it feels more comfortable. | |||
That said though, and given that I'll be using containerized solutions, it should be easy to try other options later. | |||
===Code Quality & Inspection Tools=== | |||
I would like to set these up, but haven't as yet. | |||
==Development Environment (VS Code)== | |||
* VS Code | |||
* VS Code Extensions: TBD | |||
=Progress= | |||
Old items from [[#Latest Progress| Latest Progress]] will be moved to here | |||