# ConSol Monitoring > The ConSol monitoring portfolio ## Releases -------------------------------------------------------------------------------- title: "SNClient 0.44 was released" date: "2026-06-24" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * add semicolon to default nasty characters ### Features * check_tasksched: improve performance * check_drivesize: allow specifying drive with trailing slash ### Changed * check_service: fix typo in output * move count metrics to front * update golang build chain to 1.26.4 ### Bugfixes * fix running external scripts with sudo (#388) * fix rpm installation not creating required snclient user during update (#380) * fix logfile ownership after upgrade (#380) * check_files: fix max-depth was ignored when using forward slashes or double backslashes * rpm: fix setup snclient during update from previous packages ### Download [Read more](https://omd.consol.de/blog/2026/06/24/snclient-0.44-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.43 was released" date: "2026-05-22" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * **DEPRECATION NOTICE** linux agent runs now as unprivileged `snclient` user. Read more [below](#linux-dropped-privileges). ### Features * add /api/v1/admin/log/level rest endpoint to change log level * add /api/v1/admin/log/file rest endpoint to retrieve log file * add check_ssh to test ssh server ### Changed * update node exporter to 1.11.1 * update windows exporter to 0.31.6 * use linux capabilities for check_omd * update golang build chain to 1.26.3 * make listener errors fatal during startup * check_tcp: make hostname mandatory * check_tcp: critical defaults to 10s * check_tcp: add time performance data * check_http: add certificate checks and redirect support ### Bugfixes * check_process: fix command detection when args contain colons ### Download ### Linux Dropped Privileges Since the version v0.43 the default Linux installation will run the snclient as user `snclient` with dropped privileges. However, in order to perform certain checks, the snclient binary will make use of capabilities to start some sub processes as root user. For example the `omd` check requires root permissions. #### Run as root again In case you need to run snclient as root user again, you can change the user name in an systemd override setting: Edit the snclient service by running this command: ```bash sudo systemctl edit snclient.service ``` and add the override: ```text [Service] User=root ``` [Read more](https://omd.consol.de/blog/2026/05/22/snclient-0.43-was-released/) ----------------------------------------------------------------------------------- title: "OMD 5.70 was released" date: "2026-04-16" -------------------------------------------------------------------------------- A new version of OMD was released. ### Changelog * OMD: * add build support for debian 13 / trixie * add build support for ubuntu 26.04 / resolute raccoon * add build support for opensuse leap 16.0 * drop build support for debian 11 * drop build support for ubuntu 20.04 * fix initial redirects when site apache does not use https * Thruk: * Thruk: [3.28](https://github.com/sni/Thruk/blob/master/Changes) * LMD: [2.8.0](https://github.com/sni/lmd/blob/master/Changes) * Naemon: * Naemon: [1.5.1](https://github.com/naemon/naemon-core/blob/master/NEWS) * Livestatus: [1.5.1](https://github.com/naemon/naemon-livestatus/releases) * Vimvault: [1.5.0](https://github.com/naemon/naemon-vimcrypt-vault-broker/blob/master/ChangeLog) * Plugins: * check_logfiles: [4.2](https://github.com/lausser/check_logfiles/blob/master/ChangeLog) * check_nsc_web: [0.7.6](https://github.com/ConSol-Monitoring/check_nsc_web/blob/master/Changes) * check_nwc_health: [12.12.2.3](https://github.com/lausser/check_nwc_health/blob/master/ChangeLog) * check_pdu_health: [3.3](https://github.com/lausser/check_pdu_health/blob/master/ChangeLog) * check_rittal_health: [4.1.2](https://github.com/lausser/check_rittal_health/blob/master/ChangeLog) * check_sap_health: [4.0.0.3](https://github.com/lausser/check_sap_health/blob/master/ChangeLog) * check_storage_health: [2.1](https://github.com/lausser/check_storage_health/blob/master/ChangeLog) * check_ups_health: [4.4](https://github.com/lausser/check_ups_health/blob/master/ChangeLog) * check_vsphere: [0.3.20](https://github.com/ConSol-Monitoring/check_vsphere/blob/main/CHANGES.md) * check_wut_health: [4.4.0.3](https://github.com/lausser/check_wut_health/blob/master/ChangeLog) * omdnotificationforwarder: [4.0](https://github.com/lausser/noteventificationforhandlerwarder/releases) * Gearman: * Mod-gearman: [5.2.3](https://github.com/sni/mod_gearman/blob/master/Changes) * Mod-gearman-worker-go: [1.7.0](https://github.com/ConSol-Monitoring/mod-gearman-worker-go/blob/master/Changes) * Grafana: * Grafana: [12.4.3](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) * Loki: [3.6.7](https://github.com/grafana/loki/blob/main/CHANGELOG.md) * pnp-datasource: [2.2.2](https://github.com/sni/grafana-pnp-datasource/blob/master/CHANGELOG.md) * thruk-datasource: [2.2.3](https://github.com/sni/grafana-thruk-datasource/blob/master/CHANGELOG.md) * Prometheus: * Prometheus: [3.5.2](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) * Alertmanager: [0.30.1](https://github.com/prometheus/alertmanager/blob/main/CHANGELOG.md) * blackbox_exporter: [0.27.0](https://github.com/prometheus/blackbox_exporter/blob/master/CHANGELOG.md) * node_exporter: [1.10.2](https://github.com/prometheus/node_exporter/blob/master/CHANGELOG.md) * pushgateway: [1.11.2](https://github.com/prometheus/pushgateway/blob/master/CHANGELOG.md) * all assets (css/js/templates) for prometheus console were removed from upstream * SNMP console removed as a result * Icinga2: * Icinga2: [2.15.2](https://github.com/Icinga/icinga2/blob/master/CHANGELOG.md) * disable deprecated check result writer * Golang builds updated to [1.26.2](https://go.dev/doc/devel/release) * Coshsh: [11.1.0.1](https://github.com/lausser/coshsh/blob/master/Changelog) * Histou: [0.6.2](https://github.com/ConSol-Monitoring/histou/blob/master/CHANGELOG.md) * Nagflux: [0.5.7](https://github.com/ConSol-Monitoring/nagflux/blob/master/CHANGELOG.md) * remove sakuli helper files * RRDTool: [1.9.0](https://github.com/oetiker/rrdtool-1.x/blob/master/CHANGES) * Telegraf: [1.35.1](https://github.com/influxdata/telegraf/blob/master/CHANGELOG.md) * VictoriaMetrics: [1.138.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) * mod_prometheus_status apache module: [0.3.5](https://github.com/ConSol-Monitoring/apache_mod_prometheus_status/blob/master/Changelog) [Read more](https://omd.consol.de/blog/2026/04/16/omd-5.70-was-released/) ----------------------------------------------------------------------------------- title: "Coshsh 11.0 was released" date: "2025-07-17" -------------------------------------------------------------------------------- A new version of Coshsh was released. ### Breaking Changes * - ### Features * The config can now hide secrets (like a cmdb password) in a vault. * datasources can write to an object's chronicle ### Changed * - ### Bugfixes * - ### Download [Read more](https://omd.consol.de/blog/2025/07/17/coshsh-11.0-was-released/) ----------------------------------------------------------------------------------- title: "Das war der Open-Source-Monitoring-Workshop 2025" date: "2025-07-05" -------------------------------------------------------------------------------- ## Spoiler: Schee war's Das 16. Treffen der deutschsprachigen Open-Source-Monitoring-Community fand am 2. und 3. Juli in München statt. Gastgeber war die [collaboration Factory GmbH](https://www.cplace.com/), die ihren Loft in der neuen Hopfenpost am Hauptbahnhof (sogar mit Tribüne) zur Verfügung stellte. Beziehungen muss man haben... Der Workshop brachte wie immer Monitoring-Admins und Software-Autoren zusammen, die sich in ihren Vorträgen über neue Entwicklungen und Anwendungen in die Karten schauen lassen. In München bieten sich natürlich Biergartenbesuche zum geselligen Austausch an, auch davon wurde mehrmals Gebrauch gemacht. --- ## Tag 1 - 2. Juli - 34,3 °C Nach der Begrüßung der ca. 50 Teilnehmer folgte die Keynote von Marco Pasti von der collaboration Factory GmbH. Er stellte **cplace** vor, das Flaggschiff des Unternehmens, das sich zunehmend auch als Bezeichnung für die gesamte Firma etabliert. Basierend auf einem flexiblen Datenmodell wird cplace von namhaften Konzernen für Projekt-, Portfolio- und Ressourcenplanung eingesetzt. Marco erläuterte, wie cplace durch drei Schichten unterschiedlicher Komplexität an beliebige Szenarien angepasst werden kann: - **No-Code**: Konfigurationseinstellungen und mitgelieferte Widgets ermöglichen einfache Anpassungen ohne Programmierkenntnisse. - **Low-Code**: Ergänzung der Basissoftware mit JavaScript-Funktionen, z.B. für Prozessautomatisierung und Datenimport. - **Pro-Code**: Implementierung gänzlich individueller Features direkt im Java-Code von cplace, z. B. für nicht standardmäßige API-Abfragen und Anbindungen von Fremdsystemen. Die Software kann On-Premise, aber der Trend geht klar zu gehosteten Installationen, denn der Betrieb der Plattform durch die cplace-Kunden selbst bedeutet für diese Aufwand in Form von Zeit und Hardware und geht zu Lasten des Kerngeschäfts. Man merkte, dass bei cplace Technikbegeisterung herrscht, genau wie bei den Workshopteilnehmern. --- **Softwareentwicklung in einem Open-Source-Unternehmen - Blerim Sheqa, NETWAYS** Blerim Sheqa von NETWAYS gab Einblicke in die Softwareentwicklung in einem Open-Source-Unternehmen. NETWAYS lagerte nach dem Fork von Icinga die Entwicklung in ein eigenes Unternehmen aus. Blerim erzählte von der Herausforderung, individuelle Kundenwünsche in Benefits für die Gesamtheit der Anwender umzuwandeln und sich dem Prinzip „wer zahlt schafft an“ zu verweigern, um glaubhaft zu bleiben. Die Balance zwischen Kundenwünschen und Community-Interessen zu halten ist nicht leicht. Blerim unterstrich, was man nicht oft genug betonen kann: Open-Source ist gratis, aber nicht kostenlos. Irgendjemand muss das ja alles entwickeln und supporten und hätte am Monatsende gern ein Gehalt auf dem Konto. --- **Monitoring bei cplace - Sebastian Weitzel, cplace** Sebastian Weitzel, unterstützt von Martin Rauch und Frank Büttner, alle drei von cplace, stellte die Herausforderungen und die Erkenntnisse beim Monitoring der Plattform vor. Der Vortrag zeigte, wie wichtig eine enge Zusammenarbeit zwischen Entwicklung und Betrieb ist. An mehreren Beispielen wurde verdeutlicht, wie das Bereitstellen von Metriken durch die Applikation hilft, die Performance zu verbessern und Fehlern auf die Spur zu kommen. Insbesondere als Anbieter einer gehosteten Applikation as-a-service hilft Monitoring, Probleme bei Softwareupdates zu erkennen, bevor dieses auf alle Kundeninstallationen ausgerollt wird. ![Monitoring cplace](monitoring-cplace.jpg) --- **Mittagessen im Augustiner Biergarten** Das Foto sagt wohl alles. Es war ein ziemlich heißer Sommertag. ![Augustiner Biergarten](mittag-2-7.jpg) --- **Umbrella Monitoring - Jens Michelsons, Allgeier** Jens Michelsons, Produktmanager von openITCOCKPIT, ließ Bastlerherzen höher schlagen. Er monitort sein Balkonkraftwerk wie andere ihr Rechenzentrum, mit detaillierten Metriken und Dashboards. Dazu zapft er die Api seines Home Assistant mit openITCOCKPIT an. Balkone sind weit verbreitet, Balkonkraftwerke mittlerweile auch immer mehr, aber wer Monitoring-Know-How besitzt, kann mehr draus machen. Was dem einen die Geranienkästen, sind dem anderen die Dashboards. Jens hatte am Abend viele neugierige Fragen zu beantworten. --- **Cloud-Monitoring bei einem Elektrokonzern - Ulrike Klusik, ConSol** Ulrike Klusik präsentierte eine beeindruckende Monitoringplattform, die sie bei einem großen deutschen Elektrokonzern aufgebaut hat. Applikationen, die in beliebigen EKS-Clustern deployed werden, können diese ohne großen Aufwand nutzen. Operators erstellen dazu die nötigen Ressourcen in den Kunden-Deployments, um Metriken und Logs zu einem hochverfügbaren Loki- und VictoriaMetrics-Cluster des Monitoring-Teams zu senden. Die Plattform basiert zu 100% auf Open-Source-Komponenten (läuft auf AWS, vermeidet aber dessen teure Builtin-Produkte) und nimmt Kunden das Problem, sich selbst um einen Langzeitspeicher und Guis für ihre Monitoringdaten kümmern zu müssen. Wie immer, wenn Ulrike Einblick in ihre Setups gibt, klappten die Kinnladen runter. ![Ulrike@Siemens](ulrike-siemens.jpg) --- **Grußwort vom CEO - Dr. Rupert Stuffer, cplace** Nach Ulrike Klusiks Vortrag sorgte Dr. Rupert Stuffer, CEO von cplace, für einen herzlichen Moment. Er war in seinem Büro auf die vielen Menschen aufmerksam geworden, die sich quasi in seinem Wohnzimmer breit gemacht hatten. Er ließ es sich nicht nehmen, ein paar wertschätzende Worte an die Teilnehmer zu richten. Er hatte wohl gespürt, daß es sich um Leute handelte, die sich getroffen hatten, um durch offenen Austausch unter Gleichgesinnten neue Ideen zur Verbesserung ihrer Arbeit mitzunehmen. ![Dr. Rupert Stuffer](rupert.jpg) --- **Monitoring bei der Freiwilligen Feuerwehr München - Basti Schubert** Basti Schubert, beruflich Solutions Engineer bei Grafana, engagiert sich seit frühester Jugend bei der Freiwilligen Feuerwehr München, wo er die EDV-Infrastruktur betreut. Sein Vortrag fokussierte auf einfache, nachvollziehbare und robuste Setups, die mit wenig Aufwand wartbar sind. Nachvollziehbar, da all das ehrenamtlich in der knappen Freizeit gemacht wird. Beeindruckend war, wie Basti eine IT-Infrastruktur aufgebaut hat, welches nicht mal Berufsfeuerwehren mit zigfachem Budget hinbekommen. --- **Lightning Talk: Wie man einen Synthetic Test mit dem Robot Framework entwickelt - Simon Meggle, elabit** Simon erzählte die Geschichte von armen Rudi, der für das Beschwerdeformular eines Webauftritts verantwortlich ist, bei dem etwas faul ist. Immer wieder kommt es vor, daß Beschwerden nach dem Absenden im Nirvana verschwinden. Eigentlich sollten sie alle in einer Liste im Admin-Webinterface sichtbar sein. Rudis Chef setzt ihn stark unter Druck, aber Rudi findet Hilfe in Form des Robot Frameworks. Mit diesem kann man auch ohne tiefgreifende Programmierkenntnisse sog. synthetische Tests erstellen. Rudi schreibt einen Test, der einen wütenden Kunden simuliert, welcher eine Beschwerde abschickt und sodann prüft, ob sie in der Admin-GUI angekommen ist. Mit derlei Scripts lassen sich webbasierte Interaktionen rund um die Uhr automatisiert ausführen, um Fehlern auf die Spur zu kommen. --- Am Ende des ersten Tages schlug beim Verlassen der Neuen Hopfenpost erneut die erbarmungslose Sommerhitze zu. Wir marschierten Richtung Löwenbräukeller am Stiglmairplatz, wo das Bier in ausgedörrte Kehlen rann und dabei ein Geräusch verursachte, das so ähnlich wie „coshsh“ klang. Ich kann die Schweinshaxn dort empfehlen. ## Tag 2 - 3. Juli - 23 °C --- **Netdisco und Icinga im Team - Hajo Kessener, Uni Hannover** Hajo Kessener, ein IT-Urgestein von Leibniz Universität Hannover IT Services, präsentierte, wie er die Netzwerklandschaft der Universität mit Netdisco automatisch kartographiert. Hajo pumpt anschließend per Skript diese Daten, aufbereitet als Hosts und Services, zum Icinga-Director, um sein Monitoring stets aktuell zu halten. Sein Ansatz zeigt, wie Open-Source-Tools kombiniert werden können, um komplexe Netzwerke zuverlässig zu überwachen. Netdisco findet alle IP-Adressen, bohrt dann tiefer und kann mit Hilfe der Mac-Adressen sagen, wer mit wem kommuniziert bzw. welches Gerät an welchem Switch/Controller angeschlossen ist. Mit Hilfe solcher Informationen lassen sich sogar vermisste oder auch geklaute Endgeräte aufspüren. Eine weitere Erkenntnis nach Zuruf aus dem Publikum: "Python ist älter als Perl". Ich hatte mich gefreut, dachte "Wenn mir nochmal einer blöd kommt...". Genauere Recherchen belegen aber, daß leider doch Perl die ältere Sprache ist. (Es gibt Spekulationen, daß Python anfangs intern und undokumentiert entwickelt wurde und mindestens so alt ist wie Perl, Belege dafür gibt es aber nicht) --- **Prometheus und SNMP - Gerhard Laußer, ConSol** Ich wollte ursprünglich zeigen, wie Netzwerkgeräte vorab auf bestimmte Mibs/Tables/OIDs gescannt werden, um Module für den snmp_exporter zu erstellen, die nur die nötigsten SNMP-Walks/Gets ausführen. Leider scheiterte die Demo: Der verwendete SNMP-Simulator benötigte 15 Sekunden pro Request und war unbrauchbar. Plan B, ein Setup zu zeigen, bei dem Windows-Eventlogs mit Alloy an einen Loki gesendet werden, scheiterte an fehlendem Nested KVM auf Hetzner-VMs. Plan C, dasselbe mit Linux-Logs zu zeigen, scheiterte am Vortragstag um 02:00 morgens, da der Linux-Server ein Container mit SNClient als Prozess 1 und ohne Journald – und somit ohne Logs – war. (Klar hätte man etwas mit loki.source.file basteln können, aber das hätte insgesamt zu wenig Substanz gehabt) Nachdem die Demo-Pläne nicht funktionierten, entschloss ich mich dazu, das wenige zu zeigen, das entstanden war und ansonsten ohne festgelegtes Ziel und Fokus zu improvisieren. Tatsächlich entstand eine überraschend informative Plauderei mit dem Publikum, unterstützt von Sören und Basti, bei der die Zeit im Nu verging. Ein Appell von Basti, den man im Hinterkopf behalten sollte: "Finger weg von Streaming Telemetry!" --- **Mittagessen im Augustiner Biergarten** Wie schon am Vortag gingen wir wieder zum Augustiner, diesmal bei sehr angenehmen Temperaturen. Angesichts der üppigen Portionen gab es große Augen. ![Augustiner Biergarten](mittag-3-7.jpg) --- **Icinga Notifications - Eric Lippmann, Icinga** Eric Lippmann, CTO von Icinga, stellte „Icinga Notifications“ vor, eine Lösung für die Konfiguration von Benachrichtigungen, die außerhalb des Monitoring-Cores stattfindet. Die Einrichtung von Kontakten, Eskalationen, Bereitschaften und Zeiten ist oft aufwendig, erfordert manchmal Neustarts und ist für Anwender schwer selbstständig umzusetzen. Icinga Notifications bietet eine GUI, in der Benutzer einfach selbst festlegen können, wer, wie, wann und bei welchen Störungen benachrichtigt wird. --- **Lightning Talk: Performance Data Graphs - Thilo Wening, NETWAYS** Zum Abschluss des zweiten Tages präsentierte Thilo Wening eine Alternative zu den eingebetteten Grafana-Panels in Icinga Web, welche sich in einem IFrame befinden müssen und zu denen die Authentifizierung durchgereicht werden muss. Das Web-Modul Performance Data Graphs umgeht diesen Schritt und bietet eine direkte und leichtgewichtige Integration in die Gui. --- **Fazit** Der 16. Open-Source-Monitoring-Workshop 2025 hat viel Spaß gemacht und viel Know How zusammengebracht. Wir sehen uns im nächsten Jahr! [Read more](https://omd.consol.de/blog/2025/07/05/das-war-der-open-source-monitoring-workshop-2025/) ----------------------------------------------------------------------------------- title: "Die Agenda für den Open-Source-Monitoring-Workshop 2025" date: "2025-06-26" -------------------------------------------------------------------------------- ## Vortag - 1.7.25 - **18:00 - 22:00** █ **Vorabendtreffen im Parkcafe** ## Workshop Tag 1 - 2.7.25 - **09:00 - 09:30** █ **Eintreffen/Registrierung** - **09:30 - 10:00** █ **Begrüßung und Keynote** - **10:00 - 10:45** █ **Softwareentwicklung in einem Open-Source-Unternehmen** - Blerim Sheqa, Icinga - **10:45 - 11:00** █ **Pause** - **11:00 - 11:45** █ **Monitoring bei cplace** - Sebastian Weitzel, cplace - **11:45 - 11:50** █ **Ankündigung Mittagessen und Wegbeschreibung** - **11:50 - 13:15** █ **Mittagessen im Augustiner Biergarten** - **13:15 - 14:00** █ **Umbrella Monitoring** - Jens Michelsons, it-novum - **14:00 - 14:45** █ **Cloud-Monitoring bei einem großen Elektrokonzern** - Ulrike Klusik, Consol - **14:45 - 14:55** █ **Pause** - **14:55 - 15:40** █ **Monitoring bei der Freiwilligen Feuerwehr München** - Basti Schubert - **15:40 - 16:00** █ **Lightning Talks...** - **18:00 - 22:00** █ **Löwenbräukeller** ## Workshop Tag 2 - 3.7.25 - **09:00 - 09:15** █ **Eintreffen** - **09:15 - 10:00** █ **Netdisco2 und Icinga im Team** - Hajo Kessener, Leibnitz Universität Hannover - **10:00 - 10:45** █ **Prometheus und SNMP** - Gerhard Laußer, Consol - **10:45 - 11:55** █ **Pause** - **10:55 - 11:40** █ **Icinga Notifications** - Eric Lippmann, Icinga - **11:40 - 13:10** █ **Mittagessen im Augustiner Biergarten** - **13:10 - 14:10** █ **Lightning Talks...** - **14:10 - 00:00** █ **Resümee** [Read more](https://omd.consol.de/blog/2025/06/26/die-agenda-f%C3%BCr-den-open-source-monitoring-workshop-2025/) ----------------------------------------------------------------------------------- title: "Mein Besuch beim openITCOCKPIT Community Meetup 2025 in Fulda" date: "2025-06-06" -------------------------------------------------------------------------------- ### Mein Besuch beim openITCOCKPIT Community Meetup 2025 in Fulda Am 6. Juni 2025 blicke ich zurück auf das [**openITCOCKPIT Community Meetup**](https://it-services.it-novum.com/termine/openitcockpit-community-meetup-2025/) in Fulda, an dem ich auf Einladung von it-novum teilgenommen habe – eine interessante Veranstaltung für Monitoring-Profis. Wir bei ConSol setzen auf unsere Open Monitoring Distribution, aber der Austausch mit der „Konkurrenz“ war spannend, denn ohne Scheuklappen lassen wir uns gegenseitig in die Karten schauen. Hier meine Eindrücke von Dienstagabend und Mittwochvormittag. #### Dienstagabend: Netter Start im Brauhaus Wiesenmühle Am Dienstag, den 3. Juni, trafen wir uns im Brauhaus Wiesenmühle in Fulda. Bei Bier und Essen – in Fulda isst man „Rucksack“, das wusste ich nicht – sprachen wir über Arbeitsalltag, Homeoffice und Projekte und stellten fest: Ob ConSol/OMD, Netways/Icinga, it-novum/openITCOCKPIT oder OpenNMS, unsere Themen ähneln sich, sei es Skalierung, Integration oder Performance. Das Beisammensein war angenehm, der Zusammenhalt der Community spürbar. Danke an it-novum für die Gastfreundschaft! {{< figure src="./wiesenmuehle.jpeg" width="50%" alt="Gravel Calls" class="left" >}} #### Mittwoch Vormittag: Informativer Round Table Der Mittwochvormittag, den 4. Juni, fand in einem beeindruckenden Aufnahmestudio statt, das it-novum mit einem Green Screen ausgestattet hatte – man merkte, dass sie ordentlich technischen Aufwand betreiben, um solche Events professionell zu übertragen. Zuerst präsentierte Jens Michelsons den **Live-Release von openITCOCKPIT V5** mit neuen Features – ein solider Start, und der Chat war aktiv. Danach kam der **Monitoring Round Table**, für mich das Highlight. Da saßen wir: Eric Lippmann, der CTO von Icinga, Ronny Trommer von OpenNMS, ich von ConSol und Markus und Jens von it-novum. Wir sind Konkurrenten, aber keine Rivalen. Wir tauschten uns über Themen wie CMDB, Cloud, Plugins aus und kamen überein, dass Autodiscovery zu Problemen führen kann und eine solide Datenbasis als Grundlage für das Monitoring einen vor unangenehmen Überraschungen schützt. Konfigurationen generieren tun wir alle, jeder hat da sein bevorzugtes Tool, um Datenquellen anzuzapfen. Ein weiterer Punkt, bei dem wir uns einig waren: Cloud-Umgebungen, speziell Kubernetes und Co., in ein Host-Service-Korsett zwingen zu wollen, ist ein Unding. Deshalb stellen wir neben unsere bewährten statusgetriebenen Monitoring-Cores üblicherweise Prometheus, dann wird’s in diesem Umfeld wieder stimmig. Unsere Kundenprojekte ähneln sich, und die Kooperation passt zum Open-Source-Geist. Respekt an Eric, Ronny und die anderen – das war ein guter Austausch unter Leuten, die Monitoring nicht nur als Job sehen, sondern mit Begeisterung und Herzblut bei der Sache sind, angetrieben, unseren Kunden die beste Umgebung hinzustellen. {{< figure src="./roundtable.jpeg" width="50%" alt="Gravel Calls" class="left" >}} Es folgten zwei Vorträge, der API-Deep-Dive mit Markus Stollwerk und die Modulentwicklung mit Jan Gleim. Für jemanden, der das Innenleben von openITCOCKPIT nicht so gut kennt wie ich, war deutlich erkennbar, welcher Entwicklungsaufwand da betrieben wird. #### Mein Fazit Der Dienstagabend im Brauhaus Wiesenmühle war ein netter Auftakt, und der Round Table hat gezeigt, wie fair wir in der Monitoring-Community zusammenarbeiten – trotz unterschiedlicher Firmen stehen wir vor ähnlichen Herausforderungen in unseren Projekten, haben mehr Gemeinsamkeiten bei der Umsetzung und den Projektumgebungen als Unterschiede, die rein technisch durch unsere verschiedenen Monitoring-Plattformen bedingt sind. Wer diese Atmosphäre selbst erleben will, für den haben wir demnächst zwei Tage voller Vorträge und Fachsimpelei: Der **Open-Source-Monitoring-Workshop 2025** wartet auf euch – meldet euch hier an: https://eveeno.com/osmw2025. [Read more](https://omd.consol.de/blog/2025/06/06/mein-besuch-beim-openitcockpit-community-meetup-2025-in-fulda/) ----------------------------------------------------------------------------------- title: "OMD 5.60 was released" date: "2025-05-28" -------------------------------------------------------------------------------- A new version of OMD was released. ### Changelog * OMD: * enable apache systemd service on first installation * removed build support for sles15sp4 * removed build support for sles15sp5 * use upx to reduce binary size * Thruk: * Thruk: [3.22.2](https://github.com/sni/Thruk/blob/master/Changes) * LMD: [2.5.0](https://github.com/sni/lmd/blob/master/Changes) * Naemon: * change relative paths base to site folder * fix timestamps beyond 2038 * Plugins: * check_hpasm: [4.10.1](https://github.com/lausser/check_hpasm/blob/master/ChangeLog) * check_nwc_health: [12.4](https://github.com/lausser/check_nwc_health/blob/master/ChangeLog) * check_pdu_health: [3.0.2.2](https://github.com/lausser/check_pdu_health/blob/master/ChangeLog) * check_printer_health: [1.2.0.2](https://github.com/lausser/check_printer_health/blob/master/ChangeLog) * check_rittal_health: [4.1.1](https://github.com/lausser/check_rittal_health/blob/master/ChangeLog) * check_ups_health: [4.3.2.3](https://github.com/lausser/check_ups_health/blob/master/ChangeLog) * check_vsphere: [0.3.18](https://github.com/ConSol-Monitoring/check_vsphere/blob/main/CHANGES.md) * omdnotificationforwarder: [2.7.1.3](https://github.com/lausser/noteventificationforhandlerwarder/releases) * Gearman: * Mod-gearman: [5.1.8](https://github.com/sni/mod_gearman/blob/master/Changes) * Mod-gearman-worker-go: [1.5.7](https://github.com/ConSol-Monitoring/mod-gearman-worker-go/blob/master/Changes) * Grafana: * Grafana: [11.5.2](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) * pnp-datasource: [2.2.0](https://github.com/sni/grafana-pnp-datasource/blob/master/CHANGELOG.md) * thruk-datasource: [2.2.0](https://github.com/sni/grafana-thruk-datasource/blob/master/CHANGELOG.md) * Prometheus: * Prometheus: [2.53.4](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) * Ansible: * replace shipped ansible with dependency on ansible * snmptrapd: * improve the samplicate_watch units * Golang builds updated to [1.23.7](https://go.dev/doc/devel/release) * Histou: [0.6.1](https://github.com/ConSol-Monitoring/histou/blob/master/CHANGELOG.md) * upx: [4.2.4](https://github.com/upx/upx/releases) * VictoriaMetrics: [1.116.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases) [Read more](https://omd.consol.de/blog/2025/05/28/omd-5.60-was-released/) ----------------------------------------------------------------------------------- title: "Die Anmeldung für den Monitoring-Workshop 2025 ist geöffnet" date: "2025-04-23" -------------------------------------------------------------------------------- Die Registrierungsseite für den Workshop 2025 ist offen: [Open-Source-Monitoring-Workshop 2025](https://eveeno.com/osmw2025). Anmelden! Die ersten Vorträge stehen auch schon fest: * Netdicso und Icinga2 im Team, Discovery und Provisionierung von Netzwerkgeräten im Monitoring - Hajo Kessener, Leibnitz Universität Hannover * Cloud-Monitoring bei einem großen deutschen Elektrokonzern (VictoriaMetrics, Loki, Alloy, kube-prometheus, eks-cluster) - Ulrike Klusik, ConSol * Prometheus und SNMP, MIB Discovery und Generierung von Konfigs für den snmp_exporter - Gerhard Laußer, ConSol * Monitoring in der cplace-Cloud, Zusammenarbeit zwischen Cloud Operations und SW-Entwicklung zur Definition praxisnaher Metriken - Sebastian Weitzel, cplace * Monitoring bei der Freiwilligen Feuerwehr München (Prometheus, Alertmanager und Grafana on Top) - Basti Schubert, Grafana * Softwareentwicklung in einem Open-Source Unternehmen - Blerim Sheqa, Icinga * Umbrella Monitoring – Integration von Systemen mittels APIs am Beispiel von Smart Home - Jens Michelsons, it-novum Weitere Referenten sind willkommen. Schlagt eure Präsentation im Channel **#themen** des [monitoring-portal Discord Servers](https://discord.gg/jDfPZ63FcJ) vor. Dauer eine halbe bis dreiviertel Stunde für reguläre Vorträge, zehn Minuten für Lightning Talks, egal. Hauptsache Monitoring. Erzählt, was ihr so treibt, wie ihr Probleme gelöst habt, woran ihr bastelt. [Read more](https://omd.consol.de/blog/2025/04/23/die-anmeldung-f%C3%BCr-den-monitoring-workshop-2025-ist-ge%C3%B6ffnet/) ----------------------------------------------------------------------------------- title: "SNClient 0.33 was released" date: "2025-04-11" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * check_pdh: added windows performance counter check ### Changed * check_service: fix case insensitive excludes * check_pagefile: fix crash if there is no pagefile * update windows exporter to 0.30.5 * update node exporter to 1.9.1 * check_dns: fix crash without any nameserver * change handling exporter arguments and keep quotes * change handling allias/script arguments and keep quotes * check_os_version: add kernel version/arch/hostname * change minimum golang requirement to 1.23.8 ### Bugfixes * fix timeout race condition when rebooting windows service * fix checking for allowed arguments / nasty characters for web api requests * fix checking for allowed arguments / nasty characters from external scripts (#158) ### Download [Read more](https://omd.consol.de/blog/2025/04/11/snclient-0.33-was-released/) ----------------------------------------------------------------------------------- title: "Lebenszeichen vom Monitoring-Workshop 2025" date: "2025-04-06" -------------------------------------------------------------------------------- Wir sind dieses Jahr spät dran. Bis zuletzt sah es nicht so aus, als würde sich ein Gastgeber bzw. Veranstaltungsort finden, aber dann kam der Zufall ins Spiel. Anwender und Autoren von Open-Source-Monitoring-Software und alle, die von der ehemaligen Anlaufstelle der deutschsprachigen Monitoring-Szene nagios-portal.de übrig geblieben sind, bekommen wieder Gelegenheit zum Austausch. ### Breaking Changes: * Wir ändern nichts, es soll ja so toll bleiben wie immer. ### Features * Vorträge über neue Entwicklungen * Vorstellung bestehender Installationen * Fachsimpeln * Praxis und Hilfestellung * Wirtshausbesuch Für die konkreten Themen und dem Termin bitte dem Download-Link folgen. Dort wird sich nach und nach eine Agenda formen. Und damit was vorangeht, ist jeder zum Mitmachen aufgefordert. Bitte anmelden, sich kurz vorstellen und im Kanal #themen laut rufen "ich möchte einen Vortrag über ... halten!". ### Changed * Veranstaltungsort ist wieder mal München. Der Gastgeber ist die collaboration Factory GmbH (die Fima hinter der Softwareplattform Cplace). Der Workshop wird in der neuen Hopfenpost gleich neben dem Hauptbahnhof stattfinden. Der Workshop geht über zwei Tage und zwar am 2. und 3. Juli 2025. ### Bugfixes * - ### Download ### Past Releases * 2024 - Neckarsulm, Schwarz IT * 2019 - München, ConSol * 2018 - Braunschweig, TU * 2017 - Düsseldorf, ConSol * 2016 - Kiel, Fachhochschule * 2014 - Berlin, CGI * 2013 - Hamburg, Seminarzentrum Südring * 2012 - München, ConSol * 2011 - Hannover, Kassenärztliche Vereinigung Niedersachsen * 2010 - Nürnberg, qSkills * 2009 - Kassel, Signet * 2008 - Oldenburg, BTC AG * 2007 - Karlsruhe, Forschungszentrum * 2006 - Erlangen, RZ der Friedrich-Alexander-Universität * 2005 - Kassel,Verlag Dierichs [Read more](https://omd.consol.de/blog/2025/04/06/lebenszeichen-vom-monitoring-workshop-2025/) ----------------------------------------------------------------------------------- title: "SNClient 0.32 was released" date: "2025-03-13" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * add alpine support * add new slike operator which is case sensitive * add support for str() operator * add 'snclient config check' command ### Changed * change like operator to be case insensitive * check_process: make process filter case insensitive * check_service: make service filter case insensitive * managed exporter: passthrough all urls, not just /metrics ### Bugfixes * fix check_process when empty-state is set to OK ### Download [Read more](https://omd.consol.de/blog/2025/03/13/snclient-0.32-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.31 was released" date: "2025-02-12" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features ### Changed * update windows exporter to 0.30.2 * chore: update dependencies * change minimum golang requirement to 1.23 ### Bugfixes * fix check_files thresholds on total_size ### Download [Read more](https://omd.consol.de/blog/2025/02/12/snclient-0.31-was-released/) ----------------------------------------------------------------------------------- title: "OMD 5.50 was released" date: "2024-12-19" -------------------------------------------------------------------------------- A new version of OMD was released. ### Changelog * OMD: * add -y/--yes option to accept some dialogs automatically * add build support for ubuntu 24.04 * fix bash completion for services * fix dropping root privileges on omd disable command * omd cleanup: add --dry-run / -n option * omd cleanup: add --keep / -k option * omd start -N/--no-verify to set CORE_NOVERIFY=yes * Thruk: * Thruk: [3.20.2](https://github.com/sni/Thruk/blob/master/Changes) * LMD: [2.2.5](https://github.com/sni/lmd/blob/master/Changes) * Naemon: * Naemon: [1.4.3](https://github.com/naemon/naemon-core/blob/master/NEWS) * Livestatus: [1.4.3](https://github.com/naemon/naemon-livestatus/releases) * Plugins: * check_dell_health: [1.1.0.2](https://github.com/lausser/check_dell_health/blob/master/ChangeLog) * check_hpasm: [4.10](https://github.com/lausser/check_hpasm/blob/master/ChangeLog) * check_mssql_health: [2.7.8](https://github.com/lausser/check_mssql_health/blob/master/ChangeLog) * check_nwc_health: [12.0.3.2](https://github.com/lausser/check_nwc_health/blob/master/ChangeLog) * check_pdu_health: [3.0.2.1](https://github.com/lausser/check_pdu_health/blob/master/ChangeLog) * check_printer_health: [1.2.0.1](https://github.com/lausser/check_printer_health/blob/master/ChangeLog) * check_rittal_health: [4.1](https://github.com/lausser/check_rittal_health/blob/master/ChangeLog) * check_ups_health: [4.3.2](https://github.com/lausser/check_ups_health/blob/master/ChangeLog) * check_vsphere: [0.3.11](https://github.com/ConSol-Monitoring/check_vsphere/blob/main/CHANGES.md) * check_wut_health: [4.1.2.2](https://github.com/lausser/check_wut_health/blob/master/ChangeLog) * mod_prometheus_status apache module: [0.3.2](https://github.com/ConSol-Monitoring/apache_mod_prometheus_status/blob/master/Changelog) * omdnotificationforwarder: [2.6.2.1](https://github.com/lausser/noteventificationforhandlerwarder/releases) * Gearman: * Mod-gearman: [5.1.7](https://github.com/sni/mod_gearman/blob/master/Changes) * Mod-gearman-worker-go: [1.5.6](https://github.com/ConSol-Monitoring/mod-gearman-worker-go/blob/master/Changes) * Grafana: * Grafana: [11.2.5](https://github.com/grafana/grafana/blob/main/CHANGELOG.md) * Loki: [3.3.1](https://github.com/grafana/loki/blob/main/CHANGELOG.md) * pnp-datasource: [2.1.0](https://github.com/sni/grafana-pnp-datasource/blob/master/CHANGELOG.md) * thruk-datasource: [2.1.1](https://github.com/sni/grafana-thruk-datasource/blob/master/CHANGELOG.md) * Apache: * add UnsafeAllow3F if required * snmptrapd: * improve the trap_logger * Golang builds updated to [1.22.10](https://go.dev/doc/devel/release) * Coshsh: [10.2.3](https://github.com/lausser/coshsh/blob/master/Changelog) * Histou: [0.5.4](https://github.com/ConSol-Monitoring/histou/blob/master/CHANGELOG.md) * Icinga2: [2.13.10](https://github.com/Icinga/icinga2/blob/master/CHANGELOG.md) * mod_prometheus_status apache module: [0.3.2](https://github.com/ConSol-Monitoring/apache_mod_prometheus_status/blob/master/Changelog) * Monitoring-plugins: [2.4.0](https://github.com/monitoring-plugins/monitoring-plugins/blob/master/NEWS) * Nagflux: [0.5.2](https://github.com/ConSol-Monitoring/nagflux/blob/master/CHANGELOG.md) * Node: 20.6.1 * omdnotificationforwarder: [2.6.1.3](https://github.com/lausser/noteventificationforhandlerwarder/releases) [Read more](https://omd.consol.de/blog/2024/12/19/omd-5.50-was-released/) ----------------------------------------------------------------------------------- title: "SNClient v0.29 was released" date: "2024-11-17" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features ### Changed * show error if shared ports use different ssl configuration ### Bugfixes * fix vbs wrapper in default config (#170) * fix empty performance data when using perf-config units * fix fix process state filter on windows (#169) ### Download [Read more](https://omd.consol.de/blog/2024/11/17/snclient-v0.29-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.28 was released" date: "2024-10-21" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * add check_ping plugin * add exporter to inventory list * add /api/v1/inventory/{module} rest endpoint to get specific inventory * add /api/v1/admin/updates/install endpoint to trigger update ### Changed * update windows exporter to 0.29.1 * wmi: always set en_US language in query (#156) * check_eventlog: fix time offset parsing (#157) * check_files: add version macro support for .exe/.dll files * check_process: check default empty-state to critical ### Bugfixes * fix updates from custom urls * fix agent not starting if exporter_modules folder does not exist * fix missing thresholds when using perf-syntax ### Download [Read more](https://omd.consol.de/blog/2024/10/21/snclient-0.28-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.27 was released" date: "2024-09-02" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * none ### Changed * do not use empty-state if warn/crit conditions contain check on 'count' * check_memory: distinguish between committed and swap memory * check_memory: change default type on linux to physical,swap * update windows exporter to 0.28.1 ### Bugfixes * fix evaluating nested filter conditions * fix check_drivesize missing performance data when using free_bytes threshold (#136) * fix check_http/check_tcp/check_dns help (#135) * fix using condition alias on nested filters * fix check_tasksched memory leak (#122) ### Download [Read more](https://omd.consol.de/blog/2024/09/02/snclient-0.27-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.26 was released" date: "2024-07-17" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * none ### Changed * update node exporter to 1.8.2 * chdir windows service into to shared-path folder * fill unused $ARGx$ macros with empty space (#110) ### Bugfixes * fix check_os_version windows build version (#134) * fix escaping arguments in windows ps1 scripts * fix if/else expression evaluation (#126) ### Download [Read more](https://omd.consol.de/blog/2024/07/17/snclient-0.26-was-released/) ----------------------------------------------------------------------------------- title: "OMD 5.40 was released" date: "2024-07-15" -------------------------------------------------------------------------------- A new version of OMD was released. ### Changelog * OMD: - remove /etc/init.d/omd, all systems use systemd now * Thruk: - Thruk: [3.16](https://github.com/sni/Thruk/blob/master/Changes) - LMD: [2.2.1](https://github.com/sni/lmd/blob/master/Changes) * Naemon: - Naemon: [1.4.2](https://github.com/naemon/naemon-core/blob/master/NEWS) - Livestatus: [1.4.2](https://github.com/naemon/naemon-livestatus/releases/tag/v1.4.2) * Plugins: - check_dell_health: [1.1.0.1](https://github.com/lausser/check_dell_health/blob/master/ChangeLog) - check_hpasm: [4.9.0.1](https://github.com/lausser/check_hpasm/blob/master/ChangeLog) - check_mssql_health: [2.7.7](https://github.com/lausser/check_mssql_health/blob/master/ChangeLog) - check_nsc_web: [0.7.2](https://github.com/ConSol-Monitoring/check_nsc_web/blob/master/Changes) - check_ntp_health: [1.5.0.1](https://github.com/lausser/check_ntp_health/blob/master/ChangeLog) - check_nwc_health: [11.4.0.2](https://github.com/lausser/check_nwc_health/blob/master/ChangeLog) - check_printer_health: [1.2](https://github.com/lausser/check_printer_health/blob/master/ChangeLog) - check_rittal_health: [4.0](https://github.com/lausser/check_rittal_health/blob/master/ChangeLog) - check_ups_health: [4.2.1](https://github.com/lausser/check_ups_health/blob/master/ChangeLog) - check_vsphere: [0.3.10](https://github.com/ConSol-Monitoring/check_vsphere/blob/main/CHANGES.md) - check_wut_health: [3.2.1](https://github.com/lausser/check_wut_health/blob/master/ChangeLog) * Gearman: - Mod-gearman-worker-go: [1.5.1](https://github.com/ConSol-Monitoring/mod-gearman-worker-go/blob/master/Changes) * Grafana: - Grafana: [10.4.2](https://github.com/grafana/grafana/blob/main/CHANGELOG.md#1042-2024-04-10) - thruk-datasource: [2.0.8](https://github.com/sni/grafana-thruk-datasource/blob/master/CHANGELOG.md) * Prometheus: - Prometheus: [2.53.1](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md#2531--2024-07-10) * Victoriametrics: [1.101.0 (LTS)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.1) * Coshsh: [10.2.1.3](https://github.com/lausser/coshsh/blob/master/Changelog) * omdeventhandler: [1.1.0.1](https://github.com/lausser/noteventificationforhandlerwarder/releases/tag/e1.1.0.1) * omdnotificationforwarder: [2.6.1.1](https://github.com/lausser/noteventificationforhandlerwarder/releases/tag/n2.6.1.1) * pnp4nagios: [f4a831e](https://github.com/ConSol-Monitoring/pnp/commits/master/) * Promxy: [0.0.86](https://github.com/jacksontj/promxy/releases/tag/v0.0.86) ### Download [Read more](https://omd.consol.de/blog/2024/07/15/omd-5.40-was-released/) ----------------------------------------------------------------------------------- title: "Das war der Open-Source-Monitoring-Workshop 2024" date: "2024-05-19" -------------------------------------------------------------------------------- Der Open-Source-Monitoring-Workshop, der vergangene Woche in Neckarsulm stattfand, blickt auf eine lange Tradition zurück. Seit seiner Premiere, damals noch als Wochenendtreffen von Nagios-Anwendern, im Jahr 2005 wird er jährlich in einer anderen Stadt veranstaltet. Aufgrund Covid gab es jedoch eine vierjährige Unterbrechung, aber ab diesem Jahr wird der Workshop wieder regelmäßig stattfinden. Gastgeber der diesjährigen Veranstaltung war die Schwarz IT, ein Unternehmen der Schwarz Gruppe, dem Konzern, der hinter den bekannten Einzelhandelsketten Lidl und Kaufland steht. Die Veranstaltung begann mit einer Begrüßung durch Timo Schumacher, der in seiner Präsentation Einblicke in die Schwarz Gruppe bot. Wachstums-, Mitarbeiter- und Geschäftszahlen unterstrichen die immense Bedeutung des Konzerns, der auch weitaus mehr als Supermärkte zu bieten hat. Nach der Einführung folgte Simon Meggle, der das ROBOT Framework vorstellte. Dieses auf den ersten Blick unscheinbare Tool wird von einer beachtlichen Anzahl von Entwicklern getrieben und umfasst unzählige Module zur Testautomatisierung. Simon benutzt es, um Checkmk für End-to-End-Checks fit zu machen. Anschließend führte Frank Aigner von der Schwarz IT in das Thema "End-2-End-Webseiten-Monitoring" mit dem SPACE-Tool ein, das innerhalb der Schwarz Gruppe genutzt wird, um die Performance und Verfügbarkeit von Webseiten zu überwachen. Es basiert auf dem Framework Playwright von Microsoft und wurde mittels Containerisierung in die OMD-Landschaft integriert. Nach einer kurzen Pause berichteten H. Bogner, S. Rauh und O. Gnapp über ihre Reise zum Aufbau eines Cloud-Observability-Services. Die drei gehören zum Geschäftsbereich StackIT, dem Cloud-Angebot von Schwarz. Mittel der Wahl beim Monitoring der Cloud und des physikalischen Unterbaus ist Prometheus. Die darauf folgende Mittagspause verbrachten wir in der tollen Kantine von Lidl Digital. Vielen Dank nochmal dafür, daß wir eingeladen waren. Gerhard Laußer gab einen Einblick in die Komponenten des Konfigurationsgenerators Coshsh. Tobias Kempf erzählte von der Implementierung von OMD Labs als Monitoring-Lösung für die Schwarz Gruppe und zeigte mit aktuellen Zahlen das immense Wachstum der Landschaft. {{< figure src="./osmw2024-schwarz-omd.jpg" width="50%" alt="Gravel Calls" class="left" >}} Sven Nierlein schloss den Tag mit einem Vortrag über Windows-Monitoring mit SNClient ab. Dieses Tool schafft die Securityprobleme mit dem Vorgänger NSClient++ aus der Welt und hat auch in Sachen Features und Performance mehr zu bieten. Der zweite Tag begann mit einer Übersicht über versteckte Kosten im IT-Monitoring. Martin Hirschvogel von Checkmk hatte hierfür das Kleingedruckte in den Preislisten der Cloud-Anbieter unter die Lupe genommen. Ingrida Tamošaitytė-Ehrig und Eduard Schander von Novatec setzten das Programm mit einem Vortrag über Application Performance Monitoring bei der VHV fort. Dieses basiert auf dem Tool InspectIT Ocelot, welches Java-Klassen automatisch instrumentiert. Michael Kraus entwickelt Monitoring bei Witty, einem Hersteller von Schwimmbadtechnik. Er zeigte, wie Wartungsaufwände dank Telemetriedaten reduziert werden können. Anschließend fanden Lightning Talks statt, bei denen mehrere kurze, prägnante Präsentationen zu verschiedenen Monitoring-Themen gehalten wurden. Darunter war Marc Lückert aus der Schweiz, der mit Naemon und Prometheus die IT des Nahrungsmittelverarbeitungsmaschinenherstellers Bühler überwacht. Frank Aigner zeigte noch einmal das E2E-Monitoring der Schwarz Gruppe, speziell die Grafana-Dashboards. Nach der Mittagspause gab Matthias Gallinger einen Überblick über die Verarbeitung und Analyse von Zeitreihenmetriken innerhalb von OMD. PNP, InfluxDB und Victoriametrics sind die Produkte, welche von Naemon und Prometheus mit Meßwerten befüllt und von Grafana wieder ausgelesen werden. Zwei Tage hatten Entwickler und Anwender zum regen Austausch genutzt und das sicher nicht zum letzten Mal. Wer hat Lust, 2025 Gastgeber zu sein? Wir brauchen dazu Räumlichkeiten (wenn möglich nicht in Bayern, wir waren schon zu oft an der Reihe) und eine Kaffeemaschine. Bei der Organisation des Anmeldeprozesses können wir behilflich sein. [Read more](https://omd.consol.de/blog/2024/05/19/das-war-der-open-source-monitoring-workshop-2024/) ----------------------------------------------------------------------------------- title: "SNClient 0.24 was released" date: "2024-05-12" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * add native aarch64 (arm64) packages for windows * add support for extending default filter/warn/crit ### Changed * update node exporter to 1.8.0 * check_service: use cgroup pids if no main pid available ### Bugfixes * fix memory leak in check_service on windows * fix memory leak in check_drivesize on windows * fix missing scripts in inventory ### Download [Read more](https://omd.consol.de/blog/2024/05/12/snclient-0.24-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.20 was released" date: "2024-03-15" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * check_drivesize: does not use parent folder anymore automatically. Use folder=...if required ### Features * add support for perf-syntax * add regexp replacement macro post processor * add if/else conditionals to \*-syntax templates * check_ntp_offset: add support for other languages ### Changed * check_wmi: add performance data * check_wmi: add support for empty-syntax * update check_nsc_web to v0.7.1 * update windows build framework to wix 3.14 * improve wmi stability ### Bugfixes * check_service: fix json error * fix windows ps1 wrapped scripts * fix timeout for checks over nrpe ### Download [Read more](https://omd.consol.de/blog/2024/03/15/snclient-0.20-was-released/) ----------------------------------------------------------------------------------- title: "Die Anmeldung für den Monitoring-Workshop 2024 ist geöffnet" date: "2024-03-07" -------------------------------------------------------------------------------- Es ist soweit! Wir haben jetzt eine Registrierungsseite für den Workshop: [Open-Source-Monitoring-Workshop 2024](https://eveeno.com/292154783). Haut in die Tasten! Mittlerweile sind auch einige tolle Vorträge zusammengekommen: * Telemetriedatenerfassung und Überwachung von Schwimmbädern - Michael Kraus * Application Performance Monitoring - Sascha Brechmann * SNClient - Sven Nierlein * Coshsh - Gerhard Laußer * SPACE - "End-2-End-Webseiten-Monitoring" bei der Schwarz Gruppe - Frank Aigner * STACKIT Argus, wie wir einen Cloud-Monitoring-Service gebaut haben - H. Bogner, S. Rauh, O. Gnapp * Einsatz von OMD Labs als Monitoring Lösung für die Schwarz Gruppe - Tobias Kempf Weitere Referenten sind natürlich willkommen. Schlagt eure Präsentation im Channel **#vorträge** des [monitoring-portal Discord Servers](https://discord.gg/jDfPZ63FcJ) vor. Halbe Stunde, zehn Minuten, egal. Hauptsache Monitoring. [Read more](https://omd.consol.de/blog/2024/03/07/die-anmeldung-f%C3%BCr-den-monitoring-workshop-2024-ist-ge%C3%B6ffnet/) ----------------------------------------------------------------------------------- title: "SNClient 0.19 was released" date: "2024-02-28" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * add check_os_updates ### Changed * write startup errors to default logfile * check_drivesize: add freespace-ignore-reserved option ### Bugfixes * fix wrapped scripts not found * fix check_drivesize using random volume for sub path * fix check_drivesize using root folder when checking non-existing path * fix perf-config when using REST API * fix timeout handling (fixes #96) * fix check_files skipping folder to early * fix check_eventlog using event id ### Download [Read more](https://omd.consol.de/blog/2024/02/28/snclient-0.19-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.17 was released" date: "2024-02-01" -------------------------------------------------------------------------------- A new version of SNClient was released. ### Breaking Changes * none ### Features * support += operator in config files ### Changed * create logfile folder unless it exists * update windows exporter to 0.25.1 ### Bugfixes * fix check_process username being empty sometimes * fix check_cpu_utilization calculation * fix automatic log rotation on windows ### Download [Read more](https://omd.consol.de/blog/2024/02/01/snclient-0.17-was-released/) ----------------------------------------------------------------------------------- title: "OMD 5.30 was released" date: "2024-01-29" -------------------------------------------------------------------------------- A new version of OMD was released. ### Changelog * Thruk: update to [3.12](https://github.com/sni/Thruk/blob/master/Changes) * LMD: update to [2.1.9](https://github.com/sni/lmd/blob/master/Changes) * Monitoring-Plugins: update to [2.3.5](https://github.com/monitoring-plugins/monitoring-plugins/blob/master/NEWS) * Gearman: - Mod-Gearman module: update to [5.1.3](https://github.com/sni/mod_gearman/blob/master/Changes) - Mod-gearman-worker-go: update to [1.3.8](https://github.com/ConSol-Monitoring/mod-gearman-worker-go/blob/master/Changes) * VictoriaMetrics: update to [1.93.1 (LTS)](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.93.1) * Prometheus: - Prometheus: [2.45.2](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md#2450--2023-06-23) - Alertmanager: [0.26.0](https://github.com/prometheus/alertmanager/blob/main/CHANGELOG.md#0260--2023-08-23) - Pushgateway: [1.6.0](https://github.com/prometheus/pushgateway/blob/master/CHANGELOG.md#160--2023-05-25) * Grafana: - Grafana: [10.2.3](https://github.com/grafana/grafana/blob/main/CHANGELOG.md#1023-2023-12-18) - Thruk datasource: update to [2.0.4](https://github.com/sni/grafana-thruk-datasource/blob/master/CHANGELOG.md) - Enable wal for internal sqlite db * Histou: update to [0.5.1](https://github.com/Griesbacher/histou/blob/master/CHANGELOG.md#v051---02102023) * Go builds: update to [1.21.6](https://tip.golang.org/doc/go1.21) ### Download [Read more](https://omd.consol.de/blog/2024/01/29/omd-5.30-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.16 was released" date: "2024-01-10" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes * none ### Features * add check_ntp_offset * add check_mailq * add check_connections * add check_process cpu metrics * add reload support for systemd service * add metrics to check_os_version * add metrics to check_snclient_version ### Changed * improve check_process output * improve check_service linux performance * use state from metric checks for final result * check_service: adjust default warn/crit condition if service argument is used (#72) ### Bugfixes * fix using arguments in aliased checks (#69) * fix metrics order when using check_nsc_web * fix wrong path in logrotation on linux * fix check_drivesize using percent thresholds ### Download [Read more](https://omd.consol.de/blog/2024/01/10/snclient-0.16-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.15 was released" date: "2023-12-16" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes * none ### Features ### Changed * improve counter memory consumption * change internal wmi library ### Bugfixes * fix check_temperature default thresholds * fix check_kernel_stats listing processes and threads * fix wmi based checks memory usage * fix check_eventlog memory usage ### Download [Read more](https://omd.consol.de/blog/2023/12/16/snclient-0.15-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.14 was released" date: "2023-12-12" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes * none ### Features * add cli inventory mode * add duration / age / date / human number macro post processors * add support for %{} macros and $() as well * add timeout to check_drivesize for hanging mounts * add check_cpu_utilization * add check_mount * add check_temperature * add check_kernel_stats * add fmt macro operator to formt numbers * add automatic retries to wmi querys * add /api/v1/admin/reload rest endpoint to initialize config changes * add /api/v1/admin/certs/replace rest endpoint to replace certificates ### Changed * check_service (linux): add rss/vms/cpu/task metric and remove generic mem metric * check_service: add more metrics * support different allowed hosts settings for each web handler ### Bugfixes * fix cpu check on osx not returning metrics ### Download [Read more](https://omd.consol.de/blog/2023/12/12/snclient-0.14-was-released/) ----------------------------------------------------------------------------------- title: "Monitoring-Workshop 2024 ist hiermit angekündigt" date: "2023-12-05" -------------------------------------------------------------------------------- So, Herrschaften, es geht wieder los! Die Tradition des jährlichen Monitoring-Workshops der deutschsprachigen Community (*ja, ja, ich weiß, der Satz ist feinstes Denglish*) wird wiederbelebt. 2019 fand das letzte Treffen statt und seitdem hat sich eine Menge getan. Wer Monitoring-Software schreibt ist hier genauso richtig wie jemand, der sich Rat zum Betrieb seiner Landschaft holen will. ### Breaking Changes: * Nix, es wird so toll wie eh und je. ### Features * Vorträge über neue Entwicklungen * Vorstellung bestehender Installationen * Fachsimpeln * Praxis und Hilfestellung * Wirtshausbesuch Für die konkreten Themen und dem Termin bitte dem Download-Link folgen. Dort wird sich nach und nach eine Agenda formen. Und damit was vorangeht, ist jeder zum Mitmachen aufgefordert. Bitte anmelden, sich kurz vorstellen und im Kanal #themen laut rufen "ich möchte einen Vortrag über ... halten!". ### Changed * Veranstaltungsort ist diesmal Neckarsulm (An der Stadtgrenze zu Heilbronn). Wir sind zu Gast bei der Schwarz IT (Die IT hinter Lidl und Kaufland). Der genaue Ort ist bei [Lidl Digital in der Rötelstraße 36](https://maps.app.goo.gl/NN1Uq1cgL3bxjqug9). Der Workshop geht über zwei Tage und zwar am 16. und 17. Mai 2024. ### Bugfixes * - ### Download ### Past Releases * 2019 - München, ConSol * 2018 - Braunschweig, TU * 2017 - Düsseldorf, ConSol * 2016 - Kiel, Fachhochschule * 2014 - Berlin, CGI * 2013 - Hamburg, Seminarzentrum Südring * 2012 - München, ConSol * 2011 - Hannover, Kassenärztliche Vereinigung Niedersachsen * 2010 - Nürnberg, qSkills * 2009 - Kassel, Signet * 2008 - Oldenburg, BTC AG * 2007 - Karlsruhe, Forschungszentrum * 2006 - Erlangen, RZ der Friedrich-Alexander-Universität * 2005 - Kassel,Verlag Dierichs [Read more](https://omd.consol.de/blog/2023/12/05/monitoring-workshop-2024-ist-hiermit-angek%C3%BCndigt/) ----------------------------------------------------------------------------------- title: "check_nwc_health 11.2.2 was released" date: "2023-11-30" -------------------------------------------------------------------------------- A new version of check_nwc_health was released. ### Breaking Changes: * - ### Features * The \-\-mode interface-errdisabled will show you if an interface was forcibly disabled (Cisco and Arista only) Errdisable is a feature of most switches running IOS. When a port is in err-disabled state, it is shut down and traffic can no longer pass thru. Reasons for this are numerous, it will be shown in the plugin output. The same feature was also implemented by Arista. You might find it under different names like "ErrDisabled", "Error Disabled", "Errdisable State" and so on... ``` ... --mode interface-errdisabled CRITICAL - GigabitEthernet5/0/28 (alias IP Phone)/vlan 0 is disabled, reason: stormControl, GigabitEthernet4/0/45 (alias User Port)/vlan 0 is disabled, reason: bpduGuard ... --mode interface-errdisabled --name GigabitEthernet5/0/28 CRITICAL - GigabitEthernet5/0/28 (alias IP Phone)/vlan 0 is disabled, reason: stormControl ``` ### Changed * - ### Bugfixes * - ### Download [Read more](https://omd.consol.de/blog/2023/11/30/check_nwc_health-11.2.2-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.13 was released" date: "2023-11-23" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes: * - ### Features * Add node exporter to osx packages * Add check_load plugin * Add folder size support to check_files (#38) * Add basic promethes metrics * check_eventlog: finish implementing ### Changed * Drop privileges when running node exporter * Update node exporter to 1.7.0 * Improve docs and help pages ### Bugfixes * Fix reading empty config files (#58) * Fix packaging for debian <= 12 ### Download [Read more](https://omd.consol.de/blog/2023/11/23/snclient-0.13-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.12 was released" date: "2023-11-07" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes: * - ### Features * Add built-in check_http / check_tcp / check_dns * Add check_omd check ### Changed * - ### Bugfixes * MSI windows installer: - do not change windows service during updates ### Download [Read more](https://omd.consol.de/blog/2023/11/07/snclient-0.12-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.11 was released" date: "2023-10-27" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes: * - ### Features * Add (prometheus) node/windows exporter and custom exporters * Support filter regex syntax: attr ~~ /value/ ### Changed * Return error if using unknown check arguments ### Bugfixes * Fix check_drivesize showing wrong disk on linux * Fix windows installer removing snclient.ini on update ### Download [Read more](https://omd.consol.de/blog/2023/10/27/snclient-0.11-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.10 was released" date: "2023-10-22" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes: * Windows: add firewall on program, not ports ### Features * Add prometheus exporter exporter listener * Add support for shared listeners * Add on demand macros ### Changed * - ### Bugfixes * - ### Download [Read more](https://omd.consol.de/blog/2023/10/22/snclient-0.10-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.09 was released" date: "2023-10-07" -------------------------------------------------------------------------------- A new release of SNClient was released. ### Breaking Changes: * - ### Features * check_network: add name/exclude and speed ### Changed * Use shared_path as working directory when running * check_process: fix showing all processes * check_drivesize: do not apply default filter if disk is specified ### Bugfixes ### Download [Read more](https://omd.consol.de/blog/2023/10/07/snclient-0.09-was-released/) ----------------------------------------------------------------------------------- title: "check_nwc_health 11.2 was released" date: "2023-09-15" -------------------------------------------------------------------------------- A new version of check_nwc_health was released. ### Breaking Changes: * - ### Features * The \-\-mode interface-status will now output the configured VLANs on an interface. ``` ... --mode interface-status --name GigabitEthernet0/0/32 OK - GigabitEthernet0/0/32 (alias Uplink S+, vlan(s): vlan209) is up/up ``` ### Changed * The mode hardware-health for Huawei devices caches the contents of the hwEntityTable for an hour and only requests a few rows from the hwEntityStateTable, so that the runtime is significantly reduced. (Ths also avoids hitting a rate limit) ### Bugfixes * - ### Download [Read more](https://omd.consol.de/blog/2023/09/15/check_nwc_health-11.2-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.08 was released" date: "2023-09-07" -------------------------------------------------------------------------------- A new prerelease of SNClient was released. ### Breaking Changes: * - ### Features * Powershell and BAT scripts no longer need a wrapping. ### Changed * - ### Bugfixes * Some common cmd.exe errors are caught. * Fix restart after an update from the cli. * Installation creates a logfile folder. ### Download [Read more](https://omd.consol.de/blog/2023/09/07/snclient-0.08-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.06 was released" date: "2023-07-27" -------------------------------------------------------------------------------- A new prerelease of SNClient was released. ### Breaking Changes: * - ### Features * relative include paths are now considered to be in the parent ini-file ### Changed * - ### Bugfixes * wmi: improve error handling ### Download [Read more](https://omd.consol.de/blog/2023/07/27/snclient-0.06-was-released/) ----------------------------------------------------------------------------------- title: "SNClient 0.05 was released" date: "2023-07-21" -------------------------------------------------------------------------------- A new prerelease of SNClient was released. You can now install the MSI, put an additional ini-file with your customizations in the install directory and therefore don't have to modify anything in the download package. Any local settings are made in separate from it. ### Breaking Changes: * - ### Features * add configuration dialog to .msi installer * setup windows firewall rules during installation * add wildcard support for includes * add conf.d directory support for includes * added check_tasksched * added nrpe mode to test cmd ### Changed * - ### Bugfixes * fix osx installer (#27) * fix linux check_drivesize (#14) * fix missing byte in nrpe response (#21) ### Download [Read more](https://omd.consol.de/blog/2023/07/21/snclient-0.05-was-released/) ----------------------------------------------------------------------------------- title: "Naemon 1.4.1 is here" date: "2023-02-02" -------------------------------------------------------------------------------- [statt oben als description: ist die Kurzbeschreibung besser hier aufgehoben, denn sonst wird das formatierte Zeugs da unten in unformatierter Form als Vorschau angezeigt. Der Inhalt von description: wird erst angezeigt, wenn man den Post anklickt]:: Today we released the version 1.4.1 of naemon-core and naemon-livestatus. This release includes bugfixes only and fixes a memory leak along with some packaging cleanup. ### Features * None ### Changed * Use /run/naemon instead of /var/run/naemon ### Bugfixes * Fix bashisms in scripts * Fix fg: no job control output * Fix memory leak in cmd parser * Fix closing worker filehandles ### Livestatus * Fix service parents to be always a list of host,service tupels * Fix g_tree_foreach: assertion `tree != NULL’ in logs table [Read more](https://omd.consol.de/blog/2023/02/02/naemon-1.4.1-is-here/) ----------------------------------------------------------------------------------- title: "Naemon 1.4.0 is here" date: "2022-11-11" -------------------------------------------------------------------------------- Today we released the version 1.4.0 of naemon-core and naemon-livestatus. This release includes a new internal command to expire acknowledgements, and a few bugfixes. Due to object structure changes, NEB modules should be re-built when upgrading to 1.4.0 ### Breaking Changes: * Due to object structure changes, it is recommended to recompile any NEB mobules when upgrading to 1.4.0 ### Features * Support for expiring acknowledgments ### Changed * Improvements to Debian packaging, including multiarch installs ### Bugfixes * Fix a crash for external command: CHANGE_CONTACT_HOST_NOTIFICATION_TIMEPERIOD * Fix a crash for external command: CHANGE_HOST_CHECK_TIMEPERIOD ### Livestatus * Allow multiline output when submitting passive checks by allowing escaped newlines [Read more](https://omd.consol.de/blog/2022/11/11/naemon-1.4.0-is-here/) --- ## Insights -------------------------------------------------------------------------------- title: "Labs Repository GPG Key Rotation" date: "2026-04-13" -------------------------------------------------------------------------------- Starting on April 13 2026 all .deb and .rpm files in the [labs repository](/repo) will use the `monitoring-repo-consol-de-gpg-2026.asc` instead of the old `GPG-KEY-4096` or `RPM-GPG-KEY` keys. For consistency all existing rpm and deb files have been resigned to use the new key as well. | | File | Size | ID | |----------|-------------------------------------------------------------------------------------------------------------------- |------|------------------| | **Old** | [RPM-GPG-KEY](https://labs.consol.de/repo/stable/RPM-GPG-KEY) | 1024 | F8C1CA08A57B9ED7 | | **Old** | [GPG-KEY-4096](https://labs.consol.de/repo/stable/GPG-KEY-4096) | 4096 | F0CA212FF1FFE778 | | **New** | [monitoring-repo-consol-de-gpg-2026.asc](https://labs.consol.de/repo/stable/monitoring-repo-consol-de-gpg-2026.asc) | 4096 | CBB9B38BE1B9D330 | ## Debian / Ubuntu In Debian and Ubuntu you will probably notice warnings like this when running `apt update` ```txt #>apt update Hit:1 http://labs.consol.de/repo/stable/debian trixie InRelease Err:1 http://labs.consol.de/repo/stable/debian trixie InRelease Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key E1C8FD55CA5EEEFF05E93DD3CBB9B38BE1B9D330, which is needed to verify signature. Fetched 32.5 kB in 0s (107 kB/s) All packages are up to date. Warning: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://labs.consol.de/repo/stable/debian trixie InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key E1C8FD55CA5EEEFF05E93DD3CBB9B38BE1B9D330, which is needed to verify signature. Warning: Failed to fetch http://labs.consol.de/repo/stable/debian/dists/trixie/InRelease Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key E1C8FD55CA5EEEFF05E93DD3CBB9B38BE1B9D330, which is needed to verify signature. Warning: Some index files failed to download. They have been ignored, or old ones used instead. ``` To make it work again, simple replace the key with the new one. Either follow the instructions from the repository installation again: - [stable repository](https://labs.consol.de/repo/stable/) - [stream repository](https://labs.consol.de/repo/stream/) - [testing repository](https://labs.consol.de/repo/testing/) or use these commands: ```bash curl -fsS "https://labs.consol.de/repo/testing/monitoring-repo-consol-de-gpg-2026.asc" -o /etc/apt/trusted.gpg.d/monitoring-repo-consol-de-gpg-2026.asc ``` Then make sure the key is used in the sources file: ex.: `/etc/apt/sources.list.d/labs-consol-stable.list` ```txt deb [signed-by=/etc/apt/trusted.gpg.d/monitoring-repo-consol-de-gpg-2026.asc] http://labs.consol.de/repo/stable/debian trixie main ``` The important part here is, the `signed-by` option must point to the new key file. ## RHEL / Rocky Linux / Alma On RHEL (and compatible) systems you will get an error like this: ```txt GPG key at https://labs.consol.de/repo/stable/GPG-KEY-4096 (0xF1FFE778) is already installed The GPG keys listed for the "labs_consol_stable" repository are already installed but they are not correct for this package. Check that the correct key URLs are configured for this repository.. Failing package is: omd-5.60-labs-edition-el9-1.x86_64 GPG Keys are configured as: https://labs.consol.de/repo/stable/GPG-KEY-4096 The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'yum clean packages'. Error: GPG check FAILED ``` You can simply update the repository package, for example for rhel9 with: ```bash rpm -Uvh "https://labs.consol.de/repo/stable/rhel9/x86_64/labs-consol-stable.rhel9.noarch.rpm" ``` ## OpenSuse ```txt Looking for gpg keys in repository consol_labs_stable. gpgkey=https://labs.consol.de/repo/stable/GPG-KEY-4096 New repository or package signing key received: Repository: consol_labs_stable Key Fingerprint: E1C8 FD55 CA5E EEFF 05E9 3DD3 CBB9 B38B E1B9 D330 Key Name: ConSol Monitoring Team Key Algorithm: RSA 4096 Key Created: Wed 18 Mar 2026 03:34:45 PM CET Key Expires: (does not expire) Rpm Name: gpg-pubkey-e1b9d330-69bab805 ... Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): ``` You can either trust the new key by `a` or update the repository: ```bash zypper removerepo consol_labs_stable zypper addrepo -f https://labs.consol.de/repo/stable/sles15sp6/consol-labs.repo ``` ## Alpine The alpine repository is not affected by this change. [Read more](https://omd.consol.de/blog/2026/04/13/labs-repository-gpg-key-rotation/) ----------------------------------------------------------------------------------- title: "Monitoring vSphere cluster health with check_vsphere" date: "2026-04-01" -------------------------------------------------------------------------------- ## What's new? The [`cluster-health`](/docs/plugins/check_vsphere/cmd/cluster-health/) command in **[check_vsphere](/docs/plugins/check_vsphere/)** looks at the members of a vSphere cluster, checks their state and decides whether the whole cluster is healthy. By default it treats nodes that are *disconnected* or *in maintenance* as faulty, but you can tweak that list. Use `--faulty` to customize what counts as a failure. ## How the threshold works You tell the command when to raise a warning or a critical alert with the `--cluster-threshold` flag: ``` [max_members:]warn_threshold:crit_threshold ``` * `max_members` (optional) - Apply the rule to clusters with up to this many members. * `warn_threshold` – Number or percent of faulty nodes that triggers a **WARN**. * `crit_threshold` – Number or percent that triggers a **CRIT**. You can give several `--cluster-threshold` flags for different cluster sizes. Rules apply to clusters up to their `max_members`; if multiple rules match, the smallest `max_members` wins. One rule must omit `max_members`; that one is the fallback. ## Quick examples * `3:1:1` - For clusters up to 3 nodes: a single fault triggers a critical state (warning and critical equal). * `5:1:3` - For clusters up to 5 nodes: warn at >=1 faulty node, critical at >=3. * `10:2:5` - For clusters up to 10 nodes: warn at 2 faulty nodes, critical at 5 * `50:5:15` - For clusters up to 50 nodes: warn at 5 faulty nodes, critical at 15. * `10%:20%` - Fallback for larger clusters: warning at 10% failures, critical at 20%. ## Usage snippet ```bash check_vsphere cluster-health \ --host vcenter.example.com \ -u naemon@vsphere.local \ --cluster-threshold 3:1:1 \ --cluster-threshold 5:1:3 \ --cluster-threshold 10:2:5 \ --cluster-threshold 50:5:15 \ --cluster-threshold '10%:20%' \ --cluster-name MyCluster ``` ## Naemon integration ``` define command{ command_name check_vsphere_cluster_health command_line VSPHERE_PASS=$ARG4$ $USER2$/check_vsphere cluster-health \ -u $ARG3$ \ --host $ARG1$ \ --cluster-name $ARG2$ \ --cluster-threshold 3:1:1 \ --cluster-threshold 5:1:3 \ --cluster-threshold 10:2:5 \ --cluster-threshold 50:5:15 \ --cluster-threshold '10%:20%' } define service{ use generic-service host_name vcenter.example.com service_description vSphere Cluster Health check_command check_vsphere_cluster_health!vcenter.example.com!MyCluster!user!pw } ``` [Read more](https://omd.consol.de/blog/2026/04/01/monitoring-vsphere-cluster-health-with-check_vsphere/) ----------------------------------------------------------------------------------- title: "Reliable RKE2 Certificate Expiry Alerts for Clusters Managed by SUSE Rancher" date: "2026-03-24" -------------------------------------------------------------------------------- This shows how to combine Kubernetes Events and Rancher Metadata for precise, actionable alerting. --- ## Alert ManagedClusterRKE2ExpiringCertificates SUSE Rancher provides an RKE2 cluster management platform for application teams. As an infrastructure team operating a SUSE Rancher instance, we want to notify teams about expiring RKE2 certificates, as their rotation requires manual intervention. Unfortunately, the information about expiring certificates is only available in the Kubernetes Events of the Rancher-managed RKE2 clusters. Hence, we need to automatically install Kubernetes Event log collection via Fleet on the managed clusters. For notification purposes, we need additional information about the cluster owner. This cannot be easily propagated down to managed clusters, so the event logs contain no reference to the Rancher project they belong to nor to the owning application. Therefore, we need to enrich the alert with the **application** label that we added to the Rancher **Project** object. This enables routing of the alert to the correct team via Alertmanager. --- ## Setting the Stage: Rancher, Projects and Managed Clusters 1. **Rancher deployment** – A single RKE2 cluster hosts the Rancher application. 2. **Rancher Project management** – The infrastructure team creates *Rancher Projects* (e.g., `proj1`, `proj2`), grants permission to the Application team and adds a custom label `application=` that denotes the owning product. 3. **Managed Cluster** – Application teams operate their own RKE2 clusters **inside** those projects. These clusters are managed by Rancher. The monitoring stack (Loki + Prometheus) runs in the Rancher host cluster, while the managed clusters need to forward their Kubernetes events as logs to the central Loki instance. These logs must include the cluster name. --- ### Deploy Loki in the Rancher Cluster Loki can be easily deployed via Helm chart. For the recording rules, we need additional **Ruler configuration** - configuration of rules via ConfigMap **Remote write** – Configure Loki to forward its streams to the Prometheus installed via Helm rancher-monitoring in RKE2 clusters This is achieved by the following values. ```yaml # values.yaml snippet for Loki Helm loki: # requirement: # storage.bucketNames.rules is not set! # Remote write configuration config: ... # Remote write to Prometheus remote_write: - url: http://prometheus-operated.cattle-monitoring-system.svc:9090/api/v1/write timeout: 30s # configuration of rules taken from configmaps: rulerConfig: storage: type: local local: directory: /rules rule_path: /var/loki/rules-temp # configuration of rules taken from configmaps supporting different tenants: sidecar: rules: folderAnnotation: tenant ``` ### Deploy the Loki recording rule The following ConfigMap specifies the recording rules for logs from tenant "managed-cluster", where the Kubernetes events should be written to. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: loki-recording-rules namespace: monitoring annotations: tenant: managed-cluster labels: # marks config map to be used by the ruler: loki_rule: "true" data: rke2-recording-rules.yaml: | groups: - name: rke2-recording-rules interval: 10m rules: - record: loki:recorded:rke2_cert_expiration_timestamp expr: | max by (rancher_cluster,managed_cluster,cert) (max_over_time( {source="kubernetes_events"} | json | reason = "CertificateExpirationWarning" | line_format "{{ .msg }}" | regexp "(?P[^,:]+\\.crt:.*?(expire within .+ days|expired) at [^,]+)" | label_format cert=`{{ regexReplaceAll "(?P[^:]+\\.crt):.*" .pair "${cert}" }}` | label_format expire=`{{ regexReplaceAll ".*(expire within .+ days|expired) at (?P[^,]+)" .pair "${date}" }}` | label_format managed_cluster=cluster | label_format expires_ts=`{{ unixEpoch (toDate "2006-01-02T15:04:05Z07:00" .expire) }}s` | unwrap duration_seconds(expires_ts) [30m])) ``` This log query does the heavy lifting by creating one metric point per mentioned certificate from a single log line, as ``` { "count": 638, "eventRV": "496410095", "kind": "Node", "msg": "Node certificates require attention - restart rke2 on this node to trigger automatic rotation: controller-manager/kube-controller-manager.crt: certificate CN=kube-controller-manager will expire within 120 days at 2026-06-08T07:38:41Z, scheduler/kube-scheduler.crt: certificate CN=kube-scheduler will expire within 120 days at 2026-06-08T07:38:41Z", "name": "rke2-eu-cluster1-controlplane-976f6b69-lt7lj", "reason": "CertificateExpirationWarning", "reportingcontroller": "rke2-cert-monitor", "reportinginstance": "rke2-eu-cluster1-controlplane-976f6b69-lt7lj", "sourcecomponent": "rke2-cert-monitor", "sourcehost": "rke2-eu-cluster1-controlplane-976f6b69-lt7lj", "type": "Warning" } ``` with the following labels ``` cluster=rke2-eu-cluster1 rancher_cluster=rancher-eu-prod service_name=kubernetes_events source=kubernetes_events ``` Let's examine the steps of the query: - `{source="kubernetes_events"} | json | reason = "CertificateExpirationWarning"` selects the log lines for Kubernetes events with the reason "CertificateExpirationWarning" and converts them to JSON format - `| line_format "{{ .msg }}"` keeps only the "msg" value from the JSON structure, i.e., the event message - `| regexp "(?P[^,:]+\\.crt:.*?(expire within .+ days|expired) at [^,]+)"`: `(?P … )` captures the part "certname … expire … at DATE" until the last date into a temporary label called pair - `| label_format cert= ... | label_format expire= ...`: `regexReplaceAll` creates a new line for each occurrence of the patterns in `pair` and metadata labels `cert` and `expire` - `| label_format managed_cluster=cluster`: simply renames the label to `managed_cluster` - `| label_format expires_ts=`{{ unixEpoch (toDate "2006-01-02T15:04:05Z07:00" .expire) }}s`: - `toDate` parses the ISO string using Go's layout format. - `unixEpoch` converts the `time.Time` into seconds since epoch. - The trailing `s` is required because Loki expects the value to be a float representing seconds; we keep the `s` suffix for readability (it is stripped later). Now we have a numeric label `expires_ts = 1713168000` (for the example above). - `| unwrap duration_seconds(expires_ts)`: creates a sample with: - value – the result of `duration_seconds(expires_ts)`, i.e., the timestamp expressed as a float number of seconds. - labels – everything that exists at this point (`rancher_cluster`, `managed_cluster`, `cert`, etc.). - Outer `max by (… (max_over_time(....[30m]))` The inner `max_over_time` still returns a series per unique set of labels that existed for each inner log line. The outer `max` by collapses any residual duplication (e.g., if the same cert appears on two nodes) into a single series per certificate. The result is a canonical, monotonic timestamp series that can be safely joined with other metrics. The longer duration 30m is used to avoid missing logs due to transmission latencies from the managed cluster. - `interval: 10m`: To limit processing effort and handle delayed log ingestion, the rule is executed only every 10 minutes, analyzing logs from the last 30 minutes. The resulting metric points are: ``` loki:recorded:rke2_cert_expiration_timestamp{cert="controller-manager/kube-controller-manager.crt", managed_cluster="rke2-eu-cluster1", rancher_cluster="rancher-eu-prod"} 1780904321 loki:recorded:rke2_cert_expiration_timestamp{cert="scheduler/kube-scheduler.crt", managed_cluster="rke2-eu-cluster1", rancher_cluster="rancher-eu-prod"} 1780904236 ``` --- ### Collect Kubernetes Events from Managed Clusters For collecting and forwarding Kubernetes Events from the managed cluster to the central Loki, we can use Grafana Alloy deployed via Helm chart. We need to add the managed cluster name and Rancher cluster as external labels, e.g., via environment variables: ``` # extract from alloy values.yaml alloy: configMap: content: |- loki.source.kubernetes_events "cluster_events" { log_format = "json" forward_to = [ loki.process.cluster_events.receiver ] } ... loki.write "default" { endpoint { url = } external_labels = { cluster = sys.env("CLUSTER_NAME"), rancher_cluster = sys.env("RANCHER_CLUSTER"), } } ``` To ensure we get Kubernetes events automatically from all managed clusters, we can use a Fleet GitRepo within the Rancher cluster. Here we can have the managed cluster name in Fleet variables and the Rancher cluster set in the repo file. --- ### Export Rancher-derived Labels as Metrics For alerting routing, we need metrics that show the associated project and its application label for each managed cluster. However, the connection between managed clusters and projects is not trivial, as it involves several intermediate objects - such as cloud credentials and namespaces - between them. Since SUSE Rancher does not provide these relationships as metrics, we had to write a customer-specific Prometheus metrics exporter. However, it would be too complex to include here. So let's assume for now we have the combined metrics available as a single series per managed cluster: ```text rancher_managed_cluster_owner_info{managed_cluster="rke2-eu-cluster1",project_id="proj1",application="app1"} 1 rancher_managed_cluster_owner_info{managed_cluster="rke2-eu-cluster2",project_id="proj2",application="app1"} 1 rancher_managed_cluster_owner_info{managed_cluster="rke2-eu-cluster3",project_id="proj3",application="app2"} 1 ``` This is an info metric – the value is always `1`. This design allows us to use `group_left` to attach the `application` label to any rule that joins on `managed_cluster`. This exporter is deployed in the Rancher RKE2 cluster and scraped by the operator-managed Prometheus via a ServiceMonitor. For example: ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: custom-rancher-exporter namespace: monitoring spec: endpoints: - port: http jobLabel: app selector: matchLabels: app.kubernetes.io/instance: custom-rancher-exporter app.kubernetes.io/name: custom-rancher-exporter ``` ### Defining The Alert Rule Now we can combine the log-derived timestamp with the Rancher label metric in the following Alert rule. This is deployed via CR PrometheusRule in the Rancher cluster, to be added to the Prometheus from SUSE Rancher-monitoring Helm chart. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: customer-alerts namespace: customer-monitoring spec: groups: - name: rke2-alerts rules: - alert: ManagedClusterRKE2ExpiringCertificates annotations: action: | The certificate rotation needs to be triggered manually. Follow the instructions in https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/manage-clusters/rotate-certificates description: | The certificate {{ $labels.cert }} on cluster {{ $labels.managed_cluster }} expires on {{ $value | humanizeTimestamp }} impact: | When the certificate is not renewed, some Kubernetes services will no longer be accessible. expr: | (max_over_time(loki:recorded:rke2_cert_expiration_timestamp[30m]) * on(managed_cluster) group_left(application) rancher_managed_cluster_owner_info ) < (time() + 5 * 60 * 60 * 24) for: 5m labels: severity: critical ``` **Explanation** - `max_over_time(...[30m])`: Pull the most recent expiry timestamp for each certificate. A 30-minute look-back is needed since we evaluate logs only every 10 minutes. - `* on(managed_cluster) group_left(application) rancher_managed_cluster_owner_info`: Join the timestamp series with the Rancher label metric, copying the `application` label onto the result. - `< (time() + 5*60*60*24)`: Trigger when the stored expiry is *earlier* than "now + 5 days". With the `application` label present, Alertmanager can now route the alert to the appropriate receiver (e.g., Slack channel `#app1-ops`). --- ## Conclusion We have shown a rather complex setup needed to reliably alert customers about expiring RKE2 certificates in their managed clusters. This required specifically: 1. **Gathering Events from the managed clusters** (Alloy Kubernetes Event collection deployed via Fleet from Rancher Cluster) 2. **Parsing logs into a recorded time series** (Loki recording rules), 3. **Exporting business metadata as informational metrics** (custom Rancher Exporter), and 4. **Joining both metrics in a Prometheus alert rule**, to obtain alerts that are both *precise* (exact expiry timestamps) and *actionable* (routed to the correct owners). The approach scales across any number of managed clusters. *Happy monitoring!* [Read more](https://omd.consol.de/blog/2026/03/24/reliable-rke2-certificate-expiry-alerts-for-clusters-managed-by-suse-rancher/) ----------------------------------------------------------------------------------- title: "Using check_sap_health with SNC" date: "2025-12-02" -------------------------------------------------------------------------------- To use check_sap_health with SNC (Secure Network Communication), you need to install the SAP NetWeaver RFC SDK and the SAP Cryptographic Library, along with the Perl module sapnwrfc. This guide walks you through the complete setup process within an OMD site. ## Prerequisites You need to download the following archives from the SAP Software Download Center: - **SAP NetWeaver RFC SDK**: `nwrfc750P_17-70002752.zip` (or newer version) - **SAP Cryptographic Library**: `SAPCRYPTOLIBP_8561-20011697.SAR` - **SAPCAR**: Tool to extract SAR archives Place these files in a preparation directory within your OMD site: ```bash OMD[sapmon@omdmuc08]:~$ cd prep_sap/ OMD[sapmon@omdmuc08]:~/prep_sap$ ls -l total 27488 -rw-r--r--. 1 demo demo 20054340 Dec 2 10:14 nwrfc750P_17-70002752.zip -rwxr-xr-x. 1 demo demo 5712584 Dec 2 10:14 SAPCAR* -rw-r--r--. 1 demo demo 2373692 Dec 2 10:14 SAPCRYPTOLIBP_8561-20011697.SAR ``` ## Step 1: Extract the SAP NetWeaver RFC SDK Unzip the RFC SDK archive: ```bash OMD[sapmon@omdmuc08]:~/prep_sap$ unzip nwrfc750P_17-70002752.zip ``` This creates a `nwrfcsdk/` directory containing the libraries, headers, and tools needed for RFC communication. ## Step 2: Extract the SAP Cryptographic Library Use SAPCAR to extract the cryptographic library: ```bash OMD[sapmon@omdmuc08]:~/prep_sap$ ./SAPCAR -xf SAPCRYPTOLIBP_8561-20011697.SAR SAPCAR: processing archive SAPCRYPTOLIBP_8561-20011697.SAR (version 2.01) SAPCAR: 6 file(s) extracted ``` This extracts the following files: - `libsapcrypto.so` - Main cryptographic library - `libslcryptokernel.so` - Cryptographic kernel - `sapgenpse` - Tool for managing Personal Security Environments ```bash OMD[sapmon@omdmuc08]:~/prep_sap$ ls -l lib* -rwxr-xr-x. 1 demo demo 5843176 Aug 25 15:33 libsapcrypto.so* -rwxr-xr-x. 1 demo demo 499679 Sep 8 08:02 libslcryptokernel.so* -rw-r--r--. 1 demo demo 166 Sep 8 08:16 libslcryptokernel.so.sha256 ``` ## Step 3: Clone and Build perl-sapnwrfc The perl-sapnwrfc module is a fork of the well-known Perl binding for SAP NetWeaver RFC SDK, originally maintained by Piers Harding. The original repository was the standard way to create Perl bindings to the SAP NWRFCSDK for many years, but it was last updated in 2013 and the maintainer seems to have abandoned the project. With the advent of modern C compilers (C23 standard, GCC 14+) on recent Linux distributions, the old C-Perl bindings can no longer be successfully compiled. This is why my fork of the module is not available on CPAN and must be cloned from GitHub, which has been updated to work with contemporary compilers while maintaining backward compatibility with legacy systems. Clone the perl-sapnwrfc repository from GitHub: ```bash OMD[sapmon@omdmuc08]:~/prep_sap$ git clone https://github.com/lausser/perl-sapnwrfc.git Cloning into 'perl-sapnwrfc'... remote: Enumerating objects: 292, done. remote: Total 292 (delta 156), done. ``` Navigate to the cloned directory and build the Perl module: ```bash OMD[sapmon@omdmuc08]:~/prep_sap$ cd perl-sapnwrfc/ OMD[sapmon@omdmuc08]:~/prep_sap/perl-sapnwrfc$ perl Makefile.PL --source $(pwd)/../nwrfcsdk/ ``` The configure script will detect the SAP libraries and generate the Makefile: ``` ================================================ BUILD INFORMATION ================================================ OS: linux source opt: /omd/sites/sapmon/prep_sap/perl-sapnwrfc/../nwrfcsdk/ sapnwrfc dir: /omd/sites/sapmon/prep_sap/perl-sapnwrfc/../nwrfcsdk/ libraries: -lm -ldl -lrt -lpthread -lsapnwrfc -lsapucum include dir: /omd/sites/sapmon/prep_sap/perl-sapnwrfc/../nwrfcsdk//include ================================================ ``` Now compile the module: ```bash OMD[sapmon@omdmuc08]:~/prep_sap/perl-sapnwrfc$ make ``` ## Step 4: Install the Perl Module and Libraries Install the compiled Perl module into your OMD site's local Perl library: ```bash OMD[sapmon@omdmuc08]:~/prep_sap/perl-sapnwrfc$ make install Files found in blib/arch: installing files in blib/lib into architecture dependent library tree Installing /omd/sites/sapmon/local/lib/perl5/lib/perl5/x86_64-linux-thread-multi/auto/SAPNW/Connection/Connection.so Installing /omd/sites/sapmon/local/lib/perl5/lib/perl5/x86_64-linux-thread-multi/sapnwrfc.pm ... ``` Copy all required libraries to your OMD site's local lib directory: ```bash OMD[sapmon@omdmuc08]:~/prep_sap$ cp nwrfcsdk/lib/lib* lib* ~/local/lib OMD[sapmon@omdmuc08]:~/prep_sap$ cp sapgenpse ~/local/bin ``` This copies: - The RFC SDK libraries (`libsapnwrfc.so`, `libsapucum.so`) - The ICU libraries (`libicudata.so.50`, `libicui18n.so.50`, `libicuuc.so.50`) - The cryptographic libraries (`libsapcrypto.so`, `libslcryptokernel.so`) - The `sapgenpse` tool for managing Personal Security Environments ## Step 5: Verify the Installation Clean up the preparation directory: ```bash OMD[sapmon@omdmuc08]:~/prep_sap$ cd .. OMD[sapmon@omdmuc08]:~$ rm -rf prep_sap ``` Verify that the Perl module is properly linked to all required libraries: ```bash OMD[sapmon@omdmuc08]:~$ ldd /omd/sites/sapmon/local/lib/perl5/lib/perl5/x86_64-linux-thread-multi/auto/SAPNW/Connection/Connection.so linux-vdso.so.1 (0x00007feef9b0e000) libsapnwrfc.so => /omd/sites/sapmon/local/lib/libsapnwrfc.so (0x00007feef8df0000) libsapucum.so => /omd/sites/sapmon/local/lib/libsapucum.so (0x00007feef8a93000) libc.so.6 => /lib64/libc.so.6 (0x00007feef888b000) ... ``` All libraries should resolve to your local installation paths, confirming that the dependencies are properly installed. ## Using check_sap_health with SNC Now that the dependencies are installed, you can use check_sap_health with SNC (Secure Network Communication) for encrypted and authenticated connections to your SAP systems. ### SNC-Related Parameters check_sap_health provides several command-line parameters to enable and control SNC: **`--snc`** Enables the SNC protocol for communication. If this flag is not set, all other SNC parameters are ignored. When `--snc` is set, the environment variable `SNC_MODE` is set to `1`. **`--secudir`** Sets the environment variable `SECUDIR`, which specifies the folder where the `SAPSNCS.pse` file (Personal Security Environment) is expected. This file contains the certificates and keys for SNC authentication. **`--snc-lib`** Sets the environment variable `SNC_LIB`. By default, if this parameter is not set, check_sap_health searches the `LD_LIBRARY_PATH` until a file `libsapcrypto.so` is found and sets `SNC_LIB` accordingly. Use `--snc-lib` if `libsapcrypto.so` is installed in an unusual location. **`--snc-myname`** Sets the environment variable `SNC_MYNAME`, which identifies your own SNC name (the monitoring system's identity). **`--snc-partnername`** Sets the environment variable `SNC_PARTNERNAME`, which identifies the SAP system you want to connect to. This is the Distinguished Name (DN) of the SAP system's certificate. **`--snc-qop`** Sets the environment variable `SNC_QOP` (Quality of Protection). The default value is `3`. This parameter controls the level of security with values ranging from `1` to `9`: - `1` = Authentication only - `2` = Integrity protection - `3` = Privacy protection (encryption) - `8` = Use default protection - `9` = Maximum protection **`--saprouter`** Specifies the SAP Router to use for the connection (e.g., `/H/e4u.sap.consol.de`). This parameter was implemented alongside SNC support and enables a powerful combination: when used together with SNC, the plugin can perform encrypted communication through a single communication partner—the SAP Router. This simplifies network architecture and security policies, as all SAP communication can be routed through one central gateway. ### Example Usage Here's a complete example of using check_sap_health with SNC to monitor CCMS MTE values: ```bash check_sap_health \ --username rfc-nagios --password qfiqh43fiqf4i \ --mshost k-e4u-ci.sap.consol.de --r3name E4U --client 000 --msserv 3628 \ --mode ccms-mte-check --mtelong \ --name "Consol IT SAP-System (local)" \ --name2 "Produktivsystem" --regexp --name3 "UsersLoggedIn" \ --snc \ --secudir /omd/sites/sapmon/etc/check_sap_health/sec \ --snc-partnername "p/secude:EMAIL=CON-SAP-COMPETENCE-CENTER@MAIL.CONSOL,CN=SAPNCKE4U,O=CONSOL IT,C=DE" ``` In this example: - The `--snc` flag enables SNC communication - The `--secudir` parameter points to the directory containing the `SAPSNCS.pse` file - The `--snc-partnername` specifies the Distinguished Name of the SAP system's certificate - The SNC library (`libsapcrypto.so`) is automatically detected from the `LD_LIBRARY_PATH` With these parameters, check_sap_health establishes a secure, encrypted connection to the SAP system using the certificates and keys configured in your Personal Security Environment. [Read more](https://omd.consol.de/blog/2025/12/02/using-check_sap_health-with-snc/) ----------------------------------------------------------------------------------- title: "A Brief Overview Of The check_vsphere Plugin" date: "2025-11-18" -------------------------------------------------------------------------------- ## What is it? [check\_vsphere](https://github.com/consol-monitoring/check_vsphere) is a plugin for Naemon, Icinga, and Nagios-compatible systems. It checks various aspects of ~VMware~Broadcom vCenter or ESX hosts. For a long time, this was done using `check_vmware_esx.pl` or `check_esx.pl`. However, Broadcom (formerly VMware) has decided to deprecate the Perl SDK for vCenter. Therefore, we decided to rewrite the parts our customers use in Python using the [pyVmomi](https://github.com/vmware/pyvmomi/) library. In this article, I will provide an overview of what the plugin can do and delve into some of its features. Development happens at [Github](https://github.com/consol-monitoring/check_vsphere). Feel free to open issues or pull requests. ## Authentication Currently, only user/password-based authentication is supported. The common options needed to establish a connection are: * `-u USERNAME` * `-p PASSWORD` can be omitted in favor of the `VSPHERE_PASS` environment variable * `-s ADDR` hostname of the vCenter or ESX host * `-nossl` whether TLS verification should be skipped So a command line has at least this basic structure: ``` check_vsphere subcommand -u user -p pass -s addr [subcommand options] ``` In this document `[AUTH]` just means: `-u user -p pass -s addr`. ## Checks Here is a brief overview of some features, to see the full list please see [the documentation](https://omd.consol.de/docs/plugins/check_vsphere/cmd/). ### VSAN The [vsan](/docs/plugins/check_vsphere/cmd/vsan/) command offers two modes: * `healthtest` – shows exactly what you see under **Cluster → Monitor → vSAN → Skyline Health** in vCenter. * `objecthealth` – performs a detailed check of vSAN object health. Please try them, they are not used very much and may need some fine tuning. ### Host checks There are several host checks in `check_vsphere`: * **[host-runtime](/docs/plugins/check_vsphere/cmd/host-runtime/)** offers a few modes: * **status** – vCenter calculates an overall host status. This mode just maps the colors to exit codes (green → OK, yellow → warning, red → critical). * **con** – checks whether the host can still talk to the vCenter. * **health** – runs various health checks exposed by the API for the host (memory, voltage, fans, …) and reports any problems. * **temp** – walks through the temperature sensors and reports issues. The state is determined by the vCenter/ESX host itself. * **[host-nic](/docs/plugins/check_vsphere/cmd/host-nic/)** - This check verifies if all network interfaces are connected * **[host-service](/docs/plugins/check_vsphere/cmd/host-service/)** - This check can verify if various services are running on a host, like ntp, DCUI, vpxa etc. ### VM checks * **[media](/docs/plugins/check_vsphere/cmd/media/)** – spots VMs that still have a CD‑ROM attached. * **[vm-tools](/docs/plugins/check_vsphere/cmd/vmtools/)** – flags VMs without guest tools installed. * **[vm‑net‑dev](/docs/plugins/check_vsphere/cmd/vmnetdev/)** – finds VMs that contain unused network devices. * **[snapshots](/docs/plugins/check_vsphere/cmd/snapshots/)** – reports VMs with an unexpected number of snapshots or snapshots that are too old. * **[vm‑guestfs](/docs/plugins/check_vsphere/cmd/vmguestfs/)** – monitors filesystem usage of VM volumes via vCenter. ### PerfCounters #### Overview The vCenter has a variety of [performance counters](https://dp-downloads.broadcom.com/api-content/apis/API_VWSA_001/8.0U3/html/ReferenceGuides/vim.PerformanceManager.html). These counters may be related to VirtualMachines, HostSystems, Datacenters, ClusterComputeResources, and possibly more. `check_vmware_esx` had many hard-coded options for specific performance counters. We decided to generalize this so any performance counter can be checked with `check_vsphere`. To get a list of performance counters available on a vCenter, the `list-metrics` command can be used. ``` check_vsphere list-metrics [AUTH] ``` If you're coming from `check_vmware_esx`, [the documentation](/docs/plugins/check_vsphere/cmd/perf/#rosetta) has a list of all the performance counters that were supported by `check_vmware_esx` and their counterparts in `check_vsphere`. However, as mentioned earlier, you can check any performance counter. For example, to monitor the power consumption of an ESX host: ``` check_vsphere perf [AUTH] --perfcounter power:power:average \ --vimtype HostSystem --vimname esx-hostname \ --critical 400 ``` #### Instances `check_vmware_esx` and its related tools have a significant bug. Performance counters can have instances. For example, disk I/O counters are available for each disk, where each disk represents an instance of the counter. When you monitor this with `check_vmware_esx`, you only monitor a random disk and ignore all the others. Yes, we have been monitoring random disks for years. With `check_vsphere`, you can now check specific disks using the `--perfinstance` flag. The default instance is an empty string, which is a special value. It monitors the aggregate (average) across all instances where this is applicable. This is only available when it makes sense; for example, CPU usage can have an aggregate over all cores. However, calculating the average across several different disks is generally not meaningful, so vSphere does not provide this aggregate. You can also check each instance with `--perfinstance '*'`. In this case, the threshold is applied to each instance, and the highest criticality is returned. ``` # check disk latency # the default perfinstance is '' which is the aggregate and not available # for this counter $ check_vsphere perf -s vcenter.example.com -u naemon@vsphere.local -nossl \ --vimname esx1.int.example.com --vimtype HostSystem \ --perfcounter disk:totalLatency:average UNKNOWN: Cannot find disk:totalLatency:average for the queried resources # On that error you may want to try --perfinstance '*' # now you see all instances for this counter $ check_vsphere perf -s vcenter.example.com -u naemon@vsphere.local -nossl \ --vimname esx1.int.example.com --vimtype HostSystem \ --perfcounter disk:totalLatency:average --perfinstance '*' OK: disk:totalLatency:average_naa.6000eb3810d426400000000000000277 has value 0 Millisecond disk:totalLatency:average_naa.600605b00ba8cb0022564867b8c8cc32 has value 2 Millisecond disk:totalLatency:average_naa.6000eb3810d4264000000000000000b2 has value 0 Millisecond disk:totalLatency:average_naa.600605b00ba8cb001fd947850523e56d has value 0 Millisecond disk:totalLatency:average_naa.600605b00ba8cb0029700b163217244e has value 6 Millisecond disk:totalLatency:average_naa.6000eb3810d4264000000000000002b3 has value 1 Millisecond | 'disk:totalLatency:average_naa.6000eb3810d426400000000000000277'=0.0ms;;;; 'disk:totalLatency:average_naa.600605b00ba8cb0022564867b8c8cc32'=2.0ms;;;; ... # you can also check a single instance specifically $ check_vsphere perf -s vcenter.example.com -u naemon@vsphere.local -nossl \ --vimname esx1.int.example.com --vimtype HostSystem \ --perfcounter disk:totalLatency:average --perfinstance naa.600605b00ba8cb0022564867b8c8cc32 OK: disk:totalLatency:average_naa.600605b00ba8cb0022564867b8c8cc32 has value 2 Millisecond | 'disk:totalLatency:average_naa.600605b00ba8cb0022564867b8c8cc32'=2.0ms;;;; ``` [Read more](https://omd.consol.de/blog/2025/11/18/a-brief-overview-of-the-check_vsphere-plugin/) ----------------------------------------------------------------------------------- title: "OMD Labs Edition Not Affected by Recent Nagios XI Vulnerabilities" date: "2025-11-04" -------------------------------------------------------------------------------- ## OMD Labs Edition Not Affected by Recent Nagios XI Vulnerabilities Several vulnerabilities were recently disclosed in **Nagios XI**: - **CVE-2025-34286** – Remote Code Execution in the *Core Config Manager (CCM)* - **CVE-2025-34284** – Command Injection in the *WinRM Plugin* - **CVE-2025-34134** – Remote Code Execution in the *Business Process Intelligence (BPI) Component* These issues could allow authenticated administrators to execute arbitrary commands on affected systems, potentially leading to full host compromise. --- ### OMD Labs Edition Is **Not Affected** The **OMD Labs Edition** (Open Monitoring Distribution) is an open-source monitoring platform maintained by [ConSol Labs](https://omd.consol.de). While it is compatible with Nagios and related cores such as **Naemon** and **Icinga**, it does **not** include or depend on the proprietary components of **Nagios XI** that are impacted by these vulnerabilities. Specifically: - **CVE-2025-34286:** Targets the *Core Config Manager (CCM)* of Nagios XI. → *OMD Labs Edition does not use the CCM.* - **CVE-2025-34284:** Targets the *WinRM Plugin* of Nagios XI. → *OMD Labs Edition does not ship or depend on the WinRM plugin.* - **CVE-2025-34134:** Targets the *Business Process Intelligence (BPI)* component of Nagios XI. → *OMD Labs Edition does not contain any BPI component.* Therefore, **OMD Labs Edition is not affected** by these vulnerabilities. [Read more](https://omd.consol.de/blog/2025/11/04/omd-labs-edition-not-affected-by-recent-nagios-xi-vulnerabilities/) ----------------------------------------------------------------------------------- title: "Labs Repository GPG Key Rotation" date: "2025-08-21" -------------------------------------------------------------------------------- ## Outdated This news is **outdated** and superseded by [the key rotation from April 2026](/blog/2026/04/13/labs-repository-gpg-key-rotation/). ## Key Rotation Starting on August 18 2025 all .deb and .rpm files in the [labs repository](/repo) will use the `GPG-KEY-4096` instead of the old `RPM-GPG-KEY`. For consistency all existing rpm and deb files have been resigned to use the new key as well. | | File | Size | ID | |----------|-----------------------------------------------------------------|------|-------------------| | **Old** | [RPM-GPG-KEY](https://labs.consol.de/repo/stable/RPM-GPG-KEY) | 1024 | F8C1CA08A57B9ED7 | | **New** | [GPG-KEY-4096](https://labs.consol.de/repo/stable/GPG-KEY-4096) | 4096 | F0CA212FF1FFE778 | ## Debian / Ubuntu In Debian and Ubuntu you will probably notice warnings like this when running `apt update` ```txt Err:1 http://labs.consol.de/repo/stable/debian bookworm InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F0CA212FF1FFE778 Fetched 48.0 kB in 0s (158 kB/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done All packages are up to date. W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://labs.consol.de/repo/stable/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F0CA212FF1FFE778 ``` To make it work again, simple replace the key with the new one. Either follow the instructions from the repository installation again: - [stable repository](https://labs.consol.de/repo/stable/) - [stream repository](https://labs.consol.de/repo/stream/) - [testing repository](https://labs.consol.de/repo/testing/) or use these commands: ```bash curl -fsS "https://labs.consol.de/repo/stable/GPG-KEY-4096" -o /etc/apt/trusted.gpg.d/labs.consol.de-GPG-KEY-4096.asc ``` Then make sure the key is used in the sources file: ex.: `/etc/apt/sources.list.d/labs-consol-stable.list` ```txt deb [signed-by=/etc/apt/trusted.gpg.d/labs.consol.de-GPG-KEY-4096.asc] http://labs.consol.de/repo/stable/debian bookworm main ``` The important part here is, the `signed-by` option must point to the new key file. ## RHEL / Rocky Linux / Alma RHEL 9 and later already used the new key, so there is nothing to do here. [Read more](https://omd.consol.de/blog/2025/08/21/labs-repository-gpg-key-rotation/) ----------------------------------------------------------------------------------- title: "Tunnel SSH into an OMD server. Proof of Concept" date: "2025-07-31" -------------------------------------------------------------------------------- # Tunneling SSH over WebSocket with GOST in an OMD server: In this blog post, we'll walk through a proof-of-concept (POC) setup for tunneling SSH traffic over WebSocket using the GOST tool. This configuration runs inside an OMD (Open Monitoring Distribution) container, leveraging Apache as a reverse proxy for the WebSocket connection. The goal is to securely forward SSH requests from a local client to the container's SSH daemon (sshd) via an encrypted tunnel. This setup is useful for scenarios where direct SSH access is restricted, but HTTP/HTTPS ports are open (e.g., behind firewalls or in cloud environments). We'll use Podman to manage the container, self-signed certificates for TLS, and GOST for the tunneling logic. ## Overview The architecture involves: - **Client-side (outside the container)**: An SSH client connects to a local GOST listener, which forwards the traffic over secure WebSocket (WSS) to the container's Apache server. - **Server-side (inside the container)**: Apache proxies the WebSocket connection to a GOST server instance, which then forwards the traffic to the local sshd on port 22. Here's a high-level diagram of the flow: ```mermaid graph LR A[SSH Client
ssh -p 2222 localhost] --> B[Client GOST
tcp://:2222
forward+wss://localhost:8443?path=/ssh] B --> C[Apache httpd
ProxyPass ws://localhost:8080/ssh] C --> D[Server GOST
forward+ws://:8080?path=/ssh
tcp://127.0.0.1:22] D --> E[sshd
port 22] ``` ## Prerequisites - Podman installed on the host machine. - GOST binary available both on the host and inside the container (download from [gost.run](https://gost.run/)). - Basic knowledge of container management, Apache configuration, and SSH. ## Step-by-Step Setup ### 1. Start the OMD Container On the host machine, run the OMD container with Podman, mapping the external port 8443 to the container's internal HTTPS port 443: ```bash podman run -it -p 8443:443 --entrypoint bash docker.io/consol/omd-labs-rocky:nightly ``` This drops you into a bash shell inside the container. ### 2. Configure Certificates and SSH Inside the Container Generate a self-signed certificate for TLS using `sscg`: ```bash sscg -q \ --cert-file /etc/pki/tls/certs/localhost.crt \ --cert-key-file /etc/pki/tls/private/localhost.key \ --ca-file /etc/pki/tls/certs/localhost.crt \ --lifetime 365 \ --hostname localhost \ --email root@localhost ``` Generate SSH host keys and start the SSH daemon: ```bash /usr/libexec/openssh/sshd-keygen rsa /usr/sbin/sshd ``` Create a password for the user *demo* ```bash passwd demo ``` ### 3. Configure Apache for WebSocket Proxying Edit the Apache configuration file `/omd/apache/ssh.conf` to enable WebSocket proxying: ```apache LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so LoadModule rewrite_module modules/mod_rewrite.so RewriteEngine On RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule ^/ssh$ ws://localhost:8080/ssh [P,L] ProxyPass ws://localhost:8080/ssh retry=0 disablereuse=On ProxyPassReverse ws://localhost:8080/ssh ``` Start Apache: ```bash httpd ``` ### 4. Run Server-Side GOST Inside the container, start the GOST server to listen on WebSocket and forward to local SSH: ```bash gost -DD -L forward+ws://:8080?path=/ssh -F tcp://127.0.0.1:22 ``` ### 5. Run Client-Side GOST and Test SSH Back on the host machine (outside the container), start the client-side GOST: ```bash gost -DD -L tcp://:2222 -F forward+wss://localhost:8443?path=/ssh ``` Now, connect via SSH to the tunneled port: ```bash ssh -p 2222 -l demo localhost ``` You should be prompted for authentication and connected to the container's sshd as user 'demo'. ## Potential Issues and Troubleshooting - **Self-Signed Certs**: Use `-k` in curl tests or add the cert to your trust store for production. - **Port Mapping**: Ensure Podman's port mapping (8443:443) is correct; adjust if using a different external port. - **Logs**: Check GOST logs with `-DD` for debug info, and Apache error logs for proxy issues. - **Security**: This POC uses no authentication on the WebSocket path—add Basic Auth or other mechanisms for real-world use. ## Conclusion This setup demonstrates how to tunnel traditional protocols like SSH over modern web technologies, making it firewall-friendly and scalable. Experiment with adding authentication or multiple forwards for more advanced scenarios! If you have questions or improvements, drop a comment below. [Read more](https://omd.consol.de/blog/2025/07/31/tunnel-ssh-into-an-omd-server.-proof-of-concept/) ----------------------------------------------------------------------------------- title: "Coshsh can keep a secret" date: "2025-07-17" -------------------------------------------------------------------------------- Coshsh, as a generator for monitoring config files, needs to know which hosts and applications make up an enterprise's IT landscape. Usually, it fetches this information by querying a CMDB's API. Access is typically granted by presenting a token, or a username and password. Since this is confidential data, it should not be visible in clear text—especially if you back up your Coshsh config in a public Git repository. In an OMD setup, you can store secrets in a vault, which is used by the Naemon core. Since the release of Coshsh 11.0, Coshsh can use the vault too. Imagine the following scenario: you have two OMD sites, *prod* and *nonprod*—for production and testing, respectively. Coshsh collects inventory data by querying the ServiceNow CMDB. It then generates the corresponding configuration files, which are used by the two OMD sites. Importantly, the OMD sites themselves do not query ServiceNow for inventory; all ServiceNow communication is handled by Coshsh during config generation. However, both OMD sites can create incidents in ServiceNow as part of their monitoring workflows. The production site interacts with the main ServiceNow instance, while the testing site uses a separate development ServiceNow instance. | cmdb ServiceNow | | omd site | | Incident ServiceNow | |-------------------|--------|----------|--------|-----------------------| | itsm.example.com | → | prod | → | itsm.example.com | | itsm.example.com | → | nonprod | → | itsm-dev.example.com | Let’s further assume that the two ServiceNow instances have different API credentials: | ServiceNow instance | Username | Password | |---------------------|-------------|--------------| | main | monitoring | v3rys3cr3t | | dev | monitordev | n0ts0s3cr3t | If you don't have a Naemon Vault yet, you can create one in your OMD site with: ``` vim -x -c "set cm=blowfish2" etc/naemon/vault.cfg ``` (You can find more information on how to use the vault with the Naemon core [here](https://github.com/naemon/naemon-vimcrypt-vault-broker).) Next, edit the vault and add the ServiceNow passwords: ``` $VAULT:svcnow_pw_prod$ = v3rys3cr3t $VAULT:svcnow_pw_nonprod$ = n0ts0s3cr3t ``` In the Coshsh cookbook, we have two recipes, which generate the configs for the *prod* and *nonprod* sites. Because the variable *%RECIPE_NAME%* is replaced by the recipe's name when running the generation ("cooking"), we can reference the secrets using *svcnow_pw_%RECIPE_NAME%*. This helps to keep the cookbook shorter, because there is no need to define separate datasources with the different secret references. The credentials are used in a data source of type *svcnow_cmdb_ci*. (FYI, the datasource of type *svcnow_cmdb_ci*, which connects coshsh to the ServiceNow CMDB, also creates the definition for a contact named *servicenow*. This contact is using a notification script which creates incidents via the */api/now/table/incident* endpoint of ServiceNow. In order to separate the CMDB interface from the Incident interface, the datasource can take two different urls, *cmdb_url* and *incident_url*) ``` ############################### # etc/coshsh/conf.d/example.cfg ############################### # # A vault is a file or database where secrets are stored (in encrypted or # at least not publicly accessible form) # This section defines a vault of type Naemon Vault, which can be opened and # read using the environment variable $NAEMON_VIM_MASTER_PASSWORD as the key. # Thanks to a vault, secrets have not to be written in cleartext in this # config file. Instead, we reference them using the notation @VAULT[key] # Vault contents are kept inside coshsh in form of a key-value-dictionary. # In this example, an occurrence of @VAULT[svcnow_pw_prod] will be replaced # by "v3rys3cr3t". # [vault_naemon] type = naemon_vault file = ./etc/naemon/vault.cfg key = %NAEMON_VIM_MASTER_PASSWORD% # # A mapping is like a key-value store. It has a name ("svcnow" in this case) # and can be used in recipes', datasources' and datarecipients' attributes. # @MAPPING_SVCNOW[svcnow_url_prod] for example will resolve to # https://svcnow.example.com # @MAPPING_NAMEINCAPITALLETTERS[key] -> value # [mapping_svcnow] svcnow_user_prod = monitoring svcnow_user_nonprod = monitordev svcnow_url_prod = https://svcnow.example.com svcnow_url_nonprod = https://svcnow-dev.example.com # # This is the datasource the inventory data are read from. # The type svcnow_cmdb_ci references the Python code which actually # communicates with the ServiceNow API. # [datasource_servicenow] type = svcnow_cmdb_ci username = @MAPPING_SVCNOW[svcnow_user_%RECIPE_NAME%] password = @VAULT[svcnow_pw_%RECIPE_NAME%] # Inventory always comes from the main ServiceNow cmdb_url = @MAPPING_SVCNOW[svcnow_url_prod] # Incidents are created either in the main or the dev ServiceNow incident_url = @MAPPING_SVCNOW[svcnow_url_%RECIPE_NAME%] # # This is a recipe which can't be cooked (watch the double \_). # It's sole purpose is to be inherited by the prod and nonprod recipes in # order to avoid repetitive attributes. # [recipe__vault] objects_dir = %OMD_ROOT%/var/coshsh/configs/%RECIPE_NAME% classes_dir = %OMD_ROOT%/etc/coshsh/recipes/example/classes templates_dir = %OMD_ROOT%/etc/coshsh/recipes/example/templates datasources = servicenow vaults = naemon # # These are the actual recipes. You can run # coshsh-cook --cookbook ~/etc/coshsh/conf.d/example.cfg --recipe prod # coshsh-cook --cookbook ~/etc/coshsh/conf.d/example.cfg --recipe nonprod # which will read inventory data from the main dev servicenow and include # a contact which will create incidents in the main resp. the dev servicenow. # The config files will be written to # ~/var/coshsh/configs/prod/dynamic or ~/var/coshsh/configs/nonprod/dynamic # [recipe_prod] isa = recipe__vault [recipe_nonprod] isa = recipe__vault ``` When we cook a recipe, the data source's attributes are first updated with the actual recipe name, so we get the correct references to the mapping and vault variables. Then, these references are looked up in *mapping_svcnow* and *vault_naemon*, and are resolved to their final values. Using this new feature allows you to safely commit this *example.cfg* to a public Git repository. [Read more](https://omd.consol.de/blog/2025/07/17/coshsh-can-keep-a-secret/) ----------------------------------------------------------------------------------- title: "Forwarding Windows Eventlogs to Loki with Alloy" date: "2025-02-27" -------------------------------------------------------------------------------- ### Forwarding Windows Eventlogs to a central log console In system monitoring, logs are a valuable source for detecting upcoming or existing issues. Especially for Windows, collecting all logs in one place has not been easy. Forwarding Windows event logs to a syslog server, which then writes the logs to files, was one approach—but this is quite outdated. Here, you will set up a modern solution based on state-of-the-art observability tools. On the client side, you will use the agent [SNClient](/docs/snclient) with its helper, [Grafana Alloy](https://grafana.com/oss/alloy-opentelemetry-collector/). On the monitoring side, you will use the [Open Monitoring Distribution](/docs/omd) with [Loki](https://grafana.com/oss/loki/). ![event forwarding](./snclient-alloy-omd-loki.drawio.png) ### Step one - Install OMD and open the Loki API. To keep things short, I assume that you already have OMD installed and have created a site. In this example, use the site name *demo*. (And the OMD server is called *omd-server*) Loki is not enabled by default, so you need to run the following commands: ```bash omd stop omd config set GRAFANA on omd config set LOKI on ``` In an OMD setup, Loki listens by default only on the loopback interface. To make it accessible from Windows servers, create the file *~/etc/apache/conf.d/loki.conf* with the following content: ```apache LoadModule proxy_module /usr/lib64/httpd/modules/mod_proxy.so LoadModule proxy_http_module /usr/lib64/httpd/modules/mod_proxy_http.so ProxyPassInterpolateEnv on ProxyPass http://127.0.0.1:${CONFIG_LOKI_HTTP_PORT}/loki ProxyPassReverse http://127.0.0.1:${CONFIG_LOKI_HTTP_PORT}/loki RequestHeader set X-WEBAUTH-USER %{REMOTE_USER}e ErrorDocument 503 /503.html?LOKI=on ``` The Loki API is now accessible externally via *https://omd-server/demo/loki/api/v1/push*. Access is controlled via the Thruk login page. Using basic authentication makes the login transparent, so the client believes it is communicating directly with the API. It means, would you open this url in a browser, you would see a login page presented by OMD's default gui, Thruk. Alloy will add credentials when sending POST requests to the API and Thruk will immediately let them pass. A dedicated user and its password are created like this: ```bash htpasswd ~/etc/htpasswd loki L0k1 ``` Finally start the OMD site. ```bash omd start ``` Loki is now ready to receive Windows events. (Or any other data sent by Alloy/journald, Open Telemetry Logs, Fluent Bit, Docker, Promtail,...) ### Step two - Install the SNClient on the Windows server For the base installation, follow the instructions [here](/docs/snclient/install/windows/). Next, change the default password by creating a new file *C:\Program Files\snclient\snclient_local_auth.ini* with the following content: ```ini [/settings/default] allowed hosts = 127.0.0.1, 10.0.1.2 password = SHA256:9f86d081884... ``` The password is stored as a hashed value. Refer to the [Security page](https://omd.consol.de/docs/snclient/security/) for instructions. (*allowed hosts* restricts access to the snclient agent, i suggest you edit the list so that it consists of 127.0.0.1 and yout omd-server's ip address) After saving the file, restart the service **snclient** using the service manager or by running ```powershell net stop snclient net start snclient ``` At this point, you can monitor the Windows host with *Naemon* and the *check_nsc_web* plugin, but this article focuses on log forwarding. ### Step three - Add Alloy to SNClients exporters Create a file *C:\Program Files\snclient\snclient_local_alloy.ini* with the following contents: ```ini [/modules] ManagedExporterServer = enabled [/settings/ManagedExporter/alloy] ;password = agent path = ${shared-path}/exporter/alloy-windows-amd64.exe agent args = run ./alloy agent address = 127.0.0.1:12345 ;;agent max memory = 256M url prefix = /alloy port = ${/settings/WEB/server/port} ``` This config instructs snclient to start (and eventually restart) Grafana Alloy. Next, go to the [Grafana Alloy release page](https://github.com/grafana/alloy/releases), download *alloy-windows-amd64.exe.zip*, unpack it and move the extracted *alloy-windows-amd64.exe* to *C:\Program Files\snclient\exporter*. Then, create a folder *C:\Program Files\snclient\alloy* and add the file *windows_event.alloy* with the following content: ```alloy loki.source.windowsevent "application" { eventlog_name = "Application" use_incoming_timestamp = true exclude_event_data = true forward_to = [loki.process.windows_eventlog.receiver] labels = { job = "windows_eventlog", instance = constants.hostname, } } loki.source.windowsevent "security" { eventlog_name = "Security" use_incoming_timestamp = true forward_to = [loki.process.windows_eventlog.receiver] labels = { job = "windows_eventlog", instance = constants.hostname, } } loki.source.windowsevent "system" { eventlog_name = "System" use_incoming_timestamp = true forward_to = [loki.process.windows_eventlog.receiver] labels = { job = "windows_eventlog", instance = constants.hostname, } } loki.source.windowsevent "setup" { eventlog_name = "Setup" use_incoming_timestamp = true forward_to = [loki.process.windows_eventlog.receiver] labels = { job = "windows_eventlog", instance = constants.hostname, } } loki.process "windows_eventlog" { // In rare cases the message is empty (either empty string or nil) // To avoid fill it with "empty_message" stage.template { source = "message" template = `{{- $message := .Value -}} {{- if eq $message "" -}}empty_message {{- else if eq $message nil -}}empty_message {{- else -}}{{- $message -}}{{- end -}}` } // Loki has a builtin parser for windows messages. If it finds a field // in the message which already existed, then it will overwrite its value // instead of creating a new label. stage.eventlogmessage { source = "message" overwrite_existing = true } // at this moment stage.windowsevent is experimental, but soon it will // replace the deprecated stage.eventlogmessage //stage.windowsevent { // source = "message" // overwrite_existing = true //} // Select (as few as possible) fields in an event which should be used // as labels for Loki. // Syntax is: // The symbol on the left side will be a key name in the json. // The symbol on the right side is the name of an existing event field. // "" means, that you expect a field with the same name as the left side // and that you copy its value the the json key. // xy = "abc" means that you expect a field "abc" in the event and you // want key "xy" in the event to get its value. stage.labels { values = { // One of Application, System, Security, Setup // This could also be hard-coded in the loki.source.windowsevent channel = "", // For example, source = Windows-Security-Auditing // If you expect very few sources, you can create a label. // But usually in a normal Windows environment they are so numerous // that they have an impact on Loki's performance. // source = "", } } // We don't want to see Alloy's own logs stage.drop { source = "source" value = "Alloy" drop_counter_reason = "source_alloy" } // By default, an event's timestamp is the time when it was first // ingested by Alloy. This might not be exact enough, // so we parse the original time from the event data and // update the timestamp. stage.timestamp { source = "timeCreated" format = "2025-02-27T17:45:00.0000000Z" } forward_to = [loki.write.endpoint.receiver] } loki.write "endpoint" { endpoint { // CHANGE THIS URL // Use the hostname of your OMD server and // replace the sitename. url ="https://YOUR-OMD-SERVER/YOUR-SITE/loki/api/v1/push" tls_config { insecure_skip_verify = true } basic_auth { // These are the credentials we created with // the htpasswd command. // CHANGE THEM! RUN htpasswd AGAIN! username = "loki" password = "L0ki" } } } ``` When you have saved the file, you have to restart snclient again with **net stop snclient** and **net start snclient**. Now, your Windows event logs are forwarded to Loki via Grafana Alloy. ### Step four - Testing and Searching On the Windows server, open a PowerShell and run the command ```powershell eventcreate /t INFORMATION /id 100 /so MyApp /d "Application started successfully" ``` In Grafana, go to Explore, select the Loki data source, and then enter a LogQL query like {job="windows_eventlog", instance="YOUR_WINDOWS_SERVER_HOSTNAME"} in the query field. The url is *https://YOUR-OMD-SERVER/YOUR-SITE/grafana*. And here is your event! ![loki view](./snclient-alloy-omd-loki-grafana.png) #### Troubleshooting Logs not appearing in Grafana? Check the *C:\Program Files\snclient\snclient.log*, verify the Loki URL, and ensure the SNClient service is running. Make sure there is no firewall blocking the traffic between Windows and OMD. ### Conclusion This setup provides a modern and efficient solution for centralizing Windows Event Logs using state-of-the-art observability tools. By integrating SNClient, Grafana Alloy, and Loki within an OMD environment, logs can be seamlessly collected, processed, and analyzed in a scalable and structured way. This approach eliminates outdated methods such as simple syslog forwarding, offering a robust and future-proof logging pipeline for Windows environments. [Read more](https://omd.consol.de/blog/2025/02/27/forwarding-windows-eventlogs-to-loki-with-alloy/) ----------------------------------------------------------------------------------- title: "Duplicate users in the Grafana database" date: "2025-02-10" -------------------------------------------------------------------------------- This issue occurred on an OMD 5.40 system running Grafana 10.4.2. A customer reported that embedded Grafana panels in the Service Detail View of Thruk were not working. Instead of a graph, they were presented with the Grafana login page. In the *~/var/log/grafana/grafana.log*, the following messages appeared: ``` logger=user.sync t=2025-02-06T15:38:43.735173011+01:00 level=error msg="Failed to fetch user" error="Found a conflict in user login information. 3 users already exist with either the same login or email: [Sepp (email:Sepp, id:124), SEPP (email:SEPP, id:152), sepp (email:sepp, id:182)]." auth_module=authproxy auth_id=SEPP logger=authn.service t=2025-02-06T15:38:43.735241811+01:00 level=error msg="Failed to run post auth hook" client=auth.client.proxy id= error="[user.sync.internal] unable to retrieve user" ``` From the Thruk logs, we determined that users were logging in using different variations of their usernames - sometimes in lowercase, sometimes in uppercase, and occasionally mixed case. This was possible because Apache authenticated users via an LDAP backend that was case-insensitive. However, starting with Grafana 9.3, the software no longer treated differently cased usernames as the same user. (This issue had gone unnoticed for some time and was likely introduced with an OMD update.) ### Enforcing Lowercase Usernames in Thruk To address this, we enforced lowercase usernames in Thruk by setting: ``` make_auth_user_lowercase = 1 ``` _File: `~/etc/thruk/thruk_local.d/users_lowercase.conf`_ After restarting Thruk (`omd restart thruk`), all usernames were internally converted to lowercase, regardless of how users entered them. > **Note:** Login to an OMD monitoring system is done through a Thruk login page. If authentication is successful, the username (now converted to lowercase) is passed to Grafana via the HTTP header `X-WEBAUTH-USER`, and the user is even created in Grafana on the fly. ### Cleaning Up Conflicting Users in Grafana Next, we needed to sanitize the Grafana database by removing redundant user entries that contained uppercase letters. We followed the workflow documented [here](https://grafana.com/blog/2022/12/12/guide-to-using-the-new-grafana-cli-user-identity-conflict-tool-in-grafana-9.3/). First, we listed all conflicting user accounts: ```bash OMD[site1@tmonmuc]:~$ grafana cli \ --config=/omd/sites/site1/etc/grafana/grafana.ini \ --homepath=$HOME/share/grafana \ admin user-manager conflicts list ``` This displayed all users with login conflicts. *SEPP/Sepp/sepp* was one of them. The next step was to generate a conflict resolution file: ```bash OMD[site1@tmonmuc]:~$ grafana cli \ --config=/omd/sites/site1/etc/grafana/grafana.ini \ --homepath=$HOME/share/grafana \ admin user-manager conflicts generate-file ``` The generated file appeared as: ``` /tmp/conflicting_user_1310352894.diff ``` This file contained user conflicts in the following format: ``` conflict: sepp - id: 124, email: Sepp, login: Sepp, last_seen_at: 2024-11-15T11:15:42Z, auth_module: authproxy, conflict_email: true, conflict_login: true + id: 152, email: SEPP, login: SEPP, last_seen_at: 2024-10-22T05:43:19Z, auth_module: authproxy, conflict_email: true, conflict_login: true - id: 182, email: sepp, login: sepp, last_seen_at: 2024-10-22T05:43:19Z, auth_module: authproxy, conflict_email: true, conflict_login: true ``` We needed to edit this file so that each *conflict:* block contained exactly one line with a plus sign (`+`) for the user we wanted to keep (in our case, the lowercase entry with ID 182). The other entries were marked with a minus sign (`-`) to be deleted. Before applying changes, we validated the file: ```bash OMD[site1@tmonmuc]:~$ grafana cli \ --config=/omd/sites/site1/etc/grafana/grafana.ini \ --homepath=$HOME/share/grafana \ admin user-manager conflicts validate-file \ /tmp/conflicting_user_1310352894.diff ``` No errors were found, so we proceeded with the `ingest-file` command: ```bash OMD[site1@tmonmuc]:~$ grafana cli \ --config=/omd/sites/site1/etc/grafana/grafana.ini \ --homepath=$HOME/share/grafana \ admin user-manager conflicts ingest-file \ /tmp/conflicting_user_1310352894.diff ``` ### Unexpected Error and Solution However, after confirming *Proceed with operation?* with *Y*, we encountered an error: ``` Error: ✗ not able to merge with &{%!e(string=could not find intoUser: Found conflict in user login information. 3 users already exist with either the same login or email: [Sepp (email:Sepp, id:124), SEPP (email:SEPP, id:152), sepp (email:sepp, id:182)]}) ``` This indicated that Grafana still treated `sepp`, `Sepp`, and `SEPP` as the same user, likely due to internal case-insensitive handling. To resolve this, we explicitly configured Grafana to respect case differences by adding the following line to *grafana.ini*: ```ini [users] case_insensitive_login = false # Temporarily enforce case sensitivity allow_sign_up = false default_theme = light ``` _File: `~/etc/grafana/grafana.ini`_ With this setting in place, we successfully ingested the conflict resolution file: ```bash OMD[site1@tmonmuc]:~$ grafana cli \ --config=/omd/sites/site1/etc/grafana/grafana.ini \ --homepath=$HOME/share/grafana \ admin user-manager conflicts ingest-file \ /tmp/conflicting_user_1310352894.diff ... conflicts resolved. ``` After the `ingest-file` command ran successfully, we removed the `case_insensitive_login = false` line from `grafana.ini` to restore the original setting. ### Conclusion By enforcing lowercase usernames in Thruk and making Grafana temporarily case-sensitive, we successfully eliminated duplicate user entries and restored embedded panel functionality. This issue highlights the importance of consistent username handling across authentication layers to avoid unexpected conflicts. [Read more](https://omd.consol.de/blog/2025/02/10/duplicate-users-in-the-grafana-database/) ----------------------------------------------------------------------------------- title: "Replacing NSClient++: A Comparison of Current Alternatives" date: "2025-01-13" -------------------------------------------------------------------------------- Many of you have probably wondered how to best replace NSClient++ with a modern tool. René Gau from ASFINAG has written a [bachelor’s thesis](https://onlinecampus.fernfh.ac.at/pluginfile.php/223278/mod_data/content/298735/Gau_Rene_00230036_SS24_Jungbauer_2024-06-10.pdf) on this very topic, providing a detailed analysis of the pros and cons of the most relevant alternatives. Here’s a little spoiler: if you want to avoid vendor lock-in and exorbitant migration or licensing costs, this [website](https://omd.consol.de/docs/snclient/) has all the information you need!" [Read more](https://omd.consol.de/blog/2025/01/13/replacing-nsclient-a-comparison-of-current-alternatives/) ----------------------------------------------------------------------------------- title: "missing perl module Digest::SHA1 under Rocky 9" date: "2025-01-10" -------------------------------------------------------------------------------- Some of you might have followed the instructions at [https://labs.consol.de/repo/testing/#_9](https://labs.consol.de/repo/testing/#_9) to install the labs.consol.de repository. If you're using a minimal setup of Rocky Linux 9, you might encounter this error message: ```bash # dnf install omd-5.51.20250109-labs-edition.x86_64 Last metadata expiration check: 0:08:01 ago on Fri 10 Jan 2025 10:32:09 AM CET. Error: Problem: package omd-5.51.20250109-labs-edition-el9-1.x86_64 from labs_consol_testing requires perl-Net-SNMP, but none of the providers can be installed - conflicting requests - nothing provides perl(Digest::SHA1) >= 1.02 needed by perl-Net-SNMP-6.0.1-25.el8.1.noarch from epel (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) ``` The missing Perl module *Digest::SHA1* is included in the installation package *perl-Digest-SHA1*, which is not part of a minimal setup. Moreover, running ***dnf search perl-digest-SHA1*** yields no results. To resolve this, you need to enable the *PowerTools* repository, which contains this RPM. You can do this by running the following command: ```bash # dnf config-manager --enable powertools # Rocky 8 and later/Alma 8 # dnf config-manager --enable crb # AlmaLinux 9 and later ``` [Read more](https://omd.consol.de/blog/2025/01/10/missing-perl-module-digestsha1-under-rocky-9/) ----------------------------------------------------------------------------------- title: "Removing old Naemon logfiles" date: "2024-06-05" -------------------------------------------------------------------------------- In an OMD environment Naemon writes its logfile to *var/naemon/naemon.log*. At midnight a logrotate job moves the logfile to the folder *var/naemon/archive*, renaming *naemon.log* to *naemon.log-YYYMMDD*. By default none of the archived logs is ever deleted. If your disk space is limited, you may want to discard old logs after some months or years. Create a file *~/local/bin/purge_naemon_archive* with the following content. ```bash #!/bin/bash archive_dir=$OMD_ROOT/var/naemon/archive today=$(date +%Y%m%d) days_to_keep=${CONFIG_NAEMON_ARCHIVES_RETENTION:-365} # the oldest day in yyyymmdd format cutoff_date=$(date -d "$today - $days_to_keep days" +%Y%m%d) for file in "$archive_dir"/naemon.log-* do # extract the date from the filename file_date=$(basename "$file" | cut -d '-' -f 2) # if the file's date is older than the cutoff date, delete it # (yyyymmdd format allows integer operations) if [[ "$file_date" -lt "$cutoff_date" ]] then rm "$file" fi done ``` Make it executable with: ```bash chmod 755 ~/local/bin/purge_naemon_archive ``` Then create a cronjob by writing this line into the file *~/etc/cron.d/purge_naemon_archive*: ``` 10 0 * * * $OMD_ROOT/local/bin/purge_naemon_archive ``` Finally, restart the cron daemon: ```bash omd restart crontab ``` Now archived logfiles will be deleted if they are older than a year. You can define your own retention period by setting the environment variable *CONFIG_NAEMON_ARCHIVES_RETENTION* to the number of days you want to keep the log files. (You can add the variable to *~/etc/environment*) [Read more](https://omd.consol.de/blog/2024/06/05/removing-old-naemon-logfiles/) ----------------------------------------------------------------------------------- title: "Replacing check_http by check_curl - Lessons Learned" date: "2023-07-28" -------------------------------------------------------------------------------- In a customer's installation we had the problem, that an http-check (using the plugin check_http) could not successfully connect to a website. ``` $ check_http -H cts-freq.cloud -t 120 --ssl -u "/" --ssl --onredirect follow -vvv CRITICAL - Cannot make SSL connection. 140484279109552:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769: SSL initialized ``` A test with check_curl showed that this newer plugin was functioning as expected. ``` $ check_curl -H cts-freq.cloud -t 120 --ssl -u "/" --ssl --onredirect follow -vvv HTTP OK: HTTP/1.1 200 OK - 449 bytes in 0.605 second response time |time=0.604714s;;;0.000000;120.000000 size=449B;;;0 ``` As check_curl was designated as the successor for check_http, we decided to decommission check_http and use check_curl everywhere in the configs. Unfortunately, it turned out that it didn't go as smoothly as expected. This was the configuration we were starting with: ``` define command { command_name check_curl_ssl command_line $USER1$/check_curl -H $ARG1$ -t 120 --ssl -u "$ARG2$" $ARG3$ } define command { command_name check_http_ssl command_line $USER1$/check_http -H $ARG1$ -t 120 --ssl -u "$ARG2$" $ARG3$ } ``` The first problem that caught our attention was with a url containing a hostname that resolved to both an IPv4 and an IPv6 address through DNS. Since we were operating in a pure IPv4 environment and check_curl tool attempted to contact the IPv6 address, the result was an error. Adding the parameter _\-\-use-ipv4_ resolved this issue. The next problem was puzzling. The web server responded to a request through check_curl with the message *HTTP WARNING: HTTP/1.1 403 Forbidden*. However, a request through check_http returned the expected *HTTP OK: HTTP/1.1 200 OK*. It turned out that the web server was configured by its operators to only grant access to selected user agents. So we added _\-\-useragent "check_http"_ and the problem was solved. And here is the final configuration: ``` define command { command_name check_curl_ssl command_line $USER1$/check_curl \ -H $ARG1$ -t 120 \ --useragent "check_http" --use-ipv4 \ --ssl -u "$ARG2$" $ARG3$ } define command { command_name check_http_ssl command_line $USER1$/check_http \ -H $ARG1$ -t 120 \ --ssl -u "$ARG2$" $ARG3$ } ``` [Read more](https://omd.consol.de/blog/2023/07/28/replacing-check_http-by-check_curl-lessons-learned/) ----------------------------------------------------------------------------------- title: "Image Change Triggers for Tekton" date: "2022-03-14" --------------------------------------------------------------------------------
One of our customers is in the process of decommisioning their OpenShift v3.11 cluster. This cluster is currently still used for building customer specific base images. Over time quite a few elaborated pipeline builds (based on Jenkins) have been developed for that purpose. The customer wanted me to migrate the existing pipeline builds on their v3.11 cluster to Tekton (aka OpenShift Pipeline) builds running on their new v4.9 cluster. This task turned out to be quite pesky. Tekton is a beast in many aspects. For instance the base images that we want to build are in some cases dependent on each other. In other words we want one newly built base image to trigger the build(s) of other base image(s). Turned out there is nothing in Tekton that compares to image change triggers in BuildConfigs. We did some brainstorming internally and in parallel got in contact with Red Hat. We ended up with two options: * Conventional BuildConfigs starting pipeline builds by means of a custom build. * Taking advantage of the custom resource ApiServerSource that comes with Knative Eventing. This resource is able to transform change events from arbitrary resources into HTTP requests that in turn can be sent to some EventListener as provided by Tekton. A Red Hat engineer named Andrew Block has published this solution on [GitHub](https://github.com/sabre1041/image-trigger-tekton). Both options have their drawbacks. Custom builds as needed for the first option are [poorly](https://docs.openshift.com/container-platform/4.9/cicd/builds/understanding-image-builds.html#builds-strategy-custom-build_understanding-image-builds) [documented](https://docs.openshift.com/container-platform/4.9/cicd/builds/build-strategies.html#builds-strategy-custom-build_build-strategies). The second option requires installation of two bulky products (Knative and Knative Eventing) just to get the desired resource ApiServerSource. Our customer currently has no demand for Knative and regarded the second option as *breaking a nut with a sledgehammer*. So I focused on option one. ## Basic idea The basic idea is to use conventional BuildConfigs and take advantage of their image change trigger feature. An image change triggers a custom build (i.e. a builder pod with a custom image is started) that in turn creates an instance of PipelineRun, i.e. a pipeline build is started. ## Evolution To my surprise I didn't encounter any major obstacles. The approach worked to my satisfaction right from the beginning. It worked so well that I decided to tear down all the resources that were already in place just to get HTTP endpoints for source change triggers: EventListeners, TriggerTemplates, TriggerBindings, Triggers. I was able to get rid of all this bloat just by adding a few lines to my BuildConfig. A second improvement was to create an owned-by relation between the PipelineRun resource that was created by the custom build and the Build resource itself. It was thus possible to later just delete the Build resource only and have OpenShift take care of deletion of all depending resources, i.e. PipelineRun and all Pods running tasks of the pipeline. Another evolutionary step was to aggregate the output of all the various containers started in the course of one single pipeline run in the custom build. This way is became possible to display the output of the build with `oc logs build/...` or even do `oc start-build --follow`. No need to install the `tkn` CLI tool for Tekton. A final extra is due to the special workflow applied by the customer. New base images aren't promoted right away but require a review and approval step. Not yet approved base images are stored in a separate ImageStream with the tag *to-be-approved*. Nevertheless, these *to-be-approved* images may trigger the builds for other base images by means of image change triggers. However, when the same base image builds get triggered by a source code change the underlying base image needs to be an already approved one as the *to-be-approved* version may not be available at all, or it may be still in the approval process, or it may not have been approved for good reasons. So the following extra logic was coded into the custom build: In case the custom build has been triggered by an image change trigger this image is determined and used as the base image for the pipeline build to be started. In all other cases the base image specified in the Dockerfile by the `FROM` clause is used. (For further evolutionary steps see the update sections at the end of the article.) ## Details **Disclaimer**: The following explanations sometimes assume some basic knowledge of Tekton pipelines and tasks. Apologies for any confusion this may cause for the uninitiated. I will demonstrate the approach with a relatively simple use case. A build for a customized base image `java11-custom` that is derived from a customized UBI8 base image `ubi8-custom`. The Tekton pipeline only consists of two tasks - *git-clone* and *buildah* that are readily provided by OpenShift Pipelines. The pipeline is named *simple*. (It's basically the Tekton analogon of a BuildConfig with Docker strategy.) ### BuildConfig Let's start with the BuildConfig that is at the beginning of each build: ```yaml apiVersion: build.openshift.io/v1 kind: BuildConfig metadata: name: start-java11-pipeline spec: runPolicy: Serial serviceAccount: pipeline source: binary: {} type: Binary strategy: customStrategy: env: - name: IMAGESTREAM value: java11-custom - name: GIT_URL value: git@:/java-custom.git - name: GIT_REVISION value: jdk11 - name: NAME_PREFIX value: java11 from: kind: ImageStreamTag name: simple-pipeline-starter:latest type: Custom triggers: - imageChange: from: kind: ImageStreamTag name: ubi8-custom:to-be-approved type: ImageChange - generic: secretReference: name: webhook-token type: Generic ``` There are several details to point out. First of all this BuildConfig specifies a custom build (i.e. `spec.strategy.type: Custom`). The build pod is started with a custom image (`spec.strategy.customStrategy.from.name: simple-pipeline-starter:latest`). I'll get back to this builder image soon. The Tekton pipeline that ultimately will be started is rather generic. The concrete details about what to build and how and where to store it are determined by a bunch of environment variables specified in `spec.strategy.customStrategy.env`. (Not all possible variables are shown here. Some have reasonable defaults so that they can be ommitted.) It is necessary to start the builder image with credentials of the service account *pipeline* (`spec.serviceAccount: pipeline`). Only this way it is possible to create an instance of PipelineRun, i.e. start a Tekton pipeline. The `spec.triggers` section is the reason why this all came into being. There is a source change trigger and an image change trigger. The image change trigger monitors the ImageStreamTag `ubi8-custom:to-be-approved` which is another customized base image. As soon as this customized UBI8 image has been newly built the above BuildConfig triggers a new build for image `java11-custom`. (Mind the tag name `to-be-approved` indicating the customized UBI8 image has not been approved yet. Nevertheless, the `java11-custom` base image is built based on this customized UBI8 image.) ### Builder image The builder image for the custom build itself is built with a conventional BuildConfig with Docker build strategy. The Dockerfile looks like: ``` FROM registry.redhat.io/openshift4/ose-cli:v4.9 ARG tkn_dl_link=https://mirror.openshift.com/pub/openshift-v4/clients/pipeline/0.19.1/tkn-linux-amd64-0.19.1.tar.gz COPY build.sh /usr/bin/build.sh RUN yum -y update && yum -y install jq && yum clean all && \ ( curl $tkn_dl_link | tar -C /usr/bin -xzf - tkn ) && \ chmod 755 /usr/bin/build.sh ENTRYPOINT [ "/usr/bin/build.sh" ] ``` The builder image is based on the Red Hat provided image `ose-cli:v4.9` enabling to run the `oc` CLI tool within a container. The Dockerfile additionally installs the `tkn` CLI tool of Tekton and `jq`, a command line tool to deal with JSON formatted data. These three tools `oc`, `tkn` and `jq` are important building blocks for the actual build script `build.sh` that is also added to the builder image and specified as entrypoint. Let's have a look at it: ```bash #!/usr/bin/bash # extract image that caused image change trigger TRIGGERED_BY_IMAGE=$(echo "$BUILD" | jq -j '.spec.triggeredBy[0].imageChangeBuild.imageID') [ "$TRIGGERED_BY_IMAGE" != "null" ] && FROM_IMAGE="$TRIGGERED_BY_IMAGE" if [ -n "$FROM_IMAGE" ]; then # ... and use it as FROM in buildah build BUILD_EXTRA_ARGS="${BUILD_EXTRA_ARGS:+${BUILD_EXTRA_ARGS} }--from $FROM_IMAGE" fi BUILD_DATE=$(date --utc '+%Y-%m-%dT%H:%M:%SZ') BUILD_EXTRA_ARGS="${BUILD_EXTRA_ARGS:+${BUILD_EXTRA_ARGS} }--label build-date=${BUILD_DATE}" BUILD_KIND=$(echo "$BUILD" | jq -j '.kind') BUILD_API_VERSION=$(echo "$BUILD" | jq -j '.apiVersion') BUILD_NAME=$(echo "$BUILD" | jq -j '.metadata.name') BUILD_UID=$(echo "$BUILD" | jq -j '.metadata.uid') PR_NAME=$(oc create -f - << __EOF__ | sed 's/pipelinerun\.tekton\.dev\/\(.*\) created/\1/' apiVersion: tekton.dev/v1beta1 kind: PipelineRun metadata: generateName: ${NAME_PREFIX:-${IMAGESTREAM}}- labels: tekton.dev/pipeline: simple ownerReferences: - apiVersion: "${BUILD_API_VERSION}" kind: "${BUILD_KIND}" name: "${BUILD_NAME}" uid: "${BUILD_UID}" spec: params: - name: dockerfile value: ${DOCKERFILE:-Dockerfile} - name: git-revision value: ${GIT_REVISION:-master} - name: git-url value: ${GIT_URL} - name: imagestream value: ${IMAGESTREAM} - name: build-extra-args value: "${BUILD_EXTRA_ARGS}" - name: build-name value: ${BUILD_NAME} pipelineRef: name: simple serviceAccountName: pipeline timeout: 1h0m0s workspaces: - name: ssh-dir secret: secretName: ssh-files - name: shared-workspace volumeClaimTemplate: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: source-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Mi __EOF__ ) tkn pipelinerun logs --follow "$PR_NAME" COMPLETION_REASON="$(oc get pipelinerun/${PR_NAME} -o jsonpath='{.status.conditions[0].reason}')" test "$COMPLETION_REASON" == "Succeeded" && exit 0 test "$COMPLETION_REASON" == "Completed" && exit 0 exit 1 ``` The principal functionality is to create an instance of PipelineRun (`oc create -f -`). The output of this command is processed with `sed` to extract the name of the PipelineRun. (It is stored in `$PR_NAME`.) Tekton's CLI tool `tkn` is then used to display the output generated by the various containers started by this PipelineRun. It is thus possible to display the output with `oc logs build/` or watch the build output when started with `oc start-build start-java11-pipeline --follow`. The input to `oc create -f -` is provided by means of a so-called [here document](https://www.gnu.org/software/bash/manual/bash.html#Here-Documents) - anything between `oc create -f - << __EOF__` and `__EOF__`. It defines the YAML code of the PipelineRun resource to be created. A custom build (as any other build derived from a BuildConfig) is represented by a Build resource. For custom builds the corresponding Build resource is provided to the build process by the environment variable `$BUILD` as JSON data. `build.sh` takes advantage of this information in two ways. First, the build script extracts the URL of the image that was changed and triggered this build (`$TRIGGERED_BY_IMAGE`). In case this build was actually triggered by an image change (`"$TRIGGERED_BY_IMAGE" != "null"`) this image is used as the base image for the following PipelineRun (overriding any base image specified by the `FROM` clause in the Dockerfile). This is achieved by appending the option `--from ` to the variable `$BUILD_EXTRA_ARGS` that is handed down to `buildah` by means of a corresponding pipeline and later task parameter. Second, the build script extracts data from the JSON data about the Build instance itself (`BUILD_KIND`, `BUILD_API_VERSION`, `BUILD_NAME`, `BUILD_UID`) and adds it into the PipelineRun YAML (`metadata.ownerReferences`). This way an owned-by relation between the PipelineRun instance and the Build instance is introduced. As a consequence when the Build instance is deleted OpenShift automatically takes care of also deleting the PipelineRun instance *owned by* this Build instance (and also all other resources created by the PipelineRun). ### Relations and namings For this setup each Tekton pipeline requires a corresponding builder image, not only because of the reference to the pipeline in the build script (`spec.pipelineRef.name`) but also due to the pipeline specific set of parameters (`spec.params`) and the workspaces required by the pipeline (`spec.workspaces`). I've introduced the naming convention *<pipeline-name>-pipeline-starter* (e.g. *simple-pipeline-starter*) for the custom build images. The concrete usage of a Tekton pipeline is modelled by BuildConfigs specifying values of pipeline parameters. These are named by the pattern *start-<prefix>-pipeline* where *<prefix>* is prepended to the name of the to-be-created PipelineRun instance. It is automatically also used as a prefix for the names of all the pods that are started by the PipelineRun. ``` start-java11-pipeline BuildConfig └ start-java11-pipeline-7 Build └ start-java11-pipeline-7-build Build pod └ java11-fsr4k PipelineRun ├ java11-fsr4k-fetch-repository-cghx4-pod-b8r72 Tekton task pod └ java11-fsr4k-build-image-lvgnn-pod-56wdb Tekton task pod ``` ## Benefits I see five benefits of the outlined solution: 1. **Image change triggers**: Tekton simply lacks any support for image change triggers. With this approach you get them without much overhead. 2. **Source code change triggers**: The set of resources offered by Tekton for triggering PipelineRuns by arbitrary HTTP requests is very powerful. Nevertheless, it has to be stated that configuring such a trigger is tedious - if not painful. It involves an EventListener, a TriggerTemplate, a TriggerBinding and a Trigger. With the above approach setting up a source change trigger becomes a matter of 4 lines of YAML code in one BuildConfig. 3. **Easy starting of builds**: With one BuildConfig for each concrete usage of a Tekton pipeline spinning up a PipelineRun becomes a matter of `oc start-build ` (or a click on *Start build* in the web UI). In contrast, starting a Pipeline with `tkn pipeline start` tends to become cumbersome. 4. **Easy access to build log output**: The build log output in Tekton is scattered on (potentially) many containers in quite a few pods that are started by one PipelineRun. The custom build takes advantage of `tkn pipeline logs` thus replicating the aggregated build log in the context of the build. As a consequence the complete build log can be displayed with `oc logs`. You can even watch the build output in real time with `oc start-build --follow`. (Sidenote: You don't need `tkn` all if you just want to start and observe builds.) 5. **Customized build logic**: To a certain degree it is possible to adapt the custom build to specific needs, workflows, conventions as demonstrated above with the *to-be-approved* tag logic. (See also *Update (2022-05-30)* below.) ## Update (2022-04-21) Originally the above described BuildConfig used a webhook of Type *GitLab*. This caused a subtle bug with regard to source code change triggers. As [OpenShift's documentation](https://docs.openshift.com/container-platform/4.9/cicd/builds/triggering-builds-build-hooks.html#builds-webhook-triggers_triggering-builds-build-hooks) states: > When the push events are processed, the OpenShift Container Platform control plane host confirms if the branch reference inside the event matches the branch reference in the corresponding BuildConfig. If so, it then checks out the exact commit reference noted in the webhook event on the OpenShift Container Platform build. If they do not match, no build is triggered. Unfortunately the BuildConfig references no git repository and no branch to get its input from. It simply starts a custom build with no input except some environment variables. Thus OpenShift assumes that git branch *master* is relevant. Consequently only POST requests caused by push events on branch *master* ultimately trigger a build. All other branches cannot be used for source code change triggers. The workaround is to use a webhook of type *Generic*. With this type of webhooks the request body of the POST request is not taken account. As long as the webhook token in the URL matches the token in the secret referenced in the BuildConfig a build is triggered. ## Update (2022-05-12) The final status of the Builds created by the BuildConfig originally did not reflect the outcome of the underlying PipelineRun. In other words: Even when the PipelineRun ran into some problem and resulted in Status *Failed* the corresponding Build always showed *Complete*, not *Error*. The fix was to make sure that the build script `build.sh` terminates with a return code 0 in case everything was fine or a return code of 1 (or any other value not equal 0) in case some problem occurred. To achieve this I evaluate the status of the PipelineRun. When the `reason` field was set to *Succeeded* the script returns with 0, in all other cases with 1. See the last to lines of the above script `build.sh`. ## Update (2022-05-30) I've introduced a conditional step to the Pipeline *simple*: It sends email notifications in case any problem occurred during the PipelineRun. This had the effect that the completion reason for the good case (no problems during PipelineRun, no failure notification) changed from *Succeeded* to *Completed*. The build script `build.sh` has been adapted accordingly. Additionally our customer requested each image to be labelled with the build date. This has also been added to the build script. ## Final words My thanks go to Gerd Beyer for his initial idea. Any comments, additions, corrections are welcome: <firstname>.<lastname> at consol.de [Read more](https://omd.consol.de/blog/2022/03/14/image-change-triggers-for-tekton/) ----------------------------------------------------------------------------------- title: "A look insight Camel K" date: "2022-02-24" -------------------------------------------------------------------------------- Today software often needs to be run in cloud environments. Newly developed software, especially microservices are developed with cloud readiness in mind. But we not only have microservices in business environments, we also have integration software. This type of software is developed and designed to connect external services to internal ones. To make this kind of software cloud ready and to make it an ease to deploy them in cloud environments Camel K came up. In this article we will have a look at how Camel K works and what are the concepts behind it. ## Introduction As mentioned in the short introduction, one of the big goals in today's software development is to have the software "cloud ready". This includes packaging and sizing, as startup time becomes important within the cloud, as well as making it run in a container environment such as docker. For most modern architectures like microservices this is already the case. A special kind of software indeed is the integration software. The goal of such a software system is to integrate or connect external services to local ones. Therefore, often data conversions are needed. Assuming the following situation: You have an external service sending its data to one of your own services. The incoming data is in XML format, but your systems all work with json as an exchange format. So the incoming data has to be transformed in order to be useful for your own systems. What you can do is to set up a microservice as the entrypoint to your system which converts the incoming data to the needed json format. Then call the next service and so on. But with an integration software framework you can do better. Integration Software is tailored for converting and transforming data and to connect several systems with each other. Two famous Integration Frameworks are [Apache Camel](https://camel.apache.org) and [Spring Integration](https://spring.io/projects/spring-integration). Both of them can also be used with the famous Spring Boot Framework. In this article we will have a brief look at Apache Camel, as this is the technology where Camel K is based on. If you are already familiar with Camel you can skip the next paragraphs and start at [Kamel K](#camel-k). ## Camel Apache Camel was initially released on June 27, 2007 and has come a long way since then. The idea behind it is to have a framework with a lot of components to make it easy to connect to standardized interfaces. So for example there are components for HTTP (inbound and outbound), FTP, Mail and many more. For the complete list visit [Camel Components Reference](https://camel.apache.org/components/3.15.x/). ### Basic Camel Architecture First I will start with a short overview of the basic Camel architecture and a short explanation of each single part. [](./camel-architecture.png) *Camel Architectural Overview from [https://camel.apache.org/manual/architecture.html](https://camel.apache.org/manual/architecture.html)* The main component of Camel is the Camel Context. It holds up all the features which Camel has to offer e.g. * components * routes * processors * beans So when you start a Camel application you have to initialize the context. If you run a standalone Camel application you have to initialize the context by your own. If you use Spring Boot with the various Camel Starters the framework will do this for you. Once the context is initialized the routes has to be added. In standalone mode this must be done by yourself or in SpringBoot this will happen automatically as long as you define your RouteBuilder in a Class which is annotated with *@Configuration*. The Camel context also holds the routing engine which is responsible for running the routes. ### A Camel Route The following graphic shows an example workflow for a Camel route. [](./camel_route_example.png) *Schema of the example route* In the example we start with a http call from system A in the top left corner. The system sends a POST request with a xml body to an endpoint provided from the Camel example application. The format of the body is the following: ```java hello {{count}} ``` *XML Body of the incoming message* The incoming request is transformed to an internal Camel object, called *Exchange*. An Exchange includes several fields and objects. The main components are: ExchangeID: : the unique ID of this exchange MEP: : the pattern used for this exchange, can be InOnly or InOut. InOnly does not return a response, where InOut does. Exception: : if there occurs an exception during the processing of this exchange it will be stored in this field Properties: : key/value pairs for storing information which is bound to this exchange InMessage: : the information coming from the originating input data. In our example the request body will be stored in the message body, the request headers will be stored in the message headers OutMessage: : the message going out to the next step of the execution After the incoming data is transformed to an exchange, this exchange is routed to the next step in the process, in our example to the XPath extraction step. Within this an XPath Expression is evaluated against the incoming message body which is basically the payload from the incoming http request. The result from the XPath evaluation is the message string sent from the client. This string is the new body from the out message of the exchange and will be the input for the next step. The next step is the concatenation of the log output. This will then be the input parameter for the log component which is responsible for logging to the given logger. Here you can see the source code for this process: ```java import org.apache.Camel.Header; import org.apache.Camel.builder.RouteBuilder; import java.time.Clock; import java.time.LocalDateTime; import java.time.ZoneOffset; public class Example extends RouteBuilder { public void configure() throws Exception { from("netty-http:http://0.0.0.0:8080/myapp/myservice") .setHeader("incomingTimestamp", method(LocalDateTime.class, "now()")) .convertBodyTo(String.class) .setBody(xpath("/message", String.class)) .to("seda:secondRoute"); from("seda:secondRoute") .setHeader("duration", method(this, "calculateDuration")) .log("Message '${body}' arrived at ${headers.incomingTimestamp} and took ${headers.duration} ms "); } public long calculateDuration(@Header("incomingTimestamp") LocalDateTime incomingTimestamp) { final ZoneOffset offset = Clock.systemDefaultZone().getZone().getRules().getOffset(incomingTimestamp); return System.currentTimeMillis() - incomingTimestamp.toEpochSecond(offset) * 1000; } } ``` *Java Source Code of the example route(s)* You can see in the code that in Camel terms an integration process is called *Route*. This route is configured in a *RouteBuilder*. Every route starts with an *from* clause. Within the *from* you specify the source of this integration. This is always a Camel component which acts as a so called *Consumer* as it consumes data and converts it to something meaningful for camel. You can also see that the first route end with a *to* clause, which is most times the case. The *to* clause also specifies a camel component which than acts as a so called *Producer* as it produces objects from the internal Exchanges. So you can see that we use the *netty-http* component which sets up an incoming HTTP Endpoint listening on localhost on port 8080 and the path */myapp/myservice*. In the second line of the route definition we are using the message headers, which already contain some HTTP Headers mapped from the incoming message to Camel headers, to store some other information. In this case we use it for storing the receiving timestamp of the message in order to log the message processing duration at the end of the route. Going on with the next line we convert the message body to a string because as of now the body was an input stream which could only be read once. On this string we perform the XPath expression to extract the message string from the request and set it as the new body of our exchange. After that we route to a second route which is also defined in this same RouteBuilder. This second route starts with the *seda* Camel component which is an asynchronous way to transfer data within the same Camel context. You can imagine the *seda* component as a non-blocking method call on an arbitrary java object. In this second route we also use a header to store a second information which we want to print out in the log - the calculated message processing duration. This time we use the bean component to call an arbitrary java method. To make it easier, especially in Camel K, we define the method to be called in our RouteBuilder itself. This way no external dependency is needed. The last step of the route is to concatenate the log message with the message body and the information stored in the message headers and use this as the input for the *log* component which logs the message simply to *System.out*. --- ## Camel K ## Having looked at the very basics of Camel, we will come now to the main topic of this article - Camel K. In the next subsection we will start with the installation of Camel K with the help of the *kamel* cli. Afterwards we are going to deploy our Camel example route and have a look at what happened in the cluster. If you want to follow the steps shown here, you will have to install [Minikube](https://kubernetes.io/de/docs/tasks/tools/install-minikube/). All the commands will be sent by using `kamel` or `kubectl`. If you are using Red Hat CRC or something else, you have to look for the right command by your own. ### Installation Before you are going to install Camel K you first have to set up your `kubctl` environment. This can be done by using a development cluster like [Minikube](https://kubernetes.io/de/docs/tasks/tools/install-minikube/) or you can set up your `kubectl` command to connect to an existing external cluster. After that configuration installing Camel K is really straight forward and simple. All you have to do is to download the [*kamel* cli](https://github.com/apache/camel-k/releases), put it in a directory which is on your path and issue the following command: ``` kamel install ```

Info: The installation command will be run against the cluster you are currently connected to.

If you get errors check the [Installation Instructions ](https://camel.apache.org/Camel-k/1.8.x/installation/installation.html). You might have to do some extra work beforehand, depending on the cluster type you are using. When you issue `kubectl get pods` you should see that a new operator pod for Camel K was deployed and that this pod should be in running state. If not, you should check the logs for error messages. ### Running an Integration After the successful installation, we should run our Camel route within Camel K. This is also as straightforward as the installation was. Just create a *Example.java* file with the contents from above. Open a terminal in the same folder as the java file is located and run: ``` kamel run Example.java --dev ``` With the `--dev` your terminal get attached to the log output of the running integration. Furthermore, all changes that you make to the source code will reflect to a change in the running pod.

Info: With the cli you can also add Traits or dependencies to your Integration (if necessary).

You should see the log output of the pod which is spinning up. Depending on the speed of your cluster this might take a moment to finish. After the pod is started you should see something like this: [](./camel-k-startup.png) *Example Route startup log* As the integration is now successfully deployed, we are able to issue the first request. Therefore, we have to get the port on which our application is listening for incoming requests. This can be done with the following command: ``` kubectl get service ``` The output should look like something like this: [](./route-service-port.png) *The port our Integration is listening at* In the above picture you can see that the port for this application is 32522 and that the type is a container port. This means by sending request to this port, the request will be handled by this particular container. So we have to send our request to ``` http://CLUSTER_IP:32522/myapp/myservice ``` Here is the curl command you can send to the server to see if the integration is running as expected: ``` curl --location --request POST 'http://CLUSTER_IP:32522/myapp/myservice' \ --header 'Content-Type: text/plain' \ --data-raw ' hello 1' ``` Don't forget to replace *CLUSTER_IP* with the actual IP address or name of your cluster. And that's how the output in the log should look like: [](./example-message-log-output.png) After we have the integration running we should have a look under the hood to see how things work together. ### Camel K Architecture To get a good impression about how many components are building up Camel K, we should take look at the following architectural overview from the Camel K website: [](./camel-k-high-level.svg) *Architectural overview from [https://camel.apache.org/Camel-k/1.8.x/architecture/architecture.html](https://camel.apache.org/Camel-k/1.8.x/architecture/architecture.html)* We will focus on the following components, as they are the most important ones: * Camel K Operator * Traits * Integration * IntegrationPlatform * IntegrationKit * Build * Camel Catalog * Camel K Runtime #### Camel K Operator The Camel K Operator is implemented as a kubernetes operator and therefore it follows the controller pattern. This pattern is based on the idea that the operator will try to get the resources, for which it is responsible for, in a desired state. Therefore, it will apply changes, monitor and reconcile the resource until this state is reached. When the state is reached it will go along monitoring the resources and check if there are any changes need to be applied. [](./operator-pattern.png) *Operator Pattern* The operator is responsible for creating and gathering all the resources needed for the *Camel K Runtime* to do their work. After the Camel K Runtime has build the container, the operator will create all the necessary kubernetes resources and will observe their state. #### Traits Traits are build-in Camel K Features which will be applied to the Camel route. You can define the traits you want to apply to your route by setting them as an argument to the `kamel run` call. For example if you want to apply the *jolokia* trait to use a jolokia client within your route you can do so with the following command: ``` kamel run --trait jolokia.[key]=[value] --trait jolokia.[key2]=[value2] integration.groovy ``` You can also add this to your java file, as a so called *modeline*. ```java // Camel-k: jolokia.[key2]=[value2] import org.apache.Camel.Header; import org.apache.Camel.builder.RouteBuilder; import java.time.Clock; import java.time.LocalDateTime; import java.time.ZoneOffset; public class Example extends RouteBuilder { public void configure() throws Exception { from("netty-http:http://0.0.0.0:8080/myapp/myservice") ... ``` There are several traits which will automatically be applied to your route. These are traits which are essential to run your route on the cluster type you are on. These so called *standard traits* exist for every supported cluster type e.g. kubernetes, knative and Openshift. The information which is needed to decide which trait has to be applied is served by the *IntegrationPlatform*. A full list of the available traits can be found under [https://camel.apache.org/Camel-k/1.8.x/traits/traits.html](https://camel.apache.org/Camel-k/1.8.x/traits/traits.html). #### IntegrationPlatform The *IntegrationPlatform* is a so called *CR* (Custom Resource). Custom Resources are a mechanism to extend kubernetes. You can find more information on [https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). The IntegrationPlatform resource is responsible for storing common configuration values which the operator needs to work and which is applied to every route. The following values are stored: * The cluster type we are running on * Standard traits which should be applied * The builder which should be used to build the container (e.g. buildah, kanico, spectrum) * Which maven registry should be used to get necessary dependencies * Which container registry should be used #### Integration The *Integration* is the resource which represents the actual Camel application. It holds the following values: * The source code given to the kamel cli * all resources necessary to run the route (properties, ConfigMaps, Builder etc.) * all the dependencies which are necessary (Maven Dependencies, external dependencies like repositories) * The traits which should be applied to this particular route (the standard ones, as well the ones defined as an argument to the cli, even as trait or parameter to the kamel call) #### IntegrationKit The *Integration Kit* is a side resource which is used in combination with the container image which is the output of the operator run. It is needed to run the container image as it holds the run configuration for this particular image which means: * Properties * Secrets * ConfigMaps * Resources and Dependencies * Trait Configuration for that particular image #### Build The *Build* resource describes the process of combining the *Image*, *Integration Kit* and the *Integration* to get a running container. [](./camel-k-build.png) *The build process* #### Camel Catalog The *Camel Catalog* holds the necessary information for the Camel K Runtime. It stores the following: * which components are available * which languages are supported (java, groovy etc.) * which Camel DataConverters and DataFormats are available These are some information which in a regular Camel application is stored in the *Camel Context*. #### Camel K Runtime The *Camel K Runtime* is a java based application which gets called from the *Operator*. It is responsible for converting all the resources supplied by the operator (Traits, ConfigMaps, Properties, Secrets etc.) to something which can be understood by the regular Camel runtime which will execute the code within the container. After the conversion is done, it will build the image in which the Camel code runs. This is done with the help of *Camel Quarkus*. After the image is build it is pushed to the *Container Registry* of the cluster to be available for the regular kubernetes build. ### Startup of an integration ### After we looked at all the resources involved to get an integration running in *Camel K* we will finally take a look at the process of running an integration. [](./integration-startup-process.png) *Startup Process of an Integration in Camel K* 1. When we run the `kamel run Example.java` command, the source code and optional parameters like dependencies (only if your route needs dependencies which are not part of Camel or in particular camel-quarkus) will be handed to the operator. The operator now checks which resources are needed (Secrets, ConfigMaps) and loads the *IntegrationPlatform* resource to obtain basic configuration values. 2. The operator generates the *Integration* which is handed to the *Camel K Runtime* in the next step. 3. With the gathered information the Camel K Runtime is called to build the image. The runtime issues the builder mechanism configured in the *Integration* to build a *camel-quarkus* image. Therefore, it translates the Camel components from the route to the appropriate camel-quarkus ones. Often these are the same but in some cases the might differ. As we don't have a `pom.xml` defining our dependencies and as we - at least for most cases - don't specify dependencies with Traits or arguments, the runtime tries to figure out which Camel dependencies are needed and loads them from the configured maven repository. After the image is build it will be pushed to the registry. 4. The operator generates all the kubernetes resources needed to run the integration. Which resources are needed depends on the components used in the integration ( e.g. ingress for incoming data). 5. The operator starts the application pod. 6. After everything is started the Operator monitors the state and reconciles if necessary. ## Summary In this article we took a look under the hood of Camel K. We first talked about Camel in general and the idea of integration software. We then went to Camel K and took a tour starting with the installation, to running an example integration ending in the chapters to see how all the *magic* is done. So, is Camel K the best way to run Camel integrations in a cloud environment ? It depends. If you can start from scratch and have only simple routes with basic Camel logic and with none or only a few external dependencies, Camel K might be a good solution for you. If you have to migrate an existing Camel based software and/or if you have more complex requirements the following limitations might stop you from using Camel K: * you can not use all Camel components, only the ones supported by camel-quarkus * you can not directly reference routes defined in other integrations, you have to use some technique to transfer the data e.g. database, mqtt broker, jms or similar * you can not use other frameworks to support Camel (like Spring or Spring Boot) * you can customize the resources created by Camel k, but not to an extent like you can when you do the deployment on your own * you can not use libraries which are not available via maven * you don't have a dependency management like in maven (you can use modelines or cli parameters) So for more complex integration needs it might be better do develop a classical camel application, maybe supported by Spring or other frameworks, and deploy them as every other javan application too. ## References * Icons: https://www.flaticon.com/ * Camel in Action Second Edition * Camel K: https://camel.apache.org/Camel-k * Kamel-CLI: https://github.com/apache/Camel-k/releases * Camel Architecture: https://camel.apache.org/manual/architecture.html * Minikube: https://kubernetes.io/de/docs/tasks/tools/install-minikube/ [Read more](https://omd.consol.de/blog/2022/02/24/a-look-insight-camel-k/) ----------------------------------------------------------------------------------- title: "Visualisation and debugging of Apache Camel routes" date: "2022-02-24" --------------------------------------------------------------------------------
The documentation of software is an everyday business of a software developer and engineer. Especially for integration scenarios a diagram on the flow of a message through the system or the whole landscape is an essential illustration. Fortunately there are [standardised messaging patterns](https://www.enterpriseintegrationpatterns.com/patterns/messaging/toc.html){:target="_blank"} which can be used. Unfortunately, however, there is no tool which can create such visualizations out of the box directly of source code. In this article we will have a look at [Apache Camel](https://camel.apache.org/){:target="_blank"} and how it is possible to get a graphical representation of an integration route. We will also discuss about debugging it, as some tools have the feature to do this. ## About Apache Camel and Enterprise Integration Patterns (EIPs) The integration framework Apache Camel was created in [2008](https://projects.apache.org/json/projects/camel.json){:target="_blank"}. Since then it was continuously developed and maintained. At the moment it is versatile useable not only in standard java applications, also on Quarkus or natively on Kubernetes. The [documentation](https://camel.apache.org/docs/){:target="_blank"} is quite good and a large community available on [Zulip](https://camel.zulipchat.com/){:target="_blank"}, [stackoverflow.com](https://stackoverflow.com/questions/tagged/apache-camel){:target="_blank"} or [GitHub](https://github.com/apache/camel/){:target="_blank"} can help you if any challenge occurs. As the framework is based on [Enterprise Integration Patterns](https://camel.apache.org/components/latest/eips/enterprise-integration-patterns.html){:target="_blank"} it is easy to find the best solution for nearly every integration scenario. Additionally, the implementation is straight forward as the documentation keeps an example for each pattern to be used with Apache Camel. Here is a sample for a Apache Camel route: ```java from("jetty:http://0.0.0.0:8080/api").matchOnUriPrefix(true)) .routeId("Step1") .removeHeaders("*", "breadcrumbId") .unmarshal().json(JsonLibrary.Jackson, Map.class) .process(extractInitialBodyProcessor) .to(seda("further_processing?waitForTaskToComplete=Never")) .setHeader(Exchange.HTTP_RESPONSE_CODE, constant(200)) .setBody(constant("")); ``` For manual documentation purposes you can then either use the [EIP icons in one of this diagramming applications](https://camel.apache.org/components/latest/eips/enterprise-integration-patterns.html#_eip_icons){:target="_blank"} or a web-based tool like [Visual Paradigm Online](https://online.visual-paradigm.com/de/diagrams/templates/enterprise-integration-patterns/camel-route/){:target="_blank"}. ## Available solutions to visualize Apache Camel routes Starting with tools to only create a diagram of a Camel route we will end up with more powerful. ### [camel-net](https://github.com/BenjaminBrandmeier/camel-net){:target="_blank"} ![](camel-net.png) Camel-net parses the source code of an application and creates an interactive graph of the found routes which can be accessed through the started webserver. Pros + complete decoupled of the application + you can see the route behind an element Cons - the graph contains the same data, but always looks slightly different ### [camel-plantuml](https://camel.apache.org/blog/2021/05/plantuml-diagram/){:target="_blank"} ![](camel-plantuml.png) As the name already suggests a PlantUML code for a diagram will be generated with this tool. In order to use it you can either embed it to your application or connect remotely to it [see detailed instructions on Github](https://github.com/ncasaux/camel-plantuml){:target="_blank"}. No matter which way you use, you get the PlantUML code through a get request to the configured endpoint. Pros + lightweight solution + you can add the resulting PlantUML code to your version control + possibility to connect remotely Cons - some other tool must be used to render the PlantUML code The generated code for the image above was this:
- click to expand / collapse code section - ```text @startuml SampleCamel skinparam ArrowColor #Black skinparam rectangle { RoundCorner 20 } skinparam rectangle<> { BorderColor #6C8EBF BackgroundColor #DAE8FC } skinparam queue<> { } skinparam queue<> { BorderColor #B85450 BackgroundColor #F8CECC } skinparam queue<> { BorderColor #82B366 BackgroundColor #D5E8D4 } footer Generated with camel-plantuml on %date("dd-MM-yyyy HH:mm") ' === Some useful settings for tweaking diagram layout === 'left to right direction 'hide stereotype 'skinparam nodesep 50 'skinparam ranksep 50 'skinparam wrapWidth 250 rectangle route_1 <> as " Step1 " queue endpoint_1 <><> as " jetty://http://0.0.0.0:8080/api " queue endpoint_2 <><> as " seda://further_processing " endpoint_1 --> route_1 : from route_1 --> endpoint_2 : to @enduml ```
### [Hawtio](https://hawt.io/){:target="_blank"} ![](hawtio.png) The more powerful tool called Hawtio offers a web-based management interface like the JDK's binary `jconsole`. There are [several ways to get started](http://hawt.io/docs/get-started/){:target="_blank"}. If you embed it to your application and add the following to the `RouteBuilder` class, you can simply debug the Camel routes of the application in your browser (very useful in cases where the application is running somewhere else): ```java getContext().setDebugging(true); ``` The web interface shows directly a reached breakpoint and displays the headers and the body of the current message: ![](hawtio-debug.png) Pros + nice visualisation of the routes + offers many additional features + security of interface included + possibility to connect remotely + write our own or [use available](http://hawt.io/docs/plugins/){:target="_blank"} plugins to get additional features Cons - not as lightweight as other solutions If you want to try hawtio with [Quarkus](https://quarkus.io/){:target="_blank"}, maybe take a look at [this Github issue](https://github.com/hawtio/hawtio/issues/2643){:target="_blank"}. ## Textual debugging of Camel routes The Camel Route debugger is an IDE plugin and offers a textual solution to debug Camel routes. It is available for [IntelliJ](https://camel.apache.org/blog/2021/12/camelDebuggerForIntelliJ/){:target="_blank"}, [VS Code, Eclipse Desktop](https://camel.apache.org/blog/2022/02/camel-textual-debug-eclipse-vscode/){:target="_blank"} and [other IDEs](https://microsoft.github.io/debug-adapter-protocol/implementors/tools/){:target="_blank"}. Starting with Apache Camel version 3.15.0 a breakpoint in the IDE can be set and used for interaction directly in the route (a missing feature in the past). ![](camel-debugger.png) It also allows some evaluations and manipulations: ![](camel-debugger-tools.png) # Conclusion There are very interesting developments like the recently released IDE plugins and of course experienced tools like Hawtio. Mainly it depends on the requirements what to use. In my point of view, if you would like to - take a brief look on the routing **[camel-net](https://github.com/BenjaminBrandmeier/camel-net){:target="_blank"}** is just fine - add the results into version control **[camel-plantuml](https://camel.apache.org/blog/2021/05/plantuml-diagram/){:target="_blank"}** is a good choice - only debug your routes locally **[Camel Route debugger](https://camel.apache.org/blog/2022/02/camel-textual-debug-eclipse-vscode/){:target="_blank"}** will fit - debug your routes and get additional information about your JVM then you can best use **[Hawtio](https://hawt.io/){:target="_blank"}** For my purpose the documentation of Camel routes, as described in the beginning, the tool of choice is Hawtio. Especially for automation enthusiasts (as I am) the manual steps can be decreased with the usage of a (GUI) testing tool. The steps to implement would be in general: start the application, connect to the web interface, navigate to the route diagram page and take a screenshot ... but that is a topic for another article. Finally, from my point of view the new debugging features of Apache Camel itself allows newcomers to easily get started with debugging and prevent experienced users from adding inline [processors](https://camel.apache.org/manual/processor.html){:target="_blank"} only for debugging purposes. [Read more](https://omd.consol.de/blog/2022/02/24/visualisation-and-debugging-of-apache-camel-routes/) ----------------------------------------------------------------------------------- title: "Streaming and Messaging" date: "2021-07-14" --------------------------------------------------------------------------------
## Disclaimer This article is the author’s opinion on similarities and differences between Streaming and Messaging. ## Streaming and Messaging The first time I was busy with the terms *messaging* and *streaming* was during my master thesis in 2016. Among other things, the thesis was about different strategies of microservices integration. During that time, the term messaging was popular. Moroever, Kafka, which is a streaming platform, was popular, too. From a high-level perspective, messaging, kafka and streaming seem to be the same thing... but I never understood, why we have these two terms which are used synonymously in many contexts: messaging and streaming. This article is my answer to that question. I start with aspects that both have in common. ### Infrastructure Communication flows through a separate infrastructural component, e.g., RabbitMQ, Kafka, ActiveMQ... even an FTP server is a valid candidate. ### Temporal decoupling Producer and consumer can be available (online) at different times - just like it works with instant messaging on smartphones. ### Location transparency Producer and consumer do not have to know about each other’s location (IP address and port). They know only the infrastructural component in between. Heads up: These properties also apply to databases! Now, let’s have a look at aspects that are different between messaging and streaming. ## Data Unit In general, the size of a data unit, both in streaming and messaging tends to be small. Bytes up to a few megabytes. The transport of bigger files via a file storage could also be seen as a sort of messaging - however, this will not be part of this article. When it comes to the difference between streaming and messaging, streaming data units tend to be smaller, because they are just raw information without context - e.g. a temperature measurement. In messaging, this is typically vice versa. Data units are compositions of multiple facts, designed for a specific use case. Consequently, they tend to be bigger than data units in the context of streaming. ### Data unit production & processing flow: Streaming
### Data unit production & processing flow: Messaging
## Responsibilities of Producers and Consumers In streaming applications, producers define the structure of data units and publish them. Consequently, the consumers have to implement the processing logic on top of these data units. In messaging applications, data units are composed with a use case in mind. This composition is already a sort of processing. In summary it can be said that in streaming applications, the data units are post-processed while in messaging applications data units are pre-processed. In the real world, in most use cases, it will be somewhere in between. ## Data Rate In a streaming application, data sources publish all of their data. Consequently, the data rate will be higher than in messaging applications, because the data is not drilled down to particular use cases. If data is available, it will be published. This is not a strict law, more my impression of the typical use cases. ## Data Unit Durability & Consumption Rate In messaging applications, data units are typically consumed once. Most message brokers are also built with that in mind. E.g. RabbitMQ performs best, when no data units are stored inside the broker (“A happy rabbit is an empty rabbit”). In contrast, the streaming platform Kafka is *built for keeping* data in order to support use cases that need the data units to be consumed multiple times. ## Data Durability As said before, typical streaming platforms such as Apache Kafka are built for keeping data. Thus, the data can be stored longer. With messaging, data units are something that should be consumed, as soon as possible. Therefore, message brokers are built to get rid of data instead of keeping it. ## Broker/Consumer Relationship Streaming platforms are focused on collecting and providing data. Business logic is usually the responsibility of consumers. In contrast, messaging systems tend to be smarter. They provide features such as priority queues, message delays, message routing, filtering etc. In streaming platforms, all of this can be done as well - however, usually this has to be implemented by hand on the consumer side. ## Integration with data mining use cases When it comes to data mining and analytics, streaming applications provide great support out of the box, because the data units can be consumed multiple times and be re-composed at any time - no matter what the data is already been used for. With messaging, this tends to be not that easy, because the pre-composed data units have to be aggregated for the new data minig use cases and you will probably over-fetch data. ## Scalability Storage capacity and data throughput: nowadays streaming platforms are built to be scalable for these aspects while keeping a low latency. If the concrete use case of your streaming application is not that demanding, a messaging system can be sufficient as well. ## Summary | | Streaming | Messaging | | ------------- | ------------- | ----- | | Data Unit content | defined by data source | defined by the use case | | Processing | post | pre | | Data Unit size | smaller | bigger | | Data Rate | higher | lower | | Data Unit Durability & Consumption Rate | Consume-Once --> infinitiy | Consume-Once | | Broker/Consumer relationship | simple broker, smart consumers | smart broker, simple consumers | | Integration with data mining use cases | easy | might be harder | | Scalability | easier| harder | [newest posts on ConSol Labs]: https://labs.consol.de [Read more](https://omd.consol.de/blog/2021/07/14/streaming-and-messaging/) ----------------------------------------------------------------------------------- title: "A journey of a Helm operator to OperatorHub.io and back again" date: "2021-05-03" --------------------------------------------------------------------------------
Some time ago, I started a project to create a Helm based operator for an OpenShift application. I used the Operator SDK to create the Helm operator. The Operator SDK documentation describes the parameters pretty good, and it contains a simple tutorial. it does not, however, describe the complete development cycle. This article aims to describe everything from creating the operator to the point where you can upload your operator to OperatorHub.io. We start with a basic Helm Chart. With this, you can install Nginx as a StatefulSet. You can find the source code in my github repo. Before we can start with creating an operator, we need to fulfill some prerequisites. ## Prerequisite In order to develop the operator, the following applications need to be installed: - [operator-sdk](https://sdk.operatorframework.io/) - [git](https://git-scm.com/) - [oc](https://docs.openshift.com/container-platform/4.7/cli_reference/openshift_cli/getting-started-cli.html#cli-installing-cli_cli-developer-commands)/[kubectl](https://kubernetes.io/docs/tasks/tools/) tool - Connection to running OpenShift (4.x)/Kubernetes cluster - Be logged in to a docker repository like quay.io - [Helm (optional)](https://helm.sh/docs/intro/install/) I spare you the details of how to download the applications and set them up. There are a lot of articles describing just that. ## Create the initial operator We have everything set up and we got a Nginx Helm Chart. With this Chart, Nginx can be installed as a StatefulSet. But wouldn't it be much cooler to define a Nginx CRD and Kubernetes/OpenShift installs Nginx for us? If we change something in the CRD, the operator should take care of applying the changes, and it should make sure that a user can only change limited properties in the application. Of course, we could create an ansible- or go operator, but creating a Helm based operator might be more sufficient for a lot of use cases. Our source Helm Chart is in the subfolder `nginx-sts` of the current directory. Let's create the initial Helm operator. We do this with the command `operator-sdk init` and add the reference to the used source Helm Chart `--helm-chart=nginx-sts` and some metadata (For some reason it is not possible to describe the location of the Helm Chart like this `--helm-chart=./nginx-sts`.). In our case the command looks like this: ``` bash $>operator-sdk init --plugins=helm.sdk.operatorframework.io/v1 --group=demo --domain=consol.de --helm-chart=nginx-sts --project-name=nginx-sts Created helm-charts/nginx-sts Generating RBAC rules I0315 16:08:14.986191 199443 request.go:655] Throttling request took 1.004878088s, request: GET:https://api.crc.testing:6443/apis/coordination.k8s.io/v1beta1?timeout=32s WARN[0004] The RBAC rules generated in config/rbac/role.yaml are based on the Chart's default manifest. Some rules may be missing for resources that are only enabled with custom values, and some existing rules may be overly broad. Double check the rules generated in config/rbac/role.yaml to ensure they meet the operator's permission requirements. ``` The Operator SDK is expecting that the (source) Helm Chart is located in a different folder than where this operator files will be generated. Because of this, we need to delete the original Helm Chart folder (`nginx-sts`). Don't worry, a copy of the Helm Chart has been created in the folder `helmcharts/nginx-sts`. We get a warning that we should double check the created roles and rolebinding. In our case, we don't need to worry about that. You can, however, check it if you want to. The created file structure of the operator looks like this: ``` bash $>tree . ├── config │   ├── crd │   │   ├── bases │   │   │   └── demo.consol.de_nginxsts.yaml │   │   └── kustomization.yaml │   ├── default │   │   ├── kustomization.yaml │   │   └── manager_auth_proxy_patch.yaml │   ├── manager │   │   ├── kustomization.yaml │   │   └── manager.yaml │   ├── prometheus │   │   ├── kustomization.yaml │   │   └── monitor.yaml │   ├── rbac │   │   ├── auth_proxy_client_clusterrole.yaml │   │   ├── auth_proxy_role_binding.yaml │   │   ├── auth_proxy_role.yaml │   │   ├── auth_proxy_service.yaml │   │   ├── kustomization.yaml │   │   ├── leader_election_role_binding.yaml │   │   ├── leader_election_role.yaml │   │   ├── nginxsts_editor_role.yaml │   │   ├── nginxsts_viewer_role.yaml │   │   ├── role_binding.yaml │   │   └── role.yaml │   ├── samples │   │   ├── demo_v1alpha1_nginxsts.yaml │   │   └── kustomization.yaml │   └── scorecard │   ├── bases │   │   └── config.yaml │   ├── kustomization.yaml │   └── patches │   ├── basic.config.yaml │   └── olm.config.yaml ├── Dockerfile ├── helm-charts │   └── nginx-sts │   ├── Chart.yaml │   ├── templates │   │   ├── configmap.yaml │   │   ├── _helpers.tpl │   │   ├── NOTES.txt │   │   ├── route.yaml │   │   ├── service.yaml │   │   ├── stateful_sets.yaml │   │   └── tests │   │   └── test-connection.yaml │   └── values.yaml ├── Makefile ├── PROJECT ├── README.md └── watches.yaml 15 directories, 39 files ``` With these files, we will be able to create and deploy the operator in an OpenShift or Kubernetes cluster. First, let us modify the created CRD (config/crd/bases/demo.consol.de_nginxsts.yaml) to be more strict. By this, we can use input validation of Kubernetes when we create a resource. Also, we provide the end-user with a speaking documentation, so that we can focus on the fun parts, instead of working on support tickets. Anyhow, we are going to change the generated yaml from: {% highlight yaml%} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: nginxsts.demo.consol.de spec: group: demo.consol.de names: kind: NginxSts listKind: NginxStsList plural: nginxsts singular: nginxsts scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: NginxSts is the Schema for the nginxsts API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: Spec defines the desired state of NginxSts type: object x-kubernetes-preserve-unknown-fields: true status: description: Status defines the observed state of NginxSts type: object x-kubernetes-preserve-unknown-fields: true type: object served: true storage: true subresources: status: {} {% endhighlight %} To this strict and descriptive version (changes are highlighted): {% highlight yaml hl_lines="35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94"%} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: nginxsts.demo.consol.de spec: group: demo.consol.de names: kind: NginxSts listKind: NginxStsList plural: nginxsts singular: nginxsts scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: NginxSts is the Schema for the nginxsts API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: Spec defines the desired state of NginxSts type: object required: - replicaCount properties: image: description: Configuration of the used Nginx image type: object properties: pullPolicy: description: 'Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images' type: string pattern: '^(Always|Never|IfNotPresent)$' repository: description: The alternative image repository that should be used. type: string tag: description: The alternative image tag that should be used. type: string replicaCount: description: The number of replicas of the Nginx StatefulSet. type: integer default: 1 resources: description: The resources for the Nginx pod. type: object properties: limits: description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object properties: cpu: type: string pattern: '^(\d+m|\d+(\.\d{1,3})?)$' memory: type: string pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$' requests: description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object properties: cpu: type: string pattern: '^(\d+m|\d+(\.\d{1,3})?)$' memory: type: string pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$' service: description: Definition of the Nginx service type: object properties: port: description: The port that will be used for the Nginx container. The defined port number must be greater than 1024. type: integer minimum: 1024 type: description: type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer type: string pattern: '^(ExternalName|ClusterIP|NodePort|LoadBalancer)$' status: description: Status defines the observed state of NginxSts type: object x-kubernetes-preserve-unknown-fields: true type: object served: true storage: true subresources: status: {} {% endhighlight %} Looks much better, doesn't it? A description about how to define CRD can be found [here](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/). ## Build and run the operator Building and running the operator is fairly easy. We just need to execute the following command: ``` bash export USERNAME= make docker-build docker-push IMG=quay.io/$USERNAME/nginx-operator:v0.0.1 ``` In my case it looks like this: ``` bash export USERNAME=omeyer make docker-build docker-push IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 ``` After the upload is finished, run this command to deploy the operator manually: ``` bash make deploy IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 ``` --- **Hint** Make sure that the destination registry is public and can be accessed without password. Also, note that the deployment of the operator has the `ImagePullPolicy: IfNotPresent`, which means that the operator image might not be reloaded when the operator images change. For changes in the operator, you either need to create new tags for each change or change the `ImagePullPolicy` to `Always`. I prefer the latter. --- ## Test the operator To test the operator, we create a Nginx CRD in the cluster. This should, in return, cause the operator to deploy a new nginx-sts application. We can use the sample CRD in the folder `config/samples/demo_v1alpha1_nginxsts.yaml` to create the CRD. The file's content looks like this: ``` yaml apiVersion: demo.consol.de/v1alpha1 kind: NginxSts metadata: name: nginxsts-sample spec: replicaCount: 3 ``` To create the resource, we execute: ``` bash oc apply -f config/samples/demo_v1alpha1_nginxsts.yaml ``` --- **Hint** I modified the samples file, so that it contains only the replicas and not all other attributes from Helm's `values.yaml`. You can look up the complete CRD file in my GitHub repo. --- As you can see, 3 replicas of the Nginx StatefulSet have been started, and I am able to connect to it. I call this a success. ``` bash $> oc get pod -n default NAME READY STATUS RESTARTS AGE nginxsts-sample-nginx-sts-0 1/1 Running 0 3h17m nginxsts-sample-nginx-sts-1 1/1 Running 0 3h17m nginxsts-sample-nginx-sts-2 1/1 Running 0 3h17m $> oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD nginxsts-sample-nginx-sts nginxsts-sample-nginx-sts-default.apps-crc.testing nginxsts-sample-nginx-sts http None $> curl -v nginxsts-sample-nginx-sts-default.apps-crc.testing ... * Connected to nginxsts-sample-nginx-sts-default.apps-crc.testing (192.168.130.11) port 80 (#0) > GET / HTTP/1.1 > Host: nginxsts-sample-nginx-sts-default.apps-crc.testing > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Server: nginx/1.19.7 < Date: Thu, 18 Mar 2021 12:09:25 GMT < Content-Type: text/html < Content-Length: 612 < Last-Modified: Tue, 16 Feb 2021 16:25:34 GMT < ETag: "602bf1fe-264" < X-PodName: nginxsts-sample-nginx-sts-0 < Accept-Ranges: bytes < Set-Cookie: 25f93612584c1f45a174dd9568118c21=b331a06e7f6d6d15096677f0cf1b21fe; path=/; HttpOnly < Cache-control: private ... ``` ## Create an operator bundle So far, we created all resources to deploy the operator. Based on our CRD this operator is able to deploy applications. If we just need that, we can wrap everything up and call it a day. But wouldn't it be nice, if our customers could use OLM (Operator Lifecycle Manager) to find our operator in the OperatorHub.io and install it on their cluster through just some clicks? I think so, too. Let's get it done and let's create the resources needed . --- **Hint** If you are using an OpenShift 4.x cluster you do not need to install OLM (Operator Lifecycle Manager) on your cluster (otherwise you can find a description of how to do that [here](https://sdk.operatorframework.io/docs/olm-integration/quickstart-bundle/#enabling-olm)). It comes with the OLM included and configured. To test that it works properly, run: ``` bash $>operator-sdk olm status --olm-namespace openshift-operator-lifecycle-manager INFO[0002] Fetching CRDs for version "0.16.1" INFO[0002] Using locally stored resource manifests INFO[0002] Successfully got OLM status for version "0.16.1" NAME NAMESPACE KIND STATUS operators.operators.coreos.com CustomResourceDefinition Installed operatorgroups.operators.coreos.com CustomResourceDefinition Installed installplans.operators.coreos.com CustomResourceDefinition Installed clusterserviceversions.operators.coreos.com CustomResourceDefinition Installed subscriptions.operators.coreos.com CustomResourceDefinition Installed system:controller:operator-lifecycle-manager ClusterRole Installed aggregate-olm-edit ClusterRole Installed aggregate-olm-view ClusterRole Installed catalogsources.operators.coreos.com CustomResourceDefinition Installed olm Namespace namespaces "olm" not found olm-operator-binding-olm ClusterRoleBinding clusterrolebindings.rbac.authorization.k8s.io "olm-operator-binding-olm" not found olm-operator olm Deployment deployments.apps "olm-operator" not found catalog-operator olm Deployment deployments.apps "catalog-operator" not found olm-operator-serviceaccount olm ServiceAccount serviceaccounts "olm-operator-serviceaccount" not found operators Namespace namespaces "operators" not found global-operators operators OperatorGroup operatorgroups.operators.coreos.com "global-operators" not found olm-operators olm OperatorGroup operatorgroups.operators.coreos.com "olm-operators" not found packageserver olm ClusterServiceVersion clusterserviceversions.operators.coreos.com "packageserver" not found operatorhubio-catalog olm CatalogSource catalogsources.operators.coreos.com "operatorhubio-catalog" not found ``` --- Let's initialize the resources that are needed to add our operator to OperatorHub.io by executing this command ``` bash make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 operator-sdk generate kustomize manifests -q Display name for the operator (required): > Nginx-StatefulSet Description for the operator (required): > This will deploy Nginx as a StatefulSet. This is an example! Don't use it in production! Provider's name for the operator (required): > ConSol Software GmbH Any relevant URL for the provider name (optional): > https://www.consol.de/ Comma-separated list of keywords for your operator (required): > example,nginx Comma-separated list of maintainers and their emails (e.g. 'name1:email1, name2:email2') (required): > olaf.meyer@consol.de cd config/manager && /home/olaf/Projects/operator/helm_operator/bin/kustomize edit set image controller=quay.io/omeyer/nginx-sts-operator:v0.0.1 /home/olaf/Projects/operator/helm_operator/bin/kustomize build config/manifests | operator-sdk generate bundle -q --overwrite --version 0.0.1 operator-sdk bundle validate ./bundle INFO[0000] Found annotations file bundle-dir=bundle container-tool=docker INFO[0000] Could not find optional dependencies file bundle-dir=bundle container-tool=docker INFO[0000] All validation tests have completed successfully ``` What happened after we have executed this command? Let's have a look ``` bash $>tree ... ├── bundle │   ├── manifests │   │   ├── demo.consol.de_nginxsts.yaml │   │   ├── nginx-sts.clusterserviceversion.yaml │   │   ├── nginx-sts-controller-manager-metrics-service_v1_service.yaml │   │   └── nginx-sts-metrics-reader_rbac.authorization.k8s.io_v1_clusterrole.yaml │   ├── metadata │   │   └── annotations.yaml │   └── tests │   └── scorecard │   └── config.yaml ├── bundle.Dockerfile ├── config │   ├── crd ... ``` A new directory `bundle` and a file `bundle.Dockerfile` have been created. Before we have a closer look at these files, let's build and deploy the bundle operator and see how it looks like in OpenShift. To build the bundle operator, we run these commands: ``` bash export BUNDLE_IMG="quay.io/$USERNAME/nginx-sts-operator-bundle:v0.0.1" make bundle-build BUNDLE_IMG=$BUNDLE_IMG make docker-push IMG=$BUNDLE_IMG ``` After this, the bundle operator has been uploaded to the specified quay.io repo. Now we need to deploy it with this command: ``` bash $>operator-sdk run bundle $BUNDLE_IMG INFO[0040] Successfully created registry pod: quay-io-omeyer-nginx-sts-operator-bundle-v0-0-1 INFO[0040] Created CatalogSource: nginx-sts-catalog INFO[0040] OperatorGroup "operator-sdk-og" created INFO[0040] Created Subscription: nginx-sts-v0-0-1-sub INFO[0045] Approved InstallPlan install-8j9wf for the Subscription: nginx-sts-v0-0-1-sub INFO[0045] Waiting for ClusterServiceVersion "default/nginx-sts.v0.0.1" to reach 'Succeeded' phase INFO[0045] Waiting for ClusterServiceVersion "default/nginx-sts.v0.0.1" to appear FATA[0120] Failed to run bundle: error waiting for CSV to install: timed out waiting for the condition ``` It has failed?! Let's see for ourselves. Looks like the operator has successfully been installed. It might just have taken a bit too long. ![installed_nginx-sts_operator.jpg](installed_nginx-sts_operator.jpg) ![installed_nginx-sts_operator_details.jpg](installed_nginx-sts_operator_details.jpg) We are now able to create a new CRD. The form and yaml view look very good for a starting point. ![installed_nginx-sts_operator_new_crd_form.jpg](installed_nginx-sts_operator_new_crd_form.jpg) Let's do a couple of changes: - I would like to see the ConSol logo, when the operator has been installed. - The url `https://nginx-sts.domain` and description for it is not correct. - There is no short description for the operator. - And so on ... To do these changes, let us have a look at the folder `bundle/manifests`. It contains 4 files: 1. `demo.consol.de_nginxsts.yaml` is the file for the definition of the `nginxsts.demo.consol.de` CRD. 1. `nginx-sts.clusterserviceversion.yaml` is the ClusterServiceVersion (like stated in the name) and is used by the OLM to set up the `nginx-sts` operator. Furthermore, it contains all information that is used when installing and / or modifying the operator. 1. The other two files are used to monitor the operator. ## Modify the cluster service version To change the information that is displayed in OpenShift, we need to modify the cluster service version file. However, there are two cluster service versions in the project. The file `bundle/manifests/nginx-sts.clusterserviceversion.yaml` has been created by the file `config/manifests/bases/nginx-sts.clusterserviceversion.yaml`. If we modify the latter and run `make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1` again, the changes will be transferred to bundle file. So let us to this. I added the highlighted lines to the `nginx-sts.clusterserviceversion.yaml`: {% highlight bash hl_lines="8 9 10 11 12 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 46 66 67 69 70 73 74"%} apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: annotations: alm-examples: '[]' capabilities: Basic Install categories: Example operator containerImage: quay.io/omeyer/nginx-sts-operator:v0.0.1 createdAt: "1970-01-01T00:00:00Z" description: This will deploy Nginx as a StatefulSet. This is an example! Don't use it in production! repository: https://github.com/olaf-meyer/helm-operator support: None - Example operator name: nginx-sts.v0.0.0 namespace: placeholder spec: apiservicedefinitions: {} customresourcedefinitions: owned: - description: Definition how to setup the Nginx as a StatefulSet. displayName: Nginx-StatefulSet kind: NginxSts name: nginxsts.demo.consol.de resources: - kind: Route name: "" version: route.openshift.io/v1 - kind: Service name: "" version: v1 - kind: StatefulSet name: "" version: v1 - kind: Pod name: "" version: v1 specDescriptors: - description: Limits describes the minimum/maximum amount of compute resources required/allowed displayName: Resource Requirements path: resources x-descriptors: - urn:alm:descriptor:com.tectonic.ui:resourceRequirements version: v1alpha1 description: This will deploy Nginx as a StatefulSet. This is an example! Don't use it in production! displayName: Nginx-StatefulSet icon: - base64data: iVBORw0KGgoAAAANSUhEUgAAAQIAAAEsCAYAAAAo6b0WAAAxjUlEQVR42u19CXxU5dX+C8gmLrjUqkVFRcg6EVGUoi1arSKuVepn60JdoqIhMyEkM3dJxr1KP1Ra6x+1tlY/9Qv6uS ..... mediatype: image/png install: spec: deployments: null strategy: "" installModes: - supported: false type: OwnNamespace - supported: false type: SingleNamespace - supported: false type: MultiNamespace - supported: true type: AllNamespaces keywords: - example - nginx links: - name: ConSol Software GmbH url: https://www.consol.de/ maintainers: - email: olaf.meyer@consol.de name: Olaf Meyer maturity: alpha provider: name: ConSol Software GmbH url: https://www.consol.de/ version: 0.0.0 {% endhighlight %} Most changes are self explaining. There are a couple of changes that I want to talk about: 1. The whole attribute `customresourcedefinitions` is used for displaying the defined CRD in the Web-GUI. It is not intended as a replacement of the CRD definition. The list of resources doesn't have to be complete, but should contain the most import resources for your application. Also, you can define which kinds of controls should be used. With this we can use the OpenShift resource form for entering the limits and requests. 1. The icon attribute is truncated because even a small image can become too long for demo purposes. You can find the complete definition in my GitHub Repo for this article. 1. In a production scenario I would calculate the attribute `createdAt` and `containerImage` in a CI/CD pipeline. There is one more point that I would like to change in the generated `ClusterServiceVersion` file. The deployment is using the default service account, which can lead to some problems. We are going to change that. For this, we need to create a service account `nginx-sts-controller-manager` in the file `service_account.yaml` in the folder `config/rbac`. The next step is to add this service account to the `kustomization.yaml`, so that it will be processed when we update the bundle files. After this, we replace the `default` service account with `nginx-sts-controller-manager` in the files `config/rbac/auth_proxy_role_binding.yaml`, `config/rbac/leader_election_role_binding.yaml` and `config/rbac/role_binding.yaml`. The last part is to use this service account in the deployment of the operator. For this, we add the line `serviceAccountName: nginx-sts-controller-manager` to the file `config/manager/manager.yaml`. If we now run the command `make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1`, the files in the folder `bundle/manifests` will contain our changes and we are ready to test the latest version. Because we have changed a lot, we need to `uninstall` the current operator. The `cleanup` command requires a parameter `operatorPackageName`. You can find this value i.e. in the file `bundle/metadata/annotations.yaml` in the value `operators.operatorframework.io.bundle.package.v1`. In our case this is `nginx-sts`. So the cleanup command looks like this: ``` bash operator-sdk cleanup nginx-sts ``` After this, we can rebuild the container image and deploy it ``` bash export USERNAME=omeyer export BUNDLE_IMG="quay.io/$USERNAME/nginx-sts-operator-bundle:v0.0.1" make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 make bundle-build BUNDLE_IMG=$BUNDLE_IMG make docker-push IMG=$BUNDLE_IMG operator-sdk run bundle $BUNDLE_IMG ``` ## Verify the new version Let's have a look at the new version of the operator. From my point of view the open points are closed. In the installed operator overview there are neither empty fields nor fields filled with dummy values. ![installed_nginx-sts_operator_install_new.jpg](installed_nginx-sts_operator_install_new.jpg) On the operator overview page the description looks fine as well and shows the ConSol Logo. ![installed_nginx-sts_operator_details_new.jpg](installed_nginx-sts_operator_details_new.jpg) The last point is the changed editor for entering the resources of the pod. On the right hand side we get a documentation of the CRD. Even if it is not shown in the image, we have an auto completion in the editor as well. ![installed_nginx-sts_operator_new_crd_yaml.jpg](installed_nginx-sts_operator_new_crd_yaml.jpg) All looks good ... ## Create Scorecard tests Almost! We should add tests to the operator to further automate the build and deployment of the operator. Basic tests are already included. We can execute them with the command: ``` bash operator-sdk scorecard bundle -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-crds-have-validation] Labels: "suite":"olm" "test":"olm-crds-have-validation-test" Results: Name: olm-crds-have-validation State: pass Log: Loaded 1 Custom Resources from alm-examples Loaded CustomresourceDefinitions: [&CustomResourceDefinition{...}] -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test basic-check-spec] Labels: "suite":"basic" "test":"basic-check-spec-test" Results: Name: basic-check-spec State: pass -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-crds-have-resources] Labels: "suite":"olm" "test":"olm-crds-have-resources-test" Results: Name: olm-crds-have-resources State: pass Log: Loaded ClusterServiceVersion: nginx-sts.v0.0.1 -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-status-descriptors] Labels: "suite":"olm" "test":"olm-status-descriptors-test" Results: Name: olm-status-descriptors State: fail Errors: nginxsts.demo.consol.de does not have a status descriptor Log: Loaded ClusterServiceVersion: nginx-sts.v0.0.1 Loaded 1 Custom Resources from alm-examples -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-spec-descriptors] Labels: "suite":"olm" "test":"olm-spec-descriptors-test" Results: Name: olm-spec-descriptors State: fail Suggestions: Add a spec descriptor for replicaCount Errors: replicaCount does not have a spec descriptor Log: Loaded ClusterServiceVersion: nginx-sts.v0.0.1 Loaded 1 Custom Resources from alm-examples -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-bundle-validation] Labels: "suite":"olm" "test":"olm-bundle-validation-test" Results: Name: olm-bundle-validation State: pass Log: time="2021-03-22T16:33:04Z" level=debug msg="Found manifests directory" name=bundle-test time="2021-03-22T16:33:04Z" level=debug msg="Found metadata directory" name=bundle-test time="2021-03-22T16:33:04Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test time="2021-03-22T16:33:04Z" level=info msg="Found annotations file" name=bundle-test time="2021-03-22T16:33:04Z" level=info msg="Could not find optional dependencies file" name=bundle-test ``` Looks like there are two errors. It seems that we should provide a description for replicaCount and for the status in the cluster service version. Let's fix that. To provide a description for replicaCount, add the following section to `config/manifests/bases/nginx-sts.clusterserviceversion.yaml` ``` yaml ... specDescriptors: - description: The number of replicas of the Nginx StatefulSet. displayName: Replicas path: replicaCount x-descriptors: - urn:alm:descriptor:com.tectonic.ui:podCount - description: Limits describes the minimum/maximum amount of compute resources required/allowed displayName: Resource Requirements path: resources x-descriptors: - urn:alm:descriptor:com.tectonic.ui:resourceRequirements version: v1alpha1 ... ``` With this change we do not only resolve the error but also change the style of the input field. This is quite neat. ![installed_nginx-sts_operator_new_crd_form_new.jpg](installed_nginx-sts_operator_new_crd_form_new.jpg) Originally, I was under the impression that the second error `nginxsts.demo.consol.de does not have a status descriptor` could not be fixed. In the status descriptor we can define which values will be used in the status attribute. Also, we define which attribute and how it is displayed on the first tab (Display view). The values in the status attribute are written by the Helm operator and at the time being, it is not possible to change them. The Helm operator writes the attributes `conditions`, `manifest` and `name` to the status. Using the attribute `conditions` in the `statusDescriptors` is not working, because the condition is already displayed at the end of the page. The content of the attribute `manifest` is too long. Let's use the attribute `name` in the `statusDescriptors`: ``` yaml ... statusDescriptors: - description: Name of the created application displayName: My-Application name path: deployedRelease.name x-descriptors: - 'urn:alm:descriptor:text' ... ``` The result looks like this: ![installed_nginx-sts_operator_installed_crd_details.jpg](installed_nginx-sts_operator_installed_crd_details.jpg) If we rerun the Scorecard tests, all tests pass. --- Hint: To test the changes, we deleted the operator. After this, I recreated the bundle definition and rebuilt the operator and deployed it. The following commands were used: ``` bash operator-sdk cleanup nginx-sts export USERNAME=omeyer export BUNDLE_IMG="quay.io/$USERNAME/nginx-sts-operator-bundle:v0.0.1" make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 make bundle-build BUNDLE_IMG=$BUNDLE_IMG make docker-push IMG=$BUNDLE_IMG operator-sdk run bundle $BUNDLE_IMG ``` --- ## Create example integration test So far, the tests verified only the cluster service version. What is missing is a test to check whether the operator itself is working. For this kind of tests the Operator SDK offers an integration of the KUTTL test framework. At the time being, it is required to make the changes for the KUTTL test directly in folder `bundle` and not in the folder `config`. If we run the command `make bundle` our changes for the KUTTL test are not overwritten. Let's create a simple KUTTL test case. First we need to create a KUTTL configuration file `kuttl-test.yaml` in a new folder `bundle/tests/scorecard/kuttl`. The content of the configuration file `kuttl-test.yaml` looks like this: ``` yaml apiVersion: kudo.dev/v1beta1 kind: TestSuite parallel: 4 timeout: 120 startControlPlane: false ``` This is the default configuration of the Operator SDK. It just configures the number of parallel test executions, and the amount of time the KUTTL test is allowed to take. Next, we create a folder `install-nginx-sts` for grouping the tests. In this folder we add two files. With the first file `00-install-nginx-sts.yaml` we set up the test case. In our case this is an Nginx StatefulSet with 4 replicas. It looks like this: ```yaml apiVersion: demo.consol.de/v1alpha1 kind: NginxSts metadata: name: nginxsts-sample namespace: default spec: replicaCount: 4 ``` In the second file `00-assert.yaml` we defined how the result should look like. This means that the StatefulSet with 4 replicas is running. The file looks like this: ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: nginxsts-sample-nginx-sts namespace: default status: readyReplicas: 4 ``` In the last step we need to add the KUTTL tests to the Scorecard tests by adding the following line to the file `bundle/tests/scorecard/config.yaml`: {% highlight bash hl_lines="9 10 11 12"%} ... - entrypoint: - scorecard-test - olm-status-descriptors image: quay.io/operator-framework/scorecard-test:v1.5.0 labels: suite: olm test: olm-status-descriptors-test - image: quay.io/operator-framework/scorecard-test-kuttl:v1.5.0 labels: suite: kuttlsuite test: kuttltest1 {% endhighlight %} When calling the Scorecard tests, we need to provide the service account with sufficient permissions to create the nginx-sts CRD. Additionally, we need to add the attribute `--wait-time=60s` in order to wait a bit longer till Nginx is deployed (It is deploying quite quick, but not that quick that it passes the test). So the command to run the Scorecard and KUTTL tests looks like this: ``` bash operator-sdk scorecard bundle --service-account=nginx-sts-nginx-sts-controller-manager --wait-time=60s ``` ## How to upload operator to OperatorHub.io My initial plan was to add a section how to upload our operator to OperatorHub.io. After reviewing the documentation, however, I came to the conclusion that the details with all hooks and cranks need to be explained in a separate article. Even that would only rephrase the documentation, and it does not give much benefit. You can find the documentation, how to commit a community Kubernetes or OpenShift operator, here: [Community operators](https://operator-framework.github.io/community-operators/contributing-prerequisites/). If you want to create an OpenShift operator that is certified by Red Hat, the documentation for this can be found here: [Certified operator build guide](https://redhat-connect.gitbook.io/certified-operator-guide/what-if-ive-already-published-a-community-operator) ## Best practice I do not want to get into the details of best practice for Helm Charts. For a productive operator, we would create more test cases which verify that installations with different settings still work. If the operator is using stateful data, we would also spend some time thinking about how an update of this data can be handled. These are all application-centric best practices. There are some points that you should consider for your operator: 1. Some clusters may have very little access to the outside or have poor network connectivity. Because of this, the operator might be pulled from a different repository. The created operator should support the modification of the image repository. You can find more information on this topic here. 1. Add support for http/https proxies to the operator and to the application itself. In the enterprise environment (at least in Germany) it is not uncommon to use a proxy to connect to resources that are located outside the cluster. 1. This is not a best practice per se, but always check the log of the operator and the content of the status attribute of the CRD. It happened to me that everything seemed to work, however, the operator was missing certain permissions or the Helm Chart had a small bug. 1. Create custom goals in the makefile to save you from executing a couple of commands each time you have changed something and need to redeploy the operator. ## Summary To create an operator based on a Helm Chart is quite easy. There are some details that require some testing and not all error messages are self-explanatory. I must admit that I was quite skeptical, whether it makes sense for self written applications to create operators. After the current experience, however, I find myself thinking about a lot of situations where an operator could make my life much easier. In further articles I might go deeper, e.g. how operators can be used in a CI/CD pipeline and which advantages this might have. ## Further references - [Source code of the examples](https://github.com/olaf-meyer/helm-operator) - [Documentation of the Operator SDK](https://sdk.operatorframework.io/docs/) - [Introduction to Helm operators from Daniel Messer](https://youtu.be/N9QVJk6kjwg) - [Requirements to upload operators to OperatorHub.io](https://operatorhub.io/contribute) - [Preview of your operator at OperatorHub.io](https://operatorhub.io/preview) - [Definition of available UI components in the consol form view](https://github.com/openshift/console/blob/master/frontend/packages/operator-lifecycle-manager/src/components/descriptors/reference/reference.md) - [Definition of available UI components in the consol form view (part 2)](https://www.openshift.com/blog/openshift-4-2-declarative-dynamic-ui-for-your-operator) - [Homepage of KUTTL framework](https://kuttl.dev/docs/#pre-requisites) - [Source code of the OperatorHub.io](https://github.com/operator-framework) - [Requirements to add operators to OperatorHub.io](https://operator-framework.github.io/community-operators/contributing-where-to/) [Read more](https://omd.consol.de/blog/2021/05/03/a-journey-of-a-helm-operator-to-operatorhub.io-and-back-again/) ----------------------------------------------------------------------------------- title: "A journey of a Helm operator to OperatorHub.io and back again" date: "2021-05-03" --------------------------------------------------------------------------------
Some time ago, I started a project to create a Helm based operator for an OpenShift application. I used the Operator SDK to create the Helm operator. The Operator SDK documentation describes the parameters pretty good, and it contains a simple tutorial. it does not, however, describe the complete development cycle. This article aims to describe everything from creating the operator to the point where you can upload your operator to OperatorHub.io. We start with a basic Helm Chart. With this, you can install Nginx as a StatefulSet. You can find the source code in my github repo. Before we can start with creating an operator, we need to fulfill some prerequisites. ## Prerequisite In order to develop the operator, the following applications need to be installed: - [git](https://git-scm.com/) - [oc](https://docs.openshift.com/container-platform/4.7/cli_reference/openshift_cli/getting-started-cli.html#cli-installing-cli_cli-developer-commands)/[kubectl](https://kubernetes.io/docs/tasks/tools/) tool - Connection to running OpenShift (4.x)/Kubernetes cluster - Be logged in to a docker repository like quay.io - [Helm (optional)](https://helm.sh/docs/intro/install/) I spare you the details of how to download the applications and set them up. There are a lot of articles describing just that. ## Create the initial operator We have everything set up and we got a Nginx Helm Chart. With this Chart, Nginx can be installed as a StatefulSet. But wouldn't it be much cooler to define a Nginx CRD and Kubernetes/OpenShift installs Nginx for us? If we change something in the CRD, the operator should take care of applying the changes, and it should make sure that a user can only change limited properties in the application. Of course, we could create an ansible- or go operator, but creating a Helm based operator might be more sufficient for a lot of use cases. Our source Helm Chart is in the subfolder `nginx-sts` of the current directory. Let's create the initial Helm operator. We do this with the command `operator-sdk init` and add the reference to the used source Helm Chart `--helm-chart=nginx-sts` and some metadata (For some reason it is not possible to describe the location of the Helm Chart like this `--helm-chart=./nginx-sts`.). In our case the command looks like this: ``` bash $>operator-sdk init --plugins=helm.sdk.operatorframework.io/v1 --group=demo --domain=consol.de --helm-chart=nginx-sts --project-name=nginx-sts Created helm-charts/nginx-sts Generating RBAC rules I0315 16:08:14.986191 199443 request.go:655] Throttling request took 1.004878088s, request: GET:https://api.crc.testing:6443/apis/coordination.k8s.io/v1beta1?timeout=32s WARN[0004] The RBAC rules generated in config/rbac/role.yaml are based on the Chart's default manifest. Some rules may be missing for resources that are only enabled with custom values, and some existing rules may be overly broad. Double check the rules generated in config/rbac/role.yaml to ensure they meet the operator's permission requirements. ``` The Operator SDK is expecting that the (source) Helm Chart is located in a different folder than where this operator files will be generated. Because of this, we need to delete the original Helm Chart folder (`nginx-sts`). Don't worry, a copy of the Helm Chart has been created in the folder `helmcharts/nginx-sts`. We get a warning that we should double check the created roles and rolebinding. In our case, we don't need to worry about that. You can, however, check it if you want to. The created file structure of the operator looks like this: ``` bash $>tree . ├── config │   ├── crd │   │   ├── bases │   │   │   └── demo.consol.de_nginxsts.yaml │   │   └── kustomization.yaml │   ├── default │   │   ├── kustomization.yaml │   │   └── manager_auth_proxy_patch.yaml │   ├── manager │   │   ├── kustomization.yaml │   │   └── manager.yaml │   ├── prometheus │   │   ├── kustomization.yaml │   │   └── monitor.yaml │   ├── rbac │   │   ├── auth_proxy_client_clusterrole.yaml │   │   ├── auth_proxy_role_binding.yaml │   │   ├── auth_proxy_role.yaml │   │   ├── auth_proxy_service.yaml │   │   ├── kustomization.yaml │   │   ├── leader_election_role_binding.yaml │   │   ├── leader_election_role.yaml │   │   ├── nginxsts_editor_role.yaml │   │   ├── nginxsts_viewer_role.yaml │   │   ├── role_binding.yaml │   │   └── role.yaml │   ├── samples │   │   ├── demo_v1alpha1_nginxsts.yaml │   │   └── kustomization.yaml │   └── scorecard │   ├── bases │   │   └── config.yaml │   ├── kustomization.yaml │   └── patches │   ├── basic.config.yaml │   └── olm.config.yaml ├── Dockerfile ├── helm-charts │   └── nginx-sts │   ├── Chart.yaml │   ├── templates │   │   ├── configmap.yaml │   │   ├── _helpers.tpl │   │   ├── NOTES.txt │   │   ├── route.yaml │   │   ├── service.yaml │   │   ├── stateful_sets.yaml │   │   └── tests │   │   └── test-connection.yaml │   └── values.yaml ├── Makefile ├── PROJECT ├── README.md └── watches.yaml 15 directories, 39 files ``` With these files, we will be able to create and deploy the operator in an OpenShift or Kubernetes cluster. First, let us modify the created CRD (config/crd/bases/demo.consol.de_nginxsts.yaml) to be more strict. By this, we can use input validation of Kubernetes when we create a resource. Also, we provide the end-user with a speaking documentation, so that we can focus on the fun parts, instead of working on support tickets. Anyhow, we are going to change the generated yaml from: ```yaml --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: nginxsts.demo.consol.de spec: group: demo.consol.de names: kind: NginxSts listKind: NginxStsList plural: nginxsts singular: nginxsts scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: NginxSts is the Schema for the nginxsts API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: Spec defines the desired state of NginxSts type: object x-kubernetes-preserve-unknown-fields: true status: description: Status defines the observed state of NginxSts type: object x-kubernetes-preserve-unknown-fields: true type: object served: true storage: true subresources: status: {} ``` To this strict and descriptive version (changes are highlighted): {% highlight yaml hl_lines="35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94"%} --- apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: nginxsts.demo.consol.de spec: group: demo.consol.de names: kind: NginxSts listKind: NginxStsList plural: nginxsts singular: nginxsts scope: Namespaced versions: - name: v1alpha1 schema: openAPIV3Schema: description: NginxSts is the Schema for the nginxsts API properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: Spec defines the desired state of NginxSts type: object required: - replicaCount properties: image: description: Configuration of the used Nginx image type: object properties: pullPolicy: description: 'Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images' type: string pattern: '^(Always|Never|IfNotPresent)$' repository: description: The alternative image repository that should be used. type: string tag: description: The alternative image tag that should be used. type: string replicaCount: description: The number of replicas of the Nginx StatefulSet. type: integer default: 1 resources: description: The resources for the Nginx pod. type: object properties: limits: description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object properties: cpu: type: string pattern: '^(\d+m|\d+(\.\d{1,3})?)$' memory: type: string pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$' requests: description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object properties: cpu: type: string pattern: '^(\d+m|\d+(\.\d{1,3})?)$' memory: type: string pattern: '^(\d+(e\d+)?|\d+(\.\d+)?(e\d+)?[EPTGMK]i?)$' service: description: Definition of the Nginx service type: object properties: port: description: The port that will be used for the Nginx container. The defined port number must be greater than 1024. type: integer minimum: 1024 type: description: type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer type: string pattern: '^(ExternalName|ClusterIP|NodePort|LoadBalancer)$' status: description: Status defines the observed state of NginxSts type: object x-kubernetes-preserve-unknown-fields: true type: object served: true storage: true subresources: status: {} {% endhighlight %} Looks much better, doesn't it? A description about how to define CRD can be found [here](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/). ## Build and run the operator Building and running the operator is fairly easy. We just need to execute the following command: ``` bash export USERNAME= make docker-build docker-push IMG=quay.io/$USERNAME/nginx-operator:v0.0.1 ``` In my case it looks like this: ``` bash export USERNAME=omeyer make docker-build docker-push IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 ``` After the upload is finished, run this command to deploy the operator manually: ``` bash make deploy IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 ``` --- **Hint** Make sure that the destination registry is public and can be accessed without password. Also, note that the deployment of the operator has the `ImagePullPolicy: IfNotPresent`, which means that the operator image might not be reloaded when the operator images change. For changes in the operator, you either need to create new tags for each change or change the `ImagePullPolicy` to `Always`. I prefer the latter. --- ## Test the operator To test the operator, we create a Nginx CRD in the cluster. This should, in return, cause the operator to deploy a new nginx-sts application. We can use the sample CRD in the folder `config/samples/demo_v1alpha1_nginxsts.yaml` to create the CRD. The file's content looks like this: ``` yaml apiVersion: demo.consol.de/v1alpha1 kind: NginxSts metadata: name: nginxsts-sample spec: replicaCount: 3 ``` To create the resource, we execute: ``` bash oc apply -f config/samples/demo_v1alpha1_nginxsts.yaml ``` --- **Hint** I modified the samples file, so that it contains only the replicas and not all other attributes from Helm's `values.yaml`. You can look up the complete CRD file in my GitHub repo. --- As you can see, 3 replicas of the Nginx StatefulSet have been started, and I am able to connect to it. I call this a success. ``` bash $> oc get pod -n default NAME READY STATUS RESTARTS AGE nginxsts-sample-nginx-sts-0 1/1 Running 0 3h17m nginxsts-sample-nginx-sts-1 1/1 Running 0 3h17m nginxsts-sample-nginx-sts-2 1/1 Running 0 3h17m $> oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD nginxsts-sample-nginx-sts nginxsts-sample-nginx-sts-default.apps-crc.testing nginxsts-sample-nginx-sts http None $> curl -v nginxsts-sample-nginx-sts-default.apps-crc.testing ... * Connected to nginxsts-sample-nginx-sts-default.apps-crc.testing (192.168.130.11) port 80 (#0) > GET / HTTP/1.1 > Host: nginxsts-sample-nginx-sts-default.apps-crc.testing > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Server: nginx/1.19.7 < Date: Thu, 18 Mar 2021 12:09:25 GMT < Content-Type: text/html < Content-Length: 612 < Last-Modified: Tue, 16 Feb 2021 16:25:34 GMT < ETag: "602bf1fe-264" < X-PodName: nginxsts-sample-nginx-sts-0 < Accept-Ranges: bytes < Set-Cookie: 25f93612584c1f45a174dd9568118c21=b331a06e7f6d6d15096677f0cf1b21fe; path=/; HttpOnly < Cache-control: private ... ``` ## Create an operator bundle So far, we created all resources to deploy the operator. Based on our CRD this operator is able to deploy applications. If we just need that, we can wrap everything up and call it a day. But wouldn't it be nice, if our customers could use OLM (Operator Lifecycle Manager) to find our operator in the OperatorHub.io and install it on their cluster through just some clicks? I think so, too. Let's get it done and let's create the resources needed . --- **Hint** If you are using an OpenShift 4.x cluster you do not need to install OLM (Operator Lifecycle Manager) on your cluster (otherwise you can find a description of how to do that [here](https://sdk.operatorframework.io/docs/olm-integration/quickstart-bundle/#enabling-olm)). It comes with the OLM included and configured. To test that it works properly, run: ``` bash $>operator-sdk olm status --olm-namespace openshift-operator-lifecycle-manager INFO[0002] Fetching CRDs for version "0.16.1" INFO[0002] Using locally stored resource manifests INFO[0002] Successfully got OLM status for version "0.16.1" NAME NAMESPACE KIND STATUS operators.operators.coreos.com CustomResourceDefinition Installed operatorgroups.operators.coreos.com CustomResourceDefinition Installed installplans.operators.coreos.com CustomResourceDefinition Installed clusterserviceversions.operators.coreos.com CustomResourceDefinition Installed subscriptions.operators.coreos.com CustomResourceDefinition Installed system:controller:operator-lifecycle-manager ClusterRole Installed aggregate-olm-edit ClusterRole Installed aggregate-olm-view ClusterRole Installed catalogsources.operators.coreos.com CustomResourceDefinition Installed olm Namespace namespaces "olm" not found olm-operator-binding-olm ClusterRoleBinding clusterrolebindings.rbac.authorization.k8s.io "olm-operator-binding-olm" not found olm-operator olm Deployment deployments.apps "olm-operator" not found catalog-operator olm Deployment deployments.apps "catalog-operator" not found olm-operator-serviceaccount olm ServiceAccount serviceaccounts "olm-operator-serviceaccount" not found operators Namespace namespaces "operators" not found global-operators operators OperatorGroup operatorgroups.operators.coreos.com "global-operators" not found olm-operators olm OperatorGroup operatorgroups.operators.coreos.com "olm-operators" not found packageserver olm ClusterServiceVersion clusterserviceversions.operators.coreos.com "packageserver" not found operatorhubio-catalog olm CatalogSource catalogsources.operators.coreos.com "operatorhubio-catalog" not found ``` --- Let's initialize the resources that are needed to add our operator to OperatorHub.io by executing this command ``` bash make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 operator-sdk generate kustomize manifests -q Display name for the operator (required): > Nginx-StatefulSet Description for the operator (required): > This will deploy Nginx as a StatefulSet. This is an example! Don't use it in production! Provider's name for the operator (required): > ConSol Software GmbH Any relevant URL for the provider name (optional): > https://www.consol.de/ Comma-separated list of keywords for your operator (required): > example,nginx Comma-separated list of maintainers and their emails (e.g. 'name1:email1, name2:email2') (required): > olaf.meyer@consol.de cd config/manager && /home/olaf/Projects/operator/helm_operator/bin/kustomize edit set image controller=quay.io/omeyer/nginx-sts-operator:v0.0.1 /home/olaf/Projects/operator/helm_operator/bin/kustomize build config/manifests | operator-sdk generate bundle -q --overwrite --version 0.0.1 operator-sdk bundle validate ./bundle INFO[0000] Found annotations file bundle-dir=bundle container-tool=docker INFO[0000] Could not find optional dependencies file bundle-dir=bundle container-tool=docker INFO[0000] All validation tests have completed successfully ``` What happened after we have executed this command? Let's have a look ``` bash $>tree ... ├── bundle │   ├── manifests │   │   ├── demo.consol.de_nginxsts.yaml │   │   ├── nginx-sts.clusterserviceversion.yaml │   │   ├── nginx-sts-controller-manager-metrics-service_v1_service.yaml │   │   └── nginx-sts-metrics-reader_rbac.authorization.k8s.io_v1_clusterrole.yaml │   ├── metadata │   │   └── annotations.yaml │   └── tests │   └── scorecard │   └── config.yaml ├── bundle.Dockerfile ├── config │   ├── crd ... ``` A new directory `bundle` and a file `bundle.Dockerfile` have been created. Before we have a closer look at these files, let's build and deploy the bundle operator and see how it looks like in OpenShift. To build the bundle operator, we run these commands: ``` bash export BUNDLE_IMG="quay.io/$USERNAME/nginx-sts-operator-bundle:v0.0.1" make bundle-build BUNDLE_IMG=$BUNDLE_IMG make docker-push IMG=$BUNDLE_IMG ``` After this, the bundle operator has been uploaded to the specified quay.io repo. Now we need to deploy it with this command: ``` bash $>operator-sdk run bundle $BUNDLE_IMG INFO[0040] Successfully created registry pod: quay-io-omeyer-nginx-sts-operator-bundle-v0-0-1 INFO[0040] Created CatalogSource: nginx-sts-catalog INFO[0040] OperatorGroup "operator-sdk-og" created INFO[0040] Created Subscription: nginx-sts-v0-0-1-sub INFO[0045] Approved InstallPlan install-8j9wf for the Subscription: nginx-sts-v0-0-1-sub INFO[0045] Waiting for ClusterServiceVersion "default/nginx-sts.v0.0.1" to reach 'Succeeded' phase INFO[0045] Waiting for ClusterServiceVersion "default/nginx-sts.v0.0.1" to appear FATA[0120] Failed to run bundle: error waiting for CSV to install: timed out waiting for the condition ``` It has failed?! Let's see for ourselves. Looks like the operator has successfully been installed. It might just have taken a bit too long. ![](installed_nginx-sts_operator.jpg) ![](installed_nginx-sts_operator_details.jpg) We are now able to create a new CRD. The form and yaml view look very good for a starting point. ![](installed_nginx-sts_operator_new_crd_form.jpg) Let's do a couple of changes: - The url `https://nginx-sts.domain` and description for it is not correct. - There is no short description for the operator. - And so on ... To do these changes, let us have a look at the folder `bundle/manifests`. It contains 4 files: 1. `demo.consol.de_nginxsts.yaml` is the file for the definition of the `nginxsts.demo.consol.de` CRD. 1. `nginx-sts.clusterserviceversion.yaml` is the ClusterServiceVersion (like stated in the name) and is used by the OLM to set up the `nginx-sts` operator. Furthermore, it contains all information that is used when installing and / or modifying the operator. 1. The other two files are used to monitor the operator. ## Modify the cluster service version To change the information that is displayed in OpenShift, we need to modify the cluster service version file. However, there are two cluster service versions in the project. The file `bundle/manifests/nginx-sts.clusterserviceversion.yaml` has been created by the file `config/manifests/bases/nginx-sts.clusterserviceversion.yaml`. If we modify the latter and run `make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1` again, the changes will be transferred to bundle file. So let us to this. I added the highlighted lines to the `nginx-sts.clusterserviceversion.yaml`: {% highlight bash hl_lines="8 9 10 11 12 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 46 66 67 69 70 73 74"%} apiVersion: operators.coreos.com/v1alpha1 kind: ClusterServiceVersion metadata: annotations: alm-examples: '[]' capabilities: Basic Install categories: Example operator containerImage: quay.io/omeyer/nginx-sts-operator:v0.0.1 createdAt: "1970-01-01T00:00:00Z" description: This will deploy Nginx as a StatefulSet. This is an example! Don't use it in production! repository: https://github.com/olaf-meyer/helm-operator support: None - Example operator name: nginx-sts.v0.0.0 namespace: placeholder spec: apiservicedefinitions: {} customresourcedefinitions: owned: - description: Definition how to setup the Nginx as a StatefulSet. displayName: Nginx-StatefulSet kind: NginxSts name: nginxsts.demo.consol.de resources: - kind: Route name: "" version: route.openshift.io/v1 - kind: Service name: "" version: v1 - kind: StatefulSet name: "" version: v1 - kind: Pod name: "" version: v1 specDescriptors: - description: Limits describes the minimum/maximum amount of compute resources required/allowed displayName: Resource Requirements path: resources x-descriptors: - urn:alm:descriptor:com.tectonic.ui:resourceRequirements version: v1alpha1 description: This will deploy Nginx as a StatefulSet. This is an example! Don't use it in production! displayName: Nginx-StatefulSet icon: - base64data: iVBORw0KGgoAAAANSUhEUgAAAQIAAAEsCAYAAAAo6b0WAAAxjUlEQVR42u19CXxU5dX+C8gmLrjUqkVFRcg6EVGUoi1arSKuVepn60JdoqIhMyEkM3dJxr1KP1Ra6x+1tlY/9Qv6uS ..... mediatype: image/png install: spec: deployments: null strategy: "" installModes: - supported: false type: OwnNamespace - supported: false type: SingleNamespace - supported: false type: MultiNamespace - supported: true type: AllNamespaces keywords: - example - nginx links: - name: ConSol Software GmbH url: https://www.consol.de/ maintainers: - email: olaf.meyer@consol.de name: Olaf Meyer maturity: alpha provider: name: ConSol Software GmbH url: https://www.consol.de/ version: 0.0.0 {% endhighlight %} Most changes are self explaining. There are a couple of changes that I want to talk about: 1. The whole attribute `customresourcedefinitions` is used for displaying the defined CRD in the Web-GUI. It is not intended as a replacement of the CRD definition. The list of resources doesn't have to be complete, but should contain the most import resources for your application. Also, you can define which kinds of controls should be used. With this we can use the OpenShift resource form for entering the limits and requests. 1. The icon attribute is truncated because even a small image can become too long for demo purposes. You can find the complete definition in my GitHub Repo for this article. 1. In a production scenario I would calculate the attribute `createdAt` and `containerImage` in a CI/CD pipeline. There is one more point that I would like to change in the generated `ClusterServiceVersion` file. The deployment is using the default service account, which can lead to some problems. We are going to change that. For this, we need to create a service account `nginx-sts-controller-manager` in the file `service_account.yaml` in the folder `config/rbac`. The next step is to add this service account to the `kustomization.yaml`, so that it will be processed when we update the bundle files. After this, we replace the `default` service account with `nginx-sts-controller-manager` in the files `config/rbac/auth_proxy_role_binding.yaml`, `config/rbac/leader_election_role_binding.yaml` and `config/rbac/role_binding.yaml`. The last part is to use this service account in the deployment of the operator. For this, we add the line `serviceAccountName: nginx-sts-controller-manager` to the file `config/manager/manager.yaml`. If we now run the command `make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1`, the files in the folder `bundle/manifests` will contain our changes and we are ready to test the latest version. Because we have changed a lot, we need to `uninstall` the current operator. The `cleanup` command requires a parameter `operatorPackageName`. You can find this value i.e. in the file `bundle/metadata/annotations.yaml` in the value `operators.operatorframework.io.bundle.package.v1`. In our case this is `nginx-sts`. So the cleanup command looks like this: ``` bash operator-sdk cleanup nginx-sts ``` After this, we can rebuild the container image and deploy it ``` bash export USERNAME=omeyer export BUNDLE_IMG="quay.io/$USERNAME/nginx-sts-operator-bundle:v0.0.1" make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 make bundle-build BUNDLE_IMG=$BUNDLE_IMG make docker-push IMG=$BUNDLE_IMG operator-sdk run bundle $BUNDLE_IMG ``` ## Verify the new version Let's have a look at the new version of the operator. From my point of view the open points are closed. In the installed operator overview there are neither empty fields nor fields filled with dummy values. ![](installed_nginx-sts_operator_install_new.jpg) On the operator overview page the description looks fine as well and shows the ConSol Logo. ![](installed_nginx-sts_operator_details_new.jpg) The last point is the changed editor for entering the resources of the pod. On the right hand side we get a documentation of the CRD. Even if it is not shown in the image, we have an auto completion in the editor as well. ![](installed_nginx-sts_operator_new_crd_yaml.jpg) All looks good ... ## Create Scorecard tests Almost! We should add tests to the operator to further automate the build and deployment of the operator. Basic tests are already included. We can execute them with the command: ``` bash operator-sdk scorecard bundle -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-crds-have-validation] Labels: "suite":"olm" "test":"olm-crds-have-validation-test" Results: Name: olm-crds-have-validation State: pass Log: Loaded 1 Custom Resources from alm-examples Loaded CustomresourceDefinitions: [&CustomResourceDefinition{...}] -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test basic-check-spec] Labels: "suite":"basic" "test":"basic-check-spec-test" Results: Name: basic-check-spec State: pass -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-crds-have-resources] Labels: "suite":"olm" "test":"olm-crds-have-resources-test" Results: Name: olm-crds-have-resources State: pass Log: Loaded ClusterServiceVersion: nginx-sts.v0.0.1 -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-status-descriptors] Labels: "suite":"olm" "test":"olm-status-descriptors-test" Results: Name: olm-status-descriptors State: fail Errors: nginxsts.demo.consol.de does not have a status descriptor Log: Loaded ClusterServiceVersion: nginx-sts.v0.0.1 Loaded 1 Custom Resources from alm-examples -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-spec-descriptors] Labels: "suite":"olm" "test":"olm-spec-descriptors-test" Results: Name: olm-spec-descriptors State: fail Suggestions: Add a spec descriptor for replicaCount Errors: replicaCount does not have a spec descriptor Log: Loaded ClusterServiceVersion: nginx-sts.v0.0.1 Loaded 1 Custom Resources from alm-examples -------------------------------------------------------------------------------- Image: quay.io/operator-framework/scorecard-test:v1.5.0 Entrypoint: [scorecard-test olm-bundle-validation] Labels: "suite":"olm" "test":"olm-bundle-validation-test" Results: Name: olm-bundle-validation State: pass Log: time="2021-03-22T16:33:04Z" level=debug msg="Found manifests directory" name=bundle-test time="2021-03-22T16:33:04Z" level=debug msg="Found metadata directory" name=bundle-test time="2021-03-22T16:33:04Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test time="2021-03-22T16:33:04Z" level=info msg="Found annotations file" name=bundle-test time="2021-03-22T16:33:04Z" level=info msg="Could not find optional dependencies file" name=bundle-test ``` Looks like there are two errors. It seems that we should provide a description for replicaCount and for the status in the cluster service version. Let's fix that. To provide a description for replicaCount, add the following section to `config/manifests/bases/nginx-sts.clusterserviceversion.yaml` ``` yaml ... specDescriptors: - description: The number of replicas of the Nginx StatefulSet. displayName: Replicas path: replicaCount x-descriptors: - urn:alm:descriptor:com.tectonic.ui:podCount - description: Limits describes the minimum/maximum amount of compute resources required/allowed displayName: Resource Requirements path: resources x-descriptors: - urn:alm:descriptor:com.tectonic.ui:resourceRequirements version: v1alpha1 ... ``` With this change we do not only resolve the error but also change the style of the input field. This is quite neat. ![](installed_nginx-sts_operator_new_crd_form_new.jpg) Originally, I was under the impression that the second error `nginxsts.demo.consol.de does not have a status descriptor` could not be fixed. In the status descriptor we can define which values will be used in the status attribute. Also, we define which attribute and how it is displayed on the first tab (Display view). The values in the status attribute are written by the Helm operator and at the time being, it is not possible to change them. The Helm operator writes the attributes `conditions`, `manifest` and `name` to the status. Using the attribute `conditions` in the `statusDescriptors` is not working, because the condition is already displayed at the end of the page. The content of the attribute `manifest` is too long. Let's use the attribute `name` in the `statusDescriptors`: ``` yaml ... statusDescriptors: - description: Name of the created application displayName: My-Application name path: deployedRelease.name x-descriptors: - 'urn:alm:descriptor:text' ... ``` The result looks like this: ![](installed_nginx-sts_operator_installed_crd_details.jpg) If we rerun the Scorecard tests, all tests pass. --- Hint: To test the changes, we deleted the operator. After this, I recreated the bundle definition and rebuilt the operator and deployed it. The following commands were used: ``` bash operator-sdk cleanup nginx-sts export USERNAME=omeyer export BUNDLE_IMG="quay.io/$USERNAME/nginx-sts-operator-bundle:v0.0.1" make bundle IMG=quay.io/$USERNAME/nginx-sts-operator:v0.0.1 make bundle-build BUNDLE_IMG=$BUNDLE_IMG make docker-push IMG=$BUNDLE_IMG operator-sdk run bundle $BUNDLE_IMG ``` --- ## Create example integration test So far, the tests verified only the cluster service version. What is missing is a test to check whether the operator itself is working. For this kind of tests the Operator SDK offers an integration of the KUTTL test framework. At the time being, it is required to make the changes for the KUTTL test directly in folder `bundle` and not in the folder `config`. If we run the command `make bundle` our changes for the KUTTL test are not overwritten. Let's create a simple KUTTL test case. First we need to create a KUTTL configuration file `kuttl-test.yaml` in a new folder `bundle/tests/scorecard/kuttl`. The content of the configuration file `kuttl-test.yaml` looks like this: ``` yaml apiVersion: kudo.dev/v1beta1 kind: TestSuite parallel: 4 timeout: 120 startControlPlane: false ``` This is the default configuration of the Operator SDK. It just configures the number of parallel test executions, and the amount of time the KUTTL test is allowed to take. Next, we create a folder `install-nginx-sts` for grouping the tests. In this folder we add two files. With the first file `00-install-nginx-sts.yaml` we set up the test case. In our case this is an Nginx StatefulSet with 4 replicas. It looks like this: ```yaml apiVersion: demo.consol.de/v1alpha1 kind: NginxSts metadata: name: nginxsts-sample namespace: default spec: replicaCount: 4 ``` In the second file `00-assert.yaml` we defined how the result should look like. This means that the StatefulSet with 4 replicas is running. The file looks like this: ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: nginxsts-sample-nginx-sts namespace: default status: readyReplicas: 4 ``` In the last step we need to add the KUTTL tests to the Scorecard tests by adding the following line to the file `bundle/tests/scorecard/config.yaml`: {% highlight bash hl_lines="9 10 11 12"%} ... - entrypoint: - scorecard-test - olm-status-descriptors image: quay.io/operator-framework/scorecard-test:v1.5.0 labels: suite: olm test: olm-status-descriptors-test - image: quay.io/operator-framework/scorecard-test-kuttl:v1.5.0 labels: suite: kuttlsuite test: kuttltest1 {% endhighlight %} When calling the Scorecard tests, we need to provide the service account with sufficient permissions to create the nginx-sts CRD. Additionally, we need to add the attribute `--wait-time=60s` in order to wait a bit longer till Nginx is deployed (It is deploying quite quick, but not that quick that it passes the test). So the command to run the Scorecard and KUTTL tests looks like this: ``` bash operator-sdk scorecard bundle --service-account=nginx-sts-nginx-sts-controller-manager --wait-time=60s ``` ## How to upload operator to OperatorHub.io My initial plan was to add a section how to upload our operator to OperatorHub.io. After reviewing the documentation, however, I came to the conclusion that the details with all hooks and cranks need to be explained in a separate article. Even that would only rephrase the documentation, and it does not give much benefit. You can find the documentation, how to commit a community Kubernetes or OpenShift operator, here: [Community operators](https://operator-framework.github.io/community-operators/contributing-prerequisites/). If you want to create an OpenShift operator that is certified by Red Hat, the documentation for this can be found here: [Certified operator build guide](https://redhat-connect.gitbook.io/certified-operator-guide/what-if-ive-already-published-a-community-operator) ## Best practice I do not want to get into the details of best practice for Helm Charts. For a productive operator, we would create more test cases which verify that installations with different settings still work. If the operator is using stateful data, we would also spend some time thinking about how an update of this data can be handled. These are all application-centric best practices. There are some points that you should consider for your operator: 1. Some clusters may have very little access to the outside or have poor network connectivity. Because of this, the operator might be pulled from a different repository. The created operator should support the modification of the image repository. You can find more information on this topic here. 1. Add support for http/https proxies to the operator and to the application itself. In the enterprise environment (at least in Germany) it is not uncommon to use a proxy to connect to resources that are located outside the cluster. 1. This is not a best practice per se, but always check the log of the operator and the content of the status attribute of the CRD. It happened to me that everything seemed to work, however, the operator was missing certain permissions or the Helm Chart had a small bug. 1. Create custom goals in the makefile to save you from executing a couple of commands each time you have changed something and need to redeploy the operator. ## Summary To create an operator based on a Helm Chart is quite easy. There are some details that require some testing and not all error messages are self-explanatory. I must admit that I was quite skeptical, whether it makes sense for self written applications to create operators. After the current experience, however, I find myself thinking about a lot of situations where an operator could make my life much easier. In further articles I might go deeper, e.g. how operators can be used in a CI/CD pipeline and which advantages this might have. ## Further references - [Documentation of the Operator SDK](https://sdk.operatorframework.io/docs/) - [Introduction to Helm operators from Daniel Messer](https://youtu.be/N9QVJk6kjwg) - [Requirements to upload operators to OperatorHub.io](https://operatorhub.io/contribute) - [Preview of your operator at OperatorHub.io](https://operatorhub.io/preview) - [Definition of available UI components in the consol form view](https://github.com/openshift/console/blob/master/frontend/packages/operator-lifecycle-manager/src/components/descriptors/reference/reference.md) - [Definition of available UI components in the consol form view (part 2)](https://www.openshift.com/blog/openshift-4-2-declarative-dynamic-ui-for-your-operator) - [Homepage of KUTTL framework](https://kuttl.dev/docs/#pre-requisites) - [Source code of the OperatorHub.io](https://github.com/operator-framework) - [Requirements to add operators to OperatorHub.io](https://operator-framework.github.io/community-operators/contributing-where-to/) [Read more](https://omd.consol.de/blog/2021/05/03/a-journey-of-a-helm-operator-to-operatorhub.io-and-back-again/) ----------------------------------------------------------------------------------- title: "RabbitMQ" date: "2021-02-09" --------------------------------------------------------------------------------
The first version of RabbitMQ has been released in 2007. Back in these days, the goal was to provide *a complete open source implementation of Advanced Message Queuing Protocol (AMQP)*, aiming at *modern messaging needs* such as high availability, high performance, scalability and security. Nowadays, RabbitMQ is one of the most popular message brokers and can be found in several domains. This article lights up core concepts and compares it with ActiveMQ Artemis and AWS SQS. ## RabbitMQ The integration of multiple applications requires some kind of communication. From a high-level perspective the kinds of communication can be grouped into *direct* and *indirect* communication. Direct communication means that a direct network connection will be established between sender and receiver of data. This happens when using HTTP, REST, GraphQL or grpc. Indirect communication flows via an infrastructural component between sender and receiver of data. A file on a shared filesystem is a simple example. An intermediate FTP server, a database or a messaging system are other examples. The main reason why these architectures are used is the provided degree of decoupling between sender and receiver. With direct communication, sender and receiver have to be available at time of communication. By introducing an intermediate component, data can be stored and delivered when the receiver is available. Modern messaging solutions provide a lot more features and one of them is RabbitMQ. ### Messaging Concepts and Features RabbitMQ provides both durable and non-durable queues. While durable queues survive a node restart, messages in non-durable queues will be lost. Topics, also known as Publish/Subscribe, are also supported by RabbitMQ. With acknowledged messages, RabbitMQ provides an "at least once" delivery guarantee. If acknowledgements are disabled, e.g. to increase throughput, this will change to an "at most once" semantic. The routing of messages to queues can be delayed by a specified amount of milliseconds. This will happen entirely inside RabbitMQ. The Time-To-Live property can be used to flag messages that become irrelevant to consumers after a specified amount of time. Messages that are not consumed within the Time-To-Live will be removed from RabbitMQ. In the world of RabbitMQ, producers send messages to so-called *exchanges* and not to the target queues directly. Which queue a message will be routed to, depends on the *routing key* which links exchanges and queues. If the routing key of a message matches a routing key of a queue bound to that exchange, this message will be delivered to the queue. For queues, it is also possible to use wildcards `*` and `#` on routing keys. In that case, a queue will get all messages which match that pattern. --- Example: Messages get published with the routing keys `world.eu` and `world.na`. A queue declared with the routing key `world.*` will receive all messages. Another queue with the routing key `world.eu` will only get those messages. --- When a consumer is not able to process a message, it is possible to reject messages in order to do some error handling supported by RabbitMQ. Rejected messages go back to a position *closer to the queue head*. When using RabbitMQ with AMQP 0-9-1, transactions are also provided. In the context of RabbitMQ, transactions are a *batching* feature on publishes, acknowledgements and rejections of messages. Creation of resources such as queues are not protected by transactions. ### Management UI & Monitoring RabbitMQ comes with a built-in management UI. It provides information about consumers, queues, messages, users and the current routing configuration. Besides, basic monitoring is also included. For in-depth monitoring, version `3.8.0+`, provides support for Prometheus and Grafana. rabbit-ui ### Protocols, APIs and Clients RabbitMQ supports AMQP 0-9-1, AMQP 1 (via plugin), MQTT and STOMP and clients exist for several programming languages. For Java clients, right now, there are multiple approaches. * [A native AMQP-Client, maintained by RabbitMQ](https://www.rabbitmq.com/java-client.html) - Vanilla Java Client * [RabbitMQ JMS 1.1 Client](https://github.com/rabbitmq/rabbitmq-jms-client) - If you like to use the JMS 1.1. API with RabbitMQ. Not all features implemented, e.g. XA transactions * [Spring AMQP](https://spring.io/projects/spring-amqp) - Provides a solid integration into the Spring framework * [Spring Cloud Stream RabbitMQ Binder](https://cloud.spring.io/spring-cloud-static/spring-cloud-stream-binder-rabbit/2.2.0.M1/spring-cloud-stream-binder-rabbit.html) - Spring Cloud Stream abstraction. Makes it easier to migrate from RabbitMQ to e.g. Apache Kafka #### Code Sample with Spring AMQP Launch a local instance of rabbitmq with `docker run --rm -p 5672:5672 -p 15672:15672 docker.io/rabbitmq:3.8.9-management`. Port `5672` is for AMQP, `15672` exposes the management console. Add the `spring-boot-starter-amqp`: ```xml org.springframework.boot spring-boot-starter-amqp ``` Then create a message receiver ```java import org.springframework.amqp.rabbit.annotation.Queue; import org.springframework.amqp.rabbit.annotation.RabbitListener; import org.springframework.stereotype.Component; @Component public class Receiver { @RabbitListener(queuesToDeclare = @Queue("my-queue")) public void receiveMessage(String message) { System.out.println("Received <" + message + ">"); } } ``` And finally send and receive a message ```java import org.springframework.amqp.rabbit.core.RabbitTemplate; import org.springframework.boot.CommandLineRunner; import org.springframework.stereotype.Component; @Component public class Runner implements CommandLineRunner { private final RabbitTemplate rabbitTemplate; public Runner(RabbitTemplate rabbitTemplate) { this.rabbitTemplate = rabbitTemplate; } @Override public void run(String... args) throws Exception { System.out.println("Sending message..."); rabbitTemplate.convertAndSend("my-queue","Hello from RabbitMQ"); Thread.sleep(1000); } } ``` The log output should contain ``` Sending message... Received ``` If you read the section about exchanges and routing keys carefully, you should be wondered why and how `rabbitTemplate.convertAndSend("my-queue","Hello from RabbitMQ")` works - as said before: producers send messages together with their routing key to *exchanges* and not to queues. However, the code example did not contain an exchange, but it *seemed* to contain a queue name. It works, because Spring AMQP has a few default behaviors: 1. The method `convertAndSend(String routingKey, Object object)` sends a message to the *default exchange*. This explains, why it was not necessary to set an exchange explicitly. 2. The receiver's annotation `@RabbitListener(queuesToDeclare = @Queue("my-queue"))` created a queue with the name *my-queue*, but it also created a binding with the queue name as the routing key. Just to conclude: The code sample created a queue called *my-queue* and bound it to the default exchange with the routing key *my-queue*. The producer sent its message to the default exchange with the routing key *my-queue*. ### High Availability RabbitMQ provides two setups to increase availability: Active/Active and Active/Passive. #### Active/Active In Active/Active, HA is defined on two layers: on nodes and on queues. In a cluster that consists of multiple nodes, clients can connect to any of them. However, this does not mean that the queues are automatically high available as well. Every queue is managed by a single node, the *queue master*, and *can* be replicated to other nodes if configured. However, all operations on that queue will still be performed only by the queue master no matter which node a client connected to. When the queue master goes down, another node will take over to be the new queue master. In case of network partitioning within the cluster, RabbitMQ can either retain consistency or availability. Consistency will be retained when *pause_minority* is configured. Instances on the minority side of the partition will pause themselves and wait until the partition is resolved. During this time, these instances are not available. Availability will be retained when *auto-heal* or *ignore* is configured. When a partition is detected, all instances keep running. When the partition is resolved, the state of the majority side of the partition will be used as the new state for the whole cluster. Obviously, this can lead to loss of data. #### Active/Passive In an Active/Passive setup, instances share persistent messages via the filesystem. If a node crashes, another node will initialize with the shared file system and take over. Non-persistent messages will be lost. This approach comes with the drawback of a delay when another node takes over. Besides, the Active/Passive setup seems to be more complex to [set up](https://www.rabbitmq.com/pacemaker.html). Combinations Active/Active and Active/Passive are also possible. ### Cloud & Kubernetes AWS, GCP and Azure provide ways to setup RabbitMQ clusters. While GCP and Azure offer it via their marketplaces, AWS also provides a native service that is part of [AWS MQ](https://aws.amazon.com/amazon-mq/). For the operation on Kubernetes, there is an [operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html) available. Key features are the provisioning of single-node and multi-node clusters, management of active vs. desired state and a set of monitoring tools based on Prometheus and Grafana. Upcoming versions will also provide rolling upgrades for RabbitMQ clusters. ## Comparison with ActiveMQ Artemis
In general, ActiveMQ Artemis can be seen as similar to RabbitMQ, because most features listed in this article are also supported. The following list summarizes features that are unique to ActiveMQ Artemis. * Interceptors (Allows inspection and modification of messages entering and exiting Artemis) * REST-API and a proprietary *core* API (According to the [documentation](https://activemq.apache.org/components/artemis/documentation/latest/core.html), this API is easier to use than JMS and provides more of Artemis' features.) * JMS 2.0 * JDBC instead of file system as storage layer * Embedded Mode (Allows embedding ActiveMQ into a Java application or JUnit tests) * JTA XA transactions Regarding HA, ActiveMQ Artemis also provides both Active/Active and Active/Passive setups. In contrast to RabbitMQ, the Active/Passive configuration is not limited to the shared storage concept. There is also a *replication mode*, where the state is synchronized continuously between the Active and the Passive node. Compared to the Active/Active concept in RabbitMQ, Artemis follows an *all-or-nothing*-principle regarding replication - it is not possible to replicate just single queues. Apart from that, I want to point out the good quality of the [online documentation](https://activemq.apache.org/components/artemis/documentation/latest/). ## Comparison with AWS SQS
AWS SQS is focused on queues and doesn't provide features regarding Publish/Subscribe or routing like in RabbitMQ or ActiveMQ Artemis. In order to do that, SQS can be combined with SNS. Features provided by SQS are standard queues (optimized for throughput), FIFO queues (retain ordering), delaying of single messages or entire queues. Messages in SQS are automatically deleted after at most 14 days. By setting smaller values (but at least 1 minute), the TTL feature provided by RabbitMQ and ActiveMQ Artemis can be implemented. SQS provides clients for the most common programming languages - including, but not limited to, Java, Python, Ruby, .NET, PHP and JavaScript. A JMS 1.1 client is also available. As of the time of writing this article, AWS does not provide a (mock) implementation for local testing. [newest posts on ConSol Labs]: https://labs.consol.de [Read more](https://omd.consol.de/blog/2021/02/09/rabbitmq/) ----------------------------------------------------------------------------------- title: "How to integrate Kafka with Istio on OpenShift" date: "2021-02-02" --------------------------------------------------------------------------------
Last summer I watched the Red Hat master course about Kafka from Sébastien Blanc. The Kafka setup in Kubernetes presented in the course looked pretty easy. The Kafka client implementation for Java seemed to be easy as well. Furthermore, I wanted to use Kafka for a long time, so I got the idea to extend my Istio example. Each time a service is called, a message is sent to a topic. The service (implemented in Quarkus), as well as the Kafka cluster should be in an Istio Service Mesh and secured with mTLS. I found descriptions of Joel Takvorian that Kafka works with Istio, so I knew (or at least hoped) that my plan should work. This article will describe the overall architecture of the example and which obstacles I encountered during deployment. ## The demo application and Istio setup The first step was to set up the environment for the demo application and for Istio. I forked the Red Hat Istio tutorial and added a bunch of yaml files that: - create the projects and the needed operators, - deploy Istio and the demo application and the Istio configuration for the demo application. This Istio configuration will make sure that every connection between the application is secured with mTLS by a set of `DestinationRule`s and `VirtualService`s. The configuration locks down undefined service calls by a bunch of `AuthorizationPolicy`s. This should be a more or less simplified setup for a real live application. The yaml-files and scripts for the setup can be found in my fork of the Red Hat Istio tutorial. The details are left out for brevity. ## Setup of the Kafka cluster To set up the Kafka cluster I used the Strimzi project. The setup was pretty straight forward. I configured the listener for the Broker that the Kafka traffic is not encrypted. The next step was to set up the Kafka cluster in a way that the Istio sidecar was injected to the Kafka Broker and Zookeeper. I added a couple of labels to the Kafka Broker, Zookeeper and Entity Operator to be compliant to the Istio notation. I also added the annotation `sidecar.istio.io/inject: "true"` to the Kafka Broker, Zookeeper and Entity Operator. With this annotation, the Istio sidecar container is injected to the pod in OpenShift. From my experience, to inject the Istio sidecar container to the Entity Operator is not causing any problems. The final configuration looks like this: {% highlight yaml hl_lines="27 28 30 31 32 33 34 35 43 44 45 46 47 49 50 51 52 53 54 55 62 63 64 65 67 68 69 70 71 72"%} - apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: demo-app-log-cluster labels: {} namespace: ${DEPLOYMENT_NAMESPACE} spec: kafka: config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 log.message.format.version: '2.5' version: 2.5.0 storage: type: ephemeral replicas: 1 listeners: plain: {} tls: {} authorization: type: simple template: statefulset: metadata: labels: app: kafka-cluster-demo-app-log version: 1.0 pod: metadata: labels: app: kafka-cluster-demo-app-log version: 1.0 annotations: sidecar.istio.io/inject: "true" entityOperator: topicOperator: reconciliationIntervalSeconds: 90 userOperator: reconciliationIntervalSeconds: 120 template: deployment: metadata: labels: app: kafka-entity-operator-demo-app-log version: 1.0 app-type: entity-operator pod: metadata: labels: app: kafka-entity-operator-demo-app-log version: 1.0 app-type: entity-operator annotations: sidecar.istio.io/inject: "true" zookeeper: storage: type: ephemeral replicas: 1 template: statefulset: metadata: labels: app: kafka-zookeeper-demo-app-log version: 1.0 pod: metadata: labels: app: kafka-zookeeper-demo-app-log version: 1.0 annotations: sidecar.istio.io/inject: "true" {% endhighlight %} The highlighted sections are included to deploy Istio-compliant pods and to inject the sidecar container. The service of the Kafka Broker can only be used from within the OpenShift cluster. The configuration is not intended for a production use case, but only for testing purposes. ## Extend the demo application with a Kafka client The next steps were to extend the customer-, preference- and the three recommendation-services that each time they are called, they send out messages to Kafka topics. I added a helper Pojo containing the information about the service calls. These changes to the services are all very similar. Here is the source code of the Pojo helper class: {% highlight java linenos %} @RegisterForReflection public class RestServiceCall { public enum Services { CUSTOMER, PREFERENCE, RECOMMENDATION } private Services origin; private Services destination; private String originVersion; private String comment; public RestServiceCall() { } public RestServiceCall(final Services origin, final Services destination, final String originVersion, final String comment) { this.origin = origin; this.destination = destination; this.originVersion = originVersion; this.comment = comment; } ... getter and setter for the fields ... } {% endhighlight %} As you can see, the helper class contains an origin, a destination, the version of the origin service and a comment. Let us have a look at what has been changed in the customer service, for example: {% highlight java hl_lines="14 15 16 23 44 45 46 47 48 49 50"%} @Path("/") public class CustomerResource { private static final String VERSION="v1"; private static final String RESPONSE_STRING_FORMAT = "customer => %s\n"; private final Logger logger = LoggerFactory.getLogger(getClass()); @Inject @RestClient PreferenceService preferenceService; @Inject @Channel("restServiceCall") Emitter restServiceCall; @GET @Produces(MediaType.TEXT_PLAIN) public Response getCustomer() { try { String response = preferenceService.getPreference().trim(); writeLogMessage(response); return Response.ok(String.format(RESPONSE_STRING_FORMAT, response)).build(); } catch (WebApplicationException ex) { Response response = ex.getResponse(); logger.warn("Non HTTP 20x trying to get the response from preference service: " + response.getStatus()); return Response .status(Response.Status.SERVICE_UNAVAILABLE) .entity(String.format(RESPONSE_STRING_FORMAT, String.format("Error: %d - %s", response.getStatus(), response.readEntity(String.class))) ) .build(); } catch (ProcessingException ex) { logger.warn("Exception trying to get the response from preference service.", ex); return Response .status(Response.Status.SERVICE_UNAVAILABLE) .entity(String.format(RESPONSE_STRING_FORMAT, ex.getCause().getClass().getSimpleName() + ": " + ex.getCause().getMessage())) .build(); } } private void writeLogMessage(String comment) { RestServiceCall serviceCall= new RestServiceCall(Services.CUSTOMER,Services.PREFERENCE,VERSION,comment); String out = JsonbBuilder.create().toJson(serviceCall); logger.info("RestServiceCall"+ out); KafkaRecord msg = KafkaRecord.of(1, out); restServiceCall.send(msg); } } {% endhighlight %} I added a field `restServiceCall` which is linked to a Kafka topic. The next change is the new method `writeLogMessage` which creates a `RestServiceCall` object, serializes it to JSON, creates a Kafka message with the key `1` and the value of the serialized RestServiceCall object. Finally, this message is sent to the Kafka Broker. The configuration for the Kafka integration is done in the `application.properties`. The file looks like this: {% highlight ini hl_lines="3 4 5 6 7"%} com.redhat.developer.demos.customer.rest.PreferenceService/mp-rest/url=http://preference:8080 mp.messaging.outgoing.restServiceCall.connector=smallrye-kafka mp.messaging.outgoing.restServiceCall.topic=demo-application-log-customer mp.messaging.outgoing.restServiceCall.key.serializer=org.apache.kafka.common.serialization.IntegerSerializer mp.messaging.outgoing.restServiceCall.value.serializer=org.apache.kafka.common.serialization.StringSerializer mp.messaging.outgoing.restServiceCall.bootstrap.servers=${KAFKA_BOOTSTRAP_SERVER:demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092} quarkus.log.level=INFO quarkus.log.category."de.consol".level=DEBUG quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t{id}) %s%e%n quarkus.jaeger.service-name=${JAEGER_SERVICE_NAME:customer} quarkus.jaeger.propagation=${JAEGER_PROPAGATION:b3} quarkus.jaeger.sampler-type=const quarkus.jaeger.sampler-param=1 quarkus.jaeger.endpoint=${JAEGER_ENDPOINT:http://jaeger-collector.istio-system.svc:14268/api/traces} {% endhighlight %} I just needed to define and configure the connector `smallrye-kafka`, which is the Kafka framework that I am using. In the next line is the topic to which the message is sent, followed by the used classes for serializing the key and value. Finally, the service or URL of the Kafka bootstrap server, which is the service of my Kafka cluster. The changes in the other services look similar. With these changes I was able to connect my demo application to a Kafka cluster. Both were in the same Istio data plane. So far, so good. Let us add a consumer to the Kafka cluster. ## Add a Kafka consumer to the demo application Just writing a message to a topic is a bit boring, so I decided to create a consumer printing out a small report, showing which services were used in the last couple of minutes. For this, I created a new service. The consumer service contains the class `RestServiceCall` and the actual Kafka consumer. It looks like this: {% highlight java linenos%} @Traced @ApplicationScoped public class ConsumerResource { @ConfigProperty(name = "stream-topics") List topics; @Produces public Topology getCalledServices() { Duration windowSizeMs = Duration.ofMinutes(5); Duration gracePeriodMs = Duration.ofMinutes(1); final StreamsBuilder builder = new StreamsBuilder(); JsonbSerde restServiceCallSerde = new JsonbSerde<>(RestServiceCall.class); builder.stream( topics, Consumed.with(Serdes.Integer(), restServiceCallSerde) ) .groupBy( (key, value) -> value.emptyComment().getKey(), Grouped.with( Serdes.String(), /* key (note: type was modified) */ restServiceCallSerde) /* value */) .windowedBy(TimeWindows.of(windowSizeMs).grace(gracePeriodMs).advanceBy(windowSizeMs)) .reduce((aggValue, newValue) -> aggValue /* adder */) .suppress(Suppressed.untilWindowCloses(BufferConfig.unbounded())) .toStream() .print(Printed.toSysOut()); // .to("demo-application-log-aggregated"); return builder.build(); } } {% endhighlight %} The class is using KafkaStream to process the data. The processing is done in the method `getCalledServices`. In this method the messages for previous defined topics are grouped and aggregated with a duration of 5 minutes based on the new key (origin service + origin version + destination) and printed to system out. When I made some requests the result looked liked this: ``` bash │ 2021-01-13 16:17:44,674 INFO [org.apa.kaf.str.sta.int.RocksDBTimestampedStore] (demo-application-log-consumer-aa8c2dec-5045-464d-a83d-40217651bb2e-StreamThread-1) Opening store KSTREAM-REDUCE-STATE-STORE-0000000002.1610539200000 in r │ │ [KTABLE-TOSTREAM-0000000009]: [CUSTOMER:v1:PREFERENCE@1610472600000/1610472900000], RestServiceCall [origin=CUSTOMER, destination=PREFERENCE, originVersion=v1, comment=] │ │ [KTABLE-TOSTREAM-0000000009]: [PREFERENCE:v1:RECOMMENDATION@1610472600000/1610472900000], RestServiceCall [origin=PREFERENCE, destination=RECOMMENDATION, originVersion=v1, comment=] │ │ [KTABLE-TOSTREAM-0000000009]: [RECOMMENDATION:v1:@1610472600000/1610472900000], RestServiceCall [origin=RECOMMENDATION, destination=null, originVersion=v1, comment=] ``` ## Use mTLS for the complete data plane With these changes the demo application is now using Kafka. The next step was to turn on mTLS for the connection between the demo application (including the new consumer service) and the Kafka cluster. I tried several configurations with `DestinationRule`s and `VirtualService`s which did not work. Enforcing mTLS via `PeerAuthentication` (one for each namespace), however, worked like a charm. The `PeerAuthentication`s look like this for my namespaces/projects: {% highlight yaml linenos%} ... - kind: PeerAuthentication apiVersion: security.istio.io/v1beta1 metadata: name: demo-app namespace: demo-app spec: mtls: mode: STRICT - kind: PeerAuthentication apiVersion: security.istio.io/v1beta1 metadata: name: kafka namespace: kafka spec: mtls: mode: STRICT ... {% endhighlight %} I used `STRICT` mode to prohibit non-encrypted connections between the services and Kafka. As I looked at Kiali, I could see that the two namespaces demo-app and Kafka are marked to be secure. This is indicated by the padlock in the row `Istio Config`. However, the connection (the blue arrows) between the services and the Kafka Broker seemed to be insecure (the padlock is missing). Are these connections secured or not? ![screenshot_kiali1.jpg](screenshot_kiali1.jpg) ![screenshot_kiali2.jpg](screenshot_kiali2.jpg) My first check was to look at the envoy metrics of the customer (as an example) and see if they contain any indication if the Kafka messages were using mTLS. Here are the results: {% highlight bash hl_lines="10 20"%} ... # TYPE istio_tcp_connections_closed_total counter istio_tcp_connections_closed_total{reporter="source",source_workload="consumer", source_workload_namespace="demo-app",source_principal="spiffe://cluster.local/ns/demo-app/sa/consumer", source_app="consumer",source_version="v1",destination_workload="demo-app-log-cluster-kafka", destination_workload_namespace="kafka",destination_principal="spiffe://cluster.local/ns/kafka/sa/demo-app-log-cluster-kafka", destination_app="kafka-cluster-demo-app-log",destination_version="1", destination_service="demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local", destination_service_name="demo-app-log-cluster-kafka-bootstrap",destination_service_namespace="kafka", request_protocol="tcp",response_flags="-",connection_security_policy="unknown", source_canonical_service="consumer",destination_canonical_service="kafka", source_canonical_revision="v1",destination_canonical_revision="1"} 4 istio_tcp_connections_closed_total{reporter="source",source_workload="consumer", source_workload_namespace="demo-app",source_principal="spiffe://cluster.local/ns/demo-app/sa/consumer", source_app="consumer",source_version="v1",destination_workload="demo-app-log-cluster-kafka", destination_workload_namespace="kafka",destination_principal="spiffe://cluster.local/ns/kafka/sa/demo-app-log-cluster-kafka", destination_app="kafka-cluster-demo-app-log",destination_version="1", destination_service="demo-app-log-cluster-kafka-brokers.kafka.svc.cluster.local", destination_service_name="demo-app-log-cluster-kafka-brokers",destination_service_namespace="kafka", request_protocol="tcp",response_flags="-",connection_security_policy="unknown", source_canonical_service="consumer",destination_canonical_service="kafka", source_canonical_revision="v1",destination_canonical_revision="1"} 0 ... {% endhighlight %} This does not look like the connections with Kafka are using mTLS, otherwise the attribute `connection_security_policy` should have the value `mutual_tls`. Why it is not working would be speculation on my side. It is something that I need to investigate later on. I found another way to verify that the connection is secured or at least provides a strong indication. I opened a shell on one of the envoy proxies and queried the SSL certificate of the Kafka cluster. For the strict mode I got this reply: ``` bash istio-proxy@customer-6794675f5-gkqxm:/$ openssl s_client -showcerts -servername \ -connect demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 > 2)] = 0x16)" tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 17:15:46.667232 IP 172.17.0.24.44050 > 172.17.0.11.9092: Flags [P.], seq 3560243409:3560245459, ack 1817826881, win 502, options [nop,nop,TS val 221777789 ecr 3018265579], length 2050 17:15:46.667727 IP 172.17.0.11.9092 > 172.17.0.24.44050: Flags [P.], seq 1:174, ack 2050, win 501, options [nop,nop,TS val 3018265579 ecr 221777789], length 173 ``` The results show that an SSL handshake happened, when I started sending messages to the Kafka Broker. The best way, of course, would be to store a `tcpdump` in the envoy proxy and analyse it with Wireshark, for example. IMHO, the recording of the SSL handshake was proof enough that the connection is secure. --- *Remark:* In order to execute `tcpdump`, you need to allow `allowPrivilegeEscalation` in the configmap `istio-sidecar-injector`. --- ## Add Kafka's RBAC to the application From what I know, the Istio sidecar proxy is not forwarding the client certificate to Kafka, so I cannot use this option to combine Istio's `AuthorizationPolicy` with Kafka's RBAC. Banzai Cloud's Supertubes provide a solution, but I wanted to stick to plain Kafka and Istio. I used the `plain` RBAC without transport encryption. The transport encryption is handled by Istio via mTLS. So, I settled with using `scram-sha-512` for authentication and added KafkaUser-resources to handle the authorization. Please note, that this is the definition of Strimzi. In other Kafka distributions the definition for this might look different. The definition of the Kafka cluster including authentication looks like this: {% highlight yaml hl_lines="9 10"%} - apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: demo-app-log-cluster spec: ... listeners: plain: authentication: type: scram-sha-512 authorization: type: simple template: ... {% endhighlight %} The plain authentication is now using `scram-sha-512`. I did not need to specify an external ingress since the messages are generated from the same cluster. The authorization is `simple` with using ACLs in KafkaUsers. This is fine for my use case, because I do not need authorization via KeyCloak or Open Policy Agent. I created one KafkaUser for each topic and a separate KafkaUser for the consumer. I only show one KafkaUser because the definition is quite long and similar for the other users. ``` yaml - apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaUser metadata: name: customer-writer namespace: ${DEPLOYMENT_NAMESPACE} labels: strimzi.io/cluster: demo-app-log-cluster spec: authentication: type: scram-sha-512 authorization: type: simple acls: # Example Producer Acls for topic demo-application-log-customer - resource: type: topic name: demo-application-log-customer patternType: literal operation: Write host: "*" - resource: type: topic name: demo-application-log-customer patternType: literal operation: Create host: "*" - resource: type: topic name: demo-application-log-customer patternType: literal operation: Describe host: "*" ``` This definition contains the authentication and authorization for the user, as well as the definition which operations it can perform on which topic. ## Extend demo application for Kafka authentication and authorization So, I have set up the Kafka RBAC on my Kafka cluster. The next step is to configure the services of the demo application so that it knows which KafkaUser to use and where to get the credentials. When creating a KafkaUser like above, Strimzi will create Kubernetes secrets with the credentials for the KafkaUser in the Kafka namespace. The first step is to copy the KafkaUser secrets in our demo-app namespace with this command: ``` bash kubectl get secret --namespace kafka -l \ strimzi.io/cluster=demo-app-log-cluster,strimzi.io/kind=KafkaUser \ -o json | sed 's/"namespace"\:\s*"kafka"/"namespace": "demo-app"/g' | \ jq 'del(.items[].metadata.ownerReferences)'| \ kubectl apply -f - -n demo-app ``` --- Side note: If you have an easier command, please let me know. --- The next step is to extend our Kafka configuration in the demo app to use the correct authentication and authorization with these properties. The result looks like this: {% highlight ini hl_lines="6 7 8 9 10"%} mp.messaging.outgoing.restServiceCall.connector=smallrye-kafka mp.messaging.outgoing.restServiceCall.topic=demo-application-log-customer mp.messaging.outgoing.restServiceCall.key.serializer=org.apache.kafka.common.serialization.IntegerSerializer mp.messaging.outgoing.restServiceCall.value.serializer=org.apache.kafka.common.serialization.StringSerializer mp.messaging.outgoing.restServiceCall.bootstrap.servers=${KAFKA_BOOTSTRAP_SERVER:demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092} mp.messaging.outgoing.restServiceCall.security.protocol=SASL_PLAINTEXT mp.messaging.outgoing.restServiceCall.sasl.mechanism=SCRAM-SHA-512 mp.messaging.outgoing.restServiceCall.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ username="${KAFKA_USER_NAME:customer-writer}" \ password="${KAFKA_PASSWORD}"; {% endhighlight %} By this, I defined that I want to use `SASL_PLAINTEXT` as the protocol, and the mechanism to be `SCRAM-SHA-512`. Last but not least, which class should be used and what the user and password is. I am not a big fan of hard coding username and password in an application, so they are read from the environment variables of the container. For the last step, I needed to modify the deployment to use the copied secret in an environment variable. With the necessary changes, the deployment looks like this: {% highlight yaml hl_lines="10 11 12 13 14"%} ... containers: - env: - name: JAVA_OPTIONS value: -Xms15m -Xmx15m -Xmn15m - name: KAFKA_BOOTSTRAP_SERVER value: demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 - name: JAEGER_ENDPOINT value: http://jaeger-collector.istio-system.svc:14268/api/traces - name: KAFKA_PASSWORD valueFrom: secretKeyRef: name: customer-writer key: password image: quay.io/omeyer/istio-tutorial-customer:v1.0-kafka ... {% endhighlight %} This was the final setting. Now the demo application is using Kafka with RBAC. Both are in one data plane and are using mTLS for the communication. ## Pitfalls on the way I first started the example on Red Hat OpenShift Service Mesh v1.12. This is based on Istio 1.4.6 and Kiali 1.17. The Istio version did not include a Kafka filter. The result was that the basic integration between Istio and Kafka with mTLS was not working. I found examples to use Kafka's mTLS instead of Istio's mTLS, by excluding Kafka traffic from Istio. I did not want to do this. I switched from OpenShift to minikube 1.19, deployed Istio 1.7.4 and the latest Kiali. I translated the OpenShift templates to plain yaml files. I did not encounter any problem with the basic setup in Kubernetes. I needed to adjust the configuration for the Istio sidecar since Kubernetes supports auto injection for all pods in a namespace, whereas in OpenShift we need to add an annotation for all pods where a sidecar should be injected. For some reason, I was not able to enforce mTLS with `DestinationRule`s between the namespaces of the demo-application and Kafka. I assume, that this has to do with the fact that the Kafka client is calling the pod directly via the `StateFullSet`'s `PodDNS`. When I enforced mTLS via `PeerAuthentication` between the projects, it worked as expected. --- Update: Red Hat OpenShift Service Mesh v2.0 is using a newer version of Istio (1.6), but I still cannot enable mTLS for Kafka provided by Istio. --- ## Summary I learned a lot about Kafka and Istio as I extended the demo application. I used only a small subset of Kafka's features, for example I did not tweak the sending and receiving of messages in the Kafka client. The integration between Kafka and Istio is possible, but it is not as easy as with applications that use HTTP or HTTPS. When using TCP based protocols, make sure that Istio is supporting them, otherwise the integration becomes challenging. You can find the changes in my fork of the Red Hat tutorial. I created a document how to set up the environment for OpenShift and Kubernetes. The fork also contains a `Makefile` to build the applications and branches for each version of the recommendation service. Following is a list of tutorials I used or that I found interesting: - [Kafka Master Course](https://youtu.be/jhXRFG1896s) - [Red Hat Istio tutorial](https://redhat-scholars.github.io/istio-tutorial/) - [Link to my fork of the Red Hat Istio tutorial](https://github.com/olaf-meyer/istio-tutorial/tree/olaf-meyer-kafka-recommendation-v1) - [Banzai SuperTubes](https://banzaicloud.com/products/supertubes/) - [Capture SSL handshake](https://stackoverflow.com/questions/39624745/capture-only-ssl-handshake-with-tcpdump) [Read more](https://omd.consol.de/blog/2021/02/02/how-to-integrate-kafka-with-istio-on-openshift/) ----------------------------------------------------------------------------------- title: "How to integrate Kafka with Istio on OpenShift" date: "2021-02-02" --------------------------------------------------------------------------------
Last summer I watched the Red Hat master course about Kafka from Sébastien Blanc. The Kafka setup in Kubernetes presented in the course looked pretty easy. The Kafka client implementation for Java seemed to be easy as well. Furthermore, I wanted to use Kafka for a long time, so I got the idea to extend my Istio example. Each time a service is called, a message is sent to a topic. The service (implemented in Quarkus), as well as the Kafka cluster should be in an Istio Service Mesh and secured with mTLS. I found descriptions of Joel Takvorian that Kafka works with Istio, so I knew (or at least hoped) that my plan should work. This article will describe the overall architecture of the example and which obstacles I encountered during deployment. ## The demo application and Istio setup The first step was to set up the environment for the demo application and for Istio. I forked the Red Hat Istio tutorial and added a bunch of yaml files that: - deploy Istio and the demo application and the Istio configuration for the demo application. This Istio configuration will make sure that every connection between the application is secured with mTLS by a set of `DestinationRule`s and `VirtualService`s. The configuration locks down undefined service calls by a bunch of `AuthorizationPolicy`s. This should be a more or less simplified setup for a real live application. The yaml-files and scripts for the setup can be found in my fork of the Red Hat Istio tutorial. The details are left out for brevity. ## Setup of the Kafka cluster To set up the Kafka cluster I used the Strimzi project. The setup was pretty straight forward. I configured the listener for the Broker that the Kafka traffic is not encrypted. The next step was to set up the Kafka cluster in a way that the Istio sidecar was injected to the Kafka Broker and Zookeeper. I added a couple of labels to the Kafka Broker, Zookeeper and Entity Operator to be compliant to the Istio notation. I also added the annotation `sidecar.istio.io/inject: "true"` to the Kafka Broker, Zookeeper and Entity Operator. With this annotation, the Istio sidecar container is injected to the pod in OpenShift. From my experience, to inject the Istio sidecar container to the Entity Operator is not causing any problems. The final configuration looks like this: {% highlight yaml hl_lines="27 28 30 31 32 33 34 35 43 44 45 46 47 49 50 51 52 53 54 55 62 63 64 65 67 68 69 70 71 72"%} - apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: demo-app-log-cluster labels: {} namespace: ${DEPLOYMENT_NAMESPACE} spec: kafka: config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 log.message.format.version: '2.5' version: 2.5.0 storage: type: ephemeral replicas: 1 listeners: plain: {} tls: {} authorization: type: simple template: statefulset: metadata: labels: app: kafka-cluster-demo-app-log version: 1.0 pod: metadata: labels: app: kafka-cluster-demo-app-log version: 1.0 annotations: sidecar.istio.io/inject: "true" entityOperator: topicOperator: reconciliationIntervalSeconds: 90 userOperator: reconciliationIntervalSeconds: 120 template: deployment: metadata: labels: app: kafka-entity-operator-demo-app-log version: 1.0 app-type: entity-operator pod: metadata: labels: app: kafka-entity-operator-demo-app-log version: 1.0 app-type: entity-operator annotations: sidecar.istio.io/inject: "true" zookeeper: storage: type: ephemeral replicas: 1 template: statefulset: metadata: labels: app: kafka-zookeeper-demo-app-log version: 1.0 pod: metadata: labels: app: kafka-zookeeper-demo-app-log version: 1.0 annotations: sidecar.istio.io/inject: "true" {% endhighlight %} The highlighted sections are included to deploy Istio-compliant pods and to inject the sidecar container. The service of the Kafka Broker can only be used from within the OpenShift cluster. The configuration is not intended for a production use case, but only for testing purposes. ## Extend the demo application with a Kafka client The next steps were to extend the customer-, preference- and the three recommendation-services that each time they are called, they send out messages to Kafka topics. I added a helper Pojo containing the information about the service calls. These changes to the services are all very similar. Here is the source code of the Pojo helper class: {% highlight java linenos %} @RegisterForReflection public class RestServiceCall { public enum Services { CUSTOMER, PREFERENCE, RECOMMENDATION } private Services origin; private Services destination; private String originVersion; private String comment; public RestServiceCall() { } public RestServiceCall(final Services origin, final Services destination, final String originVersion, final String comment) { this.origin = origin; this.destination = destination; this.originVersion = originVersion; this.comment = comment; } ... getter and setter for the fields ... } {% endhighlight %} As you can see, the helper class contains an origin, a destination, the version of the origin service and a comment. Let us have a look at what has been changed in the customer service, for example: {% highlight java hl_lines="14 15 16 23 44 45 46 47 48 49 50"%} @Path("/") public class CustomerResource { private static final String VERSION="v1"; private static final String RESPONSE_STRING_FORMAT = "customer => %s\n"; private final Logger logger = LoggerFactory.getLogger(getClass()); @Inject @RestClient PreferenceService preferenceService; @Inject @Channel("restServiceCall") Emitter restServiceCall; @GET @Produces(MediaType.TEXT_PLAIN) public Response getCustomer() { try { String response = preferenceService.getPreference().trim(); writeLogMessage(response); return Response.ok(String.format(RESPONSE_STRING_FORMAT, response)).build(); } catch (WebApplicationException ex) { Response response = ex.getResponse(); logger.warn("Non HTTP 20x trying to get the response from preference service: " + response.getStatus()); return Response .status(Response.Status.SERVICE_UNAVAILABLE) .entity(String.format(RESPONSE_STRING_FORMAT, String.format("Error: %d - %s", response.getStatus(), response.readEntity(String.class))) ) .build(); } catch (ProcessingException ex) { logger.warn("Exception trying to get the response from preference service.", ex); return Response .status(Response.Status.SERVICE_UNAVAILABLE) .entity(String.format(RESPONSE_STRING_FORMAT, ex.getCause().getClass().getSimpleName() + ": " + ex.getCause().getMessage())) .build(); } } private void writeLogMessage(String comment) { RestServiceCall serviceCall= new RestServiceCall(Services.CUSTOMER,Services.PREFERENCE,VERSION,comment); String out = JsonbBuilder.create().toJson(serviceCall); logger.info("RestServiceCall"+ out); KafkaRecord msg = KafkaRecord.of(1, out); restServiceCall.send(msg); } } {% endhighlight %} I added a field `restServiceCall` which is linked to a Kafka topic. The next change is the new method `writeLogMessage` which creates a `RestServiceCall` object, serializes it to JSON, creates a Kafka message with the key `1` and the value of the serialized RestServiceCall object. Finally, this message is sent to the Kafka Broker. The configuration for the Kafka integration is done in the `application.properties`. The file looks like this: {% highlight ini hl_lines="3 4 5 6 7"%} com.redhat.developer.demos.customer.rest.PreferenceService/mp-rest/url=http://preference:8080 mp.messaging.outgoing.restServiceCall.connector=smallrye-kafka mp.messaging.outgoing.restServiceCall.topic=demo-application-log-customer mp.messaging.outgoing.restServiceCall.key.serializer=org.apache.kafka.common.serialization.IntegerSerializer mp.messaging.outgoing.restServiceCall.value.serializer=org.apache.kafka.common.serialization.StringSerializer mp.messaging.outgoing.restServiceCall.bootstrap.servers=${KAFKA_BOOTSTRAP_SERVER:demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092} quarkus.log.level=INFO quarkus.log.category."de.consol".level=DEBUG quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t{id}) %s%e%n quarkus.jaeger.service-name=${JAEGER_SERVICE_NAME:customer} quarkus.jaeger.propagation=${JAEGER_PROPAGATION:b3} quarkus.jaeger.sampler-type=const quarkus.jaeger.sampler-param=1 quarkus.jaeger.endpoint=${JAEGER_ENDPOINT:http://jaeger-collector.istio-system.svc:14268/api/traces} {% endhighlight %} I just needed to define and configure the connector `smallrye-kafka`, which is the Kafka framework that I am using. In the next line is the topic to which the message is sent, followed by the used classes for serializing the key and value. Finally, the service or URL of the Kafka bootstrap server, which is the service of my Kafka cluster. The changes in the other services look similar. With these changes I was able to connect my demo application to a Kafka cluster. Both were in the same Istio data plane. So far, so good. Let us add a consumer to the Kafka cluster. ## Add a Kafka consumer to the demo application Just writing a message to a topic is a bit boring, so I decided to create a consumer printing out a small report, showing which services were used in the last couple of minutes. For this, I created a new service. The consumer service contains the class `RestServiceCall` and the actual Kafka consumer. It looks like this: {% highlight java linenos%} @Traced @ApplicationScoped public class ConsumerResource { @ConfigProperty(name = "stream-topics") List topics; @Produces public Topology getCalledServices() { Duration windowSizeMs = Duration.ofMinutes(5); Duration gracePeriodMs = Duration.ofMinutes(1); final StreamsBuilder builder = new StreamsBuilder(); JsonbSerde restServiceCallSerde = new JsonbSerde<>(RestServiceCall.class); builder.stream( topics, Consumed.with(Serdes.Integer(), restServiceCallSerde) ) .groupBy( (key, value) -> value.emptyComment().getKey(), Grouped.with( Serdes.String(), /* key (note: type was modified) */ restServiceCallSerde) /* value */) .windowedBy(TimeWindows.of(windowSizeMs).grace(gracePeriodMs).advanceBy(windowSizeMs)) .reduce((aggValue, newValue) -> aggValue /* adder */) .suppress(Suppressed.untilWindowCloses(BufferConfig.unbounded())) .toStream() .print(Printed.toSysOut()); // .to("demo-application-log-aggregated"); return builder.build(); } } {% endhighlight %} The class is using KafkaStream to process the data. The processing is done in the method `getCalledServices`. In this method the messages for previous defined topics are grouped and aggregated with a duration of 5 minutes based on the new key (origin service + origin version + destination) and printed to system out. When I made some requests the result looked liked this: ``` bash │ 2021-01-13 16:17:44,674 INFO [org.apa.kaf.str.sta.int.RocksDBTimestampedStore] (demo-application-log-consumer-aa8c2dec-5045-464d-a83d-40217651bb2e-StreamThread-1) Opening store KSTREAM-REDUCE-STATE-STORE-0000000002.1610539200000 in r │ │ [KTABLE-TOSTREAM-0000000009]: [CUSTOMER:v1:PREFERENCE@1610472600000/1610472900000], RestServiceCall [origin=CUSTOMER, destination=PREFERENCE, originVersion=v1, comment=] │ │ [KTABLE-TOSTREAM-0000000009]: [PREFERENCE:v1:RECOMMENDATION@1610472600000/1610472900000], RestServiceCall [origin=PREFERENCE, destination=RECOMMENDATION, originVersion=v1, comment=] │ │ [KTABLE-TOSTREAM-0000000009]: [RECOMMENDATION:v1:@1610472600000/1610472900000], RestServiceCall [origin=RECOMMENDATION, destination=null, originVersion=v1, comment=] ``` ## Use mTLS for the complete data plane With these changes the demo application is now using Kafka. The next step was to turn on mTLS for the connection between the demo application (including the new consumer service) and the Kafka cluster. I tried several configurations with `DestinationRule`s and `VirtualService`s which did not work. Enforcing mTLS via `PeerAuthentication` (one for each namespace), however, worked like a charm. The `PeerAuthentication`s look like this for my namespaces/projects: {% highlight yaml linenos%} ... - kind: PeerAuthentication apiVersion: security.istio.io/v1beta1 metadata: name: demo-app namespace: demo-app spec: mtls: mode: STRICT - kind: PeerAuthentication apiVersion: security.istio.io/v1beta1 metadata: name: kafka namespace: kafka spec: mtls: mode: STRICT ... {% endhighlight %} I used `STRICT` mode to prohibit non-encrypted connections between the services and Kafka. As I looked at Kiali, I could see that the two namespaces demo-app and Kafka are marked to be secure. This is indicated by the padlock in the row `Istio Config`. However, the connection (the blue arrows) between the services and the Kafka Broker seemed to be insecure (the padlock is missing). Are these connections secured or not? ![](screenshot_kiali1.jpg) ![](screenshot_kiali2.jpg) My first check was to look at the envoy metrics of the customer (as an example) and see if they contain any indication if the Kafka messages were using mTLS. Here are the results: {% highlight bash hl_lines="10 20"%} ... # TYPE istio_tcp_connections_closed_total counter istio_tcp_connections_closed_total{reporter="source",source_workload="consumer", source_workload_namespace="demo-app",source_principal="spiffe://cluster.local/ns/demo-app/sa/consumer", source_app="consumer",source_version="v1",destination_workload="demo-app-log-cluster-kafka", destination_workload_namespace="kafka",destination_principal="spiffe://cluster.local/ns/kafka/sa/demo-app-log-cluster-kafka", destination_app="kafka-cluster-demo-app-log",destination_version="1", destination_service="demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local", destination_service_name="demo-app-log-cluster-kafka-bootstrap",destination_service_namespace="kafka", request_protocol="tcp",response_flags="-",connection_security_policy="unknown", source_canonical_service="consumer",destination_canonical_service="kafka", source_canonical_revision="v1",destination_canonical_revision="1"} 4 istio_tcp_connections_closed_total{reporter="source",source_workload="consumer", source_workload_namespace="demo-app",source_principal="spiffe://cluster.local/ns/demo-app/sa/consumer", source_app="consumer",source_version="v1",destination_workload="demo-app-log-cluster-kafka", destination_workload_namespace="kafka",destination_principal="spiffe://cluster.local/ns/kafka/sa/demo-app-log-cluster-kafka", destination_app="kafka-cluster-demo-app-log",destination_version="1", destination_service="demo-app-log-cluster-kafka-brokers.kafka.svc.cluster.local", destination_service_name="demo-app-log-cluster-kafka-brokers",destination_service_namespace="kafka", request_protocol="tcp",response_flags="-",connection_security_policy="unknown", source_canonical_service="consumer",destination_canonical_service="kafka", source_canonical_revision="v1",destination_canonical_revision="1"} 0 ... {% endhighlight %} This does not look like the connections with Kafka are using mTLS, otherwise the attribute `connection_security_policy` should have the value `mutual_tls`. Why it is not working would be speculation on my side. It is something that I need to investigate later on. I found another way to verify that the connection is secured or at least provides a strong indication. I opened a shell on one of the envoy proxies and queried the SSL certificate of the Kafka cluster. For the strict mode I got this reply: ``` bash istio-proxy@customer-6794675f5-gkqxm:/$ openssl s_client -showcerts -servername \ -connect demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 > 2)] = 0x16)" tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 17:15:46.667232 IP 172.17.0.24.44050 > 172.17.0.11.9092: Flags [P.], seq 3560243409:3560245459, ack 1817826881, win 502, options [nop,nop,TS val 221777789 ecr 3018265579], length 2050 17:15:46.667727 IP 172.17.0.11.9092 > 172.17.0.24.44050: Flags [P.], seq 1:174, ack 2050, win 501, options [nop,nop,TS val 3018265579 ecr 221777789], length 173 ``` The results show that an SSL handshake happened, when I started sending messages to the Kafka Broker. The best way, of course, would be to store a `tcpdump` in the envoy proxy and analyse it with Wireshark, for example. IMHO, the recording of the SSL handshake was proof enough that the connection is secure. --- *Remark:* In order to execute `tcpdump`, you need to allow `allowPrivilegeEscalation` in the configmap `istio-sidecar-injector`. --- ## Add Kafka's RBAC to the application From what I know, the Istio sidecar proxy is not forwarding the client certificate to Kafka, so I cannot use this option to combine Istio's `AuthorizationPolicy` with Kafka's RBAC. Banzai Cloud's Supertubes provide a solution, but I wanted to stick to plain Kafka and Istio. I used the `plain` RBAC without transport encryption. The transport encryption is handled by Istio via mTLS. So, I settled with using `scram-sha-512` for authentication and added KafkaUser-resources to handle the authorization. Please note, that this is the definition of Strimzi. In other Kafka distributions the definition for this might look different. The definition of the Kafka cluster including authentication looks like this: {% highlight yaml hl_lines="9 10"%} kind: Kafka metadata: name: demo-app-log-cluster spec: ... listeners: plain: authentication: type: scram-sha-512 authorization: type: simple template: ... {% endhighlight %} The plain authentication is now using `scram-sha-512`. I did not need to specify an external ingress since the messages are generated from the same cluster. The authorization is `simple` with using ACLs in KafkaUsers. This is fine for my use case, because I do not need authorization via KeyCloak or Open Policy Agent. I created one KafkaUser for each topic and a separate KafkaUser for the consumer. I only show one KafkaUser because the definition is quite long and similar for the other users. ``` yaml - apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaUser metadata: name: customer-writer namespace: ${DEPLOYMENT_NAMESPACE} labels: strimzi.io/cluster: demo-app-log-cluster spec: authentication: type: scram-sha-512 authorization: type: simple acls: # Example Producer Acls for topic demo-application-log-customer - resource: type: topic name: demo-application-log-customer patternType: literal operation: Write host: "*" - resource: type: topic name: demo-application-log-customer patternType: literal operation: Create host: "*" - resource: type: topic name: demo-application-log-customer patternType: literal operation: Describe host: "*" ``` This definition contains the authentication and authorization for the user, as well as the definition which operations it can perform on which topic. ## Extend demo application for Kafka authentication and authorization So, I have set up the Kafka RBAC on my Kafka cluster. The next step is to configure the services of the demo application so that it knows which KafkaUser to use and where to get the credentials. When creating a KafkaUser like above, Strimzi will create Kubernetes secrets with the credentials for the KafkaUser in the Kafka namespace. The first step is to copy the KafkaUser secrets in our demo-app namespace with this command: ``` bash kubectl get secret --namespace kafka -l \ strimzi.io/cluster=demo-app-log-cluster,strimzi.io/kind=KafkaUser \ -o json | sed 's/"namespace"\:\s*"kafka"/"namespace": "demo-app"/g' | \ jq 'del(.items[].metadata.ownerReferences)'| \ kubectl apply -f - -n demo-app ``` --- Side note: If you have an easier command, please let me know. --- The next step is to extend our Kafka configuration in the demo app to use the correct authentication and authorization with these properties. The result looks like this: {% highlight ini hl_lines="6 7 8 9 10"%} mp.messaging.outgoing.restServiceCall.connector=smallrye-kafka mp.messaging.outgoing.restServiceCall.topic=demo-application-log-customer mp.messaging.outgoing.restServiceCall.key.serializer=org.apache.kafka.common.serialization.IntegerSerializer mp.messaging.outgoing.restServiceCall.value.serializer=org.apache.kafka.common.serialization.StringSerializer mp.messaging.outgoing.restServiceCall.bootstrap.servers=${KAFKA_BOOTSTRAP_SERVER:demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092} mp.messaging.outgoing.restServiceCall.security.protocol=SASL_PLAINTEXT mp.messaging.outgoing.restServiceCall.sasl.mechanism=SCRAM-SHA-512 mp.messaging.outgoing.restServiceCall.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ username="${KAFKA_USER_NAME:customer-writer}" \ password="${KAFKA_PASSWORD}"; {% endhighlight %} By this, I defined that I want to use `SASL_PLAINTEXT` as the protocol, and the mechanism to be `SCRAM-SHA-512`. Last but not least, which class should be used and what the user and password is. I am not a big fan of hard coding username and password in an application, so they are read from the environment variables of the container. For the last step, I needed to modify the deployment to use the copied secret in an environment variable. With the necessary changes, the deployment looks like this: {% highlight yaml hl_lines="10 11 12 13 14"%} ... containers: - env: - name: JAVA_OPTIONS value: -Xms15m -Xmx15m -Xmn15m - name: KAFKA_BOOTSTRAP_SERVER value: demo-app-log-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 - name: JAEGER_ENDPOINT value: http://jaeger-collector.istio-system.svc:14268/api/traces - name: KAFKA_PASSWORD valueFrom: secretKeyRef: name: customer-writer key: password image: quay.io/omeyer/istio-tutorial-customer:v1.0-kafka ... {% endhighlight %} This was the final setting. Now the demo application is using Kafka with RBAC. Both are in one data plane and are using mTLS for the communication. ## Pitfalls on the way I first started the example on Red Hat OpenShift Service Mesh v1.12. This is based on Istio 1.4.6 and Kiali 1.17. The Istio version did not include a Kafka filter. The result was that the basic integration between Istio and Kafka with mTLS was not working. I found examples to use Kafka's mTLS instead of Istio's mTLS, by excluding Kafka traffic from Istio. I did not want to do this. I switched from OpenShift to minikube 1.19, deployed Istio 1.7.4 and the latest Kiali. I translated the OpenShift templates to plain yaml files. I did not encounter any problem with the basic setup in Kubernetes. I needed to adjust the configuration for the Istio sidecar since Kubernetes supports auto injection for all pods in a namespace, whereas in OpenShift we need to add an annotation for all pods where a sidecar should be injected. For some reason, I was not able to enforce mTLS with `DestinationRule`s between the namespaces of the demo-application and Kafka. I assume, that this has to do with the fact that the Kafka client is calling the pod directly via the `StateFullSet`'s `PodDNS`. When I enforced mTLS via `PeerAuthentication` between the projects, it worked as expected. --- Update: Red Hat OpenShift Service Mesh v2.0 is using a newer version of Istio (1.6), but I still cannot enable mTLS for Kafka provided by Istio. --- ## Summary I learned a lot about Kafka and Istio as I extended the demo application. I used only a small subset of Kafka's features, for example I did not tweak the sending and receiving of messages in the Kafka client. The integration between Kafka and Istio is possible, but it is not as easy as with applications that use HTTP or HTTPS. When using TCP based protocols, make sure that Istio is supporting them, otherwise the integration becomes challenging. You can find the changes in my fork of the Red Hat tutorial. I created a document how to set up the environment for OpenShift and Kubernetes. The fork also contains a `Makefile` to build the applications and branches for each version of the recommendation service. Following is a list of tutorials I used or that I found interesting: - [Red Hat Istio tutorial](https://redhat-scholars.github.io/istio-tutorial/) - [Link to my fork of the Red Hat Istio tutorial](https://github.com/olaf-meyer/istio-tutorial/tree/olaf-meyer-kafka-recommendation-v1) - [Banzai SuperTubes](https://banzaicloud.com/products/supertubes/) - [Capture SSL handshake](https://stackoverflow.com/questions/39624745/capture-only-ssl-handshake-with-tcpdump) [Read more](https://omd.consol.de/blog/2021/02/02/how-to-integrate-kafka-with-istio-on-openshift/) ----------------------------------------------------------------------------------- title: "Integration testing with Testcontainers" date: "2020-10-27" --------------------------------------------------------------------------------
Automatic integration tests as part of the development life cycle can save a lot of time and money. Not only when dealing with other service APIs or offering some, also if the application uses a database or other infrastructure services. We at Consol made a lot of good experience to develop the integration tests as part of the life cycle from the beginning of a project. Therefor the [Citrus framework](https://citrusframework.org) is often a good choice to do it automated. But there are other frameworks and libraries which can be useful. In this article, we'll have a look at [Testcontainers](https://testcontainers.org). By using a sample microservice, we will show how Testcontainers can be used and what chances it provides. ## About Citrus The Citrus framework was started to develop in the year 2006 and it still meets the main purpose to do automated integration testing. It offers a lot of possibilities to connect with third party systems, send or receive messages and act as a server or client. Like in a sequence diagram it allows to define and test the complete flow of a message through the systems. Additionally it can generate random data and do the validation of received messages as fine granular as desired. If you would like to get some detailed information about what exactly Citrus is and what can be tested take a look at the [documentation](http://citrusframework.org/citrus/reference/html/index.html) or directly try out some [samples](http://citrusframework.org/samples/). ## About Testcontainers Testcontainers is a Java library that: > supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container. The main purpose of it is to set up the required infrastructure (services) for unit tests. But it also supports to run frontend tests in containerized web browsers. With the approach of a `GenericContainer` it allows the usage of every available docker image. For further information you can visit: [https://www.testcontainers.org/](https://www.testcontainers.org/) ## The sample application In this article we will use a Java 11 Maven project with Spring Boot and Apache Camel. The application subscribes to a queue and stores each message to a database while it also publishes it to a topic. As a special challenge we will add a transactional behaviour: if something went wrong after storing the message to the database, the database entry should be reverted. This means we expect to have no entry in the database and no message on the topic if an error occurs. This behavior should be covered with tests. ![sequence_use_case.png](sequence_use_case.png) As infrastructure services we will use Active MQ Artemis and PostgreSQL. ## Testing it with Testcontainers Basically testcontainers definitions are unit tests. This means that the required dependencies should be only in `test` while the test classes are placed in `src/test/java`. For our microservice we only need the following dependencies: {% highlight xml %} org.testcontainers testcontainers ${testcontainers-version} test org.testcontainers junit-jupiter ${testcontainers-version} test org.testcontainers postgresql ${testcontainers-version} test {% endhighlight %} The next step is the definition of the test class, which needs a `@Testcontainers` annotation additionally to the Spring ones, to activate the active profile etc. For our scenario we need now to define the containers for the database as well as for the message broker. There is a special `PostgreSQLContainer` offered by Testcontainers which ships with some useful functionality. The message broker will be a `GenericContainer` which is also enough for us. As I personally prefer small and clean setups I decided to use the [Alpine Linux](https://alpinelinux.org) images of PostgreSQL and Active MQ Artemis. In order to prevent from race conditions the containers should wait until the necessary service has been started. {% highlight java hl_lines="3 9" %} @Container static PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer<>("postgres:alpine") .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 1)); @Container static GenericContainer activeMQContainer = new GenericContainer<>("vromero/activemq-artemis:latest-alpine") .withExposedPorts(61616) .withEnv("DISABLE_SECURITY", "true") .waitingFor(Wait.forLogMessage(".*AMQ221007: Server is now live.*\n", 1));; {% endhighlight %} At this point both containers will boot up. As soon as they are ready, the spring application will be started as well. It is important to configure the Spring application to use random allocated ports of the test containers: {% highlight java %} @DynamicPropertySource static void registerDynamicProperties(DynamicPropertyRegistry registry) { registry.add("spring.datasource.url", postgreSQLContainer::getJdbcUrl); registry.add("spring.datasource.username", postgreSQLContainer::getUsername); registry.add("spring.datasource.password", postgreSQLContainer::getPassword); registry.add("spring.artemis.port", activeMQContainer::getFirstMappedPort); } {% endhighlight %} We are now ready to test the setup and ensure that the containers started correctly as well as our system under test. This can be easily seen in the log output: {% highlight text %} [...] org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock) org.testcontainers.DockerClientFactory - Docker host IP address is localhost org.testcontainers.DockerClientFactory - Connected to docker: Server Version: 19.03.13 API Version: 1.40 Operating System: Docker Desktop Total Memory: 3940 MB org.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit org.testcontainers.DockerClientFactory - Checking the system... org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0 org.testcontainers.DockerClientFactory - ✔︎ Docker environment should have more than 2GB free disk space 🐳 [vromero/activemq-artemis:latest-alpine] - Creating container for image: vromero/activemq-artemis:latest-alpine 🐳 [vromero/activemq-artemis:latest-alpine] - Starting container with ID: d454b28985ee5654287a72ac1700d53c3fd15b331b704132ad822037da7770b7 🐳 [vromero/activemq-artemis:latest-alpine] - Container vromero/activemq-artemis:latest-alpine is starting: d454b28985ee5654287a72ac1700d53c3fd15b331b704132ad822037da7770b7 🐳 [vromero/activemq-artemis:latest-alpine] - Container vromero/activemq-artemis:latest-alpine started in PT14.199709S 🐳 [postgres:alpine] - Creating container for image: postgres:alpine 🐳 [postgres:alpine] - Starting container with ID: 8ca61a20b7fc2e089a838e4e673d7fe10cdbc6c27ee2cadd49855823b8821c23 🐳 [postgres:alpine] - Container postgres:alpine is starting: 8ca61a20b7fc2e089a838e4e673d7fe10cdbc6c27ee2cadd49855823b8821c23 🐳 [postgres:alpine] - Container postgres:alpine started in PT3.230064S . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.2.RELEASE) com.consol.testcontainers.transaction.ApplicationTest - Starting ApplicationTest on marts.local with PID 35942 (started by martin) com.consol.testcontainers.transaction.ApplicationTest - Running with Spring Boot v2.3.2.RELEASE, Spring v5.2.8.RELEASE com.consol.testcontainers.transaction.ApplicationTest - The following profiles are active: test org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.apache.camel.spring.boot.CamelAutoConfiguration' of type [org.apache.camel.spring.boot.CamelAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) org.apache.camel.support.LRUCacheFactory - Detected and using LRUCacheFactory: camel-caffeine-lrucache org.apache.camel.impl.engine.DefaultCamelBeanPostProcessor - No CamelContext defined yet so cannot inject into bean: org.apache.camel.impl.health.DefaultHealthCheckRegistry org.apache.camel.impl.engine.BaseExecutorServiceManager - Using custom DefaultThreadPoolProfile: ThreadPoolProfile[default (true) size:20-20, keepAlive:60 SECONDS, maxQueue:1000, allowCoreThreadTimeOut:true, rejectedPolicy:CallerRuns] org.apache.camel.spring.boot.SpringBootRoutesCollector - Loading additional Camel XML routes from: classpath:camel/*.xml org.apache.camel.spring.boot.SpringBootRoutesCollector - Loading additional Camel XML rests from: classpath:camel-rest/*.xml org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) is starting org.apache.camel.impl.engine.AbstractCamelContext - MDC logging is enabled on CamelContext: camel-1 org.apache.camel.impl.engine.AbstractCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html org.apache.camel.impl.engine.AbstractCamelContext - Using HealthCheck: camel-spring-boot org.apache.camel.spring.boot.CamelSpringBootApplicationListener - Starting CamelMainRunController to ensure the main thread keeps running org.apache.camel.impl.engine.InternalRouteStartupManager - Route: UC started and consuming from: jms://queue:sourceQueue org.apache.camel.impl.engine.AbstractCamelContext - Total 1 routes, of which 1 are started org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) started in 6.053 seconds com.consol.testcontainers.transaction.ApplicationTest - Started ApplicationTest in 16.737 seconds (JVM running for 38.764) {% endhighlight %} Now let's write some useful tests :-) First of all we need to initialize the database. Therefore, we simply add a setup script to the resources and configure it in the container definition `.withInitScript("db_init.sql")`. As we need to interact with the database as well as with the message broker, we will use JdbcTemplate and JmsTemplate which can easily be added with `@Autowired`. The test itself must take care of a clean environment on every run. Testing a good case scenario could look like this: {% highlight java %} // clear database jdbcTemplate.execute("TRUNCATE t_testcontainers"); // the send and expected message String message = "{\"id\": 42, \"name\": \"Just a simple test\"}"; // subscribe to topic ActiveMQTopic topic = new ActiveMQTopic("destinationTopic?consumer.retroactive=true"); // trigger the use case jmsTemplate.convertAndSend("sourceQueue", message); // get the message from the database and topic jmsTemplate.setReceiveTimeout(5000); String receivedJmsMessage = (String)jmsTemplate.receiveAndConvert(topic); String storedDbEntry = jdbcTemplate.queryForObject("SELECT * FROM t_testcontainers", String.class); //check it assertEquals(message, receivedJmsMessage); assertEquals(message, storedDbEntry); {% endhighlight %} It is even more important to test the bad case scenarios. I have added a processor on the route of the service to throw a RuntimeException on a special message. The test should then check that the database is empty. The complete test class (without comments) is now: {% highlight java %} package com.consol.testcontainers.transaction; import org.apache.activemq.artemis.jms.client.ActiveMQTopic; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.jms.core.JmsTemplate; import org.springframework.test.context.ActiveProfiles; import org.springframework.test.context.DynamicPropertyRegistry; import org.springframework.test.context.DynamicPropertySource; import org.testcontainers.containers.GenericContainer; import org.testcontainers.containers.PostgreSQLContainer; import org.testcontainers.containers.wait.strategy.Wait; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNull; @ActiveProfiles("test") @SpringBootTest @Testcontainers public class UC_Test { @Container static PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer<>("postgres:alpine") .withInitScript("db_init.sql") .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 1)); @Container static GenericContainer activeMQContainer = new GenericContainer<>("vromero/activemq-artemis:latest-alpine") .withExposedPorts(61616) .withEnv("DISABLE_SECURITY", "true") .waitingFor(Wait.forLogMessage(".*AMQ221007: Server is now live.*\n", 1));; @Autowired JmsTemplate jmsTemplate; @Autowired private JdbcTemplate jdbcTemplate; @DynamicPropertySource static void registerDynamicProperties(DynamicPropertyRegistry registry) { registry.add("spring.datasource.url", postgreSQLContainer::getJdbcUrl); registry.add("spring.datasource.username", postgreSQLContainer::getUsername); registry.add("spring.datasource.password", postgreSQLContainer::getPassword); registry.add("spring.artemis.port", activeMQContainer::getFirstMappedPort); } @Test void testOk() { jdbcTemplate.execute("TRUNCATE t_testcontainers"); String message = "{\"id\": 42, \"name\": \"Just a simple test\"}"; ActiveMQTopic topic = new ActiveMQTopic("destinationTopic?consumer.retroactive=true"); jmsTemplate.convertAndSend("sourceQueue", message); jmsTemplate.setReceiveTimeout(5000); String receivedJmsMessage = (String)jmsTemplate.receiveAndConvert(topic); String storedDbEntry = jdbcTemplate.queryForObject("SELECT * FROM t_testcontainers", String.class); assertEquals(message, receivedJmsMessage); assertEquals(message, storedDbEntry); } @Test void testRollback() { jdbcTemplate.execute("TRUNCATE t_testcontainers"); String message = "{\"id\": 42, \"name\": \"I will throw an Exception\"}"; ActiveMQTopic topic = new ActiveMQTopic("destinationTopic?consumer.retroactive=true"); jmsTemplate.convertAndSend("sourceQueue", message); jmsTemplate.setReceiveTimeout(5000); String receivedJmsMessage = (String)jmsTemplate.receiveAndConvert(topic); int storedDbEntries = jdbcTemplate.queryForObject("SELECT COUNT(*) FROM t_testcontainers", Integer.class); assertNull(receivedJmsMessage); assertEquals(0, storedDbEntries); } } {% endhighlight %} ## Testing it with Citrus (just a summary) The same tests implemented with Citrus could look like the following. It is split into a configuration and a test class: {% highlight java %} package com.consol.testcontainers.transaction.config; import com.consol.citrus.container.SequenceBeforeTest; import com.consol.citrus.dsl.endpoint.CitrusEndpoints; import com.consol.citrus.dsl.runner.TestRunner; import com.consol.citrus.dsl.runner.TestRunnerBeforeTestSupport; import com.consol.citrus.jms.endpoint.JmsEndpoint; import org.apache.activemq.ActiveMQConnectionFactory; import org.apache.commons.dbcp.BasicDataSource; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import javax.jms.ConnectionFactory; @Configuration public class CitrusConfiguration { @Value("${jms.broker.url}") private String jmsBrokerUrl; @Value("${jms.endpoint.out.uc}") private String jmsEndpointOutUc; @Value("${jms.endpoint.in}") private String jmsEndpointIn; @Value("${jms.receive.timeout}") private long jmsReceiveTimeout; @Value("${db.url}") private String dbUrl; @Value("${db.driver}") private String dbDriver; @Value("${db.user}") private String dbUser; @Value("${db.password}") private String dbPassword; @Bean public JmsEndpoint jmsEndpointOutUc() { return CitrusEndpoints.jms() .asynchronous() .connectionFactory(connectionFactory()) .destination(jmsEndpointOutUc) .build(); } @Bean public JmsEndpoint jmsEndpointIn() { return CitrusEndpoints.jms() .asynchronous() .connectionFactory(connectionFactory()) .destination(jmsEndpointIn) .pubSubDomain(true) .autoStart(true) .timeout(jmsReceiveTimeout) .build(); } @Bean public ConnectionFactory connectionFactory() { return new ActiveMQConnectionFactory(jmsBrokerUrl); } @Bean(destroyMethod = "close") public BasicDataSource datasource() { final BasicDataSource dataSource = new BasicDataSource(); dataSource.setDriverClassName(dbDriver); dataSource.setUrl(dbUrl); dataSource.setUsername(dbUser); dataSource.setPassword(dbPassword); return dataSource; } @Bean public SequenceBeforeTest beforeTest() { return new TestRunnerBeforeTestSupport() { @Override public void beforeTest(TestRunner runner) { runner.purgeQueues(purgeJmsQueueBuilder -> purgeJmsQueueBuilder .connectionFactory(connectionFactory()) .queueNames(jmsEndpointOutUc1)); runner.purgeQueues(purgeJmsQueueBuilder -> purgeJmsQueueBuilder .connectionFactory(connectionFactory()) .queue(jmsEndpointIn)); } }; } } {% endhighlight %} {% highlight java %} package com.consol.testcontainers.transaction.test; import com.consol.citrus.annotations.CitrusTest; import com.consol.citrus.dsl.testng.TestNGCitrusTestRunner; import com.consol.citrus.jms.endpoint.JmsEndpoint; import org.springframework.beans.factory.annotation.Autowired; import org.testng.annotations.Test; import javax.sql.DataSource; @Test public class UC_Test extends TestNGCitrusTestRunner { @Autowired JmsEndpoint jmsEndpointOutUc; @Autowired JmsEndpoint jmsEndpointIn; @Autowired DataSource dataSource; @Test @CitrusTest public void ok() { createVariable("message", "{\"id\": {citrus:randomNumber(10)}, \"name\": \"Just a simple test\"}"); send(action -> action.endpoint(jmsEndpointOutUc).payload("${message}")); repeatOnError().until("i = 5").index("i").autoSleep(100).actions( echo("check database: ${i} from max 5 tries"), query(action -> action.dataSource(dataSource) .statement("select count(*) as entry_found from t_testcontainers where message='${message}';") .validate("entry_found", "1")) ); receive(action -> action.endpoint(jmsEndpointIn).payload("${message}")); } @Test @CitrusTest public void rollback() { createVariable("message", "{\"id\": {citrus:randomNumber(10)}, \"name\": \"I will throw an Exception\"}"); send(action -> action.endpoint(jmsEndpointOutUc).payload("${message}")); receiveTimeout(action -> action.endpoint(jmsEndpointIn)); query(action -> action.dataSource(dataSource) .statement("select count(*) as entry_found from t_testcontainers where message='${message}';") .validate("entry_found", "0")); } } {% endhighlight %} ## Citrus and Testcontainers compared As described in the beginning, we usually use the [Citrus framework](https://citrusframework.org) for integration testing. Therefore, I would now like to discuss the differences to Testcontainers I noticed. First of all the concept of Citrus with the **test sources** as part of an independently project or module offers opportunities like a normal service deployment (on another machine). The service and the tests can therefore be individually delivered and started. To **write tests** Citrus offers a huge support for different use cases (see [documentation](https://citrusframework.org/citrus/reference/html/index.html)) and Testcontainer allows to use nearly every docker image available. For some special images there are also own container classes to use like in our example the database. A big advantage of Citrus is the **validation** (of every detail). In contrast to this has Testcontainers at the moment no special support for that and you have to write this parts for your own or even use other libraries for that. The **delivery** can be done with Citrus separately which allows a **execution** ot the tests against a normal deployed application even on another machine. On the other side Testcontainers can run nearly everywhere (a docker installation is required) since the needed services are set up as part of the tests. ## Conclusion Testcontainers is a good choice for a fast provisioning of the infrastructure. But it need support by other frameworks when it has to use the provided services. The advantages of Citrus are more likely the other wise with the validation and usage of third party systems. A combination of Testcontainers with other testing tools (for example Citrus, Wiremock, Rest-Assured, ...) to control specific protocols or allowing a validation of data structures would be a technology stack which raises integration testing to the next level. [Read more](https://omd.consol.de/blog/2020/10/27/integration-testing-with-testcontainers/) ----------------------------------------------------------------------------------- title: "Integration testing with Testcontainers" date: "2020-10-27" --------------------------------------------------------------------------------
Automatic integration tests as part of the development life cycle can save a lot of time and money. Not only when dealing with other service APIs or offering some, also if the application uses a database or other infrastructure services. We at Consol made a lot of good experience to develop the integration tests as part of the life cycle from the beginning of a project. Therefor the [Citrus framework](https://citrusframework.org) is often a good choice to do it automated. But there are other frameworks and libraries which can be useful. In this article, we'll have a look at [Testcontainers](https://testcontainers.org). By using a sample microservice, we will show how Testcontainers can be used and what chances it provides. ## About Citrus The Citrus framework was started to develop in the year 2006 and it still meets the main purpose to do automated integration testing. It offers a lot of possibilities to connect with third party systems, send or receive messages and act as a server or client. Like in a sequence diagram it allows to define and test the complete flow of a message through the systems. Additionally it can generate random data and do the validation of received messages as fine granular as desired. If you would like to get some detailed information about what exactly Citrus is and what can be tested take a look at the [documentation](http://citrusframework.org/citrus/reference/html/index.html) or directly try out some [samples](http://citrusframework.org/samples/). ## About Testcontainers Testcontainers is a Java library that: > supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container. The main purpose of it is to set up the required infrastructure (services) for unit tests. But it also supports to run frontend tests in containerized web browsers. With the approach of a `GenericContainer` it allows the usage of every available docker image. For further information you can visit: [https://www.testcontainers.org/](https://www.testcontainers.org/) ## The sample application In this article we will use a Java 11 Maven project with Spring Boot and Apache Camel. The application subscribes to a queue and stores each message to a database while it also publishes it to a topic. As a special challenge we will add a transactional behaviour: if something went wrong after storing the message to the database, the database entry should be reverted. This means we expect to have no entry in the database and no message on the topic if an error occurs. This behavior should be covered with tests. ![](sequence_use_case.png) As infrastructure services we will use Active MQ Artemis and PostgreSQL. ## Testing it with Testcontainers Basically testcontainers definitions are unit tests. This means that the required dependencies should be only in `test` while the test classes are placed in `src/test/java`. For our microservice we only need the following dependencies: ```xml org.testcontainers testcontainers ${testcontainers-version} test org.testcontainers junit-jupiter ${testcontainers-version} test org.testcontainers postgresql ${testcontainers-version} test ``` The next step is the definition of the test class, which needs a `@Testcontainers` annotation additionally to the Spring ones, to activate the active profile etc. For our scenario we need now to define the containers for the database as well as for the message broker. There is a special `PostgreSQLContainer` offered by Testcontainers which ships with some useful functionality. The message broker will be a `GenericContainer` which is also enough for us. As I personally prefer small and clean setups I decided to use the [Alpine Linux](https://alpinelinux.org) images of PostgreSQL and Active MQ Artemis. In order to prevent from race conditions the containers should wait until the necessary service has been started. {% highlight java hl_lines="3 9" %} @Container static PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer<>("postgres:alpine") .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 1)); @Container static GenericContainer activeMQContainer = new GenericContainer<>("vromero/activemq-artemis:latest-alpine") .withExposedPorts(61616) .withEnv("DISABLE_SECURITY", "true") .waitingFor(Wait.forLogMessage(".*AMQ221007: Server is now live.*\n", 1));; {% endhighlight %} At this point both containers will boot up. As soon as they are ready, the spring application will be started as well. It is important to configure the Spring application to use random allocated ports of the test containers: ```java @DynamicPropertySource static void registerDynamicProperties(DynamicPropertyRegistry registry) { registry.add("spring.datasource.url", postgreSQLContainer::getJdbcUrl); registry.add("spring.datasource.username", postgreSQLContainer::getUsername); registry.add("spring.datasource.password", postgreSQLContainer::getPassword); registry.add("spring.artemis.port", activeMQContainer::getFirstMappedPort); } ``` We are now ready to test the setup and ensure that the containers started correctly as well as our system under test. This can be easily seen in the log output: ```text [...] org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock) org.testcontainers.DockerClientFactory - Docker host IP address is localhost org.testcontainers.DockerClientFactory - Connected to docker: Server Version: 19.03.13 API Version: 1.40 Operating System: Docker Desktop Total Memory: 3940 MB org.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit org.testcontainers.DockerClientFactory - Checking the system... org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0 org.testcontainers.DockerClientFactory - ✔︎ Docker environment should have more than 2GB free disk space 🐳 [vromero/activemq-artemis:latest-alpine] - Creating container for image: vromero/activemq-artemis:latest-alpine 🐳 [vromero/activemq-artemis:latest-alpine] - Starting container with ID: d454b28985ee5654287a72ac1700d53c3fd15b331b704132ad822037da7770b7 🐳 [vromero/activemq-artemis:latest-alpine] - Container vromero/activemq-artemis:latest-alpine is starting: d454b28985ee5654287a72ac1700d53c3fd15b331b704132ad822037da7770b7 🐳 [vromero/activemq-artemis:latest-alpine] - Container vromero/activemq-artemis:latest-alpine started in PT14.199709S 🐳 [postgres:alpine] - Creating container for image: postgres:alpine 🐳 [postgres:alpine] - Starting container with ID: 8ca61a20b7fc2e089a838e4e673d7fe10cdbc6c27ee2cadd49855823b8821c23 🐳 [postgres:alpine] - Container postgres:alpine is starting: 8ca61a20b7fc2e089a838e4e673d7fe10cdbc6c27ee2cadd49855823b8821c23 🐳 [postgres:alpine] - Container postgres:alpine started in PT3.230064S . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.2.RELEASE) com.consol.testcontainers.transaction.ApplicationTest - Starting ApplicationTest on marts.local with PID 35942 (started by martin) com.consol.testcontainers.transaction.ApplicationTest - Running with Spring Boot v2.3.2.RELEASE, Spring v5.2.8.RELEASE com.consol.testcontainers.transaction.ApplicationTest - The following profiles are active: test org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.apache.camel.spring.boot.CamelAutoConfiguration' of type [org.apache.camel.spring.boot.CamelAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) org.apache.camel.support.LRUCacheFactory - Detected and using LRUCacheFactory: camel-caffeine-lrucache org.apache.camel.impl.engine.DefaultCamelBeanPostProcessor - No CamelContext defined yet so cannot inject into bean: org.apache.camel.impl.health.DefaultHealthCheckRegistry org.apache.camel.impl.engine.BaseExecutorServiceManager - Using custom DefaultThreadPoolProfile: ThreadPoolProfile[default (true) size:20-20, keepAlive:60 SECONDS, maxQueue:1000, allowCoreThreadTimeOut:true, rejectedPolicy:CallerRuns] org.apache.camel.spring.boot.SpringBootRoutesCollector - Loading additional Camel XML routes from: classpath:camel/*.xml org.apache.camel.spring.boot.SpringBootRoutesCollector - Loading additional Camel XML rests from: classpath:camel-rest/*.xml org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) is starting org.apache.camel.impl.engine.AbstractCamelContext - MDC logging is enabled on CamelContext: camel-1 org.apache.camel.impl.engine.AbstractCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html org.apache.camel.impl.engine.AbstractCamelContext - Using HealthCheck: camel-spring-boot org.apache.camel.spring.boot.CamelSpringBootApplicationListener - Starting CamelMainRunController to ensure the main thread keeps running org.apache.camel.impl.engine.InternalRouteStartupManager - Route: UC started and consuming from: jms://queue:sourceQueue org.apache.camel.impl.engine.AbstractCamelContext - Total 1 routes, of which 1 are started org.apache.camel.impl.engine.AbstractCamelContext - Apache Camel 3.4.2 (camel-1) started in 6.053 seconds com.consol.testcontainers.transaction.ApplicationTest - Started ApplicationTest in 16.737 seconds (JVM running for 38.764) ``` Now let's write some useful tests :-) First of all we need to initialize the database. Therefore, we simply add a setup script to the resources and configure it in the container definition `.withInitScript("db_init.sql")`. As we need to interact with the database as well as with the message broker, we will use JdbcTemplate and JmsTemplate which can easily be added with `@Autowired`. The test itself must take care of a clean environment on every run. Testing a good case scenario could look like this: ```java // clear database jdbcTemplate.execute("TRUNCATE t_testcontainers"); // the send and expected message String message = "{\"id\": 42, \"name\": \"Just a simple test\"}"; // subscribe to topic ActiveMQTopic topic = new ActiveMQTopic("destinationTopic?consumer.retroactive=true"); // trigger the use case jmsTemplate.convertAndSend("sourceQueue", message); // get the message from the database and topic jmsTemplate.setReceiveTimeout(5000); String receivedJmsMessage = (String)jmsTemplate.receiveAndConvert(topic); String storedDbEntry = jdbcTemplate.queryForObject("SELECT * FROM t_testcontainers", String.class); //check it assertEquals(message, receivedJmsMessage); assertEquals(message, storedDbEntry); ``` It is even more important to test the bad case scenarios. I have added a processor on the route of the service to throw a RuntimeException on a special message. The test should then check that the database is empty. The complete test class (without comments) is now: ```java package com.consol.testcontainers.transaction; import org.apache.activemq.artemis.jms.client.ActiveMQTopic; import org.junit.jupiter.api.Test; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.jms.core.JmsTemplate; import org.springframework.test.context.ActiveProfiles; import org.springframework.test.context.DynamicPropertyRegistry; import org.springframework.test.context.DynamicPropertySource; import org.testcontainers.containers.GenericContainer; import org.testcontainers.containers.PostgreSQLContainer; import org.testcontainers.containers.wait.strategy.Wait; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNull; @ActiveProfiles("test") @SpringBootTest @Testcontainers public class UC_Test { @Container static PostgreSQLContainer postgreSQLContainer = new PostgreSQLContainer<>("postgres:alpine") .withInitScript("db_init.sql") .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 1)); @Container static GenericContainer activeMQContainer = new GenericContainer<>("vromero/activemq-artemis:latest-alpine") .withExposedPorts(61616) .withEnv("DISABLE_SECURITY", "true") .waitingFor(Wait.forLogMessage(".*AMQ221007: Server is now live.*\n", 1));; @Autowired JmsTemplate jmsTemplate; @Autowired private JdbcTemplate jdbcTemplate; @DynamicPropertySource static void registerDynamicProperties(DynamicPropertyRegistry registry) { registry.add("spring.datasource.url", postgreSQLContainer::getJdbcUrl); registry.add("spring.datasource.username", postgreSQLContainer::getUsername); registry.add("spring.datasource.password", postgreSQLContainer::getPassword); registry.add("spring.artemis.port", activeMQContainer::getFirstMappedPort); } @Test void testOk() { jdbcTemplate.execute("TRUNCATE t_testcontainers"); String message = "{\"id\": 42, \"name\": \"Just a simple test\"}"; ActiveMQTopic topic = new ActiveMQTopic("destinationTopic?consumer.retroactive=true"); jmsTemplate.convertAndSend("sourceQueue", message); jmsTemplate.setReceiveTimeout(5000); String receivedJmsMessage = (String)jmsTemplate.receiveAndConvert(topic); String storedDbEntry = jdbcTemplate.queryForObject("SELECT * FROM t_testcontainers", String.class); assertEquals(message, receivedJmsMessage); assertEquals(message, storedDbEntry); } @Test void testRollback() { jdbcTemplate.execute("TRUNCATE t_testcontainers"); String message = "{\"id\": 42, \"name\": \"I will throw an Exception\"}"; ActiveMQTopic topic = new ActiveMQTopic("destinationTopic?consumer.retroactive=true"); jmsTemplate.convertAndSend("sourceQueue", message); jmsTemplate.setReceiveTimeout(5000); String receivedJmsMessage = (String)jmsTemplate.receiveAndConvert(topic); int storedDbEntries = jdbcTemplate.queryForObject("SELECT COUNT(*) FROM t_testcontainers", Integer.class); assertNull(receivedJmsMessage); assertEquals(0, storedDbEntries); } } ``` ## Testing it with Citrus (just a summary) The same tests implemented with Citrus could look like the following. It is split into a configuration and a test class: ```java package com.consol.testcontainers.transaction.config; import com.consol.citrus.container.SequenceBeforeTest; import com.consol.citrus.dsl.endpoint.CitrusEndpoints; import com.consol.citrus.dsl.runner.TestRunner; import com.consol.citrus.dsl.runner.TestRunnerBeforeTestSupport; import com.consol.citrus.jms.endpoint.JmsEndpoint; import org.apache.activemq.ActiveMQConnectionFactory; import org.apache.commons.dbcp.BasicDataSource; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import javax.jms.ConnectionFactory; @Configuration public class CitrusConfiguration { @Value("${jms.broker.url}") private String jmsBrokerUrl; @Value("${jms.endpoint.out.uc}") private String jmsEndpointOutUc; @Value("${jms.endpoint.in}") private String jmsEndpointIn; @Value("${jms.receive.timeout}") private long jmsReceiveTimeout; @Value("${db.url}") private String dbUrl; @Value("${db.driver}") private String dbDriver; @Value("${db.user}") private String dbUser; @Value("${db.password}") private String dbPassword; @Bean public JmsEndpoint jmsEndpointOutUc() { return CitrusEndpoints.jms() .asynchronous() .connectionFactory(connectionFactory()) .destination(jmsEndpointOutUc) .build(); } @Bean public JmsEndpoint jmsEndpointIn() { return CitrusEndpoints.jms() .asynchronous() .connectionFactory(connectionFactory()) .destination(jmsEndpointIn) .pubSubDomain(true) .autoStart(true) .timeout(jmsReceiveTimeout) .build(); } @Bean public ConnectionFactory connectionFactory() { return new ActiveMQConnectionFactory(jmsBrokerUrl); } @Bean(destroyMethod = "close") public BasicDataSource datasource() { final BasicDataSource dataSource = new BasicDataSource(); dataSource.setDriverClassName(dbDriver); dataSource.setUrl(dbUrl); dataSource.setUsername(dbUser); dataSource.setPassword(dbPassword); return dataSource; } @Bean public SequenceBeforeTest beforeTest() { return new TestRunnerBeforeTestSupport() { @Override public void beforeTest(TestRunner runner) { runner.purgeQueues(purgeJmsQueueBuilder -> purgeJmsQueueBuilder .connectionFactory(connectionFactory()) .queueNames(jmsEndpointOutUc1)); runner.purgeQueues(purgeJmsQueueBuilder -> purgeJmsQueueBuilder .connectionFactory(connectionFactory()) .queue(jmsEndpointIn)); } }; } } ``` ```java package com.consol.testcontainers.transaction.test; import com.consol.citrus.annotations.CitrusTest; import com.consol.citrus.dsl.testng.TestNGCitrusTestRunner; import com.consol.citrus.jms.endpoint.JmsEndpoint; import org.springframework.beans.factory.annotation.Autowired; import org.testng.annotations.Test; import javax.sql.DataSource; @Test public class UC_Test extends TestNGCitrusTestRunner { @Autowired JmsEndpoint jmsEndpointOutUc; @Autowired JmsEndpoint jmsEndpointIn; @Autowired DataSource dataSource; @Test @CitrusTest public void ok() { createVariable("message", "{\"id\": {citrus:randomNumber(10)}, \"name\": \"Just a simple test\"}"); send(action -> action.endpoint(jmsEndpointOutUc).payload("${message}")); repeatOnError().until("i = 5").index("i").autoSleep(100).actions( echo("check database: ${i} from max 5 tries"), query(action -> action.dataSource(dataSource) .statement("select count(*) as entry_found from t_testcontainers where message='${message}';") .validate("entry_found", "1")) ); receive(action -> action.endpoint(jmsEndpointIn).payload("${message}")); } @Test @CitrusTest public void rollback() { createVariable("message", "{\"id\": {citrus:randomNumber(10)}, \"name\": \"I will throw an Exception\"}"); send(action -> action.endpoint(jmsEndpointOutUc).payload("${message}")); receiveTimeout(action -> action.endpoint(jmsEndpointIn)); query(action -> action.dataSource(dataSource) .statement("select count(*) as entry_found from t_testcontainers where message='${message}';") .validate("entry_found", "0")); } } ``` ## Citrus and Testcontainers compared As described in the beginning, we usually use the [Citrus framework](https://citrusframework.org) for integration testing. Therefore, I would now like to discuss the differences to Testcontainers I noticed. First of all the concept of Citrus with the **test sources** as part of an independently project or module offers opportunities like a normal service deployment (on another machine). The service and the tests can therefore be individually delivered and started. To **write tests** Citrus offers a huge support for different use cases (see [documentation](https://citrusframework.org/citrus/reference/html/index.html)) and Testcontainer allows to use nearly every docker image available. For some special images there are also own container classes to use like in our example the database. A big advantage of Citrus is the **validation** (of every detail). In contrast to this has Testcontainers at the moment no special support for that and you have to write this parts for your own or even use other libraries for that. The **delivery** can be done with Citrus separately which allows a **execution** ot the tests against a normal deployed application even on another machine. On the other side Testcontainers can run nearly everywhere (a docker installation is required) since the needed services are set up as part of the tests. ## Conclusion Testcontainers is a good choice for a fast provisioning of the infrastructure. But it need support by other frameworks when it has to use the provided services. The advantages of Citrus are more likely the other wise with the validation and usage of third party systems. A combination of Testcontainers with other testing tools (for example Citrus, Wiremock, Rest-Assured, ...) to control specific protocols or allowing a validation of data structures would be a technology stack which raises integration testing to the next level. [Read more](https://omd.consol.de/blog/2020/10/27/integration-testing-with-testcontainers/) ----------------------------------------------------------------------------------- title: "OpenShift and Let's Encrypt" date: "2020-09-16" -------------------------------------------------------------------------------- So you have this nifty web application deployed on your OpenShift cluster and you want to make it accessible by the whole world with HTTPS under the name `coolapp.`. Unfortunately you face several issues: * Exposing the service to your web application leaves you with a route using the self-signed certificate that was generated during setup of the cluster. None of the browsers in the wild will trust this certificate. * The self-signed certificate dictates URLS of the form `https://.apps..` (or whatever domain suffix you configured). Not very nice. * You might mitigate the previous issues by getting an official certificate signed by a generally trusted institution. But you will have to pay for it. * And you will have to pay for it not only once but every year (latest every 389 days) thanks to recently tightened certificate policies installed by all major browser vendors. * Worst of it all: You must not (by any means) forget to apply for a new certificate in a timely manner and replace the certificate in your route before the old expires. Otherwise some people might get pretty angry about you. *Let's Encrypt* to the rescue! ## *Let's Encrypt* briefly [Let's Encrypt](https://letsencrypt.org/) has done a tremendous job during the last couple of years (exactly since effectively September 7 2015) to promote general transport encryption in the World Wide Web by * issuing free (as in free beer) TLS certificates generally accepted by all recent web browsers, * providing a protocol to automate the certificate issuance and renewal, * providing a reference implementation of the client side of this protocol ([certbot](https://certbot.eff.org/)) trying to make it as easy as possible for administrators to automate renewal and installation of TLS certificates and * last but not least (due to the automation presented) issueing certificates valid only for a relatively short period of time (90 days) giving a pragmatic solution to the (never properly solved) problem of certificate revocation. As a moral side-note *Let's Encrypt* also has drained the money-for-nothing-business practised by so many other certification bodies for so long basically offering the very same level of domain verification for horrendous costs. In essence to prove that the domain in question is under your control *Let's Encrypt* sends you as the supplicant a challenge (aka token) that you have to present under a well-known URL on your web server (HTTP-01) or as a TXT record with a well-known domain name in the DNS (DNS-01). Read the excellent documentation on their website about [how it works](https://letsencrypt.org/how-it-works/) and [all other details](https://letsencrypt.org/docs/). ## The OpenShift challenge We want to take advantage of *Let's Encrypt* to get a generally accepted certificate for our web application fully automated and for free. As soon as everything is set up our web application will be provided with up-to-date certificates automagically as long as it stays online. There have been early attempts to use certbot in an OpenShift cluster. But the circumstances in a cluster are so different compared to a bare metal machine runnning e.g. an nginx web server that these attempts did not gain much traction. Instead special solutions have been implemented. ## The options Three tools were taken into closer consideration: * [OpenShift-ACME](https://github.com/tnozicka/openshift-acme/) is a controller monitoring routes. Special annotations cause OpenShift-ACME to request TLS certificates for each route and taking care of its lifecycle (i.e. timely renewal). OpenShift-ACME so far only supports domain verification via HTTP. * [Certman-Operator](https://github.com/openshift/certman-operator) is an operator highly specialized for Red Hat's [dedicated cluster](https://cloud.redhat.com/) service. It only supports domain verification via DNS and only with AWS Route53, i.e. any OpenShift cluster running on something different than AWS is out of the game. * [Cert-Manager](https://cert-manager.io/) is the most advanced and flexible solution as it supports domain verification via HTTP and DNS - the latter on a long list of platforms as it comes with many plugins for various DNS servers. It's more targeted towards Kubernetes but can also be used on OpenShift. Routes are [not natively supported](https://github.com/jetstack/cert-manager/issues/1064), but this may [change](https://github.com/jetstack/cert-manager/pull/2840) in the near future. However, with the help of ingresses (yes, they exist in OpenShift too) routes can be automatically fitted with TLS certificates provided by *Let's Encrypt*. Unsurprisingly the following instructions will explain how to use Cert-Manager. We will focus on domain verification via DNS. Although more complicated to set up this approach is more flexible as it is also applicable for web applications not reachable from the Internet. From our experience this use case is quite common in enterprise environments. We will show how to employ [ACME-DNS](https://github.com/joohoi/acme-dns), a stripped down DNS server with just enough functionality to serve the TXT records needed for domain verification via DNS. With ACME-DNS we avoid any vendor (i.e. cluster provider) lock-in. We will mention alternatives and options along the way wherever we deem it worth pointing out, but will not go into details. ## Installation We assume that your OpenShift cluster is version 4. You can use Cert-Manager on OpenShift 3, but please consult the [installation instructions](https://cert-manager.io/docs/installation/openshift/) for limitations and special procedures. You have two options to install Cert-Manager. You can either install it as an operator directly from your cluster's web console (*Administrator view: Operators → OperatorHub → Search for 'cert-manager'*). By the time of writing this article (2020-09-16) the operator from the catalog was quite outdated (version 0.15.2 listed while v1.0.1 was the latest available). The installation instructions mention a [link to Red Hat's operator catalog](https://cert-manager.io/docs/installation/openshift/#installing-with-cert-manager-operator) that is defunct. The second option is to install all necessary resources old-style with a long template file. As we want to go for the latest and greatest this is our way. Log in to your cluster as cluster administrator, create a namespace for your Cert-Manager resources and apply the template to create all the resources. ``` oc login ... oc new-project cert-manager oc apply --validate=false \ -f https://github.com/jetstack/cert-manager/releases/download/v1.0.1/cert-manager.yaml ``` > Note: The --validate=false flag is added to the oc apply command above or else you will receive a validation error relating to the caBundle field of the ValidatingWebhookConfiguration resource. ## ACME-DNS Before we go into configuring Cert-Manager we have to prepare ACME-DNS. As we want to use domain verification via DNS we need to be able to programmatically set certain TXT records. More specific: We request a (new) TLS certificate from *Let's Encrypt* for the domain `coolapp.mydomain.com` and get a challenge in response. We have to present this challenge[-1-](#note1) in a TXT record with domain name `_acme-challenge.coolapp.mydomain.com`. Unfortunately most DNS servers do not provide any API to manipulate zone information programmatically. Instead we delegate the TXT record to some other domain served by ACME-DNS. The above mentioned TXT record becomes a CNAME record pointing to this new domain. ``` _acme-challenge.coolapp.mydomain.com (CNAME) ==> caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io.auth.acme-dns.io (TXT) ``` We have to register with `acme-dns.io` by sending a POST request to its REST API. ``` curl --data '{"allowfrom":["194.246.122.0/24"]}' https://auth.acme-dns.io/register | jq . > acme-dns.json ``` With this command you register a subdomain under `auth.acme-dns.io` where you can put your TXT record programmatically (as will be explained shortly). The JSON data in the request restricts access to the API for this subdomain to a certain IP range. Send an empty request (`-X POST` instead of `--data '{...}'`) if you don't need this additional security. See ACME-DNS's [usage instructions](https://github.com/joohoi/acme-dns#usage) for details. The response (stored in `acme-dns.json`) looks like ``` { "username": "9a88737a-7014-420a-90bd-de11cc3a9a6e", "password": "E5uaq4v5ftSZPwFPzQsXvT0uzDBu5rkEGfdOP6mo", "fulldomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io", "subdomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc", "allowfrom": [ "194.246.122.0/24" ] } ``` After sending a POST request to the endpoint `https://auth.acme-dns.io/update` with username and password in request headers and subdomain and the challenge in the request body ACME-DNS will serve a TXT record with name `caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io` and the challenge as content. Cert-Manager will do exactly that later. Don't forget to (manually) set the CNAME record (e.g. `_acme-challenge.coolapp.mydomain.com`) pointing to `caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io`. ACME-DNS is open source. You can [run the server by yourself](https://github.com/joohoi/acme-dns#self-hosted) in case you do not want to be dependent on `acme-dns.io`. ## Configuration Next step is to set up where certificates will be requested and how the domain verification will be performed. This is achieved by creating an instance of `Issuer` or `ClusterIssuer`. The first is a namespaced resource while the latter is valid for the whole cluster. ```yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: letsencrypt-staging namespace: coolapp spec: acme: email: "cert-master@mydomain.com" privateKeySecretRef: name: letsencrypt-account-key server: https://acme-staging-v02.api.letsencrypt.org/directory solvers: - dns01: acmeDNS: accountSecretRef: key: acme-dns-account name: acme-dns-secret host: https://auth.acme-dns.io cnameStrategy: Follow ``` This issuer will use the ACME protocol (`spec.acme`) and the endpoint will be *Let's Encrypt*'s staging endpoint (`spec.acme.server`). Your *Let's Encrypt* account credentials will be stored in the secret `letsencrypt-account-key`. You don't have to prepare anything there as Cert-Manager will take care of registration automatically the first time it contacts *Let's Encrypt*. Below `spec.acme.solvers` we define what domain verification will be used.[-2-](#note2) The key-value-pair `cnameStrategy: Follow` is important as it allows to delegate the TXT record to some other domain by means of a CNAME record. Instead of `acmeDNS` you can specify one of the other [supported DNS providers](https://cert-manager.io/docs/configuration/acme/dns01/#supported-dns01-providers). In case your OpenShift cluster is run by one of the listed cloud providers you can use their proprietary interface. If you opt for running an ACME-DNS server by yourself `host: https://auth.acme-dns.io` is the place where you have to put the URL of your self-hosted instance. The credentials we got when we registered with `acme-dns.io` have to be stored in the secret `acme-dns-secret`. Create it with this YAML file ``` kind: Secret apiVersion: v1 metadata: name: acme-dns-secret namespace: coolapp stringData: acme-dns-account: |- { "coolapp.mydomain.com": { "username": "9a88737a-7014-420a-90bd-de11cc3a9a6e", "password": "E5uaq4v5ftSZPwFPzQsXvT0uzDBu5rkEGfdOP6mo", "fulldomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io", "subdomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc" } } type: Opaque ``` Of course you have to adapt the data to match your domain, your ACME-DNS credentials, your ACME-DNS subdomain and the domain(s) you want certificates for. With Issuer configured, account created on acme-dns.io and credentials stored in a secret everything is now in place to request a certificate. ## The DNS resolver pitfall During our experiments with cert-manager we encoutered error messages in the log output of the `cert-manager` pod. ``` $ oc logs -f -n cert-manager cert-manager-75ff5bf6d6-n7sv5 .... E0910 11:30:35.262146 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="dial tcp 46.4.128.227:53: i/o timeout" .... ``` (Formatting not in the original output.) Domain verification finally failed. This is what happened in detail: We had requested a new certificate. Cert-Manager sent a request to *Let's Encrypt* and got back the challenge to be used in the following domain verification. Cert-Manager prepared the TXT record in our subdomain of `auth.acme-dns.io`. Cert-Manager was about to get back to *Let's Encrypt* to tell them *Everything is in place now. Go ahead and verify that we are in control of the domain.* Before actually doing that Cert-Manager tried to verify itself that the TXT record looked as expected. Instead of simply using the DNS resolver configured in the pod it somehow determined the authoritative nameserver for the TXT record and tried to contact it directly (supposedly to bypass any caches on the way from the local resolver to the authorative nameserver that might hold outdated records). In our case this was the observed TCP connection to `46.4.128.227` (`auth.acme-dns.io`) port 53. This connection timed out because it was blocked by our firewall. Actually this is a common firewall rule in enterprise environments to avoid confusion in 'split horizon DNS' setups, e.g. `www.mydomain.com` must deliver an internal IP address if resolved by an internal host contrary to an external IP address when resolved by some host in the Internet. The solution for this issue was to modify the deployment of the `cert-manager` pod. First find out the IP address of the resolver configured in the pod. As the images for the Cert-Manager pods are super-lean (i.e. no shell, no basic unix tools) you will have to use some other existing pod you have access to. ``` oc rsh -n cat /etc/resolv.conf ``` Then ... ``` oc edit deployment cert-manager -n cert-manager ``` Find the section with the arguments for the command run within the pod. ``` spec: containers: - args: - --v=2 - --cluster-resource-namespace=$(POD_NAMESPACE) - --leader-election-namespace=kube-system ``` Add two more lines ... ``` spec: containers: - args: - --v=2 - --cluster-resource-namespace=$(POD_NAMESPACE) - --leader-election-namespace=kube-system - --dns01-recursive-nameservers=":53" - --dns01-recursive-nameservers-only ``` This forces Cert-Manager to use the local resolver only (no direct communication with remote nameservers). The downside being that verifying the TXT record will take longer[-3-](#note3). ## Requesting a certificate Requesting a certificate is accomplished by creating an instance of `Certificate`. Here is the corresponding YAML file ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: coolapp-mydomain-com namespace: coolapp spec: dnsNames: - coolapp.mydomain.com duration: 2160h renewBefore: 360h issuerRef: kind: Issuer name: letsencrypt-staging secretName: coolapp-mydomain-com-tls subject: organizations: - "My company" usages: - server auth ``` Adapt to your data (domain name(s), organization name(s), secret name, etc.) and create the `Certificate` instance with `oc apply -f ...`. `duration` is probably obsolete as *Let's Encrypt* delivers certificates valid for 90 days in any case. As soon as the `Certificate` instance has been created some controller jumps in and starts a certification process. Behind the scenes an instance of `CertificateRequest` is created which in turn creates an instance of `Order`. You can watch these instances as the certification proceeds. Finally the certificate gets stored in the secret given in the `Certificate` instance with `spec.secretName`. ## Securing your route So far we have got a TLS certificate stored in a secret. On the other hand routes in Openshift do not read TLS data from a secret but store it in the route directly.[-4-](#note4) Kubernetes ingresses do it that way, but we are on OpenShift. By the time of writing (2020-09-16) routes were [not natively supported](https://github.com/jetstack/cert-manager/issues/1064) by Cert-Manager. A [pull request](https://github.com/jetstack/cert-manager/pull/2840) was already in a very advances stage. So probably routes will be supported in the near future. In the meantime you have to apply a little trick. The official OpenShift documentation doesn't mention it (for unknown reasons) but it is possible to create `Ingress` instances in OpenShift too. They are not processed by the ingress controller directly in a way that HAproxy is configured accordingly but instead the ingress controller automatically creates one or more `Route` instances that mimick the desired behavior of the `Ingress`. It even monitors the `Ingress` instance and adapts the routes in case anything changes with the ingress. That's the way we need to go. Delete any existing route for your application and create an instance of `Ingress` ``` apiVersion: extensions/v1beta1 kind: Ingress metadata: name: coolapp-ingress namespace: coolapp spec: rules: - host: coolapp.mydomain.com http: paths: - backend: serviceName: coolapp servicePort: 8080 path: / pathType: Prefix tls: - hosts: - coolapp.mydomain.com secretName: coolapp-mydomain.com-tls ``` Again adapt the data to your needs and then create the ingress with `oc apply -f ...`. Automagically a route comes into existance and the fields `spec.tls.certificate` and `spec.tls.key` contain the certificate data from the secret where Cert-Manager stored the certificate. ## Switching from Let's Encrypt staging to production Probably you will realize pretty fast that the route is of very limited use. Very intentionally the instructions so far consistently used the staging end point of *Let's Encrypt*. The resulting TLS certificates are signed by a fake CA not recognized by any TLS client. The reason is that the production endpoint of *Let's Encrypt* has pretty rigid rate limits. We want to avoid that the production endpoint becomes defunct for you due to e.g. too many failed verifications within one hour. We adhere to *Let's Encrypt*'s [recommendation](https://letsencrypt.org/docs/staging-environment/) to use their staging endpoint for experimentation. As soon as you are confident that everything works as expected you can switch to production as follows. Create a second instance of `Issuer` very similar to the one created above. The differences are `metadata.name: letsencrypt-prod` and `spec.acme.server: https://acme-v02.api.letsencrypt.org/directory`. Adapt the existing `Certificate` by changing `spec.issuerRef.name` to `letsencrypt-prod`. This will trigger Cert-Manager, the `CertificateRequest` will be adapted, a new `Order` instance will be created and a new verification process will start. In the end a new TLS certificate will be stored in the secret. This time signed by *Let's Encrypt*'s official certification authority. The route mimicking the ingress will be recreated by the ingress controller with the certificate data from the secret. ## Maintenance Hopefully none! Your certificate will be renewed by Cert-Manager automatically as soon as the certificate's expiration date minus the `renewBefore` value from the `Certificate` instance has been reached. Nevertheless we highly recommend that you monitor the first renewal - just to be on the safe side.
**Note 1** Actually not the challenge itself is stored in the TXT record but the challenge is mangled with your *Let's Encrypt* account id and the resulting SHA256 hash is stored as the contents of the TXT record.
**Note 2** Cert-Manager allows to configure multiple solvers here. Selection which solver is used for a specific certificate is done based on labels (of the corresponding `Certificate` resource instance) and matches against the domain name(s) or domain(s). Read the Cert-Manager [documentation](https://cert-manager.io/docs/configuration/acme/#adding-multiple-solver-types) for details.
**Note 3** TXT records served by ACME-DNS come with a time-to-live (TTL) of one second. So the difference should be neglectable. Nevertheless experiments showed that it sometimes takes significantly longer (up to 1 minute) till you see the updated TXT record - for whatever reasons.
**Note 4** There is an [issue](https://github.com/openshift/origin/issues/2162) about moving TLS certificate data from routes to secrets referenced by the routes that has been lingering in OpenShift's GitHub project since more than 5(!) years. [Read more](https://omd.consol.de/blog/2020/09/16/openshift-and-lets-encrypt/) ----------------------------------------------------------------------------------- title: "OpenShift and Let's Encrypt" date: "2020-09-16" -------------------------------------------------------------------------------- So you have this nifty web application deployed on your OpenShift cluster and you want to make it accessible by the whole world with HTTPS under the name `coolapp.`. Unfortunately you face several issues: * Exposing the service to your web application leaves you with a route using the self-signed certificate that was generated during setup of the cluster. None of the browsers in the wild will trust this certificate. * The self-signed certificate dictates URLS of the form `https://.apps..` (or whatever domain suffix you configured). Not very nice. * You might mitigate the previous issues by getting an official certificate signed by a generally trusted institution. But you will have to pay for it. * And you will have to pay for it not only once but every year (latest every 389 days) thanks to recently tightened certificate policies installed by all major browser vendors. * Worst of it all: You must not (by any means) forget to apply for a new certificate in a timely manner and replace the certificate in your route before the old expires. Otherwise some people might get pretty angry about you. *Let's Encrypt* to the rescue! ## *Let's Encrypt* briefly [Let's Encrypt](https://letsencrypt.org/) has done a tremendous job during the last couple of years (exactly since effectively September 7 2015) to promote general transport encryption in the World Wide Web by * issuing free (as in free beer) TLS certificates generally accepted by all recent web browsers, * providing a protocol to automate the certificate issuance and renewal, * providing a reference implementation of the client side of this protocol ([certbot](https://certbot.eff.org/)) trying to make it as easy as possible for administrators to automate renewal and installation of TLS certificates and * last but not least (due to the automation presented) issueing certificates valid only for a relatively short period of time (90 days) giving a pragmatic solution to the (never properly solved) problem of certificate revocation. As a moral side-note *Let's Encrypt* also has drained the money-for-nothing-business practised by so many other certification bodies for so long basically offering the very same level of domain verification for horrendous costs. In essence to prove that the domain in question is under your control *Let's Encrypt* sends you as the supplicant a challenge (aka token) that you have to present under a well-known URL on your web server (HTTP-01) or as a TXT record with a well-known domain name in the DNS (DNS-01). Read the excellent documentation on their website about [how it works](https://letsencrypt.org/how-it-works/) and [all other details](https://letsencrypt.org/docs/). ## The OpenShift challenge We want to take advantage of *Let's Encrypt* to get a generally accepted certificate for our web application fully automated and for free. As soon as everything is set up our web application will be provided with up-to-date certificates automagically as long as it stays online. There have been early attempts to use certbot in an OpenShift cluster. But the circumstances in a cluster are so different compared to a bare metal machine runnning e.g. an nginx web server that these attempts did not gain much traction. Instead special solutions have been implemented. ## The options Three tools were taken into closer consideration: * [OpenShift-ACME](https://github.com/tnozicka/openshift-acme/) is a controller monitoring routes. Special annotations cause OpenShift-ACME to request TLS certificates for each route and taking care of its lifecycle (i.e. timely renewal). OpenShift-ACME so far only supports domain verification via HTTP. * [Certman-Operator](https://github.com/openshift/certman-operator) is an operator highly specialized for Red Hat's [dedicated cluster](https://cloud.redhat.com/) service. It only supports domain verification via DNS and only with AWS Route53, i.e. any OpenShift cluster running on something different than AWS is out of the game. * [Cert-Manager](https://cert-manager.io/) is the most advanced and flexible solution as it supports domain verification via HTTP and DNS - the latter on a long list of platforms as it comes with many plugins for various DNS servers. It's more targeted towards Kubernetes but can also be used on OpenShift. Routes are [not natively supported](https://github.com/jetstack/cert-manager/issues/1064), but this may [change](https://github.com/jetstack/cert-manager/pull/2840) in the near future. However, with the help of ingresses (yes, they exist in OpenShift too) routes can be automatically fitted with TLS certificates provided by *Let's Encrypt*. Unsurprisingly the following instructions will explain how to use Cert-Manager. We will focus on domain verification via DNS. Although more complicated to set up this approach is more flexible as it is also applicable for web applications not reachable from the Internet. From our experience this use case is quite common in enterprise environments. We will show how to employ [ACME-DNS](https://github.com/joohoi/acme-dns), a stripped down DNS server with just enough functionality to serve the TXT records needed for domain verification via DNS. With ACME-DNS we avoid any vendor (i.e. cluster provider) lock-in. We will mention alternatives and options along the way wherever we deem it worth pointing out, but will not go into details. ## Installation We assume that your OpenShift cluster is version 4. You can use Cert-Manager on OpenShift 3, but please consult the [installation instructions](https://cert-manager.io/docs/installation/openshift/) for limitations and special procedures. You have two options to install Cert-Manager. You can either install it as an operator directly from your cluster's web console (*Administrator view: Operators → OperatorHub → Search for 'cert-manager'*). By the time of writing this article (2020-09-16) the operator from the catalog was quite outdated (version 0.15.2 listed while v1.0.1 was the latest available). The installation instructions mention a [link to Red Hat's operator catalog](https://cert-manager.io/docs/installation/openshift/#installing-with-cert-manager-operator) that is defunct. The second option is to install all necessary resources old-style with a long template file. As we want to go for the latest and greatest this is our way. Log in to your cluster as cluster administrator, create a namespace for your Cert-Manager resources and apply the template to create all the resources. ``` oc login ... oc new-project cert-manager oc apply --validate=false \ -f https://github.com/jetstack/cert-manager/releases/download/v1.0.1/cert-manager.yaml ``` > Note: The --validate=false flag is added to the oc apply command above or else you will receive a validation error relating to the caBundle field of the ValidatingWebhookConfiguration resource. ## ACME-DNS Before we go into configuring Cert-Manager we have to prepare ACME-DNS. As we want to use domain verification via DNS we need to be able to programmatically set certain TXT records. More specific: We request a (new) TLS certificate from *Let's Encrypt* for the domain `coolapp.mydomain.com` and get a challenge in response. We have to present this challenge[-1-](#note1) in a TXT record with domain name `_acme-challenge.coolapp.mydomain.com`. Unfortunately most DNS servers do not provide any API to manipulate zone information programmatically. Instead we delegate the TXT record to some other domain served by ACME-DNS. The above mentioned TXT record becomes a CNAME record pointing to this new domain. ``` _acme-challenge.coolapp.mydomain.com (CNAME) ==> caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io.auth.acme-dns.io (TXT) ``` We have to register with `acme-dns.io` by sending a POST request to its REST API. ``` curl --data '{"allowfrom":["194.246.122.0/24"]}' https://auth.acme-dns.io/register | jq . > acme-dns.json ``` With this command you register a subdomain under `auth.acme-dns.io` where you can put your TXT record programmatically (as will be explained shortly). The JSON data in the request restricts access to the API for this subdomain to a certain IP range. Send an empty request (`-X POST` instead of `--data '{...}'`) if you don't need this additional security. See ACME-DNS's [usage instructions](https://github.com/joohoi/acme-dns#usage) for details. The response (stored in `acme-dns.json`) looks like ``` { "username": "9a88737a-7014-420a-90bd-de11cc3a9a6e", "password": "E5uaq4v5ftSZPwFPzQsXvT0uzDBu5rkEGfdOP6mo", "fulldomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io", "subdomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc", "allowfrom": [ "194.246.122.0/24" ] } ``` After sending a POST request to the endpoint `https://auth.acme-dns.io/update` with username and password in request headers and subdomain and the challenge in the request body ACME-DNS will serve a TXT record with name `caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io` and the challenge as content. Cert-Manager will do exactly that later. Don't forget to (manually) set the CNAME record (e.g. `_acme-challenge.coolapp.mydomain.com`) pointing to `caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io`. ACME-DNS is open source. You can [run the server by yourself](https://github.com/joohoi/acme-dns#self-hosted) in case you do not want to be dependent on `acme-dns.io`. ## Configuration Next step is to set up where certificates will be requested and how the domain verification will be performed. This is achieved by creating an instance of `Issuer` or `ClusterIssuer`. The first is a namespaced resource while the latter is valid for the whole cluster. ```yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: letsencrypt-staging namespace: coolapp spec: acme: email: "cert-master@mydomain.com" privateKeySecretRef: name: letsencrypt-account-key server: https://acme-staging-v02.api.letsencrypt.org/directory solvers: - dns01: acmeDNS: accountSecretRef: key: acme-dns-account name: acme-dns-secret host: https://auth.acme-dns.io cnameStrategy: Follow ``` This issuer will use the ACME protocol (`spec.acme`) and the endpoint will be *Let's Encrypt*'s staging endpoint (`spec.acme.server`). Your *Let's Encrypt* account credentials will be stored in the secret `letsencrypt-account-key`. You don't have to prepare anything there as Cert-Manager will take care of registration automatically the first time it contacts *Let's Encrypt*. Below `spec.acme.solvers` we define what domain verification will be used.[-2-](#note2) The key-value-pair `cnameStrategy: Follow` is important as it allows to delegate the TXT record to some other domain by means of a CNAME record. Instead of `acmeDNS` you can specify one of the other [supported DNS providers](https://cert-manager.io/docs/configuration/acme/dns01/#supported-dns01-providers). In case your OpenShift cluster is run by one of the listed cloud providers you can use their proprietary interface. If you opt for running an ACME-DNS server by yourself `host: https://auth.acme-dns.io` is the place where you have to put the URL of your self-hosted instance. The credentials we got when we registered with `acme-dns.io` have to be stored in the secret `acme-dns-secret`. Create it with this YAML file ``` kind: Secret apiVersion: v1 metadata: name: acme-dns-secret namespace: coolapp stringData: acme-dns-account: |- { "coolapp.mydomain.com": { "username": "9a88737a-7014-420a-90bd-de11cc3a9a6e", "password": "E5uaq4v5ftSZPwFPzQsXvT0uzDBu5rkEGfdOP6mo", "fulldomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc.auth.acme-dns.io", "subdomain": "caeb38f2-6592-4128-d8be-da5e6039d1bc" } } type: Opaque ``` Of course you have to adapt the data to match your domain, your ACME-DNS credentials, your ACME-DNS subdomain and the domain(s) you want certificates for. With Issuer configured, account created on acme-dns.io and credentials stored in a secret everything is now in place to request a certificate. ## The DNS resolver pitfall During our experiments with cert-manager we encoutered error messages in the log output of the `cert-manager` pod. ``` $ oc logs -f -n cert-manager cert-manager-75ff5bf6d6-n7sv5 .... E0910 11:30:35.262146 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="dial tcp 46.4.128.227:53: i/o timeout" .... ``` (Formatting not in the original output.) Domain verification finally failed. This is what happened in detail: We had requested a new certificate. Cert-Manager sent a request to *Let's Encrypt* and got back the challenge to be used in the following domain verification. Cert-Manager prepared the TXT record in our subdomain of `auth.acme-dns.io`. Cert-Manager was about to get back to *Let's Encrypt* to tell them *Everything is in place now. Go ahead and verify that we are in control of the domain.* Before actually doing that Cert-Manager tried to verify itself that the TXT record looked as expected. Instead of simply using the DNS resolver configured in the pod it somehow determined the authoritative nameserver for the TXT record and tried to contact it directly (supposedly to bypass any caches on the way from the local resolver to the authorative nameserver that might hold outdated records). In our case this was the observed TCP connection to `46.4.128.227` (`auth.acme-dns.io`) port 53. This connection timed out because it was blocked by our firewall. Actually this is a common firewall rule in enterprise environments to avoid confusion in 'split horizon DNS' setups, e.g. `www.mydomain.com` must deliver an internal IP address if resolved by an internal host contrary to an external IP address when resolved by some host in the Internet. The solution for this issue was to modify the deployment of the `cert-manager` pod. First find out the IP address of the resolver configured in the pod. As the images for the Cert-Manager pods are super-lean (i.e. no shell, no basic unix tools) you will have to use some other existing pod you have access to. ``` oc rsh -n cat /etc/resolv.conf ``` Then ... ``` oc edit deployment cert-manager -n cert-manager ``` Find the section with the arguments for the command run within the pod. ``` spec: containers: - args: - --v=2 - --cluster-resource-namespace=$(POD_NAMESPACE) - --leader-election-namespace=kube-system ``` Add two more lines ... ``` spec: containers: - args: - --v=2 - --cluster-resource-namespace=$(POD_NAMESPACE) - --leader-election-namespace=kube-system - --dns01-recursive-nameservers=":53" - --dns01-recursive-nameservers-only ``` This forces Cert-Manager to use the local resolver only (no direct communication with remote nameservers). The downside being that verifying the TXT record will take longer[-3-](#note3). ## Requesting a certificate Requesting a certificate is accomplished by creating an instance of `Certificate`. Here is the corresponding YAML file ``` apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: coolapp-mydomain-com namespace: coolapp spec: dnsNames: - coolapp.mydomain.com duration: 2160h renewBefore: 360h issuerRef: kind: Issuer name: letsencrypt-staging secretName: coolapp-mydomain-com-tls subject: organizations: - "My company" usages: - server auth ``` Adapt to your data (domain name(s), organization name(s), secret name, etc.) and create the `Certificate` instance with `oc apply -f ...`. `duration` is probably obsolete as *Let's Encrypt* delivers certificates valid for 90 days in any case. As soon as the `Certificate` instance has been created some controller jumps in and starts a certification process. Behind the scenes an instance of `CertificateRequest` is created which in turn creates an instance of `Order`. You can watch these instances as the certification proceeds. Finally the certificate gets stored in the secret given in the `Certificate` instance with `spec.secretName`. ## Securing your route So far we have got a TLS certificate stored in a secret. On the other hand routes in Openshift do not read TLS data from a secret but store it in the route directly.[-4-](#note4) Kubernetes ingresses do it that way, but we are on OpenShift. By the time of writing (2020-09-16) routes were [not natively supported](https://github.com/jetstack/cert-manager/issues/1064) by Cert-Manager. A [pull request](https://github.com/jetstack/cert-manager/pull/2840) was already in a very advances stage. So probably routes will be supported in the near future. In the meantime you have to apply a little trick. The official OpenShift documentation doesn't mention it (for unknown reasons) but it is possible to create `Ingress` instances in OpenShift too. They are not processed by the ingress controller directly in a way that HAproxy is configured accordingly but instead the ingress controller automatically creates one or more `Route` instances that mimick the desired behavior of the `Ingress`. It even monitors the `Ingress` instance and adapts the routes in case anything changes with the ingress. That's the way we need to go. Delete any existing route for your application and create an instance of `Ingress` ``` apiVersion: extensions/v1beta1 kind: Ingress metadata: name: coolapp-ingress namespace: coolapp spec: rules: - host: coolapp.mydomain.com http: paths: - backend: serviceName: coolapp servicePort: 8080 path: / pathType: Prefix tls: - hosts: - coolapp.mydomain.com secretName: coolapp-mydomain.com-tls ``` Again adapt the data to your needs and then create the ingress with `oc apply -f ...`. Automagically a route comes into existance and the fields `spec.tls.certificate` and `spec.tls.key` contain the certificate data from the secret where Cert-Manager stored the certificate. ## Switching from Let's Encrypt staging to production Probably you will realize pretty fast that the route is of very limited use. Very intentionally the instructions so far consistently used the staging end point of *Let's Encrypt*. The resulting TLS certificates are signed by a fake CA not recognized by any TLS client. The reason is that the production endpoint of *Let's Encrypt* has pretty rigid rate limits. We want to avoid that the production endpoint becomes defunct for you due to e.g. too many failed verifications within one hour. We adhere to *Let's Encrypt*'s [recommendation](https://letsencrypt.org/docs/staging-environment/) to use their staging endpoint for experimentation. As soon as you are confident that everything works as expected you can switch to production as follows. Create a second instance of `Issuer` very similar to the one created above. The differences are `metadata.name: letsencrypt-prod` and `spec.acme.server: https://acme-v02.api.letsencrypt.org/directory`. Adapt the existing `Certificate` by changing `spec.issuerRef.name` to `letsencrypt-prod`. This will trigger Cert-Manager, the `CertificateRequest` will be adapted, a new `Order` instance will be created and a new verification process will start. In the end a new TLS certificate will be stored in the secret. This time signed by *Let's Encrypt*'s official certification authority. The route mimicking the ingress will be recreated by the ingress controller with the certificate data from the secret. ## Maintenance Hopefully none! Your certificate will be renewed by Cert-Manager automatically as soon as the certificate's expiration date minus the `renewBefore` value from the `Certificate` instance has been reached. Nevertheless we highly recommend that you monitor the first renewal - just to be on the safe side.
**Note 1** Actually not the challenge itself is stored in the TXT record but the challenge is mangled with your *Let's Encrypt* account id and the resulting SHA256 hash is stored as the contents of the TXT record.
**Note 2** Cert-Manager allows to configure multiple solvers here. Selection which solver is used for a specific certificate is done based on labels (of the corresponding `Certificate` resource instance) and matches against the domain name(s) or domain(s). Read the Cert-Manager [documentation](https://cert-manager.io/docs/configuration/acme/#adding-multiple-solver-types) for details.
**Note 3** TXT records served by ACME-DNS come with a time-to-live (TTL) of one second. So the difference should be neglectable. Nevertheless experiments showed that it sometimes takes significantly longer (up to 1 minute) till you see the updated TXT record - for whatever reasons.
**Note 4** There is an [issue](https://github.com/openshift/origin/issues/2162) about moving TLS certificate data from routes to secrets referenced by the routes that has been lingering in OpenShift's GitHub project since more than 5(!) years. [Read more](https://omd.consol.de/blog/2020/09/16/openshift-and-lets-encrypt/) ----------------------------------------------------------------------------------- title: "How to add an application to a Red Hat OpenShift Service Mesh" date: "2020-08-07" --------------------------------------------------------------------------------
During a discussion with a customer, we talked about which steps are necessary to add an application to a services mesh. Which should be no big deal. Unfortunately, there is not a simple guideline how to do that for the Red Hat OpenShift Service Mesh. Furthermore, I was not sure how the requests for the application would look like in Jaeger. To clarify these points, I created a small application. Which I then deployed on OpenShift and added it to a service mesh control plane. This is the documentation of the steps that I have done.
## The demo application The created demo application can call other applications and mocks internal processing. Hence, I'm fluent in Java, I created a Java REST-Service that takes a message and a counter. The response returns the message. The counter determines the number of recursive method calls. The environment variable `BACKEND_SERVICE_URL` points to a demo application instance that serves as a backend. If the environment variable is not defined, then a method is called recursively. The request flow of the application looks like this: ![](sequence_request_call.png) The source code of the REST-API looks like this: ```java @Path("/") public class TestResource { @Inject WorkerService worker; @GET @Path("message/{message}") @Produces(MediaType.TEXT_PLAIN) public String message(@PathParam String message, @QueryParam Integer counter) { if(message== null){ message="hello"; } if(counter== null){ counter=10; } return worker.doStuff(message,counter); } } ``` The source code of the service looks like this: ```java @ApplicationScoped public class WorkerService { @Inject @RestClient BackendService backendService; private static final Logger log = LoggerFactory.getLogger(WorkerService.class); public String doStuff(String message, int counter) { log.debug("Message: {}, counter: {}", message, counter); try { Thread.sleep((long) (Math.random() * 1000)); } catch (InterruptedException e) { } if (counter == 0) { log.debug("Counter is zero, do nothing", message, counter); return message; } message = message + " " + counter; counter--; if (System.getenv("BACKEND_SERVICE_URL") != null) { log.debug("URL for backend service is: " + System.getenv("BACKEND_SERVICE_URL")); message = backendService.getMessage(message, counter); } else { log.debug("Call doStuff recursively"); message = doStuff(message, counter); } return message; } } ``` The source code of the backend service looks like this: ```java @RegisterRestClient(configKey = "backend-service") @RegisterProvider(LogClientRequestFilter.class) public interface BackendService { @Path("/message/{message}") @GET @Produces("text/plain") public String getMessage(@PathParam String message,@QueryParam Integer counter); } ``` As you can see, it is pretty straight forward and involves no magic. It is standard Java code using Quarkus as an Eclipse MicroProfile framework. ## Current setup of the application The current setup of the application consists of two deployments. The first is the frontend, which takes the request and calls the backend application. This backend application is performing some calculations (calling recursively the function `doStuff`) and returning the result to the frontend. A route is exposing the frontend application. A typical request and response looks like this: ```bash $ curl messager-entrypoint-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10 Hello World 10 9 8 7 6 5 4 3 2 1 ``` ## Add application to service mesh We are at a point where we have an existing application up and running in one project. The Service mesh control plane is installed in another project. Let's bring them together by creating a resource of the type ServiceMeshMemberRoll. It contains the name of the service mesh control plane project and the projects of the service mesh data plane. The data plane consists only of the project with the demo application. Our setup looks like this: {% highlight yaml hl_lines="5 8"%} apiVersion: maistra.io/v1 kind: ServiceMeshMemberRoll metadata: name: default namespace: olaf-smcp spec: members: - test-tracing {% endhighlight %} Line 5 contains the project for the control plane. The project of the data plane is defined in line 8. So far, so good. Let's call the demo application to test if everything still works. The result looks like this: ```bash $ curl -v messager-entrypoint-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10 * Trying 10.4.10.20... * TCP_NODELAY set * Connected to messager-entrypoint-test-tracing.apps.ocp.acme.com (10.4.10.20) port 80 (#0) > GET /message/Hello%20World?counter=10 HTTP/1.1 > Host: messager-entrypoint-test-tracing.apps.ocp.acme.com > User-Agent: curl/7.58.0 > Accept: */* > * HTTP 1.0, assume close after body < HTTP/1.0 503 Service Unavailable < pragma: no-cache < cache-control: private, max-age=0, no-cache, no-store < content-type: text/html < .... ``` So, what happened? When adding a project to the data plane, then the Istio Operator adds networks rules to the project. One rule allows pods from the control plane to access the project, which makes perfect sense. How else should Istio instrument the pods, if it doesn't have access to them? Another rule defines that namespaces with the label `network.openshift.io/policy-group: ingress` can access pods with the label `maistra.io/expose-route: "true"`. An example of this is the namespace with the OpenShift Router or ingress pods (`openshift-ingress`). Our deployments of the demo application don't have this label. Hence, the route can't access the demo application. ## Add labels to deployment and pod for route To fix this, add the label `maistra.io/expose-route: "true"` to the deployments of the demo application. Do this with these commands: ```bash $ oc patch deployment messager-backend -p '{"spec":{"template":{"metadata":{"labels":{"maistra.io/expose-route":"true"}}}}}' deployment.apps/messager-backend patched $ oc patch deployment messager-entrypoint -p '{"spec":{"template":{"metadata":{"labels":{"maistra.io/expose-route":"true"}}}}}' deployment.apps/messager-backend patched ``` If the Router/Ingress controllers of your OpenShift cluster are running on the host network, you need to add the label `network.openshift.io/policy-group: ingress` to the default project as well. A detailed explanation for this is [here](https://docs.openshift.com/container-platform/4.4/networking/configuring-networkpolicy.html). After this, our demo application can be called via the defined route. --- *Remark:* Labeling the pods with `maistra.io/expose-route: "true"` is an optional step. If it is not required to access applications via a route, then you don't need to label the pods. In this case, I recommend to remove the routes of the application from the project as well. --- ## Add annotation for sidecar auto injection to deployment Now we adjust the deployments, so that if a pod gets deployed then the Istio sidecar container is injected as well. Execute the following command to do this: ```bash $ oc patch deployment messager-backend -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}' deployment.apps/messager-backend patched $ oc patch deployment messager-entrypoint -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}' deployment.apps/messager-backend patched ``` This command will add an annotation `sidecar.istio.io/inject: "true"` to the pod template of the deployment. With this annotation, the service mesh control plane will inject Istio sidecar container in the pod. If you want, you can of course use `istioctl` to do this. After adding the annotation, a new deployment should be started. If this is not the case, delete the replica sets. After the restart of the pods they should contain the Istio sidecar. To verify this, get a list of all pods. ```bash oc get po NAME READY STATUS RESTARTS AGE messager-backend-85cc85d87-hsx8d 2/2 Running 0 4m26s messager-entrypoint-7f75457fcc-f7qc7 2/2 Running 0 11s ``` The number of containers is increased from one (only our application container) to two (Istio sidecar plus application container). ## Create ingress gateway with new route It is good practice to use the Istio ingress gateway instead of calling the application pod via route. For this, we need to add two resources: 1. A gateway: This determines the ingress point. Whether it is using HTTP or HTTPS, the hostname certificates and so on ... 1. A virtual service which routes the incoming traffic to a service. The demo application looks like this: ```yaml --- apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: tracingdemo-entrypoint namespace: test-tracing spec: selector: istio: ingressgateway servers: - hosts: - messager-entrypoint2-test-tracing.apps.ocp.acme.com port: name: http number: 80 protocol: HTTP --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: tracingdemo-entrypoint-vs namespace: test-tracing spec: gateways: - tracingdemo-entrypoint hosts: - messager-entrypoint2-test-tracing.apps.ocp.acme.com http: - route: - destination: host: messager-entrypoint port: number: 8080 ``` So lets add these two resources to the demo application namespaces: ```bash oc create -f setup_gateway.yml ``` --- *Remark:* If your service mesh control plane hasn't enabled IOR, then you need to create a route that uses the defined hostname of the gateway and points to the ingress service of the service mesh control plane. --- So, finally, the demo application can be called with the bare minimum Istio setup. Let's try to call the application URL from the command line: ```bash $ curl messager-entrypoint2-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10 Hello World 10 9 8 7 6 5 4 3 2 1 ``` This looks good. The request and response is as expected. ## Display calls in Kiali and Jaeger To view only a single request in Kiali is not feasible, let's put the previous command in a loop and run it: ```bash $ while :; do curl messager-entrypoint2-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10; sleep 1; done Hello World 10 9 8 7 6 5 4 3 2 1Hello World 10 9 8 7 6 5 4 3 2 1Hello World 10 9 8 7 6 5 4 3 2 1 ``` The graph for our demo in Kiali looks like this: ![](kiali_overview_without_opentracing.jpg) This matches the request flow described above. This is the point to start exploring more features of Istio and extent the configuration bit by bit. Let's have a look at Jaeger to see how an initial application without tracing support looks like. For a single request the graph looks like this in Jaeger: ![](jaeger_without_tracing.jpg) The screenshot above shows that Jaeger gets only information about the request to the ingress gateway. It doesn't show any information about the frontend or backend application. This is not helpful for debugging an application call flow. Istio is generating HTTP headers for tracing. These headers are not processed by the Java application and used in calls of subsequent applications. So, Jaeger gets only information about the incoming request to the container and not how the applications are actually connected. ## How to fix the Jaeger issue? How can we fix this issue? After some try and error, I came up with this modification in my source code, which will add the tracing HTTP header to outgoing calls as well as sent tracing information about internal calls to Jaeger: The source code of the REST-API look like this: {% highlight java hl_lines="1" %} @Traced @Path("/") public class TestResource { @Inject WorkerService worker; @GET @Path("message/{message}") @Produces(MediaType.TEXT_PLAIN) public String message(@PathParam String message, @QueryParam Integer counter) { if(message== null){ message="hello"; } if(counter== null){ counter=10; } return worker.doStuff(message,counter); } } {% endhighlight %} The source code of the service looks like this {% highlight java hl_lines="1" %} @Traced @ApplicationScoped public class WorkerService { @Inject @RestClient BackendService backendService; private static final Logger log = LoggerFactory.getLogger(WorkerService.class); public String doStuff(String message, int counter) { log.debug("Message: {}, counter: {}", message, counter); try { Thread.sleep((long) (Math.random() * 1000)); } catch (InterruptedException e) { } if (counter == 0) { log.debug("Counter is zero, do nothing", message, counter); return message; } message = message + " " + counter; counter--; if (System.getenv("BACKEND_SERVICE_URL") != null) { log.debug("URL for backend service is: " + System.getenv("BACKEND_SERVICE_URL")); message = backendService.getMessage(message, counter); } else { log.debug("Call doStuff recursively"); message = doStuff(message, counter); } return message; } } {% endhighlight %} The source code of the backend service looks like this: {% highlight java hl_lines="3" %} @RegisterRestClient(configKey = "backend-service") @RegisterProvider(LogClientRequestFilter.class) @Traced public interface BackendService { @Path("/message/{message}") @GET @Produces("text/plain") public String getMessage(@PathParam String message,@QueryParam Integer counter); } {% endhighlight %} To submit the tracing information, we need to provide the application, the Jaeger Endpoint (URL of the Jaeger server) and the name of the service in Jaeger. Environment variables are providing these values. The `application.properties` looks like this: {% highlight properties hl_lines="4 6" %} quarkus.log.level=INFO quarkus.log.category."de.consol".level=DEBUG quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t{id}) %s%e%n backend-service/mp-rest/url=${BACKEND_SERVICE_URL:http://localhost:8080} backend-service/mp-rest/scope=javax.inject.Singleton quarkus.jaeger.service-name=${JAEGER_SERVICE_NAME:default-tracing-test} quarkus.jaeger.sampler-param=1 {% endhighlight %} The OpenTracing annotations will take the tracing header parameter from request and add them to the outgoing request call. With this information, Jaeger can build a complete graph of the request, called application and Java methods. After the changes in the source code, the Jaeger graph looks like this: ![](jaeger_with_tracing.jpg) The graph for a single requests looks like this: ![](jaeger_with_tracing_spans.jpg) As you can see, the name of the spans has changed and also the number of spans (the number in the brackets). ## Summary At this point in the article, you should have a sound idea of how to add an application to a control plane. We went through a couple of steps to instrument your applications in Istio and to take full advantage of Jaeger. You can find more about the touched topic here: * [Quarkus](https://quarkus.io/) * [OpenTracing with Quarkus](https://quarkus.io/guides/opentracing) * [Kiali](https://kiali.io/) * [Jaeger](https://www.jaegertracing.io/) [Read more](https://omd.consol.de/blog/2020/08/07/how-to-add-an-application-to-a-red-hat-openshift-service-mesh/) ----------------------------------------------------------------------------------- title: "How to add an application to a Red Hat OpenShift Service Mesh" date: "2020-08-07" --------------------------------------------------------------------------------
During a discussion with a customer, we talked about which steps are necessary to add an application to a services mesh. Which should be no big deal. Unfortunately, there is not a simple guideline how to do that for the Red Hat OpenShift Service Mesh. Furthermore, I was not sure how the requests for the application would look like in Jaeger. To clarify these points, I created a small application. Which I then deployed on OpenShift and added it to a service mesh control plane. This is the documentation of the steps that I have done.
## The demo application The created demo application can call other applications and mocks internal processing. Hence, I'm fluent in Java, I created a Java REST-Service that takes a message and a counter. The response returns the message. The counter determines the number of recursive method calls. The environment variable `BACKEND_SERVICE_URL` points to a demo application instance that serves as a backend. If the environment variable is not defined, then a method is called recursively. The request flow of the application looks like this: ![sequence_request_call.png](sequence_request_call.png) The source code of the REST-API looks like this: {% highlight java %} @Path("/") public class TestResource { @Inject WorkerService worker; @GET @Path("message/{message}") @Produces(MediaType.TEXT_PLAIN) public String message(@PathParam String message, @QueryParam Integer counter) { if(message== null){ message="hello"; } if(counter== null){ counter=10; } return worker.doStuff(message,counter); } } {% endhighlight %} The source code of the service looks like this: {% highlight java %} @ApplicationScoped public class WorkerService { @Inject @RestClient BackendService backendService; private static final Logger log = LoggerFactory.getLogger(WorkerService.class); public String doStuff(String message, int counter) { log.debug("Message: {}, counter: {}", message, counter); try { Thread.sleep((long) (Math.random() * 1000)); } catch (InterruptedException e) { } if (counter == 0) { log.debug("Counter is zero, do nothing", message, counter); return message; } message = message + " " + counter; counter--; if (System.getenv("BACKEND_SERVICE_URL") != null) { log.debug("URL for backend service is: " + System.getenv("BACKEND_SERVICE_URL")); message = backendService.getMessage(message, counter); } else { log.debug("Call doStuff recursively"); message = doStuff(message, counter); } return message; } } {% endhighlight %} The source code of the backend service looks like this: {% highlight java %} @RegisterRestClient(configKey = "backend-service") @RegisterProvider(LogClientRequestFilter.class) public interface BackendService { @Path("/message/{message}") @GET @Produces("text/plain") public String getMessage(@PathParam String message,@QueryParam Integer counter); } {% endhighlight %} As you can see, it is pretty straight forward and involves no magic. It is standard Java code using Quarkus as an Eclipse MicroProfile framework. ## Current setup of the application The current setup of the application consists of two deployments. The first is the frontend, which takes the request and calls the backend application. This backend application is performing some calculations (calling recursively the function `doStuff`) and returning the result to the frontend. A route is exposing the frontend application. A typical request and response looks like this: {% highlight bash %} $ curl messager-entrypoint-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10 Hello World 10 9 8 7 6 5 4 3 2 1 {% endhighlight %} ## Add application to service mesh We are at a point where we have an existing application up and running in one project. The Service mesh control plane is installed in another project. Let's bring them together by creating a resource of the type ServiceMeshMemberRoll. It contains the name of the service mesh control plane project and the projects of the service mesh data plane. The data plane consists only of the project with the demo application. Our setup looks like this: {% highlight yaml hl_lines="5 8"%} apiVersion: maistra.io/v1 kind: ServiceMeshMemberRoll metadata: name: default namespace: olaf-smcp spec: members: - test-tracing {% endhighlight %} Line 5 contains the project for the control plane. The project of the data plane is defined in line 8. So far, so good. Let's call the demo application to test if everything still works. The result looks like this: {% highlight bash %} $ curl -v messager-entrypoint-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10 * Trying 10.4.10.20... * TCP_NODELAY set * Connected to messager-entrypoint-test-tracing.apps.ocp.acme.com (10.4.10.20) port 80 (#0) > GET /message/Hello%20World?counter=10 HTTP/1.1 > Host: messager-entrypoint-test-tracing.apps.ocp.acme.com > User-Agent: curl/7.58.0 > Accept: */* > * HTTP 1.0, assume close after body < HTTP/1.0 503 Service Unavailable < pragma: no-cache < cache-control: private, max-age=0, no-cache, no-store < content-type: text/html < .... {% endhighlight %} So, what happened? When adding a project to the data plane, then the Istio Operator adds networks rules to the project. One rule allows pods from the control plane to access the project, which makes perfect sense. How else should Istio instrument the pods, if it doesn't have access to them? Another rule defines that namespaces with the label `network.openshift.io/policy-group: ingress` can access pods with the label `maistra.io/expose-route: "true"`. An example of this is the namespace with the OpenShift Router or ingress pods (`openshift-ingress`). Our deployments of the demo application don't have this label. Hence, the route can't access the demo application. ## Add labels to deployment and pod for route To fix this, add the label `maistra.io/expose-route: "true"` to the deployments of the demo application. Do this with these commands: {% highlight bash %} $ oc patch deployment messager-backend -p '{"spec":{"template":{"metadata":{"labels":{"maistra.io/expose-route":"true"}}}}}' deployment.apps/messager-backend patched $ oc patch deployment messager-entrypoint -p '{"spec":{"template":{"metadata":{"labels":{"maistra.io/expose-route":"true"}}}}}' deployment.apps/messager-backend patched {% endhighlight %} If the Router/Ingress controllers of your OpenShift cluster are running on the host network, you need to add the label `network.openshift.io/policy-group: ingress` to the default project as well. A detailed explanation for this is [here](https://docs.openshift.com/container-platform/4.4/networking/configuring-networkpolicy.html). After this, our demo application can be called via the defined route. --- *Remark:* Labeling the pods with `maistra.io/expose-route: "true"` is an optional step. If it is not required to access applications via a route, then you don't need to label the pods. In this case, I recommend to remove the routes of the application from the project as well. --- ## Add annotation for sidecar auto injection to deployment Now we adjust the deployments, so that if a pod gets deployed then the Istio sidecar container is injected as well. Execute the following command to do this: {% highlight bash %} $ oc patch deployment messager-backend -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}' deployment.apps/messager-backend patched $ oc patch deployment messager-entrypoint -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"true"}}}}}' deployment.apps/messager-backend patched {% endhighlight %} This command will add an annotation `sidecar.istio.io/inject: "true"` to the pod template of the deployment. With this annotation, the service mesh control plane will inject Istio sidecar container in the pod. If you want, you can of course use `istioctl` to do this. After adding the annotation, a new deployment should be started. If this is not the case, delete the replica sets. After the restart of the pods they should contain the Istio sidecar. To verify this, get a list of all pods. {% highlight bash %} oc get po NAME READY STATUS RESTARTS AGE messager-backend-85cc85d87-hsx8d 2/2 Running 0 4m26s messager-entrypoint-7f75457fcc-f7qc7 2/2 Running 0 11s {% endhighlight %} The number of containers is increased from one (only our application container) to two (Istio sidecar plus application container). ## Create ingress gateway with new route It is good practice to use the Istio ingress gateway instead of calling the application pod via route. For this, we need to add two resources: 1. A gateway: This determines the ingress point. Whether it is using HTTP or HTTPS, the hostname certificates and so on ... 1. A virtual service which routes the incoming traffic to a service. The demo application looks like this: {% highlight yaml %} --- apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: tracingdemo-entrypoint namespace: test-tracing spec: selector: istio: ingressgateway servers: - hosts: - messager-entrypoint2-test-tracing.apps.ocp.acme.com port: name: http number: 80 protocol: HTTP --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: tracingdemo-entrypoint-vs namespace: test-tracing spec: gateways: - tracingdemo-entrypoint hosts: - messager-entrypoint2-test-tracing.apps.ocp.acme.com http: - route: - destination: host: messager-entrypoint port: number: 8080 {% endhighlight %} So lets add these two resources to the demo application namespaces: {% highlight bash %} oc create -f setup_gateway.yml {% endhighlight %} --- *Remark:* If your service mesh control plane hasn't enabled IOR, then you need to create a route that uses the defined hostname of the gateway and points to the ingress service of the service mesh control plane. --- So, finally, the demo application can be called with the bare minimum Istio setup. Let's try to call the application URL from the command line: {% highlight bash %} $ curl messager-entrypoint2-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10 Hello World 10 9 8 7 6 5 4 3 2 1 {% endhighlight %} This looks good. The request and response is as expected. ## Display calls in Kiali and Jaeger To view only a single request in Kiali is not feasible, let's put the previous command in a loop and run it: {% highlight bash %} $ while :; do curl messager-entrypoint2-test-tracing.apps.ocp.acme.com/message/Hello%20World?counter=10; sleep 1; done Hello World 10 9 8 7 6 5 4 3 2 1Hello World 10 9 8 7 6 5 4 3 2 1Hello World 10 9 8 7 6 5 4 3 2 1 {% endhighlight %} The graph for our demo in Kiali looks like this: ![kiali_overview_without_opentracing.jpg](kiali_overview_without_opentracing.jpg) This matches the request flow described above. This is the point to start exploring more features of Istio and extent the configuration bit by bit. Let's have a look at Jaeger to see how an initial application without tracing support looks like. For a single request the graph looks like this in Jaeger: ![jaeger_without_tracing.jpg](jaeger_without_tracing.jpg) The screenshot above shows that Jaeger gets only information about the request to the ingress gateway. It doesn't show any information about the frontend or backend application. This is not helpful for debugging an application call flow. Istio is generating HTTP headers for tracing. These headers are not processed by the Java application and used in calls of subsequent applications. So, Jaeger gets only information about the incoming request to the container and not how the applications are actually connected. ## How to fix the Jaeger issue? How can we fix this issue? After some try and error, I came up with this modification in my source code, which will add the tracing HTTP header to outgoing calls as well as sent tracing information about internal calls to Jaeger: The source code of the REST-API look like this: {% highlight java hl_lines="1" %} @Traced @Path("/") public class TestResource { @Inject WorkerService worker; @GET @Path("message/{message}") @Produces(MediaType.TEXT_PLAIN) public String message(@PathParam String message, @QueryParam Integer counter) { if(message== null){ message="hello"; } if(counter== null){ counter=10; } return worker.doStuff(message,counter); } } {% endhighlight %} The source code of the service looks like this {% highlight java hl_lines="1" %} @Traced @ApplicationScoped public class WorkerService { @Inject @RestClient BackendService backendService; private static final Logger log = LoggerFactory.getLogger(WorkerService.class); public String doStuff(String message, int counter) { log.debug("Message: {}, counter: {}", message, counter); try { Thread.sleep((long) (Math.random() * 1000)); } catch (InterruptedException e) { } if (counter == 0) { log.debug("Counter is zero, do nothing", message, counter); return message; } message = message + " " + counter; counter--; if (System.getenv("BACKEND_SERVICE_URL") != null) { log.debug("URL for backend service is: " + System.getenv("BACKEND_SERVICE_URL")); message = backendService.getMessage(message, counter); } else { log.debug("Call doStuff recursively"); message = doStuff(message, counter); } return message; } } {% endhighlight %} The source code of the backend service looks like this: {% highlight java hl_lines="3" %} @RegisterRestClient(configKey = "backend-service") @RegisterProvider(LogClientRequestFilter.class) @Traced public interface BackendService { @Path("/message/{message}") @GET @Produces("text/plain") public String getMessage(@PathParam String message,@QueryParam Integer counter); } {% endhighlight %} To submit the tracing information, we need to provide the application, the Jaeger Endpoint (URL of the Jaeger server) and the name of the service in Jaeger. Environment variables are providing these values. The `application.properties` looks like this: {% highlight properties hl_lines="4 6" %} quarkus.log.level=INFO quarkus.log.category."de.consol".level=DEBUG quarkus.log.console.format=%d{HH:mm:ss} %-5p traceId=%X{traceId}, spanId=%X{spanId}, sampled=%X{sampled} [%c{2.}] (%t{id}) %s%e%n backend-service/mp-rest/url=${BACKEND_SERVICE_URL:http://localhost:8080} backend-service/mp-rest/scope=javax.inject.Singleton quarkus.jaeger.service-name=${JAEGER_SERVICE_NAME:default-tracing-test} quarkus.jaeger.sampler-param=1 {% endhighlight %} The OpenTracing annotations will take the tracing header parameter from request and add them to the outgoing request call. With this information, Jaeger can build a complete graph of the request, called application and Java methods. After the changes in the source code, the Jaeger graph looks like this: ![jaeger_with_tracing.jpg](jaeger_with_tracing.jpg) The graph for a single requests looks like this: ![jaeger_with_tracing_spans.jpg](jaeger_with_tracing_spans.jpg) As you can see, the name of the spans has changed and also the number of spans (the number in the brackets). ## Summary At this point in the article, you should have a sound idea of how to add an application to a control plane. We went through a couple of steps to instrument your applications in Istio and to take full advantage of Jaeger. You can find more about the touched topic here: * [Quarkus](https://quarkus.io/) * [OpenTracing with Quarkus](https://quarkus.io/guides/opentracing) * [Kiali](https://kiali.io/) * [Jaeger](https://www.jaegertracing.io/) [Read more](https://omd.consol.de/blog/2020/08/07/how-to-add-an-application-to-a-red-hat-openshift-service-mesh/) ----------------------------------------------------------------------------------- title: "Unofficial guideline to get the latest and greatest version of Kiali in OpenShift" date: "2020-05-18" --------------------------------------------------------------------------------
During this year's Red Hat Summit I had the chance to get a glimpse of the latest version of Kiali. This version had some nice features, like the traffic flow of the application graph during a time period (Graph replay). It also contains wizards to create destination rules and virtual services. This demo has struck my curiosity to get the hands on this Kiali version. One obstacle for me was that my Kiali is running in Red Hat OpenShift Service Mesh and is controlled by the Kiali operator. Currently, it is using version 1.12. The version that I wanted to try was the latest release version (1.17). The Red Hat OpenShift Service Mesh does not support this version. This article describes what we need to do in order to replace the Kiali version of an Red Hat OpenShift Service Mesh with the latest version of Kiali.
--- *Disclaimer: The following changes of the Red Hat OpenShift Service Mesh are not supported by Red Hat. I cannot guarantee that the changes won't break the Red Hat OpenShift Service Mesh or your application. You do this at your own risk!* --- Let's start with a short detour: How does the Kiali operator determines which versions of Kiali are supported in Red Hat OpenShift Service Mesh? If we look into the definition of the `ServiceMeshControlPlane`, it is possible to provide an image name and container registry. With this, we are able to change the source location of the Kiali image. This value is not helpful to get the latest version of Kiali in our service mesh since we cannot define the tag of the Kiali image that we would like to use. If we now look at the Kiali resource definition, we will notice that this contains a version key. This version key is coupled with `KIALI_IMAGE_*` environment variables of the Kiali operator which determines the supported versions. To get these supported versions, we can use this command: ```bash $ oc set env po kiali-operator-5fd5c849b9-zkl49 -n openshift-operators --list|grep -i kiali_ KIALI_IMAGE_default=registry.redhat.io/openshift-service-mesh/kiali-rhel7@sha256:76667b3532df11a511b03c4efa165723cff48aa5fb2e56a2ceb693c02a6bce7a KIALI_IMAGE_v1_0=registry.redhat.io/openshift-service-mesh/kiali-rhel7@sha256:76667b3532df11a511b03c4efa165723cff48aa5fb2e56a2ceb693c02a6bce7a KIALI_IMAGE_v1_12=registry.redhat.io/openshift-service-mesh/kiali-rhel7@sha256:e1fb3df10a7f7862e8549ad29e4dad97b22719896c10fe5109cbfb3b98f56900 ``` Therefore, we can use Kiali version v1.0 and v1.12. In order to use the latest version of Kiali, we need to add an environment variable with the desired version in the Operator and change the Kiali resource to use it. However, as soon as the Operator is updated, it is very likely that our environment variable will be removed. Let us use a different approach: Update the version in the Kiali deployment which is controlled by the Kiali Operator. The advantage is that we only need to modify one resource and not two or more. Furthermore, the change is very easy and will only effect one namespace. The drawback is that the Kiali operator will overwrite the change in the deployment, in case a new version of the Kiali operator has been installed or the Kiali operator resource has been modified. So let's change the Kiali version used by patching the desired Kiali deployment. For this we execute the following command: ```bash $ oc patch deployment kiali --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"quay.io/kiali/kiali:v1.17"}]' -n ``` --- *Hint:* You can replace the version with a later version, if you like. --- After this, the Kiali pod should be restarted. If this is successful and we open the Kiali application in a browser, we should see in the `About` the following dialog: ![](screenshot_kiali_about.jpg) ![](screenshot_kiali_about_dialog.jpg) --- *Hint:* You might need to delete the browser cache in order to get the desired version running in your browser, because the css and JavaScript files may be cached. --- So with this we have the latest and greatest version of Kiali running in our Red Hat OpenShift Service Mesh. So far, I have not encountered any problem when using the latest version of Kiali in my test environment. One last remark: The feature that I liked most in Kiali 1.17 is the Graph replay feature. More information about this can be found here: - [Video of Kiali and Jaeger Sprint #34 Demo - Service Mesh observability](https://youtu.be/04fGMBjHZ68?t=365) [Read more](https://omd.consol.de/blog/2020/05/18/unofficial-guideline-to-get-the-latest-and-greatest-version-of-kiali-in-openshift/) ----------------------------------------------------------------------------------- title: "Unofficial guideline to get the latest and greatest version of Kiali in OpenShift" date: "2020-05-18" --------------------------------------------------------------------------------
During this year's Red Hat Summit I had the chance to get a glimpse of the latest version of Kiali. This version had some nice features, like the traffic flow of the application graph during a time period (Graph replay). It also contains wizards to create destination rules and virtual services. This demo has struck my curiosity to get the hands on this Kiali version. One obstacle for me was that my Kiali is running in Red Hat OpenShift Service Mesh and is controlled by the Kiali operator. Currently, it is using version 1.12. The version that I wanted to try was the latest release version (1.17). The Red Hat OpenShift Service Mesh does not support this version. This article describes what we need to do in order to replace the Kiali version of an Red Hat OpenShift Service Mesh with the latest version of Kiali.
--- *Disclaimer: The following changes of the Red Hat OpenShift Service Mesh are not supported by Red Hat. I cannot guarantee that the changes won't break the Red Hat OpenShift Service Mesh or your application. You do this at your own risk!* --- Let's start with a short detour: How does the Kiali operator determines which versions of Kiali are supported in Red Hat OpenShift Service Mesh? If we look into the definition of the `ServiceMeshControlPlane`, it is possible to provide an image name and container registry. With this, we are able to change the source location of the Kiali image. This value is not helpful to get the latest version of Kiali in our service mesh since we cannot define the tag of the Kiali image that we would like to use. If we now look at the Kiali resource definition, we will notice that this contains a version key. This version key is coupled with `KIALI_IMAGE_*` environment variables of the Kiali operator which determines the supported versions. To get these supported versions, we can use this command: {% highlight bash %} $ oc set env po kiali-operator-5fd5c849b9-zkl49 -n openshift-operators --list|grep -i kiali_ KIALI_IMAGE_default=registry.redhat.io/openshift-service-mesh/kiali-rhel7@sha256:76667b3532df11a511b03c4efa165723cff48aa5fb2e56a2ceb693c02a6bce7a KIALI_IMAGE_v1_0=registry.redhat.io/openshift-service-mesh/kiali-rhel7@sha256:76667b3532df11a511b03c4efa165723cff48aa5fb2e56a2ceb693c02a6bce7a KIALI_IMAGE_v1_12=registry.redhat.io/openshift-service-mesh/kiali-rhel7@sha256:e1fb3df10a7f7862e8549ad29e4dad97b22719896c10fe5109cbfb3b98f56900 {% endhighlight %} Therefore, we can use Kiali version v1.0 and v1.12. In order to use the latest version of Kiali, we need to add an environment variable with the desired version in the Operator and change the Kiali resource to use it. However, as soon as the Operator is updated, it is very likely that our environment variable will be removed. Let us use a different approach: Update the version in the Kiali deployment which is controlled by the Kiali Operator. The advantage is that we only need to modify one resource and not two or more. Furthermore, the change is very easy and will only effect one namespace. The drawback is that the Kiali operator will overwrite the change in the deployment, in case a new version of the Kiali operator has been installed or the Kiali operator resource has been modified. So let's change the Kiali version used by patching the desired Kiali deployment. For this we execute the following command: {% highlight bash %} $ oc patch deployment kiali --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"quay.io/kiali/kiali:v1.17"}]' -n {% endhighlight %} --- *Hint:* You can replace the version with a later version, if you like. --- After this, the Kiali pod should be restarted. If this is successful and we open the Kiali application in a browser, we should see in the `About` the following dialog: ![screenshot_kiali_about.jpg](screenshot_kiali_about.jpg) ![screenshot_kiali_about_dialog.jpg](screenshot_kiali_about_dialog.jpg) --- *Hint:* You might need to delete the browser cache in order to get the desired version running in your browser, because the css and JavaScript files may be cached. --- So with this we have the latest and greatest version of Kiali running in our Red Hat OpenShift Service Mesh. So far, I have not encountered any problem when using the latest version of Kiali in my test environment. One last remark: The feature that I liked most in Kiali 1.17 is the Graph replay feature. More information about this can be found here: - [Kiali sprint 34 overview; Search for 'Graph replay'](https://medium.com/kialiproject/kiali-sprint-34-overview-86027b13b27d) - [Video of Kiali and Jaeger Sprint #34 Demo - Service Mesh observability](https://youtu.be/04fGMBjHZ68?t=365) [Read more](https://omd.consol.de/blog/2020/05/18/unofficial-guideline-to-get-the-latest-and-greatest-version-of-kiali-in-openshift/) ----------------------------------------------------------------------------------- title: "Debugging Istio" date: "2020-05-07" --------------------------------------------------------------------------------
In the article, I'm going to describe what we can do, if we configured our application to use Istio, but it is not working like intended. Originally, I wanted to give a detailed description what problems I encountered during the creation of my webinar and how I fixed them. However, I came to a point where this would be a very long one. I hope that you don't mind that I shortened it and just describe which tools are available to `debug` the Istio configuration. In my previous [article](/blog/2020/05/07/simple-example-how-to-use-istio-and-keycloak/) I described how to configure Keycloak for my webinar. So without further ado, let's start.
## Certificate problem After deploying my application and the Istio configuration, everything seemed to run. There were no errors in the logs of my pods. I ran a small script to call my demo application in a loop with the access_token from Keycloak. Instead of successful response, I got the following error: ```bash Origin authentication failed. ``` The reason for that error is the policy that enforces a JWT authentication and which looks like this: ```yaml kind: Policy metadata: name: customerjwt namespace: olaf-customer spec: targets: - name: customer - name: preference - name: recommendation peers: - mtls: {} peerIsOptional: ~ origins: - jwt: audiences: - customer issuer: 'https://skeycloak-olaf-sso.apps.acme.de/auth/realms/customer' jwksUri: 'https://skeycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs' principalBinding: USE_ORIGIN ``` After verifying three times that my access token and the Keycloak url were correct, it dawned slowly on me, that there might be a problem with my Istio infrastructure configuration. After checking the Istio Ingress Gateway and the istio-proxy in the customer pod, I found these log entries in the discovery container of the Istio Pilot pod: ```bash 2020-02-17T12:57:34.675759Z error model Failed to fetch public key from "https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs": Get https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs: x509: certificate signed by unknown authority 2020-02-17T12:57:34.675778Z warn Failed to fetch jwt public key from "https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs" ``` So why is the error in the Istio pilot pod and not in the Istio Ingress Gateway, for example? The reason is that the JWT is validated by the public key that is provided by the jksURI. It is sufficient to get this key before the first request. After this, Istio can cache the public key and save network calls. All this is done in Pilot, which then "caches" the result values in the Envoy configuration of the Istio-Proxy container. A stored configuration looks like this: ``` json "http_filters": [ { "name": "jwt-auth", "config": { "allow_missing_or_failed": true, "rules": [ { "forward": true, "audiences": [ "customer1" ], "issuer": "https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer", "local_jwks": { "inline_string": "{\"keys\":[{\"kid\":\"Ek-wxUoZGSSp5eePlJBMuAGzyolk6aI5HSyB0gceFZY\",\"kty\":\"RSA\",\"alg\":\"RS256\",\"use\":\"sig\",\"n\":\"jXNMZfp-AI2LpnTuSAD6mG5Kkv390YAqJF0Y3OOeDqAYDI-t4glGHku_nVPqdn2_LNNxumMPY94ytsoWYJDiHlIvmmnQukYL0kVE5DmdbYhiJ0djh0K6E4WePcmKhBugxukY1N9zmQZjvly0EenyTrZFnLLdsZpyupCzdhdL6xZrxXX3Gjr1JlC1-2BgvHkyg_a3tPHmcL559KLuJk-darJentwJEi3B5RYIyYLKmX-1cC3GUeRSnhnaRlSUUnY5XKFpMt2W636XOmRY77HL90PyoIU-7FjnIV0PenWlxdq_nGJ5YBZqGKJTJmKcquhO4WkSn0PPZx_KRiwjub-MgQ\",\"e\":\"AQAB\"}]}" }, "forward_payload_header": "istio-sec-544a41081254dc82aed9a0d81261fdc6f8cdd75d" } ] } }, ``` So we know what is causing the problem. Let's fix it! There are two options: 1. Instead of `jwksUri`, in the Policy we could use an attribute called `jwks`. You can get the content from the `jwksUri`. Just paste the value in the policy with the correct indentation. An example looks like this: ``` yaml jwks: {"keys":[{"kid":"WyXEwTpUURQsNanFbktq-KIHRM1io_0","kty":"RSA","alg":"RS256","use":"sig", "n":"yQRHyWzFqXZ5DegUpf_9QA2kr5Ys2N6XXUhNdAHTh_CjQT_YfhvFKfQOYdbQH4pzLPSlBoOaADVqIgE55qrvoysft_ KgAvUr2W7b4vWy13PKzBdcec6EDxxod3MmphT0K_9bit-KPMXEeLKlKfRVYAYEdM8FQJpUuO4HQv-xCP-KhYcR_Yvipxhpt hDkQqmZiAn6b4QffK-9brDlnhgCsQjP92GMPUU9AZ-","e":"AQAB"}]} ``` 1. Add the signer certificates in the pilot pod, which is a bit more complex. 1. (Optional) Download the wildcard certificate of your OpenShift cluster with the following command: ```bash openssl s_client \ -showcerts \ -servername console-openshift-console.apps.acme.de \ -connect console-openshift-console.apps.acme.de:443 /dev/null \ | openssl x509 -outform PEM >openshift-wildcard.pem ``` 1. Create a secret with the certificate. The filename in the secret has to be `extra.pem`: ```bash oc create secret generic openshift-wildcard \ --from-file=extra.pem=openshift-wildcard.pem \ -n olaf ``` 1. The last step is to use the volume discovery container of the Istio Pilot pod. The easiest way to do that is with the command: ```bash oc set volumes deployment/istio-pilot \ --add \ --name=extracacerts \ --mount-path=/cacerts \ --secret-name=openshift-wildcard \ --containers=discovery ``` One remark on the second solution: When I started writing this article, OpenShift Istio (Maistra 1.0.x) didn't support addition CA certificates. With Openshift Istio (Maistra 1.1.x) it is possible to define addition CA certificates in the ServiceMeshControlPlane before installing OpenShift Istio. At the time being, it is not possible to add additional CA certificates after the installation out of the box. So for now after the installation of OpenShift Istio, you need to choose between option 1 and 2. Also, keep in mind that after updates of the Istio operator you might need to mount the secret again in the deployment because Istio Operator might overwrite it. ## How to analyze problems of the Istio configuration The previous problem that I encountered was a rather specific one with respect to the Istio version used and the OpenShift operator. The other techniques that I will describe are more generic and not so much specific for the Istio and OpenShift environment. ### Get the configuration of the Envoy proxy of a pod During the creation of my sample application, I encountered a scenario were I needed to verify the configuration of an Envoy proxy in a pod. To a certain degree, the Istio configuration will end up in an Envoy proxy as its configuration. A look at the Envoy configuration helps us to verify, if our Istio configuration has been translated to to the desired Envoy configuration. Furthermore, by looking at the Envoy configuration we gain a deeper understanding of the Istio internal behavior. Okay, enough talk, how can we get the configuration of an Istio-proxy (Envoy proxy) for a pod? The answer is very simple. By using this command: ```bash oc exec -c istio-proxy -- curl -kv localhost:15000/config_dump> _Envoy_config.json ``` If we take a look at the written file, we will get e.g. the jwk config (like seen above), the retries for endpoints, the routing and many more. At this point I have to admit that I could not create the Envoy configuration from scratch. However I find the configuration easy enough to understand when focusing on certain aspects. ### Increase the logging level of the Envoy proxy Besides, the configuration there is a nice feature that I started to appreciate very much: To increase the logging for certain features of the Envoy proxy without increase the log output for other components. Let's have a look which components are available: |admin|aws|assert|backtrace|client|config|connection|conn_handler| |dubbo|file|filter|forward_proxy|grpc|hc|health_checker|http| |http2|hystrix|init|io|jwt|kafka|lua|main| |misc|mongo|quic|pool|rbac|redis|router|runtime| |stats|secret|tap|testing|thrift|tracing|upstream|udp| |wasm||||||| There are quite a lot of components from which we can choose, if we want to increase the log level. To change the level for only the RBAC component, we can execute the following command: ```bash oc exec -it -c istio-proxy -- sh -c 'curl -k -X POST localhost:15000/logging?rbac=debug' ``` We should get a response like this from the pod: ```bash active loggers: admin: warning aws: warning assert: warning backtrace: warning client: warning config: warning ... pool: warning rbac: debug redis: warning router: warning runtime: warning stats: warning secret: warning tap: warning testing: warning thrift: warning tracing: warning upstream: warning udp: warning wasm: warning ``` If we really wanted, we could set the log level for all components of envy proxy to `debug` with this command: ```bash oc exec -it -c istio-proxy -- sh -c 'curl -k -X POST localhost:15000/logging?level=debug' ``` The output should look like this: ```bash active loggers: admin: debug aws: debug assert: debug backtrace: debug client: debug config: debug ... pool: debug rbac: debug redis: debug router: debug runtime: debug stats: debug secret: debug tap: debug testing: debug thrift: debug tracing: debug upstream: debug udp: debug wasm: debug ``` Hint: Be careful when changing the log level of all components for an Envoy proxy, because this is generating quite a lot of log entries (even in a test environment). The Envoy proxy offers a minimal UI. We can have a look at the UI, if we do a port-forward to our local computer from an istio-proxy with either: ```bash oc port-forward 15000 ``` or ```bash istioctl dashboard envoy ``` If we open this web-site [`https://localhost:15000`](https://localhost:15000) in our browser we see the following website: We get quite some options here, what we can see or even change in the Envoy proxy. ### Increase the logging of an Istio component So far, now I have showed you, what we can do to get more information from the Envoy proxy. I have not touched the control plane of Istio so far. It took me some time to find it, however, each Istio control plane component has a built in GUI (at least for Mixer, Pilot, Citadel, and Galley that is). We can have a look at it by forwarding the port of the Istio pod to our local computer. Again, we do a port-forward this time but instead of the port 15000 the port is now 9876: ```bash oc port-forward 9876 ``` For the Istio pilot it looks like this: ```bash oc port-forward istio-pilot-775d8bfc6b-zb25d 9876 ``` If we open this website [`http://localhost:9876/scopez/`](http://localhost:9876/scopez/) in our browser, we see the following GUI: ![](screenshot_pilot_gui.jpg) So, this time we have a GUI to change the logging setting of a component, which is quite nice. Also, we can get more information about the memory usage, the metrics and much more. ## Check the Istio configuration Besides changing the logging level, Istio offers another powerful tool to see if something is wrong. WE can use the command line or Kiali to verify our Istio configuration. It might not be perfect, but it still can save us from a lot of trouble. ### Option a): the command line tool #### Get proxy configuration of a specific pod As I have pointed out above, we can get the complete Envoy configuration using `oc exec`. There is another way we can use `istioctl proxy-config`. This command allows us to get subset of the proxy configuration like `endpoints` or `clusters` for example. To get a list of all end points for a proxy, we can run the following command: ```bash istioctl proxy-config endpoints ``` This can be quite useful, as it shows us which endpoints are available for the pod and whether they are healthy. For one of my pods, it looks like this: ```bash $ istioctl proxy-config endpoints customer-56d65d78d9-n5sq9 ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.10.10.39:8060 HEALTHY OK outbound|8060||istio-citadel.olaf.svc.cluster.local 10.10.10.39:15014 HEALTHY OK outbound|15014||istio-citadel.olaf.svc.cluster.local 10.10.10.40:3001 HEALTHY OK outbound|9090||prometheus.olaf.svc.cluster.local 10.10.10.41:8443 HEALTHY OK outbound|443||istio-galley.olaf.svc.cluster.local 10.10.10.41:9901 HEALTHY OK outbound|9901||istio-galley.olaf.svc.cluster.local 10.10.10.41:15014 HEALTHY OK outbound|15014||istio-galley.olaf.svc.cluster.local 10.10.10.42:9091 HEALTHY OK outbound|9091||istio-policy.olaf.svc.cluster.local 10.10.10.42:15004 HEALTHY OK outbound|15004||istio-policy.olaf.svc.cluster.local 10.10.10.42:15014 HEALTHY OK outbound|15014||istio-policy.olaf.svc.cluster.local 10.10.10.43:8080 HEALTHY OK outbound|80||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.43:8443 HEALTHY OK outbound|443||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.43:15020 HEALTHY OK outbound|15020||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.43:15443 HEALTHY OK outbound|15443||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.45:20001 HEALTHY OK outbound|20001||kiali.olaf.svc.cluster.local 10.10.10.51:8080 HEALTHY OK outbound|8080|version-v1|customer.olaf-customer.svc.cluster.local 10.10.10.51:8080 HEALTHY OK outbound|8080||customer.olaf-customer.svc.cluster.local 10.10.10.53:8080 HEALTHY OK outbound|8080|version-v1|recommendation.olaf-customer.svc.cluster.local 10.10.10.53:8080 HEALTHY OK outbound|8080||recommendation.olaf-customer.svc.cluster.local 10.10.11.100:8080 HEALTHY OK outbound|8080|version-v2|recommendation.olaf-customer.svc.cluster.local 10.10.11.100:8080 HEALTHY OK outbound|8080||recommendation.olaf-customer.svc.cluster.local 10.10.11.35:8443 HEALTHY OK outbound|443||jaeger-query.olaf.svc.cluster.local 10.10.11.35:9411 HEALTHY OK outbound|9411||jaeger-collector.olaf.svc.cluster.local 10.10.11.35:9411 HEALTHY OK outbound|9411||zipkin.olaf.svc.cluster.local 10.10.11.35:14250 HEALTHY OK outbound|14250||jaeger-collector.olaf.svc.cluster.local 10.10.11.35:14267 HEALTHY OK outbound|14267||jaeger-collector.olaf.svc.cluster.local 10.10.11.35:14268 HEALTHY OK outbound|14268||jaeger-collector.olaf.svc.cluster.local 10.10.11.94:9091 HEALTHY OK outbound|9091||istio-telemetry.olaf.svc.cluster.local 10.10.11.94:15004 HEALTHY OK outbound|15004||istio-telemetry.olaf.svc.cluster.local 10.10.11.94:15014 HEALTHY OK outbound|15014||istio-telemetry.olaf.svc.cluster.local 10.10.11.94:42422 HEALTHY OK outbound|42422||istio-telemetry.olaf.svc.cluster.local 10.10.11.95:8080 HEALTHY OK outbound|8080||istio-pilot.olaf.svc.cluster.local 10.10.11.95:15010 HEALTHY OK outbound|15010||istio-pilot.olaf.svc.cluster.local 10.10.11.95:15011 HEALTHY OK outbound|15011||istio-pilot.olaf.svc.cluster.local 10.10.11.95:15014 HEALTHY OK outbound|15014||istio-pilot.olaf.svc.cluster.local 10.10.11.96:8080 HEALTHY OK outbound|80||istio-egressgateway.olaf.svc.cluster.local 10.10.11.96:8443 HEALTHY OK outbound|443||istio-egressgateway.olaf.svc.cluster.local 10.10.11.96:15443 HEALTHY OK outbound|15443||istio-egressgateway.olaf.svc.cluster.local 10.10.11.98:8443 HEALTHY OK outbound|443||istio-sidecar-injector.olaf.svc.cluster.local 10.10.11.98:15014 HEALTHY OK outbound|15014||istio-sidecar-injector.olaf.svc.cluster.local 10.10.11.99:3001 HEALTHY OK outbound|3000||grafana.olaf.svc.cluster.local 10.131.1.60:8080 HEALTHY OK outbound|8080|version-v1|preference.olaf-customer.svc.cluster.local 10.131.1.60:8080 HEALTHY OK outbound|8080||preference.olaf-customer.svc.cluster.local 127.0.0.1:8080 HEALTHY OK inbound|8080|http-customer|customer.olaf-customer.svc.cluster.local 127.0.0.1:15000 HEALTHY OK prometheus_stats 127.0.0.1:15020 HEALTHY OK inbound|15020|mgmt-15020|mgmtCluster 172.30.1.216:9411 HEALTHY OK zipkin 172.30.174.171:15010 HEALTHY OK xds-grpc ``` To get a list of all available sub-command, run: ``` bash istioctl proxy-config --help ``` #### Proxy status in istio Another useful command is `istioctl proxy-status`. This command returns the sync status of the pod with respect to the central configuration of Istio (pilot). If a problem with the proxy configuration occurs, it is a good starting point to check whether the proxies are in sync with pilot. For my cluster, the result of the proxy status looks like this: ```bash $ istioctl proxy-status -i olaf NAME CDS LDS EDS RDS PILOT VERSION customer-56d65d78d9-n5sq9.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 istio-egressgateway-85cd64f885-q99ml.olaf SYNCED SYNCED SYNCED NOT SENT istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 istio-ingressgateway-bf6644dd5-bdjwc.olaf SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 preference-v1-6f449bc44-82l4d.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 recommendation-v1-56ddb779f-lw9hb.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 recommendation-v2-5d85578cbf-49tx4.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 ``` This looks good so far. Don't worry about the value `NOT SENT` for the egress proxy, because there are no services defined for the egress, so no need to sync anything. The meaning of the columns CDS, LDS, EDS and RDS are here: > - CDS: [Envoy Cluster Discovery Service](https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cds.html). Envoy defines a “cluster” as a group of hosts that accept traffic for a given endpoint or service. Don’t confuse an Envoy cluster with a Kubernetes cluster – in a Kubernetes cluster, an Envoy “cluster” will typically, though not always, be a Kubernetes deployment or other replica set. > - LDS: [Envoy Listener Discovery Service](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/lds). Envoy defines a “listener” for each configured host:port endpoint. A Kubernetes cluster will typically have an Envoy listener for each target service port in an Envoy cluster. > - EDS: Envoy Endpoint Discovery Service. Envoy defines an “endpoint” as a member of an Envoy cluster, to which it can connect for a service. In a Kubernetes cluster, an Envoy endpoint would be a pod backing the destination service. > - RDS: [Envoy Route Discovery Service](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/rds.html). Envoy defines a “route” as the set of Envoy rules that match a service or virtual host to an Envoy “cluster.” (Quote from [How to make Istio work with your app](https://securityboulevard.com/2019/11/how-to-make-istio-work-with-your-apps)) #### False positives If we run the command `istioctl experimental analyze -k` it will analyze the Istio configuration for our cluster. In my case, I use OpenShift 4.3. This will return an error because the injection of an Istio sidecar container can only be done in the deployment etc. and it is not possible base on setting in the namespace. So do not worry, if you see this error message on an OpenShift cluster: ```bash istioctl experimental analyze -k Warn [IST0102] (Namespace olaf-customer) The namespace is not enabled for Istio injection. Run 'kubectl label namespace olaf-customer istio-injection=enabled' to enable it, or 'kubectl label namespace olaf-customer istio-injection=disabled' to explicitly mark it as not needing injection ``` ### Option b): Kiali Here is an example how Kiali is verifying a Istio configuration. When setting up my demo application, I wanted to make sure that the traffic between the pods is using mTLS by default. To ensure this, I created a default Destination Route that enables mTLS. When I look at the `Istio Config`, I notice the following error. ![](screenshot_kiali_istio_config_overview.jpg) When I opened the YAML file, I got a detail description what went wrong. ![](screenshot_kiali_istio_config__error_detail.jpg) In my case, I forgot to create a global ServiceMeshPolicy. Please note, that this is only a fraction of the tests that Kiali does; but to show all of them would go beyond the scope of the article. A complete list of the validation can be found [here](https://kiali.io/documentation/validations/). In general, I like the combination of Kiali and Jaeger a lot to debug configuration issues. We either get a visual feedback that something is not working like intended (i.e. mTLS is not switched on) or we get a validation error. ## Summary and further reading material When I started with Istio, I was wondering what the purpose of the `istioctl` was, because we can setup the istio configuration by only using `oc` or `kubectl`. After some playing around with applications that were using Istio features, I notice that the strength of `istioctl` is analysis of the Istio runtime environment. Istioctl together with Kiali and Jaeger, it forms a very good team if we need to hunt down errors and this is something, that we all know, happens often sooner than we like. - [More information on the logging of the Envoy proxy](https://www.envoyproxy.io/docs/envoy/latest/operations/admin#post--logging) - [More info about the REST admin API of the Envoy proxy](https://www.envoyproxy.io/docs/envoy/latest/operations/admin) - [Istio documentation about the Web-GUI for the Istio Components](https://istio.io/docs/ops/diagnostic-tools/controlz/) - [How to make Istio work with your app](https://securityboulevard.com/2019/11/how-to-make-istio-work-with-your-apps) [Read more](https://omd.consol.de/blog/2020/05/07/debugging-istio/) ----------------------------------------------------------------------------------- title: "Debugging Istio" date: "2020-05-07" --------------------------------------------------------------------------------
In the article, I'm going to describe what we can do, if we configured our application to use Istio, but it is not working like intended. Originally, I wanted to give a detailed description what problems I encountered during the creation of my webinar and how I fixed them. However, I came to a point where this would be a very long one. I hope that you don't mind that I shortened it and just describe which tools are available to `debug` the Istio configuration. In my previous [article](/blog/2020/05/07/simple-example-how-to-use-istio-and-keycloak/) I described how to configure Keycloak for my webinar. So without further ado, let's start.
## Certificate problem After deploying my application and the Istio configuration, everything seemed to run. There were no errors in the logs of my pods. I ran a small script to call my demo application in a loop with the access_token from Keycloak. Instead of successful response, I got the following error: ```bash Origin authentication failed. ``` The reason for that error is the policy that enforces a JWT authentication and which looks like this: ```yaml - apiVersion: authentication.istio.io/v1alpha1 kind: Policy metadata: name: customerjwt namespace: olaf-customer spec: targets: - name: customer - name: preference - name: recommendation peers: - mtls: {} peerIsOptional: ~ origins: - jwt: audiences: - customer issuer: 'https://skeycloak-olaf-sso.apps.acme.de/auth/realms/customer' jwksUri: 'https://skeycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs' principalBinding: USE_ORIGIN ``` After verifying three times that my access token and the Keycloak url were correct, it dawned slowly on me, that there might be a problem with my Istio infrastructure configuration. After checking the Istio Ingress Gateway and the istio-proxy in the customer pod, I found these log entries in the discovery container of the Istio Pilot pod: ```bash 2020-02-17T12:57:34.675759Z error model Failed to fetch public key from "https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs": Get https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs: x509: certificate signed by unknown authority 2020-02-17T12:57:34.675778Z warn Failed to fetch jwt public key from "https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer/protocol/openid-connect/certs" ``` So why is the error in the Istio pilot pod and not in the Istio Ingress Gateway, for example? The reason is that the JWT is validated by the public key that is provided by the jksURI. It is sufficient to get this key before the first request. After this, Istio can cache the public key and save network calls. All this is done in Pilot, which then "caches" the result values in the Envoy configuration of the Istio-Proxy container. A stored configuration looks like this: ``` json "http_filters": [ { "name": "jwt-auth", "config": { "allow_missing_or_failed": true, "rules": [ { "forward": true, "audiences": [ "customer1" ], "issuer": "https://keycloak-olaf-sso.apps.acme.de/auth/realms/customer", "local_jwks": { "inline_string": "{\"keys\":[{\"kid\":\"Ek-wxUoZGSSp5eePlJBMuAGzyolk6aI5HSyB0gceFZY\",\"kty\":\"RSA\",\"alg\":\"RS256\",\"use\":\"sig\",\"n\":\"jXNMZfp-AI2LpnTuSAD6mG5Kkv390YAqJF0Y3OOeDqAYDI-t4glGHku_nVPqdn2_LNNxumMPY94ytsoWYJDiHlIvmmnQukYL0kVE5DmdbYhiJ0djh0K6E4WePcmKhBugxukY1N9zmQZjvly0EenyTrZFnLLdsZpyupCzdhdL6xZrxXX3Gjr1JlC1-2BgvHkyg_a3tPHmcL559KLuJk-darJentwJEi3B5RYIyYLKmX-1cC3GUeRSnhnaRlSUUnY5XKFpMt2W636XOmRY77HL90PyoIU-7FjnIV0PenWlxdq_nGJ5YBZqGKJTJmKcquhO4WkSn0PPZx_KRiwjub-MgQ\",\"e\":\"AQAB\"}]}" }, "forward_payload_header": "istio-sec-544a41081254dc82aed9a0d81261fdc6f8cdd75d" } ] } }, ``` So we know what is causing the problem. Let's fix it! There are two options: 1. Instead of `jwksUri`, in the Policy we could use an attribute called `jwks`. You can get the content from the `jwksUri`. Just paste the value in the policy with the correct indentation. An example looks like this: ``` yaml jwks: {"keys":[{"kid":"WyXEwTpUURQsNanFbktq-KIHRM1io_0","kty":"RSA","alg":"RS256","use":"sig", "n":"yQRHyWzFqXZ5DegUpf_9QA2kr5Ys2N6XXUhNdAHTh_CjQT_YfhvFKfQOYdbQH4pzLPSlBoOaADVqIgE55qrvoysft_ KgAvUr2W7b4vWy13PKzBdcec6EDxxod3MmphT0K_9bit-KPMXEeLKlKfRVYAYEdM8FQJpUuO4HQv-xCP-KhYcR_Yvipxhpt hDkQqmZiAn6b4QffK-9brDlnhgCsQjP92GMPUU9AZ-","e":"AQAB"}]} ``` 1. Add the signer certificates in the pilot pod, which is a bit more complex. 1. (Optional) Download the wildcard certificate of your OpenShift cluster with the following command: ```bash openssl s_client \ -showcerts \ -servername console-openshift-console.apps.acme.de \ -connect console-openshift-console.apps.acme.de:443 /dev/null \ | openssl x509 -outform PEM >openshift-wildcard.pem ``` 1. Create a secret with the certificate. The filename in the secret has to be `extra.pem`: ```bash oc create secret generic openshift-wildcard \ --from-file=extra.pem=openshift-wildcard.pem \ -n olaf ``` 1. The last step is to use the volume discovery container of the Istio Pilot pod. The easiest way to do that is with the command: ```bash oc set volumes deployment/istio-pilot \ --add \ --name=extracacerts \ --mount-path=/cacerts \ --secret-name=openshift-wildcard \ --containers=discovery ``` One remark on the second solution: When I started writing this article, OpenShift Istio (Maistra 1.0.x) didn't support addition CA certificates. With Openshift Istio (Maistra 1.1.x) it is possible to define addition CA certificates in the ServiceMeshControlPlane before installing OpenShift Istio. At the time being, it is not possible to add additional CA certificates after the installation out of the box. So for now after the installation of OpenShift Istio, you need to choose between option 1 and 2. Also, keep in mind that after updates of the Istio operator you might need to mount the secret again in the deployment because Istio Operator might overwrite it. ## How to analyze problems of the Istio configuration The previous problem that I encountered was a rather specific one with respect to the Istio version used and the OpenShift operator. The other techniques that I will describe are more generic and not so much specific for the Istio and OpenShift environment. ### Get the configuration of the Envoy proxy of a pod During the creation of my sample application, I encountered a scenario were I needed to verify the configuration of an Envoy proxy in a pod. To a certain degree, the Istio configuration will end up in an Envoy proxy as its configuration. A look at the Envoy configuration helps us to verify, if our Istio configuration has been translated to to the desired Envoy configuration. Furthermore, by looking at the Envoy configuration we gain a deeper understanding of the Istio internal behavior. Okay, enough talk, how can we get the configuration of an Istio-proxy (Envoy proxy) for a pod? The answer is very simple. By using this command: ```bash oc exec -c istio-proxy -- curl -kv localhost:15000/config_dump> _Envoy_config.json ``` If we take a look at the written file, we will get e.g. the jwk config (like seen above), the retries for endpoints, the routing and many more. At this point I have to admit that I could not create the Envoy configuration from scratch. However I find the configuration easy enough to understand when focusing on certain aspects. ### Increase the logging level of the Envoy proxy Besides, the configuration there is a nice feature that I started to appreciate very much: To increase the logging for certain features of the Envoy proxy without increase the log output for other components. Let's have a look which components are available: |admin|aws|assert|backtrace|client|config|connection|conn_handler| |dubbo|file|filter|forward_proxy|grpc|hc|health_checker|http| |http2|hystrix|init|io|jwt|kafka|lua|main| |misc|mongo|quic|pool|rbac|redis|router|runtime| |stats|secret|tap|testing|thrift|tracing|upstream|udp| |wasm||||||| There are quite a lot of components from which we can choose, if we want to increase the log level. To change the level for only the RBAC component, we can execute the following command: ```bash oc exec -it -c istio-proxy -- sh -c 'curl -k -X POST localhost:15000/logging?rbac=debug' ``` We should get a response like this from the pod: ```bash active loggers: admin: warning aws: warning assert: warning backtrace: warning client: warning config: warning ... pool: warning rbac: debug redis: warning router: warning runtime: warning stats: warning secret: warning tap: warning testing: warning thrift: warning tracing: warning upstream: warning udp: warning wasm: warning ``` If we really wanted, we could set the log level for all components of envy proxy to `debug` with this command: ```bash oc exec -it -c istio-proxy -- sh -c 'curl -k -X POST localhost:15000/logging?level=debug' ``` The output should look like this: ```bash active loggers: admin: debug aws: debug assert: debug backtrace: debug client: debug config: debug ... pool: debug rbac: debug redis: debug router: debug runtime: debug stats: debug secret: debug tap: debug testing: debug thrift: debug tracing: debug upstream: debug udp: debug wasm: debug ``` Hint: Be careful when changing the log level of all components for an Envoy proxy, because this is generating quite a lot of log entries (even in a test environment). The Envoy proxy offers a minimal UI. We can have a look at the UI, if we do a port-forward to our local computer from an istio-proxy with either: ```bash oc port-forward 15000 ``` or ```bash istioctl dashboard envoy ``` If we open this web-site [`https://localhost:15000`](https://localhost:15000) in our browser we see the following website: ![screenshot_envoy_gui.jpg](screenshot_envoy_gui.jpg) We get quite some options here, what we can see or even change in the Envoy proxy. ### Increase the logging of an Istio component So far, now I have showed you, what we can do to get more information from the Envoy proxy. I have not touched the control plane of Istio so far. It took me some time to find it, however, each Istio control plane component has a built in GUI (at least for Mixer, Pilot, Citadel, and Galley that is). We can have a look at it by forwarding the port of the Istio pod to our local computer. Again, we do a port-forward this time but instead of the port 15000 the port is now 9876: ```bash oc port-forward 9876 ``` For the Istio pilot it looks like this: ```bash oc port-forward istio-pilot-775d8bfc6b-zb25d 9876 ``` If we open this website [`http://localhost:9876/scopez/`](http://localhost:9876/scopez/) in our browser, we see the following GUI: ![screenshot_pilot_gui.jpg](screenshot_pilot_gui.jpg) So, this time we have a GUI to change the logging setting of a component, which is quite nice. Also, we can get more information about the memory usage, the metrics and much more. ## Check the Istio configuration Besides changing the logging level, Istio offers another powerful tool to see if something is wrong. WE can use the command line or Kiali to verify our Istio configuration. It might not be perfect, but it still can save us from a lot of trouble. ### Option a): the command line tool #### Get proxy configuration of a specific pod As I have pointed out above, we can get the complete Envoy configuration using `oc exec`. There is another way we can use `istioctl proxy-config`. This command allows us to get subset of the proxy configuration like `endpoints` or `clusters` for example. To get a list of all end points for a proxy, we can run the following command: ```bash istioctl proxy-config endpoints ``` This can be quite useful, as it shows us which endpoints are available for the pod and whether they are healthy. For one of my pods, it looks like this: ```bash $ istioctl proxy-config endpoints customer-56d65d78d9-n5sq9 ENDPOINT STATUS OUTLIER CHECK CLUSTER 10.10.10.39:8060 HEALTHY OK outbound|8060||istio-citadel.olaf.svc.cluster.local 10.10.10.39:15014 HEALTHY OK outbound|15014||istio-citadel.olaf.svc.cluster.local 10.10.10.40:3001 HEALTHY OK outbound|9090||prometheus.olaf.svc.cluster.local 10.10.10.41:8443 HEALTHY OK outbound|443||istio-galley.olaf.svc.cluster.local 10.10.10.41:9901 HEALTHY OK outbound|9901||istio-galley.olaf.svc.cluster.local 10.10.10.41:15014 HEALTHY OK outbound|15014||istio-galley.olaf.svc.cluster.local 10.10.10.42:9091 HEALTHY OK outbound|9091||istio-policy.olaf.svc.cluster.local 10.10.10.42:15004 HEALTHY OK outbound|15004||istio-policy.olaf.svc.cluster.local 10.10.10.42:15014 HEALTHY OK outbound|15014||istio-policy.olaf.svc.cluster.local 10.10.10.43:8080 HEALTHY OK outbound|80||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.43:8443 HEALTHY OK outbound|443||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.43:15020 HEALTHY OK outbound|15020||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.43:15443 HEALTHY OK outbound|15443||istio-ingressgateway.olaf.svc.cluster.local 10.10.10.45:20001 HEALTHY OK outbound|20001||kiali.olaf.svc.cluster.local 10.10.10.51:8080 HEALTHY OK outbound|8080|version-v1|customer.olaf-customer.svc.cluster.local 10.10.10.51:8080 HEALTHY OK outbound|8080||customer.olaf-customer.svc.cluster.local 10.10.10.53:8080 HEALTHY OK outbound|8080|version-v1|recommendation.olaf-customer.svc.cluster.local 10.10.10.53:8080 HEALTHY OK outbound|8080||recommendation.olaf-customer.svc.cluster.local 10.10.11.100:8080 HEALTHY OK outbound|8080|version-v2|recommendation.olaf-customer.svc.cluster.local 10.10.11.100:8080 HEALTHY OK outbound|8080||recommendation.olaf-customer.svc.cluster.local 10.10.11.35:8443 HEALTHY OK outbound|443||jaeger-query.olaf.svc.cluster.local 10.10.11.35:9411 HEALTHY OK outbound|9411||jaeger-collector.olaf.svc.cluster.local 10.10.11.35:9411 HEALTHY OK outbound|9411||zipkin.olaf.svc.cluster.local 10.10.11.35:14250 HEALTHY OK outbound|14250||jaeger-collector.olaf.svc.cluster.local 10.10.11.35:14267 HEALTHY OK outbound|14267||jaeger-collector.olaf.svc.cluster.local 10.10.11.35:14268 HEALTHY OK outbound|14268||jaeger-collector.olaf.svc.cluster.local 10.10.11.94:9091 HEALTHY OK outbound|9091||istio-telemetry.olaf.svc.cluster.local 10.10.11.94:15004 HEALTHY OK outbound|15004||istio-telemetry.olaf.svc.cluster.local 10.10.11.94:15014 HEALTHY OK outbound|15014||istio-telemetry.olaf.svc.cluster.local 10.10.11.94:42422 HEALTHY OK outbound|42422||istio-telemetry.olaf.svc.cluster.local 10.10.11.95:8080 HEALTHY OK outbound|8080||istio-pilot.olaf.svc.cluster.local 10.10.11.95:15010 HEALTHY OK outbound|15010||istio-pilot.olaf.svc.cluster.local 10.10.11.95:15011 HEALTHY OK outbound|15011||istio-pilot.olaf.svc.cluster.local 10.10.11.95:15014 HEALTHY OK outbound|15014||istio-pilot.olaf.svc.cluster.local 10.10.11.96:8080 HEALTHY OK outbound|80||istio-egressgateway.olaf.svc.cluster.local 10.10.11.96:8443 HEALTHY OK outbound|443||istio-egressgateway.olaf.svc.cluster.local 10.10.11.96:15443 HEALTHY OK outbound|15443||istio-egressgateway.olaf.svc.cluster.local 10.10.11.98:8443 HEALTHY OK outbound|443||istio-sidecar-injector.olaf.svc.cluster.local 10.10.11.98:15014 HEALTHY OK outbound|15014||istio-sidecar-injector.olaf.svc.cluster.local 10.10.11.99:3001 HEALTHY OK outbound|3000||grafana.olaf.svc.cluster.local 10.131.1.60:8080 HEALTHY OK outbound|8080|version-v1|preference.olaf-customer.svc.cluster.local 10.131.1.60:8080 HEALTHY OK outbound|8080||preference.olaf-customer.svc.cluster.local 127.0.0.1:8080 HEALTHY OK inbound|8080|http-customer|customer.olaf-customer.svc.cluster.local 127.0.0.1:15000 HEALTHY OK prometheus_stats 127.0.0.1:15020 HEALTHY OK inbound|15020|mgmt-15020|mgmtCluster 172.30.1.216:9411 HEALTHY OK zipkin 172.30.174.171:15010 HEALTHY OK xds-grpc ``` To get a list of all available sub-command, run: ``` bash istioctl proxy-config --help ``` #### Proxy status in istio Another useful command is `istioctl proxy-status`. This command returns the sync status of the pod with respect to the central configuration of Istio (pilot). If a problem with the proxy configuration occurs, it is a good starting point to check whether the proxies are in sync with pilot. For my cluster, the result of the proxy status looks like this: ```bash $ istioctl proxy-status -i olaf NAME CDS LDS EDS RDS PILOT VERSION customer-56d65d78d9-n5sq9.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 istio-egressgateway-85cd64f885-q99ml.olaf SYNCED SYNCED SYNCED NOT SENT istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 istio-ingressgateway-bf6644dd5-bdjwc.olaf SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 preference-v1-6f449bc44-82l4d.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 recommendation-v1-56ddb779f-lw9hb.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 recommendation-v2-5d85578cbf-49tx4.olaf-customer SYNCED SYNCED SYNCED SYNCED istio-pilot-fbd4b6b5-48b7r maistra-1.1.0 ``` This looks good so far. Don't worry about the value `NOT SENT` for the egress proxy, because there are no services defined for the egress, so no need to sync anything. The meaning of the columns CDS, LDS, EDS and RDS are here: > - CDS: [Envoy Cluster Discovery Service](https://www.envoyproxy.io/docs/envoy/latest/configuration/upstream/cluster_manager/cds.html). Envoy defines a “cluster” as a group of hosts that accept traffic for a given endpoint or service. Don’t confuse an Envoy cluster with a Kubernetes cluster – in a Kubernetes cluster, an Envoy “cluster” will typically, though not always, be a Kubernetes deployment or other replica set. > - LDS: [Envoy Listener Discovery Service](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/lds). Envoy defines a “listener” for each configured host:port endpoint. A Kubernetes cluster will typically have an Envoy listener for each target service port in an Envoy cluster. > - EDS: Envoy Endpoint Discovery Service. Envoy defines an “endpoint” as a member of an Envoy cluster, to which it can connect for a service. In a Kubernetes cluster, an Envoy endpoint would be a pod backing the destination service. > - RDS: [Envoy Route Discovery Service](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/rds.html). Envoy defines a “route” as the set of Envoy rules that match a service or virtual host to an Envoy “cluster.” (Quote from [How to make Istio work with your app](https://securityboulevard.com/2019/11/how-to-make-istio-work-with-your-apps)) #### False positives If we run the command `istioctl experimental analyze -k` it will analyze the Istio configuration for our cluster. In my case, I use OpenShift 4.3. This will return an error because the injection of an Istio sidecar container can only be done in the deployment etc. and it is not possible base on setting in the namespace. So do not worry, if you see this error message on an OpenShift cluster: ```bash istioctl experimental analyze -k Warn [IST0102] (Namespace olaf-customer) The namespace is not enabled for Istio injection. Run 'kubectl label namespace olaf-customer istio-injection=enabled' to enable it, or 'kubectl label namespace olaf-customer istio-injection=disabled' to explicitly mark it as not needing injection ``` ### Option b): Kiali Here is an example how Kiali is verifying a Istio configuration. When setting up my demo application, I wanted to make sure that the traffic between the pods is using mTLS by default. To ensure this, I created a default Destination Route that enables mTLS. When I look at the `Istio Config`, I notice the following error. ![screenshot_kiali_istio_config_overview.jpg](screenshot_kiali_istio_config_overview.jpg) When I opened the YAML file, I got a detail description what went wrong. ![screenshot_kiali_istio_config__error_detail.jpg](screenshot_kiali_istio_config__error_detail.jpg) In my case, I forgot to create a global ServiceMeshPolicy. Please note, that this is only a fraction of the tests that Kiali does; but to show all of them would go beyond the scope of the article. A complete list of the validation can be found [here](https://kiali.io/documentation/validations/). In general, I like the combination of Kiali and Jaeger a lot to debug configuration issues. We either get a visual feedback that something is not working like intended (i.e. mTLS is not switched on) or we get a validation error. ## Summary and further reading material When I started with Istio, I was wondering what the purpose of the `istioctl` was, because we can setup the istio configuration by only using `oc` or `kubectl`. After some playing around with applications that were using Istio features, I notice that the strength of `istioctl` is analysis of the Istio runtime environment. Istioctl together with Kiali and Jaeger, it forms a very good team if we need to hunt down errors and this is something, that we all know, happens often sooner than we like. - [More information on the logging of the Envoy proxy](https://www.envoyproxy.io/docs/envoy/latest/operations/admin#post--logging) - [More info about the REST admin API of the Envoy proxy](https://www.envoyproxy.io/docs/envoy/latest/operations/admin) - [Istio documentation about the Web-GUI for the Istio Components](https://istio.io/docs/ops/diagnostic-tools/controlz/) - [How to make Istio work with your app](https://securityboulevard.com/2019/11/how-to-make-istio-work-with-your-apps) [Read more](https://omd.consol.de/blog/2020/05/07/debugging-istio/) ----------------------------------------------------------------------------------- title: "Simple example how to use Istio and Keycloak" date: "2020-05-07" --------------------------------------------------------------------------------
Some time ago, I did a webinar about the RedHat Service Mesh, which is based on Istio. For this webinar, I prepared a demo application. Among other things, I wanted to show how to do the authentication with JWT token in general and, more specific, with Keycloak. This article will describe how to configure Keycloak. In the second article, I will show you what problems I encountered running the application in Istio and how I figured out what was wrong in my configuration. You can find the article [here](/blog/2020/05/07/debugging-istio/)
## Set up the demo application As basis for my demo application I used the application of the awesome [Red Hat Istio tutorial](https://redhat-developer-demos.github.io/istio-tutorial/istio-tutorial/1.3.x/index.html). The only exception I made was to use my own docker repo and to create a single yaml file to setup the application along with the Istio configuration. The yaml with the OpenShift resources and with the Istio configuration file can be found [here](https://raw.githubusercontent.com/olaf-meyer/openshift-talks/master/setup_demo_app.yaml). The demo application consists of four services. The entry point is the customer service. The customer service calls the recommendation service. Which calls version 1 and version 2 of preference services. The following picture shows an overview of the demo application. ![](screenshot_app_overview.jpg) Istio configuration for the demo application is done in that manner that it enforces mTLS and RBAC. With the current policy, the communication between the services is always encrypted. Furthermore, this configuration ensures that the services can only communicate the designated endpoints and all other communication will be blocked. Also, I wanted to secure the entry point of the application so that it can only be called by users that have been authenticated by a Keycloak instance. The bearer/access token and the public key of other Istio examples that I found were almost exclusively hard coded. Furthermore, the authorization was done by the `audience` attribute in the JWT token and not by the roles assigned to the user. To be a bit closer to a real world setup, I decided to use roles instead of hard coded `audience`. To do this, I configured a realm, client, role and a user in Keycloak. As a starting point for my Keycloak configuration I used a previous version of the Red Hat Istio tutorial. The example can be found [here](https://github.com/redhat-developer-demos/istio-tutorial/blob/1a46e933f0c78f007cabb67b6b9ff079e3300fda/documentation/modules/advanced/pages/jwt.adoc). In a newer version of the tutorial, it used a hard coded access token and a public key. ## Deploy Keycloak For my webinar I used a Keycloak which was deployed on an OpenShift 4.3 cluster. However, you should be able to use RedHat SSO instead of Keycloak as well. To deploy Keycloak to my OpenShift cluster I used a template provided by Keycloak. Please notice that this template is ephemeral and the used password should not be used in production! ```bash oc process \ -f https://raw.githubusercontent.com/keycloak/keycloak-containers/master/openshift-examples/keycloak-https.json \ -p NAMESPACE=olaf-sso \ -p KEYCLOAK_USER=admin \ -p KEYCLOAK_PASSWORD=admin \ | oc create -f - ``` ## Setup Keycloak as an identity provider Let's start with log into Keycloak and setup the Istio configuration. ### Create a security realm The first step is to create a security realm. In order to do this, press "Add realm" and enter the name "customer", then press "Create". Piece of cake, so far. ![](screenshot_add_realm.jpg) ### Create a custom client Next, create a client with the name "istio". You do not need to add more information and leave the `Client Protocol` as `openid-connect`. At least, not at the moment. ![](screenshot_add_client.jpg) ### Create a role To be able to select user by roles, we need to create at least one role. Create a role called "customer". If you want, you can create another role for test purpose. ![](screenshot_add_role.jpg) ### Create a test user We are almost there; we just need to create a user. Which is pretty straightforward. In the view "Users" press the button "Add user". In my case, I added the following values in the form and pressed save. ![](screenshot_add_user.jpg) In the user we need to do two more things: - In the next step, we need to assign a role to the user. For this, select the tab "Role Mappings" and assign the realm role "customer" to the new user.![](screenshot_add_user_role.jpg) ### Verify the result Sow far, we created everything that we need to get an access_token from Keycloak. To do exactly this, I called this script in the bash (The URL and the values may vary in your environment): ```bash curl \ -sk \ --data "username=olaf&password=olaf&grant_type=password&client_id=istio" \ https://secure-keycloak-olaf-sso.apps.poc2.int.consol.de/auth/realms/customer/protocol/openid-connect/token \ | jq ".access_token" "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJZQlNMT3V1VkdMWkpDaXVQUUVrQTRTTzZEWE5qTGJKNkgyV0tSOXh2LUhRIn0.eyJqdGkiOiJiOWVjOTg4Yy05MmFhLTQ4YzUtOGQxOS00YWQ1NDgyMTgxOTkiLCJleHAiOjE1ODQxMTM1OTMsIm5iZiI6MCwiaWF0IjoxNTg0MTEzMjkzLCJpc3MiOiJodHRwczovL3NlY3VyZS1rZXljbG9hay1vbGFmLXNzby5hcHBzLnBvYzIuaW50LmNvbnNvbC5kZS9hdXRoL3JlYWxtcy9jdXN0b21lciIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiJjZDEwYTMyZC1iMGVhLTRhZWMtYTViNi0yNzFjMWExNGEwOWQiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJpc3RpbyIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6ImZlYTdjMDlmLTA1ZDUtNDNiOS05MmJhLTVjNGY0NGJiYzhkMyIsImFjciI6IjEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsImN1c3RvbWVyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsIm5hbWUiOiJPbGFmIE1leWVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoib2xhZiIsImdpdmVuX25hbWUiOiJPbGFmIiwiZmFtaWx5X25hbWUiOiJNZXllciIsImVtYWlsIjoib2xhZi5tZXllckBjb25zb2wuZGUifQ.Qec8ywW7fhYGOFzh3YfmJsSWZm96GnJg0XZWjjBwAp1Zap8UfUQaFpkSl-ey2-k12nptpYMuKU9PXPIXq8dnfuLUNYK5mZHLxEx77-giFls-mdIJ0LVvcZHqVDvQKWyr_l--zCGsxd4fmKhWdT85FrA0mcMOwJbzE547fQJ8ooNCx4VUo0O1bzR2f-gnX22Whf00Il9esvX8-HllXbBP3_t32OticXWySeOKI2oaVz1f4h38DTK0q22Cdnb-4IsZypGPUe24DR0ZGQUSkhM05ZPX232FKenC1bHOVBV6cXCqVhH2rbs8zcURPIq70txL5aOf31PEhidSwb4VZ2IuPA" ``` My preference to view the content of access_token is the website [https://jwt.io/](https://jwt.io/). Of course, this is not recommended for productive token. The returned access token looks like this: ![](screenshot_access_token1.jpg) There are two points in the JWT token that need to be changed. The first point is that the audience of the token is "account". This value is caused by adding the role "roles" to the access_token. There are many examples that show how to use Istio with JWT, containing almost exclusive checks of the audience in the token. To check if the JWT contains the value "account" is not helpful, so let's modify it. Second, Istio (at least with the version 1.1.7 or Red Hat Service Mesh 1.0.9) can't handle nested attributes. Unfortunately, Keycloak is using by default nested attributes for the assigned roles of the user. This means that we have to create a root level attribute for the roles that are assigned to a user. ### Adjusting the values returned in attribute `aud` / `audience` To replace the complete value of the attribute `aud` (`audience`) is not that easy and could bring side effects with it. So let's add just a value to the `audience`. To do this, open client `istio` and select the tab `Mappers` and press on the button `Create`. In the next form, we create a mapper that adds a hard coded value to the audience. My mapping looks like this: ![](screenshot_access_token2.jpg) After I added the mapping to the Istio client, the resulting access token looks like this: ![](screenshot_access_token3.jpg) As you can see, the attribute `audience` now contains two values `customer` and `account`. Now we can do a check in Istio whether the audience contains the value `customer`. The audience `customer` is only returned, if the client "istio" has been used; but not for other clients. So only one more problem to solve. ### Change location were roles of the user are returned The final change in Keycloak is to make the user roles a top-level attribute. Which is not hard. Like for the `audience` we add a mapper in the client. This time, we are going to use a mapper of the type `User realm role`. With this, the user realm roles will be duplicated to a custom attribute. In my case I create a top level attribute called `roles`. The role mapper that I created looks like this: ![](screenshot_access_token4.jpg) The access token of my user looks like this: ![](screenshot_access_token5.jpg) As you can see, the access token has a new attribute `roles` with `offline_access`,`uma_authorization` and `customer`. With theses changes, we should able to log into Istio with the access token. In the second article, I will show you what problems I encountered running the application in Istio and how I figured out what was wrong in my configuration.You can find the article [here](/blog/2020/05/07/debugging-istio/) [Read more](https://omd.consol.de/blog/2020/05/07/simple-example-how-to-use-istio-and-keycloak/) ----------------------------------------------------------------------------------- title: "OpenShift 4.3 installation on VMware vSphere with static IPs" date: "2020-01-31" --------------------------------------------------------------------------------
In this article, I will show you how to install Red Hat OpenShift Container Platform 4.3 (OCP) on VMware vSphere with static IPs addresses using the openshift installer in UPI mode and terraform. In contrast to the official OpenShift 4.3 install documentation, we will not use DHCP for the nodes and will not setup the nodes manually - instead we will use static IP addresses and terraform to setup the virtual machines in our vCenter. * TOC {:toc} Since this article will not describe basic requirements like the minimum VMware vSphere version, a good start would be to consult the official OpenShift documentation: [https://docs.openshift.com/container-platform/4.3/installing/installing_vsphere/installing-vsphere.html](https://docs.openshift.com/container-platform/4.3/installing/installing_vsphere/installing-vsphere.html){:target="_blank"} # Infrastructure / Nodes Let's take a look at the hosts we will be faced with in this setup. ## External nodes (non OpenShift cluster machines) * **bastion / install host**: the bastion host will be the host, where we execute our openshift-installer and terraform commands. This node can be your workstation or a temporary Linux VM. After a successful installation, this node can be removed - please ensure to save your installation configuration (ssh-key, ignition files, etc.) into an external location (fileshare, git-repo, ...). * **http host**: the http host will provide the ignition file for out bootstrap node via http. Of course you can setup an http server on the bastion / install host. After bootstrapping OpenShift, this host can be removed as well. * **external loadbalancer (lb)**: this node is optional. If you have some kind of real hardware loadbalancer like F5 BigIP, use this one. For this test installation we will setup a very basic HA-Proxy on a Linux VM. ## VMware vSphere nodes (OpenShift nodes) The following nodes will be provisioned on your VMware vSphere environment: * **bootstrap node**: the bootstrap node will be provisioned by our installer and will be responsible for setting up the OpenShift 4 cluster - fully automated. After the installation, the bootstrap node can be removed. * **control pane nodes (OpenShift master nodes)**: * control-plane-0 * control-plane-1 * control-plane-2 * **worker nodes (OpenShift compute nodes)**: * compute-0 * compute-1 # Architecture This picture shows our landscape while installation: *(Loadbalancer A and Loadbalancer B can reside on the same hard-/software dispatcher)* # Prerequisites Let's talk about some very important prerequisites for a smooth installation. The most important point will be the DNS configuration. ## OCP cluster id First of all, lets define an OCP cluster id. This id (string) will be relevant for the DNS configuration. Keep in mind that OpenShift 4 has a concept of a "clusterid" that will be incorporated into your clusters DNS records. Your DNS records will all have `.` in them. In other words, your "cluster id" will end up being part of your FQDN. Read the official documentation for more information. We'll use `ocp4-cluster-001` as a cluster id. So later on, a node's fqdn will be composed of a node name, the cluster id an the base domain: `..` Sample host name (FQDN): `compute-0.ocp4-cluster-001.int.consol.de` ## Base domain The base domain is the subdomain, where your nodes will be placed in. The name depends on your infrastructure. Our base domain: `int.consol.de` ## DNS configuration Proper DNS setup is imperative for a successful OpenShift installation. DNS is used for name resolution (A records), certificate generation (PTR records), and service discovery (SRV records). Our DNS configuration: ```bash # BOOTSTRAP NODE bootstrap.ocp4-cluster-001.int.consol.de has address 10.0.5.58 # MASTER / CONTROL PLANE NODES control-plane-0.ocp4-cluster-001.int.consol.de has address 10.0.5.59 control-plane-1.ocp4-cluster-001.int.consol.de has address 10.0.5.60 control-plane-2.ocp4-cluster-001.int.consol.de has address 10.0.5.61 # WORKER NODES compute-0.ocp4-cluster-001.int.consol.de has address 10.0.5.62 compute-1.ocp4-cluster-001.int.consol.de has address 10.0.5.63 # LOADBALANCER lb.ocp4-cluster-001.int.consol.de has address 10.0.5.64 ``` ## Terraform For the fully automated VM provisioning, we'll use an infrastructure-as-code tool called [terraform](https://www.terraform.io/){:target="_blank"}. Take care of the version - currently only version 11.x of terraform is working with the installer github template. Version 12.x will most likely fail (at the time of writing). Link: [https://www.terraform.io/downloads.html](https://www.terraform.io/downloads.html){:target="_blank"} Quick setup: ```bash $ mkdir ~/bin $ cd /tmp/ $ export TERRAFORM_VERSION=0.11.14 $ curl -O -L https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip $ unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d ~/bin/ $ terraform -v Terraform v0.11.14 ``` ## govc govc is a VMware/vSphere CLI. For faster deployment of an OVA Image, we will use govc. Alternatively you can deploy the template manually via vSphere web console. Link: [https://github.com/vmware/govmomi/tree/master/govc](https://github.com/vmware/govmomi/tree/master/govc){:target="_blank"} Quick setup: ```bash $ mkdir ~/bin $ export GOVC_URL=https://github.com/vmware/govmomi/releases/download/v0.22.1/govc_linux_amd64.gz $ curl -L ${GOVC_URL} | gunzip > ~/bin/govc $ chmod +x ~/bin/govc $ govc version govc 0.22.1 ``` ## SSH keys Since our CoreOS based VMs will be configured automatically, we have to provide a ssh public key to be able to log on to the nodes via ssh as user `core`. Generate a ssh private key: ```bash $ ssh-keygen -f ~/.ssh/id_rsa_ocp4_vcenter ``` Use the generated ssh public key `~/.ssh/id_rsa_ocp4_vcenter.pub` for the installation. ## LB / HAProxy Setup As loadbalancer for our POC setup, we will use a HAProxy on a CentOS VM. Don't do this in production environments! Our loadbalancer will listen on network address `lb.ocp4-cluster-001.int.consol.de:`. ### Install HAProxy ```bash $ yum install haproxy -y ``` ### Basic HAProxy setup This configuration will add the following load balancer entries to HA-Proxy: * openshift-api-server *(port 6443)* * machine-config-server *(port 22623)* * ingress-http *(port 80)* * ingress-https *(port 443)* Add the following configuration lines to `/etc/haproxy/haproxy.cfg`: *(replace the IP addresses!)* ```bash frontend openshift-api-server bind *:6443 default_backend openshift-api-server mode tcp option tcplog backend openshift-api-server balance source mode tcp server bootstrap 10.0.5.58:6443 check server control-plane-0 10.0.5.59:6443 check server control-plane-1 10.0.5.60:6443 check server control-plane-2 10.0.5.61:6443 check frontend machine-config-server bind *:22623 default_backend machine-config-server mode tcp option tcplog backend machine-config-server balance source mode tcp server bootstrap 10.0.5.58:22623 check server control-plane-0 10.0.5.59:22623 check server control-plane-1 10.0.5.60:22623 check server control-plane-2 10.0.5.61:22623 check frontend ingress-http bind *:80 default_backend ingress-http mode tcp option tcplog backend ingress-http balance source mode tcp server compute-0 10.0.5.62:80 check server compute-1 10.0.5.63:80 check frontend ingress-https bind *:443 default_backend ingress-https mode tcp option tcplog backend ingress-https balance source mode tcp server compute-0 10.0.5.62:443 check server compute-1 10.0.5.63:443 check ``` Please ensure, that no services are listening on port 80 or 443 on your HAproxy machine (e.g. `lsof -i:80` or `ss -lnt sport = :80`) Restart HA-Proxy: `systemctl restart haproxy` *(optional)* Enable haproxy service for permanent use: `systemctl enable haproxy` ## HTTP Server for serving ignition files For serving the ignition file of the bootstrap node, we will setup a simple HTTP server on port 8080 on our loadbalancer host (lb.ocp4-cluster-001.int.consol.de). ```bash $ yum install httpd -y ``` In `/etc/httpd/conf/httpd.conf`, change the listen port from `Listen 80` to `Listen 8080`. Restart httpd: `systemctl restart httpd` ## OCP pull secret For installing Red Hat OpenShift Container Platform 4.3, you'll need a pull secret: Link: [https://cloud.redhat.com/openshift/install/vsphere/user-provisioned](https://cloud.redhat.com/openshift/install/vsphere/user-provisioned){:target="_blank"} ## OCP openshift-installer From [https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/](https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/){:target="_blank"}, fetch the appropriate openshift-installer version and extract the binary to `~/bin/`, e.g. `openshift-install-linux-4.3.0.tar.gz`. ## oc client From [https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/](https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/){:target="_blank"}, fetch the appropriate oc version and extract the binary to `~/bin/`, e.g. `openshift-client-linux-4.3.0.tar.gz`. ## VMware vSphere / vCenter For a fully automated provisioning of your VMs on VMware vSphere, you'll need an vCenter account with the appropriate permissions and information: * vCenter URL * vCenter login * vCenter password * datacenter name * cluster name * datastore name A ressource group will be generated automatically and named like your OCP cluster id. ## OVA image The base image for all our CoreOS VMs will be a Red Hat CoreOS Image 4.3. This image can be found here: [https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/4.3.0/rhcos-4.3.0-x86_64-vmware.ova](https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/4.3.0/rhcos-4.3.0-x86_64-vmware.ova){:target="_blank"} # Installation ## OVA image upload The first step of the installation is to upload the OVA image to our vCenter datacenter and mark it as a template. All CoreOS VMs will be cloned from this template. Ensure you have the correct image downloaded: ```bash $ ls -la rhcos-4.3.0-x86_64-vmware.ova -rw-rw-r-- 1 zisis zisis 829542400 Jan 27 12:59 rhcos-4.3.0-x86_64-vmware.ova ``` Set govc environment variables if you want to upload the image via govc. Alternatively you can upload the image via vCenter web console. ```bash export GOVC_URL='ocp-vcenter.int.consol.de' export GOVC_USERNAME='administrator@vsphere.local' export GOVC_PASSWORD='mypassword' export GOVC_INSECURE=1 export GOVC_DATASTORE='vsanDatastore' ``` Check if your login works: ```bash $ govc about Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 ... ``` Upload OVA image and mark as template. The name (rhcos-4.3.0) is important for later use, since we will reference the name of the template in our terraform plan (terraform.tfvars). Pull spec: ```bash $ govc import.spec /home/zisis/rhcos-4.3.0-x86_64-vmware.ova | python -m json.tool > rhcos.json ``` Customize the Network you want to use in `rhcos.json`: ```bash "NetworkMapping": [ { "Name": "VM Network", "Network": "VM Network" } ``` List resource pools: ```bash $ govc find / -type p /Datacenter/host/cluster001/Resources /Datacenter/host/openshift-test-esx-3.int.consol.de/Resources ``` Upload the rhcos-4.3.0 image to the resource pool: ```bash $ govc import.ova -options=./rhcos.json -name=rhcos-4.3.0 \ -pool=/Datacenter/host/cluster001/Resources /home/zisis/rhcos-4.3.0-x86_64-vmware.ova [27-01-20 17:27:21] Uploading disk.vmdk... OK ``` Mark the VM as a template: ```bash $ govc vm.markastemplate vm/rhcos-4.3.0 ``` ## Ignition files (openshift-installer) The following steps will: * create an installation directory * create an install-config.yaml * create a backup of install-config.yaml, since the install-config.yaml will be deleted automatically * create the ignition files with openshift-installer For the install-config.yaml, you need the following input: * base domain * OCP cluster id * OCP pull secret * ssh ***public*** key (~/.ssh/id_rsa_ocp4_vcenter.pub) * vCenter host * vCenter user * vCenter password * vCenter datacenter * vCenter datastore Commands: ```bash $ mkdir ocpinstall $ cd ocpinstall # create an install-config.yaml $ cat < install-config.yaml apiVersion: v1 baseDomain: int.consol.de metadata: name: ocp4-cluster-001 platform: vSphere: vcenter: ocp-vcenter.int.consol.de username: administrator@vsphere.local password: mypassword datacenter: Datacenter defaultDatastore: vsanDatastore pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"........"}}}' sshKey: ssh-rsa AAAAB3N..... EOF # make a backup $ cp install-config.yaml install-config.`date '+%s'`.bak # create ignition configs $ openshift-install create ignition-configs INFO Consuming "Install Config" from target directory ``` Now you should have a directory structure like this: ```bash $ tree . ├── auth │   ├── kubeadmin-password │   └── kubeconfig ├── bootstrap.ign ├── install-config.1579194442.bak ├── master.ign ├── metadata.json └── worker.ign ``` Caveat: ignition files are valid for 24 hours - so if your installation takes longer than 24 hours due to issues, you have to generate new ignition files. ## Copy ignition files to your HTTP server Copy the generated bootstrap.ign file to your HTTP server and ensure, that the file can be downloaded with http: Copy: ```bash $ scp bootstrap.ign http-user@lb.ocp4-cluster-001.int.consol.de:/var/www/html/ ``` Check, if download would succeed from your http server: ```bash $ curl -I http://lb.int.consol.de:8080/bootstrap.ign HTTP/1.1 200 OK ... ``` ## Prepare the terraform installer For the installation, we will use a modified version of the UPI vSphere installer, which is available at: [https://github.com/openshift/installer/tree/master/upi/vsphere](https://github.com/openshift/installer/tree/master/upi/vsphere){:target="_blank"} Clone the repo and change to the upi vSphere directory: ```bash $ git clone https://github.com/openshift/installer.git $ cd installer/upi/vsphere ``` Copy the example file to terraform.tfvars an adjust the variables. In sections `control_plane_ignition / END_OF_MASTER_IGNITION` and `compute_ignition / END_OF_WORKER_IGNITION`, insert / copy&paste the contents of the ignitions files (master and worker) we generated before: * END_OF_MASTER_IGNITION -> master.ign * END_OF_WORKER_IGNITION -> worker.ign Our terraform.tfvars: ```bash cluster_id = "ocp4-cluster-001" cluster_domain = "ocp4-cluster-001.int.consol.de" base_domain = "int.consol.de" vsphere_server = "ocp-vcenter.int.consol.de" vsphere_user = "administrator@vsphere.local" vsphere_password = "mypassword" vsphere_cluster = "cluster001" vsphere_datacenter = "Datacenter" vsphere_datastore = "vsanDatastore" vm_template = "rhcos-4.3.0" machine_cidr = "10.0.5.0/24" vm_network = "VM Network" control_plane_count = 3 compute_count = 2 bootstrap_ignition_url = "http://lb.int.consol.de:8080/bootstrap.ign" control_plane_ignition = < In this phase, the bootstrap node is set up, which in turn will setup the cluster: ```bash $ cd ~/ocpinstall/ $ openshift-install --dir=. wait-for bootstrap-complete --log-level debug DEBUG OpenShift Installer v4.3.0 DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3e INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp4-cluster-001.int.consol.de:6443... DEBUG Still waiting for the Kubernetes API: the server could not find the requested resource ... DEBUG Still waiting for the Kubernetes API: the server could not find the requested resource DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4-cluster-001.int.consol.de:6443/version?timeout=32s: EOF INFO API v1.16.2 up INFO Waiting up to 30m0s for bootstrapping to complete... DEBUG Bootstrap status: complete INFO It is now safe to remove the bootstrap resources ... ``` If you want to see more details about installers progress, you can start a `tail -f` on `~/ocpinstall/.openshift_install.log`. Since the bootstrap node has given control over to the forming cluster, we can remove the bootstrap node: ```bash $ cd installer/upi/vsphere $ terraform apply -auto-approve -var 'bootstrap_complete=true' ``` After the bootstrap node was removed, you can adjust your loadbalancer and remove the bootstrap node from your dispatcher configuration. The OCP4 cluster will continue to finish its installation by using a lot of cool operators: ```bash $ cd ~/ocpinstall/ $ openshift-install --dir=. wait-for install-complete INFO Waiting up to 30m0s for the cluster at https://api.ocp4-cluster-001.int.consol.de:6443 to initialize... INFO Waiting up to 10m0s for the openshift-console route to be created... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/zisis/ocpinstall/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4-cluster-001.int.consol.de INFO Login to the console with user: kubeadmin, password: AHTUN-2vGkj-Fo0Ba-Ctae9 ``` A new star is born. ## Kubeconfig for oc commands For being able to execute oc commands, you have to use the kubeconfig, which was generated by the installer. You either use this kubeconfig by exporting the environment variable KUBECONFIG or copy the file auth/kubeconfig to your home directory in .kube/config (be sure not to overwrite any other kubeconfig): ```bash $ export KUBECONFIG=~/ocpinstall/auth/kubeconfig ``` or ```bash $ cp ~/ocpinstall/auth/kubeconfig ~/.kube/config ``` ## Wait for clusteroperators Wait for cluster operators to get ready: ```bash $ oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.3.0 True False False 26h cloud-credential 4.3.0 True False False 27h cluster-autoscaler 4.3.0 True False False 26h console 4.3.0 True False False 26h dns 4.3.0 True False False 26h image-registry 4.3.0 True False False 26h ingress 4.3.0 True False False 26h insights 4.3.0 True False False 26h kube-apiserver 4.3.0 True False False 26h kube-controller-manager 4.3.0 True False False 26h kube-scheduler 4.3.0 True False False 26h machine-api 4.3.0 True False False 26h machine-config 4.3.0 True False False 26h marketplace 4.3.0 True False False 26h monitoring 4.3.0 True False False 26h network 4.3.0 True False False 26h node-tuning 4.3.0 True False False 26h openshift-apiserver 4.3.0 True False False 26h openshift-controller-manager 4.3.0 True False False 26h openshift-samples 4.3.0 True False False 26h operator-lifecycle-manager 4.3.0 True False False 26h operator-lifecycle-manager-catalog 4.3.0 True False False 26h service-ca 4.3.0 True False False 26h service-catalog-apiserver 4.3.0 True False False 26h service-catalog-controller-manager 4.3.0 True False False 26h storage 4.3.0 True False False 26h ``` Check your nodes: ```bash $ oc get nodes NAME STATUS ROLES AGE VERSION compute-0 Ready worker 8m2s v1.14.6+cebabbf4a compute-1 Ready worker 7m54s v1.14.6+cebabbf4a control-plane-0 Ready master 10m13s v1.14.6+cebabbf4a control-plane-1 Ready master 11m26s v1.14.6+cebabbf4a control-plane-2 Ready master 10m54s v1.14.6+cebabbf4a ``` ## Setup image registry (postinstall) In order to complete the installation, you need to add storage to the image registry. For test setups, you can set this to emptyDir (for more permanent storage, please see the official documentation for more information). So after a successful installation, only the image registry operator should be running in the openshift-image-registry namespace: ```bash $ oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-f9697f69d-n47j8 2/2 Running 0 62s ``` Patch the image registry operator to use an emptyDir as storage: ```bash $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' config.imageregistry.operator.openshift.io/cluster patched ``` *(for OCP 4.3)* When there is no Object Storage available at initial setup of the registry, you will have to set manually the managementState for OpenShift bringing up the image registry successfully, see [https://docs.openshift.com/container-platform/4.3/registry/configuring-registry-storage/configuring-registry-storage-vsphere.html#registry-removed_configuring-registry-storage-vsphere](https://docs.openshift.com/container-platform/4.3/registry/configuring-registry-storage/configuring-registry-storage-vsphere.html#registry-removed_configuring-registry-storage-vsphere){:target="_blank"}: ```bash $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState": "Managed"}}' config.imageregistry.operator.openshift.io/cluster patched ``` Now you should see the registry coming up: ```bash $ oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-f9697f69d-n47j8 2/2 Running 0 3m54s image-registry-7766c98447-pgdwp 0/1 ContainerCreating 0 65s image-registry-9fd47f8fb-gs5kc 0/1 ContainerCreating 0 65s node-ca-2g5j9 0/1 ContainerCreating 0 66s node-ca-2l85v 0/1 ContainerCreating 0 66s node-ca-798mx 0/1 ContainerCreating 0 66s node-ca-nmrhw 0/1 ContainerCreating 0 66s node-ca-wcp5g 0/1 ContainerCreating 0 66s ``` # Uninstall cluster If you want to remove the cluster, you can use the terraform destroy command: ```bash $ cd installer/upi/vsphere $ terraform destroy -auto-approve ``` # Additional information Since your VMware administrator will ask you for sure about the required permissions for the installation, maybe this is a good list to start the discussion. I'm sure, not all of these permissions are needed for the installation, so if you have time, strip it down ;) vSphere roles / permissions: ```bash Datastore Allocate space Low level file operations Folder Create folder Delete folder Network Assign network Resource Assign vApp to resource pool Assign virtual machine to resource pool Create resource pool Remove resource pool vApp Clone View OVF environment vApp application configuration vApp instance configuration vApp resource configuration Virtual machine Change Configuration Acquire disk lease Add existing disk Add new disk Add or remove device Advanced configuration Change CPU count Change Memory Change Settings Change Swapfile placement Change resource Configure Host USB device Configure Raw device Configure managedBy Display connection settings Extend virtual disk Modify device settings Query Fault Tolerance compatibility Query unowned files Reload from path Remove disk Rename Reset guest information Set annotation Toggle disk change tracking Toggle fork parent Upgrade virtual machine compatibility Edit Inventory Create from existing Create new Move Register Remove Unregister Guest operations Guest operation alias modification Guest operation alias query Guest operation modifications Guest operation program execution Guest operation queries Interaction Answer question Backup operation on virtual machine Configure CD media Configure floppy media Connect devices Console interaction Create screenshot Defragment all disks Drag and drop Guest operating system management by VIX API Inject USB HID scan codes Install VMware Tools Pause or Unpause Perform wipe or shrink operations Power off Power on Record session on virtual machine Replay session on virtual machine Reset Resume Fault Tolerance Suspend Suspend Fault Tolerance Test failover Test restart Secondary VM Turn off Fault Tolerance Turn on Fault Tolerance Provisioning Allow disk access Allow file access Allow read-only disk access Allow virtual machine download Allow virtual machine files upload Clone template Clone virtual machine Create template from virtual machine Customize guest Deploy template Mark as template Mark as virtual machine Modify customization specification Promote disks Read customization specifications ``` # Contact If you have any questions or remarks, feel free to contact me via zisis(dot)lianas(at)consol(dot)de or [Twitter](https://twitter.com/zisisli){:target="_blank"}. If you need professional OpenShift support, visit us at [https://www.consol.de/it-consulting/openshift/](https://www.consol.de/it-consulting/openshift/){:target="_blank"}. [Read more](https://omd.consol.de/blog/2020/01/31/openshift-4.3-installation-on-vmware-vsphere-with-static-ips/) ----------------------------------------------------------------------------------- title: "OpenShift 4.3 installation on VMware vSphere with static IPs" date: "2020-01-31" --------------------------------------------------------------------------------
In this article, I will show you how to install Red Hat OpenShift Container Platform 4.3 (OCP) on VMware vSphere with static IPs addresses using the openshift installer in UPI mode and terraform. In contrast to the official OpenShift 4.3 install documentation, we will not use DHCP for the nodes and will not setup the nodes manually - instead we will use static IP addresses and terraform to setup the virtual machines in our vCenter. * TOC {:toc} Since this article will not describe basic requirements like the minimum VMware vSphere version, a good start would be to consult the official OpenShift documentation: [https://docs.openshift.com/container-platform/4.3/installing/installing_vsphere/installing-vsphere.html](https://docs.openshift.com/container-platform/4.3/installing/installing_vsphere/installing-vsphere.html){:target="_blank"} # Infrastructure / Nodes Let's take a look at the hosts we will be faced with in this setup. ## External nodes (non OpenShift cluster machines) * **bastion / install host**: the bastion host will be the host, where we execute our openshift-installer and terraform commands. This node can be your workstation or a temporary Linux VM. After a successful installation, this node can be removed - please ensure to save your installation configuration (ssh-key, ignition files, etc.) into an external location (fileshare, git-repo, ...). * **http host**: the http host will provide the ignition file for out bootstrap node via http. Of course you can setup an http server on the bastion / install host. After bootstrapping OpenShift, this host can be removed as well. * **external loadbalancer (lb)**: this node is optional. If you have some kind of real hardware loadbalancer like F5 BigIP, use this one. For this test installation we will setup a very basic HA-Proxy on a Linux VM. ## VMware vSphere nodes (OpenShift nodes) The following nodes will be provisioned on your VMware vSphere environment: * **bootstrap node**: the bootstrap node will be provisioned by our installer and will be responsible for setting up the OpenShift 4 cluster - fully automated. After the installation, the bootstrap node can be removed. * **control pane nodes (OpenShift master nodes)**: * control-plane-0 * control-plane-1 * control-plane-2 * **worker nodes (OpenShift compute nodes)**: * compute-0 * compute-1 # Architecture This picture shows our landscape while installation: *(Loadbalancer A and Loadbalancer B can reside on the same hard-/software dispatcher)* # Prerequisites Let's talk about some very important prerequisites for a smooth installation. The most important point will be the DNS configuration. ## OCP cluster id First of all, lets define an OCP cluster id. This id (string) will be relevant for the DNS configuration. Keep in mind that OpenShift 4 has a concept of a "clusterid" that will be incorporated into your clusters DNS records. Your DNS records will all have `.` in them. In other words, your "cluster id" will end up being part of your FQDN. Read the official documentation for more information. We'll use `ocp4-cluster-001` as a cluster id. So later on, a node's fqdn will be composed of a node name, the cluster id an the base domain: `..` Sample host name (FQDN): `compute-0.ocp4-cluster-001.int.consol.de` ## Base domain The base domain is the subdomain, where your nodes will be placed in. The name depends on your infrastructure. Our base domain: `int.consol.de` ## DNS configuration Proper DNS setup is imperative for a successful OpenShift installation. DNS is used for name resolution (A records), certificate generation (PTR records), and service discovery (SRV records). Our DNS configuration: {% highlight bash %} # BOOTSTRAP NODE bootstrap.ocp4-cluster-001.int.consol.de has address 10.0.5.58 # MASTER / CONTROL PLANE NODES control-plane-0.ocp4-cluster-001.int.consol.de has address 10.0.5.59 control-plane-1.ocp4-cluster-001.int.consol.de has address 10.0.5.60 control-plane-2.ocp4-cluster-001.int.consol.de has address 10.0.5.61 # WORKER NODES compute-0.ocp4-cluster-001.int.consol.de has address 10.0.5.62 compute-1.ocp4-cluster-001.int.consol.de has address 10.0.5.63 # LOADBALANCER lb.ocp4-cluster-001.int.consol.de has address 10.0.5.64 {% endhighlight %} ## Terraform For the fully automated VM provisioning, we'll use an infrastructure-as-code tool called [terraform](https://www.terraform.io/){:target="_blank"}. Take care of the version - currently only version 11.x of terraform is working with the installer github template. Version 12.x will most likely fail (at the time of writing). Link: [https://www.terraform.io/downloads.html](https://www.terraform.io/downloads.html){:target="_blank"} Quick setup: {% highlight bash %} $ mkdir ~/bin $ cd /tmp/ $ export TERRAFORM_VERSION=0.11.14 $ curl -O -L https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip $ unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d ~/bin/ $ terraform -v Terraform v0.11.14 {% endhighlight %} ## govc govc is a VMware/vSphere CLI. For faster deployment of an OVA Image, we will use govc. Alternatively you can deploy the template manually via vSphere web console. Link: [https://github.com/vmware/govmomi/tree/master/govc](https://github.com/vmware/govmomi/tree/master/govc){:target="_blank"} Quick setup: {% highlight bash %} $ mkdir ~/bin $ export GOVC_URL=https://github.com/vmware/govmomi/releases/download/v0.22.1/govc_linux_amd64.gz $ curl -L ${GOVC_URL} | gunzip > ~/bin/govc $ chmod +x ~/bin/govc $ govc version govc 0.22.1 {% endhighlight %} ## SSH keys Since our CoreOS based VMs will be configured automatically, we have to provide a ssh public key to be able to log on to the nodes via ssh as user `core`. Generate a ssh private key: {% highlight bash %} $ ssh-keygen -f ~/.ssh/id_rsa_ocp4_vcenter {% endhighlight %} Use the generated ssh public key `~/.ssh/id_rsa_ocp4_vcenter.pub` for the installation. ## LB / HAProxy Setup As loadbalancer for our POC setup, we will use a HAProxy on a CentOS VM. Don't do this in production environments! Our loadbalancer will listen on network address `lb.ocp4-cluster-001.int.consol.de:`. ### Install HAProxy {% highlight bash %} $ yum install haproxy -y {% endhighlight %} ### Basic HAProxy setup This configuration will add the following load balancer entries to HA-Proxy: * openshift-api-server *(port 6443)* * machine-config-server *(port 22623)* * ingress-http *(port 80)* * ingress-https *(port 443)* Add the following configuration lines to `/etc/haproxy/haproxy.cfg`: *(replace the IP addresses!)* {% highlight bash %} frontend openshift-api-server bind *:6443 default_backend openshift-api-server mode tcp option tcplog backend openshift-api-server balance source mode tcp server bootstrap 10.0.5.58:6443 check server control-plane-0 10.0.5.59:6443 check server control-plane-1 10.0.5.60:6443 check server control-plane-2 10.0.5.61:6443 check frontend machine-config-server bind *:22623 default_backend machine-config-server mode tcp option tcplog backend machine-config-server balance source mode tcp server bootstrap 10.0.5.58:22623 check server control-plane-0 10.0.5.59:22623 check server control-plane-1 10.0.5.60:22623 check server control-plane-2 10.0.5.61:22623 check frontend ingress-http bind *:80 default_backend ingress-http mode tcp option tcplog backend ingress-http balance source mode tcp server compute-0 10.0.5.62:80 check server compute-1 10.0.5.63:80 check frontend ingress-https bind *:443 default_backend ingress-https mode tcp option tcplog backend ingress-https balance source mode tcp server compute-0 10.0.5.62:443 check server compute-1 10.0.5.63:443 check {% endhighlight %} Please ensure, that no services are listening on port 80 or 443 on your HAproxy machine (e.g. `lsof -i:80` or `ss -lnt sport = :80`) Restart HA-Proxy: `systemctl restart haproxy` *(optional)* Enable haproxy service for permanent use: `systemctl enable haproxy` ## HTTP Server for serving ignition files For serving the ignition file of the bootstrap node, we will setup a simple HTTP server on port 8080 on our loadbalancer host (lb.ocp4-cluster-001.int.consol.de). {% highlight bash %} $ yum install httpd -y {% endhighlight %} In `/etc/httpd/conf/httpd.conf`, change the listen port from `Listen 80` to `Listen 8080`. Restart httpd: `systemctl restart httpd` ## OCP pull secret For installing Red Hat OpenShift Container Platform 4.3, you'll need a pull secret: Link: [https://cloud.redhat.com/openshift/install/vsphere/user-provisioned](https://cloud.redhat.com/openshift/install/vsphere/user-provisioned){:target="_blank"} ## OCP openshift-installer From [https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/](https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/){:target="_blank"}, fetch the appropriate openshift-installer version and extract the binary to `~/bin/`, e.g. `openshift-install-linux-4.3.0.tar.gz`. ## oc client From [https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/](https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/){:target="_blank"}, fetch the appropriate oc version and extract the binary to `~/bin/`, e.g. `openshift-client-linux-4.3.0.tar.gz`. ## VMware vSphere / vCenter For a fully automated provisioning of your VMs on VMware vSphere, you'll need an vCenter account with the appropriate permissions and information: * vCenter URL * vCenter login * vCenter password * datacenter name * cluster name * datastore name A ressource group will be generated automatically and named like your OCP cluster id. ## OVA image The base image for all our CoreOS VMs will be a Red Hat CoreOS Image 4.3. This image can be found here: [https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/4.3.0/rhcos-4.3.0-x86_64-vmware.ova](https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.3/4.3.0/rhcos-4.3.0-x86_64-vmware.ova){:target="_blank"} # Installation ## OVA image upload The first step of the installation is to upload the OVA image to our vCenter datacenter and mark it as a template. All CoreOS VMs will be cloned from this template. Ensure you have the correct image downloaded: {% highlight bash %} $ ls -la rhcos-4.3.0-x86_64-vmware.ova -rw-rw-r-- 1 zisis zisis 829542400 Jan 27 12:59 rhcos-4.3.0-x86_64-vmware.ova {% endhighlight %} Set govc environment variables if you want to upload the image via govc. Alternatively you can upload the image via vCenter web console. {% highlight bash %} export GOVC_URL='ocp-vcenter.int.consol.de' export GOVC_USERNAME='administrator@vsphere.local' export GOVC_PASSWORD='mypassword' export GOVC_INSECURE=1 export GOVC_DATASTORE='vsanDatastore' {% endhighlight %} Check if your login works: {% highlight bash %} $ govc about Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 ... {% endhighlight %} Upload OVA image and mark as template. The name (rhcos-4.3.0) is important for later use, since we will reference the name of the template in our terraform plan (terraform.tfvars). Pull spec: {% highlight bash %} $ govc import.spec /home/zisis/rhcos-4.3.0-x86_64-vmware.ova | python -m json.tool > rhcos.json {% endhighlight %} Customize the Network you want to use in `rhcos.json`: {% highlight bash %} "NetworkMapping": [ { "Name": "VM Network", "Network": "VM Network" } {% endhighlight %} List resource pools: {% highlight bash %} $ govc find / -type p /Datacenter/host/cluster001/Resources /Datacenter/host/openshift-test-esx-3.int.consol.de/Resources {% endhighlight %} Upload the rhcos-4.3.0 image to the resource pool: {% highlight bash %} $ govc import.ova -options=./rhcos.json -name=rhcos-4.3.0 \ -pool=/Datacenter/host/cluster001/Resources /home/zisis/rhcos-4.3.0-x86_64-vmware.ova [27-01-20 17:27:21] Uploading disk.vmdk... OK {% endhighlight %} Mark the VM as a template: {% highlight bash %} $ govc vm.markastemplate vm/rhcos-4.3.0 {% endhighlight %} ## Ignition files (openshift-installer) The following steps will: * create an installation directory * create an install-config.yaml * create a backup of install-config.yaml, since the install-config.yaml will be deleted automatically * create the ignition files with openshift-installer For the install-config.yaml, you need the following input: * base domain * OCP cluster id * OCP pull secret * ssh ***public*** key (~/.ssh/id_rsa_ocp4_vcenter.pub) * vCenter host * vCenter user * vCenter password * vCenter datacenter * vCenter datastore Commands: {% highlight bash %} $ mkdir ocpinstall $ cd ocpinstall # create an install-config.yaml $ cat < install-config.yaml apiVersion: v1 baseDomain: int.consol.de metadata: name: ocp4-cluster-001 platform: vSphere: vcenter: ocp-vcenter.int.consol.de username: administrator@vsphere.local password: mypassword datacenter: Datacenter defaultDatastore: vsanDatastore pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"........"}}}' sshKey: ssh-rsa AAAAB3N..... EOF # make a backup $ cp install-config.yaml install-config.`date '+%s'`.bak # create ignition configs $ openshift-install create ignition-configs INFO Consuming "Install Config" from target directory {% endhighlight %} Now you should have a directory structure like this: {% highlight bash %} $ tree . ├── auth │   ├── kubeadmin-password │   └── kubeconfig ├── bootstrap.ign ├── install-config.1579194442.bak ├── master.ign ├── metadata.json └── worker.ign {% endhighlight %} Caveat: ignition files are valid for 24 hours - so if your installation takes longer than 24 hours due to issues, you have to generate new ignition files. ## Copy ignition files to your HTTP server Copy the generated bootstrap.ign file to your HTTP server and ensure, that the file can be downloaded with http: Copy: {% highlight bash %} $ scp bootstrap.ign http-user@lb.ocp4-cluster-001.int.consol.de:/var/www/html/ {% endhighlight %} Check, if download would succeed from your http server: {% highlight bash %} $ curl -I http://lb.int.consol.de:8080/bootstrap.ign HTTP/1.1 200 OK ... {% endhighlight %} ## Prepare the terraform installer For the installation, we will use a modified version of the UPI vSphere installer, which is available at: [https://github.com/openshift/installer/tree/master/upi/vsphere](https://github.com/openshift/installer/tree/master/upi/vsphere){:target="_blank"} Clone the repo and change to the upi vSphere directory: {% highlight bash %} $ git clone https://github.com/openshift/installer.git $ cd installer/upi/vsphere {% endhighlight %} Copy the example file to terraform.tfvars an adjust the variables. In sections `control_plane_ignition / END_OF_MASTER_IGNITION` and `compute_ignition / END_OF_WORKER_IGNITION`, insert / copy&paste the contents of the ignitions files (master and worker) we generated before: * END_OF_MASTER_IGNITION -> master.ign * END_OF_WORKER_IGNITION -> worker.ign Our terraform.tfvars: {% highlight bash %} cluster_id = "ocp4-cluster-001" cluster_domain = "ocp4-cluster-001.int.consol.de" base_domain = "int.consol.de" vsphere_server = "ocp-vcenter.int.consol.de" vsphere_user = "administrator@vsphere.local" vsphere_password = "mypassword" vsphere_cluster = "cluster001" vsphere_datacenter = "Datacenter" vsphere_datastore = "vsanDatastore" vm_template = "rhcos-4.3.0" machine_cidr = "10.0.5.0/24" vm_network = "VM Network" control_plane_count = 3 compute_count = 2 bootstrap_ignition_url = "http://lb.int.consol.de:8080/bootstrap.ign" control_plane_ignition = < In this phase, the bootstrap node is set up, which in turn will setup the cluster: {% highlight bash %} $ cd ~/ocpinstall/ $ openshift-install --dir=. wait-for bootstrap-complete --log-level debug DEBUG OpenShift Installer v4.3.0 DEBUG Built from commit 2055609f95b19322ee6cfdd0bea73399297c4a3e INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp4-cluster-001.int.consol.de:6443... DEBUG Still waiting for the Kubernetes API: the server could not find the requested resource ... DEBUG Still waiting for the Kubernetes API: the server could not find the requested resource DEBUG Still waiting for the Kubernetes API: Get https://api.ocp4-cluster-001.int.consol.de:6443/version?timeout=32s: EOF INFO API v1.16.2 up INFO Waiting up to 30m0s for bootstrapping to complete... DEBUG Bootstrap status: complete INFO It is now safe to remove the bootstrap resources ... {% endhighlight %} If you want to see more details about installers progress, you can start a `tail -f` on `~/ocpinstall/.openshift_install.log`. Since the bootstrap node has given control over to the forming cluster, we can remove the bootstrap node: {% highlight bash %} $ cd installer/upi/vsphere $ terraform apply -auto-approve -var 'bootstrap_complete=true' {% endhighlight %} After the bootstrap node was removed, you can adjust your loadbalancer and remove the bootstrap node from your dispatcher configuration. The OCP4 cluster will continue to finish its installation by using a lot of cool operators: {% highlight bash %} $ cd ~/ocpinstall/ $ openshift-install --dir=. wait-for install-complete INFO Waiting up to 30m0s for the cluster at https://api.ocp4-cluster-001.int.consol.de:6443 to initialize... INFO Waiting up to 10m0s for the openshift-console route to be created... INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/zisis/ocpinstall/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4-cluster-001.int.consol.de INFO Login to the console with user: kubeadmin, password: AHTUN-2vGkj-Fo0Ba-Ctae9 {% endhighlight %} A new star is born. ## Kubeconfig for oc commands For being able to execute oc commands, you have to use the kubeconfig, which was generated by the installer. You either use this kubeconfig by exporting the environment variable KUBECONFIG or copy the file auth/kubeconfig to your home directory in .kube/config (be sure not to overwrite any other kubeconfig): {% highlight bash %} $ export KUBECONFIG=~/ocpinstall/auth/kubeconfig {% endhighlight %} or {% highlight bash %} $ cp ~/ocpinstall/auth/kubeconfig ~/.kube/config {% endhighlight %} ## Wait for clusteroperators Wait for cluster operators to get ready: {% highlight bash %} $ oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.3.0 True False False 26h cloud-credential 4.3.0 True False False 27h cluster-autoscaler 4.3.0 True False False 26h console 4.3.0 True False False 26h dns 4.3.0 True False False 26h image-registry 4.3.0 True False False 26h ingress 4.3.0 True False False 26h insights 4.3.0 True False False 26h kube-apiserver 4.3.0 True False False 26h kube-controller-manager 4.3.0 True False False 26h kube-scheduler 4.3.0 True False False 26h machine-api 4.3.0 True False False 26h machine-config 4.3.0 True False False 26h marketplace 4.3.0 True False False 26h monitoring 4.3.0 True False False 26h network 4.3.0 True False False 26h node-tuning 4.3.0 True False False 26h openshift-apiserver 4.3.0 True False False 26h openshift-controller-manager 4.3.0 True False False 26h openshift-samples 4.3.0 True False False 26h operator-lifecycle-manager 4.3.0 True False False 26h operator-lifecycle-manager-catalog 4.3.0 True False False 26h service-ca 4.3.0 True False False 26h service-catalog-apiserver 4.3.0 True False False 26h service-catalog-controller-manager 4.3.0 True False False 26h storage 4.3.0 True False False 26h {% endhighlight %} Check your nodes: {% highlight bash %} $ oc get nodes NAME STATUS ROLES AGE VERSION compute-0 Ready worker 8m2s v1.14.6+cebabbf4a compute-1 Ready worker 7m54s v1.14.6+cebabbf4a control-plane-0 Ready master 10m13s v1.14.6+cebabbf4a control-plane-1 Ready master 11m26s v1.14.6+cebabbf4a control-plane-2 Ready master 10m54s v1.14.6+cebabbf4a {% endhighlight %} ## Setup image registry (postinstall) In order to complete the installation, you need to add storage to the image registry. For test setups, you can set this to emptyDir (for more permanent storage, please see the official documentation for more information). So after a successful installation, only the image registry operator should be running in the openshift-image-registry namespace: {% highlight bash %} $ oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-f9697f69d-n47j8 2/2 Running 0 62s {% endhighlight %} Patch the image registry operator to use an emptyDir as storage: {% highlight bash %} $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' config.imageregistry.operator.openshift.io/cluster patched {% endhighlight %} *(for OCP 4.3)* When there is no Object Storage available at initial setup of the registry, you will have to set manually the managementState for OpenShift bringing up the image registry successfully, see [https://docs.openshift.com/container-platform/4.3/registry/configuring-registry-storage/configuring-registry-storage-vsphere.html#registry-removed_configuring-registry-storage-vsphere](https://docs.openshift.com/container-platform/4.3/registry/configuring-registry-storage/configuring-registry-storage-vsphere.html#registry-removed_configuring-registry-storage-vsphere){:target="_blank"}: {% highlight bash %} $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState": "Managed"}}' config.imageregistry.operator.openshift.io/cluster patched {% endhighlight %} Now you should see the registry coming up: {% highlight bash %} $ oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-f9697f69d-n47j8 2/2 Running 0 3m54s image-registry-7766c98447-pgdwp 0/1 ContainerCreating 0 65s image-registry-9fd47f8fb-gs5kc 0/1 ContainerCreating 0 65s node-ca-2g5j9 0/1 ContainerCreating 0 66s node-ca-2l85v 0/1 ContainerCreating 0 66s node-ca-798mx 0/1 ContainerCreating 0 66s node-ca-nmrhw 0/1 ContainerCreating 0 66s node-ca-wcp5g 0/1 ContainerCreating 0 66s {% endhighlight %} # Uninstall cluster If you want to remove the cluster, you can use the terraform destroy command: {% highlight bash %} $ cd installer/upi/vsphere $ terraform destroy -auto-approve {% endhighlight %} # Additional information Since your VMware administrator will ask you for sure about the required permissions for the installation, maybe this is a good list to start the discussion. I'm sure, not all of these permissions are needed for the installation, so if you have time, strip it down ;) vSphere roles / permissions: {% highlight bash %} Datastore Allocate space Low level file operations Folder Create folder Delete folder Network Assign network Resource Assign vApp to resource pool Assign virtual machine to resource pool Create resource pool Remove resource pool vApp Clone View OVF environment vApp application configuration vApp instance configuration vApp resource configuration Virtual machine Change Configuration Acquire disk lease Add existing disk Add new disk Add or remove device Advanced configuration Change CPU count Change Memory Change Settings Change Swapfile placement Change resource Configure Host USB device Configure Raw device Configure managedBy Display connection settings Extend virtual disk Modify device settings Query Fault Tolerance compatibility Query unowned files Reload from path Remove disk Rename Reset guest information Set annotation Toggle disk change tracking Toggle fork parent Upgrade virtual machine compatibility Edit Inventory Create from existing Create new Move Register Remove Unregister Guest operations Guest operation alias modification Guest operation alias query Guest operation modifications Guest operation program execution Guest operation queries Interaction Answer question Backup operation on virtual machine Configure CD media Configure floppy media Connect devices Console interaction Create screenshot Defragment all disks Drag and drop Guest operating system management by VIX API Inject USB HID scan codes Install VMware Tools Pause or Unpause Perform wipe or shrink operations Power off Power on Record session on virtual machine Replay session on virtual machine Reset Resume Fault Tolerance Suspend Suspend Fault Tolerance Test failover Test restart Secondary VM Turn off Fault Tolerance Turn on Fault Tolerance Provisioning Allow disk access Allow file access Allow read-only disk access Allow virtual machine download Allow virtual machine files upload Clone template Clone virtual machine Create template from virtual machine Customize guest Deploy template Mark as template Mark as virtual machine Modify customization specification Promote disks Read customization specifications {% endhighlight %} # Contact If you have any questions or remarks, feel free to contact me via zisis(dot)lianas(at)consol(dot)de or [Twitter](https://twitter.com/zisisli){:target="_blank"}. If you need professional OpenShift support, visit us at [https://www.consol.de/it-consulting/openshift/](https://www.consol.de/it-consulting/openshift/){:target="_blank"}. [Read more](https://omd.consol.de/blog/2020/01/31/openshift-4.3-installation-on-vmware-vsphere-with-static-ips/) ----------------------------------------------------------------------------------- title: "Installing MongoDB on OpenShift" date: "2020-01-14" --------------------------------------------------------------------------------
So here is another one of our series *Installing Blahblahblah on OpenShift*. This time it is about getting MongoDB to run on OpenShift - the way recommended and promoted by the MongoDB guys. The whole setup is still in beta stage as indicated on [these](https://access.redhat.com/containers/?tab=overview#/registry.connect.redhat.com/mongodb/mongodb-enterprise-ops-manager) [two](https://access.redhat.com/containers/?tab=overview#/registry.connect.redhat.com/mongodb/mongodb-enterprise-appdb) entries in Red Hat's container image catalog. You *can* get your MongoDB instance up and running on OpenShift. But most of the required steps have to be performed on the command line, contrary to the impression given by MongoDB, Inc that once you get the MongoDB Operations Manager up and running everything can be achieved via this tool's GUI. Some operations in the Operations Manager simply do not work (yet) on OpenShift. When I last had to work with MongoDB back in 2015 and 2016 (version 3.0 and 3.2) installing this NoSQL database was a matter of running a single installer. You got a standalone instance nearly instantly that was ready for use. The downside was that tens of thousands of MongoDB installations where [accessible via the Internet with default passwords](https://uds.cispa.saarland/wp-content/uploads/2015/02/MongoDB_documentation.pdf). Since then a lot has changed. MongoDB has become far more enterprise-ish. Your MongoDB deployments are now administered with the *MongoDB Operations Manager* that takes advantage of hosts running the *MongoDB Agent*. In the Kubernetes / OpenShift world for getting a running Operations Manager and a bunch of pods equipped with the MongoDB Agent requires dealing with the *MongoDB Enterprise Operator for Kubernetes*. I got most of my input from the following documentations: * [MongoDB Enterprise Operator for Kubernetes Documentation](https://docs.mongodb.com/kubernetes-operator/) * [OpenShift Installation of the MongoDB Enterprise Operator for Kubernetes](https://github.com/mongodb/mongodb-enterprise-kubernetes/blob/master/openshift-install.md) * [MongoDB Operations Manager Documentation](https://docs.opsmanager.mongodb.com/) Initially I had also included the blog article [Introducing the MongoDB Enterprise Operator for Kubernetes](https://www.mongodb.com/blog/post/introducing-the-mongodb-enterprise-operator-for-kubernetes) to this list. But be warned. It's pretty messed up. You won't get far with it on its own. Be careful not to get confused as some notions exist in the OpenShift world and the MongoDB world with distinct meanings: cluster, replica set, project, deployment. The documentation found in the GitHub project of the MongoDB Enterprise Operator about installing the operator on OpenShift turned out to be not very accurate. I've created a [fork](https://github.com/makuhama/mongodb-enterprise-kubernetes) and updated the documentation. Hopefully this update will sometime find its way back into the parent project. The Operations Manager Documentation does not reference the shortcomings of running this tool on Kubernetes / OpenShift. You have to find out yourself and how to work around these drawbacks. I performed the installation on an OpenShift v3.11 cluster as this will be the customer's target platform. As the procedure is operator based I expect this also works on OpenShift v4.x. OpenShift versions earlier than 3.11 are definitely out of the game. ## Project and custom resources Start with a blank project. Either create it yourself (sufficient permissions provided) or ask your friendly cluster administrator to create it for you. We will call it `mongodb`. ``` oc login https:// oc new-project mongodb ``` Create a bunch of custom resource definitions (CRDs) used by the MongoDB Operator. ``` oc create -f https://github.com/mongodb/mongodb-enterprise-kubernetes/raw/master/crds.yaml ``` ## Pull secret to access Red Hat registries Next is an OpenShift special. Prepare a secret with your credentials you need to pull images from the registries `registry.redhat.io` and `registry.connect.redhat.com`. To achieve this visit [https://access.redhat.com/terms-based-registry/](https://access.redhat.com/terms-based-registry/), choose the appropriate account and download the OpenShift pull secret (to be found under the *OpenShift Secret* tab). This secret has one entry with key `.dockerconfigjson`. The value is base64 encoded. Use your favorite text editor to extract this value into a separate file. Let's call it `dockerconfig.b64`. Decode the value with ``` base64 -d < dockerconfig.b64 | jq . > dockerconfig.json ``` The resulting file `dockerconfig.json` should look similar to ```json { "auths": { "registry.redhat.io": { "auth": "RNVpqSTBPVEV3WldZMFl6ZGh..." } } } ``` This is the access token needed to access `registry.redhat.io`. In order to be able to also access `registry.connect.redhat.com` duplicate the entry with name `registry.redhat.io` and change the key to `registry.connect.redhat.com`. You should end up with something like ```json { "auths": { "registry.redhat.io": { "auth": "RNVpqSTBPVEV3WldZMFl6ZGh..." }, "registry.connect.redhat.com": { "auth": "RNVpqSTBPVEV3WldZMFl6ZGh..." } } } ``` Don't forget the comma between the two entries under `auths`! We now create the definition of an OpenShift secret with this JSON file as value. ```yaml apiVersion: v1 kind: Secret metadata: name: openshift-pull-secret stringData: .dockerconfigjson: | { "auths": { "registry.redhat.io": { "auth": "RNVpqSTBPVEV3WldZMFl6ZGh..." }, "registry.connect.redhat.com": { "auth": "RNVpqSTBPVEV3WldZMFl6ZGh..." } } } type: kubernetes.io/dockerconfigjson ``` We assume that you store this YAML file as `openshift-pull-secret.yaml`. Finally create this secret with ``` oc create -f openshift-pull-secret.yaml ``` ## Create the MongoDB Enterprise Operator Create the MongoDB Enterprise Operator and a few associated resources (service accounts, roles, role bindings) with ``` oc create -f https://github.com/mongodb/mongodb-enterprise-kubernetes/raw/master/mongodb-enterprise-openshift.yaml ``` The pod with name `enterprise-operator-xxxx-yyy` will fail to start due to missing permissions when pulling the image. Link the pull secret to some service accounts and restart the pod ``` oc secret link default openshift-pull-secret --for=pull oc secret link enterprise-operator openshift-pull-secret --for=pull oc secret link mongodb-enterprise-appdb openshift-pull-secret --for=pull oc delete $(oc get pod -l=app=enterprise-operator -o name) ``` Make sure the pod starts properly. The MongoDB Enterprise Operator is now in place. So far we have reached step 1 of the bootstrap procedure to get MongoDB up and running on OpenShift. Next is to get a usable Operations Manager. ## Install the MongoDB Operations Manager First we have to prepare two secrets with parameters for the Operations Manager. One is for the credentials of the super-user of the Operations Manager. ```yaml apiVersion: v1 kind: Secret stringData: FirstName: Operations LastName: Manager Password: Username: opsman type: Opaque metadata: name: opsman-admin-credentials ``` Store this in a file called `opsman-admin-credentials.yaml`. The given values are just an example. Choose whatever you prefer. Mind that the password must match the Operations Manager's password policy, i.e. 8 characters minimum, one letter minimum, one digit minimum, one special character minimum. This is documented nowhere except in the Operations Manager GUI that you only get to see in case you managed to configure a password that complies to the above rules. In case you intend to use LDAP or SAML for user authentication in Operations Manager (instead of a user database provided by Operations Manager itself) keep in mind that the username given above must already exist in LDAP or SAML. (Again this is documented nowhere and you have to learn the hard way when its too late.) You need another secret with the password Operations Manager uses to access its own database called *appdb*. ```yaml apiVersion: v1 kind: Secret stringData: password: type: Opaque metadata: name: opsman-db-password ``` Store this in a file called `opsman-db-password.yaml`. Now we create a custom resource of type `MongoDBOpsManager` that will trigger the MongoDB Enterprise Operator and make it install the Operations Manager. ```yaml apiVersion: mongodb.com/v1 kind: MongoDBOpsManager metadata: name: ops-manager spec: replicas: 1 version: 4.2.4 adminCredentials: opsman-admin-credentials backup: enabled: false applicationDatabase: members: 3 version: 4.2.0 passwordSecretKeyRef: name: opsman-db-password ``` Store this in a file called `opsman-instance.yaml`. For the time being we will keep things simple and omit the backup feature as it introduces a bunch of additional resources that we have to provide. The versions of the Operations Manager and its application database seem to be more or less in sync. Have a look at Red Hat's image catalog [here](https://access.redhat.com/containers/?tab=tags#/registry.connect.redhat.com/mongodb/mongodb-enterprise-ops-manager) and [here](https://access.redhat.com/containers/?tab=tags#/registry.connect.redhat.com/mongodb/mongodb-enterprise-appdb) on what's available. I had to try a little bit to find the above combination that actually worked. For instance using the combination 4.2.4 / 4.2.3 produced the following error message in the log output of the appdb pods ``` Cluster config did not pass validation for pre-expansion semantics : MongoDB version 4.2.3 for process = ops-manager-db-0 was not found in the list of available versions ``` followed by a very long list of available tarballs for various versions of MongoDB for various platforms. Apparently MongoDB's own download server is not in sync with Red Hat's container image catalog (or the other way round). Now it's time to create all that in OpenShift and trigger the Operator. ``` oc create -f opsman-admin-credentials.yaml oc create -f opsman-db-password.yaml oc apply -f opsman-instance.yaml ``` We intentionally use `oc apply` here to be able to modify the MongoDBOpsManager resource later, e.g. to add backup functionality. Watch the installation unfold, e.g. with `oc get pods -w` or `oc logs ops-manager-0 -f`. Be patient as it might take a couple of minutes until the Operations Manager is operational. Create a route to make the Operations Manager's GUI accessible from the outside. ``` oc expose svc ops-manager-svc ``` ## First time setup of Operations Manager Access the GUI and log in with the credentials given with the above secret `opsman-admin-credentials.yaml`. You will be confronted with a first time setup of this tool. We won't go into details. Finally your MongoDB Operations Manager is running and usable. We have completed step 2 of the setup procedure. Additional users might now register themselves via the *Register* link on the login page. ## Install a MongoDB instance The obvious way to get an instance of MongoDB up and running would be 1. Create an Operations Manager organization. 2. Create an Operations Manager project within the organization. 3. Create a MongoDB deployment (an actual database) within the project. all via the Operations Manager GUI. The required elements exist and are functional. Unfortunately the resulting project will be no good in the OpenShift world. The deployment will be rejected with more or less helpful error messages. As [stated](https://docs.mongodb.com/kubernetes-operator/stable/tutorial/create-operator-credentials/) in the Operator's documentation > Unlike earlier Kubernetes Operator versions, use the Operator to create your Ops Manager project. The Operator adds additional metadata to Projects that it creates to help manage the deployments. you have to create the organization via the GUI. The rest has to be accomplished on the command line. So make sure you are logged into the Operations Manager GUI. Click on the very top right *Operations* and *Organizations*. Click the green Button *NEW ORGANIZATION*. Give a name (we choose *test-orga* here). On the next page you have the option to add additional users to your organization with certain roles. Click *Create Organization* and the organization has come into being. You now need the organization ID. Go to the page of your newly created organization (e.g. via *Operations* in the top right corner, *Organizations* and then the corresponding entry in the list of organizations). Click *Settings* in the left menu column and note down the organization ID given on that page. Next you must create an API key that allows the Operator to access the Operations Manager. Click *Access* on the left and then select the tab *API keys*. Here click the big green button *Create API Key*. Note down the 8 character code below *Public Key*. Give a name to the API key and choose a permission - either *Organization Owner* or *Organization Project Creator*. Click *Next*. Here is the only time when you see the private key completely in clear text. Write it down - you will need it shortly. Add a whitelist entry to allow the operator to access the Operations Manager API. Use the Operations Manager pod ID address. You can get it with ``` oc get pod -l app=enterprise-operator -o jsonpath='{.items[0].status.podIP}' ``` (You may have realized that this white list doesn't work anymore as soon as the Operator gets restarted, i.e. it's run in a pod with a different IP address. You may specify the IP address range used for pods (e.g. 10.0.0.0/8). The problem is that it's impossible for mere mortals to determine this IP address range.) Now it's time to create a secret and a config map in preparation of the deployment of a real MongoDB instance. The secret holds the credentials to access the Operations Manager API. ```yaml apiVersion: v1 kind: Secret stringData: user: publicApiKey: type: Opaque metadata: name: test-orga-api-key ``` Store this in a file named `test-orga-api-key.yaml`. The config map has information about the project to create (provided it doesn't already exist), the organization where the project resides in and the URL to access the Operations Manager. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: test-project-config data: projectName: "Test Project" orgId: baseUrl: http://..svc.cluster.local:8080 ``` Store this in a file named `test-project-config.yaml`. The last resource we have to prepare is of type `MongoDB`. ```yaml apiVersion: mongodb.com/v1 kind: MongoDB metadata: name: test-replica-set spec: members: 3 version: 4.2.0 opsManager: configMapRef: name: test-project-config credentials: test-orga-api-key type: ReplicaSet persistent: true ``` Store this in a file named `mongodb-instance.yaml`. Finally we create the secret, the config map and the MongoDB resource in OpenShift. The latter will trigger the Operator and make it create the MongoDB replica set. ``` oc create -f test-orga-api-key.yaml oc create -f test-project-config.yaml oc create -f mongodb-instance.yaml ``` As with the Operations Manager it takes a couple of minutes until the MongoDB instance is fully deployed. You can watch it unfold with `oc get pods -w`. Or you can use the Operations Manager GUI to monitor your project and MongoDB replica set. Pew! A hell lot of fiddling! As stated above running the Operations Manager on OpenShift is still beta and requires quite a few manual steps to get things up and running. ## Accessing the database To connect to this replica set (from within the OpenShift cluster) you have to provide an URL with credentials and a list of all members, e.g. ``` mongodb://:@test-replica-set-0.test-replica-set-svc.mongodb.svc.cluster.local,test-replica-set-1.test-replica-set-svc.mongodb.svc.cluster.local,test-replica-set-2.test-replica-set-svc.mongodb.svc.cluster.local ``` ## Cleanup Remove all files with sensitive data * `openshift_pull_secret.yaml` * `opsman_admin_credentials.yaml` * `opsman_db_passwd.yaml` * `test-orga-api-key.yaml` ## Final considerations The Operations Manager is still beta for OpenShift. As already mentioned this has an impact on how you create Operations Manager projects and MongoDB deployments. Additionally I had mixed experiences when trying to delete MongoDB deployments. This is supposed to be accomplished by deleting the corresponding `mongodb` resource. I tried it twice. One time with a lot of clicking around in the Operations Manager GUI to see the impact of this operation. The result was a totally confused Operations Manager. Not even restarting the Operations Manager healed the problem. Apparently the application DB was corrupted. The second time I left the Operations Manager GUI untouched for quite a while. In the end the deployment appeared to be (mostly) removed. A lingering server was still reported on the deployment overview page. And I was irritated by the message *One or more agents are out of date* on top of the page. Nevertheless I was able to deploy another replication set and then everything was fine again. On a more general level it has to be asked whether running MongoDB within OpenShift is actually advisable. After deploying the Operations Manager I persistently got emails with ``` ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. ** We suggest setting it to 'never' ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. ** We suggest setting it to 'never' ``` I don't think it is a good idea to tweak the kernel's memory page settings of your cluster nodes just because you have MongoDB running somewhere. It might impact the performance of ordinary (i.e. non-DB) applications. Additionally if performance of your MongoDB deployments is of concern (and it probably is) then running MongoDB within your OpenShift cluster using volumes provided by some cloud storage solution is not the ideal setup. You better stick to bare metal, i.e. physical disks accessed via some hardware interface. That said it still might be a viable option to run your Operations Manager in OpenShift dutifully managing your MongoDB installation on some physical machines. Mind that the statement to prefer bare metal installations does not only apply to MongoDB but basically to all database-like applications. [Read more](https://omd.consol.de/blog/2020/01/14/installing-mongodb-on-openshift/) ----------------------------------------------------------------------------------- title: "CodeReady Containers on Ubuntu" date: "2019-11-29" --------------------------------------------------------------------------------
With the release of OpenShift 4.x Red Hat left no stone unturned (compared to previous 3.x versions). Among many things [Minishift](https://www.okd.io/minishift/) became [Red Hat CodeReady Containers](https://developers.redhat.com/products/codeready-containers). Having been a big fan of *Minishift* I recently wanted to give *CodeReady Containers* (aka CRC) a try. Turned out this is not that easy - at least if you want to run CRC on a Linux that does not come from Red Hat (or its community). This article gives instructions for all those people out there who want to run *CodeReady Containers* on Ubuntu. **Update 2020-12-17:** According to [this comment](https://github.com/code-ready/crc/issues/549#issuecomment-747434667) on GitHub by one of the maintainers / developers of [Red Hat CodeReady Containers](https://developers.redhat.com/products/codeready-containers) the issues with Ubuntu have been resolved in the latest version of CRC. First lesson I had to learn was that *CodeReady Containers* does not run with VirtualBox - at least not on Linux. The only supported virtualization technology (on Linux) is *KVM* in combination with *libvirt*. I've been happily using VirtualBox for years. But the prospect of getting a virtualization with near-native performance made me switch to *KVM / libvirt*. Next obstacle: > On Linux, CodeReady Containers is only supported on Red Hat Enterprise Linux/CentOS 7.5 or newer (including 8.x versions) and on the latest two stable Fedora releases. That's what you can read in the [Getting Started Guide](https://access.redhat.com/documentation/en-us/red_hat_codeready_containers/1.0/html/getting_started_guide/getting-started-with-codeready-containers_gsg) of *CodeReady Containers*. And further on: > Ubuntu 18.04 LTS or newer and Debian 10 or newer are not officially supported and may require manual set up of the host machine. That's sad. I have been using Ubuntu (Kubuntu to be precise) for at least ten years and contrary to the virtualization technology I was not willing to easily give up my beloved working horse. Dastardly the Red Hat guys give no further hint on what *manual set up of the host machine* actually means. Surprisingly I also couldn't find any instructions on the Internet about this issue. I had to find out myself. This is the conclusion of my findings: According to the *Getting Started Guide* you have to install a bunch packages in preparation of setting up CRC. I found out that it is sufficient to only install one package. The list given by Red Hat is automatically pulled in via dependencies. This has the advantage that you can get rid of the whole virtualization by removing just this single one package (probably doing a `apt autoremove` afterwards). As `root` do ``` # apt install virt-manager ``` I found out that you have to restart your machine after that to be on the safe side. (Probably it would be much faster just to start or restart two or three services with `systemctl`. But I was not motivated to dig into that detail. So I just recommend a restart.) Next download the CRC executable. Similar to *Minishift* there is no true installer. Just a big self-contained executable that does all the heavy lifting. Extract the downloaded TAR archive and move the `crc` executable to an appropriate place. Make sure that this place (i.e. folder) is covered by your `$PATH` variable. (I use `$HOME/bin` for those instances. On Ubuntu it is automatically added to your `$PATH` if it exists.) ``` wget https://mirror.openshift.com/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz tar -xvJf crc-linux-amd64.tar.xz mv crc-linux-1.2.0-amd64/crc $HOME/bin rm -rf crc-linux-amd64.tar.xz crc-linux-1.2.0-amd64 ``` The exact name of the folder `crc-linux-1.2.0-amd64` will probably change in the future. You also need the so-called *pull secret*. It's a collection of personal access tokens for 4 different image registries. You can get it via ``` https://cloud.redhat.com/openshift/install/crc/installer-provisioned ``` Log in with your Red Hat account and download your pull secret. It's a JSON file. You will need its contents shortly. Now it's time for the actual setup of CRC. (Do not run this command as root!) ``` crc setup ``` You will be asked for your linux password in the course of the procedure. This allows `crc setup` to manipulate your system's DNS setup. Mind the two output lines ``` INFO Will use root access: write NetworkManager config in /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf INFO Will use root access: write dnsmasq configuration in /etc/NetworkManager/dnsmasq.d/crc.conf ``` (Nearly) everything is in place now. We can try to start the local OpenShift cluster. ``` crc start --nameserver ``` I highly recommend to specify your local nameserver with the option `--nameserver`. Otherwise CRC will use Google's DNS server 8.8.8.8. You will be asked to provide the contents of your pull secret during the first run of `crc start`. Just copy and paste the whole JSON. So far everything was in line with the instructions given in the *Getting Started Guide*. Unfortunately `crc start` ends with the following error message: ``` .... INFO Starting OpenShift cluster ... [waiting 3m] ERRO Error approving the node csr Not able to get csr names (exit status 1 : Unable to connect to the server: dial tcp: lookup api.crc.testing: no such host ``` This is where the subtle differences between Red Hat Linux and Ubuntu come into play. Both distributions use systemd and NetworkManager. But Ubuntu uses systemd-resolved for name resolution while Red Hat Linux uses dnsmasq. `crc setup` blindly assumes a Red Hat Linux and happily patches NetworkManager's setup to give NetworkManager control over dnsmasq. (Remember the two above mentioned output lines?) This fails miserably on Ubuntu. The first config file (`crc-nm-dnsmasq.conf`) makes sure that dnsmasq is started under the control of NetworkManager while the second file (`crc.conf`) delegates name resolution for names ending in `apps-crc.testing` or `crc.testing` to a DNS server running in the local OpenShift cluster. The very same setup could be achieved by extending the configuration of systemd-resolved. It would have been no big deal for `crc start` to check what kind of name resolution is already in place and either patch the configuration of NetworkManager or systemd-resolved. Unfortunately this is not the case. Even worse `crc start` checks every time whether `crc-nm-dnsmasq.conf` and `crc.conf` are still in place and were not modified. So the only solution is to disable systemd-resolved and give name resolution in the hands of dnsmasq. This is fairly easy to achieve. (This has to be done as root.) ``` # systemctl disable systemd-resolved.service # rm /etc/resolv.conf # systemctl restart NetworkManager ``` To be honest I feel a bit uncomfortable with this solution. I would have guessed dnsmasq being available a bit later than systemd-resolved and thus expected problems during system start. But so far I could not observe any problems. So now it's time to restart your CRC. ``` crc stop # ... wait some time and make sure you VM has actually terminated crc start --nameserver ``` The output should end with ``` INFO To access the cluster, first set up your environment by following 'crc oc-env' instructions INFO Then you can access it by running 'oc login -u developer -p developer https://api.crc.testing:6443' INFO To login as an admin, username is 'kubeadmin' and password is XXXXX-XXXXX-XXXXX-XXXXX INFO INFO You can now run 'crc console' and use these credentials to access the OpenShift web console Started the OpenShift cluster WARN The cluster might report a degraded or error state. This is expected since several operators have been disabled to lower the resource usage. For more information, please consult the documentation ``` I haven't used `crc oc-env` as recommended but added the `oc` command to `$HOME/bin`. ``` cd $HOME/bin ln -s ../.crc/bin/oc oc ``` Have fun with your CRC cluster! I did all my investigations on Kubuntu 18.04.3. I guess that my findings also apply for later versions of Ubuntu as the general setup of name resolution has not changed significantly. **Update 2020-01-28** After several weeks I found one drawback of this setup. Ordinary docker containers refuse to find any hosts by name. Some investigation revealed that this is due to `/etc/resolv.conf` looking like ``` # Generated by NetworkManager search nameserver 8.8.8.8 nameserver 8.8.4.4 ``` See docker's [documentation](https://docs.docker.com/v17.09/engine/userguide/networking/default_network/configure-dns/) about the nifty details. The nameservers `8.8.8.8` and `8.8.4.4` are inserted as a last resort in case docker's logic when composing the container's `/etc/resolv.conf` left no name servers. I my corporate environment DNS traffic to external nameservers is blocked. So `8.8.8.8` and `8.8.4.4` are of no use. The workaround is to use docker's command line option `--dns=...`, i.e. ``` docker run -it --dns= busybox /bin/sh ``` [Read more](https://omd.consol.de/blog/2019/11/29/codeready-containers-on-ubuntu/) ----------------------------------------------------------------------------------- title: "Introduction to AWS CDK" date: "2019-11-04" --------------------------------------------------------------------------------
AWS Cloud Development Kit (CDK) is a relatively new kid on the block. It is a tool for defining Infrastructure as Code (IaC) and is considered to be the future successor of AWS CloudFormation. This article overviews the IaC approach, introduces a reader to the AWS CDK, shows what problems it aims to solve and presents a simple example application implemented with it. ## tl;dr * IaC is all about automating the provisioning process of an infrastructure. * This automation helps you iterate faster. * AWS CDK has reached a stable 1.x version. However, at the time of writing many of its comprising modules are still experimental. * CloudFormation is still the "official" answer to the question "How do I do IaC in AWS?" * If you are using CDK and unhappy with it, you can let CDK generate CloudFormation templates for you, commit them in git, drop CDK and use CloudFormation for further development. * One of the main reasons for using CDK is that it enables you to use your favorite programming language in your favorite IDE and apply knowledge and skills that you already have. ## Infrastructure as Code [Infrastructure as Code][iac-aws-whitepaper] AWS whitepaper describes IaC the following way: > Rather than relying on manually performed steps, both administrators and developers can instantiate infrastructure using configuration files. Infrastructure as Code treats these configuration files as software code. These files can be used to produce a set of artifacts, namely the compute, storage, network, and application services that comprise an operating environment. Infrastructure as Code eliminates configuration drift through automation, thereby increasing the speed and agility of infrastructure deployments. So the idea is that you describe your VPC configuration, RDS instances, S3 buckets, API gateway endpoints and other AWS resources in text files, store them in git and then use a tool which reads them and provisiones the corresponding cloud resources (i.e. the infrastructure) for you:
Since the infrastructure configuration files are stored in a git repository they are treated essentially as code (which may be stored in the same repository) and hence the name -- Infrastructure as Code (IaC). This IaC approach gives you the following advantages: * git gives you versioning out of the box. You can e.g. tag commits according to some versioning scheme such as [semantic versioning][semver]. * If you already use [Gitflow][gitflow] on your project, you can apply it to your infrastructure as well because it "lives" now in a repository. And, in general, all your git-related skills, processes and operations can be "reused" i.e. you can review / merge / tag / revert / cherry-pick your infrastructure. * Human errors are minimized. Since your infrastructure is created automatically by a tool, the risk of making a typo is much lower. * Faster development iterations can be achieved because you do not need to create your infrastructure manually but rather it is created automatically for you. Saved time can be invested in development and other important job. ## Short History of IaC In Context of AWS I'll try to draw analogy between development of IaC approaches and time periods of human history in a humorous way.
In the Early Middle Ages people used to rely on bash scripting heavily. Developers would script AWS CLI commands, commit them to git and then let a Jenkins pipeline run them. This old technique has all the key characteristics of an IaC approach: * Provisioning of an infrastructure is automated. * The bash scripts are stored in git and hence the infrastructure is treated as code. * The scripts are versioned by git and thus the infrastructure is also versioned. Aside from the Great Plague, poor hygiene and permanent wars, back in those dismal days there were also the following problems: * Ugly echo-based debugging sessions with bash scripts. * Inability to run bash scripts locally if you happened to have Windows. * Constant googling "How to implement X in bash?" and applying trial and error approach to writing bash scripts. * Difficulties with supporting and developing large code bases of interdependent bash scripts. * Weak or no code completion and inspection support in your IDE.
A little later, in the Late Middle Ages the idea of using the boto3 started to spread in masses. [boto3][boto3] is a Python library which provides API to AWS cloud services. At that time people used to write Python scripts which gave advantages over bash scripting such as: * Better IDE support. It was still not perfect because Python is dynamically typed. * Better cross-platformness thanks to Python's ability to run on all major OSes. * Less shooting oneself in the foot because of [bash gotchas][bash-gotchas]. Things got a little bit better. However it was still a scripting-based imperative IaC solution but this time with Python instead of bash.
And then the Industrial Age came with the invention of the ~~steam engine~~ [CloudFormation][cloudformation]. CloudFormation was and currently still is the "official" answer to the question "How do I do IaC in AWS?" (we'll get to that later). Some of the great advantages of CloudFormation are: * CloudFormation introduced the concept of stacks which are basically bundles of AWS resources. These bundled resources can be deleted all at once by deleting the corresponding stack. * CloudFormation checks whether you've set all the necessary properties to the AWS resources and if something is missing no resources will be provisioned. Back in the Late Middle Ages you would have an exception in the middle of a script and your infrastructure would have been provisioned only partially i.e. it would have been in an inconsistent state. * CloudFormation is declarative. You describe the end result and CloudFormation applies the steps necessary to reach it. You don't need to write create and wait commands as it was the case with the scripting, but rather you can concentrate on architecting activities.
In 2018 during its yearly re:Invent conference AWS [introduced the AWS CDK][aws-cdk-at-re-invent] which should lead us hopefully to the bright cloud future. CDK combines the best of both worlds. It is declarative like CloudFormation and yet you can use full power of your favorite programming language and work in your favorite IDE. ## AWS CDK AWS CDK is a CLI tool and a set of libraries. The CDK libraries let you define your AWS infrastructure by writing code in one of five supported programming languages: Typescript, Javascript, Java, Python and C#. The CDK CLI tool generates then CloudFormation templates from the code you've written and provisiones the infrastructure. Although you do write code to define your infrastructure, it is still defined in a declarative manner because this code is different from the scripting that we used to have in the Middle Ages. You don't write instructions like "create a S3 bucket" and "poll and wait until S3 bucket is fully provisioned" but rather you write instructions like "add S3 bucket to my infrastructure with properties X, Y and Z". There are three main concepts in CDK: * application * stack * construct An application comprises all the cloud resources that you are going to provision in AWS. It represents basically all your workloads that you are going to deploy to AWS in order to solve the particular business problem you have (or your customer has). An application could consist e.g. of a VPC, RDS instances running databases, EC2 instances running your application and web servers and ALBs distributing load over the EC2 instances. The application consists of stacks which correspond to CloudFormation stacks. Stacks are there to logically bundle resources which should be deployed together as one unit. You could have e.g. stack A for VPC, Subnets, Security Groups, Network ACLs, Route Tables and other VPC-related configuration, stack B for RDS instances, stack C for application servers and their ALBs and stack D for web servers and their ALBs. Each stack can include AWS resources either directly or as constructs. A construct is a set of AWS resources which represents a reusable component which may be included in multiple other CDK applications. [Dynamo table viewer][cdk-dynamo-table-viewer] is an example of a construct. OK, so you start writing a CDK application. You add a couple of stacks to it. The stacks consist of various AWS resources such as SQS queues, Lambdas, S3 buckets, DynamoDB tables etc. You add those resources using the corresponding CDK libraries such as [@aws-cdk/aws-ec2][aws-ec2]. And then you use the CDK tool to deploy your application. The tool executes your code, finds the stacks, generates a CloudFormation template for each of them and deploys these templates producing a CloudFormation stack for each of them:
And that's it. Your app is in the cloud. That's how CDK works on a high level. AWS CDK is [open source][aws-cdk-github] and is written primarily in Typescript. Libraries for the rest of the languages are generated together with bindings (except for Javascript) which delegate execution to Node.js engine as described in the [CDK developer guide][aws-cdk-dev-guide]: > The AWS CDK is developed in TypeScript and transpiled to JavaScript. Bindings for the other supported languages make use of the AWS CDK engine running on Node.js, as does the cdk command-line tool. The main reasons why you might consider CDK for development of AWS-based cloud applications are the following: * You can use full power of your favorite programming language (if it's one of the five supported) in which you are proficient while still staying declarative when defining your infrastructure. * Your favorite IDE supports you with code completion, snippets, smart refactorings and other goodies. * The CDK app defining your infrastracture is usually much shorter than the generated CloudFormation template which you would have had to write if you wouldn't have used CDK. This is achieved by leveraging the CDK helper methods such as [this one][dynamodb-table-helper-method] (`Table.grantReadWriteData`) which generates IAM roles and policies with all the necessary permissions. CDK has reached a stable version 1.x. However, there are many CDK modules with experimental status such as [@aws-cdk/aws-s3-deployment][aws-s3-deployment] (at the time of writing it has experimental status). So the tool is not yet produciton ready and that's why I mentioned earlier that the "official" answer to the question "How do I do IaC in AWS?" is still the battle-tested CloudFormation. However, this should not discourage you from trying out CDK e.g. on pilot or small internal projects. If you've stated to use CDK and then you realize that you are unhappy with it, you can do the following: * let CDK generate CloudFormation YAML templates for you * commit the generated templates to git * drop CDK and use CloudFormation for futher development Since Typescript is the main CDK's language [the demo CDK application][demo-app] described further has been written in it. ## Demo App The demo app consists of two stacks, one for front-end and another one for back-end:
The front-end stack represents a simple HTML / CSS / Javascript application hosted on an S3 bucket. This application consists of three pages for viewing, creating and editing orders. The orders are saved in a DynamoDB Table. Logic is executed by Lambdas which are triggered via the API Gateway. In order to deploy the app, you need the following: * Node.js (version 10.3.0+) * AWS CDK CLI (`npm install -g aws-cdk@1.14.0`) * AWS environment variables with AWS access key and some other settings: ``` export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY export AWS_DEFAULT_REGION=us-east-1 export AWS_DEFAULT_OUTPUT=json ``` * [AWS CLI][aws-cli-install] (optional but recommended) The application can be deployed the following way: * `git clone https://github.com/ConSol/aws-cdk-and-lambda-workshop.git` * `cd aws-cdk-and-lambda-workshop` * `npm install` * `npm run build` * Let CDK bootstrap an environment for itself: `cdk bootstrap` * After bootstrapping you'll see that CDK has created a S3 staging bucket: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws s3 ls 2019-11-03 10:34:31 cdktoolkit-stagingbucket-1s73ctmtae8rs ``` This staging bucket is used e.g. as an intermediate buffer to store a Lambda's code. * First deploy the backend stack with: `cdk deploy BackendStack` * Answer `y` when you see the prompt "Do you wish to deploy these changes?" * After deployment has finished, you'll see the URL of the provisioned API Gateway endpoint in the console output: ``` Outputs: BackendStack.OrderApiEndpoint315A9F2C = https://gwvft2cty1.execute-api.us-east-1.amazonaws.com/prod/ ``` * Open the `webui/code.js` and edit the `ORDERS_ENDPOINT_URL` to point to the newly created endpoint: ``` const ORDERS_ENDPOINT_URL = "https://gwvft2cty1.execute-api.us-east-1.amazonaws.com/prod/"; ``` * Then deploy the front-end stack: `cdk deploy FrontendStack` * Answer with `y` again when you see the prompt. * Unfortunately CDK doesn't currently output URL of the public S3 bucket, created as part of the FrontendStack, so you need to find it e.g. using AWS CLI: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws s3 ls 2019-11-03 10:34:31 cdktoolkit-stagingbucket-1s73ctmtae8rs 2019-11-03 10:48:54 frontendstack-webui258d87a5-dp03mz7lo8hb ``` * Construct the URL based on the S3 bucket name and open it in your browser: `https://frontendstack-webui258d87a5-dp03mz7lo8hb.s3.amazonaws.com/index.html` * You'll see the main page which looks like this:
* You can create an order on the corresponding "Create Order" page:
* After hitting "Submit" you should see the order object which has been saved to a DynamoDB Table:
* If you copy the "orderId" and, go to the "View Order" page, paste the ID to the corresponding input and hit "Submit" you should see the same order which has been read by a Lambda from DynamoDB and sent back to the web page:
* The rest "Edit Order" page lets you edit properties of an order. * If you examine the CloudFormation stacks, you'll see three of them: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws cloudformation describe-stacks | grep -i stackname "StackName": "FrontendStack", "StackName": "BackendStack", "StackName": "CDKToolkit", ``` Two stacks have been created to deploy the demo app and the "CDKToolkit" stack has been created by the `cdk bootstrap` command. * Let's copy the contents of the staging S3 bucket and examine them: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws s3 cp --recursive s3://cdktoolkit-stagingbucket-1s73ctmtae8rs cdktoolkit-stagingbucket-1s73ctmtae8rs mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ cd cdktoolkit-stagingbucket-1s73ctmtae8rs/ mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop/cdktoolkit-stagingbucket-1s73ctmtae8rs$ tree . . ├── assets │   ├── 5fe8c75cbaa0f422a51f1bca34363659ace9170913b7d4d29389e1612e12651e │   │   └── e2b9658164b03c078c7e06e5da82a09b297b220e5a8e19229b883a2cf4618077.zip │   ├── 6416c21be320b522db64c705872c0a54d788e3df57b34a5f0d1e8602d7521430 │   │   └── ecc015fefc60848dbfe64452efab244d7d337e4d0c9d0effabfd9bd0ecffcdf6.zip │   └── bc86562e57cb99459725e82c8c6a382f9a6a6639ece5d0a056b29400aeaea588 │   └── bf2f3df4eb41f9d92ee603498716dcb8b3e0b82068f7938cbfb60db26c385666.zip └── cdk ├── BackendStack │   └── d9956d67a6a21540132f235fc971ca743e145b5c4669c9bd42194e698f6c533b.yml └── FrontendStack └── 896e7586f0d351cfcebc19b091df6b84dfd63ad91eb99f4c1f3625e6eafb5db9.yml 7 directories, 5 files ``` You can see there are 2 CloudFormation YAML templates generated by the CDK from the Typescript code defining the infrastructure (we'll get to the code later) and 3 ZIP archives. The first one contains the HTML / CSS / Javascript front-end app: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop/cdktoolkit-stagingbucket-1s73ctmtae8rs$ zip -v --test assets/5fe8c75cbaa0f422a51f1bca34363659ace9170913b7d4d29389e1612e12651e/e2b9658164b03c078c7e06e5da82a09b297b220e5a8e19229b883a2cf4618077.zip Archive: assets/5fe8c75cbaa0f422a51f1bca34363659ace9170913b7d4d29389e1612e12651e/e2b9658164b03c078c7e06e5da82a09b297b220e5a8e19229b883a2cf4618077.zip testing: code.js OK testing: create-order.html OK testing: edit-order.html OK testing: favicon.png OK testing: index.html OK testing: styles.css OK testing: vendor/bootstrap.min.css OK testing: vendor/bootstrap.min.js OK testing: vendor/jquery-3.3.1.min.js OK testing: vendor/popper.min.js OK testing: view-order.html OK No errors detected in compressed data of assets/5fe8c75cbaa0f422a51f1bca34363659ace9170913b7d4d29389e1612e12651e/e2b9658164b03c078c7e06e5da82a09b297b220e5a8e19229b883a2cf4618077.zip. test of assets/5fe8c75cbaa0f422a51f1bca34363659ace9170913b7d4d29389e1612e12651e/e2b9658164b03c078c7e06e5da82a09b297b220e5a8e19229b883a2cf4618077.zip OK ``` The second one contains Python code for an auxiliary Lambda which copies data between S3 buckets: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop/cdktoolkit-stagingbucket-1s73ctmtae8rs$ zip -v --test assets/6416c21be320b522db64c705872c0a54d788e3df57b34a5f0d1e8602d7521430/ecc015fefc60848dbfe64452efab244d7d337e4d0c9d0effabfd9bd0ecffcdf6.zip Archive: assets/6416c21be320b522db64c705872c0a54d788e3df57b34a5f0d1e8602d7521430/ecc015fefc60848dbfe64452efab244d7d337e4d0c9d0effabfd9bd0ecffcdf6.zip testing: awscli/ OK testing: awscli/alias.py OK testing: awscli/paramfile.py OK ... lots of other Python files ... testing: rsa/_compat.py OK testing: rsa/core.py OK No errors detected in compressed data of assets/6416c21be320b522db64c705872c0a54d788e3df57b34a5f0d1e8602d7521430/ecc015fefc60848dbfe64452efab244d7d337e4d0c9d0effabfd9bd0ecffcdf6.zip. test of assets/6416c21be320b522db64c705872c0a54d788e3df57b34a5f0d1e8602d7521430/ecc015fefc60848dbfe64452efab244d7d337e4d0c9d0effabfd9bd0ecffcdf6.zip OK ``` This Lambda comes from the [@aws-cdk/aws-s3-deployment][aws-s3-deployment] CDK library: > This is what happens under the hood: > 1. When this stack is deployed (either via cdk deploy or via CI/CD), the contents of the local website-dist directory will be archived and uploaded to an intermediary assets bucket. If there is more than one source, they will be individually uploaded. > 2. The BucketDeployment construct synthesizes a custom CloudFormation resource of type Custom::CDKBucketDeployment into the template. The source bucket/key is set to point to the assets bucket. > 3. The custom resource downloads the .zip archive, extracts it and issues aws s3 sync --delete against the destination bucket (in this case websiteBucket). If there is more than one source, the sources will be downloaded and merged pre-deployment at this step. And the third ZIP contains code of the Lambda which processes orders: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop/cdktoolkit-stagingbucket-1s73ctmtae8rs$ zip -v --test assets/bc86562e57cb99459725e82c8c6a382f9a6a6639ece5d0a056b29400aeaea588/bf2f3df4eb41f9d92ee603498716dcb8b3e0b82068f7938cbfb60db26c385666.zip Archive: assets/bc86562e57cb99459725e82c8c6a382f9a6a6639ece5d0a056b29400aeaea588/bf2f3df4eb41f9d92ee603498716dcb8b3e0b82068f7938cbfb60db26c385666.zip testing: main.d.ts OK testing: main.js OK testing: main.ts OK No errors detected in compressed data of assets/bc86562e57cb99459725e82c8c6a382f9a6a6639ece5d0a056b29400aeaea588/bf2f3df4eb41f9d92ee603498716dcb8b3e0b82068f7938cbfb60db26c385666.zip. test of assets/bc86562e57cb99459725e82c8c6a382f9a6a6639ece5d0a056b29400aeaea588/bf2f3df4eb41f9d92ee603498716dcb8b3e0b82068f7938cbfb60db26c385666.zip OK ``` Now let's examine the code from which those ZIPs and CloudFormation YAML templates are created. The project has been generated by the `cdk init --language=typescript` command. This command has basically created an NPM package. You can recognize that by the `package.json` and `.npmignore` files. Since the Typescript has been selected, the `tsconfig.json` has also been generated. You can read more on how to create a simple CDK project in the official [AWS CDK workshop][aws-cdk-workshop]. When `cdk deploy` command executes it looks into the `bin/app.ts` file: ``` #!/usr/bin/env node import "source-map-support/register"; import cdk = require("@aws-cdk/core"); import { EchoStack } from "../lib/echo-stack"; import { BackendStack } from "../lib/backend-stack"; import { FrontendStack } from "../lib/frontend-stack"; const app = new cdk.App(); const echoStack = new EchoStack(app, "EchoStack"); const backendStack = new BackendStack(app, "BackendStack"); const frontendStack = new FrontendStack(app, "FrontendStack"); const ownerTag = new cdk.Tag("owner", "Max.Mustermann"); [echoStack, backendStack, frontendStack].forEach(stack => ownerTag.visit(stack)); ``` You can see an instance of a CDK app is created and 3 CDK stacks are added to it. We have used only `BackendStack` and `FrontendStack` stacks. The `EchoStack` contains a simple Lambda behind an API Gateway which returns internal `event` object (in HTTP response body as JSON) passed to it by the AWS Lambda runtime. The last two lines are there to tag all resources with "owner" tag equal to "Max.Mustermann". The `BackendStack` which has been deployed first with `cdk deploy BackendStack` is defined in Typescript the following way: ``` import cdk = require("@aws-cdk/core"); import lambda = require("@aws-cdk/aws-lambda"); import apigw = require("@aws-cdk/aws-apigateway"); import dynamodb = require("@aws-cdk/aws-dynamodb"); export class BackendStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) { super(scope, id, props); const orders = new dynamodb.Table(this, "Orders", { partitionKey: { name: "OrderId", type: dynamodb.AttributeType.STRING } }); const orderResourceHandler = new lambda.Function(this, "OrderResourceHandler", { runtime: lambda.Runtime.NODEJS_10_X, code: lambda.Code.asset("order-resource-handler"), handler: "main.handler", environment: { TABLE_NAME: orders.tableName } }); orders.grantReadWriteData(orderResourceHandler); new apigw.LambdaRestApi(this, "OrderApi", { handler: orderResourceHandler }); } } ``` This stack declares a DynamoDB Table, a Lambda function and an API Gateway which calls the function and passes an HTTP request to it. The implementation of the Lambda is located in the `order-resource-handler` folder. The usage of the helper method `orders.grantReadWriteData(orderResourceHandler)` is worth noting. Such methods spare a lot of effort compared to conventional CloudFormation templates because they generate all the necessary IAM roles and policies which would have been needed to be written by hand in the case of CloudFormation. The FrontendStack is defined the following way: ``` import cdk = require("@aws-cdk/core"); import s3 = require("@aws-cdk/aws-s3"); import s3Deploy = require("@aws-cdk/aws-s3-deployment"); export class FrontendStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) { super(scope, id, props); const bucket = new s3.Bucket(this, "WebUI", { websiteIndexDocument: "index.html", publicReadAccess: true }); new s3Deploy.BucketDeployment(this, "WebUIDeployment", { sources: [s3Deploy.Source.asset("webui")], destinationBucket: bucket }); } } ``` It declares an S3 bucket which is going to be used for hosting an HTML / CSS / Javascript app (`publicReadAccess: true`). The bucket is populated by the files located in the `webui` folder (`sources: [s3Deploy.Source.asset("webui")]`). If you execute `cdk synth BackendStack` CDK will generate (synthesize) a CloudFormation template for the BackendStack. The generated template has approximately 400 lines of YAML and the original Typescript code has only approximately 30 lines of code i.e. more than 10 times less. How to clean up created resources: * destroy front-end stack: ``` cdk destroy FrontendStack ``` * Answer `y` to acknowledge that you want to delete stack's resources. * If you examine CloudFormation stacks, you'll see that the FrontendStask is gone: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws cloudformation describe-stacks | grep -i stackname "StackName": "BackendStack", "StackName": "CDKToolkit", ``` * However, the S3 bucket for the front-end app is still there: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws s3 ls 2019-11-03 10:34:31 cdktoolkit-stagingbucket-1s73ctmtae8rs 2019-11-03 10:48:54 frontendstack-webui258d87a5-dp03mz7lo8hb ``` Currently CDK does not delete non-empty resources such as S3 buckets. The bucket can be deleted with AWS CLI though: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws s3 rb --force s3://frontendstack-webui258d87a5-dp03mz7lo8hb delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/code.js delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/favicon.png delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/edit-order.html delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/create-order.html delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/styles.css delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/vendor/bootstrap.min.js delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/index.html delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/vendor/jquery-3.3.1.min.js delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/vendor/popper.min.js delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/vendor/bootstrap.min.css delete: s3://frontendstack-webui258d87a5-dp03mz7lo8hb/view-order.html remove_bucket: frontendstack-webui258d87a5-dp03mz7lo8hb ``` * BackendStack can be deleted next with `cdk destroy BackendStack` command. * However, the DynamoDB Table is still there. Just like S3 bucket it has not been deleted because it is not empty: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws dynamodb list-tables { "TableNames": [ "BackendStack-OrdersA9B65338-1ULLFMB1DGX6A" ] } ``` * The table has to be deleted separately: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws dynamodb delete-table --table-name BackendStack-OrdersA9B65338-1ULLFMB1DGX6A { "TableDescription": { "TableName": "BackendStack-OrdersA9B65338-1ULLFMB1DGX6A", "TableStatus": "DELETING", "ProvisionedThroughput": { "NumberOfDecreasesToday": 0, "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, "TableSizeBytes": 0, "ItemCount": 0, "TableArn": "arn:aws:dynamodb:us-east-1:251705002408:table/BackendStack-OrdersA9B65338-1ULLFMB1DGX6A", "TableId": "5d2f7ed1-6f90-4b42-a8d5-0b6a1a651cc9" } } ``` * As the last step the `CDKToolkit` CloudFormation stack needs to be deleted. The corresponding S3 bucket has to be also deleted separately: ``` mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws cloudformation delete-stack --stack-name CDKToolkit mmustermann@mustermachine:~/aws-cdk-and-lambda-workshop$ aws s3 rb --force s3://cdktoolkit-stagingbucket-1s73ctmtae8rs delete: s3://cdktoolkit-stagingbucket-1s73ctmtae8rs/cdk/BackendStack/d9956d67a6a21540132f235fc971ca743e145b5c4669c9bd42194e698f6c533b.yml delete: s3://cdktoolkit-stagingbucket-1s73ctmtae8rs/assets/6416c21be320b522db64c705872c0a54d788e3df57b34a5f0d1e8602d7521430/ecc015fefc60848dbfe64452efab244d7d337e4d0c9d0effabfd9bd0ecffcdf6.zip delete: s3://cdktoolkit-stagingbucket-1s73ctmtae8rs/cdk/FrontendStack/896e7586f0d351cfcebc19b091df6b84dfd63ad91eb99f4c1f3625e6eafb5db9.yml delete: s3://cdktoolkit-stagingbucket-1s73ctmtae8rs/assets/bc86562e57cb99459725e82c8c6a382f9a6a6639ece5d0a056b29400aeaea588/bf2f3df4eb41f9d92ee603498716dcb8b3e0b82068f7938cbfb60db26c385666.zip delete: s3://cdktoolkit-stagingbucket-1s73ctmtae8rs/assets/5fe8c75cbaa0f422a51f1bca34363659ace9170913b7d4d29389e1612e12651e/e2b9658164b03c078c7e06e5da82a09b297b220e5a8e19229b883a2cf4618077.zip remove_bucket: cdktoolkit-stagingbucket-1s73ctmtae8rs ``` ## Conclusion AWS CDK will probably supersede CloudFormation in the next 10-15 years. The tool combines best features of CloudFormation (declarativeness) and scripting (expressive power of modern programming languages). Currently CDK leans on CloudFormation by generating CloudFormation templates. However, AWS develops together with CDK the [Cloud Assembly specification][cloud-assembly-spec] which basically defines a ZIP archive containing artifacts deployable to a cloud. Cloud Assembly's ZIP archive is very similar to a WAR. The latter is deployed to a Java application server and the former is deployed to a cloud. It looks that the Cloud Assembly mechanism will become default for the CDK tool. CDK is AWS-only IaC tool. There is also the [Pulumi][pulumi] which is a rival multi-cloud-capable solution. It'll be exciting to see how these modern IaC tools gain more developers mindshare. [iac-aws-whitepaper]: https://d1.awsstatic.com/whitepapers/DevOps/infrastructure-as-code.pdf [semver]: https://semver.org [gitflow]: https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow [boto3]: https://boto3.amazonaws.com/v1/documentation/api/latest/index.html [bash-gotchas]: http://tldp.org/LDP/abs/html/gotchas.html [cloudformation]: https://aws.amazon.com/cloudformation/faqs [aws-cdk-at-re-invent]: https://www.youtube.com/watch?v=Lh-kVC2r2AU [cdk-dynamo-table-viewer]: https://github.com/eladb/cdk-dynamo-table-viewer [aws-ec2]: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ec2-readme.html [aws-cdk-github]: https://github.com/aws/aws-cdk [aws-cdk-dev-guide]: https://docs.aws.amazon.com/en_pv/cdk/latest/guide/getting_started.html [dynamodb-table-helper-method]: https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-dynamodb.Table.html#grant-wbr-read-wbr-write-wbr-datagrantee [aws-s3-deployment]: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html [demo-app]: https://github.com/ConSol/aws-cdk-and-lambda-workshop [aws-cli-install]: https://docs.aws.amazon.com/en_pv/cli/latest/userguide/cli-chap-install.html [aws-s3-deployment]: https://docs.aws.amazon.com/cdk/api/latest/docs/aws-s3-deployment-readme.html [aws-cdk-workshop]: https://cdkworkshop.com [cloud-assembly-spec]: https://github.com/aws/aws-cdk/blob/master/design/cloud-assembly.md [pulumi]: https://www.pulumi.com [Read more](https://omd.consol.de/blog/2019/11/04/introduction-to-aws-cdk/) ----------------------------------------------------------------------------------- title: "Introduction to Spring Boot and GraphQL for API Design Pt. 2" date: "2019-09-26" --------------------------------------------------------------------------------
GraphQL is a nice way to publish a highly customizable API. In combination with Spring Boot, which makes development really easy and offers features like database integration and security, you can quickly build your API service from scratch. This is the second part of the series in which we will create a REST-Service based on Spring Boot which will be translated in a GraphQL Service in the 3rd part of this little series. ## Introduction [In the first part](https://labs.consol.de/blog/2019/05/02/graphql_spring_boot_introduction.html) of this series we already learned about REST and GraphQL. I explained what each of them is good for and in which case you should use REST and when you should use GraphQL instead. In this part we will create a REST WebService based on Spring Boot which will be transformed to GraphQL in the third part of this series. The service which we will build is a simple backend for a guestbook. To keep it very simple we will build only the backend, which will give us the opportunity to get a list of guestbook entries via a REST interface which then can be consumed by every client like a website or a mobile app. Furthermore it will allow us to add a new entry, as well as delete an existing one. The delete operation will be secured by basic auth, as we don't want any user to delete entries from our guestbook. Each guestbook entry consists of a title, a message and the name of the author. To keep it simple we will not use any database. The first message is hardcoded and stored in a simple list, only kept in memory. All new entries added via the REST interface will be added to this list. Also the credentials to log in are stored within the code. For authentication we will use the features provided by the Spring Framework. Furthermore we don't use specific *Service* and *Repository* layers. You should do so if you are going to write a real application intended for productive use, but we, in this little demo, will not. In order to follow the setup, you will need a Java8 Runtime and a working installation of maven. If you want to download the source from the github repository you will need a git client. Furthermore you need a Java IDE of your choice. I will use IntelliJ, so things might look a little different on your machine, at least if your setup differs from mine. The source code we develop here can also be downloaded from [https://github.com/ConSol/graphqlguestbook](https://github.com/ConSol/graphqlguestbook). So buckle up and let's start. ## Basic Setup To setup the project from scratch we start at [https://start.spring.io/](https://start.spring.io/) to select our needed components. On the time writing this article the following settings should be the most actual ones After you initialized the application with the above mentioned settings, download and extract the zip-File to a destination of your choice and open it in your preferred IDE. You should see something like this: As you can see, the initializer has created an empty project with all the necessary code to run an simple SpringBoot application. We have an GuestbookApplication class which is responsible for startup and configuration, as well as an simple test for even this application class. ## Implement the REST Endpoint for the guestbook Let's begin by implementing our guestbook REST-endpoint. ### Implement the controller Therefore we create an new class under the *de.consol.blog.graphqlspringboot.guestbook.rest* package and call it *GuestbookController*. The class should be annotated with *@RestController* in order to show Spring that this is an endpoint class. For our first step we also need a method which has no parameters and returns a list of strings. Let's call this method *getEntries*. In the end this method will return a list of *Entry* object, but for now strings will be sufficient enough. This method will also be called when you open the endpoint in the browser as an index page. The complete listing of the class should look like this: ```java package de.consol.blog.graphqlspringboot.guestbook.rest; import java.util.ArrayList; import java.util.List; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @RestController public class GuestbookEndpoint { @GetMapping() public List getEntries(){ List result = new ArrayList<>(); result.add("Hello World"); return result; } } ``` If you start the application by running the main method in the *GuestbookApplication* class and try to access *localhost:8080* on your browser, you will be prompted with an login dialog. > Note: This is only the case if you follow this article along and code step by step. When you got the files from the github repository this is already fixed. > This is because we are using *SpringSecurity*. So by default all endpoints will be secured by username and password. For now we will leave it as it is. Later on we will disable security for all endpoints except some selected ones such as the endpoint for delete. If you want to access the endpoint and see the *hello world* message in your browser you can login with username *user* and the password which is prompted in the startup log of the application. Search for a line like this (your password will be different): ### Implement the model In order to get rid of the hardcoded *hello world* return value with the **Hello World** message we are going on by implementing the *Entry* model of our guestbook. It consists of some basic string attributes: * title * message + author Let's implement the *Entry* class in a separate model package under *de.consol.blog.graphqlspringboot.guestbook.model* and give it the following content: ```java package de.consol.blog.graphqlspringboot.guestbook.model; import java.util.Objects; public class Entry { private String title, message,author; /** * Gets message * * @return value of message */ public String getMessage() { return message; } /** * Sets message */ public void setMessage(final String message) { this.message = message; } /** * Gets author * * @return value of author */ public String getAuthor() { return author; } /** * Sets author */ public void setAuthor(final String author) { this.author = author; } /** * Gets title * * @return value of title */ public String getTitle() { return title; } /** * Sets title */ public void setTitle(final String title) { this.title = title; } @Override public String toString() { return "Entry{" + "title='" + title + '\'' + ", message='" + message + '\'' + ", author='" + author + '\'' + '}'; } @Override public boolean equals(final Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } final Entry entry = (Entry) o; return Objects.equals(title, entry.title) && Objects.equals(message, entry.message) && Objects .equals(author, entry.author); } @Override public int hashCode() { return Objects.hash(title, message, author); } } ``` After adding the class, instead of returning a simple string, we can return a list of *Entry* instances in our controller. So we change the *getEntries* method in the controller to something like this: ```java public List getEntries(){ final Entry firstEntry = new Entry(); firstEntry.setAuthor("Me"); firstEntry.setMessage("Hello World"); firstEntry.setTitle("The first message here"); List result = new ArrayList<>(); result.add(firstEntry); return result; } ``` If you restart the application and have a look at your browser again you will see, that now a list of *JSON* objects is returned instead of simple strings. That is because SpringBoot transforms java objects to their json representation out of the box. ### Security Configuration Before we implement the missing endpoint methods for adding new entries and delete existing ones, we first have to deal with the security settings of our application. By now, every time we start the application a new password is generated for the user *user*. We will change this to have a static password for the user and permit all users, even anonymous ones, to see the entries and add new entries as well. For the security configuration we will have a *SecurityConfiguration* class under *de.consol.blog.graphqlspringboot.guestbook.configuration* where we will add our user and password to an InMemory authentication scheme. > This should only be used in demo or test cases, because it is of course not secure at all > See the following listing: ```java package de.consol.blog.graphqlspringboot.guestbook.configuration; import org.springframework.context.annotation.Configuration; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; @Configuration @EnableWebSecurity public class SecurityConfiguration extends WebSecurityConfigurerAdapter { @Override protected void configure(AuthenticationManagerBuilder auth) throws Exception { auth.inMemoryAuthentication().withUser("user").password("secret").roles("OWNER"); } @Override public void configure(final HttpSecurity http) throws Exception { http.csrf().disable(); http.authorizeRequests().antMatchers("/", "/entry").permitAll(); super.configure(http); } } ``` As you can see, we will ignore our *root* endpoint, which is for listing the entries, as well as the (not yet existing) *entry* endpoint which is for adding new entries. When you restart the application and give it a try, you will see that there is no login form anymore and you can see all the existing entries without giving a username or password. ### New endpoint for adding new entries Adding the possibility to add new entries is very simple and straight forward. All we have to do is to add a new method in our controller, add the annotation that this will be a *PostMapping* and implement the logic. To do so, we will refactor the code written above to have a managed list to share between the *getEntries*-Method and the *addEntry*-Method. To make the life easier on creating *Entry*-Instances, will add a few new methods in the *Entry* class as well: ```java ... /** * Sets message */ public Entry message(final String message) { this.message = message; return this; } ... /** * Sets author */ public Entry author(final String author) { this.author = author; return this; } ... /** * Sets title */ public Entry title(final String title) { this.title = title; return this; } ``` Now with the above shown changes we are able to build our entry instances as shown in the following listing of the *GuestbookController* class: ```java package de.consol.blog.graphqlspringboot.guestbook.rest; import java.util.ArrayList; import java.util.List; import javax.servlet.http.HttpServletResponse; import org.springframework.http.HttpStatus; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RestController; import de.consol.blog.graphqlspringboot.guestbook.model.Entry; @RestController public class GuestbookController { private static List result = new ArrayList<>(); static { final Entry firstEntry = new Entry().author("Me").title("Hello World").message("This is the first message here"); result.add(firstEntry); } @GetMapping() public List getEntries() { return result; } @PostMapping(path = "/entry") public void addEntry(@RequestBody final Entry newEntry, HttpServletResponse response) { if (result != null && doesNotContainEntry(newEntry)) { result.add(newEntry); } else{ response.setStatus(HttpStatus.NOT_ACCEPTABLE.value()); } } private boolean doesNotContainEntry(final Entry newEntry) { return result.stream().filter(entry -> entry.getTitle().equals(newEntry.getTitle())).count() == 0; } } ``` You can see that we share the result list globally in the controller. Furthermore you see the implementation of the *addEntry* method. Other than the *getEntries* method, this one is annotated with *@PostMapping* because this will be a post request sending us the data for a new entry. The method checks if the given entry is null or if an entry with the same title already exists. If one of this is true, the client will get an 406 error response which says that the given entity is not applicable. Otherwise the entry will be added to the list and the client will get *200* indicating that the operation was successful. ### Delete an entry with the given Title Last but not least we will implement the missing method to delete an entry by its title. If we would use a database to store our guestbook entries we would of course use the primary key of the entry to find the right one. But as we only have an in memory list, the title is sufficient enough for our demo purpose. So all we have to do is to create a new method in our controller, which gets the title of the entry to be deleted. Then lookup the entry in the list and delete if it exists, otherwise return a http *404* to the client. Let's have a look in the code: ```java @DeleteMapping(path = "/deleteMessage/{title}") public void deleteEntry(@PathVariable("title")final String title, HttpServletResponse response){ Entry found = findEntryByTitle(title); if(found != null){ result.remove(found); }else{ response.setStatus(HttpStatus.NOT_FOUND.value()); } } private boolean doesNotContainEntry(final Entry newEntry) { return result.stream().filter(entry -> entry.getTitle().equals(newEntry.getTitle())).count() == 0; } ``` As you can see, we use another mapping method, the *@DeleteMapping*. This is because we want to get a request with the *HTTP Delete* method. Furthermore we use a *PathVariable* for giving us the title of the entry to be deleted. Give it a try in your REST tool of choice and use *DELETE* as your request method. You will notice when you try to call the *deleteMessage* endpoint, that you have to pass the user and password which we set up in the security configuration above. Otherwise the application will return a *401* indicating that you are not authorized to call this endpoint. ## Summary In this post we have created a REST WebService on SpringBoot which can list and add guestbook entries, as well as delete a single one by giving the title. To secure the service we used SpringSecurity, one of the many components available for SpringBoot. In the next episode we will convert this service to use GraphQL. ## Sources * [https://github.com/ConSol/graphqlguestbook](https://github.com/ConSol/graphqlguestbook) [Read more](https://omd.consol.de/blog/2019/09/26/introduction-to-spring-boot-and-graphql-for-api-design-pt.-2/) ----------------------------------------------------------------------------------- title: "Introduction to Spring Boot and GraphQL for API Design Pt. 2" date: "2019-09-26" --------------------------------------------------------------------------------
GraphQL is a nice way to publish a highly customizable API. In combination with Spring Boot, which makes development really easy and offers features like database integration and security, you can quickly build your API service from scratch. This is the second part of the series in which we will create a REST-Service based on Spring Boot which will be translated in a GraphQL Service in the 3rd part of this little series. ## Introduction [In the first part](https://labs.consol.de/blog/2019/05/02/graphql_spring_boot_introduction.html) of this series we already learned about REST and GraphQL. I explained what each of them is good for and in which case you should use REST and when you should use GraphQL instead. In this part we will create a REST WebService based on Spring Boot which will be transformed to GraphQL in the third part of this series. The service which we will build is a simple backend for a guestbook. To keep it very simple we will build only the backend, which will give us the opportunity to get a list of guestbook entries via a REST interface which then can be consumed by every client like a website or a mobile app. Furthermore it will allow us to add a new entry, as well as delete an existing one. The delete operation will be secured by basic auth, as we don't want any user to delete entries from our guestbook. Each guestbook entry consists of a title, a message and the name of the author. To keep it simple we will not use any database. The first message is hardcoded and stored in a simple list, only kept in memory. All new entries added via the REST interface will be added to this list. Also the credentials to log in are stored within the code. For authentication we will use the features provided by the Spring Framework. Furthermore we don't use specific *Service* and *Repository* layers. You should do so if you are going to write a real application intended for productive use, but we, in this little demo, will not. In order to follow the setup, you will need a Java8 Runtime and a working installation of maven. If you want to download the source from the github repository you will need a git client. Furthermore you need a Java IDE of your choice. I will use IntelliJ, so things might look a little different on your machine, at least if your setup differs from mine. The source code we develop here can also be downloaded from [https://github.com/ConSol/graphqlguestbook](https://github.com/ConSol/graphqlguestbook). So buckle up and let's start. ## Basic Setup To setup the project from scratch we start at [https://start.spring.io/](https://start.spring.io/) to select our needed components. On the time writing this article the following settings should be the most actual ones After you initialized the application with the above mentioned settings, download and extract the zip-File to a destination of your choice and open it in your preferred IDE. You should see something like this: As you can see, the initializer has created an empty project with all the necessary code to run an simple SpringBoot application. We have an GuestbookApplication class which is responsible for startup and configuration, as well as an simple test for even this application class. ## Implement the REST Endpoint for the guestbook Let's begin by implementing our guestbook REST-endpoint. ### Implement the controller Therefore we create an new class under the *de.consol.blog.graphqlspringboot.guestbook.rest* package and call it *GuestbookController*. The class should be annotated with *@RestController* in order to show Spring that this is an endpoint class. For our first step we also need a method which has no parameters and returns a list of strings. Let's call this method *getEntries*. In the end this method will return a list of *Entry* object, but for now strings will be sufficient enough. This method will also be called when you open the endpoint in the browser as an index page. The complete listing of the class should look like this: {% highlight java %} package de.consol.blog.graphqlspringboot.guestbook.rest; import java.util.ArrayList; import java.util.List; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @RestController public class GuestbookEndpoint { @GetMapping() public List getEntries(){ List result = new ArrayList<>(); result.add("Hello World"); return result; } } {% endhighlight %} If you start the application by running the main method in the *GuestbookApplication* class and try to access *localhost:8080* on your browser, you will be prompted with an login dialog. > Note: This is only the case if you follow this article along and code step by step. When you got the files from the github repository this is already fixed. > This is because we are using *SpringSecurity*. So by default all endpoints will be secured by username and password. For now we will leave it as it is. Later on we will disable security for all endpoints except some selected ones such as the endpoint for delete. If you want to access the endpoint and see the *hello world* message in your browser you can login with username *user* and the password which is prompted in the startup log of the application. Search for a line like this (your password will be different): ### Implement the model In order to get rid of the hardcoded *hello world* return value with the **Hello World** message we are going on by implementing the *Entry* model of our guestbook. It consists of some basic string attributes: * title * message + author Let's implement the *Entry* class in a separate model package under *de.consol.blog.graphqlspringboot.guestbook.model* and give it the following content: {% highlight java %} package de.consol.blog.graphqlspringboot.guestbook.model; import java.util.Objects; public class Entry { private String title, message,author; /** * Gets message * * @return value of message */ public String getMessage() { return message; } /** * Sets message */ public void setMessage(final String message) { this.message = message; } /** * Gets author * * @return value of author */ public String getAuthor() { return author; } /** * Sets author */ public void setAuthor(final String author) { this.author = author; } /** * Gets title * * @return value of title */ public String getTitle() { return title; } /** * Sets title */ public void setTitle(final String title) { this.title = title; } @Override public String toString() { return "Entry{" + "title='" + title + '\'' + ", message='" + message + '\'' + ", author='" + author + '\'' + '}'; } @Override public boolean equals(final Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } final Entry entry = (Entry) o; return Objects.equals(title, entry.title) && Objects.equals(message, entry.message) && Objects .equals(author, entry.author); } @Override public int hashCode() { return Objects.hash(title, message, author); } } {% endhighlight %} After adding the class, instead of returning a simple string, we can return a list of *Entry* instances in our controller. So we change the *getEntries* method in the controller to something like this: {% highlight java %} public List getEntries(){ final Entry firstEntry = new Entry(); firstEntry.setAuthor("Me"); firstEntry.setMessage("Hello World"); firstEntry.setTitle("The first message here"); List result = new ArrayList<>(); result.add(firstEntry); return result; } {% endhighlight %} If you restart the application and have a look at your browser again you will see, that now a list of *JSON* objects is returned instead of simple strings. That is because SpringBoot transforms java objects to their json representation out of the box. ### Security Configuration Before we implement the missing endpoint methods for adding new entries and delete existing ones, we first have to deal with the security settings of our application. By now, every time we start the application a new password is generated for the user *user*. We will change this to have a static password for the user and permit all users, even anonymous ones, to see the entries and add new entries as well. For the security configuration we will have a *SecurityConfiguration* class under *de.consol.blog.graphqlspringboot.guestbook.configuration* where we will add our user and password to an InMemory authentication scheme. > This should only be used in demo or test cases, because it is of course not secure at all > See the following listing: {% highlight java %} package de.consol.blog.graphqlspringboot.guestbook.configuration; import org.springframework.context.annotation.Configuration; import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder; import org.springframework.security.config.annotation.web.builders.HttpSecurity; import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity; import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter; @Configuration @EnableWebSecurity public class SecurityConfiguration extends WebSecurityConfigurerAdapter { @Override protected void configure(AuthenticationManagerBuilder auth) throws Exception { auth.inMemoryAuthentication().withUser("user").password("secret").roles("OWNER"); } @Override public void configure(final HttpSecurity http) throws Exception { http.csrf().disable(); http.authorizeRequests().antMatchers("/", "/entry").permitAll(); super.configure(http); } } {% endhighlight %} As you can see, we will ignore our *root* endpoint, which is for listing the entries, as well as the (not yet existing) *entry* endpoint which is for adding new entries. When you restart the application and give it a try, you will see that there is no login form anymore and you can see all the existing entries without giving a username or password. ### New endpoint for adding new entries Adding the possibility to add new entries is very simple and straight forward. All we have to do is to add a new method in our controller, add the annotation that this will be a *PostMapping* and implement the logic. To do so, we will refactor the code written above to have a managed list to share between the *getEntries*-Method and the *addEntry*-Method. To make the life easier on creating *Entry*-Instances, will add a few new methods in the *Entry* class as well: {% highlight java %} ... /** * Sets message */ public Entry message(final String message) { this.message = message; return this; } ... /** * Sets author */ public Entry author(final String author) { this.author = author; return this; } ... /** * Sets title */ public Entry title(final String title) { this.title = title; return this; } {% endhighlight %} Now with the above shown changes we are able to build our entry instances as shown in the following listing of the *GuestbookController* class: {% highlight java %} package de.consol.blog.graphqlspringboot.guestbook.rest; import java.util.ArrayList; import java.util.List; import javax.servlet.http.HttpServletResponse; import org.springframework.http.HttpStatus; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestBody; import org.springframework.web.bind.annotation.RestController; import de.consol.blog.graphqlspringboot.guestbook.model.Entry; @RestController public class GuestbookController { private static List result = new ArrayList<>(); static { final Entry firstEntry = new Entry().author("Me").title("Hello World").message("This is the first message here"); result.add(firstEntry); } @GetMapping() public List getEntries() { return result; } @PostMapping(path = "/entry") public void addEntry(@RequestBody final Entry newEntry, HttpServletResponse response) { if (result != null && doesNotContainEntry(newEntry)) { result.add(newEntry); } else{ response.setStatus(HttpStatus.NOT_ACCEPTABLE.value()); } } private boolean doesNotContainEntry(final Entry newEntry) { return result.stream().filter(entry -> entry.getTitle().equals(newEntry.getTitle())).count() == 0; } } {% endhighlight %} You can see that we share the result list globally in the controller. Furthermore you see the implementation of the *addEntry* method. Other than the *getEntries* method, this one is annotated with *@PostMapping* because this will be a post request sending us the data for a new entry. The method checks if the given entry is null or if an entry with the same title already exists. If one of this is true, the client will get an 406 error response which says that the given entity is not applicable. Otherwise the entry will be added to the list and the client will get *200* indicating that the operation was successful. ### Delete an entry with the given Title Last but not least we will implement the missing method to delete an entry by its title. If we would use a database to store our guestbook entries we would of course use the primary key of the entry to find the right one. But as we only have an in memory list, the title is sufficient enough for our demo purpose. So all we have to do is to create a new method in our controller, which gets the title of the entry to be deleted. Then lookup the entry in the list and delete if it exists, otherwise return a http *404* to the client. Let's have a look in the code: {% highlight java %} @DeleteMapping(path = "/deleteMessage/{title}") public void deleteEntry(@PathVariable("title")final String title, HttpServletResponse response){ Entry found = findEntryByTitle(title); if(found != null){ result.remove(found); }else{ response.setStatus(HttpStatus.NOT_FOUND.value()); } } private boolean doesNotContainEntry(final Entry newEntry) { return result.stream().filter(entry -> entry.getTitle().equals(newEntry.getTitle())).count() == 0; } {% endhighlight %} As you can see, we use another mapping method, the *@DeleteMapping*. This is because we want to get a request with the *HTTP Delete* method. Furthermore we use a *PathVariable* for giving us the title of the entry to be deleted. Give it a try in your REST tool of choice and use *DELETE* as your request method. You will notice when you try to call the *deleteMessage* endpoint, that you have to pass the user and password which we set up in the security configuration above. Otherwise the application will return a *401* indicating that you are not authorized to call this endpoint. ## Summary In this post we have created a REST WebService on SpringBoot which can list and add guestbook entries, as well as delete a single one by giving the title. To secure the service we used SpringSecurity, one of the many components available for SpringBoot. In the next episode we will convert this service to use GraphQL. ## Sources * [https://github.com/ConSol/graphqlguestbook](https://github.com/ConSol/graphqlguestbook) [Read more](https://omd.consol.de/blog/2019/09/26/introduction-to-spring-boot-and-graphql-for-api-design-pt.-2/) ----------------------------------------------------------------------------------- title: "Installing GitLab on OpenShift" date: "2019-07-31" --------------------------------------------------------------------------------
We recently had to install a bunch of applications on a customer's shiny new OpenShift 3.11 cluster. Among others also GitLab. Turned out getting GitLab up and running on OpenShift is not so easy. What I found on the Internet about installing GitLab on OpenShift was partly outdated and not 100% accurate. Most information was about getting GitLab into a Kubernetes cluster. So I had to adapt these information to the situation in an OpenShift cluster. This article is the conclusion of all these findings and efforts and gives a step-by-step recipe on how to install GitLab on OpenShift. GitLab will be installed by means of a group of Helm charts initially intended for Kubernetes. With some tweaks this can also be used for an OpenShift installation. Some links * [Installing GitLab on OpenShift](https://docs.gitlab.com/charts/installation/cloud/openshift.html) * [GitLab Helm Chart Deployment Guide](https://docs.gitlab.com/charts/installation/deployment.html) * Many [configuration settings](https://docs.gitlab.com/charts/charts/globals.html) plus many pages linked from there * [Tillerless Helm](https://rimusz.net/tillerless-helm) Always be aware that the relevant documentation found in the WWW might not be 100% acurate or simply outdated. The installation will be performed on OpenShift 3.11. With the exception concerning Helm versions I will not cover any details about other OpenShift versions. Your mileage will vary in case you try to install GitLab on older OpenShift clusters. GitLab will be set up to rely on an Active Directory server for user authentication. The setup for LDAP is basically the same. I disabled GitLab's own image registry, Nginx ingress and certificate manager as recommended on [Installing GitLab on OKD (OpenShift)](https://docs.gitlab.com/charts/installation/cloud/openshift.html). **UPDATE 2019-09-06** After one month we observed that the Prometheus server that is installed together with GitLab was continously crashing. I've added a [chapter](#configure-data-retention-for-prometheus) about how to prevent this behaviour. ## Install and configure Helm Get the release archive of Helm from [https://github.com/helm/helm/releases/tag/v2.12.3](https://github.com/helm/helm/releases/tag/v2.12.3). It is mandatory not to use the latest version of Helm but v2.12.x as this fits best to OpenShift v3.11. Extract the self-contained helm binary to a directory in your path. Verify it can be executed and has the right version with ``` helm version ``` Now set up Helm, install the Tiller plugin and add the GitLab repository. ``` helm init --client-only helm plugin install https://github.com/rimusz/helm-tiller helm repo add gitlab https://charts.gitlab.io/ helm repo update ``` ## Prepare the installation Latest at this point you have to make sure you are already logged in to the desired target OpenShift cluster. ``` oc login https:// ``` > In case you have already tried to install GitLab and want to start over for some reason here is how to get rid of all remains of the previous installation. > > ``` > oc delete project gitlab > > oc delete clusterrole gitlab-prometheus-kube-state-metrics > oc delete clusterrole gitlab-prometheus-server > oc delete clusterrolebinding gitlab-prometheus-alertmanager > oc delete clusterrolebinding gitlab-prometheus-kube-state-metrics > oc delete clusterrolebinding gitlab-prometheus-node-exporter > oc delete clusterrolebinding gitlab-prometheus-server > ``` GitLab will be installed in its own OpenShift project (aka namespace) named `gitlab`. ``` oc new-project gitlab ``` The Helm chart is intended for installing GitLab in a Kubernetes cluster, so will clash with some of the stricter restrictions in OpenShift. To get around this you need to assign some additional permissions. ``` oc adm policy add-scc-to-user anyuid -z gitlab-shared-secrets oc adm policy add-scc-to-user anyuid -z gitlab-gitlab-runner oc adm policy add-scc-to-user anyuid -z gitlab-prometheus-server oc adm policy add-scc-to-user anyuid -z default ``` > This is one of the examples where information from the [Internet](https://docs.gitlab.com/charts/installation/cloud/openshift.html) is definitively not correct. I tried it! Decide what URL will finally be used to access your GitLab instance (e.g. https://gitlab.<my-domain>). Get the certificate and key used for the host name and store them in files named `tls.crt` and `tls.key`. Chances are you use a wildcard certificate, e.g. `*.` Mind that `tls.crt` must contain the complete certificate chain up to (but not including) the root CA certificate. > In case you want to get along with the self-signed certificates provided by your OpenShift installation extract the certificate (chain) and the server key from the secret named `router-certs` in the `default` namespace > > ``` > oc get -n default secret router-certs -o jsonpath='{.data.tls\.crt}' | base64 -d > tls.crt > oc get -n default secret router-certs -o jsonpath='{.data.tls\.key}' | base64 -d > tls.key > ``` > > You should end up with two text files named `tls.crt` and `tls.key` like > > ``` > -----BEGIN CERTIFICATE----- > MIIDNjCCA.... > ``` > > ``` > -----BEGIN RSA PRIVATE KEY----- > MIIEowIBAA.... > ``` > > Mind that for this scenario you have to adapt the value `global.hosts.domain` in `gitlab-values.yml` to the default wildcard domain for your cluster applications, e.g. > > ``` > global: > .... > hosts: > domain: apps. > ``` Create two secrets required for the installation. ``` oc create secret tls gitlab-certs --cert=tls.crt --key=tls.key oc create secret generic gitlab-ldap-secret --from-literal=password= ``` Create a file named `gitlab-values.yml` with the input for the Helm charts. Use the following example as a starting point. ``` certmanager: install: false global: appConfig: ldap: servers: main: base: # may not be necessary in your case bind_dn: host: label: