Final Thesis: Elasticity Concept for Microservice-based System

Abstract: Software Elasticity is the concept of adapting available resources to the current or expected workload. This concept fits modern and stateless microservice architectures, which are scalable by design. Their scalability is closely related to Software Resilience and places new demands on cloud architectures. The JValue Open Data Service (JValue ODS) is an open data platform with focus on Extract, Transform, Load (ETL) pipelines and aims to make the usage of open data easy, reliable and safe. For the success of the ODS, an Elastic and therefore Resilient hosting is mandatory. This thesis deploys the ODS to an on-premise Kubernetes cluster to improve the uptime guarantee, discusses different deployment strategies, elaborates horizontal microservice scaling techniques and operates the necessary infrastructure. This thesis presents Peffer’s Design Research Process to build a concept for Elasticity in microservice-based architectures. The concept is demonstrated and evaluated in the context of the JValue ODS.

Keywords: Microservices, elasticity, scalability, kubernetes, devops

PDF: Master Thesis

Reference: Aron Metzig. Elasticity Concept for a Microservice-based System. Master Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2022.

Final Thesis: Testing Microservice Integration with Consumer-Driven Contract Tests

Abstract: Microservice-Systeme bestehen aus eigenständigen, verteilten Services, die über Netzwerkverbindungen miteinander kommunizieren. Das Testen von Service-Integrationen kann bei derartigen Systemen eine Herausforderung darstellen, da hierzu mehrere Services zur selben Zeit ausgeführt werden müssen und es viele potenzielle Quellen für falsch-negative Testergebnisse gibt.

Consumer-Driven Contract Testing (CDCT) ist ein Ansatz, der dazu verwendet werden kann, beide Seiten einer Service-Integration unabhängig voneinander zu testen. Dies wird erreicht, indem die beiden Seiten der Integration mithilfe eines Vertrags (engl. contract) voneinander entkoppelt werden, wobei der Contract als Vermittler fungiert. Dieser wird durch den Service vorgegeben, welcher die Schnittstelle eines anderen Services beansprucht, und drückt dessen Erwartungen an die verwendete Schnittstelle aus.

Diese Arbeit erforscht, inwieweit CDCT zur Testung von Microservice-Integratio- nen beitragen kann, indem Vorteile, Nachteile, Herausforderungen und Richtlinien erfasst werden, die im Zusammenhang zu CDCT für Microservice-Systeme stehen. Für die initiale Theoriebildung wurde zunächst eine strukturierte Literaturrecherche durchgeführt. Im Anschluss wurde Aktionsforschung betrieben, bei der Consumer-Driven Contract Tests für ein Open-Source Microservice-System entwickelt wurden. Zuletzt, nach der abschließenden Evaluation der Aktionsforschung, wurden die Inhalte, die im Rahmen der strukturierten Literaturrecherche erhoben wurden, mit den Erfahrungen aus der Aktionsforschung abgeglichen.

Keywords: Microservices, integration testing, consumer-driven contract tests, pact

PDF: Master Thesis

Reference: Felix Quast. Testing Microservice Integration with Consumer-Driven Contract Tests. Master Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2022.

Final Thesis: Konzept und Implementierung zur Observability für microservicebasierte Anwendungen

Abstract: ‘Microservices’ sind in der heutigen Zeit ein bekanntes und beliebtes Architekturmuster. Viele weltbekannte Tech-Unternehmen haben sich für diese entschieden. Die Entkopplung und die Aufteilung der Aufgaben in kleinere Services bringen neben daraus resultierenden Vorteilen auch Herausforderungen mit sich. Einen zentralen Negativpunkt hinsichtlich der Entwicklung dieser Dienste stellen die erschwerte Fehlersuche sowie die Schwierigkeit dar, den Überblick über die Anwendung als Gesamtes zu behalten.

In dieser Arbeit werden Softwaretools zur Überwachung und zur Aggregation von Log-Informationen vorgestellt. Darüber hinaus wird eine Kombination von Programmen gewählt, um ein Konzept zu entwickeln und eine beispielhafte Implementierung dieser Werkzeuge in ein bereits laufendes Open-Source-Projekt zu präsentieren.

Keywords: Microservices, observability, monitoring

PDF: Bachelor Thesis

Reference: Daniel Fabrikantow. Konzept und Implementierung zur Observability für microservicebasierte Anwendungen. Bachelor Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2021.

Job / Abschlussarbeit Model Compilation to Streaming Backends

Wir suchen jemanden kompetent im Compilerbau, der oder die Lust hat, sich eines wichtigen Spezialthemas anzunehmen, nämlich offene Daten nutzbar zu machen. Es folgt eine Aufgabenbeschreibung für eine Abschlussarbeit, aber wir bieten das für alles an: Studentischer Job, Abschlussarbeit, Promotion / Wimi Stelle:

Model Compilation to Streaming Backends

The goal of the thesis is to develop a compiler that turns an ETL pipeline model (the “program”) into a configuration for an event streaming framework (the “target architecture”). To start things easy, we want to compile SQL to Kafka Streams in such a way that an SQL schema definition configures a Kafka Streams instance so that the Kafka instance can load a CSV file and save it into a PostgreSQL database. If this works well, we will increase complexity: Not just a source schema (the “E” in ETL) but also transformation rules (the “T”) and a target schema (the “L”); not just Kafka Streams as a target architecture, but also Spark, Flink, and others.

This thesis is part of the JValue project. The mission of the JValue project is to make open data easy, reliable, and safe to use.

Thesis Description – Model Compilation to Streaming Backend

Bei Interesse gern direkt an mich wenden.

Dirk Riehle

Final Thesis: Value Types in TypeScript for JValue

Abstract: Over the past years, TypeScript has increasingly been gaining popularity due to its nature of providing functionalities to ease the development of scalable and robust applications whilst syntactically being a superset of JavaScript. With the growing complexity of data-driven environments, it is essential for programming languages to cope with value types beyond their primitive data types to capture the semantics of intangible data, such as systems of measurement, thus increasing readability and solidity across the codebase. By creating a test-driven framework in TypeScript, this thesis lays out different methods to efficiently implement value types, discusses their benefits as well as drawbacks, and ensures the reliability of the framework by integrating it into an existing data-driven service.

Keywords: Value types, JValue, TypeScript

PDF: Bachelor Thesis

Reference: Mert Baran. Value Types in Typescript for JValue. Bachelor Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2021. 

Final Thesis: Hierarchical Open Data Source Import for the JValue ODS

Abstract: Open Data has become more popular in the last few years due to its value to society. Governments, institutions, companies or individuals can make use of Open Data and add to economic growth or extract new knowledge from publicly available data. The Open Data Service (ODS) is a software developed by the Professorship of Open Source that aims to simplify the consumption of Open Data and make it more reliable.

The goal of this thesis is to extend the functionality of the ODS by the support of hierarchically structured data sources, in particular, File Transfer Protocol (FTP) based data sources. Due to the simplicity and reliability of the FTP, it is an appropriate solution for providing Open Data. This thesis aims to enable the user to explore and configure FTP data sources by developing a new microservice with a proof-of-concept user interface. As a result, consuming Open Data from FTP data sources is simplified and becomes more flexible.

Keywords: Open data, FTP, JValue ODS, microservices

PDF: Master Thesis

Reference: Benjamin Fischer. Hierarchical Open Data Source Import for the JValue ODS. Master Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2021.

Final Thesis: Fehlertoleranzanalyse von Microservice basierten Softwarearchitekturen – Konzept und Anwendung am JValue ODS

Abstract: Microservice-based software architectures play an essential role in building sizeable scalable cloud systems. The main advantage of microservices compared to the traditional software monoliths is the independent development, deployment, and scaling of the individual microservices, which allows innovations at a higher speed. Because microservice-based architectures are distributed systems, complexity is shifted from code to the network and communication layer. Therefore, additional failures like service outage or network connectivity loss arise, which must be tolerated to keep the system healthy and running. Within this thesis, a reusable concept is developed to analyse the fault tolerance of microservice-based software architectures. This allows for revealing weaknesses in the architecture that negatively affects the system’s reliability and resilience. For frequent problems, solution proposals are provided. The concept’s applicability and effectiveness are evaluated by applying it at the JValue Open Data Service (ODS). The analysis revealed several issues regarding the ODS’s fault tolerance, which could be fixed with the provided solutions.

Keywords: Microservices, fault tolerance, dependency graph, transactional outbox pattern

PDF: Master Thesis

Reference: Jonas Schüll. Fehlertoleranzanalyse von Microservice basierten Softwarearchitekturen – Konzept und Anwendung am JValue ODS. Master Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2021.

Final Thesis: Design and Implementation of Parameterizable Data Import for the JValue ODS

Abstract: Governments have recognized that the publication of open data is of great economic and social value. Collecting and using this data is challenging because it is not always available in an easy to process format. Minimizing these challenges is the task of the JValue Open Data Service (ODS), a system that makes data consumption easy. Yet the location of a resource and the time of a data import is statically defined.

This thesis presents a concept how the ODS can be extended by parameterizable datasources and how the data import can be triggered manually. This addresses the challenge of rapidly changing data on the Internet and adapts the ODS in order to deal with the emerging problems. With parameterizable datasources it is viable to dynamically describe the location of resources. The possibility for manual data imports ensures that data is only retrieved when it is really needed. The design decisions and the implementation of these functionalities for the ODS are covered in this thesis.

Keywords: open data; etl; JValue ODS; RESTful APIs

PDF: Bachelor Thesis

Reference: Jens Wächtler. Design and Implementation of Parameterizable Data Import for the JValue ODS. Bachelor Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2020.

Final Thesis: GraphQL-based Generic and Domain Specific Query Interfaces for the JValue ODS

Abstract: The JValue Open Data Service (ODS) is an open source application, founded and developed by the professorship of Open Source Software of the Friedrich Alexander Universität Erlangen Nürnberg (FAU). The application aims to become the go-to place for developers and data scientists around the world when building applications by using Open Data. The JValue ODS’ business model consists of collecting data from various different Open Data APIs, apply processing and cleansing and offering the improved version of the data back to the user. At the time of writing the thesis, the ODS only offers a rudimentary auto generated REST Interface for accessing the resulting data which does not fulfill the various requirements of its targeted user base. Therefore, this thesis aims to offer a solution on how a responsive and clean user API could be implemented using GraphQL as query language. As the ODS itself is a collection of many different docker native micro-services, the relevant functionality for providing the APIs is developed using the same core principles.

In order to grant the user the ability to fetch the raw data, each creation of a new pipeline, which configures the result of the attached data endpoint and a customizable transformation script, can be further configured to generate a read only GraphQL endpoint on the basis of the underlying PostgreSQL schema. This API is capable of filtering and combining all the fields of the collected data. To give the users, who are more knowledgeable in the domain they are working in, the possibility to add business logic to this data a node.js template project is provided. After implementing the desired functionality and hosting it on their preferred hosting solution, the users can register their custom endpoint through the JValue ODS Web Interface which are then validated and merged with the existing schema.

Keywords: GraphQL, JValue ODS, open data

PDF: Bachelor Thesis

Reference: Kai Malte von Rönne. GraphQL-based Generic and Domain Specific Query Interfaces for the JValue ODS. Bachelor Thesis. Friedrich-Alexander-Universität Erlangen-Nürnberg: 2020.

Final Thesis: Applying Event-Driven Architecture to the JValue ODS

Abstract: In a world of “Big Data” and data transparency “open data” might not be considered as an unknown term anymore. As the name suggests, it describes data, that is commonly available and free to use for everyone.

The vision of the software “JValue Open Data Service (ODS)” is to provide a solid solution for the community to achieve data availability and homogeneity in representation, regarding open data. With the provision of a user interface to configure the way open data is extracted and how it is presented, ETL (Extract-Transform-Load) processes are exceedingly facilitated.

The challenges of scaling large for a platform, based on a service-orientated architecture, demand a solution that can handle a large set of interactions. The purpose of this thesis is to apply an “Event-Driven Architecture” to the already existing model of the software to fulfill these requirements and furthermore, avoid the bottlenecks caused by the central orchestration of the former structure. This is done by reevaluating the current software design that is composed of distributed microservices and identify events of its current architecture with a technique, called “Event-Storming”. The results lead to an architecture design that will be implemented in the solution of “JValue ODS” and finally evaluated.

PDF: Bachelor Thesis

Reference: Hannes Fleischer. Applying Event-Driven Architecture to the JValue ODS. Bachelor Thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg: 2020