Cloud Native Diary #11
Exciting projects with Java, LLMs, and Generative AI. Conference news from KubeCon, Voxxed Days, and JFokus. Platforms and supply chain security.
In Cloud Native Diary, I periodically share my journey working with application development, platform engineering, and cloud native technologies.
The past couple of months have been hectic. In this issue, you'll find lots of content about Java, Generative AI, supply chain security, and news from events and my book. Ready? Let's get to it!
Cloud Native Spring in Action
I'm excited to share that the Chinese edition of my book Cloud Native Spring in Action - With Spring Boot and Kubernetes is officially out! Many thanks to Levin Zhang for the translation and for sharing some pictures of the printed book while waiting to receive my copy. I'm so glad that my book can now reach a wider audience!
Generative AI, LLMs, and Java
Spring AI
Spring AI is the latest addition to the Spring portfolio for building LLM-powered applications, bringing integrations with inference services and vector stores. It's under active development and continuously introducing new integrations and capabilities. Since version 0.8.0, you can find all the Spring AI modules directly in the Spring Initializr when bootstrapping new Spring Boot projects.
I've recently contributed a few enhancements to the Spring AI project:
- Implemented the auto-configuration for the OpenAI integration with the Image Generation API (gh-289).
- Updated the OpenAI integration to keep it in sync with the OpenAI API (gh-555).
- Updated the Ollama integration to keep it in sync with the Ollama API (gh-503, gh-553, gh-558).
Furthermore, I've submitted the first version of an implementation for integrating Spring AI with the Service Binding API and automatically binding inference services and vector stores when running on Kubernetes.
spring-projectsLangChain4j
LangChain4j is another framework for building LLM-powered Java applications. Over the past few months, I've been working on a library to bring LangChain4j support in Spring Boot.
ThomasVitaleUsing this project, you gain the following advantages over using the vanilla LangChain4j libraries in Spring Boot:
- Autoconfiguration and unified configuration properties for models and vector stores.
- HTTP infrastructure with RestClient, WebClient, and Jackson for all integrations.
- Built-in observability with Micrometer, including LLM-specific metrics and traces.
- Dev services with Docker Compose and Testcontainers for models and vector stores.
- Service bindings for automatic connection configuration when running on Kubernetes.
Ollama and Testcontainers
Ollama is a great way to run LLMs locally, saving money and keeping your data private. It quickly became one of the tools I used the most every day. I have it running via the native Ollama application (available for macOS, Linux, and Windows), and I also integrate with it from Spring AI or LangChain4j.
While working on the Ollama integration in Spring AI and LangChain4j, I spotted an error in the Ollama documentation, allowing me to submit my first pull request to the Ollama project.
When using Ollama from a Java application, you can run Ollama in a container for development and integration testing. Testcontainers provides an official Ollama module for that.
While working on the Weaviate integration in Spring AI and LangChain4j, I also submitted my first pull request to the Testcontainers project to extend the Weaviate module and support the new container registry where the Weaviate project publishes the official images.
Java for AI
As you can tell by now, you don't need to know Python to build LLM-powered applications and integrate AI capabilities. That's what I told Oleg Šelajev (Docker), who briefly interviewed me on the subject when we met at Voxxed Days Zurich.
Spring I/O
If you'd like to know more about how to build LLM-powered applications with Java and Spring Boot, consider attending the Spring I/O Conference in May.
- Introducing Spring AI by Mark Pollack and Christian Tzolov.
- Generative AI - Zero to Hero with Spring AI by Christian Tzolov and Adobe Saikali.
I will also give a presentation about advanced usages of Spring AI: Concerto for Java and AI - Building Production-Ready LLM Applications.
GOTO Copenhagen
In the fall, I will join GOTO Copenhagen, where I will also cover the subjects of Generative AI and Java. I will teach a masterclass and give a presentation about building AI Applications with Java and LLMs.
Trifork
Jfokus
Last February, I attended the Jfokus conference in Stockholm for the first time, and it was a fantastic experience! There were brilliant sessions, excellent organization, and amazing people. After the conference, I went on a wonderful ski trip with the organizers and fellow speakers. It was so much fun!
At the conference, I gave a presentation about building multitenant applications with Java (slides, source code). I covered how to introduce a tenant context into enterprise applications, manage data with Flyway and Hibernate, define observability with Micrometer and OpenTelemetry, and secure an application with Spring Security and Keycloak.
Voxxed Days Zurich
Last March, I was pleased to join Voxxed Days Zurich for the first time. It's been such a great experience! I attended exciting sessions, met old and new friends, and gave a presentation about Securing the Supply Chain for Your Java Applications (source code).
KubeCon Paris: AI, Platforms, and Developer Experience
KubeCon Paris was a blast! It was fantastic catching up with old friends, meeting new ones, attending exciting presentations, and even joining the first-ever KubeJam, where I played some blues and rock tunes on the piano!
Building AI-Ready Platforms - Symphony for Developer and Platform Engineer
LLM-powered applications require integrations with inference services and vector stores. Is your cloud native platform ready to support and enhance this new type of workload? At KubeCon Paris, Lize Raes and I went on a musical journey to get there (source code). It was a fun experience, including film music composition and opera singing!
The developer experience is of utmost importance after clearly understanding the app teams' requirements. We used Backstage for the developer portal, Crossplane for dev-friendly platform APIs, and Buildpacks for building production-ready container images without Dockerfiles.
Application frameworks are available in different languages and stacks to orchestrate AI applications. In the demo, we used LangChain4j and Spring Boot in Java, but the strategy we adopted supports polyglot applications. The local development workflow is powered by Testcontainers, which is fundamental for running inference services and vector stores locally.
Using a Crossplane Composition, we designed a supply chain to configure the application deployment based on Knative Serving and the Service Binding API to bind applications to inference services and vector stores.
Thanks to Carvel, the platform is composable and portable, even supporting air-gapped environments. Each platform capability is packaged using Carvel's native Kubernetes package management features. Finally, Flux is used for continuous deployment.
New capabilities like inference services and vector stores can be quickly packaged with Carvel. For extra convenience, we defined a golden path consumed via a Backstage template to build new platform packages because the platform engineer experience also counts!
Unlocking New Platform Experiences with Open Interfaces
Building platforms that provide end-to-end experiences to application teams comes with a series of challenges. Mauricio Salatino and I explored those problems at KubeCon Paris and presented solutions based on open interfaces. It was great being back on the KubeCon stage with Mauricio!
Onboarding new projects should be quick and easy, allowing developers to start working on the application business logic immediately instead of spending time with boilerplate and complex configuration. We highlighted solutions based on Backstage and Knative Functions.
Is Kubernetes required for local development workflows? We want to provide a great developer experience without adding extra cognitive load. Projects like Testcontainers and Buildpacks help us reach that goal.
Applications rarely run in isolation. Working with distributed systems requires careful design to manage the state, ensure eventual consistency, and cope with failures. Dapr abstracts that complexity via dev-friendly APIs and addresses those issues at the platform level.
Using Knative Serving, we can run HTTP-based applications with dynamic scaling, including scaling to/from zero. KEDA complements the solution by managing autoscaling for event-driven workloads.
Observability is a critical requirement before going to production. OpenTelemetry provides a comprehensive solution to instrument and collect telemetry signals, and Grafana offers a fully open-source platform for working with them.
The platform can be composed and distributed using Carvel, which offers a Kubernetes-native package management functionality. Continuous deployment and operations can be powered via GitOps using Flux.
Open Source AI
While being in Paris for KubeCon, I had the chance to attend the first Ollama meetup in Europe. It was a pleasure to chat with the Ollama maintainers, and it was inspiring to see all the use cases and capabilities unlocked by local and open-source AI frameworks and models, including Ollama, Testcontainers, Mistral, and LlamaIndex. If interested, you can watch the recordings of the lightning talks on YouTube.
Carvel
During KubeCon, I had the opportunity to catch up with Soumik Majumder and Praveen Rewar, maintainers of the Carvel project. We talked about how to make kapp-controller more extensible and shared some ideas I'm currently formalizing into a proposal for the project.
It was also interesting to talk with Varsha Prasad from the Operator Framework team about further integrating the two projects. That's one of the things I like the most about open source: collaborating across projects to achieve even better results and produce more value.
If you'd like to know more about Carvel, watch Daniel Garnier-Moiroux and Soumik Majumder's presentation, Carvel: Composable Tools for Application Management, at KubeCon Paris. For a quick introduction to the project, check out this lightning talk by Soumik Majumder and Praveen Rewar at KubeCon Paris.
KCD Munich
It seems a good time to tell you that Mauricio and I will be at KCD Munich in July. If you missed our presentation at KubeCon, you can join us in Munich!
Supply Chain Security for Java
Supply chain security is one of the areas I've been continuously working on for a long time now.
On May 10th, I will present about Securing the Supply Chain for Your Java Applications at Devoxx UK, in London.
On May 29th, I will teach a workshop about supply chain security for Spring Boot at Spring I/O.
I will also teach a masterclass on this subject at GOTO Copenhagen on September 30th.
Cloud Native Denmark
After visiting the Cloud Native community in Aarhus and Aalborg, I completed the tour in Denmark by attending the meetup in Copenhagen and showing how to supercharge your Kubernetes platform with Carvel.
If you're curious about achieving safe deployments, advanced configuration, and building portable platforms, you can watch the recording of my presentation from the Aarhus meetup.
I'm looking forward to meeting with the Aarhus community again this month to talk about cloud native development with Java and Spring Boot
KCD Denmark
After KubeCon, we have officially announced KCD Denmark 2024. This year, the conference will take place in Copenhagen at the Bella Center, and it will be a 2-day event (November 19-20). Stay tuned for information about the CFP!
Cover image generated with DALL-E 3.