Application observability - knowing the behavior of your application, on a per user basis - is a critical skill for any tech company. At my current role Iā€™ve been using New Relic to monitor our applications in production. New Relic is great, however recently Honeycomb has broken onto the scene with a new observability platform. Their CEO, Charity Majors, has been leading the conversation around application observability for a while now so I wanted to give their platform a try.

As my experience has been using New Relic to monitor Spring Boot applications, Iā€™ll do the same here. Iā€™ll walk through setting up a basic Spring Boot application, and instrumenting it with Honeycomb. This review should be taken with a grain of salt, as itā€™s not a formal evaluation. I wrote this as I went, so it reflects my thoughts at the time.

You can check out the source code here

Starting an application

First thingā€™s first - we need something to monitor! Letā€™s use the gold standard for bootstrapping a Spring Boot application, Spring Initializr

Letā€™s build a Java 11, Spring Boot 2.3 service. Weā€™ll call our application honeycomb-demo, and include Spring Web, Rest Repositories, Spring Data JPA & H2 Database. These libraries will allow us to quickly build functionality to persist objects to the database via REST endpoints.

Spring Initializr's Bootstrapping Page with the configuration filled in

If you want to try something similar, this link has my configuration.

After downloading, unzipping, and making sure I had configured jenv to use the right java version, we can give the application a run with mvn spring-boot:run

The terminal output from the running Spring Boot server

Then from a seperate terminal window, we can run curl localhost:8080/ and see a response from our app:

The JSON response from the Spring Boot server

Lets add some functionality

Seeing as instrumenting an autogenerated application endpoint wouldnā€™t be that interesting, letā€™s add a POST and a GET endpoint. As weā€™re evaluating Honeycomb, letā€™s build a product for beekeepers! Weā€™ll expose a POST endpoint for persisting Bees to our database, and a GET endpoint for seeing all the bees. Each Bee will have a name, and a bee catchphrase (duh!).

After a trip to my favorite Spring tutorial site Baeldung for a refresher on Spring Data Rest, letā€™s get started.

First, weā€™ll make a Bee domain object. Each Bee will have an autogenerated Id, a name, and itā€™s catchprase. Since weā€™re not doing Kotlin, the domain object will also have some autogenerated getters and setters that Iā€™ll spare you from seeing.

The code for the Bee Domain Object

Weā€™ll also need to define a Data Repository, which will autoconfigure our Bee CRUD endpoints for us:

The DataRepository object

Now that the repository and the domain obect are setup, after rerunning the application we can use curl to send a POST request to persist a Bee object:

The response from using curl to POST a Bee Object

Spring Data repository is incredibly powerful. With only those two classes, itā€™s configured CRUD endpoints for interacting with Bee objects and collections. It even sets up pagination over the collection of Bees for us.

Setting up Honeycomb

Now that we have something built, letā€™s intrument it šŸ”Ž

Honeycomb recently restructured their pricing tiers, and now offer a generous free tier. It includes 20 million events per month and 60 days data retention. More than plenty for our purposes! The only thing not available in the free tier is their SLO product.

The Honeycomb home page

Upon setting up an account, you land on their above homepage. For a free account, your stuck with a blue top bar prompting you to upgrade. However if it bothers you as much as it bothers me, you can use Stylebot to hide the element, so you donā€™t have to see it. Maybe itā€™s things like this that makes it impossible for me to stay focused on one taskā€¦

Well once youā€™re finished ruining their carefully crafted css, we can click ā€œCreate a Datasetā€, and choose Java as the language.

Setting up a Java dataset on the Honeycomb UI

They bring us over to a page with instructions for integrating with Java. Particularly, they call out:

ā€œThe Java Beeline provides automatic instrumentation for Spring Boot v2ā€

Score! That makes it easy. Letā€™s follow these instructions.

First - we need to add the Java Beeline to our pom.

Add the Beeline to our pom

Their guide specifies version 1.0.0 - but are they really only on version 1.0.0? No updates since its been published at all? Looking on Maven Central, it appears so - they havenā€™t updated it since it was originally published in early 2019. I guess they got it right the first time!

Next, we need to add some properties to our Spring application.properties file. Honeycomb provides the configuration as part of their setup page, with API key and dataset names already filled in. Itā€™s touches like that that really make you a happy non-paying customer!

Adding the Honeycomb properties to the application.properties file

I did have to modify their properties slightly - they use a : in their properties file as a delimiter, which causes IntelliJ to complain about ā€˜breaking code style conventionā€™. Switching them to = instead did the trick, as IntelliJ has default code styles itā€™s enforcing. According to the Java Properties documentation, either are perfectly valid deliminators.

There were also some optional config I removed to make things simpler. Hopefully I remember to delete that API key before commiting this to source control šŸ¤ž

Now, after rebuilding the server and sending a test POST request from the terminal, our app is instrumented and Honeycomb is receiving data!! šŸŽ‰šŸŽ‰

Adding the Honeycomb properties to the application.properties file

Other than the slight visual bug where the y axis labels are squished, it looks great. I particularly like the emphasis on the p50, p95 & p99 traffic. That already gave me my first insight - why did one of my curl requests take 218 ms?

Honeycomb heat map showing the slow request

Clicking into the Latency chart, we can zoom in on the requests in question. There we see that it was the first request to the server that took 218 ms - which makes sense. The server had recently started, and Spring often needs a request or two to establish connections and ā€œwarm upā€.

Mocking some user traffic

Great! Now that we are instrumented, we need some actual data rather than me running curl commands over and over again.

My performance testing tool of choice is Artillery. Itā€™s powerful and flexible - it can be configured via command line arguments directly, or a configuration file. You can follow the setup guide here to get started using it.

I put together a configuration file to run against our local server. It will ramp up users over 60 seconds, have peak load for 60 seconds, than ramp back down. Each user will POST a Bee, and then make a GET to get a list of all the Bees.

The Artillery load testing configuration file

After giving it a run with artillery run bee-test.yml (and going to get some more tea) we can see the results in Honeycomb!

The Honeycomb UI showing the load test results

Perfect! In the Total Spans section we can clearly see the ramp up, the stable period, and the ramp down.

However, only ramping our Artillery test to 100 users? Thatā€™s boring. Letā€™s tweak that config to 11:

Load test with 10000 users instead of 100

10,000 users in 1 minute? Now were cooking with gas!

After plugging my laptop in, running artillery run bee-test-insane.yml, and crossing my fingers, here are our Honeycomb results:

Honeycomb results from the mega load test

Andā€¦theyā€™re not great. It looks like our server maxed out at receiving around 250 events per second, when it should of been much higher. Artillery was likely the issue - the P90 response time of the demo application never exceeded 30 ms, it didnā€™t seem to have a problem processing the load. That implies Artillery was the problem, and it hit a limit on the number of requests it could generate concurrently. Alas, running a 10,000 user/minute test from a local machine doesnā€™t yet seem possible.

How much did that cost?

Now that weā€™ve finished our load test, letā€™s look at our Honeycomb usage. If thereā€™s one thing Iā€™ve learned from working with AWS - itā€™s never forget to check the bill. Luckily, Honeycomb makes it quite easy to see your usage.

Hooneycomb events usage

With some pretty hefty load tests weā€™ve used only 0.125% of our events budget for the month. The free tier Honeycomb offers is super generous.

The End

So thatā€™s my journey through instrumenting a Spring Boot app with Honeycomb. Since we had a lot of ground to cover weā€™ve stuck to the surface level of Honeycomb in this post. In future posts weā€™ll dig deeper into the Honeycomb query builder and custom instrumentation. For now - happy monitoring šŸšŸšŸ