Ulzurrun de Asanza i Sàez

Structured JSON logging in AWS Lambda with Kotlin and Log4j2

JVM ecosystem has a steep learning curve: missing a Maven/Gradle dependency might end up being a cryptic runtime exception about a package that is partially referenced in an XML file. I experienced this recently at work, trying to set up Log4j2 to write JSON logs in an AWS Lambda function.

My team inherited a Kotlin service running on AWS Lambda. The service used Log4j2 to handle logging and it was set up following the example provided in AWS documentation. Our src/main/resources/log4j2.xml looked like:

<Configuration status="WARN">
  <Appenders>
    <Lambda name="Lambda">
      <PatternLayout>
          <pattern>%d{yyyy-MM-dd HH:mm:ss} %X{AWSRequestId} %-5p %c{1} - %m%n</pattern>
      </PatternLayout>
    </Lambda>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="Lambda"/>
    </Root>
    <Logger name="software.amazon.awssdk" level="WARN" />
    <Logger name="software.amazon.awssdk.request" level="DEBUG" />
  </Loggers>
</Configuration>

Code language: HTML, XML (xml)

It worked but logs were unstructured plaintext lines, which made it harder to parse them and find logs related to specific events (we were trying to debug an issue that we couldn’t reproduce). We wanted to log things as a JSON string so AWS CloudWatch could parse it automatically and allow us to search by specific fields such as error code or other business-specific identifiers we were logging.

Setting up Log4j2 to use JSON format

The first thing we had to do is updating our src/main/resources/log4j2.xml and replace the PatternLayout with a more suitable JsonLayout. We wanted to log also some important business identifiers and we could do that adding a KeyValuePair as a JsonLayout child node for each value we wanted to log.

Our src/main/resources/log4j2.xml ended up looking like this:

<Configuration status="WARN">
  <Appenders>
    <Lambda name="Lambda">
      <JsonLayout compact="true" eventEol="true" properties="true" stacktraceAsString="true">
        <KeyValuePair key="awsRequestId" value="$${ctx:AWSRequestId}"/>
        <KeyValuePair key="ourImportantBusinessId" value="$${ctx:importantBusinessId}"/>
      </JsonLayout>
    </Lambda>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="Lambda"/>
    </Root>
    <Logger name="software.amazon.awssdk" level="WARN" />
    <Logger name="software.amazon.awssdk.request" level="DEBUG" />
  </Loggers>
</Configuration>

Code language: HTML, XML (xml)

However, this won’t work. We need to add some dependencies to be able to use JsonLayout and KeyValuePair. Otherwise we get a java.lang.IllegalStateException:

2022-10-25 14:38:24,671 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.layout.JsonLayout for element JsonLayout: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.layout.JsonLayout java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.layout.JsonLayout
Code language: CSS (css)

It’s clear that we are missing some dependency that provides org.apache.logging.log4j.core.layout.JsonLayout, but working mostly in the JavaScript ecosystem I didn’t find clear instructions on the next steps, which were adding required dependencies.

Adding required dependencies

People familiar with Gradle will notice the absence of tasks, testing framework and so on in the following code examples. That’s for simplicity sake.

Our service was using Kotlin and Gradle and our config lived in the build.gradle.kts file. It looked like: 

plugins {
    kotlin("jvm") version "1.5.30"
}

group = "com.myCompany.myService"
version = "1.0"

val log4jVersion = "2.17.2"

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    implementation(kotlin("stdlib"))
    implementation("com.amazonaws:aws-lambda-java-core:1.2.1")
    implementation("com.amazonaws:aws-lambda-java-events:3.10.0")

    // logs
    implementation("org.apache.logging.log4j:log4j-api:$log4jVersion")
    implementation("org.apache.logging.log4j:log4j-core:$log4jVersion")
    implementation("org.apache.logging.log4j:log4j-slf4j-impl:$log4jVersion")
    runtimeOnly("com.amazonaws:aws-lambda-java-log4j2:1.2.0")
}
Code language: Kotlin (kotlin)

We added the following dependencies (listed in Log4j Runtime Dependencies page):

Leaving our build.gradle.kts file like:

plugins {
    kotlin("jvm") version "1.5.30"
}

group = "com.myCompany.myService"
version = "1.0"

val log4jVersion = "2.17.2"

repositories {
    mavenCentral()
    mavenLocal()
}

dependencies {
    implementation(kotlin("stdlib"))
    implementation("com.amazonaws:aws-lambda-java-core:1.2.1")
    implementation("com.amazonaws:aws-lambda-java-events:3.10.0")

    // logs
    implementation("org.apache.logging.log4j:log4j-api:$log4jVersion")
    implementation("org.apache.logging.log4j:log4j-core:$log4jVersion")
    implementation("org.apache.logging.log4j:log4j-slf4j-impl:$log4jVersion")
    implementation("com.fasterxml.jackson.core:jackson-core:2.13.4")
    implementation("com.fasterxml.jackson.core:jackson-databind:2.13.4.2")
    implementation("com.fasterxml.jackson.core:jackson-annotations:2.13.0")
    runtimeOnly("com.amazonaws:aws-lambda-java-log4j2:1.2.0")
}

Code language: Kotlin (kotlin)

Logging custom keys

So far this worked but we wanted to log some custom important business identifiers to ease us debugging. We had to modify the code of our AWS Lambda function handler so it added these values. We achieved it using MDC.put static method.

// First import MDC
import org.slf4j.MDC

// Later on you can use it inside your handler like
MDC.put("ourImportantBusinessId", "The value we wanted to log")
Code language: Kotlin (kotlin)

That was enough to get structured JSON logs in AWS CloudWatch from our Kotlin and Log4j based AWS Lambda

No replies on “Structured JSON logging in AWS Lambda with Kotlin and Log4j2”

There are no comments yet.

Leave a Reply

Your email address will not be published.

Required fields are marked *

Your avatar