Paul Butcher (original) (raw)

Quick and Easy Clojure on AWS Lambda Part 2

This follows on from my previous article which described how to get a simple Clojure Ring application running on AWS Lambda. This article shows how to connect it to a database.

The accompanying code is here.

CloudFormation

AWS SAM provides direct support for DynamoDB, but not for more traditional databases like PostgreSQL, so that means dropping into CloudFormation. This is, sadly, rather wordy, because we'll need to configure all the neccessary AWS machinery (including setting up a VPC, security group, and database credentials) ourselves, but it's mostly standard boilerplate which is pretty well documented:

You can see the full CloudFormation template here. We'll explain the various sections in more detail below.

VPC

To avoid having to make our database publicly visible, we're going to put both our Lambda function and database in a shared VPC. Here's how we create that VPC:

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.0.0.0/16

We also need to define a couple of subnets (RDS requires at least two subnets, in two different avaialability zones):

  Subnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: 10.0.1.0/24
      AvailabilityZone: !Select [0, Fn::GetAZs: !Ref "AWS::Region"]
  Subnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: 10.0.2.0/24
      AvailabilityZone: !Select [1, Fn::GetAZs: !Ref "AWS::Region"]

This is using a little CloudFormation magic to select the first two availability zones in whichever region we're deploying to. You could just as easily hardcode the availability zones if you prefer.

And we need a security group which allows things within the VPC to access Postgres:

  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub "Security group for ${AWS::StackName}"
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432
          CidrIp: !GetAtt VPC.CidrBlock

We put our Lambda function in the VPC we've created by adding the following to its Properties:

      VpcConfig:
        SecurityGroupIds: [!Ref SecurityGroup]
        SubnetIds: [!Ref Subnet1, !Ref Subnet2]
      Policies: [AWSLambdaVPCAccessExecutionRole]

Database

We now have everything we need to create a database instance:

  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: !Sub "DBSubnet group for ${AWS::StackName}"
      SubnetIds: [!Ref Subnet1, !Ref Subnet2]
  Database:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceClass: db.t4g.micro
      Engine: postgres
      EngineVersion: 14.15
      DBName: example_lambda_app
      AllocatedStorage: 20
      StorageEncrypted: true
      ManageMasterUserPassword: true
      MasterUsername: postgres
      KmsKeyId: !Ref DatabaseKey
      VPCSecurityGroups: [!Ref SecurityGroup]
      DBSubnetGroupName: !Ref DBSubnetGroup

Most of this is pretty obvious: we're creating an RDS database running on a db.t4g.micro instance, with 20GB of storage, encrypted at rest, and with a master user called postgres. We're adding it to the security group we created earlier, and letting it know about the subnets we created via a DBSubnetGroup.

Key Management

We've asked RDS to manage the database password for us (ManageMasterUserPassword) and store the credentials in a Secrets Manager (KMS) secret. Here's how we create that secret:

  DatabaseKey:
    Type: AWS::KMS::Key
    Properties:
      Description: DatabaseKey
      EnableKeyRotation: false
      KeyPolicy:
        Version: 2012-10-17
        Id: !Sub "key-${AWS::StackName}"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action: ["kms:*"]
            Resource: "*"

I've chosen to disable key rotation because I'll be passing the key to the Lambda function as an environment variable. An alternative would be to modify the Lambda function to use the Secrets Manager API to retrieve the password, but I wanted to keep the code as simple as possible. The rest is simple boilerplate taken from the article mentioned above.

To use this secret in our Lambda function, we add the following to the function's Properties:

      Environment:
        Variables:
          DB_HOST: !GetAtt Database.Endpoint.Address
          DB_PASSWORD: !Sub "{{resolve:secretsmanager:${Database.MasterUserSecret.SecretArn}:SecretString:password}}"

Deployment

Deploying is exactly the same as before: build the uberjar and then sam deploy. The first time you do this it'll take a while because it's creating the database instance, but subsequent deployments will be much quicker.

Published: 2025-01-24

Tagged: clojure

Quick and Easy Clojure on AWS Lambda in 2025

I recently found myself starting a new project and was looking for the quickest and easiest way to get something up and running. In the past I might have used Heroku, and I looked briefly at Fly.io, but it turns out that Clojure now runs much better on Lambda than it used to (cold starts are no longer an issue), and it's easy to get up and running with AWS SAM which gives us simple serverless Infrastructure as Code.

This article describes how to get a simple Clojure Ring application running. A followup article shows how to connect it to a database. The code is available here.

The Code

Let's start with a simple Clojure Ring app with a sprinkling of interactivity through HTMX:

(ns example.lambda-app
  (:require [compojure.core :refer [defroutes GET]]
            [compojure.route :as route]
            [hiccup2.core :refer [html]]
            [ring.logger :refer [wrap-with-logger]]
            [ring.middleware.defaults :refer [site-defaults wrap-defaults]]
            [ring.middleware.params :refer [wrap-params]]))

(defn index-page
  []
  (str (html [:head [:title "HTMX Example"]
              [:script {:src "https://unpkg.com/htmx.org@2.0.4"}]]
             [:body [:h1 "HTMX Example"]
              [:div#greeting {:hx-get "/greet" :hx-trigger "load"}]])))

(defn greet [] (str (html [:div "Hello, World!"])))

(defroutes app-routes
  (GET "/" [] (index-page))
  (GET "/greet" [] (greet))
  (route/not-found "Not Found"))

(def app
  (-> app-routes
      wrap-params
      (wrap-defaults site-defaults)
      wrap-with-logger))

This is all entirely standard: nothing different from any other Ring app. We're going to implement two different ways to serve this app, one using ring-jetty-adapter for local development, and one using ring-lambda-adapter for deployment to AWS Lambda.

For local development, we'll create dev/user.clj:

(ns user
  (:require [ring.adapter.jetty :as jetty]
            [example.lambda-app :refer [app]]))

(defn -main [& _]
  (jetty/run-jetty #'app {:port 8080 :host "0.0.0.0" :join? false}))

And for deployment to AWS Lambda, we'll create lambda.clj:

(ns example.lambda
  (:gen-class :implements
              [com.amazonaws.services.lambda.runtime.RequestStreamHandler])
  (:require [paulbutcher.ring-lambda-adapter :refer [handle-request]]
            [example.lambda-app :refer [app]]))

(defn -handleRequest [_ is os _] (handle-request app is os))

This implements the RequestStreamHandler interface defined by the AWS Java Lambda runtime. The handle-request function is provided by ring-lambda-adapter and does the work of converting the input and output streams to and from Ring requests and responses.

Finally, here's our deps.edn:

{:paths ["src" "resources"]
 :deps {com.amazonaws/aws-lambda-java-core {:mvn/version "1.2.3"}
        com.amazonaws/aws-xray-recorder-sdk-slf4j {:mvn/version "2.18.2"}
        com.paulbutcher/ring-lambda-adapter {:mvn/version "1.0.7"}
        compojure/compojure {:mvn/version "1.7.1"}
        hiccup/hiccup {:mvn/version "2.0.0-RC4"}
        org.clojure/clojure {:mvn/version "1.12.0"}
        ring-logger/ring-logger {:mvn/version "1.1.1"}
        ring/ring-core {:mvn/version "1.13.0"}
        ring/ring-defaults {:mvn/version "0.5.0"}}
 :aliases {:run {:main-opts ["-m" "user"]}
           :build {:deps {io.github.clojure/tools.build {:mvn/version "0.10.6"}}
                   :ns-default build}
           :dev {:extra-paths ["dev"]
                 :extra-deps {org.slf4j/slf4j-simple {:mvn/version "2.0.16"}
                              ring/ring-jetty-adapter {:mvn/version "1.13.0"}}}}}

This is all very standard apart from:

You should now be able to run locally with clojure -M:dev:run.

Deployment

To deploy to AWS, you'll need to have the AWS SAM CLI installed. This sits on top of AWS CloudFormation and simplifies the process of deploying serverless applications. Our application is described via a template.yaml file:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: target/standalone.jar
      Handler: example.lambda::handleRequest
      Runtime: java21
      FunctionUrlConfig:
        AuthType: NONE
      AutoPublishAlias: live
      SnapStart:
        ApplyOn: PublishedVersions
      Timeout: 20
      MemorySize: 512
      Tracing: Active
    Metadata:
      SkipBuild: true

Outputs:
  Endpoint:
    Value: !GetAtt FunctionUrl.FunctionUrl

To deploy, first build the jar file with clojure -T:build uber (see build.clj in GitHub), then deploy with sam deploy --guided. This will ask you some questions such as which region you want to deploy to and then eventually output the URL of your function's endpoint. Connect to that URL, and you should see exactly what you saw when you ran locally.

Monitoring

There are a number of different ways to keep an eye on how your Lambda function is performing.

Development

Local development works just like any other Ring app. To deploy a new version, just build a new jar file and run sam deploy.

Conclusion

The combination of SAM, which makes Lambda function deployment so simple, and SnapStart, which removes the cold start problem, means that AWS Lambda is my new default for getting started quickly.

In the next article, we'll look at how to connect our Lambda function to a database.

Credits

This was all heavily inspired by A Recipe for Plain Clojure Lambdas.

Published: 2025-01-23

Tagged: clojure

An introduction to Datalog in Flix: Part 4

This is part 4 of a series. [Part 1 | Part 2 | Part 3 | Part 4]

The code to accompany this series is available here.

In the previous part of this series, we saw how to use lattice semantics to calculate all our degrees of separation within the Game of Thrones dataset in one pass. But we didn't quite get to our final solution because we weren't yet calculating counts of the number of characters at each degree.

In this part, we'll tie off that loose end, and discover why lattice semantics are "lattice" semantics in the process.

Of course, we could just count up our values in Flix (not Datalog) code, and that would be a perfectly reasonable approach. But as we'll see Datalog gives us a really elegant solution by leveraging partial ordering.

Partial Ordering

So far we've been working with simple integers, and integers are (surprise!) ordered. So 1 is less than 2 and 3 is less than 126 and so on.

In fact, not only are integers ordered, they totally ordered. You can pick any pair of numbers a and b, and at least one of a <= b or b <= a will be true (they might both be true if a equals b).

But, not every set of values is totally ordered. A real world example of a partial ordering is ancestry; given two people a and b, it might easily be the case that neither "a is an ancestor of b" nor "b is an ancestor of a" is true.

For example, "Eddard Stark is an ancestor of Arya Stark" is true. But neither "Cersei Lannister is an ancestor of Tyrion Lannister" nor "Tyrion Lannister is an ancestor of Cersei Lannister" are true.

However, we can ask "who is the most recent common ancestor of Cersei and Tyrion" (in this case Tywin Lannister). We generally refer to this as the least upper bound.

Least Upper Bound

In part 3, we said that lattice semantics chose the "maximum" value from all possible values. This was a simplification: they actually chose the least upper bound.

For integers, which we were working with in part 3, the least upper bound is the maximum value. But for other types, those that are partially ordered (as we saw with ancestry above), the least upper bound could be something else.

An interesting example is sets. It may not be the case that either "set a is a subset of set b" or "set b is a subset of set a" is true. But they will always have a least upper bound which is the union of a and b.

For example, the least upper bound of Set#{"Tyrion Lannister", "Cersei Lannister"} and Set#{"Tyrion Lannister", "Arya Stark"} is Set#{"Tyrion Lannister", "Cersei Lannister", "Arya Stark"}.

This is where the "lattice" in lattice semantics comes from: a partially ordered set which defines a least upper bound is called a lattice) in mathematics.

Happily, this is exactly what we need to solve our degrees of separation problem.

Six Degrees of Separation, Take 3

As a reminder, here were the rules we used in part 3 of this series:

        Degree(x; Down(0)) :- Root(x).
        Degree(x; n + Down(1)) :- Degree(y; n), Related(y, x).

Here are the new rules we're going to add:

        AggregatedDegree(n; Set#{x}) :- fix Degree(x; n).
        DegreeCount(n, Set.size(s)) :- fix AggregatedDegree(n; s).

We start by inferring new AggregatedDegree facts from the Degree facts we calculated previously. We're using the fact that the least upper bound of a collection of sets is the union of those sets, so the value on the right side of the semicolon will be the union of all of the character names in Degree.

You might be wondering what the fix is for in our new rules? If you look at the two sides of the rule (either side of the :-) you can see that on the left we're using n as a normal value (it's on the left of the semicolon), whereas on the right it's a lattice value (it's on the right of the semicolon). Flix requires that we use fix if we ever mix a value this way; it ensures that we completely calculate all the Degree facts before starting to create AggregateDegree facts.

So now we have a number of AggregatedDegree facts, one for each degree of separation, where the right hand side is a set of all the characters of that degree. The final step is to convert those facts into DegreeCount facts by finding the size of each set.

Here's the whole thing. As you can see, it's even more elegant than the solution we came up with in part 2 (which was already much simpler than we could have achieved without using Datalog).

def main(): Unit \ IO =

    let relationshipTypes = "parents" :: "parentOf" :: "killed" :: "killedBy" ::
        "serves" :: "servedBy" :: "guardianOf" :: "guardedBy" :: "siblings" ::
        "marriedEngaged" :: "allies" :: Nil;
    let relationships = Json.getRelationships(relationshipTypes);

    let related = inject relationships into Related;

    let rules = #{
        Degree(x; Down(0)) :- Root(x).
        Degree(x; n + Down(1)) :- Degree(y; n), Related(y, x).
        AggregatedDegree(n; Set#{x}) :- fix Degree(x; n).
        DegreeCount(n, Set.size(s)) :- fix AggregatedDegree(n; s).
    };

    let root = inject "Tyrion Lannister" :: Nil into Root;

    query rules, related, root select (d, c) from DegreeCount(d, c) |>
        List.foreach(match (d, c) -> println("Separated by degree <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi><mo>:</mo></mrow><annotation encoding="application/x-tex">{d}: </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6944em;"></span><span class="mord"><span class="mord mathnormal">d</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span></span></span></span>{c}"))

And, for completeness, here's what it outputs:

Separated by degree 6: 2
Separated by degree 5: 14
Separated by degree 4: 60
Separated by degree 3: 104
Separated by degree 2: 56
Separated by degree 1: 6
Separated by degree 0: 1

Conclusion

That's it for our journey through Datalog and Flix. Please do experiment with other problems which can leverage Datalog: I'd love to see what you come up with!

[Part 1 | Part 2 | Part 3 | Part 4]

Published: 2022-10-26

Tagged: flix datalog logic-programming