Working with Firebase Documents in ClojureScript

In a project I’m currently working on we’re making use of Google's Firebase to store domain data and run cloud functions.

In Firestore, which is Firebase’s database offering, every document is essentially a Javascript object. While interop in ClojureScript is pretty good we ended up converting the raw data of these documents to ClojureScript data structures using js->clj. This also meant we’d need to convert them back to JS objects before writing them to Firestore.

Because IDs are technically not part of the document the project adopted a pattern of representing documents as tuples:

[id (js->clj firestore-data)]

This works but isn’t particularly extensible. What if we also wanted to retain the “Firestore Reference” specifying a documents location inside the database? (Firestore stores data in a tree-like structure.)

It also leads to some funky gymnastics when working with collections of documents:

(sort-by (comp :join_dt second) list-of-document-tuples)

Could be worse... but also could be better.

This blogpost will compare various approaches approach to address the problems above using cljs-bean, basic ClojureScript data structures, custom protocols and :extend-via-metadata.

cljs-bean

With the recent release of cljs-bean we have an interesting alternative to js->clj. Instead of eagerly walking the structure and converting all values to their ClojureScript counterparts (i.e. persistent data structures) the original object is wrapped in a thin layer that allows us to use it as if it were a ClojureScript-native data structure:

(require '[cljs-bean.core :as cljs-bean])

(-> (cljs-bean/bean #js {"some_data" 1, :b 2})
    (get :some_data)) ; => 1

Given a Firestore QueryDocumentSnapshot we can make the JS object representing the data easily accessible from ClojureScript:

(-> (cljs-bean/->clj (.data query-document-snapshot))
    (get :some_field))

;; (cljs-bean/->clj data) is roughly the same as
;; (cljs-bean/bean data :recursive true)

The bean is immutable and can be used in client side app-state as if it is one of ClojureScript’s persistent data structures.

Caveat: Updating a bean using assoc or similar will create a copy of the object (Copy-on-Write). This is less performant and more GC intensive than with persistent data structures. Given that the data is usually quite small and that the document representations in our app state mostly aren’t written to directly this is probably ok (cljs-bean #72).

Whenever we want to use the raw object to update data in Firestore we can simply call ->js on the bean. Conveniently this will fall back to clj->js when called on ClojureScript data structures.

(.set some-ref (cljs-bean/->js our-bean))

Arguably the differences to using plain clj->js aren’t monumental but working with a database representing data as JS objects it is nice to retain those original objects.

Integrating Firestore Metadata

Now we got beans. But they still don’t contain the document ID or reference. In most places we don’t care about a documents ID or reference. So how could we enable the code below while retaining ID and reference?

(sort-by :join_dt participants)

Let’s compare the various options we have.

Tuples and Nesting

I already described the tuple-based approach above. Another, similar, approach achieves the same by nesting the data in another map. Both fall short on the requirement to make document fields directly accessible.

;; structure
{:id "some-id", :ref "/events/some-id", :data document-data}
;; usage (including gymnastics)
(sort-by (comp :join_dt :data) participants)

I’m not too fond of either approach since they both expose a specific implementation detail, that the actual document data is nested, at the call site. In a way my critique of this approach is similar to why Eric Normand advocated for getters in his IN/Clojure ’19 talk — as far as I understand anyways.

Addition of a Special Key

Another approach could be to add metadata directly to the document data.

(defn doc [query-doc-snapshot]
  (-> (cljs-bean/->clj (.data query-doc-snapshot))
      (assoc ::meta {:id (.-id query-doc-snapshot
                     :ref (.-ref query-doc-snapshot})))

This is reasonable and makes document fields directly accessible. However it also requires us to separate document fields and metadata before passing the data to any function writing to Firestore.

;; before writing we need to remove ::meta
(.set some-ref (cljs-bean/->js (dissoc document-data ::meta))

I think this is a reasonable solution that improves upon some of the issues with the tuple and nesting approach. I realize that this isn’t a huge change but this inversion of how things are nested does give us that direct field access that the nesting approach did not.

Protocols and :extend-via-metadata

An approach I’ve found particularly interesting to play with makes use of a protocol that can be implemented via metadata, as enabled by the new :extend-via-metadata option. This capability was added in Clojure 1.10 and subsequently added to ClojureScript with the 1.10.516 release:

(defprotocol IFirestoreDocument
  :extend-via-metadata true
  (id [_] "Return the ID (string) of this document")
  (ref [_] "Return the Firestore Reference object"))

(defn doc [query-doc-snapshot]
  (with-meta
    (cljs-bean/->clj (.data query-doc-snapshot))
    {`id (fn [_] (.-id query-doc-snapshot))
     `ref (fn [_] (.-ref query-doc-snapshot))}))

Using with-meta we extend a specific instance of a bean to implement the IFirestoreDocument protocol. This allows direct access to document properties while retaining important metadata:

(:name participant) ; => "Martin"
(firebase/id participant) ; => "some-firebase-id"

At call sites we use a well-defined API (defined by the protocol) instead of reaching into nested maps whose structure may need to change as our program evolves. This arguably could also be achieved with plain functions.

Sidenote: A previous iteration of this used specify!. Specify modifies the bean instance however, meaning that whenever we’d update a bean the protocol implementation got lost. In contrast metadata is carried over across updates.

Summary

Using cljs-bean we’ve enabled idiomatic property access for JS data structures without walking the entire document and converting it to a persistent data structure. We also retain the original Javascript object making it easy to use for Firestore API calls.

We’ve compared different ways of attaching additional metadata to those documents using compound structures as well as the new and shiny :extend-via-metadata. Using it we’ve extended instances of beans to support a custom protocol allowing open ended extension without hindering the ergonomics of direct property access.

While I really enjoyed figuring out how to extend beans using :extend-via-metadata it turned out that any approach storing data in “unusual places” (i.e. metadata) causes notable complexity when also wanting to serialize the data.

Serializing metadata is something that has been added to Transit quite some time ago but compared to the plug and play serialization we get when working with plain maps it did not seem worth it. Even if set up properly the protocol implementations, which are functions, are impossible to serialize.

Ultimately we ended up with plain beans and storing metadata under a well known key that is removed before writing the data to Firestore again:

(defn doc [query-doc-snapshot]
  (-> (cljs-bean/->clj (.data query-doc-snapshot))
      (assoc ::meta {:id (.-id query-doc-snapshot)
                     :ref (.-ref query-doc-snapshot)})))

(defn id [doc]
  (-> doc ::meta :id))

(defn ref [doc]
  (-> doc ::meta :ref))

(defn data [doc]
  (cljs-bean/->js (dissoc doc ::meta)))

If you're using Firebase or comparable systems, I'd be curious to hear if you do something similar on ClojureVerse.

Thanks to Matt Huebert and Mike Fikes for their feedback & ideas.

Other Posts

  1. 4 Small Steps Towards Awesome Clojure DocstringsJanuary 2019
  2. Sustainable Open Source: Current EffortsJanuary 2018
  3. Maven SnapshotsJune 2017
  4. Requiring Closure NamespacesMay 2017
  5. Simple Debouncing in ClojureScriptApril 2017
  6. Making Remote WorkMarch 2017
  7. Just-in-Time Script Loading With React And ClojureScriptNovember 2016
  8. Props, Children & Component Lifecycle in ReagentMay 2016
  9. Om/Next Reading ListNovember 2015
  10. Parameterizing ClojureScript BuildsAugust 2015
  11. ClojureBridge BerlinJuly 2015
  12. Managing Local and Project-wide Development Parameters in LeiningenJune 2015
  13. Formal Methods at AmazonApril 2015
  14. (lisp keymap)February 2015
  15. CLJSJS - Use Javascript Libraries in Clojurescript With EaseJanuary 2015
  16. Why Boot is Relevant For The Clojure EcosystemNovember 2014
  17. S3-Beam — Direct Upload to S3 with Clojure & ClojurescriptOctober 2014
  18. Patalyze — An Experiment Exploring Publicly Available Patent DataOctober 2014
  19. Running a Clojure Uberjar inside DockerSeptember 2014
  20. Using core.async and Transducers to upload files from the browser to S3September 2014
  21. Emacs & VimJuly 2014
  22. Heroku-like Deployment With Dokku And DigitalOceanMarch 2014
  23. Woodworking MasterclassesFebruary 2014
  24. Early Adopters And Inverted Social ProofFebruary 2014
  25. Living SmallFebruary 2014
  26. Sending You a TelegramJanuary 2014
  27. Running a Marathon, Or NotJanuary 2014
  28. Code SimplicityJanuary 2014
  29. What do we need to know?December 2013
  30. Sculley's DiseaseDecember 2013
  31. A Resurrection PostDecember 2013
  32. A Trip To The USSeptember 2013
  33. Analytics DataApril 2013
  34. Asynchronous CommunicationApril 2013
  35. From Zero to Marathon in Six MonthtsMarch 2013
  36. Git Information in Fish Shell’s PromptDecember 2012
  37. When We Build StuffAugust 2012
  38. Models, Operations, Views and EventsJuly 2012
  39. The Twelve Factor AppJune 2012
  40. Paris And BackMay 2012
  41. A Friend Is Looking For A Summer InternshipMay 2012
  42. Kandan Team ChatMay 2012
  43. Entypo Icon SetMarch 2012
  44. Startups, This Is How Design WorksMarch 2012
  45. Hosting A Static Site On Amazon S3February 2012
  46. Exim4 Fix Wrongly Decoded Mail SubjectJanuary 2012