Externs: The Bane of every Release Build

Ever run into errors like these?

Cannot read property ‘c’ of null

ln.S is not a function

Your app runs fine in development and but when trying :advanced it blows up?

Externs or rather the lack thereof are often to blame here and possibly the most frequent issue coming up in the #clojurescript Slack (and elsewhere).

Externs?

Externs if you don’t know what they are can be quite challenging and frustrating to learn. ClojureScript is optimized by the Closure Compiler which attempts to optimize your whole program. Sometimes it doesn’t have access to the whole program, eg. when using a foreign JS library, so we need to tell the Compiler about those. This is what externs do. They inform the Compiler about the structure of some external JavaScript it does not see. The externs protect the given property names so Closure does not attempt to shorten (or remove) them.

On top of that the Closure Compiler acts as a type checker and the Closure Library is fully typed. The type information enables some more :advanced optimizations that are not possible without. In theory this means that your :externs also need to be typed which is not so great if you are otherwise writing untyped code. ClojureScript is untyped and so is most code available on npm.

Fortunately whenever the Closure Compiler finds a property but cannot determine the type of the object it is on it will be conservative and check if the property it defined anywhere in the externs. If that is the case the compiler will not rename the property and all is well.

Writing Externs manually

Unfortunately we still need to tell the compiler about those properties which isn’t always easy. shadow-cljs has some built-in utilities to ease this process.

We can create a release build with the --pseudo-names option which will still rename everything but in a way that is still human-readable.

shadow-cljs release app --pseudo-names

# turns
Uncaught TypeError: ln.S is not a function
# into
Uncaught TypeError: something.$foo$ is not a function

If you have done this before this tells you that foo needs to be defined in externs. This process can be quite tedious and you need to recompile your app with every added externs and not all things happen directly on startup. Some errors only happen after clicking a thing after clicking three other things.

I added shadow-cljs check app to remove some of this burden as it runs all the code through the Closure Compiler type checker so you’d get warnings about externs during compilation. This could look like

------ WARNING #1 --------------------------------------------------------------
 File: ~/.m2/repository/org/clojure/clojurescript/1.9.946/clojurescript-1.9.946.jar!/clojure/string.cljs:33:35
--------------------------------------------------------------------------------
  29 |   (let [r (js/RegExp. (.-source re)
  30 |                       (cond-> "g"
  31 |                         (.-ignoreCase re) (str "i")
  32 |                         (.-multiline re) (str "m")
  33 |                         (.-unicode re) (str "u")))]
-----------------------------------------^--------------------------------------
 Property unicode never defined on re
--------------------------------------------------------------------------------
  34 |     (.replace s r replacement)))
  35 |
  36 | (defn- replace-with
  37 |   [f]
  38 |   (fn [& args]
--------------------------------------------------------------------------------

Unfortunately this is not totally accurate either and often complained about non-issues given that it works off a mostly untyped codebase.

Manually writing externs is not fun. Period.

CLJSJS

CLJSJS is a community effort to pre-bundle existing JS libraries and making them usable by the ClojureScript compiler. As a bonus it is possible to bundle externs as well, which means someone only needs to write externs once and everyone using this library benefits from them.

This is way better but what if the JS code you want to use is not available via CLJSJS?

Generating Externs

What if we generated externs?

David Nolen implemented the Externs Inference for the ClojureScript compiler. This works pretty well but sometimes relies on the user annotating the code in a couple of places which is not something you’d do (well, me at least) naturally. So you typically still run into the ln.S is not a function issue, then enable “pseudo names” and then annotate your code. In a way thats still writing externs manually.

Going for 100% automation

As described in my previous post I implemented a custom JS bundler for shadow-cljs. This means that it processes all the JS code from npm and bundles it in a way we can use from ClojureScript. To do this it also runs the code through :simple optimizations which is similar to what UglifyJS in the JS world would do.

Since it is compiling the JS anyways we can collect all the names used in the external JS and use those as externs when running the rest of the build through :advanced compilation.

Combining this with some of the information gathered by the externs inference code we get pretty reliable results and typically do not need to write manual externs.

Well, almost 100%

The caveat is that this only works when shadow-cljs is processing ALL of the JS. It cannot do this when emitting code for node.js since that just maps directly to require and lets node deal with resolving that. There are also perfectly reasonable scenarios in :browser builds where code won’t be processed by shadow-cljs (eg. including some JS via a <script> directly from a CDN).

Covering the last few percent

Sometimes you are just going to need to write some manual externs. To at least ease this process shadow-cljs now supports writing simplified externs files which look something like

# comments start with #

# properties are just listed one per line
propertyA
propertyB

# globals are prefixed by global:
global:someGlobal

You only need to list all the properties you want to preserve and any globals you might use. No need to do any type annotations.

You put these into a file at <project-root>/externs/your-build-id.txt and shadow-cljs will pick them up automatically. Of course the usual :compiler-options {:externs ["path/to/externs.js"]} still works and can be combined with all of the above.

Conclusion

I hope that the new externs generation features of shadow-cljs can reduce some of the pain caused by externs. I have done a few tests where I was able to remove all of the externs I had to add manually before. Getting to truly 100% is not possible due to the dynamic nature some of the JS on npm.

I do think that all the tools shadow-cljs provides do make this whole process a whole lot less painful. I’m curious to gather more data on this and want to hear about your experiences when you try it. Everything mentioned here is available since [email protected], it is enabled by default so no configuration required.