How to publish Deno modules to NPM

I wrote oak, a full featured HTTP middleware/router framework. It is the most used HTTP framework for Deno and powers many sites; for example doc.deno.land.

A lot of people want to use Deno as their primary development platform but also want to be able to share code with-in the Node ecosystem. This is easily possible using dnt. In this article, I will show you how I published Oak to NPM module in a way that it is useable in Node.

• Overview of oak
• dnt
• Getting oak to run on Node.js
  • Web standards
  • ESM and Node.js
  • Supporting HTTP on Node.js
  • Other tidbits
  • Building, testing and CI
  • Publishing to npm
• Conclusions

Note: This article is accurate at the time of publishing, Deno, oak, dnt, and Node.js will continue to evolve and specific technical details and statements may not be accurate in the future.

Overview of oak

If you aren’t familiar with oak, it is a koa inspired HTTP middleware framework along with a router. It’s main purpose is to provide a structured way of handling HTTP requests. While the fundamentals of oak have been consistent since it was released in December of 2018 (which Deno was v0.2 at the time), it has continued to evolve as the Deno CLI evolved, including migrating to the native HTTP server and ensuring the Deno Deploy is supported.

Basic usage in Deno is straight forward:

import { Application } from "https://deno.land/x/oak/mod.ts";

const app = new Application();

app.use((ctx) => {
  ctx.response.body = "Hello from oak";
});

app.listen({ port: 8000 });

But it was always designed with Deno in mind, leveraging built-in Deno APIs to be able to not only handle HTTP requests and responses, but also do advanced features like generating ETag signatures for static files and complicated form data body handling. Running it on Node.js was never a design consideration.

dnt

dnt provides a build pipeline to transform Deno code into a Node-compatible NPM page.

The pipeline does the following at a high level:

• Transforms Deno code to Node.js compatible TypeScript code:
  • Rewrites Deno’s extensioned module specifiers to ones compatible with Node.js module resolution.
  • Injects shims for any Deno namespaces APIs detected, as well as other globals which can be configured.
  • Rewrites remote imports from Skypack or esm.sh as bare specifier imports and adds them to the package.json as dependencies.
  • Other remote imports are downloaded and included in the package.
• Type checks the transformed TypeScript code with tsc.
• Writes out the package as a set of ESM, CommonJS and TypeScript type declaration files along with a package.json.
• Runs the final output in Node.js through a test runner which supports the Deno.test() API.

This means you can develop and test all of your code in Deno and when you want to publish it to npm and make it available for Node.js, you use dnt to export it and validate that it works as expected, all with minimal or no changes to your source code.

Once you have it setup, you can continue to iterate on your package, easily publishing it for both consumption in Deno as well as via npm.

Getting oak to run on Node.js

There were a lot of lessons learned in getting it to work, and not everything is done yet. Support for HTTP/2, web sockets and handling for FormData files need to be added, but the main functionality works.

Web standards

Deno fully leverages web standards and the web platform. Node.js is increasingly adding web platform APIs, but they are often not exposed globally and some still require dependencies. These were ones that needed to be addressed with oak:

• Web compatible Streams are globally available in Deno, but only available via built-in module "stream/web" on Node.js. These needed to be exposed via the dnt configuration:

  {
    shims: {
      custom: [{
        package: {
          name: "stream/web",
        },
        globalNames: ["ReadableStream", "TransformStream"],
      }],
    }
  }

• Web crypto is globally available in Deno, but only available via webcrypto symbol via the "crypto" built-in module on Node.js. This needed to be exposed via the crypto: true option in the shims section of the dnt configuration.

• While not used directly in oak, some of the dependencies needed the Blob global made available from the builtin "buffer" module. This is exposed via the blob: true option in the shims section of the dnt configuration.

• Deno exposes the web standard fetch() and Headers which are used by oak. Node.js currently doesn’t have them available built-in (though that is changing) and so they needed to be exposed via the "undici" package, which dnt supports via the undici: true options in the shims section of the dnt configuration.

• oak extends the web standard ErrorEvent for internal errors which is available globally in Deno. Node.js does not have this and so I had to create an ErrorEvent class which extends the globally available Event in Node.js and is loaded by dnt as a global shim.

ESM and Node.js

Deno was built around ES modules and while Node.js has un-flagged support for ES Modules for an extended period of time, it is still a complicated issue. For many workloads dnt can easily down-emit your code to CommonJS and ESM to make it possible for a consumer of your package to choose how to load it in Node.js. For oak, this was a bit more complex. There is one situation where Node.js cannot support ES modules, and that is the situation of top-level await.

I had a usage of top-level await to initialize a variable that I needed to refactor in order to support CommonJS.

Supporting HTTP on Node.js

The biggest difference between Deno and Node.js is on how they handle HTTP requests. Deno has evolved, where initially it only supported HTTP via the Deno std library, it now provides a native implementation built around the web platform fetch() APIs. Node.js’s builtin "http" module as a very low level API that has large remained unchanged since the early days of Node.js.

When oak migrated from the std library HTTP server to the native one, I implemented an abstraction layer for handling the requests and responses. Even when the support for the std library HTTP server was dropped from oak, the abstraction was retained. This abstraction, with a bit of refactoring and improvement, allowed support of the low-level Node.js "http" server with very little changes in the rest of the code base.

If you are interested, the code is in the http_server_node.ts module.

After some refactoring, I was able to use the dnt and the mapping feature to “swap out” the Deno abstraction with the Node.js abstraction. This is a minimal amount of platform specific code (about 160 lines for Deno and 220 lines for Node.js).

Other tidbits

One of the initial test failures I had when running the tests under Node.js was for the custom inspect logic that oak has for many of its classes. In Deno, the custom inspection method is defined by the "Deno.customInspect" symbol, but in Node.js it is available as "nodejs.util.inspect.custom" symbol and also has a slightly different API. So I added a "nodejs.util.inspect.custom" symbol method for each of the classes I had a Deno custom inspect upon and adjusted it to better align to the Node.js API.

Even with all that, there are still subtle differences in the output of inspect, and so it was one of the few places in oak where I actually had to branch test code based on if it running under Deno or Node.js

Also, the undici Headers class varies from the Deno Headers class. Deno’s version passes all the WHATWG tests, and undici’s was designed with Node.js first in mind. There is a fundamental problem with the web standard and server side JavaScript though and the use of Headers, specifically it has to do with the special treatment of Set-Cookie header, which is only set server side. Both Deno and undici independently solve this problem, and so this results in different behaviors when running on Deno and Node.js. I think this behaviors won’t impact people directly, but they may, and I might have to make changes in the future to accommodate for that in oak.

Building, testing and CI

dnt a build pipeline you would typically integrate into a build process. For oak I chose to create a _build_npm.ts script, which in addition to running the dnt pipeline, does some other build steps that fall outside of the scope of dnt.

Generally you only need to write tests once, and by default dnt will run your tests under the integrated Deno like test harness under Node.js for you. With oak, I had a few tests that were specific to Deno or Node.js, as well as some tests results that varied between the platforms. I tried my best to avoid branching or ignoring tests, but in some cases they are unavoidable. I created a simple utility function for detecting the environment:

export function isNode(): boolean {
  return "process" in globalThis && "global" in globalThis;
}

Integrating it all into CI was very straight forward, as all that needed to be done was to run the build script as part of the CI process, which will build, type check, and test the package for me.

Publishing to npm

Once you have dnt building your package for you, it is ready for publishing to npm. All I need to do with oak is change to the output directory of my build script followed by npm publish.

Conclusions

I am biased obviously, but I think Deno is a great development platform as well as a great runtime for JavaScript and TypeScript. With a rich set of modern built-in APIs, lots of support for web platforms APIs, and built-in support for authoring code in TypeScript, it is a great runtime. Add in the “batteries included” testing, binary redistribution packaging, debugging, and IDE language server, it is hard to find a better experience.

One of the big blockers for adoption though has been how to develop code in Deno but share it with Node.js without duplicating lots of code, or having to do lots of work yourself to make things work. I feel dnt goes a really long way of taking that barrier away, and taking something like oak and being able to write code once in Deno but share it easily with Node.js is a really good example of this.

We would really love to see more projects start to follow this path. Inevitably we will discover things that need to be changed or improved with dnt, and we would love to hear feedback and experiences with it.