Module Federation Dev Server uses env variables to pass data, this breaks caching

[statop]

Current Behavior

If I build a module, and then run a federation dev server that uses that module as a static remote, the static remote does not work.

Expected Behavior

If I build a module, and then run a federation dev server that uses that module as a static remote, the static remote works.

GitHub Repo

No response

Steps to Reproduce

See above

Nx Report

>  NX   Report complete - copy this into the issue template

   Node   : 18.17.1
   OS     : darwin-arm64
   pnpm   : 8.14.3

   nx                 : 17.2.8
   @nx/js             : 17.2.8
   @nx/jest           : 17.2.8
   @nx/eslint         : 17.2.8
   @nx/workspace      : 17.2.8
   @nx/angular        : 17.2.8
   @nx/devkit         : 17.2.8
   @nx/eslint-plugin  : 17.2.8
   @nx/node           : 17.2.8
   @nx/plugin         : 17.2.8
   @nx/react          : 17.2.8
   @nx/rollup         : 17.2.8
   @nx/storybook      : 17.2.8
   @nx/vite           : 17.2.8
   @nx/web            : 17.2.8
   @nx/webpack        : 17.2.8
   typescript         : 5.2.2
   ---------------------------------------
   Community plugins:
   @openadp/tools-plugin : 24.5.0
   @storybook/angular    : 7.6.10
   ---------------------------------------
   Local workspace plugins:
   	 @openadp/tools-plugin

Failure Logs

No response

Package Manager Version

pnpm 8.14.3

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

This is the bad code - https://github.com/nrwl/nx/blob/master/packages/react/src/executors/module-federation-dev-server/module-federation-dev-server.impl.ts#L162

Basically env variables are ignored by the nx cache, so building the static remote will pull from cache even if the cached build did not include that env variable.

[Coly010]

Hi @statop !

You can fix this easily by adding the following to the "targetDefaults" object in nx.json.

For Angular:

"@nx/angular:webpack-browser": {
      "inputs": [{
        "env": "NX_MF_DEV_SERVER_STATIC_REMOTES"
      }]
    },

For React:

"@nx/webpack:webpack": {
      "inputs": [{
        "env": "NX_MF_DEV_SERVER_STATIC_REMOTES"
      }]
    },

I will create a PR to add this to the nx.json when an MF generator is invoked.

[statop]

@Coly010

The problem I see with that is that It requires a new build of all deps.

I actually just ended up writing our own dev server for various reasons.

What I did was create a proxy expressjs instance for each static remote on the correct port that just forwards to the "master" static remote server and pre-pends the project name to the url.

That way the build output is always the same for static and non-static builds.

[Coly010]

@statop this would mean you need an express instance for every static remote though right?

[statop]

Correct. I don't think that is too bad. Certainly not as bad as a whole nx nodejs process. Maybe a more lightweight http server could be used, but express is about as light as they get.

[Coly010]

So we did investigations around this, and we created forked processes to spin up individual servers for each remote.

the result was that RAM usage increases as number of remotes increases. As such, it’s not a scalable solution. ~50 remotes is enough to kill a 2019 MacBook Pro (Intel) with 16gb RAM. My laptop is evidence to this.

Which is what led us to the solution we currently have.

a new build is only needed if the development configuration for the remote(s) has been built in a separate process not via shell.

This shouldn’t occur often, and therefore cache will be hit more often than not.

[statop]

Creating forked processes, yes it would do that.

But multiple expressjs in the same process will not do that.

I just ran this code on my machine and the node process is running at 149mb of ram.

const { createProxyMiddleware } = require('http-proxy-middleware');
const express  = require('express');


for (var x = 0; x < 10000; ++x) {

    const expressApp = express();

    expressApp.use(
      createProxyMiddleware({
        target: `http://localhost:${1000}`,
        changeOrigin: true,
        pathRewrite: { '^/': `/${x}/` }
      })
    );
    expressApp.listen(10000 + x);
}

[statop]

To be clear, in my dev server, all the epressjs instances are in the same process as the dev server.

[Coly010]

Ok I understand more clearly what you’re doing now.

This seems like a neat solution. Let us evaluate it.

[Coly010]

Reopening this until we explore the option of multiple express servers in a single process further

[Coly010]

Converted to discussion as the core issue has been solved, but a new approach using express servers in a single process might be more valuable and is worth discussing

[barrymichaeldoyle]

A little bit of a side question here... Should it be changed to NX_PUBLIC_MF_DEV_SERVER_STATIC_REMOTES with v19's new breaking change for env variables where everything with the NX_ prefix needs to be changed to the NX_PUBLIC_ prefix?