Failed detection of Node.js-environment in Next.js 14+ for orama/plugin-data-persistence

[tom-bywild]

Describe the bug

The plugin @orama/plugin-data-persistence is currently not compatible with Next.js 14 when used on the server. It looks like the Node.js-environment is not correctly detected, therefore some logic reserved to Deno gets executed, thus leading to a crash.

The specific location is in a switch statement in server.ts.

This error happens during runtime when the module gets loaded or during build time.

A quick workaround is to use package-patch and comment-out the relevant statement. But this doesn't work in a monorepo setup due to limitations of package-patch.

To Reproduce

1. Setup Next.js application (version 14 or newer)
2. Install Orama + @orama/plugin-data-persistence
3. Import restoreFromFile from @orama/plugin-data-persistence/server in a server-side function (or RSC)
4. See error

Expected behavior

Import & usage of module doesn't crash due to invalid Deno-detection.

Environment Info

OS: MacOS 14.3.1
Node: 21.6.2
Orama: 2.0.8
Plugin: 2.0.7

Affected areas

Initialization

Additional context

orama/packages/plugin-data-persistence/src/server.ts

Line 71 in 8e5e8ef

case 'deno': {

[micheleriva]

Hi @tom-bywild, we do not support Node.js <18, since Node.js 14 went out of maintenance mode in April 2023 and Node.js 16 did the same in September 2023 (if I recall correctly).

If that is the only issue though, we could probably find a quick fix.

I find this error strange though, since we use this function to detect the runtime:

orama/packages/plugin-data-persistence/src/utils.ts

Line 3 in 8e5e8ef

export function detectRuntime(): Runtime {

And I can't see how it doesn't correctly detect Node. Any suggestion? We'll also take a look.

[tom-bywild]

Many thanks for taking a look @micheleriva! But I think there's a small confusion 😄:

• my ticket is about Next.js (React Framework using their app-router) of version 14
• I'm using Node.js 21 in my project

The issue might be that w/ Next.js 14, Vercel started to enforce Web-compliant APIs, basically the server-endpoints have access to a confusing mix of Web and some Node.js-APIs. The main motivation behind this design is that any endpoint can run on new serverless architecture, and such architecture might not support Node.js at all.

Either way, from a naive POV a function that runs on the server conceptually should work with the @orama/plugin-data-persistence/server-exports as well.

Therefore the detection of the persistence-adapter might have to be updated, or issue lies somewhere else.

[micheleriva]

OMG sorry, I'm never replying late at night again 😂

Let me try to reproduce on Next.js 14 then. Let me tell you immediately, this won't work on edge routes (since they use Cloudflare Workers), but this should work just fine on any other API route.

[hassankhan]

Hi, just adding the logs I get when trying to restore a binary file:

 ⚠ Fast Refresh had to perform a full reload due to a runtime error.
 ⨯ https://deno.land/std/path/mod.ts
Module build failed: UnhandledSchemeError: Reading from "https://deno.land/std/path/mod.ts" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "https:" URIs.
Import trace for requested module:
https://deno.land/std/path/mod.ts
../../node_modules/@orama/plugin-data-persistence/dist/server.js
./src/app/api/search/route.ts
 ○ Compiling /_error ...
 ⨯ https://deno.land/std/path/mod.ts
Module build failed: UnhandledSchemeError: Reading from "https://deno.land/std/path/mod.ts" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "https:" URIs.
Import trace for requested module:
https://deno.land/std/path/mod.ts
../../node_modules/@orama/plugin-data-persistence/dist/server.js
./src/app/api/search/route.ts

My API route:

import { search } from '@orama/orama';
import { NextResponse } from 'next/server';
import { restoreFromFile } from '@orama/plugin-data-persistence/server';

const indices = {
  'repos': './repos.msp',
  'comments': './comments.msp',
  'posts': './posts.msp',
}

const hydrateIndices = async () => {
  return {
    repos: await restoreFromFile('binary', indices['repos'], 'node'),
    comments: await restoreFromFile('binary', indices['comments'], 'node'),
    posts: await restoreFromFile('binary', indices['posts'], 'node'),
  }
}

export async function GET(request: Request) {
  
  const {searchParams} = new URL(request.url);
  const query = searchParams.get('q');
  const hydratedIndices = await hydrateIndices();
  
  const r1 = search(hydratedIndices['repos'], { term: query });
  const r2 = search(hydratedIndices['comments'], { term: query });
  const r3 = search(hydratedIndices['posts'], { term: query });
  
  return NextResponse.json({ r1, r2, r3 });
}

I even exposed the detectRuntime() util to my application (by modifying code in my node_modules/) and it says that its node.

[tom-bywild]

What worked for me was not using '@orama/plugin-data-persistence/server' but instead the (polymorphic?) client-JS import: '@orama/plugin-data-persistence'

link

[hassankhan]

Thanks for posting that workaround @tom-bywild 👍

I got around the issue by persisting the index as JSON, then using JSON.parse() and passing the parsed object to load() from @orama/orama:

import { load } from '@orama/orama';

const deserialized = JSON.parse(file);
await load(db, deserialized);

[addisonj]

I am also hitting this issue, but it seems like the workaround suggested by @tom-bywild does not work as the code here throws an exception when this method is accessed... I am instead just doing the IO manually, but this was a weird one to figure out why it is failing to detect the environment in next 14

[micheleriva]

Adding a bounty for this bug

/bounty 100

[algora-pbc]

💎 $100 bounty • OramaSearch

Steps to solve:

1. Start working: Comment /attempt #649 with your implementation plan
2. Submit work: Create a pull request including /claim #649 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to askorama/orama!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @valstu		#696

[neo773]

Problem is you're trying to do a fs call in edge runtime which isn't supported.

This works

import { search } from '@orama/orama';
import { restoreFromFile } from '@orama/plugin-data-persistence/server';
import type { NextApiRequest, NextApiResponse } from 'next'

const indices = {
  'repos': './repos.msp',
}

const hydrateIndices = async () => {
  return {
    repos: await restoreFromFile('binary', indices['repos'], 'node'),
  }
}

export const runtime = 'nodejs'

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  const hydratedIndices = await hydrateIndices();
  const r1 = search(hydratedIndices['repos'], { term: 'who' });
  return res.json({ r1: r1 });
}

[neo773]

Making this work on edge will require building a middleware that implements a virtual file system over the network.

[valstu]

I added PR which should help with the original Next.js issue. Or that's how I fixed this earlier when I faced the same problem. If you want to try this out you can install this npm package https://www.npmjs.com/package/@valstu/plugin-data-persistence which includes this small change.

[micheleriva]

That's awesome thank you @valstu!

@tom-bywild please let me know if that solves the issue for you.

[valstu]

I added PR which should help with the original Next.js issue. Or that's how I fixed this earlier when I faced the same problem. If you want to try this out you can install this npm package https://www.npmjs.com/package/@valstu/plugin-data-persistence which includes this small change.

@tom-bywild did this work for you?