Bug: Incompatible with serverless-plugin-typescript

[dustyjewett]

When I add serverless-plugin-typescript to the project, I get some errors from typescript trying to unlink directories.

PR for a breaking build:
dustyjewett#1
(I wasn't able to run yarn test:cli before or after my changes, so I'm just assuming the tests break, as I'm able to run serverless package in the test directory on master, but it fails in my branch)

Serverless: Compiling with Typescript...
Serverless: Typescript compiled.
 
  Error --------------------------------------------------
 
  Error: EISDIR: illegal operation on a directory, unlink '/mnt/5aa17a51-a0c2-414d-aca1-4b6a28f5f4a0/projects/serverless-jetpack/test/packages/monorepo/yarn/.build/node_modules'
      at Object.unlinkSync (fs.js:956:3)
      at TypeScriptPlugin.<anonymous> (/mnt/5aa17a51-a0c2-414d-aca1-4b6a28f5f4a0/projects/serverless-jetpack/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/src/index.ts:195:12)
      at Generator.next (<anonymous>)
      at /mnt/5aa17a51-a0c2-414d-aca1-4b6a28f5f4a0/projects/serverless-jetpack/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:7:71
      at new Promise (<anonymous>)
      at __awaiter (/mnt/5aa17a51-a0c2-414d-aca1-4b6a28f5f4a0/projects/serverless-jetpack/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:3:12)
      at TypeScriptPlugin.copyDependencies (/mnt/5aa17a51-a0c2-414d-aca1-4b6a28f5f4a0/projects/serverless-jetpack/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:166:16)
      at TypeScriptPlugin.<anonymous> (/mnt/5aa17a51-a0c2-414d-aca1-4b6a28f5f4a0/projects/serverless-jetpack/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/src/index.ts:45:20)
      at Generator.next (<anonymous>)
      at fulfilled (/mnt/5aa17a51-a0c2-414d-aca1-4b6a28f5f4a0/projects/serverless-jetpack/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:4:58)
      at process._tickCallback (internal/process/next_tick.js:68:7)

[ryan-roemer]

Thanks for the report. I'm digging in to see if I can repro your PR.

Is the plugin actually failing a simple build? A reproduction of that would be way easier to diagnose if possible if you could put one up there? (The failure here appears to be symlink-related in the monorepo fixture. Jetpack doesn't do any symlinking, so it may just be an artifact of yarn workspaces + lerna symlinks fighting with the TS plugin or something).

And what version of jetpack are you using?

[ryan-roemer]

... I checked out your branch. With this diff:

$ git diff serverless.yml
diff --git a/test/packages/monorepo/yarn/serverless.yml b/test/packages/monorepo/yarn/serverless.yml
index 145bff8..38ff27f 100644
--- a/test/packages/monorepo/yarn/serverless.yml
+++ b/test/packages/monorepo/yarn/serverless.yml
@@ -24,7 +24,6 @@ plugins:
   modules:
     - serverless-plugin-typescript
     - serverless-offline
-    - ../../../plugins/wrapper
 
 provider:
   name: aws

Which completely removes serverless-jetpack under any condition and this package command:

$ SLS_DEBUG="*"   STAGE="sandbox"   AWS_REGION="us-east-1"   serverless package

I get your error, which means it's not jetpack specific I think:

Serverless: Compiling with Typescript...
Serverless: Typescript compiled.
 
  Error --------------------------------------------------
 
  Error: EPERM: operation not permitted, unlink '/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/.build/node_modules'
      at Object.unlinkSync (fs.js:956:3)
      at TypeScriptPlugin.<anonymous> (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/src/index.ts:195:12)
      at Generator.next (<anonymous>)
      at /Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:7:71
      at new Promise (<anonymous>)
      at __awaiter (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:3:12)
      at TypeScriptPlugin.copyDependencies (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:166:16)
      at TypeScriptPlugin.<anonymous> (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/src/index.ts:45:20)
      at Generator.next (<anonymous>)
      at fulfilled (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless-plugin-typescript/dist/src/index.js:4:58)
      at process._tickCallback (internal/process/next_tick.js:68:7)
  From previous event:
      at PluginManager.invoke (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/lib/classes/PluginManager.js:489:22)
      at getHooks.reduce.then (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/lib/classes/PluginManager.js:524:24)
  From previous event:
      at PluginManager.run (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/lib/classes/PluginManager.js:524:8)
      at variables.populateService.then (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/lib/Serverless.js:115:33)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:126:23)
  From previous event:
      at Serverless.run (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/lib/Serverless.js:102:74)
      at serverless.init.then (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/bin/serverless.js:72:30)
      at /Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/node_modules/fs-extra/node_modules/graceful-fs/graceful-fs.js:111:16
      at /Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:111:16
      at /Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:45:10
      at FSReqWrap.args [as oncomplete] (fs.js:140:20)
  From previous event:
      at initializeErrorReporter.then (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/bin/serverless.js:72:8)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:126:23)
  From previous event:
      at Object.<anonymous> (/Users/rye/scm/vendor/jp-dustyjewett-ts-issue/test/packages/monorepo/yarn/node_modules/serverless/bin/serverless.js:61:4)
      at Module._compile (internal/modules/cjs/loader.js:778:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
      at Module.load (internal/modules/cjs/loader.js:653:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
      at Function.Module._load (internal/modules/cjs/loader.js:585:3)
      at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
      at startup (internal/bootstrap/node.js:283:19)
      at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com
 
  Your Environment Information ---------------------------
     Operating System:          darwin
     Node Version:              10.16.3
     Framework Version:         1.57.0
     Plugin Version:            3.2.4
     SDK Version:               2.2.1
     Components Core Version:   1.1.2
     Components CLI Version:    1.4.0

Let's maybe have you turn to a minimal repository that produces the issue specific to jetpack so I can clone it, install, and follow your steps to a jetpack-specific error? Thanks!

[ryan-roemer]

Oh, and as a helpful tip, if the TS plugin relies on setting package.artifact it's likely not worthwhile using with jetpack as jetpack ignores configurations that have .artifact already set.

And it the TS plugin creates a zip in lieu of serverless, then it will be incompatible jetpack, because that's pretty much all jetpack does. (If by contrast, it just creates JS transpiled files to get picked up my normal packaging, it should work just fine).

[dustyjewett]

Thanks Ryan!

I think the issue might be the monorepo + serverless + jetpack + typescript... So it might be that the typescript plugin might not be compatible with jetpack's monorepo support? If I take jetpack out completely, the monorepo will stop working. So... the diff between "working" and "broken" will be a little bit more involved than I'd like.

I'm just digging into serverless, trying to share TS models between my backend lambda and frontend react inside a lerna repo... I ran into an issue, found a bug, and saw your reply suggesting jetpack.

[ryan-roemer]

I've taken a skim of serverless-plugin-typescript and I think you've got some fundamental issues with that one.

First, the TS plugin appears to just straight up not work with lerna / yarn workspaces that do symlinks. Vanilla serverless should be able to package monorepo code, just not efficiently (you'll have to do lots of patterns and it won't correctly exclude devDeps). But it should be able to run. Here, with just vanilla serverless + TS plugin with no jetpack, things break. That needs to be solved first. I'd review upstream TS issues -- maybe this one?

Second, if I'm reading things right, the TS plugin appears to build TS and deps in a separate path completely outside of serverless mechanics, then in a weirdly late hook (after:package:createDeploymentArtifacts) override package.artifact. I might be wrong about how this works but it looks like that the TS plugin controls the whole build and there's nothing really that jetpack (or vanilla serverless) can do. (I'm not even sure how they exclude devDeps in a normal project).

... so unless there's something more to the second issue, I think you're looking at:

1. Solving these issues upstream in serverless-plugin-typescript allowing (1) the plugin to work with vanilla serverless with your monorepo, and (2) maybe copying jetpack's prod dep filtering logic (a lot of it is abstracted to the inspectdep lib which could be consumed by the TS plugin).

2. Doing the TS build manually via script commands before packaging/deploying with Serverless. Then you can use all of normal sls and jetpack (and is what we've done many times for clients). If you want everything in the sls lifecycle, you could use something like https://www.npmjs.com/package/serverless-scriptable-plugin with maybe a config of:

   plugins:
     - serverless-scriptable-plugin
   
   custom:
     scriptHooks:
       # These are all the places that jetpack could be invoked that would
       # ensure the TS files are built right before jetpack kicks in.
       before:package:createDeploymentArtifacts: yarn run build-ts-files
       before:package:function:package: yarn run build-ts-files
       before:jetpack:package:package: yarn run build-ts-files

Good luck and hope that helps!

[dustyjewett]

Manual ts build worked great! Thanks for the help!

[wuelcas]

I know this is closed but I had this exact issue on Jenkins after I added a new npm package to the project. Looks like Jenkins was persisting the old TS compiled files and link to the node_modules folder. What I did to solve it was deleting the output folder (my folder is named .build, so I just did a rm -rf .build) for the compiled files and then run serverless. Solve it for me.

[top-dev-0727]

In my case, there was anproblem in the serverless.yml file. There I added package: exclude: ["node_modules"] and this error had gone.