databricks
Deprecated Buffer() Usage Warning in @databricks/sql v1.7.1
Issue Description
I'm encountering multiple issues with the @databricks/sql library version 1.7.1 in a Node.js 20.x Lambda function environment:
Deprecation warning when using the library:
[DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
When attempting to upgrade from v1.7.1, I encounter a dependency resolution error:
Error: Could not resolve "../build/Release/xxhash"
node_modules/lz4/lib/utils.js:4:18:
4 │ var XXH = require('../build/Release/xxhash')
Steps to Reproduce
- Use @databricks/sql v1.7.1 in a Node.js 20.x environment
- Execute any operation that uses the library's connection functionality
- When attempting to upgrade to a newer version, the xxhash dependency fails to resolve
Current Behavior
- v1.7.1 generates deprecation warnings about unsafe Buffer usage
- Upgrading to newer versions results in dependency resolution errors with the lz4 package and its xxhash dependency
Expected Behavior
- The library should use the recommended Buffer methods instead of the deprecated Buffer constructor
- Dependencies should be properly resolved during installation without errors
Environment Information
Node.js version: 20.x @databricks/sql version: 1.7.1 Runtime environment: AWS Lambda Architecture: x86_64 Function memory: 3008MB
Additional Context
I'm caught between two issues:
- Current version (1.7.1) works but produces deprecation warnings
- Attempting to upgrade to newer versions fails with dependency resolution errors
This suggests there might be issues with how the library's dependencies are packaged or with compatibility in Node.js 20.x environments.
Impact
These issues affect the reliability and maintainability of applications using this library, especially in serverless environments where dependency resolution is critical.
Hi, this is still a problem for me in 1.12.0. I was looking into the cause of this warning and found that it comes from the thrift package. Databricks driver currently uses 0.16.0 version of thrift but manually updating it to 0.22.0 also doesn't solve the problem. Looking at the source code of thrift, plain Buffer seems to be used quite a few time including in thirft's deprecated dependency called node-int64. I would consider rising an issue in the thrift repo but for the lack of time I am going to patch my esbuild bundle with a custom plugin that replaces all occurrences of Buffer() usage with Buffer.from().
P.S. My nodejs version is v22.15.0 running in a local dev environment
Well, I ended up with roughly the following monstrosity if this helps someone else:
const fs = require('fs');
const path = require('path');
const { parse } = require('@babel/parser');
const traverse = require('@babel/traverse').default;
const t = require('@babel/types');
const generate = require('@babel/generator').default;
const outfile = "mybundle.mjs"
const polyfillPlugin = {
name: 'polyfill',
setup(build) {
build.onLoad({ filter: /\.[cm]?[jt]sx?$/ }, async (args) => {
const contents = await fs.promises.readFile(args.path, 'utf8');
const ext = path.extname(args.path);
const loaderMap = {
'.js': 'js',
'.mjs': 'js',
'.cjs': 'js',
'.jsx': 'jsx',
'.ts': 'ts',
'.tsx': 'tsx',
};
const loader = loaderMap[ext] ?? 'js';
const ast = parse(contents, {
sourceType: 'unambiguous',
plugins: ['typescript', 'jsx'],
});
traverse(ast, {
NewExpression(path) {
const { callee, arguments: args } = path.node;
if (!t.isIdentifier(callee, { name: 'Buffer' })) return;
if (args.length === 1) {
const x = args[0];
// typeof x === 'number' ? Buffer.alloc(x) : Buffer.from(x)
const cond = t.binaryExpression(
'===',
t.unaryExpression('typeof', t.parenthesizedExpression(x), true),
t.stringLiteral('number')
);
const alloc = t.callExpression(
t.memberExpression(t.identifier('Buffer'), t.identifier('alloc')),
[x]
);
const from = t.callExpression(
t.memberExpression(t.identifier('Buffer'), t.identifier('from')),
[t.cloneNode(x, true)]
);
path.replaceWith(t.conditionalExpression(cond, alloc, from));
} else {
// Buffer.from(...args)
path.replaceWith(
t.callExpression(
t.memberExpression(t.identifier('Buffer'), t.identifier('from')),
args
)
);
}
},
});
const modifiedContents = generate(
ast,
{ retainLines: true, compact: false },
contents
).code;
return {
contents: modifiedContents,
loader: loader,
};
});
},
};
require('esbuild')
.build({
entryPoints: ['index.ts'],
bundle: true,
platform: 'node',
minify: false,
target: 'node18',
format: 'esm',
outfile,
plugins: [polyfillPlugin],
})
.catch((error) => {
console.error('Error:', error);
process.exit(1);
});
We need to replace new Buffer(<number>) to Buffer.alloc(<number>) and new Buffer(<not number (e.g. array, string) or multiple args>) to Buffer.from(<...>). So what I do is I find all places with multiple args passed into Buffer and replace them with Buffer.from(<...args>) directly. Then for cases where a single parameter is passed into the Buffer constructor, we need to match the see if the argument is a number or not so I replace with: typeof <arg> === 'number' ? Buffer.alloc(<arg>) : Buffer.from(<arg>).
AST is needed because matching parenthesis and other language specific things with regex is limited.
A relevant issue inside the Thrift project already exists on their issue tracker: THRIFT-5224 There was also a PR to fix it, which unfortunately was closed due to inactivity. I might try to push this topic again when I have more time, but in case I don't get back to this, I am leaving these links for others interested in contributing and fixing this problem for good