brianc
7.18.2: "Connection terminated unexpectedly" when using client.query with a pool when pool has been idle for 10 minutes (running in AWS Lambda)
Code is running in a lambda. When we run the function below it runs fine. But if we run it and then wait 10 minutes then run it again, we get an error every time. We are querying Aurora Postgres on AWS.
Code:
const {Pool} = require('pg');
// Global Connection, can be re-used!!
const pgPool = new Pool({
user: process.env.PG_USER,
host: process.env.PG_HOST,
database: process.env.PG_DATABASE,
password: process.env.PG_PASSWORD,
port: process.env.PG_PORT,
max: process.env.MAX_CLIENTS
});
pgPool.on('error', (err, client) => {
console.error('Unexpected error in Postgress connection pool', err);
});
async function performQuery(event) {
let queryString = null;
let args = null;
try {
// some setup code here...
const client = await pgPool.connect();
try {
const res = await client.query(queryString, args);
return res.rows;
} finally {
client.release();
}
} catch (err) {
console.error('Problem executing export query:');
console.error(err); // <-- this line is in the log below
throw err;
}
}
This is what I see in the cloudwatch logs:
{
"errorType": "Error",
"errorMessage": "Connection terminated unexpectedly",
"stack": [
"Error: Connection terminated unexpectedly",
" at Connection.<anonymous> (/var/task/node_modules/pg/lib/client.js:255:9)",
" at Object.onceWrapper (events.js:312:28)",
" at Connection.emit (events.js:223:5)",
" at Connection.EventEmitter.emit (domain.js:475:20)",
" at Socket.<anonymous> (/var/task/node_modules/pg/lib/connection.js:78:10)",
" at Socket.emit (events.js:223:5)",
" at Socket.EventEmitter.emit (domain.js:475:20)",
" at TCP.<anonymous> (net.js:664:12)"
]
}
I've tried a few variations on this but the constant is the 10 minutes and the use of the Pool. To me this code is almost identical to the code in https://node-postgres.com/features/pooling.
So far it looks like the problem has been solved by using a Client instead:
const client = new Client({
user: process.env.PG_USER,
host: process.env.PG_HOST,
database: process.env.PG_DATABASE,
password: process.env.PG_PASSWORD,
port: process.env.PG_PORT
});
await client.connect();
const res = await client.query(queryString, args);
await client.end();
return res.rows;
@jcollum this is also happening for us and a number of others (see this thread https://github.com/knex/knex/issues/3636).
We're seeing this in Node 10 / 12 in the Lambda environment. Only solution at the moment was to run a custom Node 8 runtime for our Lambda functions.
dealing with the same issue on production, working with serverless, over AWS lambdas, node v12 (over typeorm).
@briandamaged @jamesdixon any solution? did not manage to set the runtime to node8...
please help
It looks like it's related to the lambda environment or something and not pg. According to this it's hitting both pg & mysql....which makes it somewhat unlikely to be caused by this library. I'm not sure there's much I can do w/o having steps to reproduce, unfortunately. Looking at @jcollum's example it looks like it's somehow related to the lambda going idle and killing the open connections to the db. Weirdly if you only run the lambda once every 10 minutes that should be well outside the idle timeout and the pool should have closed all it's connections. Anyways I can make lots of assumptions about what's happening but w/o steps to reproduce I'm not gonna be able to provide much concrete support. I'll follow along if more info comes up I'll look into it. Also: PRs are always welcome for issues like this if you figure out what it is! 😉
Thanks for the input, @brianc!
According to this it's hitting both pg & mysql....which makes it somewhat unlikely to be caused by this library
That was the original thought but later on in the thread you referenced, one of the Knex maintainers pointed out that the MySQL error was actually different and may not be related.
The one thing we do know is that the issue isn't present in Node 8 but rears its ugly head in Node 10/12. Are you aware of any major changes in those versions related to this library that might be exacerbated in a Lambda environment?
@jamesdixon : Howdy! Just to clarify: what I meant was that Knex did not appear to be directly related to the pg-specific control path. However, I suspect that the same lower-level issue is affecting both "pg" and "mysql".
My guess is that either the server or the network is aggressively closing the connections. This is then causing the client-side libraries to "get confused" when they discover that the connection is already closed. (At least, that appears to be the case for "pg". The Client class checks the this._ending flag to see whether or not it was expecting the connection to be closed)
Got it! Thanks for the clarification!
On Tue, Feb 25, 2020 at 11:02 AM Brian Lauber [email protected] wrote:
@jamesdixon https://github.com/jamesdixon : Howdy! Just to clarify: what I meant was that Knex did not appear to be directly related to the pg-specific control path. However, I suspect that the same lower-level issue is affecting both "pg" and "mysql".
My guess is that either the server or the network is aggressively closing the connections. This is then causing the client-side libraries to "get confused" when they discover that the connection is already closed. (At least, that appears to be the case for "pg". The Client class checks the this._ending flag to see whether or not it was expecting the connection to be closed)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/issues/2112?email_source=notifications&email_token=AAA24IXYNC5ZRZGUCSJDUOTREVMJ5A5CNFSM4K2TYUC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM44G7I#issuecomment-590988157, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA24ITEVY7NCGYWPMDCCKDREVMJ5ANCNFSM4K2TYUCQ .
I fixed it by not using a pool:
try {
// trimmed code here
const client = new Client({
user: process.env.PG_USER,
host: process.env.PG_HOST,
database: process.env.PG_DATABASE,
password: process.env.PG_PASSWORD,
port: process.env.PG_PORT
});
await client.connect();
const res = await client.query(queryString, args);
await client.end();
return res.rows;
} catch (err) {
console.error('Problem executing export query:');
console.error(err);
throw err;
}
Minor performance hit there.
@jcollum : My guess is that the issue isn't actually "fixed", but rather "avoided". As in: whenever your code needs to perform a query, it is:
- Establishing a new connection (via the freshly-created
Clientinstance) - Performing the query
- Ending the connection cleanly. (by invoking
client.end())
BTW: it looks like there is a connection leak in that snippet. It should probably do this:
try {
// trimmed code here
const client = new Client({
user: process.env.PG_USER,
host: process.env.PG_HOST,
database: process.env.PG_DATABASE,
password: process.env.PG_PASSWORD,
port: process.env.PG_PORT
});
await client.connect();
try {
const res = await client.query(queryString, args);
return res.rows;
} finally {
await client.end();
}
} catch (err) {
console.error('Problem executing export query:');
console.error(err);
throw err;
}
This will ensure that the connection will be closed even if an error occurs.
Yah opening a new client every time is something you can do, but the latency overhead of opening a client is aprox 20x the overhead of sending a query....so ideally you'd use a pool. You don't have to, particularly if your lambda is going cold then pooling really likely isn't buying you much. There might be a way to simulate this in test by creating a pool, reaching into the internals, and severing the streams manually behind the scenes. Anyone care to make a first pass crack at that in a gist or something? If it reproduces the issue I can spin it into a test, make it pass, and ship a fix!
@jcollum : My guess is that the issue isn't actually "fixed", but rather "avoided". As in: whenever your code needs to perform a query, it is:
Well yes it's not a fix it's a workaround. Thanks for catching that potential leak.
@brianc : Thinking about this a bit more... I suspect that both "pg" and "knex" are encountering the same conceptual issue in their Pool implementations. Specifically: they are not re-checking the validity of the Client instances before yielding them from the pool. Here's what I'm seeing on the "knex" side, at least:
https://github.com/knex/knex/blob/8c07192ade0dde137f52b891d97944733f50713a/lib/client.js#L333-L335
Instead, this function should probably delegate to a pg-specific function that confirms that the Client instance is still valid.
I'll check the "pg" codebase in a few mins to see if I can spot a similar behavior there.
@brianc : So, it looks like the "pg" pool has some logic for removing idle clients; however, I suspect that it does not get invoked unless the underlying Client instance actually emits an "error". For ex:
https://github.com/brianc/node-postgres/blob/1c8b6b93cfa108a0ad0e0940f1bb26ecd101087b/packages/pg-pool/index.js#L216
However, it looks like the Client doesn't necessarily emit the "error" event when it detects that the connection has been closed:
https://github.com/brianc/node-postgres/blob/1c8b6b93cfa108a0ad0e0940f1bb26ecd101087b/packages/pg/lib/client.js#L252-L279
So, if this analysis is correct, then it means there is a way for the Pool to contain Client instances that have already been disconnected. (Or rather: their underlying connection instance has already been ended)
@brianc : Here's a snippet that seems to confirm my suspicion:
const {Pool} = require('pg');
function createPool() {
return new Pool({
user: process.env.PG_USER,
host: process.env.PG_HOST,
database: process.env.PG_DATABASE,
password: process.env.PG_PASSWORD,
port: process.env.PG_PORT,
max: process.env.MAX_CLIENTS
});
}
async function run() {
const pool = createPool();
const c1 = await pool.connect();
// Place the Client back into the pool...
await c1.release();
// Now intentionally end the lower-level Connection while Client is already in the pool
c1.connection.end();
// Attempt to obtain the connection again...
const c2 = await pool.connect();
try {
// Explodes
const res = await c2.query("select * from accounts", null);
console.log("Yay");
} catch(err) {
console.log("Initial error")
console.log("----------------")
console.log(err);
} finally {
c2.release();
}
// Attempt to obtain the connection again...
const c3 = await pool.connect();
try {
// Surprisingly, it succeeds this time. This is because the pool was already
// "fixed" thanks to the 'error' event that it overheard.
const res = await c3.query("select * from accounts", null);
console.log("Yay");
} catch(err) {
console.log("Second error")
console.log("----------------")
console.log(err);
} finally {
c3.release();
}
}
run();
Sample output:
Initial error
----------------
Error: write after end
at writeAfterEnd (_stream_writable.js:236:12)
at Socket.Writable.write (_stream_writable.js:287:5)
at Socket.write (net.js:711:40)
at Connection.query (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/connection.js:234:15)
at Query.submit (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/query.js:164:16)
at Client._pulseQueryQueue (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/client.js:446:43)
at Client.query (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/client.js:541:8)
at run (/Users/brianlauber/Documents/src/knex/oops.js:43:26)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:189:7)
(node:57118) UnhandledPromiseRejectionWarning: Error: write after end
at writeAfterEnd (_stream_writable.js:236:12)
at Socket.Writable.write (_stream_writable.js:287:5)
at Socket.write (net.js:711:40)
at Connection.query (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/connection.js:234:15)
at Query.submit (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/query.js:164:16)
at Client._pulseQueryQueue (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/client.js:446:43)
at Client.query (/Users/brianlauber/Documents/src/knex/node_modules/pg/lib/client.js:541:8)
at run (/Users/brianlauber/Documents/src/knex/oops.js:43:26)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:189:7)
(node:57118) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:57118) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Yay
FYI: It looks like you can trigger the issue one layer deeper as well. Ex:
// Added the `.stream` portion here
c1.connection.stream.end();
@jcollum agreed.
Shout out to @briandamaged for digging into this. I'm so thankful for people smarter than I am 🤣
I'm impressed with how quickly this is being looked at
I've been trying to work on pg at least 5 days a week this year. I'll be looking at this first thing tomorrow morning. I'm not 100% sure that making the pool evict closed clients will fix the actual specific issue in lamdas but...it's good behavior to have either way.
Thanks for being so on top of this.
On Tue, Feb 25, 2020 at 8:33 PM Brian C [email protected] wrote:
I'm impressed with how quickly this is being looked at
I've been trying to work on pg at least 5 days a week this year. I'll be looking at this first thing tomorrow morning. I'm not 100% sure that making the pool evict closed clients will fix the actual specific issue in lamdas but...it's good behavior to have either way.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/issues/2112?email_source=notifications&email_token=AAFMAAVJEGOUR5O3PGT3XZTREXWLJA5CNFSM4K2TYUC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM6YE3I#issuecomment-591233645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFMAAQNUNKE5UJFFXNV7CDREXWLJANCNFSM4K2TYUCQ .
@brianc thanks for building such an important library. Your efforts are truly appreciated. Cheers!
:heart: thanks. Looking forward to making lots of folks node apps faster in the next few months when I do some work on perf!
On Tue, Feb 25, 2020 at 10:37 PM James Dixon [email protected] wrote:
@brianc https://github.com/brianc thanks for building such an important library. Your efforts are truly appreciated. Cheers!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/issues/2112?email_source=notifications&email_token=AAAMHIMS2WOYO4X37VPYTO3REXWXLA5CNFSM4K2TYUC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM6YKYY#issuecomment-591234403, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMHIOKFBCV2UFEUSWWVRLREXWXLANCNFSM4K2TYUCQ .
@brianc : Here's a variation on the earlier script that provides some more evidence around the issue:
const {Pool} = require('pg');
function delay(t) {
return new Promise(function(resolve) {
setTimeout(resolve, t);
});
}
function createPool() {
return new Pool({
user: process.env.PG_USER,
host: process.env.PG_HOST,
database: process.env.PG_DATABASE,
password: process.env.PG_PASSWORD,
port: process.env.PG_PORT,
max: process.env.MAX_CLIENTS
});
}
async function run() {
const pool = createPool();
async function doStuff(label) {
const client = await pool.connect();
try {
console.log(`${label}: START`)
const res = await client.query("select 1", null);
console.log(`${label}: SUCCEEDED`);
} catch(err) {
console.log(`${label}: ERROR -> ${err}`)
} finally {
client.release();
}
}
const c1 = await pool.connect();
await c1.query("select 1", null);
await c1.release();
// !!!
// FYI: If you comment out this event handler, then the script
// will fail 100% of the time. (Unhandled 'error' event)
pool.on('error', function() {
console.log("Pool handled error cleanly!")
})
// c1 is in the pool. We're intentionally ending the
// lower-level connection.
c1.connection.stream.end();
// !!!
// FYI: With a 1 second delay, everything appears to work fine.
// However, if you reduce this delay too much, then you'll
// observe the errors that were reported.
console.log("Delaying...");
await delay(1000);
console.log("Finished delay");
await doStuff("ONE");
await doStuff("TWO");
}
run();
There are 2 key things you can play around with here:
- Removing the
pool.on('error', ...)expression (which will cause the script to explode) - Adjusting / removing the delay. (Errors will begin to occur when the delay is set too low)
Even with a delay of 1 second, the problem seems to disappear. So, now I'm unsure if this script is accurately recreating the scenario described by this issue.
Wrapping the pool clients would solve a lot of these problems. Right now pool clients are the raw client with additional functions to release the connection back to the pool. That means that code that directly manipulates the connection, ex: c1.connection.stream.end(), can break things internally. Ditto for continuing to use a client after you've told the pool that you're done with it. With a more restricted interface, say just a .query(...) and .release(...), it's harder to shoot yourself in the foot (though with JS it's never completely impossible).
You also need to ensure to call .release(err: Error) to evict broken clients:
async function doStuff(label) {
const client = await pool.connect();
try {
console.log(`${label}: START`)
const res = await client.query("select 1", null);
console.log(`${label}: SUCCEEDED`);
} catch(err) {
console.log(`${label}: ERROR -> ${err}`)
} finally {
client.release();
}
}
Calling .release() (without a truthy error) instructs the pool that you're done with client and it's okay to return to the pool for reuse. With proper eviction you'll still get parts of your coding potentially receiving broken clients, but at least they'll eventually evict themselves out of the pool.
@sehrope : Right. The purpose of the code snippet was to recreate a scenario where the low-level socket was being closed/ended outside of the Pool's managed lifecycle. So, you're correct: normally, you would not want to do that.
As for the .release(..) part: Javascript does not support try / catch / else syntax. So, expecting the developer to call .release(..) differently depending upon whether or not an exception was raised seems like a usability issue. Ex: it's not possible to do this:
async function doStuff(label) {
const client = await pool.connect();
try {
console.log(`${label}: START`)
const res = await client.query("select 1", null);
console.log(`${label}: SUCCEEDED`);
} catch(err) {
console.log(`${label}: ERROR -> ${err}`)
client.release(err);
} else {
client.release();
}
// Other logic that needs to run after doing stuff w/ `client`
// ....
}
(Okay, fine. It's technically possible, but it forces the developer to create/check didErrorOccur variables)
In either case: the snippet's error is actually triggered by the first call to client.query(..). So, although the detail you pointed out about client.release(..) appears to be true, I'm not sure if it really impacts this snippet in any way. (Side note: the README.md for pg-pool doesn't seem to mention this detail about the call to .release(..), but the code itself looks like it aligns w/ what you're saying. https://github.com/brianc/node-postgres/tree/master/packages/pg-pool )
The main concept is: it's possible for Clients that are already in the Pool to lose their Connections. It's the Pool's responsibility to double-check the validity of a Client before handing it off to a caller. Based upon the issue reported here, it looks like there might be a corner case where this is not occurring.
(Or, alternatively: perhaps several people missed this detail about .release(..), and everybody has been inserting disconnected Clients back into the pool???)
As for the
.release(..)part: Javascript does not supporttry / catch / elsesyntax. So, expecting the developer to call.release(..)differently depending upon whether or not an exception was raised seems like a usability issue.
Yes it's a bit tricky and hopefully when the pool interface gets improved it'll be easier to use. That "pass an error to evict" logic for the pool has been for all the years I've used this driver. All my usage of the driver has a local wrapper that handles that logic so it's in one place in each app, but you're right that everybody kind of has to handle it themselves.
(Okay, fine. It's technically possible, but it forces the developer to create/check
didErrorOccurvariables)
Yes that's the logic to which I'm referring. Though rather than checking if an error occurred the usual approach is to have different release(...) calls based on whether the task resolved or rejected:
const pg = require('pg');
const pool = new Pool({ /* ... */ });
const withClient = async (task) => {
const client = await pool.connect();
try {
const result = await task(client);
// If we get here then the task was successful so assume the client is fine
client.release();
return result;
} catch (err) {
// If we get here then the task failed so assume the client is broken
client.release(err);
throw err;
}
};
That logic is a bit over zealous as it'll evict potentially salvageable client errors (ex: after a successful ROLLBACK we'd expect the client to be usable again) but it's a decent default. If you expect to have lots of constraint violations that could thrash your pool you can make it a bit more intelligent to deal with that situation or have a transaction wrapper handle that.
In either case: the snippet's error is actually triggered by the first call to
client.query(..). So, although the detail you pointed out aboutclient.release(..)appears to be true, I'm not sure if it really impacts this snippet in any way.
It's important to evict broken connections or your pool will never heal and you'll keep getting the same broken connections. Evicting them will force your pool to (eventually) create new working connections.
(Side note: the
README.mdforpg-pooldoesn't seem to mention this detail about the call to.release(..), but the code itself looks like it aligns w/ what you're saying. https://github.com/brianc/node-postgres/tree/master/packages/pg-pool )
Yes we should fix that. The docs site (https://node-postgres.com/api/pool#releaseCallback) has the right usage but the README must be really out of date.
The main concept is: it's possible for Clients that are already in the Pool to lose their Connections. It's the Pool's responsibility to double-check the validity of a Client before handing it off to a caller. Based upon the issue reported here, it looks like there might be a corner case where this is not occurring.
The end usage always needs to deal with broken connections as even if the pool tests the connection, it's a race condition between the pool testing the connection and the end usage actually doing some work with the connection:
- Client asks to check out connection
- Pool tests connection and verifies it's okay
- Connection is killed
- Client receives connection from pool
(Or, alternatively: perhaps several people missed this detail about
.release(..), and everybody has been inserting disconnected Clients back into the pool???)
That's possible. For something like Lambda that can freeze your app and effectively kill any open sockets in the background, I'd suggest not using a pool at all if it's an option. Or you can add your own wrapper that repeatedly tries to fetch and test the connection prior to giving it to the rest of your code. Also, setting the idleTimeoutMillis on the pool to something lower than the Lambda freeze-after-inactive-seconds should limit the problem a bit as you'll be less likely to have any clients int he pool when the Lambda gets frozen.
@sehrope : Agreed -- there is always a race condition betw/ the handoff and the usage. But, based upon what has been reported, it seems that the Client/Connection has already become invalid prior to the handoff.
There is a generic pooling library named Tarn that mitigates this issue by re-validating a resource as part of the handoff. If it discovers that the resource has become invalid, it will:
- Immediately remove the invalid resource from the pool
- Attempt to provide the caller w/ another resource. (ie: grab another one from the pool if possible, or instantiate a new one)
Perhaps a similar strategy can be put in place within pg-pool?
Cross-posting from knex/knex:
There's a good chance that it's a piece of network infrastructure that is discarding the connection silently. The hope is that there is some way to detect this condition. If so, then the pool implementations can verify that the connections are still valid before handing them off to the caller.
I think this is the gist of it. AWS freezes a Lambda for later usage, which includes memory pointers (variables and such) that have been created in a global-ish scope (this post on Medium summarizes it). I would assume this includes pooled clients, which never get a chance to react to closing Sockets.
From the AWS documentation:
After a Lambda function is executed, AWS Lambda maintains the execution context for some time in anticipation of another Lambda function invocation. In effect, the service freezes the execution context after a Lambda function completes, and thaws the context for reuse, if AWS Lambda chooses to reuse the context when the Lambda function is invoked again. This execution context reuse approach has the following implications: Objects declared outside of the function's handler method remain initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. We suggest adding logic in your code to check if a connection exists before creating one.
People praise AWS Lambda, without knowing that it is the only hosting environment in the world that implements aggressive connection-recycle policy.
Nobody it seems is greedy enough as to inconvenience their clients so bad, to decide to drop live connections instead of what everyone else is doing - extending the IO capacity. It's just dumb corporate greed, backed up by self-assured market dominance, to maximize profit by reducing the cost, without increasing the capacity.
That's why issues like this one keep polluting the Internet. That's why I do not use AWS Lambda.
If you are using AWS Lambda for long running processes you're using it wrong.
On Thu, Feb 27, 2020 at 11:37 AM Vitaly Tomilov [email protected] wrote:
People praise AWS Lambda, without knowing that it is the only hosting environment in the world that executes aggressive connection drop policy.
Nobody it seems is greedy enough as to inconvenience their clients so bad, to decide to drop live connections instead of what everyone else is doing - extending the IO capacity. It's just dumb corporate greed, backed up by self-assured market dominance.
That's why issues like this one keep polluting the Internet. That's why I do not use LWS Lambda.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brianc/node-postgres/issues/2112?email_source=notifications&email_token=AAFMAAX2BYUDXURW6K6WBRTRFAI6VA5CNFSM4K2TYUC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENFVN5Y#issuecomment-592140023, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFMAAUWJDDF3RNT73IQTHLRFAI6VANCNFSM4K2TYUCQ .