SELECT with large IN clause creates java.lang.Error from JNI [CORE5897] #6155
Comments
|
Commented by: Karl Miller (searchengine27) removing {code} annotations for JIRA since they have no effect. |
Modified by: Karl Miller (searchengine27)description: I have a SQL query that looks something like: I found out quick that Firebird's IN clause has a limit of 1500, so I started batching by increments of 1500. However, when using the max (and 'near' to the max), I end up with a java.lang.Error that originates from the JNI: I also attempted to set my max primary keys to load to 1499, to see if maybe there was an off by 1 error somewhere, but same result occurred. Setting my max number of keys within the IN clause to 1000 allowed the statement to continue. My connection itself is through an embedded connection, using the Jaybird 3.0.3 driver, but I don't think that's the issue. I also tried increasing the heap space of my JVM to 4GB (admittedly my knowledge of JNI behind the scenes is lacking, but without Googling, I'm assuming JNI calls still occur within the same heapspace/process), but that also appeared to have no effect Workaround is to limit keys within IN clauses to 1k or lower, for now. => I apologize for the raw code and stack trace; I thought this was JIRA and tried to use {code} annotations, but it stays unformatted, so I'm not sure how to make this look better. I have a SQL query that looks something like: I found out quick that Firebird's IN clause has a limit of 1500, so I started batching by increments of 1500. However, when using the max (and 'near' to the max), I end up with a java.lang.Error that originates from the JNI: (FYI I removed the parts of the stack trace that aren't relevant to the bug. My code starts at the stackframe below FBConnection.java:324) I also attempted to set my max primary keys to load to 1499, to see if maybe there was an off by 1 error somewhere, but same result occurred. Setting my max number of keys within the IN clause to 1000 allowed the statement to continue. My connection itself is through an embedded connection, using the Jaybird 3.0.3 driver, but I don't think that's the issue. I also tried increasing the heap space of my JVM to 4GB (admittedly my knowledge of JNI behind the scenes is lacking, but without Googling, I'm assuming JNI calls still occur within the same heapspace/process), but that also appeared to have no effect Workaround is to limit keys within IN clauses to 1k or lower, for now. |
|
Commented by: @dyemanov You need to increase the stack size for your application. |
|
Commented by: @livius2 Nothing about issue itself, |
Modified by: Sean Leyne (seanleyne)status: Open [ 1 ] => Resolved [ 5 ] resolution: Won't Fix [ 2 ] |
Modified by: @pcisarstatus: Resolved [ 5 ] => Closed [ 6 ] |
|
Commented by: Karl Miller (searchengine27) Yes and thank you to both comments. I confirmed that increasing the JVM Thread's stack size resolved my issue and confirmed this was all on my end via being the JVM was misconfigured. And thanks for the suggestion for temp tables; I still don't fully understand how it would work with VIEWs in Firebird (without doing an explicit select still), but I'll look into it a bit more to see if I can't figure it out. |
Submitted by: Karl Miller (searchengine27)
I apologize for the raw code and stack trace; I thought this was JIRA and tried to use {code} annotations, but it stays unformatted, so I'm not sure how to make this look better.
I have a SQL query that looks something like:
PreparedStatement prepareStatement =
conn.prepareStatement("SELECT a.PRIMARY_KEY, "
+ "a.COLUMN_ONE_NAME, a.COLUMN_TWO_NAME, "
+ "a.COLUMN_THREE_NAME, a.COLUMN_FOUR_NAME, a.COLUMN_FIVE_NAME, "
+ "a.COLUMN_SIX_NAME from TABLE_NAME a "
+ "where a.PRIMARY_KEY IN ("
+ Joiner.on(", ")
.join(Iterables.limit(
Iterables.cycle("?"),
numberOfKeysToLoad))
+ ")");
Where the Guava portion is because I want to do a query of several thousand keys at a time, specifically for each PRIMARY_KEY.
I found out quick that Firebird's IN clause has a limit of 1500, so I started batching by increments of 1500.
However, when using the max (and 'near' to the max), I end up with a java.lang.Error that originates from the JNI:
java.lang.Error: Invalid memory access
at com.sun.jna.Native.invokeLong(Native Method)
at com.sun.jna.Function.invoke(Function.java:421)
at com.sun.jna.Function.invoke(Function.java:354)
at com.sun.jna.Library$Handler.invoke(Library.java:244)
at com.sun.proxy.$Proxy70.isc_dsql_prepare(Unknown Source)
at org.firebirdsql.gds.ng.jna.JnaStatement.prepare(JnaStatement.java:144)
at org.firebirdsql.jdbc.FBStatement.prepareFixedStatement(FBStatement.java:1368)
at org.firebirdsql.jdbc.AbstractPreparedStatement.prepareFixedStatement(AbstractPreparedStatement.java:1230)
at org.firebirdsql.jdbc.AbstractPreparedStatement.<init>(AbstractPreparedStatement.java:148)
at org.firebirdsql.jdbc.FBPreparedStatement.<init>(FBPreparedStatement.java:50)
at org.firebirdsql.jdbc.FBConnection.prepareStatement(FBConnection.java:1191)
at org.firebirdsql.jdbc.FBConnection.prepareStatement(FBConnection.java:961)
at org.firebirdsql.jdbc.FBConnection.prepareStatement(FBConnection.java:915)
at org.firebirdsql.jdbc.FBConnection.prepareStatement(FBConnection.java:324)
(FYI I removed the parts of the stack trace that aren't relevant to the bug. My code starts at the stackframe below FBConnection.java:324)
I also attempted to set my max primary keys to load to 1499, to see if maybe there was an off by 1 error somewhere, but same result occurred.
Setting my max number of keys within the IN clause to 1000 allowed the statement to continue.
My connection itself is through an embedded connection, using the Jaybird 3.0.3 driver, but I don't think that's the issue.
I also tried increasing the heap space of my JVM to 4GB (admittedly my knowledge of JNI behind the scenes is lacking, but without Googling, I'm assuming JNI calls still occur within the same heapspace/process), but that also appeared to have no effect
Workaround is to limit keys within IN clauses to 1k or lower, for now.
The text was updated successfully, but these errors were encountered: