Completed

Multiple IO Errors from existing SQL scripts after upgrade to version 9.5.2.2224

We are getting multiple IO errors in PROOF for existing scripts and queries after upgrading to version 9.5.2.2224 yesterday against our OLTP Databases for our DB Proxy accounts, these issues were not occurring when the same scripts are ran manually and OPS's DBA staff is not seeing any connection issues with the Proxy accounts; this is only happening in PROOF on Local Run and is intermittent.

This is a blocker for our execution sets, we have majority of our test cases incorporated with SQL queries to retrieve data from our OLTP databases; these IO errors are failing our test cases; please help priorities this issue.

Here are the two IO Errors - 

1) IO Error: Got minus one from a read call 9.5.2.2224
 
Please see screenshot below, this is from a simple DB query, again this was not an issue prior to upgrade to 9.5.2.2224; when the step is re-ran the error goes away.

2) IO Error: An existing connection was forcibly closed by the remote host, Authentication lapse 0 ms.

Please see screenshot below, this is from a simple DB query, again this was not an issue prior to upgrade to 9.5.2.2224; when the step is re-ran the error goes away.

0
13 comments
Avatar
Rex Feizi

Stephen, 

We recently upgraded the Oracle JDBC jar file to 19C. What is the version of Oracle that you are running?

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

GM Rex, OPS confirms our current version as 19.13.0.0.0; please see below

08:08:43 sershthn_db@oltpdev(oltpdev)> desc v$instance
Name Null? Type
----------------------------------------------------- -------- --------------------------------
INSTANCE_NUMBER NUMBER
INSTANCE_NAME VARCHAR2(16)
HOST_NAME VARCHAR2(64)
VERSION VARCHAR2(17)
VERSION_LEGACY VARCHAR2(17)
VERSION_FULL VARCHAR2(17)
STARTUP_TIME DATE
STATUS VARCHAR2(12)
PARALLEL VARCHAR2(3)
THREAD# NUMBER
ARCHIVER VARCHAR2(7)
LOG_SWITCH_WAIT VARCHAR2(15)
LOGINS VARCHAR2(10)
SHUTDOWN_PENDING VARCHAR2(3)
DATABASE_STATUS VARCHAR2(17)
INSTANCE_ROLE VARCHAR2(18)
ACTIVE_STATE VARCHAR2(9)
BLOCKED VARCHAR2(3)
CON_ID NUMBER
INSTANCE_MODE VARCHAR2(11)
EDITION VARCHAR2(7)
FAMILY VARCHAR2(80)
DATABASE_TYPE VARCHAR2(15)

 

08:09:12 sershthn_db@oltpdev(oltpdev)> select instance_name, version_full from v$instance;

 

INSTANCE_NAME VERSION_FULL
---------------- -----------------
OLTPDEV 19.13.0.0.0

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Done

 

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Hello, we have not been able to reproduce the issue any more, please proceed to close this ticket, thanks.

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Hi Joseph, unfortunately the 'Multiple IO Errors from existing SQL scripts after upgrade to version' issue is occurring again, we are on 9.5.5.2337.

On  recently upgraded the Oracle JDBC jar file to 19C,is it possible that updates were made in relation to the idle time for Oracle connections? As you can see in the screenshot below as provided by our OPS dept, our Proxy accounts have unlimited connections and idle time; and we never had these connection issues with our Proxy accounts until we upgraded to version 9.5.2.2224 this March.

 

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Hi Joseph, is there any update on this issue?

0
Comment actions Permalink
Avatar
Rex Feizi

Stephen, 

There have not been any changes on our side since the last release. Even last time, we didn't do anything and the issue went away. 

Could you please check with the Oracle admin to see what has been changed in the last 2 months or in between? 

--Rex

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Hi Rex, nothing has changed on our end; I would recommend we go over this when we do our upgrade; but OPS has confirmed that is is not a eRA issue. Thanks.

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Good morning, I would like to request a gotmeeting session with either Rex or a developer to triage and demo this issue; please let me know what availability this week is available. Thanks.

0
Comment actions Permalink
Avatar
Rex Feizi

Updates: 

We had a call with Stephen. And here at the steps, we took for troubleshooting: 

  1. On Stephen's personal machine, with the local run, we see the above errors irregularly. 
  2. Stephen mentioned that cloud executions don't have such issues. This prompts us to log in to one of their cloud machines (executors) and try local run from that machine. 
  3. On this other machine, we were not able to reproduce the issue. This concluded that the issue is not with the Local run. 
  4. We found this link from Microsoft that we think might be relevant to this issue. 
  5. Stephen is reaching out to their IT to see if they can find the root cause for this issue. 

 

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Update - We worked with our OPS Network group and they could not figure out the reported issue.

I went back to a previous workaround provided by Joseph as shown below....

Something that might help uncover issues, can you please download the attached file, rename it to ojdbc8-19.13.0.0.1.jar and place it in the local player installation directory in lib folder? After that, restart local player and execute tests again to see what happens. If issue occurs, please provide logs also.

...I had done that previously and it resolved the issue, but the ojdbc8-19.13.0.0.1.jar file got overidden during the next PROOF Upgrade when I updated the Subject7 Local Player. I tried the same resolution again and it worked!!

I do not want to have to manually update/replace the ojdbc8-19.13.0.0.1.jar for 60+ users every time there is an upgrade, is there some way to fix this jar to work properly for future upgrades? 

I can send the old jar file if that help troubleshoot the issue, thanks.

 

 

0
Comment actions Permalink
Avatar
Joseph Hamdan

Marking this request as Completed.

Custom drivers can be added into custom_lib folder once local player is upgraded to the latest version.

Regards,
Subject7 Team

0
Comment actions Permalink

Please sign in to leave a comment.