Answered

Space issue with the PROOF Server

 We are getting failures in our execution sets - 


Rejected due to error. Error during uploading assignment object. Message: /home/internal/upload/proof/assignments/test_case_5116_8_FIRE_FOX_ANDROID_API_26_OREO_LG_NEXUS_5X_0_1637600405860 (No space left on device)

...OPS is working to resolve the issue, we have about1GB of space left on our AWS servewr...

Still 100% but more than  1 G free  space
root@era-iss-33-nprd database_backups]# df -h .

Filesystem      Size  Used Avail Use% Mounted on

/dev/nvme0n1p2  150G  149G  1.3G 100% /  

.. they are also deleting DB backups, and I have asked to increase the total storage to 200 GB.

Any input on how to resolve this would be appreciated, can we also get input as to what we are losing so much space?

 

 

0
6 comments
Avatar
Stephen Akinpelumia

Update - 

Available space now at under 14GB... after deleting backups except the latest one and changing backup delete schedule to the last 15 days...

[root@era-iss-33-nprd ~]# df -h /

Filesystem      Size  Used Avail Use% Mounted on

/dev/nvme0n1p2  150G  137G   14G  92% /

 

 

 

0
Comment actions Permalink
Avatar
Rex Feizi

I don't see anything wrong here. The amount of space used on your on-prem installation (AWS cloud) is directly related to usage. There should be processes in place to clear the server logs, execution artifacts (images, videos, and logs). Also, there should be cleanup cron jobs that clean directory with old data, as the assignments directory. Files that are older than 5 days must be deleted in this folder as well. 

 

Hope this helps

 

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Hi Rex, we are now getting this error...

 
Error during uploading assignment object. Message: /home/internal/upload/proof/assignments/test_group_852_CHROME_ANDROID_API_26_OREO_LG_NEXUS_5X_0_1638285302922 (No space left on device)

 

...and we have 59 executions currently stuck and showing this error.

 

Please let us know your availability today to help fix this issue.

 

0
Comment actions Permalink
Avatar
Rex Feizi

There is a space issue on the host machine. This is an operations issue, not sure how I can help. 

Either way, please let me know. 

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Ok, thanks Rex.

0
Comment actions Permalink
Avatar
Stephen Akinpelumia

Good morning Rex, so I found out that the cronjobs and processes that were is place for the previous on-prem linux server were decommissioned after the new AWS server was created; instead of configuring them to work with the new server, which explained why the server is losing space currently. And no one knw how to set up these processes and cronjobs the same way as they were done previously.

Please help provide information on how to have OPS create these processes and cronjobs, as well as file locations and frequency. i would appreciate it. thanks.

0
Comment actions Permalink

Please sign in to leave a comment.