View Issue Details

IDProjectCategoryView StatusLast Update
0000146Adam4EVEBugpublic2016-10-05 11:33
ReporterSuccessor Assigned ToSuccessor  
PriorityhighSeveritymajorReproducibilityhave not tried
Status resolvedResolutionfixed 
Product Version0.0.2 
Target Version0.0.3Fixed in Version0.0.3 
Summary0000146: Rework import backend to remove double data and improve performance
DescriptionCurrently the are two issues with the CREST import aswell as imports from other sources:

1) Data is being imported twice, once for DEV and another for PRD. This is unneccessary strain on the sources and the local server

2) The server has full CPU utilization during major (all region) imports. Find a solution for this in order that the frontend is not affected during an import.
TagsNo tags attached.

Activities

Successor

2016-09-18 08:22

administrator   ~0000079

Imported data and their calculated values are now stored in a new DB, that is shared by DEV and PRD.

Furthermore the import scripts will be rewritten to capture and store statistics about their runtime.

In a first step this enabled to identify 8000 SELECTs made in the analysis of the imported CREST orders. By making a bulk SELECT, the runtime was reduced to 1/10th of the previous time.

Further import scripts to be included.

Successor

2016-09-27 07:49

administrator   ~0000085

Rewritten zKillboard import and decom of dedicated Kills table, since shared table is used.

Successor

2016-10-05 11:33

administrator   ~0000089

Scripts optimized and data consolidated into shared database

Issue History

Date Modified Username Field Change
2016-09-18 08:20 Successor New Issue
2016-09-18 08:20 Successor Status new => assigned
2016-09-18 08:20 Successor Assigned To => Successor
2016-09-18 08:20 Successor Status assigned => confirmed
2016-09-18 08:22 Successor Note Added: 0000079
2016-09-27 07:49 Successor Note Added: 0000085
2016-10-05 11:33 Successor Note Added: 0000089
2016-10-05 11:33 Successor Status confirmed => resolved
2016-10-05 11:33 Successor Fixed in Version => 0.0.3
2016-10-05 11:33 Successor Resolution open => fixed