User bio
404 bio not found
Member since Dec 3, 2018
Replies:

Kevin, all the benchmarks have been run by me at one of my workstations (Intel Core i7-4790).

The same Seagate ST1000DM010-2EP102 7200 rpm HDD was dedicated to host the data files over all experiments, with no other read or write operations interrupting it during each session.

So, summing it up, there has been no relation to ISC infrastructure.

I can share the detailed hardware specs if you need them. Or you can try and run the benchmark on your own hardware.

Joel, the code just reads one line at a time, immediately storing its parsed value into the database (within the same execution thread).

Strictly speaking, this is a mix of a CPU-bound and I/O-bound tasks within the same thread, and we should have 3 threads instead:

  • one reading from the local disk (I/O);
  • the 2nd one parsing the CSV line, and
  • the last one storing the parsed value into the (potentially, remote) database.

The 3-thread approach should improve the timings (for all the databases being tested), but the relative values are unlikely to reveal any substantial changes.

Additionally, I have to remark that the auto-commit flag is off over each session, with a single commit issued after all the data are saved. In a real-life application, one should additionally issue intermediate commits (say, after each 1000 or 10000 of records).

Certifications & Credly badges:
Andrey has no Certifications & Credly badges yet.
Global Masters badges:
Andrey has no Global Masters badges yet.
Followers:
Andrey has no followers yet.
Following:
Andrey has not followed anybody yet.