Looking For Way To Speed Up mp3Tag's Disk Reading

I have a NTFS formatted external hybrid drive (SSD+hard disk), in a USB 3.1 enclosure, connected to a USB 3.1 port on my Intel i9 desktop. It takes 11 minutes for mp3Tag V3.16 x64 to read the mp3 directories into memory. Since the number of files will more than double this year, any ideas if there are settings or options in mp3Tag to speed up NTFS directory reads?

Switch to an internal SSD.
The connection via USB is the second slowest, I think. Slower is only the access to data via the network.
Sometimes it helps to switch on the library but don't expect any wonders. The bottleneck is the USB connection

He is absolutely right

You can read this Speeding up the loading process - its talks about slightly outdated hardware but the principals stand

But even if you would switch to M.2 NVMe instead of SSD [which even this previous disk technology is pricey when going above 2 TB], you can also alleviate your initial [i.e. after booting up of the operating system] loading time with solution from this topic

And if you like it but it is too complicated for you to implement, then you can vow your opinions it this topic

and / or you can take a look at the issue of re-opening of Mp3tag thus of re-loading of files

I don't believe the USB 3.1 transfer rate is the problem. After the initial load of the directories (and I don't add too many new files to too many directories), the load time is around 23 seconds from the SSD part of the hybrid drive when I do a mp3Tag F5 refresh.

I already backup all that data to a SSD at the end of the day that I have left over from an old PC. The big concern is the reliability of that SSD. Having worked for a hard drive and SSD manufacturer, SSD's have a limited lifespan in terms of number of TBs written (it's in the fine print of SSD warranties, if one bothers to read it). I'm already past that 10 TB write threshold limit for that MLC-type SSD from that particular manufacturer, so I can expect to start seeing failures on those NAND flash memory cells on the chip soon, which means lost of data for that file that was just written.

Based on the number of write TBs I will do in the near future, following the suggestion of using a SSD will require an 8 TBs MLC-type SSD (never an 8 TB TLC-type SSD), for that SSD to last a several of years before reaching the write TB limit on that drive (since the write limit is based on drive size and number of levels used in the NAND flash memory cells).

So I'm back to my original question. Is there something that mp3Tag can do to speed up how it reads NTFS directories? I've thought about larger cluster sizes for the hybrid drive, but this will affect how much space is available for the other data that is backed up. And benchmarks I've seen show loss of performance (and usable space) for non-mp3 files past 16K cluster sizes.

But actual read and write performance via USB is always slower no matter what drive is in use. How many files are we talking about?

If this is mainly the drive used, you can enable the library setting in the options menu and point it to this directory. It will take a similar amount of time to open the first time after enabled, but be significantly faster afterwards. Unless you tell mp3tag to refresh, which will again update the full library and take some time.

I think you can see the speed in the progress dialogue.
How many MBps do you get? Compare that to the 600 MBps of the SATA bus?
Divide that by the amount of data that you need to copy and that is the time that your system will need.
This has nothing to do with MP3tag, it's the way you have set up your environment.

[We are sidestepping here]

Here are my own [inconclusive] personal test results in regard to that issue:

[I since then till this day I keep my system and music on an M2 NVMe - but all of my other drives that are online are SSDs connected through USB. And I only connect physically via SATA my HDD archives when needed]

When I was buying over a year ago for mt online archival purposes first a 4TB SSD and then a 8TB SSD, I notices that as times goes and capacities provided by manufacturers for such drives increase- they apparently have to be upgraded from MLC to TLC and then QLC. If I am not mistaken, I could not find at that time a big SSD drive that did not utilize QLC NAND Flash Memory. And I heard they were to start utilizing next level

So is this true? Or will with big SSD drives will be available in the MLC standard? Or are we in a mater of fact forever doomed to having one a single drive more space but with less durability?

I understand your answer, but it doesn't apply in this case, because the drive has to be an external one for several reasons.

First, this is a also a backup drive, which means every month it gets swapped with another backup drive from a bank vault. So having any type of internal drive doesn't work in this environment.

Second, as I mentioned in previous entries, my internal 2TB SSDs (SATA and PCIe 4.0 NVMe) are reaching their write TB warranty limits way before the other part of the warranty of 3 years will be reached. So I need to do every write and update I can to my external hybrid drive, and then copy any new/changed files to one of the internal SSDs at the end of the day. That will save anywhere between 1 - 35 GBs of write activity per day the SSDs won't have to do.

Third, since I'm ripping or updating only one file at a time, the USB 3.1 read and write transfer rates are perfectly satisfactory for this type of operation.

Fourth, I have also been doing some research because of the write warranty issue. My next PCIe drive will probability be one of the 4x2TB MLC (8TB) drives that's out in the market. With 4 internal chip sets packaged together, they're wider than normal and come surrounded by a large heat sink type package to cool it. Unfortunately, they're starting to be close to the size of a GeForce RTX 3090.

So I'm back to my original question. Is there some optimization mp3Tag can do to speed up the reading of NTFS directories?

You have asked this question now 3 times and got more or less the same answer every time:
It's your system environment esp. in repsect to hardware. Change that or that's it.

Data transfer speeds are no MP3tag problem.

I can tell by your response that you still may not understand my question.

The first directory & mp3 tag info read from the external hybrid drive takes 11 minutes. The second read now takes 23 seconds because that info has been read into the hybrid's flash buffers. That's an overhead of ~ 96% due to how mp3tag is performing its reads. From my experience, that overhead is rotational misses, head seeks and the native data transfer rate off the physical medium.

I had the same issue when I did mainframe basic machine coding. But in basic machine coding, you had the option of using "command chaining" for reads and/or writes. Command chaining allow me to have each individual read request transfers 195 data clusters at a time, with no rotational misses or head seeks. That meant the read response time was only the native data transfer rate off the physical medium and the transfer rate across the port.

I haven't done much Intel basic machine coding so far, but Windows' Data Access does support the equivalent feature to command chaining, but it's called FILEAPI.H, where you can read as many clusters as you want, only limited by the number of buffers you allocate.

I have ~ 33 MB of mp3 directories and tag info on that drive. It's native read rate off the physical medium is 24 MB/sec. So the theoretical best case would be 23 + 2 seconds, instead of 11 minutes. Since I keep my directories de-fragged, that would give me somewhere between 1 - 2 minutes of initial startup time instead of the current 11 minutes with an expanded use of FILEAPI.H feature. Looking at mp3tag.exe machine code, it appears to be written (at least part of it) in Visual C++. I'm not sure if Visual C++ supports the FILEAPI.H function (since my 40+ years of programming has always been in basic machine code, since it gives the maximum microprocessor performance with the smallest program size). If it doesn't support it, then with a little research on how to add that machine code function to do the reading may be possible in a future version of this useful tool.

I doubt that any of this really has to do with MP3tag.
Or instead of theoretical assumptions - how long does it take to copy the very same folder to a folder on a built-in drive?
If you have very slow disc reading performance, then check this thread

I just went window-shopping in a Europe-oriented price engine

And I found that

  • of the 3 available SLC SSDs their size is only up to 1.6 TB for a price of "only" ~5250$ to ~10000$; and utilizing U.2/SFF-8639 connection
  • from scores of available MLC SSDs connected through SATA only 2 of them are as big as 4TB and have a price a price of ~830$ and ~1500$ attached to them
  • there are only 2 8TB MLC SSDs and there are connected through PCIe 3.0 x8 and U.2/SFF-8639, both at the cost of ~5100$

I would never spent so much money on a single drive. I myself draw the line at paying ~750$ for 8TB and only because government had given my business some COVID subvention and I had to start spending it

If I were in your position I would rather buy whatever-SSDs-8TB- that I could get my hands on for the lowest price and run them in a NAS / RAID or some other backup arrangement

I keep a backup copy of the mp3 files on my PCIe 4.0 NVMe SSD. Most of the time the mp3 files are sync'ed with the external drive at the end of the day, or maybe a day or two later. But I did run mp3tag again that SSD. I had my anti-virus software and Windows Defender disable for the test. It took just under 2 minutes (1:57) for mp3tag to read those mp3 directories and tag info.

5.5 times read speed is certainly an improvement, but not at the expense of replacing that PCIe SSD every 18 months. (But that read improvement was still a little disappointing, since the "command chaining" I did from the late 1990's with that level of technology of disks and transfer rates did beat the current SSD time by more than 15 times.) But if Flo isn't able to implement the FILEAPI.H level programming, then there isn't much one can do to improve mp3tag's "start up" read time using hybrid external drives.

As far as I understand WIndows the benefit of this OS is that an application developer does not have to invent the wheel over and over again and cater for all the interfaces that hardware may offer but only address certain routines provided by the OS.
This means that MP3tag only issues requests to the OS and the OS then provides the

and

Which means: you have to tune your OS as I bet that all other applications that have to transfer that much data will have to go through the same bottleneck and it is not an MP3tag problem but an environment problem. MP3tag is the messenger, not the cause.
Another note but I actually do not want to dig into interface speed tests here

looks to me like a USB 2.0 connection as I get some 80MBps via a USB 3.0 connection at both ends.

1 Like