Interested in a web application instead?
See the Lacinato WebABX/Shootouter.
Lacinato ABX is the only cross-platform ABX and shootout blind testing software for audio. It allows double-blind ABX testing of audio files (seeing if you can tell which file is being played), as well as "shootouts" (picking your favorite from a shuffled, anonymized list). Or you can just use it as a convenient way to switch between files (maintaining the current play position, if you like.) It is free to download and use (see license.). Although it's Java-based, you do not need to install Java to use it. Runs on Linux/Windows/Mac.
Current version: 2.37 -- changelist is here
- ABX testing
- ABCD..X testing (more than two files)
- Shootouts, including fine-grained scoring of files for each shootout
- Jump from one file to the next while maintaining play position
- Automatically detects changes in audio files and prompts to refresh or ignore -- very handy for rapid cycling of versions that you want to re-test. (Unfortunately this doesn't work on Windows due to a Java bug.)
- Drag-and-drop audio files or groups of audio files or *.abx files into the program
- Per-file gain adjustment
- Per-file time offset (to line files up with each other)
- Configurable play range / looping
- Extensive keyboard shortcuts
- Save and recall sets of files/gains/offsets/loop points/etc. for recall or sharing with others
- Supports a fair variety of file types, including MP3, WAV, AAC, AIFF, and can even play audio streams from many video files.
- Calculates the p-value (which leads to the "confidence that results are better than chance")
- Will play high sample rates and bit-depths (limited by your audio system only) but make sure your operating system isn't downsampling. E.g. in Windows you must set your fancy sound interface as the default interface, and you need to choose the default sample rate / bit depth to use for the device. Lacinato ABX/Shootouter will not change the sample rate or bit depth as it plays: it is at the mercy of your operating system. You can set your OS to the highest rate you will be testing, but if you play files of a lower sampling rate, upsampling will occur.
Simply download one of the all-in-one packages below and it will run self-contained as a normal app (no installation or setup of anything is required -- super easy).
If you do have Java 8 installed (with the JavaFX component that is included in the Oracle version) you can just download the much smaller .jar file and run that ("java -jar ABX.jar").
Contact me if you have questions/corrections/suggestions.
All-in-one packages (57 to 71MB):
Or download just the .jar file (64k), but that requires Oracle's version of Java 8.
- 1 through 0: play file in slots 1 through 10; hold shift to choose the file
- a through j: play file in slots 1 through 10; hold shift to choose the file
- r: show/hide results
- x: Start ABX, or play "X", or go to next "X"
- s: start shootout, or go to next shootout
- k: toggle "keep pos."
- l: toggle loop
- n: go to next "X" or next shootout, if in either
- o: open fileset
- up/down arrow keys: (when gain/offset fields are focused) increase/decrease gain or time offset; hold ctrl for fine adjust
Generally shootouts are for choosing which file is your favorite. In that case, you can just click "choose" for each shootout, and ignore the scoring sliders. Sometimes you may want to score the files as you go: for each shootout, you can apply a score from 0 to 10, which you can also think of as awarding "points". When you view the results later, you can see what the average score was for each file, the total "points" allocated to each file, as well what percentage of the total allocated points each file received.
Important notes on ABX testing:
- Pick two or more files that vary only in the way you want to test yourself for (except for time offset and levels -- those can be adjusted.)
- Click "Start AB*X" or "Start Shootout" to start the tests, and click "choose" to make your choice.
- You can use the per-file "play" buttons while ABX testing, but to do a more proper test, that could be considered cheating. (You must use them for a Shootout, of course.)
- After 15 or so tests, click "Show/Hide Results" to see how you did.
- For ABX testing, the "Confidence" value is the percentage chance that your results are better than chance. E.g. if you got a 95% accuracy result but the Confidence value is 60%, that means that the accuracy result is not statistically significant. A common standard is to require 95% or better confidence in a result before considering it meaningful. For example, choosing 12 out of 16 correctly results in an accuracy of 75% with a a confidence of 96% -- this is a much more meaningful result, implying strongly that you can identify 75% correctly and that that result is not due to chance.
- Some recommend not to do more than 25 or so tests for a given pair of files, because listening fatigue will start to bias your results.
- Taking sets of tests over and over for the same pair of files until you get the result you are looking for renders your results meaningless (after all, even a 95% confidence result is wrong 5% of the time, so if you run sets of trials 20 times, guessing completely at random, one of them will give you that impressive-seeming result.) You should be able to reliably repeat a result for it to count. Pulling up a couple files and getting a 12 out of 15 doesn't mean much unless you can repeat it. Try doing multiple tests over several days
- On that theme: the internet is a big place, so if you post two totally identical files and 300 people download them and ABX test them in a trial of 20 comparisons, 15 of those people will likely get 15 or more correct with a 95%+ confidence result, purely by chance, and will proudly post their results. This obviously does not mean that a small group of people have golden ears that can hear the difference between two digitally-identical files, so beware the impressions you get from reading forum posts.
- Lacinato ABX uses an exact binomial distribution algorithm to calculate the p-value.
Lacinato ABX/Shootouter should "just work" on most linux distributions, but some releases may need some additional packages installed. You can see the official supported systems list, but any distribution with pulseaudio installed, and most without it installed, should work.
Lacinato ABX uses Oracle's Java 8, which requires libavcodec and libavformat -- more recent Linux distributions use more recent versions of those packages, which may cause MP3 files not to load. Test it first before bothering to install anything else -- it might work. If not, you do not need to uninstall the more recent versions: the older versions can co-exist, and shouldn't cause any problems. (Some distributions will, by default, lack libavformat and libavcodec entirely.)
How to install those packages will depend on your distribution, but for distros based on Ubuntu, entering something like the following into a terminal should work:
$ sudo apt-add-repository "deb http://us.archive.ubuntu.com/ubuntu/ precise-updates main"
$ sudo apt-get install libavutil51 libavcodec53 libavformat53
(then remove the added repository if desired)
The first line adds the Ubuntu 12.04 repository to your repository list. The second line installs the older versions of the packages. You might want to also remove the added repository from your repository list -- this can be done easily with whatever package manager you use ("software center", synaptic, etc.)
If you're not using an Ubuntu-based distribution, see if you can find a source of repositories for older versions of your distribution that used the correct package versions. Failing that, you can seek out .deb package files for them and install them (e.g. with "dpkg -i") but I have not had much luck with that. If you find a solution for a particular distribution, or can't figure one out, please get in touch.
If you don't have have pulseaudio installed -- i.e. you're just using ALSA -- then things should work fine if you are using the default sound interface. If you have configured a sound interface in .asoundrc, e.g. for an external sound interface, then you might run in to issues if you don't configure it to be able to handle multiple audio streams at the same time.
To test if your interface can handle multiple streams at the same time, try playing a video in YouTube while you play an audio file in a music player. If both streams play through the interface simultaneously, then Lacinato ABX/Shootouter should work fine.
For example, Debian Wheezy 7.7 uses ALSA. This ABX software creates a different audio stream for each media file you are comparing. If you configure an external interface and don't set up multi-stream mixing, you will only be able to have one audio file open in the ABX program at a time, making it useless. You could read up on dmix -- I believe that enabling it in .asoundrc will fix that problem. Let me know if it does. Installing pulseaudio should also solve the issue, but read up on the subject before jumping in.
2.37 fixes MP3 playback in some recent operating system versions
2.36 adds optional scoring of files during shootouts, auto-detection of changed audio files (except on Win due to Java bug), keyboard shortcuts, various UI improvements including protection against loss of test results
2.35 prevents freezes on Mac OS X (please report if you discover otherwise)
2.34 improves a lot of UI stuf
2.33 fixes some number parsing bugs
2.32 supports opening .abx files directly when associated in OS
2.31 adds drag/drop for *.abx files