This test part of the RealDB project. See site at http://www.gillius.org/realdb/

Based on revision 57 of CorruptionTest:

Ran the test a handful of times on the following equipment:
 * Generic "ALLIN1 2.0" multi card reader connected by USB 2.0
 * Ubuntu Linux hardy heron 8.04
 * Java 1.6.0_06 JRE to compile, GCJ 4.2.3 to compile generated JAR into native executable
 * SanDisk 128MB CompactFlash card, probably purchased circa mid 2004, copyright date on label is 2000
   * Formatted Fat16
 * BLock file with 1024 blocks of 4096 bytes each

Performed several tests locally to ensure that Sun JVM and GCJ exe worked on test files on local drive. All of the tests from this point are done with the GCJ-compiled versions. I read existing test file left over from July 10 test and it had a little over half of its blocks good, and the rest empty. This matches what I validated in Windows.

I cleared the file, unmounted it cleanly, and validated it to ensure that the file was written properly. It was. I noticed (and I expected this from past experience), that unlike Windows, Linux performs very heavy caching of IO to the flash card, so whenever I did a write or clear I saw it complete instantly. A validate would take the expected time (a few seconds) the first time I ran it on a newly mounted card, and then complete instantly after that. I also saw that if I mounted, did a clear or write, then validate, validate would return immediately. This matches my previous experience with Linux that when you write to the disk, a read on the same data will return cached memory, even if no physical IO operations have been performed.

I decided to try pulling out the drive during a write. On the first attempt, write completed immediately up to 100%, then I pulled the drive out. I must not have been fast enough, because the file came out with 1024 good blocks. On the following test, I cleared the file then unmounted cleanly, remounted to validate and saw that the file was clear. I then performed a write and pulled out the drive much faster (about 1-2 seconds after write). This time I saw corruption after inserting the card again:
validate
......25%.....50%.....75%.....100%
Blocks good=0, bad=0, clear=1024
Completed validate
write
......25%.....50%.....75%.....100%
Completed write
<I pull the card at this point>
exit
$ CorruptionTest /media/CANON_DC/CorruptionTestBlockFile 4096
Using file /media/CANON_DC/CorruptionTestBlockFile
  with blocksize = 4096 bytes
<snip instructions>
validate
......25%.....50%.....75%.....100%
Blocks good=699, bad=3, clear=322

This is the first time that I've seen actual corruption from pulling out the card. 3 of the 4096 byte blocks did not match their checksums (but were not clear).

Conclusion:
The assumption that I am trying for is to get to the point where I only lose one block (whatever I happen to be writing at the time of failure).

The results of the testing with Linux bother me, for a few reasons:

 1. There were multiple blocks corrupted, 12K worth. That means that either more than one block can be corrupted, or my assumption that the hardware block size is <= 4096 bytes is wrong, that a larger block size will achieve what I want.
 2. Linux caches the drive heavily. It tells the program everything is done and fine, but the card is still writing. This may still be possible to work around if I structure writes properly such that as long as the writes occur in order I can arrive to some proper database.

There are some tasks that I need to try:
 1. Print more details on which blocks were good/bad -- I would expect 699 good blocks followed by 3 bad followed by 322 clear, but this may not be the case and I should verify it.
 2. Try opening the file in "synchronous" mode to make sure there is a flush after every write, and see if Linux obeys it (i.e. writing takes non-"instant" time).
 3. Examine the bad blocks to see if there is any pattern to how they are bad.
 4. Try adjusting the block size to see if the amount of corruption on power loss has an upper-bound for this hardware