This test part of the RealDB project. See site at http://www.gillius.org/realdb/ Environment: * Generic "ALLIN1 2.0" multi card reader connected by USB 2.0 * Windows XP SP2 * SanDisk 128MB CompactFlash card, probably purchased circa mid 2004, copyright date on label is 2000 * Formatted Fat16 * Block file with 16384 blocks of 256 bytes each, mode "rwd" * Revision 67,68 of CorruptionTest ---------------- I only ran 1 test with the 1 MB block size, so I decided to try a few more times, in case I just got super lucky. I did clear, validate, then write. Failed during block 2. Then validate: validate ...........20%..........40%..........60%..........80%..........100% Blocks good=2, bad_data=1, bad_block_num=0, clear=5 New timestamp Thu Jul 24 21:28:54 EDT 2008 found at block 0 New timestamp Thu Jul 24 21:28:40 EDT 2008 found at block 3 Block 2 failed CRC check. Completed validate in 2.667081659 seconds This is exactly what I wanted to see, because it makes sense. Block 2 is partially written, because its timestamp is the same as block 0 and 1. Blocks 3 and onwards are clear. I updated the code (revision 68) to print out non-zero byte counts of 4096 byte segments as an indication of how far we got in writing: Block 2, segment 0 has 4074 non-zero bytes. Block 2, segment 1 has 4082 non-zero bytes. Block 2, segment 2 has 4080 non-zero bytes. Block 2, segment 3 has 4082 non-zero bytes. Block 2, segment 4 has 4072 non-zero bytes. Block 2, segment 5 has 4073 non-zero bytes. Block 2, segment 6 has 4078 non-zero bytes. Block 2, segment 7 has 4089 non-zero bytes. Block 2, segment 8 has 4081 non-zero bytes. Block 2, segment 9 has 4082 non-zero bytes. Block 2, segment 10 has 4083 non-zero bytes. Block 2, segment 11 has 4076 non-zero bytes. Block 2, segment 12 has 4084 non-zero bytes. Block 2, segment 13 has 4079 non-zero bytes. Block 2, segment 14 has 4072 non-zero bytes. Block 2, segment 15 has 4078 non-zero bytes. Block 2, segment 16 has 4081 non-zero bytes. Block 2, segment 17 has 4082 non-zero bytes. Block 2, segment 18 has 4082 non-zero bytes. Block 2, segment 19 has 4082 non-zero bytes. Block 2, segment 20 has 4080 non-zero bytes. Block 2, segment 21 has 4078 non-zero bytes. Block 2, segment 22 has 4077 non-zero bytes. Block 2, segment 23 has 4083 non-zero bytes. Block 2, segment 24 has 4076 non-zero bytes. Block 2, segment 25 has 4086 non-zero bytes. Block 2, segment 26 has 4084 non-zero bytes. Block 2, segment 27 has 4080 non-zero bytes. Block 2, segment 28 has 4079 non-zero bytes. Block 2, segment 29 has 4070 non-zero bytes. Block 2, segment 30 has 4084 non-zero bytes. Block 2, segment 31 has 4084 non-zero bytes. Block 2, segment 32 has 4069 non-zero bytes. Block 2, segment 33 has 4081 non-zero bytes. Block 2, segment 34 has 4084 non-zero bytes. Block 2, segment 35 has 4080 non-zero bytes. Block 2, segment 36 has 4082 non-zero bytes. Block 2, segment 37 has 4081 non-zero bytes. Block 2, segment 38 has 4078 non-zero bytes. Block 2, segment 39 has 4085 non-zero bytes. Block 2, segment 40 has 4082 non-zero bytes. Block 2, segment 41 has 4081 non-zero bytes. Block 2, segment 42 has 4087 non-zero bytes. Block 2, segment 43 has 4072 non-zero bytes. Block 2, segment 44 has 4080 non-zero bytes. Block 2, segment 45 has 4078 non-zero bytes. Block 2, segment 46 has 4081 non-zero bytes. Block 2, segment 47 has 4081 non-zero bytes. Block 2, segment 48 has 4075 non-zero bytes. Block 2, segment 49 has 4080 non-zero bytes. Block 2, segment 50 has 4080 non-zero bytes. Block 2, segment 51 has 4081 non-zero bytes. Block 2, segment 52 has 4076 non-zero bytes. Block 2, segment 53 has 4079 non-zero bytes. Block 2, segment 54 has 4076 non-zero bytes. Block 2, segment 55 has 4075 non-zero bytes. Block 2, segment 56 has 4078 non-zero bytes. Block 2, segment 57 has 4082 non-zero bytes. Block 2, segment 58 has 4082 non-zero bytes. Block 2, segment 59 has 4075 non-zero bytes. Block 2, segment 60 has 4082 non-zero bytes. Block 2, segment 61 has 4072 non-zero bytes. Block 2, segment 62 has 4081 non-zero bytes. Block 2, segment 63 has 4086 non-zero bytes. Block 2, segment 64 has 4075 non-zero bytes. Block 2, segment 65 has 4072 non-zero bytes. Block 2, segment 66 has 4078 non-zero bytes. Block 2, segment 67 has 4073 non-zero bytes. Block 2, segment 68 has 4078 non-zero bytes. Block 2, segment 69 has 4079 non-zero bytes. Block 2, segment 70 has 4081 non-zero bytes. Block 2, segment 71 has 4081 non-zero bytes. Block 2, segment 72 has 4085 non-zero bytes. Block 2, segment 73 has 4081 non-zero bytes. Block 2, segment 74 has 4088 non-zero bytes. Block 2, segment 75 has 4076 non-zero bytes. Block 2, segment 76 has 4083 non-zero bytes. Block 2, segment 77 has 4077 non-zero bytes. Block 2, segment 78 has 4081 non-zero bytes. Block 2, segment 79 has 4083 non-zero bytes. Block 2, segment 80 has 4079 non-zero bytes. Block 2, segment 81 has 4087 non-zero bytes. Block 2, segment 82 has 4081 non-zero bytes. Block 2, segment 83 has 4074 non-zero bytes. Block 2, segment 84 has 4082 non-zero bytes. Block 2, segment 85 has 4079 non-zero bytes. Block 2, segment 86 has 4079 non-zero bytes. Block 2, segment 87 has 4083 non-zero bytes. Block 2, segment 88 has 4075 non-zero bytes. Block 2, segment 89 has 4089 non-zero bytes. Block 2, segment 90 has 4084 non-zero bytes. Block 2, segment 91 has 4084 non-zero bytes. Block 2, segment 92 has 4075 non-zero bytes. Block 2, segment 93 has 4074 non-zero bytes. Block 2, segment 94 has 4081 non-zero bytes. Block 2, segment 95 has 4072 non-zero bytes. Block 2, segment 96 has 4081 non-zero bytes. Block 2, segment 97 has 4088 non-zero bytes. Block 2, segment 98 has 4076 non-zero bytes. Block 2, segment 99 has 4079 non-zero bytes. Block 2, segment 100 has 4079 non-zero bytes. Block 2, segment 101 has 4070 non-zero bytes. Block 2, segment 102 has 4080 non-zero bytes. Block 2, segment 103 has 4083 non-zero bytes. Block 2, segment 104 has 4077 non-zero bytes. Block 2, segment 105 has 4079 non-zero bytes. Block 2, segment 106 has 4080 non-zero bytes. Block 2, segment 107 has 4082 non-zero bytes. Block 2, segment 108 has 4082 non-zero bytes. Block 2, segment 109 has 4076 non-zero bytes. Block 2, segment 110 has 4080 non-zero bytes. Block 2, segment 111 has 4076 non-zero bytes. Block 2, segment 112 has 4073 non-zero bytes. Block 2, segment 113 has 4086 non-zero bytes. Block 2, segment 114 has 4085 non-zero bytes. Block 2, segment 115 has 4083 non-zero bytes. Block 2, segment 116 has 4077 non-zero bytes. Block 2, segment 117 has 4080 non-zero bytes. Block 2, segment 118 has 4084 non-zero bytes. Block 2, segment 119 has 4081 non-zero bytes. Block 2, segment 120 has 4079 non-zero bytes. Block 2, segment 121 has 4080 non-zero bytes. Block 2, segment 122 has 4083 non-zero bytes. Block 2, segment 123 has 4079 non-zero bytes. Block 2, segment 124 has 4078 non-zero bytes. Block 2, segment 125 has 4087 non-zero bytes. Block 2, segment 126 has 4075 non-zero bytes. Block 2, segment 127 has 4076 non-zero bytes. Block 2, segment 128 has 4077 non-zero bytes. Block 2, segment 129 has 4075 non-zero bytes. Block 2, segment 130 has 4074 non-zero bytes. Block 2, segment 131 has 4075 non-zero bytes. Block 2, segment 132 has 4078 non-zero bytes. Block 2, segment 133 has 4080 non-zero bytes. Block 2, segment 134 has 4085 non-zero bytes. Block 2, segment 135 has 4082 non-zero bytes. Block 2, segment 136 has 4083 non-zero bytes. Block 2, segment 137 has 4080 non-zero bytes. Block 2, segment 138 has 4083 non-zero bytes. Block 2, segment 139 has 4081 non-zero bytes. Block 2, segment 140 has 4083 non-zero bytes. Block 2, segment 141 has 4072 non-zero bytes. Block 2, segment 142 has 4081 non-zero bytes. Block 2, segment 143 has 4091 non-zero bytes. Block 2, segment 144 has 4084 non-zero bytes. Block 2, segment 145 has 4083 non-zero bytes. Block 2, segment 146 has 4084 non-zero bytes. Block 2, segment 147 has 4081 non-zero bytes. Block 2, segment 148 has 4084 non-zero bytes. Block 2, segment 149 has 4079 non-zero bytes. Block 2, segment 150 has 4081 non-zero bytes. Block 2, segment 151 has 4077 non-zero bytes. Block 2, segment 152 has 4080 non-zero bytes. Block 2, segment 153 has 4079 non-zero bytes. Block 2, segment 154 has 4078 non-zero bytes. Block 2, segment 155 has 4078 non-zero bytes. Block 2, segment 156 has 4086 non-zero bytes. Block 2, segment 157 has 4074 non-zero bytes. Block 2, segment 158 has 4086 non-zero bytes. Block 2, segment 159 has 4085 non-zero bytes. Block 2, segment 160 has 4074 non-zero bytes. Block 2, segment 161 has 4083 non-zero bytes. Block 2, segment 162 has 4073 non-zero bytes. Block 2, segment 163 has 4078 non-zero bytes. Block 2, segment 164 has 4081 non-zero bytes. Block 2, segment 165 has 4080 non-zero bytes. Block 2, segment 166 has 4080 non-zero bytes. Block 2, segment 167 has 4076 non-zero bytes. Block 2, segment 168 has 4086 non-zero bytes. Block 2, segment 169 has 4078 non-zero bytes. Block 2, segment 170 has 4084 non-zero bytes. Block 2, segment 171 has 4081 non-zero bytes. Block 2, segment 172 has 4076 non-zero bytes. Block 2, segment 173 has 4086 non-zero bytes. Block 2, segment 174 has 4077 non-zero bytes. Block 2, segment 175 has 4077 non-zero bytes. Block 2, segment 176 has 4081 non-zero bytes. Block 2, segment 177 has 4083 non-zero bytes. Block 2, segment 178 has 4075 non-zero bytes. Block 2, segment 179 has 4083 non-zero bytes. Block 2, segment 180 has 4070 non-zero bytes. Block 2, segment 181 has 4086 non-zero bytes. Block 2, segment 182 has 4085 non-zero bytes. Block 2, segment 183 has 4079 non-zero bytes. Block 2, segment 184 has 4079 non-zero bytes. Block 2, segment 185 has 4077 non-zero bytes. Block 2, segment 186 has 4078 non-zero bytes. Block 2, segment 187 has 4078 non-zero bytes. Block 2, segment 188 has 4074 non-zero bytes. Block 2, segment 189 has 4084 non-zero bytes. Block 2, segment 190 has 4074 non-zero bytes. Block 2, segment 191 has 4082 non-zero bytes. Block 2, segment 192 has 0 non-zero bytes. Block 2, segment 193 has 0 non-zero bytes. Block 2, segment 194 has 0 non-zero bytes. Block 2, segment 195 has 0 non-zero bytes. Block 2, segment 196 has 0 non-zero bytes. Block 2, segment 197 has 0 non-zero bytes. Block 2, segment 198 has 0 non-zero bytes. Block 2, segment 199 has 0 non-zero bytes. Block 2, segment 200 has 0 non-zero bytes. Block 2, segment 201 has 0 non-zero bytes. Block 2, segment 202 has 0 non-zero bytes. Block 2, segment 203 has 0 non-zero bytes. Block 2, segment 204 has 0 non-zero bytes. Block 2, segment 205 has 0 non-zero bytes. Block 2, segment 206 has 0 non-zero bytes. Block 2, segment 207 has 0 non-zero bytes. Block 2, segment 208 has 0 non-zero bytes. Block 2, segment 209 has 0 non-zero bytes. Block 2, segment 210 has 0 non-zero bytes. Block 2, segment 211 has 0 non-zero bytes. Block 2, segment 212 has 0 non-zero bytes. Block 2, segment 213 has 0 non-zero bytes. Block 2, segment 214 has 0 non-zero bytes. Block 2, segment 215 has 0 non-zero bytes. Block 2, segment 216 has 0 non-zero bytes. Block 2, segment 217 has 0 non-zero bytes. Block 2, segment 218 has 0 non-zero bytes. Block 2, segment 219 has 0 non-zero bytes. Block 2, segment 220 has 0 non-zero bytes. Block 2, segment 221 has 0 non-zero bytes. Block 2, segment 222 has 0 non-zero bytes. Block 2, segment 223 has 0 non-zero bytes. Block 2, segment 224 has 0 non-zero bytes. Block 2, segment 225 has 0 non-zero bytes. Block 2, segment 226 has 0 non-zero bytes. Block 2, segment 227 has 0 non-zero bytes. Block 2, segment 228 has 0 non-zero bytes. Block 2, segment 229 has 0 non-zero bytes. Block 2, segment 230 has 0 non-zero bytes. Block 2, segment 231 has 0 non-zero bytes. Block 2, segment 232 has 0 non-zero bytes. Block 2, segment 233 has 0 non-zero bytes. Block 2, segment 234 has 0 non-zero bytes. Block 2, segment 235 has 0 non-zero bytes. Block 2, segment 236 has 0 non-zero bytes. Block 2, segment 237 has 0 non-zero bytes. Block 2, segment 238 has 0 non-zero bytes. Block 2, segment 239 has 0 non-zero bytes. Block 2, segment 240 has 0 non-zero bytes. Block 2, segment 241 has 0 non-zero bytes. Block 2, segment 242 has 0 non-zero bytes. Block 2, segment 243 has 0 non-zero bytes. Block 2, segment 244 has 0 non-zero bytes. Block 2, segment 245 has 0 non-zero bytes. Block 2, segment 246 has 0 non-zero bytes. Block 2, segment 247 has 0 non-zero bytes. Block 2, segment 248 has 0 non-zero bytes. Block 2, segment 249 has 0 non-zero bytes. Block 2, segment 250 has 0 non-zero bytes. Block 2, segment 251 has 0 non-zero bytes. Block 2, segment 252 has 0 non-zero bytes. Block 2, segment 253 has 0 non-zero bytes. Block 2, segment 254 has 0 non-zero bytes. Block 2, segment 255 has 0 non-zero bytes. It appears that we went to segment 191. Based on the distribution of bytes, the entire segment 191 was likely written. That means that the upper bound for corruption (which I hypothesize is the flash block size) divides 4096 evenly, or is a multiple of 4096. All I need to make RealDB feasible is to prove that there is hardware for which the amount of corruption from power loss has an upper bound for some size block, and that I can confirm that data was committed entirely after each block (synchronous writes). ---------------- Trying test again, clear then write, failed writing block 4 when I pulled it out. This time I got 5 good blocks; no corruption. Again: failed on block 5. Got 5 good blocks, and block 5 (counting from 0) failed block check. Block 5, segment 62 has 4078 non-zero bytes. Block 5, segment 63 has 4080 non-zero bytes. Block 5, segment 64 has 0 non-zero bytes. Block 5, segment 65 has 0 non-zero bytes. So far the evidence still supports my requirement (I haven't lost more than 1 block). ---------------- I will try a test with 512 byte blocks. If the upper bound on corruption size is 4096, it should be possible to see multiple corrupted blocks (maybe, depending on how the sync writes go, if the data is either all the old or all the new data, I may not see any corruption). allocate 8192 Completed allocate 8192 in 0.02808806 seconds clear Creating file with timestamp of Thu Jul 24 22:15:18 EDT 2008 ...........20%..........40%..........60%..........80%..........100% Completed clear in 38.864925983 seconds write Creating file with timestamp of Thu Jul 24 22:16:11 EDT 2008 ...........20%Write failed while writing block 1803 Write time is 32.86 seconds for 4MB. Got 1804 good blocks Trying again: rewrite file, failed at block 1184. Got 1185 blocks. It is possible, especially given how much longer the write takes at 512 byte chunks, that the synchronous writing method actually spends almost all of its time idle, which means this could be an exercise in futility. ----------------- 16K chunks, to try to guess that the block size is 4096, and to try to see a half-written block. Write time is 2.43 seconds for 4MB. Error writing block 178: 179 good blocks 123 good blocks 239 good 113 good 51 good 173 good 44 good I give up trying to cause failure. If it spends most time idle, then actual corruption events will be low, but it still seems that corruption should be bound. ----------------- 1MB chunks again, but increase segment size to 131072 bytes: 2.21 seconds to clear 4MB. Error writing block 1 Block 1 failed CRC check. Block 1, segment 0 has 130533 non-zero bytes. Block 1, segment 1 has 130536 non-zero bytes. Block 1, segment 2 has 0 non-zero bytes. Block 1, segment 3 has 0 non-zero bytes. Block 1, segment 4 has 0 non-zero bytes. Block 1, segment 5 has 0 non-zero bytes. Block 1, segment 6 has 0 non-zero bytes. Block 1, segment 7 has 0 non-zero bytes. The corruption is on a 128K boundary. This corresponds also to the segment 191 and 63 above, as they are on 32 (4k) segment boundaries, which means they are also 128k segment boundaries. It could be that the atomic block size is 128k. Try again: all blocks succeed Again: succeed Block 3 failed CRC check. Block 3, segment 0 has 130602 non-zero bytes. Block 3, segment 1 has 130528 non-zero bytes. Block 3, segment 2 has 130558 non-zero bytes. Block 3, segment 3 has 130552 non-zero bytes. Block 3, segment 4 has 130553 non-zero bytes. Block 3, segment 5 has 130549 non-zero bytes. Block 3, segment 6 has 0 non-zero bytes. Block 3, segment 7 has 0 non-zero bytes. Again, on a 128k boundary.