Custom Format Problem

[flyman02]

Good morning,
I wanted to congratulate the great work done to develop the project.
I am trying to use this project on Arduino Uno. In practice I need to read and write a floppy custom format, which uses the same rules as Fat12 but uses 10 SECTOR X Tracks and 1024 Bytes X Sector. The problem is that I modify FF_MAX_SS = 1024 in the FFconf.h (Fatfs) form, it gives me a compilation error because it uses too much memory for variables, meter at 512 does not give error.
Is it possible to streamline variables in order to recover at least 600 memory bytes? Or is it possible to force the size of 1024 bytes x senctor?
The format I am trying to read is of an Gem S2 musical keyboard and the disc is 1.63MB 3.5 "HD
Thank you
FLY

[dhansel]

FatFS allocates two blocks of memory of size FF_MAX_SS for each open file - one that holds information about where on disk the file's sectors are located and one to hold actual sector data of the file. Since the Uno only has 2048 bytes of RAM, holding two buffers of size 1024 (plus even a minimum amount of other variables) is not possible.

However, FatFS has an option FF_FS_TINY (in ffconf.h). If that is set to "1" then FatFS only allocates one block of memory of size FF_MAX_SS - which will make it fit into Uno RAM constraints. The downside is that now FatFS has to return the head to track 0 after reading each sector to read the information (again) on where to find the next sector. Very inefficient when reading larger files. A better way to handle this would probably be to use an Arduino Mega which has 6k of RAM.

Also note that the sector size of 512 bytes is (somewhat) hard-coded in ArduinoFDC.cpp, mostly in functions readSector and writeSector. Search for "513", "514" and "515" and replace with 1025, 1026 and 1027 respectively. The 512 sector size is also hard-coded in format_track and a bit harder to change there. But hopefully you won't have to format disks in that format.

Finally, you should probably set both FF_MAX_SS and FF_MIN_SS to 1024 since if the two are not the same then FatFS expects function disk_ioctl (in file diskio.cpp) to figure out the actual sector size for the currently inserted disk.

Note that I haven't tried this here so I don't know if there are other issues with changing the sector size.

[flyman02]

I tried to modify Re FF_FS_TINY and not give more error of lack of RAM, but also by changing the variables "513", "514" and "515", always formatted at 512byte per Sector. I tried to search in the various files includes all the 512 items and replace them with 1024, but always form at 512byte. Perhaps it is a limit of the floppy unit.

[dhansel]

As I mentioned in my previous message, the changes I suggested are only for reading and writing. Changing the
format_track function is harder because it will involve a number of changes to set the proper size in the id record,
write the proper number of data bytes and the corresponding data checksum. All of which is currently hard-coded.

[GSoftwareDevelopment]

I wanted to join the topic :) and expand on the issue of formatting with different sector sizes.

You write that it is more complicated because some of the information is hardwired in, i.e. sector size. This is pre-calculated before starting formatting:

      *ptr++ = 0xFE; // ID mark
      *ptr++ = track; // cylinder number
      *ptr++ = side; // side number
      *ptr++ = sectorOrder[i]; // sector number
      *ptr++ = 2; // sector length <--- this line
      uint16_t crc = calc_crc(ptr-5, 5);
      *ptr++ = crc / 256; // CRC
      *ptr++ = crc & 255; // CRC
      *ptr++ = 0x4E; // first byte of post-data gap

The checksum for the header is also calculated here.

And in the single track formatting routine (assembler code).

          ldi r26, 0" // write 2*256+0 = 512 bytes
          ldi r27, 2 `n

Here is my question. How to retrieve the sector size in assembler code from this array? Sorry, but I don't know how to in the ATMEL assembler.
Or more simply:
Is it enough just to change in the assembler lines the values passed to the registers (r26, r27)?

I also have technical questions :)
Can the sector size be completely arbitrary?
Does it convert linearly to the number of sectors on the path? e.g., 256 bytes is 36 sectors; 128 bytes is 72?
How can you detect the number of tracks on a floppy disk?

Sorry for the number of questions, I want to understand :)

[dhansel]

It's been a while since I looked at this code or thought about floppy disk formats so take this with a grain of salt.

It should be much easier to pass the byte count high byte (2 for 512 bytes or 1 for 256 bytes) into r27 than trying to read it from the pre-computed buffer. You can't pass anything into r26 because that register is clobbered (changed) in the code before writing the data bytes. But you don't need it since the sector size should be a multiple of 256 anyways. Look at line 1080 to see how to pass data into the assembly code. You should be able to directly say "r27" to tell the compiler to use that specific register instead of just saying "r" and letting the compiler pick one.

The sector size could be made arbitrary. I think it already is for the read/write sector functions. For the format_track function you'd just have to use another register input to the assembly code to pass in the low byte count. Of course that then doesn't match the IBM format since the sector header only stores the high byte (1, 2 or 4) for the sector size.

The format routine doesn't do any calculations to figure out how many sectors will fit. It just writes as many sectors as you told it to. If it is too many then the last sectors will overwrite the first ones. The calculation is not exactly linear because there are gaps between the ID records and data record. So the gap lengths play a role in the computation of how many sectors you can fit. It is advisable to leave a longer gap at the end of the track to account for disk drives spinning at slightly different rates. If the drive is a bit slow and you don't have enough of a buffer (gap) at the end then you'll overwrite the first sector.

As far as I remember there is no way to know how many tracks there are on a disk. At least in the IBM format that information is not stored on the disk and there is no way (that I know of) to query a disk drive what it is capable of. A controller bases the number of tracks on the format that the drive and controller were designed for. You could of course just try and read starting at track 0 and see how many formatted tracks there are. When formatting, you just have to know how many tracks will fit on the disk. That depends on the disk drive itself and is usually either a maximum of 40 or 80. But many formats use fewer tracks (e.g. 35 or 76).

[GSoftwareDevelopment]

It should be much easier to pass the byte count high byte (2 for 512 bytes or 1 for 256 bytes) into r27 than trying to read it from the pre-computed buffer.

So I understand that it is better to do separate formatting routines for sectors 128, 256 and 512 bytes?
I will honestly say that I was about to do just that :) Although this is the simplest solution and a bit suboptimal in terms of memory occupation, but.... I take it as a last resort.

But you don't need it since the sector size should be a multiple of 256 anyways.

That's what I understand, you can't create sectors smaller than 256 bytes, yes? or is it just because the value 0 (zero) is written to the r26 register and hence a multiple of 256? So to put it in simpler terms. If I do this: (we are talking about a hard code)

ldi r26, 128
ldi r27, 0

This will create a 128 byte sector for me?

---

Regarding the number of sectors (relatively their size), does the parameter describing the bit length m_bitLength make a difference? What happens if I format the track, e.g. for 256 bytes/sector, but read it with the procedure for 512? :) Will it read it over those 128, giving rubbish, or will it come out with an error? And similarly with a write, where the sector is 256 bytes and I try to write 512 to it? :)

---

I noticed that there is one byte of the sector block that seems to be a purely software issue. It refers to the first byte of the buffer array buffer[0] - its value is $FD. When reading, it is compared, so I understand that its value is only related to format compatibility for PC/IBM?
Can I put a different value in its place? Obviously following the rule where I check it on reading, or ignoring it? ;) I thought I could use this byte to put in simple floppy disk format information. Using a suitable arrangement of bits, I could include information about: number of tracks, number of sectors on a track, sector size or even, use of two sides of the floppy disk. :)

---

Regarding checking the number of tracks available. That's what I assumed, that there is rather no one method for that. I was even thinking myself, that when a new disk is inserted - the flag you mentioned #11 and pin 34 :) will be set - then it checks the first two tracks. Why two? I'm only assuming a 40 and 80 track layout, and consequently, when I format for example for 40, I format the unused track with zeros alone - I even set the CRC to zero to get a physical error as a result of reading it. Then, checking these two tracks will give me the answer whether I'm dealing with a 40 or 80 track floppy. Correct me if I'm wrong. :)
Of course, the above argument about checking the number of tracks could be ignored when it would be possible to put the information in buffer[0] when writing :)

---

Anyway, I think the project is really interestingly done and I also wanted to thank you for your support - I appreciate it.

[dhansel]

So I understand that it is better to do separate formatting routines for sectors 128, 256 and 512 bytes?

What I was trying to say is that you can still use just the one formatting routine but change line 1080 such that the proper value is set for r27, i.e. add ',"r27"(bytesHi)' to line 1080 where bytesHi is a uint8 variable in C that contains either 1 for 256 bytes or 2 for 512 bytes.

Creating multiple copies of the format code will take up a lot of code space on the Arduino UNO. If I remember correctly, my ArduDOS sketch was already getting close to the limit.

That's what I understand, you can't create sectors smaller than 256 bytes, yes?

You can create smaller sectors (with the method you are describing). But of course if you hard-code 128 bytes/sector then you can't create other sizes anymore. So it would be better to pass both the low and high byte counts for the sector length from C into the assembly code (similar to what I described above). Making these changes will require learning more about the ATMEL assembler and it's interface to C code.

I noticed that there is one byte of the sector block that seems to be a purely software issue. It refers to the first byte of the buffer array buffer[0]. [...]. Can I put a different value in its place?

That first byte (in the IBM disk format) is a marker for whether the following block is a sector id header (0xFE) or sector data (0xFB). It's used in a couple places in the code to identify what's being read so you'll have to take care when changing it. Otherwise the code might not find any sectors anymore.

Of course as with all the changes you are proposing, this deviates from the IBM format so other drives won't be able to read data or work with the disk.

Regarding checking the number of tracks available

You could do what you propose but keep in mind that some floppy drives physically only have 40 tracks. If your drive only supports 40 tracks then it won't work. But that only applies to older (DD - double density) drives. Also, writing a disk on a 80-track drive and then trying to read the data from a 40-track drive can lead to read errors because the 80-track drive's head is smaller than the 40-track drive head. But as long as you are only working with "modern" HD (high density) 80-track drives it shouldn't be a problem.

[GSoftwareDevelopment]

add ', "r27"(bytesHi)' to line 1080 where bytesHi is a uint8 variable in C that contains either 1 for 256 bytes or 2 for 512 bytes.

The original represents:

     : "r"(bitlen), "r"(numsec), "r"(datagaplen), "z"(buffer), // inputs (z=r30/r31)
     : "r16", "r17", "r18", "r19", "r20", "r21", "r26", "r27"); // clobbers

I entered according to what you specified:

     : "r"(bitlen), "r"(numsec), "r"(datagaplen), "z"(buffer), "r27"(bytesHi) // inputs (z=r30/r31)
     : "r16", "r17", "r18", "r19", "r20", "r21", "r26", "r27"); // clobbers

and I changed the function header to:

static byte format_track(byte *buffer, byte driveType, byte bitlen, byte track, byte side, byte bytesHi=2)

However, I get an error:

/home/pebe/Arduino/ArduFDD2SIO/ArduinoFDC.cpp: In function 'byte format_track(byte*, byte, byte, byte, byte, byte, byte)':
/home/pebe/Arduino/ArduFDD2SIO/ArduinoFDC.cpp:1090:63: error: matching constraint references invalid operand number
      : "r16", "r17", "r18", "r19", "r20", "r21", "r26", "r27"); // clobbers
                                                               ^
/home/pebe/Arduino/ArduFDD2SIO/ArduinoFDC.cpp:1090:63: error: matching constraint references invalid operand number

However, when I change - which is probably wrong - to:

     : "r"(bitlen), "r"(numsec), "r"(datagaplen), "z"(buffer), "r"(bytesHi) // inputs (z=r30/r31)
     : "r16", "r17", "r18", "r19", "r20", "r21", "r26", "r27"); // clobbers

The compilation passes, but I have absolutely no idea if it is right.

Sorry, the ATMEL assembler in the C implementation is magic to me.

Creating multiple copies of the format code will take up a lot of code space on the Arduino UNO. If I remember correctly, my ArduDOS sketch was already getting close to the limit.

Yes, I realize this is memory-inefficient - I mentioned it :)
I do not use anything more than the ArduinoFDC class, so this memory I have "some" at my disposal. :)

You can create smaller sectors (...) will require learning more about the ATMEL assembler and it's interface to C code.

Heh... maybe by the method of deduction I can figure something out ;)
Theoretically, I could elaborate a bit on the meaning of the extra parameter in the function format_track(...byte bytesHi=2) When bytesHi would point to a value of 1 or 2, it would mean sectors 256 and 512 - that is, the MSB byte would be set. However, if its value was 128, only the LSB byte would be set, and the MSB byte would be zero :)
Heh, I know how to do it for 6502 processor ;)

   lda bytesHi  // get value of bytesHi to Accu (equivalent of r27)
   bmi set128   // branch if N flag set (7 bit) if Accu equals 128 or more then jump
   tax          // transfer Accu to X-Register
   ldy #0       // Y-Register equals 0
   beq set      // branch if Z flag set (Zero) Always jump becouse Y-Reg is equal 0 - faster & shorter than JMP

set128:
   tay          // transfer Accu to Y-Register
   ldx #0       // X-Register equals 0

set:
// at this point we already have in X and Y registers respectively MSB & LSB of sector size :)

In general, I want to use only 3.5" MF2HD 1.44MB floppy disks. Not being able to read them after reformatting in a PC, will even be advisable, since the media will be compatible only with the Atari and its systems (there are some :P)
Currently, it already cooperates with the Atari computer (video - I'm "ranting" a bit in Polish, but don't worry :D ), however, this cooperation is at a very basic level. The maximum capacity of the floppy disk is not used. No write optimization - a sector of 128 bytes written in 512 bytes....
But thanks to your help, and others from the ranks of retro geeks, the project is moving forward :)

[dhansel]

/home/pebe/Arduino/ArduFDD2SIO/ArduinoFDC.cpp:1090:63: error: matching constraint references invalid operand number

I think this error occurs because after adding "r27" in the inputs list you now have "r27" in both the inputs and clobbers list (the clobbers list informs the compiler which registers are used by your assembly code so the compiler can't count on them being the same after your code finishes. Remove "r27" from the clobbers list and try again.

If you leave out the "27" then the compiler will just choose a register. That's a possibility too but then you'll have to modify the assembly code to use the register that the compiler chose. If you look at the assembly code you can see how I did that with the "bitlen", "numsec" etc. values.

[GSoftwareDevelopment]

Remove "r27" from the clobbers list and try again.

     : "r"(bitlen), "r"(numsec), "r"(datagaplen), "z"(buffer), "r27"(bytesHi) // inputs (z=r30/r31)
     : "r16", "r17", "r18", "r19", "r20", "r21", "r26"); // clobbers

Unfortunately, it gave exactly the same error :/.

/home/pebe/Arduino/ArduFDD2SIO/ArduinoFDC.cpp:1090:56: error: matching constraint references invalid operand number
      : "r16", "r17", "r18", "r19", "r20", "r21", "r26"); // clobbers

As such, the whole thing compiles Only when there is such a notation:

     : "r"(bitlen), "r"(numsec), "r"(datagaplen), "z"(buffer), "r"(bytesHi) // inputs (z=r30/r31)
     : "r16", "r17", "r18", "r19", "r20", "r21", "r26", "r27"); // clobbers

If you leave out the "27" then the compiler will just choose a register.

Ok. I looked at it and I hope I made a good observation :)
The bitlen is allocated to the r register. In the code, access to its value is through %0.
numsec similarly, and access is through %1.
datagaplen = %2
The buffer will "sit" in r30 and r31 - I don't understand this z a bit :) However, %3 is no longer present, but instead there is Z+ in the code.

Simple conclusion.
Passing variables through r gives access through %n, where n is the next number (counting from zero) of the parameter, passed in the list:

//        %0           %1           %2               ? but r30/r31
    : "r"(bitlen), "r"(numsec), "r"(datagaplen), "z"(buffer)

That is, if I add to this list "r"(bytesHi)....

    : "r"(bitlen), "r"(numsec), "r"(datagaplen), "r"(bytesHi), "z"(buffer)

...then the value of the variable, will be available through %3.

That is, in the place where the number of bytes per sector is set in registers r26 and r27, I should do so:

    ldi r26, 0
    ldi r27, %3

Do I understand correctly? :)

---

If the above is true then taking into account the sector of 128 bytes, you could do for example like this:

    ldi r26, 0  // when `bytesHi` > 0, 
    mov r27, %3 // sets the sector size for 256 and 512 (or multiples of 256)
    brne beginSec
    ldi r26, 128 // when `bytesHi` = 0
    ldi r27, 0   // assumes a sector size of 128 bytes
beginSec:
    WRTPS
    rjmp dskip
dloop: ...

I would appreciate your reference to the above "deduction" :)

[dhansel]

Unfortunately, it gave exactly the same error :/.

As I said it's been a while since I worked on this code (or any ATMEL assembly). Not sure what else could be the problem.
You may want to take a look here: https://www.nongnu.org/avr-libc/user-manual/inline_asm.html
That's where I got a lot of info as to how the "asm" statement works and how to pass values.

Passing variables through r gives access through %n, where n is the next number (counting from zero) of the parameter, passed in the list

That's the general idea. It should work but I can't vouch for it without trying it by myself. One way to check whether the compiler did the right thing is to take a look at the generated assembly code. You can dump the assembly code using this command: avr-objdump.exe -S ArduinoFDC.ino.elf > ArduinoFDC.s [where ArduinoFDC.ino.elf is the .elf file produced by the compiler and ArduinoFDC.s will be the resulting assembly file.

The result isn't exactly easy to navigate but if you look closely you can check whether it's using the right registers.

you can also un-comment the "#define DEBUG" statement in ArduinoFDC.cpp to get some debugging info dumped
to the Serial port. It's not a lot but may be helpful. Make sure you have the serial port set to 115200 baud, otherwise
the debug output will slow down the code too much and it won't work anymore.

Let us know if/how this worked out for you!

[GSoftwareDevelopment]

Ok, thanks for the support of all kinds - I very much appreciate the information given to me. If I can get more details via avr-objdump I will certainly try to come to some consensus.

Thanks very much and I am not prolonging the agony caused by tiresome questions and back to the past ;)

I wish you good health.