• AVR Freaks

DEE Emulation - garbage collection?

Author
dom
New Member
  • Total Posts : 7
  • Reward points : 0
  • Joined: 2011/08/30 22:34:32
  • Location: 0
  • Status: offline
2013/02/04 20:29:07 (permalink)
5 (1)

DEE Emulation - garbage collection?

 
The PackEE() routine in 16b version of Microchip's Data EEPROM Emulation v2.2 library happily copies over all emulated address entries, even if their value is in 'electrically erased state' (0xFFFF). This means if an address gets written once, it will forever be carried over through page packing. There is apparently no way to tell the DEE Emulation to forget that address - something akin to TRIM on SSDs. It only takes adding a condition as in the code snippet bellow to skip an emulated EEPROM address during packing routine if it holds erased value.
 

DEE Emulation 16-bit.c:
393: else if(~latchData) //program data only if not erased, otherwise skip to next address

 


#1

9 Replies Related Threads

    Neiwiertz
    Super Member
    • Total Posts : 2094
    • Reward points : 0
    • Joined: 2004/09/01 02:58:52
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/05 00:24:09 (permalink)
    0
    Can you please enter a Support Ticket at Microchip about this potential hazard ?
    Therefore the writer(s) of DEE can take care,

    Is it also within newer then 16b ?
    Can you post a snippet of the PackEE here and mention the particular related lines
    how it can occur that it forever be carried over through page packing your discover ?
     
    Regards
    post edited by Neiwiertz - 2013/02/05 00:29:28

    Flying With --|Explorer 16|HardWare|SoftWare|-- Fav(s) Gallery Lists
    #2
    dom
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2011/08/30 22:34:32
    • Location: 0
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/05 02:25:17 (permalink)
    5 (1)
    Posting snippets is a royal pain on these Microchip forums - when I paste it strips all the newlines! (Chrome)
     
    How can it happen? The 3 byte wide flash instruction word holds the virtual (emulated) EEPROM address in 8b MSB and the 16b value under said virtual address in LSB. The packing routine goes through all the virtual addresses and carries over to the new packed page all the virtual addresses that exist in the old page, regardless of the data value stored under the virtual address. This way once a user writes to a particular virtual address, it will forever remain in the flash and survive page packing even though it's data field has been 'erased' to 0xFFFF.
     
    This won't cause corruption of the emulated array, however it will potentially decrease the endurance and increase the incidence of page packing. Most devices have 512 word page and can store up to 255 virtual address entries, if each of them get written at least once, all the 255 addresses will be found in flash and will get carried over by the packing routine, leaving only 256 empty spots in a newly packed page. No matter whether any of those virtual addresses remain in use.
     
    Looking at the PIC32 version of the Microchip library, it's even worse - DataEERead() returns zero for virtual addresses not present in the page! It should return 0xFFFF just like an erased flash/EEPROM would. This is a logical fallacy, but if we played along, any virtual address found in the emulated page holding a value of zero should *not* get copied over to the newly packed page. So same issue here, plus a mind boggling reversal of conventions *and* more damage done by actually writing the flash to zero rather than just leaving it in an erased state.
     
    #3
    Neiwiertz
    Super Member
    • Total Posts : 2094
    • Reward points : 0
    • Joined: 2004/09/01 02:58:52
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/05 05:31:38 (permalink)
    0
    Hello okay i found 16b this means 16Bit at v2.2 now i understand.
    Thanks for your explain, can you provide a fix at the PackEE ?,
     
    I will enter a support ticket they should pick it up and if verify fix in a update at DEE
    for all PackEE routines. This is orginal DEE 16Bit at v2.2 PackEE
    int PackEE(unsigned char bank)
    {
        int currentPage;        //Array row (PM page) of active DEE page
        int packedPage;         //Array row (PM page) of packed page
        int savedTBLPAG;        //Context save of TBLPAG value. Current and packed page are on same page.
        int currentOffset;      //Current page offset
        int packedOffset;       //Packed page offset
        int i;
        unsigned char latchAddr;
        unsigned int latchData;
        unsigned char dataEEFlags_sh;

        savedTBLPAG = TBLPAG;

        // Find the active page.
        for (currentPage = 0;
             (currentPage < NUM_DATA_EE_PAGES) &&
             (GetPageStatus(bank, currentPage, STATUS_CURRENT) == PAGE_NOT_CURRENT);
             currentPage++) {}


        if (currentPage == NUM_DATA_EE_PAGES)
        {
            TBLPAG = savedTBLPAG;
            SetPagePackBeforeInit(1);
            return(3);      // Error - no active page
        }
        else
        {
            // Find the next unexpired page to use
            packedPage = currentPage + 1;
            if (packedPage == NUM_DATA_EE_PAGES)
            {
                packedPage = 0;
            }
            while(GetPageStatus(bank, packedPage, STATUS_EXPIRED) == PAGE_EXPIRED)
            {
                packedPage++;
                if (packedPage == NUM_DATA_EE_PAGES)
                {
                    packedPage = 0;
                }
                if(packedPage == currentPage)
                {
                    TBLPAG = savedTBLPAG;
                    SetPageExpiredPage(1);
                    return(1);      // Error - all pages expired
                }
            }
        }

        // Point to first location in packed page
        TBLPAG = DEE_PAGE_TBL(bank, packedPage);
        packedOffset = DEE_PAGE_OFFSET(bank, packedPage);

        if(GetNextAvailCount(bank))
        {
            SetPagePackBeforePageFull(1);           // Pack called before the page was full
        }

        dataEEFlags_sh = dataEEFlags.val;
        SetaddrNotFound(0);                 // Initialize flag
        i = 0;
        NVMCON = PROGRAM_ROW;

        WritePMLow(0xFFFF, packedOffset);
        WritePMHigh(0xFF, packedOffset);
        packedOffset += 2;

        latchAddr = 0;
        i++;

        do
        {
            while((latchAddr != DATA_EE_SIZE) && (i < NUMBER_OF_INSTRUCTIONS_IN_ROW))
            {
                latchData = DataEERead((255 * bank) + latchAddr);
                if(GetaddrNotFound())       //if address is unwritten, skip to next address
                {
                    SetaddrNotFound(0);
                }
                else
                {
                    WritePMLow(latchData, packedOffset);
                    WritePMHigh(latchAddr, packedOffset);
                    packedOffset += 2;
                    i++;
                }
                latchAddr++;
                while((latchAddr == DATA_EE_SIZE) && (i < NUMBER_OF_INSTRUCTIONS_IN_ROW))
                {
                    WritePMLow(0xFFFF, packedOffset);
                    WritePMHigh(0xFF, packedOffset);
                    packedOffset += 2;
                    i++;
                }
            }
            UnlockWrite();
            i = 0;
        }
        while(latchAddr != DATA_EE_SIZE);

        dataEEFlags.val = dataEEFlags_sh;   //Restore status flags

        //Verify data was written correctly into packed page

        // Point to first location after status
        TBLPAG = DEE_PAGE_TBL(bank, packedPage);
        packedOffset = DEE_PAGE_OFFSET(bank, packedPage) + 2;

        latchAddr = ReadPMHigh(packedOffset++);
        latchData = ReadPMLow(packedOffset++);

        while(latchAddr != 0xFF)
        {
            if(DataEERead((255 * bank) + latchAddr) != latchData)
            {
                TBLPAG = savedTBLPAG;
                SetPageWriteError(1);
                return(7);          //Error - data does not match
            }
           latchAddr = ReadPMHigh(packedOffset++);
           latchData = ReadPMLow(packedOffset++);
        }


        //Program page status
        currentOffset = DEE_PAGE_OFFSET(bank, currentPage);
        packedOffset = DEE_PAGE_OFFSET(bank, packedPage);

        // Point to proper TBLPAG
        TBLPAG = DEE_PAGE_TBL(bank, currentPage);
        latchData = ReadPMLow(currentOffset);
        latchAddr = ReadPMHigh(currentOffset);
        if(packedPage == 0)
        {
            latchData++;        //Increment E/W counter
        }

        if (latchData >= ERASE_WRITE_CYCLE_MAX - 1)
        {
            SetPageExpiredPage(1);
            latchAddr &= 0b11101111;
        }

        // Point to proper TBLPAG
        TBLPAG = DEE_PAGE_TBL(bank, packedPage);
        WritePMHigh(latchAddr, packedOffset);
        WritePMLow(latchData, packedOffset);

        NVMCON = PROGRAM_WORD;
        UnlockWrite();

        if((latchAddr != ReadPMHigh(packedOffset)) ||
            (latchData != ReadPMLow(packedOffset)))
        {
            TBLPAG = savedTBLPAG;
            SetPageWriteError(1);
            return(7);
        }

        //Erase active page
        ErasePage(bank, currentPage);

        TBLPAG = savedTBLPAG;
        return(GetPageExpiredPage());
    }


    Flying With --|Explorer 16|HardWare|SoftWare|-- Fav(s) Gallery Lists
    #4
    dom
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2011/08/30 22:34:32
    • Location: 0
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/05 07:34:27 (permalink)
    0
    int PackEE(unsigned char bank) 
    {
    ....
       // Find the active page.
    ....
       // Point to first location in packed page
    ....
       do
       {
           while((latchAddr != DATA_EE_SIZE) && (i < NUMBER_OF_INSTRUCTIONS_IN_ROW))
           {
               latchData = DataEERead((255 * bank) + latchAddr);
               if(GetaddrNotFound())       //if address is unwritten, skip to next address
               { 
                   SetaddrNotFound(0);
               }
               else if(~latchData)       //program data only if not erased (0xFFFF), otherwise skip to next address 
               {
                   WritePMLow(latchData, packedOffset);
                   WritePMHigh(latchAddr, packedOffset);
                   packedOffset += 2;
                   i++;
               }
               latchAddr++;
               while((latchAddr == DATA_EE_SIZE) && (i < NUMBER_OF_INSTRUCTIONS_IN_ROW))
               {
                   WritePMLow(0xFFFF, packedOffset);
                   WritePMHigh(0xFF, packedOffset);
                   packedOffset += 2;
                   i++;
               }
           }
           UnlockWrite();
           i = 0;
       }
       while(latchAddr != DATA_EE_SIZE);
    ....
      //Verify data was written correctly into packed page
    ....
    ....
    }

     
    It's there along with a comment, unfortunately cannot be highlighted since it's in a code block.
     
    The side-effect of this is that you get addrNotFound flag when accessing such 'garbage collected' virtual address. I don't think that should be of any importance to the application - there is no such thing as missing address in a real EEPROM after all, it just reads 0xFF, which is what happens here.
     
    The 32b code should be modified accordingly - actually it should be changed to return 0xFFFF for virtual addresses not found in the emulated bank, however I have the feeling such a change, however sane, could break some applications that rely on the current behavior.
     
    #5
    Neiwiertz
    Super Member
    • Total Posts : 2094
    • Reward points : 0
    • Joined: 2004/09/01 02:58:52
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/07 16:50:55 (permalink)
    0
    Hello Dom

    I found your Improve which is a Single Line, i added original Line and disabled it,
    yeah lets see if we can highlight it see below, i am also struggle with BB Code Tags the code and link tags are working but bold for example is unrecognized here, for this post i used forums build in msg editor adjusted font and used bold and highlight
     
        do
        {
            while((latchAddr != DATA_EE_SIZE) && (i < NUMBER_OF_INSTRUCTIONS_IN_ROW))
            {
                latchData = DataEERead((255 * bank) + latchAddr);
                if(GetaddrNotFound())       //if address is unwritten, skip to next address
                {
                    SetaddrNotFound(0);
                }
                //else
                else if(~latchData)       //program data only if not erased (0xFFFF), otherwise skip to next address
                {
                    WritePMLow(latchData, packedOffset);
                    WritePMHigh(latchAddr, packedOffset);
                    packedOffset += 2;
                    i++;
                }
                latchAddr++;
                while((latchAddr == DATA_EE_SIZE) && (i < NUMBER_OF_INSTRUCTIONS_IN_ROW))
                {
                    WritePMLow(0xFFFF, packedOffset);
                    WritePMHigh(0xFF, packedOffset);
                    packedOffset += 2;
                    i++;
                }
            }
            UnlockWrite();
            i = 0;
        }
        while(latchAddr != DATA_EE_SIZE);

    Flying With --|Explorer 16|HardWare|SoftWare|-- Fav(s) Gallery Lists
    #6
    Neiwiertz
    Super Member
    • Total Posts : 2094
    • Reward points : 0
    • Joined: 2004/09/01 02:58:52
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/07 17:05:20 (permalink)
    0
    Hmmm Looks like good within Preview Window by plain text by left out style end tag
    but when post message it fails color to be painted for highlight,
    yeah it skips pgd code within code block i see ;) 
           
    [style="background-color: #ffff99;"][b]//else[/b]
    [style="background-color: #ccffcc;"][b]else if(~latchData)       //program data only if not erased (0xFFFF), otherwise skip to next address[/b]

    //else
    else if(~latchData)       //program data only if not erased (0xFFFF), otherwise skip to next address

     

    else if(latchData!=((unsigned int)0xFFFF))
      will be the same behaviour as
    else if(~latchData)
    post edited by Neiwiertz - 2013/02/07 17:26:39

    Flying With --|Explorer 16|HardWare|SoftWare|-- Fav(s) Gallery Lists
    #7
    Neiwiertz
    Super Member
    • Total Posts : 2094
    • Reward points : 0
    • Joined: 2004/09/01 02:58:52
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/07 18:06:20 (permalink)
    0
    domThe PackEE() routine in 16b version of Microchip's Data EEPROM Emulation v2.2 library
    happily copies over all emulated address entries, even if their value is in 'electrically erased state' (0xFFFF).
    This means if an address gets written once, it will forever be carried over through page packing.
    There is apparently no way to tell the DEE Emulation to forget that address - something
    akin to TRIM on SSDs.
    It only takes adding a condition as in the code snippet bellow to skip an emulated EEPROM address
    during packing routine if it holds erased value.

    Good catch of the yeare :), hope they can introduce your feature 'to forget that address' if possible

    domLooking at the PIC32 version of the Microchip library, it's even worse...

    They should look at it as a heads up improve it correctly if possible

    dom
    The side-effect of this is that you get addrNotFound flag when accessing such 'garbage collected'
    virtual address.
    I don't think that should be of any importance to the application - there is no such thing as
    missing address in a real EEPROM after all, it just reads 0xFF, which is what happens here.

    I agree and can live with 'get addrNotFound flag' maybe rename this flag addrUnUsed will do
    and solve the side effect or way out to avoid addrNotFound solution
    and "accessing such 'garbage collected' virtual address." should be avoid in first place if that make sence don't know

    Thanks for your detailed explain maybe idea to add some of your explain at manual would be usefull
    post edited by Neiwiertz - 2013/02/07 18:11:18

    Flying With --|Explorer 16|HardWare|SoftWare|-- Fav(s) Gallery Lists
    #8
    Neiwiertz
    Super Member
    • Total Posts : 2094
    • Reward points : 0
    • Joined: 2004/09/01 02:58:52
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/07 18:13:40 (permalink)
    0
     I included this one here at E16 DEE Data EEPROM Emulating And Read Write FLASH
     
    Regards
    post edited by Neiwiertz - 2013/02/07 18:18:31

    Flying With --|Explorer 16|HardWare|SoftWare|-- Fav(s) Gallery Lists
    #9
    Neiwiertz
    Super Member
    • Total Posts : 2094
    • Reward points : 0
    • Joined: 2004/09/01 02:58:52
    • Status: offline
    Re:DEE Emulation - garbage collection? 2013/02/20 15:30:45 (permalink)
    0
    dom
    The side-effect of this is that you get addrNotFound flag when accessing such 'garbage collected' virtual address.
    I don't think that should be of any importance to the application

    Hello Dom,  I did have an Answer on my Ticket and we overlooked one More ("nasty") side-effect that if we do so,
    It will Not support Anymore a Value of 0xFFFF to be stored at an address by the application,
    Therefore your discover is still an improve but will take amount of changes to be implemented in a correct way,
     
    Wisdom to keep Microchip DEE as is but i will keep your Discover in mind wink
    post edited by Neiwiertz - 2013/02/20 15:35:10

    Flying With --|Explorer 16|HardWare|SoftWare|-- Fav(s) Gallery Lists
    #10
    Jump to:
    © 2019 APG vNext Commercial Version 4.5