Welcome lantudou!
The sample application you mentioned is memory-intensive, and uses the DSP library for some realtime signal processing. It was presented at the MASTERs conference in 2004.
When MPLAB C30 was released in early 2005, a new memory allocator was introduced which supports several new combinations of memory attributes, including "Ymemory" and "near." It turns out this particular combination can be troublesome on certain devices. Also, it can occur unexpectedly because the default memory model specifies near data.
The C30 migration document (available from the C30 download page) explains your error message as follows:
Error: Could not allocate data memory
This linker error message is displayed when a section's attribute and alignment
requirements can not be satisfied. If the project previously linked successfully, the
problem may be related to variables allocated in Y memory. This issue is limited to
devices with 8K of RAM, where only a portion of Y memory qualifies as near.
C30 allocates all variables in near memory space unless the -mlarge-data option has
been selected. In previous versions, variables defined in Y memory were excluded from
this rule. In v1.33, some projects may fail to link on devices with 8K of RAM because a
solution for ymemory,near can not be found. The solution is to add the far attribute to
some or all of the Y memory variables, or to specify the -mlarge-data option.
There are two ways to solve this problem. The easy way is to open "Build Options:MPLAB C30:Memory model" and select "large data model."
Another approach is to edit file frequency.c, and apply the far attribute to the definition of buffer FFT_result:
/* fractcomplex _YBSS(256) FFT_result[64]; */
fractcomplex __attribute__((far)) _YBSS(256) FFT_result[64];
When writing a new application, the general rule is to specify the far attribute whenever an array will be allocated in Y memory. Near addressing implies a scalar context, and would rarely be useful with arrays.
Hope this helps!