FPGA to solve the two most annoying problems on usenet - Suggestions Welcome

As you all know, downloading files from usenet leaves you with two sets of files.. The rar files from what you're downloading and the par2 files for incomplete file repair.

If anyone has attempted to download anything in the 8GB range, you'll find that well.. if you're missing enough parts of the file, the par2 recovery can be a painful, painful process taking up to three hours in some cases.

I'm sick of it.

So, here's the idea. Use a FPGA to do the reed-solomon decoding to accelerate the PAR2 repair/recovery process. The system should utilize a USB connection to pipe data directly from the disk to the FPGA that will do the offboard processing of the data. The data transfer should be controlled by an application on the computer.

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264. Converting from x.264 to h.264 could be done offboard on an FPGA because it takes for-ever to complete on my system (8 hours). This should have the same premise as the previous issue, minus using a x.

264 decoding core and possibly directly converting it to h.264 or doing a decompression-recompression..

SPECS: The development system i'm using is the XILINX ML-505 board with the Virtex 5 chip. This is a open-source project being done for fun and learning btw.

Suggestions / Comments / Complaints?

Reply to
Charles Xavier
Loading thread data ...

Seems like a really nice idea. Please notify your progress.

Reply to
Atilla Filiz

There are chips that will do this already. Here's one:

formatting link

Cheers, Jon

Reply to
Jon Beniston

Thanks for the reply. Aspects of this design can certainly be used and incorporated into this design, but as you know.. Their product isn't free/open source. As this is also a learning project, I think it would be better to build it from the ground up (using some pre-existing open source IP if possible).

Reply to
Charles Xavier

Why USB ? Gigabit Ethernet would be much faster ;) Don't tell me a Virtex5 can't handle a measly 100 MB/s ;)

And... is it a PCI express slot that I see there ? That's even better !

I was under the impression that x264 was h264 in disguise. Perhaps tweaking the file format would be enough ?

Also you could recode the thing in a much simpler codec like MPEG2 or something, who cares if it takes 10x the size, just stream it in realtime ;)

Stick a DVI port on your FPGA !

Reply to
PFC

I have a hard time believing that those are the "two most annoying problems on usenet". I've been using Usenet since 1984, and I've never personally encountered either problem.

I've never tried to download an 8GB file from Usenet, and don't have any clue why anyone would want to do such a thing. There are much better ways to distributed 8GB files. But if I *did* want to download an 8GB file from Usenet, I can't imagine that waiting three hours for a reconstruction of missing pieces would really bother me that much. There's never been a single thing that I've needed from Usenet so urgently that it couldn't have waited three more hours.

I watch a fair number of movies, but I've never had any problem with x.264. It sounds like you just need better player software. That has nothing to do with Usenet.

Since it doesn't appear that there is any real problem here, it also doesn't appear that there is any need for an FPGA-based "solution".

Reply to
Eric Smith

s
y

Thank you for sharing. Let your next post be in contribution to the project.

If it doesn't apply to you and you don't see the need, why even reply?

Reply to
Charles Xavier

Hi Charles, Maybe to share the benefit of his experience to help you avoid wasting your time on a solution without a problem? Good luck, Syms.

Reply to
Symon

r

Engineers can often be of a single track mind, so let me restate. This is a problem that I am having. I could certainly get a faster computer, or try to finesse the algorithms for the decoding, but I want to learn about this specific algorithm implementation on an FPGA, so I'm using the FPGA to solve this problem (that is a problem for me).

To me, this is a practical solution and an excellent gateway for myself and others who want to get into FPGA-based hardware acceleration of software algorithms. You give me a better way that solves a practical problem for _me_ and I'll take it.

This reminds of the idiots at In-N-Out Burger who created the drive- through fast food restaurant (they claim). It doesn't affect the geriatric who only wants to eat inside because they can't imagine driving while eating. But apparently it turned out to be a good idea.

Responding to this type of feedback is the real waste of time.

Reply to
Charles Xavier

I'm not sure what the problem with x.264 is, but lets just say that it is far from trivial to write a H.264 decoder/encoder. The decoder is especially tricky as you need to be able to parse all parts of the standard whereas you can select only the parts you care about for the encoder.

If you are an expert on video codecs and a newcomer to FPGAs, this would probably be a quite ambitiouis project to start with. If you are not an expert on video codecs, you will have to spend a lot of time to understand H.264 before you can even get started on the FPGA parts.

This is written from personal experience by the way, I have written a grayscale H.264 decoder in C to get an understanding of how the standard works. Lets just say that it took way more time than I wanted to...

In summary: I don't recommend a H.264 codec as a first FPGA project.

The Reed-Solomon application seems like a much more reasonable project. You will still have a lot of things to figure out like how you are going to communicate with your FPGA board, but the problem itself is much more limited.

/Andreas

Reply to
Andreas Ehliar

Just program one of these:

formatting link
?articleID=208404063

240 cores, teraflop.

John

Reply to
John Larkin

I have an idea simular or not to your mind..(but I still thinking....hope you are now starting ..your project). my project is that

FILE COMPRESSION (zip or other) read files from SD card, then compress it, and write a new zipped file back to SD card.

it will not > As you all know, downloading files from usenet leaves you with two

Reply to
Aiken

"Suggestions / Comments / Complaints?"

Reply to
Mike Treseler

Consider using rateless codes, reed-solomon and ldpc based codes do provide erasure correction capabilities, however rateless codes such those based on LT class of codes (tornado etc) ,or more generally digital fountain codes, will have better performance due to their near linear decoding complexity as opposed to say reed-solomon erasure only decoding complexity which is around O(n^2) for naive implementations.

Arash Partow __________________________________________________ Be one who knows what they don't know, Instead of being one who knows not what they don't know, Thinking they know everything about all things.

formatting link

Reply to
Arash Partow

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.