Video scaler for Spartan 3E?

Hi Just want to do a feasibility study on whether it is possible to design and implement a video scaler on a Spartan 3E? Well my tutor kind of came up with this proposal for a project of mine but then on my tutor's side, he has experience with fpga but I'm not sure whether he had designed video scaling so may not know the complexity of it.

So, feasible? And with which chip as well? (in the best of best case, i hope can use just a spartan starter kit :) )

Reply to
Ken Soon
Loading thread data ...

Oh if for what images to scale.. hmm maybe videos from a camera? or would it possible to scale video from a vcd player?

Reply to
Ken Soon

For that you need to sample a 250kB image into RAM and read it back in a different order. You need to do that 25 or 30 times a second. As long as you have 500kB of SRAM that can handle 16MB/S random access this should be easy. For better results you need to interpolate multiple pixels on readback which increases the bandwidth requirement. Rotating the image requires not much more hardware than scaling. Both is done with two bresenham units.

Kolja Sulimma

Reply to
comp.arch.fpga

as a starter you may look at "rotazoom" IP included in demo designs of spartan-3A kit.

it takes bitmap image from memory and display it zoomed and rotated on display.

Antti

Reply to
Antti

On a sunny day (3 May 2007 01:20:54 -0700) it happened "comp.arch.fpga" wrote in :

I dunno, I think you can do that with much less memory, say we scale 1 to 2, then you only need to strore 2 picture lines of the original:

ABCDEFG HIJKLMN

Now you start averaging first pixel out = (A+B+H+I) / 4, second pixel out = (B+C+I+J) / 4.

etc, In this case you only need to store 2 lines. For other factors you need some more lines. Do not forget the aliasing factor, you are sub sampling, and need to lowpass at below f/2.

Reply to
Jan Panteltje

Ken,

Just out of curiosity, what is your timeline on this project?? You've been asking about this for months on here! (That's what all your DDR interface questions were about, no?)

There's all sorts of considerations here...

What is your input data format? Composite video? RGB? etc.. Some sort of computer driven RGB input would probably be your simplest as you'll be receiving your three colors seperately and you can just sample them as 3 input streams. But you're gonna have lots of learning and educating yourself to do on sync timing and all that fun stuff - Keep in mind, most analog video signals were designed to be fairly crude signals to drive a big honking electromagnetic coil that certainly didnt respond anything close to the clock rate of your spartan. Therefore, putting them into the digital world in some sorta of useful form can be a bit tricky - basically you need to look for your syncs and count out to where your first pixel is supposed to be.

What are your input & output resolutions? frame rates? If you're going 10x10 to 20x20 at 5 fps, I'd bet you could do that in a Spartan3... 640x480 input at a 60hz refresh, I don't really know - that's where you're going to have to sit down and run some calcuations. Go to your school's library and take out some books on image processing or multi-dimensional DSP... you're gonna need to learn some basic image processing first to get an idea of what operations need to be done before you can begin to grasp the size of the part you need.

How accurate do your interpolations need to be? Think basic 1- dimensional DSP.. how do you upsample a signal? insert zeros between samples and LPF, right? Or do the same thing in the F-domain by zero padding the ends of the FFT and inverse-FFT'ing. Either of these methods in 2 dimensions is going to be a lot of computation and a lot of taking stuff in and out of memory in different orders. DDR memories like you to take stuff out in the same order you put them in... they slow down big time when you try to jump around.. So if you do this, you will need to some up with some clever methods of read out pieces from DDR in the incorrect order, and then re-reading from a local, smaller, block ram in the order you actually want. This will take careful planning and a lot of simulation, even an experienced designer would have a tricky time with this and probably get it wrong in simulation the first shot.

If your up/down sampling doesn't need to be terribly accurate, this simplifies things. Consider a simple first order interpolation... look left, look right and figure out what goes in between. You would do the row dimension as the data flows in, just put two rows into memory at a time and then do your column dimension. Obviously, you will need to calculate bandwidth here and make sure you've got plenty enough (with margin!) on your memory.

Also, remember that multiplication & division are tricky in the binary world. The spartan3 does have some built in multipliers. But for a school project consider limiting your scope to make your life a little easier. Maybe you only scale by powers of 2, so everything becomes a Left/Right shift instead of mult/div? Maybe you limit it even more, you take 640x480 60Hz in and you ouptut 1280x960 60Hz, and do nothing else. The fewer possible cases you need to accoutn for the more likely it will be to fit in a smaller part.

Lastly, are you trying to do this on the spartan3 starter kit? becuase that has no video in, and only a crude video out (no DAC, you can just turn on and off the RGB pixels in combinations.. so 2^3 = 8 colors, of which 1 is black and 1 is white)... If you are building your own board you are going to be limitted by the size FPGA you can get in a quad flat pack part - i assume you dont have the facilities to do a BGA - and those QFPs are still VERY fine pitch to solder by hand. Let me know if you've found an eval board already that's suitable for this application, I;m curious - I don't think any of the digilent spartan boards have video input.

Good luck

Reply to
Paul

I have 2 data points for you nased on existing designs with Spartan 2e and Virtex 2. The latter is more closely related to the Spartan 3. In that design I fit a scaler / combiner taking 4 RGB inputs each 800 x

600 at up to 120 fps. The inputs were not synchronized so they needed to be stored in SDRAM first. The output was scaled between 1:2 and 2:1 (50 - 200%) and mixed into a single 800 x 600 output frame DMA'd into a PC using a PLX9656 local bus adapter. This fit pretty handily in a Virtex 2 1000 using DDR SDRAM (1 64-bit SO-DIMM).

The other design was standalone using a Spartan 2e 400. This design takes two 640 x 480 inputs stores them in single-data-rate SDRAM and then combines them into a scaled 1024 by 768 output at

60 fps.

Both designs use a low-pass filter followed by linear interpolation in each dimension (horizontal and vertical). For the horizontal filtering I used distributed arithmetic filters to save logic.

So, given enough time and design expertise I would think you could easily come up with a scaler for TV (even HDTV) video in a Spartan 3 400 or larger assuming you have the other required resources. Usually you'll find that you run out of block RAM before you run out of fabric in this sort of design. Also without the DA FIR, you could run out of multipliers.

Have fun, Gabor

Reply to
Gabor

You remembered my posts! Ok, anyway, well actually I had started with this project for my attachment which lasted for about 2-3 months and it nearing its end and don't forsee completing it. And next year, I had another full-fledged project assignment which will take about ,taking on the safe side, about 10months? So I thought of trying to progressing on for on my current project design (trying to learn how it works) or maybe even create my own video scaler myself on a cheaper 3E starter kit than the 1600E inrevium board that i'm using now.

Hmm, well, not sure on the input data format. I would like to scale a composite video but if i can scale RGB format, I would be satisfied as I'm still really new in this field. Sync timing, hmm I did some tiny demo using other people's design which had a hv sync generator, maybe have to study it more.

Input & output resolutions and frame rates? I would like to start with

640x480 and 60Hz since it seems basic enough to start with this (though i would like to cater to lower resolution with progress) Gabor mentioned that he did it with Spartan 2e 400 which took in two 640 x 480 inputs, stores them in single-data-rate SDRAM and then combines them into a scaled 1024 by 768 output at 60 fps.

Woah, ok I'm totally lost in here. But no problem, I will take note of these points and consult my profs or someone

First step i got to take will just be a simple fixed resolution scaling of one resolution set, like you said 1280x960. Well I'm not sure if i will be lent a expensive board with the 1600E on it or the starter kit. Oh I saw that the starter kit does have a DAC and also an ADC. Oh and it does have a Hirose FX2 connector then maybe I can add the digilent video decoder for S video,

Thanks alot Paul for providing the considerations that I will need to take note and the immense amount of advices. Btw, just a side note though, for my current project, I have reduced the Htaps and Vtaps to 4 & 3. Minimum frequency is about 177Mhz, above the minimum requirement os 167Mhz. Now i'm just stuck with how to assign the pins (headers, h v coefficients, seq lut address,....) and how to actually use this scaler.

Hmm Gabor, even a scaler for HDTV on a Spartan 3e?!... oh btw, combining two 640x480 to a scaled 1024x768, isn't the usual way to interpolate from a single 640x480 source to a 1024x768?

oh and regarding SDRAM, i should have enough on even a starter kit with

512Mbit SDRAM available.

And aliasing factor, sub-sampling, and lowpass at below f/2...hmm ok will take notes of these too, Jan

Thanks alot guys for the help!~ much appreciated :)

Reply to
Ken Soon

As another poster mentioned: If input and output run at the same resolution and frame rate you only need to store a small number of video lines. You can do that in internal memory. Start with a project that samples the video input and directly outputs it in the desired format. Than do the same with one video line delay. Once you have that working your scaler is more or less done.

Kolja Sulimma

Reply to
comp.arch.fpga

OK... think time domain signal... say your signal x = [1 10 5 3] and you want to upsample this to twice the sampling frequency.... if you did a simple linear interpolation you'd get x_upsampled = [1 5.5 10

7.5 5 4 3 ?] that last datapoint{?} is kinda arbitrary, presumably you'd just set it to 3. However, signals are defined as a sum of sinusoids, and generally they fit that model fairly well - when you have something that doesn't, it takes more sinusoids to fit it.. thats the general theory behind a fourier transform, right? So linear interpolation isn't really you're best solution. You can mathematically prove that if you had an infinitely long signal, and an infinitely perfect low pass filter, the "perfect" upsampling is done by inserting zeros x_zero = [1 0 10 0 5 0 3 0], and low pass fitlering it. Obviously, neither the infinite sequence or the perfect LPF is realistic. If you do the math (or plug it into matlab), the FFT of x (shfited to place zero frequency in the center), X=[-7 -4+7j 19 -4-7j] If you take the FFT of x_zero, X_ZERO = [19 -4-7j -7

-4+7j 19 -4-7j -7 -4+7j] Note, this is simply X repeated. So how do you get back your correct frequency spectrum? An ideal LPF returns X_UPSAMPLE=[0 0 -7 -4+7j 19 -4-7j 0 0]. Now note this... if you use matlab and take the inverse FFT of this (the shifted version of this actually) you get [1.0000 5.8107

10.0000 8.6391 5.0000 3.6893 3.0000 0.8609] Actually... you get this divided by 2 (you need to multiply by your upsampling facor, that comes out in the math) with a risidual imaginary part due to the fact that it's an even length sequence, so you just look at the real part. Point being however, note that the interpolated values are NOT what you get using straight linear interpolation. this is because this method takes into account the curvature of the rest of the sequence and stuff.

That being said.... for this project, I would imagine that simple linear interpolation is plenty adequate! A 2 dimensional version of this example above would be quite tricky and take a lot of processing power. But read the chapter in your DSP book on upsampling - it will cover all this :-) Actually, not positive, but the stock, every college in the world uses, oppenheim & shaeffer signals & systems book MIGHT even go into it in one of the later chapters.

Reply to
Paul

Yeh Kolja, I will definitely start with something simple first. Since I haven't begin to understand the concept of coefficient and upsampling.

Ah thanks Paul, for taking out the time to explain some concept. (trying to digest all of them though >_ OK... think time domain signal... say your signal x = [1 10 5 3] and

Reply to
Ken Soon

I definately was not saying that linear interpolation results will be "very" poor... in fact... I'd bet some commercial scalers do it that way.... At any rate... I'd be cautious of shooting for the stars - you're better off with an obtainable project... Here's some considerations...

640x480 = 307200points... which, if you're talking about RGB input, means 307200 points of Red, Green & Blue each.... 307200points x 60fps = 18.4MSample/s per color... lets assume you're processing the three colors independently... I know the Spartan3 development boards have 50MHz clocks on them. I probably wouldn't try to push a Spartan design much past 100MHz unless it was a VERY empty part (the higher the utilization the more difficult it is to place and route it at a higher speed). Now, assuming 100MHz, processing 3 colors independently, that basically means you can do 5 "operations" in a sample time... if you want to double your output rate (i.e. - scale up to 1280x960) that may be as low as 2.5 operations, depending what the operation is and if it needs to be done at the higher or lower rate. In short what that gets down to is that anymore operations than that must be done in different "blocks" of the design (i.e. they use more resources). And that is going to quickly run you up against the limit in terms of the number of multipliers in the spartan probably.

Now those are all TOTALLY ballpark, off the cuff numbers. My gut instinct would be that linear interpolation will be NO problem at all in the spartan at 100MHz, I'd bet you could even do it at those resolutions at 50MHz. But my other gut instinct there is also that any real significant amount of processing beyond linear interpolation is going to get really tight, really quick. I think as a student you're better off not trying to attempt a really tightly packed design. The reason I say that is that to get a really tight design to place and route correctly and meet timing you need to really know all the ins and outs of the software tools (the Xilinx software). You have to set all sorts of settings, possibly go in an manually place critical parts of the design, etc... etc.... And that's REALLY not what you want to be spending your time on as a student. Your purpose is to learn about the logical part of FPGA design, and that's going to be tricky enough on its own - I really wouldnt recommend working yourself into that tight a position... You're really going to have a lot on your plate with this design as it is.

At any rate.. that's my advice.. anyone else is welcome to disagree...

Reply to
Paul

ok... I guess i better start small anyway since I really new to this. Well I'm going to take a DSP module in school soon so it will definitely help me along.

(the higher the utilization the more difficult it is to place and route it at a higher speed). Yes so true. I kept getting timing constraint problem for my current video scaler project. Well though i learnt that timing delay could be reduced by adding output registers to fit the timing constraint.

Oh yah consulted my colleague who said that my video scaler (polyphase actually, did i mention it before?) is good in a sense that it consumes lesser resources. Well I could be working on this project further on, though i will still need to understand some of the theory in video scaling as well.

Well really thank alot for the time and effort to advise me and give me some direction.

Oh by the way, can I ask if that once i kind of burn the chip and little smoke came off, is the chip spoilt? The board's power indicators are fine and I programmed successfully into the chip through a parallel IV cable but nothing was displayed though. No self-test or whatsoever is available though...so scared...

Reply to
Ken Soon

[]

yes, smoke is rather bad usually :(

when really visible smoke comes, its usually really some dead things on board already, possible plastic housing cracked or some trace burned. sometimes before smoke comes you can feel with the nose, that something smells like hot, in that case the damage may be not fatal. real smoke is usually fatal.

the only hope that the damage is reversible is if there was some PCB trace burned (and all chips are ok), but the PCB trace burn doesnt make smoke, only some bad smell and sometimes visible light for short time. oh well depend on PCB and power supply, I guess PCB tracks can make smoke also..

as soon as something burns, better take immediate visual and smell observation, to see what burned, if PCB looks burnt, or if some IC has plastic cracked on top (reverse volcano where waporized silicium comes out)..

Antti

formatting link

Reply to
Antti

Everyone knows:

The smoke is what makes the part work.

If you let the smoke out, the part doesn't work anymore.

QED

DO NOT let the smoke out.

You can not put the smoke back in.

Austin

Reply to
austin

Yes, it's really difficult to get the smoke into a chip. Putting the smoke in at the factory requires a a fab costing upwards of a billion dollars!

Reply to
Eric Smith

Oh dear smoke is bad huh... Hmm it is the chip that burned, on the outer casing. I cant really see any hole on the chip so I kind of hope (hope) that everything is fine and that only because it got too hot for the casing that it burnt and gave off smoke. well I could program successfully into using IMpact though...right?

Reply to
Ken Soon

When discussing the way semiconductors *really* work (magic smoke) with friends years ago, the question arose:

What is the difference between a 3.3V 1A regulator and a 3.3V 2A regulator?

Almost everyone guessed that it was how much smoke is inside the regulator, but that is not correct.

Reply to
Eric Smith

Ken,

Seriously: any evidence of smoke or damage to the package, and it is all over. It is dead. We call it "electrical over-stress" which is a polite way of saying "you blew its brains out."

Austin

Reply to
austin

ok bummer... probably need to find the maker of the board for replacement of chip or something..

thanks guys ..sigh

Reply to
Ken Soon

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.