Xojo Conferences
MBSSep2018MunichDE
XDCMay2019MiamiUSA

Fast Frustum Culling (Real Studio games Mailinglist archive)

Back to the thread list
Previous thread: Webkit and browsers
Next thread: SpriteSurface and Clicking


[ANN] Preview of RBD 2.4   -   Marc Zeedar
  Fast Frustum Culling   -   Lo Saeteurn
   Re: Fast Frustum Culling   -   Joseph J. Strout
   Re: Fast Frustum Culling   -   Lo Saeteurn
   Re: Fast Frustum Culling   -   Joseph J. Strout
   Re: Fast Frustum Culling   -   Lo Saeteurn
   Re: Fast Frustum Culling   -   Joseph J. Strout
   Re: Fast Frustum Culling   -   Lo Saeteurn
   Re: Fast Frustum Culling   -   Joseph J. Strout
   Re: Fast Frustum Culling   -   Jeff Quan
   Re: Fast Frustum Culling   -   Lo Saeteurn
   Re: Fast Frustum Culling   -   Lo Saeteurn
   Re: Fast Frustum Culling   -   Lo Saeteurn

Fast Frustum Culling
Date: 02.06.05 16:02 (Thu, 2 Jun 2005 08:02:49 -0700)
From: Lo Saeteurn
I'm developing my own frustum culling that would be more efficient
than RB's slow inview test.

This has stumped me for the past two days now. It seems to cull
objects off much earlier than it is suppose to (objects that are not
quite out of view yet). Everything seems correct. Maybe it is my
field of view test ( d=(90-(View.FieldOfView*0.5))*(kPi/180) )?
Please help!!

The field of view is about 50 or 60.

Here is the code:
dim d as Double

//Cull FOV left/right
d = (90-(View.FieldOfView*0.5))*(kPi/180)//amount to yaw/pitch
base on FOV
View.Camera.Yaw -d
CullObjectsBehindRemovable mObjectsUnculled,View//left plane
View.Camera.Yaw d*2
CullObjectsBehindRemovable mObjectsUnculled,View//right plane
View.Camera.Yaw -d
//Cull FOV top/bottom
View.Camera.Pitch -d
CullObjectsBehindRemovable mObjectsUnculled,View//bottom plane
View.Camera.Pitch d*2
CullObjectsBehindRemovable mObjectsUnculled,View//top plane
View.Camera.Pitch -d

//Cull View distance
View.Camera.MoveForward View.Yon
View.Camera.Yaw kPi
CullObjectsBehindRemovable mObjectsUnculled,View
View.Camera.Yaw -kPi
View.Camera.MoveForward -View.Yon

Sub CullObjectsBehindRemovable(grp As Group3D, view As Rb3DSpace)

// Set any elements within the group which are not in our
// view frustum to invisible. Return the number of objects
// which are in the view.

Dim i As Integer
Dim obj As Object3D, obj2 as Object3DCullable, obj3 as
Group3DCullable
Dim r,z as Double

count = 1
if grp <> NIL then
for i = grp.Count-1 DownTo 0
obj = grp.Item(i)
if obj <> NIL and obj.Visible then

if obj IsA Object3DCullable then
obj2 = Object3DCullable(obj)
r = obj2.mRadius'*2
z = view.Camera.WorldToObjectTransform(obj2.Position.Plus
(obj2.mCenterPosition)).z
elseif obj IsA Group3DCullable then
obj3 = Group3DCullable(obj)
r = obj3.mRadius'*2
z=view.Camera.WorldToObjectTransform(obj3.Position.Plus
(obj3.mCenterPosition)).z
else
r z end if

if not(z+r > r or abs(z) < r) then
obj.Visible = false
grp.Remove i
end if

end if
next
end if

End Sub

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

Re: Fast Frustum Culling
Date: 02.06.05 17:32 (Thu, 2 Jun 2005 11:32:55 -0500)
From: Joseph J. Strout
At 8:02 AM -0700 6/2/05, Lo Saeteurn wrote:

>I'm developing my own frustum culling that would be more efficient
>than RB's slow inview test.

That'd be impressive -- InView is actually quite efficient. Why do
you believe it is slow?

Of course, you can speed things up substantially by testing fewer
objects (e.g., grouping your objects and testing whole groups at
once). But that's true regardless of how the actual in-frustum test
is done.

Best,
- Joe

Re: Fast Frustum Culling
Date: 02.06.05 17:50 (Thu, 2 Jun 2005 09:50:33 -0700)
From: Lo Saeteurn
Really? Well it seems to be really slow with my tests? Maybe it is
the bounds that are slow? There is something that is really slowing
down the game in your Frustum Culling code (it doesn't happen in
Renegades). Testing just a hundred objects really slows down the game
(or even 50).

So you are saying that the problem is not with the InView test, but
rather something else? Is calculating the bounds of objects slow?
Based on my performace tests, it seems that the slowness of the
processing is from the FrustumCulling method? Any ideas what I may be
doing that would cause this method to be extremely slow?

> That'd be impressive -- InView is actually quite efficient. Why do
> you believe it is slow?
>
> Of course, you can speed things up substantially by testing fewer
> objects (e.g., grouping your objects and testing whole groups at
> once). But that's true regardless of how the actual in-frustum
> test is done.

Yes, sort of what I'm doing, but grouping objects that move seems
like it may slow down the game.

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

Re: Fast Frustum Culling
Date: 02.06.05 19:04 (Thu, 2 Jun 2005 13:04:49 -0500)
From: Joseph J. Strout
At 9:50 AM -0700 6/2/05, Lo Saeteurn wrote:

>Really? Well it seems to be really slow with my tests? Maybe it is
>the bounds that are slow? There is something that is really slowing
>down the game in your Frustum Culling code (it doesn't happen in
>Renegades). Testing just a hundred objects really slows down the
>game (or even 50).

What Frustum Culling code are you referring to?

>So you are saying that the problem is not with the InView test, but
>rather something else? Is calculating the bounds of objects slow?

Yes, that's a pretty expensive operation. If you have a lot of
objects, you may need to put some work into not recomputing or
updating their bounds more than absolutely necessary.

> Based on my performace tests, it seems that the slowness of the
>processing is from the FrustumCulling method? Any ideas what I may
>be doing that would cause this method to be extremely slow?

Not offhand, unless you're computing the bounds of each object within
this method.

>Yes, sort of what I'm doing, but grouping objects that move seems
>like it may slow down the game.

True. Such grouping is mainly useful for static geometry, like the
scenery objects.

Best,
- Joe

Re: Fast Frustum Culling
Date: 02.06.05 19:22 (Thu, 2 Jun 2005 11:22:08 -0700)
From: Lo Saeteurn
>> Really? Well it seems to be really slow with my tests? Maybe it is
>> the bounds that are slow? There is something that is really
>> slowing down the game in your Frustum Culling code (it doesn't
>> happen in Renegades). Testing just a hundred objects really slows
>> down the game (or even 50).
>>
> What Frustum Culling code are you referring to?

CullByFrustrum

> Yes, that's a pretty expensive operation. If you have a lot of
> objects, you may need to put some work into not recomputing or
> updating their bounds more than absolutely necessary.

Well it only recomputes them at loading and updates them when they
move. The only thing in my test map that is moving is the player.

>> Based on my performace tests, it seems that the slowness of the
>> processing is from the FrustumCulling method? Any ideas what I may
>> be doing that would cause this method to be extremely slow?
>>
> Not offhand, unless you're computing the bounds of each object
> within this method.

Well I'm really confused with what is causing this slow down. I guess
I'll have to recheck my code if there is a bug somewhere that is
causing this; maybe somehow forcing objects to constantly update
their bounds. When I disable this CullByFrustrum checking, the
processing of the world is about 100-200 FPS.

I get a Render FPS of 40-80 FPS. The processing of the Frustum
culling alone is about 20-30 FPS without my other optimizations which
slows my final game frame rate to about 15-25 FPS in a smaller sized
map and 6-10 FPS in a very large map. All on a 800mhz iBook G4.

It runs without a problem on a faster 2Ghz G5 processor, but I want
lower end machines to at least process the game if their graphic card
cannot handle it.

>> Yes, sort of what I'm doing, but grouping objects that move seems
>> like it may slow down the game.
>>
> True. Such grouping is mainly useful for static geometry, like the
> scenery objects.

In most most causes, there won't be much objects moving. I'm
basically dividing the world into 25x25 (meters) tiles so in this
case, it would only have to update objects when they move out of the
20x20 tile.

I'll be fine if I use my own frustum culling code. Did you spot
anything wrong with it? I suspect my FOV calculation, but I really
don't see anything wrong with it. Maybe I'm just miscalculating the
Bounds of all the objects?

dim d as Double

//Cull FOV left/right
d = (90-(View.FieldOfView*0.5))*(kPi/180)//amount to yaw/pitch
base on FOV
View.Camera.Yaw -d
CullObjectsBehindRemovable mObjectsUnculled,View//left plane
View.Camera.Yaw d*2
CullObjectsBehindRemovable mObjectsUnculled,View//right plane
View.Camera.Yaw -d
//Cull FOV top/bottom
View.Camera.Pitch -d
CullObjectsBehindRemovable mObjectsUnculled,View//bottom plane
View.Camera.Pitch d*2
CullObjectsBehindRemovable mObjectsUnculled,View//top plane
View.Camera.Pitch -d

//Cull View distance
View.Camera.MoveForward View.Yon
View.Camera.Yaw kPi
CullObjectsBehindRemovable mObjectsUnculled,View
View.Camera.Yaw -kPi
View.Camera.MoveForward -View.Yon

Sub CullObjectsBehindRemovable(grp As Group3D, view As Rb3DSpace)

// Set any elements within the group which are not in our
// view frustum to invisible. Return the number of objects
// which are in the view.

Dim i As Integer
Dim obj As Object3D, obj2 as Object3DCullable, obj3 as
Group3DCullable
Dim r,z as Double

count = 1
if grp <> NIL then
for i = grp.Count-1 DownTo 0
obj = grp.Item(i)
if obj <> NIL and obj.Visible then

if obj IsA Object3DCullable then
obj2 = Object3DCullable(obj)
r = obj2.mRadius'*2
z = view.Camera.WorldToObjectTransform(obj2.Position.Plus
(obj2.mCenterPosition)).z
elseif obj IsA Group3DCullable then
obj3 = Group3DCullable(obj)
r = obj3.mRadius'*2
z=view.Camera.WorldToObjectTransform(obj3.Position.Plus
(obj3.mCenterPosition)).z
else
r z end if

if not(z+r > r or abs(z) < r) then
obj.Visible = false
grp.Remove i
end if

end if
next
end if

End Sub

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

Re: Fast Frustum Culling
Date: 02.06.05 20:10 (Thu, 2 Jun 2005 14:10:26 -0500)
From: Joseph J. Strout
At 11:22 AM -0700 6/2/05, Lo Saeteurn wrote:

>>What Frustum Culling code are you referring to?
>
>CullByFrustrum

But you say it's not slow in Renegades -- this is the same code
Renegades is using, isn't it? So I'm confused.

>>Yes, that's a pretty expensive operation. If you have a lot of
>>objects, you may need to put some work into not recomputing or
>>updating their bounds more than absolutely necessary.
>
>Well it only recomputes them at loading and updates them when they
>move. The only thing in my test map that is moving is the player.

And you're storing these where? If they're stored in the
Object3D.Bounds property, then they're updated every frame.

>Well I'm really confused with what is causing this slow down. I
>guess I'll have to recheck my code if there is a bug somewhere that
>is causing this; maybe somehow forcing objects to constantly update
>their bounds. When I disable this CullByFrustrum checking, the
>processing of the world is about 100-200 FPS.

I didn't mean that the frustum check wasn't slow (if you call it on a
whole bunch of objects); just that it is about as fast as it can be.
I doubt you could write something in RB that would be any faster.

But in a large world with lots of objects, you still have to go to
some effort to avoid testing objects more than you really need to. A
great deal of the code in commercial game engines is dedicated to
exactly that, using things like octtrees and so forth that allow the
code to reject large groups of objects at once.

In Renegades, I chose an occlusion approach, where I define large
occlusion planes and test the geometry against that. Though at the
moment, I can't remember if I do this before or after frustum culling.

>I'll be fine if I use my own frustum culling code.

Why do you think so?

> Did you spot anything wrong with it?

No, I didn't study it. But since it's giving incorrect results,
there must be something wrong somewhere.

Best,
- Joe

Re: Fast Frustum Culling
Date: 02.06.05 21:23 (Thu, 2 Jun 2005 13:23:03 -0700)
From: Lo Saeteurn
> But you say it's not slow in Renegades -- this is the same code
> Renegades is using, isn't it? So I'm confused.

Exact same method in the CullingUtil Module.

>> Well it only recomputes them at loading and updates them when they
>> move. The only thing in my test map that is moving is the player.
>>
> And you're storing these where? If they're stored in the
> Object3D.Bounds property, then they're updated every frame.

In the mBounds property. The slow down is not when it renders
(mainview.update), but when calling the FrustumCull (or
OcclusionalCull) methods. The render rate is great 40-70 FPS.

> I didn't mean that the frustum check wasn't slow (if you call it on
> a whole bunch of objects); just that it is about as fast as it can
> be. I doubt you could write something in RB that would be any faster.

That method is maybe called on for maybe a maximum of 100 objects for
a very large world.

> But in a large world with lots of objects, you still have to go to
> some effort to avoid testing objects more than you really need to.
> A great deal of the code in commercial game engines is dedicated to
> exactly that, using things like octtrees and so forth that allow
> the code to reject large groups of objects at once.

Yes. My tiling technique takes out much of the world. It also uses a
quick frustum tests as well. The tiles are much cheaper to process.
In your huge and complex Level 4 map, there are on average 400-500
GameObjs to test depending on the size of the tile. A 10x10 tile
would have only 250-300 GameObjs. A 1x1 tile would have about 100-300
GameObjs. Smaller tiles would mean more tiles to check.

Is 400-500 objects too much to check? If it is, maybe I should
organize the world with a quad tree or maybe the OctTree?

> In Renegades, I chose an occlusion approach, where I define large
> occlusion planes and test the geometry against that. Though at the
> moment, I can't remember if I do this before or after frustum culling.

I recall after the frustum--at least that's how it is in my code.

>> I'll be fine if I use my own frustum culling code.
>>
> Why do you think so?

Because my FrustumCulling method is almost instant, but it is not as
accurate as a test as the InView test (CullByFrustrum). I use that in
the end to test the remaining objects that are questionable. It runs
over 400 FPS alone without the CullByFrustrum test.

The remaining objects that the CullByFrustrum method tests are
anywhere from 0 to 50 with a mean of about 15. This way it runs
fairly decent, but my culling takes out objects before it is
completely out of view. I'm not sure where my miscalculation occurred.

>> Did you spot anything wrong with it?
>
> No, I didn't study it. But since it's giving incorrect results,
> there must be something wrong somewhere.

Yes I've been searching and I not as familiar with the underlying
engine as you. I must have messed up the engine somewhere.

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

Re: Fast Frustum Culling
Date: 02.06.05 22:02 (Thu, 2 Jun 2005 16:02:52 -0500)
From: Joseph J. Strout
At 1:23 PM -0700 6/2/05, Lo Saeteurn wrote:

>>But you say it's not slow in Renegades -- this is the same code
>>Renegades is using, isn't it? So I'm confused.
>
>Exact same method in the CullingUtil Module.

So I remain confused. Why then do you think it's slow in your app,
but not in Renegades?

>>And you're storing these where? If they're stored in the
>>Object3D.Bounds property, then they're updated every frame.
>
>In the mBounds property.

Well, that's the right thing. I wonder if there is some bug causing
these to update more than they should. Have you tried running Shark
on your app?

>Is 400-500 objects too much to check? If it is, maybe I should
>organize the world with a quad tree or maybe the OctTree?

It's a lot. Hard to say exactly what "too much" would be. But note
that in Renegades on Level 4, we had to get creative and do some
additional higher-level culling to get the performance we want; the
level is divided into five groups, and we turn off all but one or two
of them at any given time.

>>>I'll be fine if I use my own frustum culling code.
>>
>>Why do you think so?
>
>Because my FrustumCulling method is almost instant, but it is not as
>accurate as a test as the InView test (CullByFrustrum).

InView isn't perfectly accurate either; it returns true for any
object that *might* be in view. It's doing very simple tests and is
very quick; I don't see how your method could be significantly
faster, if it produces a similar answer. Of course, I guess it's
always possible that there's a bug in our code that makes it take
longer than it should... but it's pretty simple code.

Best,
- Joe

Re: Fast Frustum Culling
Date: 02.06.05 22:39 (Thu, 2 Jun 2005 14:39:51 -0700)
From: Jeff Quan
On Jun 2, 2005, at 2:02 PM, Joseph J. Strout wrote:
>
>> Is 400-500 objects too much to check? If it is, maybe I should
>> organize the world with a quad tree or maybe the OctTree?
>
> It's a lot. Hard to say exactly what "too much" would be. But note
> that in Renegades on Level 4, we had to get creative and do some
> additional higher-level culling to get the performance we want; the
> level is divided into five groups, and we turn off all but one or two
> of them at any given time.

Hmm. I think Joe just pointed out a very important part of culling from
Renegades.

One big trick that Renegades uses is that whole groups of objects are
culled out via the hidegrp/showgrp commands. These are strategically
embedded in navnodes around each level in order to control the
visibility of entire regions. Each region is hand-grouped into rough
rooms, so if, say, you're in the kitchen and walk over a "hidegrp
living room" navnode, then all objects contained in the "living room"
group are hidden regardless if they're InView or not.

Of course, there needs to be a corresponding navnode tagged "showgrp
living room" so that the player doesn't suddenly see empty space where
the living room used to be. It's up to the level editor to make sure
areas are correctly set to be hidden/unhidden at the appropriate
moment.

If you run around Renegades enough, you might see these as hiccups in
the game engine while the game hides/unhides whole groups. It's because
of such commands that Renegades' level 4 runs as well as it does.

As a benchmark for you, I tried to keep the number of visible objects
to well below 200.

=Jeff Quan
<email address removed>

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

Re: Fast Frustum Culling
Date: 03.06.05 00:10 (Thu, 2 Jun 2005 16:10:53 -0700)
From: Lo Saeteurn
> Hmm. I think Joe just pointed out a very important part of culling
> from Renegades.
>
> One big trick that Renegades uses is that whole groups of objects
> are culled out via the hidegrp/showgrp commands. These are
> strategically embedded in navnodes around each level in order to
> control the visibility of entire regions. Each region is hand-
> grouped into rough rooms, so if, say, you're in the kitchen and
> walk over a "hidegrp living room" navnode, then all objects
> contained in the "living room" group are hidden regardless if
> they're InView or not.
>
> Of course, there needs to be a corresponding navnode tagged
> "showgrp living room" so that the player doesn't suddenly see
> empty space where the living room used to be. It's up to the level
> editor to make sure areas are correctly set to be hidden/unhidden
> at the appropriate moment.
>
> If you run around Renegades enough, you might see these as hiccups
> in the game engine while the game hides/unhides whole groups. It's
> because of such commands that Renegades' level 4 runs as well as it
> does.

Yes, I noticed this, but sadly I had to remove it to support more
than one player.

> As a benchmark for you, I tried to keep the number of visible
> objects to well below 200.

Thanks for the advise.

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

Re: Fast Frustum Culling
Date: 03.06.05 00:13 (Thu, 2 Jun 2005 16:13:31 -0700)
From: Lo Saeteurn
> So I remain confused. Why then do you think it's slow in your app,
> but not in Renegades?

Well I made a lot of changes. Renegades was made to support only a
single player

> Well, that's the right thing. I wonder if there is some bug
> causing these to update more than they should.

There can't be because if I disable the cullbyfrustum line in my
code, it runs much faster in terms of the processing, but just a bit
slower in the rendering. The .update method is very fast (40-70 FPS
in other levels, 24-60 FPS in level 4).

> Have you tried running Shark on your app?

No. I will try that.

> It's a lot. Hard to say exactly what "too much" would be. But
> note that in Renegades on Level 4, we had to get creative and do
> some additional higher-level culling to get the performance we
> want; the level is divided into five groups, and we turn off all
> but one or two of them at any given time.

Yes, I realized that. I had to remove this functionality in my game
engine to support multiple players. Instead I just broke the world
down into tiles and which ever tiles all the players are near (to the
yon distance) would be the only ones to be processed for AI
(including navNodes and occluders). All the tiles within your
player's view and in front would be the only ones to be checked for
culling and rendering.

Maybe this was a bad choice for me?

>> Because my FrustumCulling method is almost instant, but it is not
>> as accurate as a test as the InView test (CullByFrustrum).
>>
> InView isn't perfectly accurate either; it returns true for any
> object that *might* be in view. It's doing very simple tests and
> is very quick; I don't see how your method could be significantly
> faster, if it produces a similar answer. Of course, I guess it's
> always possible that there's a bug in our code that makes it take
> longer than it should... but it's pretty simple code.

Maybe because my method doesn't use a bounds3D to test the view, but
rather a preprocessed radius and a preprocessed central position?
It's basically testing if the objects are within the 6 planes of the
camera. Maybe my bounds calculations are off. I'm not sure if the
inview test does exactly the same thing.

Is this not how you should check if an object is behind you?

z=view.Camera.WorldToObjectTransform(obj.bounds.center).z
radius= obj.bounds.radius

if not(z+radius> radius or abs(z) < radius) then
obj.Visible = false
end if

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>

Re: Fast Frustum Culling
Date: 03.06.05 02:22 (Thu, 2 Jun 2005 18:22:39 -0700)
From: Lo Saeteurn
Thanks for all your help. I guess I'll try dividing my world using
the Octree algorithm. Hopefully I can see some performance boosts.

On Jun 2, 2005, at 2:02 PM, Joseph J. Strout wrote:

> At 1:23 PM -0700 6/2/05, Lo Saeteurn wrote:
>
>>> But you say it's not slow in Renegades -- this is the same code
>>> Renegades is using, isn't it? So I'm confused.
>>>
>> Exact same method in the CullingUtil Module.
>>
> So I remain confused. Why then do you think it's slow in your app,
> but not in Renegades?
>
>>> And you're storing these where? If they're stored in the
>>> Object3D.Bounds property, then they're updated every frame.
>>>
>> In the mBounds property.
>>
> Well, that's the right thing. I wonder if there is some bug
> causing these to update more than they should. Have you tried
> running Shark on your app?
>
>> Is 400-500 objects too much to check? If it is, maybe I should
>> organize the world with a quad tree or maybe the OctTree?
>>
> It's a lot. Hard to say exactly what "too much" would be. But
> note that in Renegades on Level 4, we had to get creative and do
> some additional higher-level culling to get the performance we
> want; the level is divided into five groups, and we turn off all
> but one or two of them at any given time.
>
>>>> I'll be fine if I use my own frustum culling code.
>>>>
>>>
>>> Why do you think so?
>>>
>> Because my FrustumCulling method is almost instant, but it is not
>> as accurate as a test as the InView test (CullByFrustrum).
>>
> InView isn't perfectly accurate either; it returns true for any
> object that *might* be in view. It's doing very simple tests and
> is very quick; I don't see how your method could be significantly
> faster, if it produces a similar answer. Of course, I guess it's
> always possible that there's a bug in our code that makes it take
> longer than it should... but it's pretty simple code.
>
> Best,
> - Joe
>
> --
> Joe Strout REAL Software, Inc.
>
> Vote for REALbasic (twice!) in the LinuxWorld Reader's Choice Awards:
> http://linux.sys-con.com/general/readerschoice.htm
> _______________________________________________
> Unsubscribe or switch delivery mode:
> <http://www.realsoftware.com/support/listmanager/>
> Search the archives of this list here:
> <http://support.realsoftware.com/listarchives/lists.html>

_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>

Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>