We don't have to go all-or-nothing...  The common C code can support the
cache-dirty mechanism, and the callout can do a board-specific solution.
The startup can indicate what support the callout provides and whether
or not there is a good/cheap address to use for the cache-dirtying
solution, and if the startup indicates nothing (i.e. unoptimized
startup, or older startup) the kernel can fall back to a generic
cache-dirtying mechanism that uses a MAP_PHYS.

I think this works, with whatever optimizations we provide for specific
boards, for everything but the hypothetical multi-way cache that does
random way selection.  Is there such a beast?

Doug


-----Original Message-----
From: Adam Mallory [mailto:community-noreply@qnx.com] 
Sent: Friday, November 28, 2008 9:57 AM
To: ostech-core_os
Subject: RE: Thoughts on flushing the dcache more efficiently

> Your understanding of the 'lines' field is correct - in 6.4.0 we did 
> add a 'ways' field (not filled in by all the startups yet) so that 
> procnto could figure out the associativity.

Good to know, thanks.

> Anyhow, I don't think you can do it this way - as Doug pointed out, 
> with a multi-way cache, there's no generic mechanism you can do to 
> ensure that all the ways have been flushed (e.g a cache using random 
> replacement).

From my experience (limited as it may be) that ability tends to be an
option in configuration amongst others such as LRU.  Do we currently
have boards that do random replacement per way and we configure it that
way?

> I think the place for the solution is in the callouts. They know what 
> the replacement policy is and they'd know if there's a fast mechanism 
> to flush/invalidate the full cache. If they see a big request, they 
> can just do their quick thing and return zero - the upper levels treat

> that as an indication that whole cache is done and early out. The 
> downside is, of course, that we have to implement the fix multiple 
> times and in assembler but I think it's the best way of making sure 
> that the cache is actually flushed/invalidated.

Putting aside that I don't agree with the constant writing of code in
assembler and having to repeatedly do the same thing over and over but
slightly different each time (and the inherent bugs that come with doing
it this way).  I think we're limited in what the callouts can do.  For
the sake of argument say I wanted to pollute the cache as I've done in
the kernel (and I know the cache is no random replacement).  What base
address could I use to do this?  The startup writers wouldn't know of a
common area that could be 'abused' so they would be forced to mmap the
memory which is wasteful and it would have to be done via a patcher
routine which is really unfortunate.

I guess if we could provide such an address to the callouts, they could
optionally use it.  I'll have to marinate on that a little longer.

IMHO, all of this startup work presents a barrier.  Is it
insurmountable? No, but I think it's fair to say that given the option
of implementing an optional 'flush all' code path or new callout will
result in startups that likely don't bother since it works either way.
Even if every new startup starts doing the new flush, older and current
startups are unlikely to be reworked and it will be left until a
customer complains about system startup performance of drivers etc.

I guess I agree from a pure technical standpoint that custom code should
belong in the startup arena but the realist and practical side of me
says I hate doing things many times and if it could benefit everyone it
should be done in the OS.

In any case, I'll think about it some more...

> A bit of a side issue, but there is an optimization to the upper
layers
> that could be done that might help in some cases. If it's just a flush

> request (with no invalidation), the code should look to see...

On Fri, Nov 28, 2008 at 09:56:33AM -0500, Adam Mallory wrote:
> From my experience (limited as it may be) that ability tends to be an
> option in configuration amongst others such as LRU.  Do we currently
> have boards that do random replacement per way and we configure it that
> way?

We may never set it up that way, but we can't stop external BSP developers
from doing it - procnto should be able to deal with it.

> > I think the place for the solution is in the callouts. They know what
> > the replacement policy is and they'd know if there's a fast mechanism
> > to flush/invalidate the full cache. If they see a big request, they
> > can just do their quick thing and return zero - the upper levels treat
> > that as an indication that whole cache is done and early out. The
> > downside
> > is, of course, that we have to implement the fix multiple times and in
> > assembler but I think it's the best way of making sure that the cache
> > is actually flushed/invalidated.
> 
> Putting aside that I don't agree with the constant writing of code in
> assembler and having to repeatedly do the same thing over and over but
> slightly different each time (and the inherent bugs that come with doing
> it this way).  I think we're limited in what the callouts can do.  For
> the sake of argument say I wanted to pollute the cache as I've done in
> the kernel (and I know the cache is no random replacement).  What base
> address could I use to do this?  The startup writers wouldn't know of a
> common area that could be 'abused' so they would be forced to mmap the
> memory which is wasteful and it would have to be done via a patcher
> routine which is really unfortunate.

True, but there don't tend to be too many cache callouts written - it tends
to be one per architecture, aside from chip bug workarounds (ARM being
the main exception). That means that we correct the callouts in our
library and everyone will benefit.

> I guess if we could provide such an address to the callouts, they could
> optionally use it.  I'll have to marinate on that a little longer.

You're assuming that the callouts would use the poison method.
They typically wouldn't - they'll have better mechanisms for flushing 
the cache. Even if they do need to do loads, on SH/PPC/MIPS they
already have a one-to-one mapping area that they can use for the load
address. On ARM, there's a special address range just for doing this
kind of thing. On X86, the cache is super smart and we don't have a
callout at all.

> IMHO, all of this startup work presents a barrier.  Is it
> insurmountable? No, but I think it's fair to say that given the option
> of implementing an optional 'flush all' code path or new callout will
> result in startups that likely don't bother since it works either way.
> Even if every new startup starts doing the new flush, older and current
> startups are unlikely to be reworked and it will be left until a
> customer complains about system startup performance of drivers etc.
> 
> I guess I agree from a pure technical standpoint that custom code should
> belong in the startup arena but the realist and practical side of me
> says I hate doing things many times and if it could benefit everyone it
> should be done in the OS.

I basically agree, but this isn't the interrupt callouts where every
board has it's own thing - there's a much more limited set that need
to be dealt with. If it weren't for wacky hardware developers who
did things like writing custom cache controllers in ASIC's we probably
wouldn't even have a cache callout - everything would have been self 
contained in procnto.

	Brian

-- 
Brian Stecher (bstecher@qnx.com)        QNX Software Systems
phone: +1 (613) 591-0931 (voice)        175 Terence Matthews Cr.
       +1 (613) 591-3579 (fax)         ...