Your engineering team tells you that you can have 2 MB of on-chip cache total. You have a choice of making it one big L1 cache, two level L1 and L2 caches, split I- and D-caches at several levels, etc. What would be your choice and why? Consider hit time, miss rate, associativity, and pipeline structure in making your design decisions. For this problem, it is sufficient if you give qualitative explanation for your design decisions.

What will be an ideal response?

The L1 will be 128 KB. with split I- and D- caches.
I-Cache and D-cache are 64KB each and are two way set associative caches. This type of cache mapping is preferred because neither direct mapped or fully associative cache would perform as well. Because the N-way set associative cache is the middle ground between these two extreme cases, it is a much better choice. However, as the L1 cache is meant to be optimized, the n-factor would have to be small. In addition, due to the search required to determine the correct location of the address, the n-factor must be small. The hit time will be very low due to the smaller cache size; however the miss rate will be higher in comparison to L2 . The I- and D- cache will be separate due to the need of having this level of cache match the processor clock cycle time as closely as possible. This increases the miss rate, but that is required for having the speed desired. The I-cache will be separate also due the pipeline structure. If an instruction is not located in the I-cache, there must be NOPs inserted to effectively pause the processor. This is the same for the D-cache, as if the correct address is not located; the write/load must be paused as well. So due to the high priority of having the processor as fast as possible, the caches are separate.
The L2 would be 1MB with unified I- and D- cache with 8-way associativity.
The higher degree of associativity (8) will lead to a lower miss rate. However, due to the fact that the search will be longer, the hit time will be increased, as per the locations that must be checked must also be increased. The unified I- and D- cache are for lowering the miss rate, which is the primary purpose of the L2 cache. Most decisions for the L2 are made to increasing the hit rate and try to keep the hit time as low as possible.