OpenLayers is large by default, but can be made more trim. Prune away the vector and raster formats you won't need, and you can get it to a more reasonable size. It's also not true that OpenLayers must pull image tiles from a WMS or TileCache instance. It's my understanding that the EveryBlock project has written a custom image layer that fetches tiles from an HTTP server or proxy, which is what Kapil is looking for. The big imagery projects on my radar are going to involve collaborative annotation, requiring some easy digitization and drawing tools and RESTful protocols for publishing captured features. OpenLayers provides these tools.