By rubble I mean the broken sides of the pool.
As per the excellent advice above, knock some holes in the base and IMO sand would be fine, water it in as you fill then the topsoil.
You may need to top dress with soil again after a few weeks or months, but there won’t be major work required.