Remove the WP core colors palette from all core blocks:

• Group block
• Columns block
• Cover block
• Row block
• Query loop block
• Button block
• Paragraph block
• Heading block
• ...

Keep only the P4 color palette.

Can easily be done for all blocks at once via theme.json by adding:

...
  "settings": {
    "color": {
      "defaultPalette": false,

...