I've found it !
Using the TaskColorsBinding property, it works well... Sad it doesn't work by design, only by code : I lost about an hour on this ! ^^
Now I've got another question around this problem : using the TaskBarBackgroundTemplate, I display the WorkingHoursBar in my Gantt. Is it possible to bind the WorkingHoursBar.TaskBrush to something using the result of the CustomColorProvider defined in TaskColorsBinding ? A TemplateBinding may, or something else ?
Because currently, I bind the same CustomColorProvider to TaskColorsBinding and WWorkingHoursBar.TaskBrush, so the work is done twice for each task...